"Living in the IT Era" refers to the current period in which Information Technology, Slides of Information Technology

Download "Living in the IT Era" refers to the current period in which Information Technology and more Slides Information Technology in PDF only on Docsity! LITE MATTERS 2023 ANGELES, GALVEZ, GAN, REYES, ROXAS, SUMALA, VILLAFUERTE 1 LESSON 4.1 BIG DATA ANALYTICS Data is the fingerprint of creation; and Analytics is the new "Queen of Sciences." Hardly any human activity, business decision, strategy, or physical entity does not produce data or involve data analytics to inform it. As a result, data analytics has become core to our endeavors, from business to medicine, research, management, product development, and all life facets. From a business perspective, data is now viewed as the new gold—and data analytics, the machinery that mines, molds, and mints it. Data analytics is a set of computer-enabled analytics methods, processes, and disciplines of extracting and transforming raw data into meaningful insight, discovery, and knowledge that helps make more effective decisions. Another definition describes it as the discipline of extracting and analyzing data to deliver new insights about past performance and current operations and predict future events. Data analytics is gaining significant prominence not just for improving business outcomes or operational processes; it certainly is the new tool to improve quality, reduce costs and improve customer satisfaction. But it's fast becoming necessary for operational, administrative, and even legal reasons. Since then, data analytics has come a long way and is gaining popularity thanks to the eruption of five new SMAC technologies: social media, mobility, analytics, and cloud computing. You might add another for sensors and the internet of things (IoT). Each technology is significant in transforming the business and the data they generate. What is Data? Data (plural, data: datum), as defined by Merriam-Webster Dictionary, refers to the following: factual information (such as measurements or statistics) used as a basis for reasoning, discussion, or calculation, information in digital form that can be transmitted or processed, or information output by a sensing device or organ that includes both useful and irrelevant or redundant information and must be processed to be meaningful. In computing, data is information translated into a form efficient for movement or processing. Relative to today's computers and transmission media, data is information converted into binary digital form. It is acceptable for data to be used as a singular subject or a plural subject. Raw data is a term used to describe data in its most basic digital format. The concept of data in the context of computing has its roots in the work of Claude Shannon, an American mathematician known as the father of information theory. He ushered in binary digital concepts by applying two-value Boolean logic to electronic circuits. Binary digit formats underlie the CPUs, semiconductor memories, disk drives, and many peripheral LITE MATTERS 2023 ANGELES, GALVEZ, GAN, REYES, ROXAS, SUMALA, VILLAFUERTE 2 devices standard in computing today. Early computer input for control and data took punch cards, magnetic tape, and hard disk. Data's importance in business computing became apparent early on by the popularity of the terms "data processing" and "electronic data processing," which, for a time, came to encompass the whole gamut of what is now known as information technology. Over the history of corporate computing, specialization occurred, and a distinct data profession emerged along with the growth of corporate data processing. Recall: How is data stored? Computers represent data, including video, images, sounds, and text, as binary values using patterns of just two numbers: 1 and 0. A bit is the smallest data unit and represents just a single value. Usually, storage and memory are measured in megabytes and gigabytes. The units of data measurement continue to grow as the amount of data collected and stored grows. For example, the relatively new term "brontobyte" is data storage equal to 10 to the 27th power of bytes. Data can be stored in file formats, as in mainframe systems using ISAM and VSAM. Other file formats for data storage, conversion, and processing include comma-separated values. These formats continued to find uses across various machine types, even as more structured-data-oriented approaches gained footing in corporate computing. Greater specialization developed as databases, database management systems, and relational database technology arose to organize information. UNIT VALUE 1 byte 8 bits (binary digits) 1 kilobyte 1024 bytes 1 megabyte 1024 kilobytes 1 gigabyte 1024 megabytes 1 terabyte 1024 gigabytes 1 petabyte 1024 terabytes 1 exabyte 1024 petabytes 1 zettabyte 1024 exabytes 1 yottabyte 1024 zettabytes 1 brontobyte 1024 yottabytes Table 1 – Common Data Storage Measurements What is Analytics? Analytics is a broad term that encompasses the processes, technologies, frameworks, and algorithms to extract meaningful insights from data. Raw data does not have a meaning until it is contextualized and processed into useful information. Analytics is the process of extracting and creating information from raw data by filtering, processing, categorizing, LITE MATTERS 2023 ANGELES, GALVEZ, GAN, REYES, ROXAS, SUMALA, VILLAFUERTE 5 and the best course of action for each outcome. Prescriptive analytics aims to answer the question - What can we do to make it happen? Prescriptive analytics can predict possible outcomes based on the current choice of actions. Therefore, we can consider prescriptive analytics as a type of analytics that uses different prediction models for different inputs. Prescriptive analytics prescribes actions or the best option from the available options. The examples that illustrate the uses of predictive analytics are the following: ▪ prescribe the best medicine for treating a patient based on the outcomes of various medications for similar patients ▪ to suggest the best mobile data plan for a customer based on the customer's browsing patterns. What is Big Data? Big data is a collection of datasets whose volume, velocity, or variety is so large that storing, managing, processing, and analyzing the data using traditional databases and data processing tools is complex. In recent years, there has been an exponential growth in structured and unstructured data generated by information technology, industrial, healthcare, the Internet of Things, and other systems. According to an estimate by IBM, 2.5 quintillion bytes of data are created every day. The estimated volume of data created worldwide in 2022, according to Statista, is 97 zettabytes, compared to the 79 zettabytes of data generated in 2021. In 2025 the amount generated in 2021 is expected to double. Of all of the data in the world at the moment, approximately 90% of it is replicated, with only 10% being genuine, new data. Based on a report by DOMO, these things happen on the web in just 60 seconds to see the volume and speed at which we create data online. • 5.9 million Google searches happen. • Instagram users share 66,000 photos. • Facebook users post 1.7 million pieces of content. • People send 231.4 million emails. • YouTubers upload 500 hours of videos. • Snapchat users send 4.3 million snaps. • Twitter users write 347,200 tweets. • People send 16 million texts. • Venmo users transfer $437,600. • Amazon shoppers spend $443,000. Big Data can power the next generation of smart applications that will leverage the power of the data to make the applications intelligent. Big data applications span a wide range of web, retail and marketing, banking and financial, industrial, healthcare, environmental, Internet of Things, and cyber-physical systems. LITE MATTERS 2023 ANGELES, GALVEZ, GAN, REYES, ROXAS, SUMALA, VILLAFUERTE 6 Big data analytics involves collecting, storing, processing, and analyzing this massive-scale data. Specialized tools and frameworks are required for big data analysis when: 1. the volume of data involved is so large that it is difficult to store, process, and analyze data on a single machine, 2. the velocity of data is very high, and the data needs to be analyzed in real-time, 3. there is a variety of data involved, which can be structured, unstructured, or semi- structured, and is collected from multiple data sources, 4. various types of analytics need to be performed to extract value from the data, such as descriptive, diagnostic, predictive, and prescriptive analytics. Big data tools and frameworks have distributed and parallel processing architectures and can leverage the storage and computational resources of a large cluster of machines. Some examples of big data are listed as follows: ▪ Data generated by social networks, including text, images, audio, and video data ▪ Clickstream data generated by web applications such as e-Commerce to analyze user behavior ▪ Machine sensor data collected from sensors embedded in industrial and energy systems for monitoring their health and detecting failures ▪ Healthcare data collected in electronic health record (EHR) systems ▪ Logs generated by web applications ▪ Stock markets data ▪ Transactional data generated by banking and financial applications Types of Big Data Big data can come in multiple forms, including structured and non-structured data such as financial, text, multimedia, and genetic mappings. Contrary to much traditional data analysis organizations perform, most Big Data is unstructured or semi-structured, requiring different techniques and tools to process and analyze. Distributed computing environments and massively parallel processing (MPP) architectures enabling parallelized data to ingest and analyzed are the preferred approaches to processing such complex data. LITE MATTERS 2023 ANGELES, GALVEZ, GAN, REYES, ROXAS, SUMALA, VILLAFUERTE 7 1. Structured Data Any data stored, accessed, and processed in a fixed format is termed 'structured' data. Over time, talent in computer science has achieved tremendous success in developing techniques for working with such kinds of data (where the format is well known in advance) and deriving value from it. However, nowadays, we are foreseeing issues when such data grows to a vast extent; typical sizes are in the range of multiple zettabytes. Data containing a defined data type, format, and structure (transaction data, online analytical processing [OLAP] data cubes, traditional RDBMS, CSV files, and even simple spreadsheets) is an example of structured data. 2. Semi-structured Data Semi-structured data can contain both forms of data. We can see semi-structured data as a structured form, but it is not defined with, e.g., a table definition in relational DBMS. An example of semi-structured data is data represented in an XML file. LITE MATTERS 2023 ANGELES, GALVEZ, GAN, REYES, ROXAS, SUMALA, VILLAFUERTE 10 Variety refers to the forms of the data. Big data comes in different forms, such as structured, unstructured, or semi-structured, including text data, image, audio, video, and sensor data. Big data systems must be flexible enough to handle such a variety of data. 4. Veracity Veracity refers to how accurate the data is. The data needs to be cleaned to remove noise to extract the value. Data-driven applications can reap the benefits of big data only when the data is meaningful and accurate. Therefore, data cleansing is important so incorrect and faulty data can be filtered out. 5. Value The value of data refers to the usefulness of data for its intended purpose. It is related to the veracity or accuracy of the data. The end goal of any big data analytics system is to extract value from the data. For some applications, value also depends on how fast we can process the data. Domain-Specific Examples of Big Data Big data applications span a wide range of domains, including (but not limited to) homes, cities, environment, energy systems, retail, logistics, industry, agriculture, Internet of Things, and healthcare. 1. Web a. Web Analytics b. Performance Monitoring c. Ad Targeting and Analytics d. Content Recommendation 2. Financial a. Credit Risk Modeling b. Fraud Detection 3. Healthcare a. Epidemiological Surveillance b. Patient Similarity-based Decision Intelligence Application c. Adverse Drug Events Prediction d. Detecting Claim Anomalies e. Evidence-based Medicine f. Real-time health monitoring 4. Internet of Things a. Intrusion Detection b. Smart Parking c. Smart Roads d. Structural Health Monitoring e. Smart Irrigation 5. Environment a. Weather Monitoring LITE MATTERS 2023 ANGELES, GALVEZ, GAN, REYES, ROXAS, SUMALA, VILLAFUERTE 11 b. Air Pollution Monitoring c. Noise Pollution Monitoring d. Forest Fire Detection e. River Floods Detection f. Water Quality Monitoring 6. Logistics and Transportation a. Real-time Fleet Tracking b. Shipment Monitoring c. Remote Vehicle Diagnostics d. Route Generation and Scheduling e. Hyper-local Delivery f. Cab/Taxi Aggregators 7. Industry a. Machine Diagnosis and Prognosis b. Risk Analysis of Industrial Operations c. Production Planning and Control 8. Retail a. Inventory Management b. Customer Recommendation c. Store Layout Optimization d. Forecasting Demand How big data analytics works? Big data analytics involves collecting, processing, cleaning, and analyzing large datasets to help organizations operationalize their big data. 1. Collect data. Data collection looks different for every organization. With today's technology, organizations can gather structured and unstructured data from various sources — from cloud storage to mobile applications to in-store IoT sensors. Some data will be stored in data warehouses where business intelligence tools and solutions can access it easily. Raw or unstructured data that is too diverse or complex for a warehouse may be assigned metadata and stored in a data lake. 2. Process data. Once data is collected and stored, it must be appropriately organized to get accurate analytical queries, especially when it's large and unstructured. In addition, available data is growing exponentially, making data processing challenging for organizations. One processing option is batch processing, which looks at large data blocks over time. Batch processing is useful when there is a longer turnaround time between collecting and analyzing data. LITE MATTERS 2023 ANGELES, GALVEZ, GAN, REYES, ROXAS, SUMALA, VILLAFUERTE 12 Stream processing looks at small batches of data at once, shortening the delay time between collection and analysis for quicker decision-making. However, stream processing is more complex and often more expensive. 3. Clean data. Data, big or small, requires scrubbing to improve data quality and get more robust results; all data must be formatted correctly. Any redundant or irrelevant data must be eliminated or accounted for. Dirty data can obscure and mislead, creating flawed insights. 4. Analyze data. Getting big data into a usable state takes time. However, advanced analytics processes can turn big data into big insights once it's ready. Some of these big data analysis methods include: (1) Data mining sorts through large datasets to identify patterns and relationships by identifying anomalies and creating data clusters. (2) Predictive analytics uses an organization's historical data to make predictions, identifying upcoming risks and opportunities. (3) Deep learning imitates human learning patterns using artificial intelligence and machine learning to layer algorithms and finds patterns in the most complex and abstract data. Benefits of (Big) Data Analytics 1. Decision-making improves. Companies may use the information they obtain from data analytics to guide their decisions, leading to improved results. Data analytics removes much guesswork from preparing marketing plans, deciding what material to make, creating goods, and more. With advanced data analytics technologies, new data can be constantly gathered and analyzed to enhance your understanding of changing circumstances. 2. Marketing becomes more effective. When businesses understand their customers better, they can sell to them more efficiently. Data analytics also gives businesses invaluable insights into how their marketing campaigns work so that they can fine-tune them for better results. 3. Customer service improves. Data analytics provides businesses with deeper insight into their clients, helping them to customize customer experience to their needs, offer more customization, and create better relationships with them. 4. The efficiency of operations increases.