Analyzing and modeling complex and big data | Professor Maria Fasli | TEDxUniversityofEssex

Summary

This presentation explores the evolution of computational power, comparing Apollo 11's computers to modern smartphones, highlighting the exponential growth in processing speed and memory. It delves into the concept of Big Data, characterized by volume, velocity, and variety, emphasizing the challenge of extracting knowledge from vast data sets. The speaker discusses the importance of data exploration, visualization, and predictive modeling to gain insights and inform decision-making. Examples from financial markets and social networks illustrate the complexity of modeling agent interactions and the potential for discovering patterns and behaviors within data.

Takeaways

• 🚀 The Apollo 11 computers used a unique two-digit command interface, one for verbs (actions) and one for nouns (data).
• 📱 Modern smartphones like the iPhone 5 are 1,270 times faster and 2 million times more memory-efficient than Apollo 11 computers, reflecting rapid advancements in technology.
• 📈 Moore's Law predicts that the number of transistors on integrated circuits doubles roughly every two years, contributing to the increased computational power we have today.
• 💾 The amount of data being generated is rapidly growing, with over 2.5 billion gigabytes of data created every day.
• 🌐 By 2018, the number of internet users was projected to reach 4 billion, with 21 billion network devices contributing to an explosion in data exchange.
• 📊 Big Data is characterized by the '3 Vs': Volume (the amount of data), Velocity (the speed of data generation), and Variety (the different types of data, both structured and unstructured).
• 🧠 Data alone doesn’t equate to knowledge; data needs to be processed and interpreted to extract meaningful insights.
• 📉 Predictive modeling and data exploration help organizations make informed decisions by identifying patterns in large datasets and validating hypotheses.
• 🤖 Complex models, particularly agent-based systems, can simulate interactions and behaviors that emerge in unpredictable ways, allowing for deeper understanding of complex environments.
• 🛒 Recommender systems and personalization, like those used by Amazon and supermarkets, rely on analyzing patterns in user data to suggest products or services, driven by the principle of homophily (connecting with similar users).

Q & A

What was the role of the Apollo 11 computers during the moon mission?

The Apollo 11 computers were used by astronauts to input commands and perform tasks essential for the mission. Astronauts had to be specially trained to enter commands using a two-digit system: the first digit was the verb (the action to be performed), and the second was the noun (the type of data affected by the action).

How does the computational power of the iPhone 5 compare to the Apollo 11 computers?

The iPhone 5 is 1270 times faster and has 2 million times more processing memory than the Apollo 11 computers. It also has significantly more storage and is about 300 times lighter than the Apollo 11 systems.

What is Moore’s Law, and how does it relate to the advancements in technology?

Moore’s Law suggests that every two years, the number of transistors on integrated circuits doubles. This increase in transistor density has led to the significant advancements in computational power, as seen in modern devices like smartphones compared to earlier technologies.

How much data is produced globally every day, according to the transcript?

It is estimated that every day, we produce about 2.5 billion gigabytes of data. This amounts to millions or even trillions of bytes generated daily.

What are the 'Vs' of Big Data, and how do they characterize the data we produce?

The 'Vs' of Big Data are Volume (the sheer amount of data), Velocity (the speed at which data is produced), and Variety (the different types of data, both structured and unstructured). These factors contribute to the complexity of managing and interpreting Big Data.

Why is data not equivalent to knowledge, according to the speaker?

Data is simply a collection of recorded transactions or events. Information results from processing this data, and only when information is analyzed and understood does it become knowledge. Thus, having Big Data does not automatically lead to 'big knowledge.'

What is the significance of predictive modeling in data analysis?

Predictive modeling allows researchers to hypothesize why certain patterns emerge in the data, build models to test these hypotheses, and use the results to inform decisions. This process can help predict future trends and support decision-making in various fields.

What are agent-based models, and why are they important in understanding complex systems?

Agent-based models use independent software processes (agents) that interact with each other to simulate complex systems. These models are important because they reveal emergent behaviors and patterns that cannot be predicted just by analyzing individual agents, providing deeper insights into complex phenomena.

How do algorithmic traders operate in financial markets, and what are their limitations?

Algorithmic traders use machine learning algorithms to identify patterns in financial data and make decisions (buy/sell). However, they cannot fully understand why patterns emerge or the underlying behaviors of market participants, which limits their predictive capabilities.

How can social networks be modeled, and what principle guides their formation?

Social networks can be modeled by simulating agents that seek out like-minded individuals based on the principle of homophily, where agents connect with others who share similar characteristics. This leads to the formation of clusters within the network as agents interact and identify similarities.