Microsoft Promises a 'Whale' for GPT-5, Anthropic Delves Inside a Model’s Mind and Altman Stumbles

Summary

The video script discusses the rapid advancements in AI models, highlighting Microsoft's investment in AI, OpenAI's challenges, and Google's underreported Gemini models. It emphasizes the exponential growth in AI capabilities, cost-efficiency, and the potential for AI to become ubiquitous. The script also covers Google's adaptive compute approach to enhance reasoning and Anthropic's deep dive into understanding large language models, revealing their inner workings and the ethical considerations of AI control and misuse. Amidst the technical discussions, the video touches on the internal turmoil at OpenAI and the pressing need for serious consideration of AGI implications.

Takeaways

• 🌐 Kevin Scott, CCO of Microsoft, suggests that AI models are far from reaching diminishing returns, indicating exponential growth in computational power applied to AI training since 2012.
• 🐋 Microsoft is developing a 'whale-sized' AI model, presumably GPT 5, emphasizing the significant advancements and capabilities expected from scaling up compute.
• 📉 The cost of using AI models has dramatically decreased; for instance, it's 12 times cheaper and six times faster to use the latest GPT-4 compared to its predecessor.
• 🔍 Google has introduced updates to their Gemini models, with new capabilities and improvements detailed in a comprehensive report that went largely unnoticed.
• 📈 Google's Gemini 1.5 models can process various inputs and have shown significant advancements in quantitative reasoning by allowing extended inference time.
• 🏆 Gemini 1.5 Pro achieved a new record score on a math benchmark, although the benchmark's validity is under some scrutiny.
• 💡 Anthropic, a rival AI lab, has made strides in understanding the inner workings of large language models, shedding light on neuron activations and their meanings.
• 🤖 Anthropic's research indicates that by manipulating specific neuron activations, it's possible to influence the AI's responses, suggesting both potential and risks.
• 🚫 There's ongoing controversy and internal strife at OpenAI, with key figures leaving and expressing concerns about the organization's direction and commitment to safety.
• 🗓️ OpenAI has experienced setbacks, including delays in the release of certain features and a need for greater seriousness regarding the implications of AGI.

Q & A

What did Kevin Scott, the CCO of Microsoft, claim about the diminishing returns of AI models?

Kevin Scott stated that we are not even close to diminishing returns with the power of AI models. Since about 2012, the rate of increase in compute when applied to training has been increasing exponentially, and we are nowhere near the point of diminishing marginal returns on how powerful we can make AI models as we increase the scale of compute.

How has the cost of using GPT-4 compared to its predecessor?

The cost of using GPT-4 has significantly decreased. It is mentioned that it's 12 times cheaper to make a call to GPT-4 than the original GPT model, and it's also six times faster in terms of time to first token response.

What is the significance of the whale-sized supercomputer mentioned in the script?

The whale-sized supercomputer is a metaphor used to describe the scale of the system that Microsoft has deployed to train its AI models. It is said to be much larger than previous systems, akin to the size of a whale compared to a shark or an orca, and capable of building a significant amount of AI capabilities.

What is the potential impact of AI models becoming faster and cheaper?

The potential impact includes the widespread adoption and ubiquity of AI models. As AI continues to get cheaper, it could become pervasive in various devices, from toasters and security cameras to laptops, making artificial intelligence an integral part of everyday technology.

What updates did Google reveal about their Gemini models?

Google revealed powerful new details about their Gemini models, including the ability to accept video, image, and text input up to 1 million tokens, which is significantly more than GPT-4. They also discussed improvements to their Gemini 1.5 Pro and Gemini 1.5 Flash models, which were trained to imitate the output of Gemini 1.5 Pro.

What is the significance of the adaptive compute mentioned in the context of Google's Gemini models?

Adaptive compute refers to allowing models to think for longer periods, which is a promising direction for advancing the intelligence of models. Google aimed to emulate the contemplation that mathematicians often benefit from by training a math-specialized model and providing it with additional inference time, allowing it to explore a wider range of possibilities.

What was the new record score achieved on the math benchmark by Google's specialized model?

The new record score achieved on the math benchmark by Google's specialized model was 91.1%, which was considered impressive enough to be tweeted by Google's CEO, Sunder Pichai.

What is Anthropic and what did they reveal about the inner workings of their large language models?

Anthropic is a rival AGI lab to Google DeepMind and OpenAI. They revealed details about the inner workings of their large language models, including the use of a sparse autoencoder to map out patterns within the activations of neurons. They also discussed the ability to manipulate these activations, such as increasing the code error feature to produce error responses.

What concerns were raised by the departure of Ilya Sutskever from OpenAI?

Ilya Sutskever's departure raised concerns about the direction and safety of AI development at OpenAI. His statement upon leaving expressed confidence in OpenAI's leadership but also hinted at potential issues, such as the fear of losing equity for speaking out against the company's practices.

What controversy surrounded the use of Scarlett Johansson's voice in OpenAI's GPT-4?

The controversy was that OpenAI had used a voice similar to Scarlett Johansson's for their GPT-4 model, which led to accusations of emulation without permission. OpenAI subsequently apologized and removed the voice, pushing back the timeline for the voice mode feature to the coming months.