"Compute is the New Oil", Leaving Google, Founding Groq, Agents, Bias/Control (Jonathan Ross)

Jonathan Ross, founder and CEO of Groq, discusses his journey from Google to founding his own company. Ross, who was instrumental in developing Google's Tensor Processing Unit (TPU), explains the constraints he felt within a large corporation and the freedom he found in starting his own venture. He highlights Groq's unique approach to chip design, focusing on efficiency and avoiding memory bottlenecks, which has resulted in impressive inference speeds. Ross also touches on the future of compute power, likening it to the new oil and emphasizing the importance of efficiency and sustainability. He is optimistic about AI's potential to enhance human discourse with subtlety and nuance, while also acknowledging concerns about decision-making and control. Ross concludes by stressing Groq's mission to empower human agency in the age of AI.

Takeaways

• 🚀 **Innovation at Google**: Jonathan Ross, founder and CEO of Groq, previously worked at Google where he invented the Tensor Processing Unit (TPU), which powers AI at Google.
• 💡 **Founding Groq**: Ross left Google to found Groq, motivated by the desire for more ambition and freedom outside of a large corporate structure.
• 🤖 **Groq's Unique Approach**: Groq focuses on creating a more efficient and faster AI chip, with an initial emphasis on developing an easy-to-use compiler, which provided a unique advantage.
• 🔋 **Energy Efficiency**: Groq's chips are designed to be energy-efficient, using about one-third of the power compared to GPUs, which is beneficial for the environment and cost-effectiveness.
• 💻 **Cloud Services**: Groq offers cloud services where developers can easily start using their technology without the need for significant initial investment in hardware.
• 📈 **Market Growth**: Groq has seen rapid growth in users and developers leveraging their platform, indicating a strong market interest in their technology.
• 🔩 **Hardware Utilization**: Groq's hardware is designed to be highly utilized, which contrasts with the typical low utilization rates of GPUs, thereby offering better value for money.
• 🧠 **AI as the New Frontier**: Ross likens compute power to the new oil, suggesting that generative AI will be a significant driver of future technological and economic growth.
• 🤖 **The Role of Agents**: Ross is particularly bullish on the potential of AI agents, which can benefit greatly from Groq's high inference speeds, enabling more sophisticated and interactive applications.
• 🛠️ **Optimizing for Groq**: Model builders can optimize their AI models for Groq's hardware by taking advantage of its low-latency architecture and automated compiler.
• 🌟 **The Future of AI**: Ross is hopeful that AI will bring more nuance and curiosity to human discourse, while being cautious about the potential for AI to centralize decision-making if not managed properly.

Q & A

Why did Jonathan Ross decide to leave Google and found Groq?

Jonathan Ross decided to leave Google because he felt constrained by the corporate environment and realized that outside of Google, he could be more ambitious and bold. He didn't need to get multiple internal approvals to pursue his ideas, but could instead seek funding from any one of thousands of venture capitalists.

What is the unique advantage Groq has with its chip design?

Groq's unique advantage comes from its focus on compiler development before chip design, which allowed for a very efficient and easy-to-use software interface. The chip is known for its high inference speed, which is beneficial for applications that require rapid processing of AI models.

How does Groq's chip architecture differ from traditional GPU-based systems?

Groq's chip architecture is designed to be more efficient by reducing reliance on external memory, which is a bottleneck in GPU-based systems. Groq's chips are part of a larger system where data processing is faster and more streamlined, akin to an assembly line, as opposed to the slower, more fragmented approach of GPUs.

What is the business model for companies considering the use of Groq hardware?

Companies can start by using Groq's cloud service, which is accessible and requires no initial investment. For companies requiring massive scale and processing millions of tokens per second, Groq can discuss on-premises hardware deployment. However, Groq does not intend to rent individual chips but will allow users to upload their models for execution on Groq's infrastructure.

Why is compute power considered the new oil, according to Jonathan Ross?

Compute power is considered the new oil because it is the limiting factor for the creation of new content in the moment through generative AI. Unlike the Information Age, where data was copied and distributed, generative AI requires significant computational resources to create something new, making compute power a valuable and limiting resource.

What are the biggest bottlenecks that Jonathan Ross sees in the future of the AI industry?

The biggest bottlenecks in the future of the AI industry, as seen by Jonathan Ross, are likely to be related to compute power and the ability to efficiently utilize it. As AI models and applications become more complex and data-intensive, the need for faster and more efficient computing will grow.

What is Jonathan Ross's perspective on the future role of AI in human discourse?

Jonathan Ross is hopeful that AI will bring subtlety and nuance to human discourse, provoking curiosity and helping people to understand different viewpoints. He believes that AI can help people to see situations with more complexity and depth, leading to better communication and understanding.

What is the most value-adding layer in the AI space for startups, according to Jonathan Ross?

Jonathan Ross suggests that the most value could be extracted at the infrastructure layer, where handling the 'drudgery' of AI operations can lead to significant business opportunities. He also mentions that while creating generative AI models might offer higher expected value, it also comes with higher variance and uncertainty.

How does Groq's inference speed impact the use case for agents?

Groq's high inference speed is particularly beneficial for agents as it allows for real-time, interactive applications that can respond quickly to user inputs. This speed can enable agents to work together more effectively, providing rapid feedback and enhancing the overall user experience.

What are some techniques model builders can use to optimize for Groq hardware?

Model builders can optimize for Groq hardware by taking advantage of the company's automated compiler, exploring architectures that benefit from low latency, and utilizing Groq's ability to perform quantized numerics. Additionally, they can leverage the faster interconnect dimensions that Groq's hardware offers compared to traditional GPUs.

What is Groq's approach to handling the control and bias of AI models?

Groq's mission is to ensure that AI models augment human decision-making rather than replacing it. They aim to curate models that help users understand and make their own decisions, rather than models that make decisions for them. This approach is intended to preserve human agency in the age of AI.

Outlines

00:00

🚀 Founding GROCK and the Shift from Google

Jonathan Ross, the founder and CEO of GROCK, discusses his transition from Google, where he was instrumental in developing the Tensor Processing Unit (TPU). He shares his motivations for leaving Google to start his own company, emphasizing the constraints of working within a large corporation and the freedom that comes with running a startup. Ross also talks about the initial focus on developing a compiler for ease of software use, which provided a unique advantage for GROCK's chip design.

05:01

💡 GROCK's Innovative Chip Design and Business Model

The conversation delves into GROCK's chip architecture, known for its high inference speed but lower memory per chip. Ross explains the counterintuitive decision to design for more chips rather than fewer, drawing an analogy with car manufacturing efficiency. He discusses the advantages of GROCK's system, which allows for better chip utilization and lower costs. The topic of cloud computing versus on-premises hardware acquisition for businesses is also explored, with Ross advocating for starting with GROCK Cloud and considering on-premises deployment at scale.

10:03

🌐 The Future of Compute and AI

Ross and the interviewer ponder the future of the industry, considering compute as the new limiting resource, akin to oil. They discuss the challenges that lie ahead in terms of electricity and silicon as potential bottlenecks. Ross provides advice for entrepreneurs looking to start a business in the AI space, weighing the pros and cons of different layers of the AI ecosystem, from silicon to applications.

15:03

🤖 The Role of Agents and GROCK's Inflection Point

The discussion highlights GROCK's inference speed as a pivotal feature for powering AI agents. Ross is optimistic about the potential of agents and the various use cases unlocked by GROCK's technology. He emphasizes the importance of speed in user experience, comparing the excitement of broadband to the potential of GROCK's capabilities. The rapid growth of users and developers on GROCK's platform is also mentioned, showcasing the demand for high-speed AI applications.

20:03

🌟 Hopes and Fears for AI's Impact on Society

Ross expresses his hopes for AI to bring more subtlety and nuance to human discourse, fostering curiosity and understanding. He addresses fears about AI, drawing a parallel with the historical reaction to Galileo's telescope. Ross stresses the importance of maintaining human agency in AI decisions and ensuring that GROCK's models assist rather than dictate choices, aiming to broaden perspectives rather than control them.

Keywords

Groq

Groq is a company founded by Jonathan Ross, which is focused on developing custom silicon for artificial intelligence applications. The company's name is derived from the acronym GROCK, which stands for 'Graphics Output'. In the video, Ross discusses his decision to leave Google and found Groq, emphasizing the freedom and ambition he could pursue outside of a large corporation.

Tensor Processing Unit (TPU)

The Tensor Processing Unit (TPU) is a custom silicon chip developed by Google, which powers AI applications by accelerating machine learning tasks. Jonathan Ross, before founding Groq, was instrumental in the creation of the TPU at Google. The TPU is a key technology in AI, enabling faster and more efficient processing of neural network models.

Inference Speed

Inference speed refers to how quickly a machine learning model can make predictions or decisions based on input data. Groq is known for its exceptionally fast inference speeds, which can reach 5, 6, 7, or more than 100 tokens per second. This speed is crucial for real-time applications and enhances user experience by reducing latency.

Memory per Chip

Memory per chip is the amount of memory available on a single chip or processor. The discussion in the video mentions that Groq chips have a lower memory per chip compared to some other solutions, which influences how they are used in business applications. This design choice leads to the need for multiple chips to handle large-scale operations, as each chip operates efficiently for a short period.

Cloud Provider

A cloud provider is a company that offers resources and services through the internet, typically on a subscription basis. In the context of the video, Jonathan Ross talks about the Groq Cloud, which allows developers to start using Groq's technology immediately without the need for significant initial investment in hardware.

AI Chip

An AI chip is a type of semiconductor designed specifically to handle the complex computations required for artificial intelligence applications, such as neural network processing. Ross discusses the development of AI chips at Groq and the strategic decisions behind their design, focusing on efficiency and ease of use.

Compiler

A compiler is a program that translates code written in one programming language into another language. In the video, Ross mentions that Groq spent the first six months working on the compiler before designing the chip. This focus on software development was a strategic move that gave Groq a unique advantage by making the software easier to use.

Utilization Rate

Utilization rate refers to the percentage of time that a resource, such as a chip or server, is being used for its intended purpose. Ross points out that the utilization rate of GPUs is often as low as 25%, meaning that a significant portion of their processing time is wasted. Groq aims to improve this by providing a more efficient use of hardware resources.

Silicon

In the tech industry, 'silicon' often refers to the material used in semiconductors and, by extension, to the chips or microprocessors themselves. Ross discusses the challenges and opportunities in the silicon industry, particularly in the context of creating new AI chips and the competitive landscape.

Generative AI

Generative AI refers to artificial intelligence systems that can create new content, such as text, images, or music, as opposed to simply recognizing or responding to existing content. Ross is bullish on the potential of generative AI and discusses how Groq's technology can support the development and operation of AI agents.

AI Agents

AI agents are autonomous entities that can act on behalf of users or other systems, often powered by AI. They can perform tasks, make decisions, and interact with digital environments. Ross highlights the potential of AI agents, especially when paired with Groq's high-speed inference capabilities, to revolutionize how tasks are automated and how users interact with technology.

Highlights

Jonathan Ross, founder and CEO of Groq, discusses his transition from Google to founding his own company.

Ross invented the Tensor Processing Unit (TPU) at Google, which powers AI software.

He left Google to pursue more ambitious projects unconstrained by corporate limitations.

Groq's unique advantage came from focusing on compiler development before chip design.

Groq chips are known for their high inference speed, processing hundreds of tokens per second.

The lower memory per chip in Groq's design requires businesses to purchase multiple units for large-scale operations.

Ross explains the counterintuitive decision to design for more chips rather than fewer for efficiency and avoiding memory bottlenecks.

Groq's business model involves offering cloud services with the option for companies to deploy hardware on-premises at scale.

The company aims to provide high utilization rates of their hardware, unlike the typical 25% utilization rate of GPUs.

Ross predicts compute power will be a limiting factor in the future, likening it to the new oil.

He advises AI entrepreneurs to focus on infrastructure or model layers for higher potential value in the future.

Groq is particularly interested in the potential of agents and how their inference speed can facilitate agent collaboration.

Ross is hopeful that AI will bring subtlety and nuance to human discourse, promoting curiosity and understanding.

He expresses concern about the potential for AI to centralize decision-making, advocating for models that assist human choice rather than replace it.

Groq's mission is to ensure AI augments human agency, not replace it.

The company is focused on curating models that enhance user decision-making capabilities.

Ross highlights the importance of preserving human autonomy in the face of increasingly capable AI systems.