Is the AI Boom Real?

Summary

The author traveled to Japan and the U.S. to learn about the AI chip industry. Much buzz surrounds Sam Altman raising huge funds for a new AI chip venture. The concept of 'scaling laws' suggests that with more data and compute, AI models like GPT continue improving. This drives massive investment. Nvidia faces competition but responds aggressively. Giants like Microsoft vertically integrate to cut costs. OpenAI's ChatGPT generated $2B revenue, but it's unclear if consumer demand can sustain the AI boom. The industry hopes Microsoft's Copilot subscription catches on. If not, some assumptions need rethinking. AI's main impact could just be improving ad targeting for giants like Google and Meta.

Takeaways

• 😲 The chip venture Sam Altman is working on could involve up to $1-7 trillion in investments over several years, largely for data centers and manufacturing
• 📈OpenAI's 'scaling laws' show that combining more data and compute leads to better AI model performance
• 👀Nvidia is aggressively updating its AI chips to stay ahead of new competitors
• 🚨Tech giants like Microsoft and Google want to cut costs by making their own custom AI chips
• 🤔It's unclear if demand from mainstream consumers will sustain the massive investments expected in AI
• 🙌ChatGPT is one of the fastest growing consumer tech products ever at over $2 billion annual revenue
• 🔎All eyes are on Microsoft's Copilot subscription to indicate enterprise demand for AI
• 💰AI promises automation that could greatly boost advertising revenue
• 😕But AI may just make rich companies even richer, not necessarily transform industries
• ✈The author plans more trips to learn about and discuss the AI landscape

Q & A

What was the purpose of the author's recent travel to Japan and the United States?

The purpose was part sight-seeing and part learning tour to understand more about the AI and AI chip boom landscape outside of Taiwan.

What figure is Sam Altman said to be seeking for investments related to AI chips?

Initially it was reported to be billions, but more recently a figure of $1-7 trillion was reported, likely referring to total investments over multiple years related to manufacturing, real estate, power, etc.

What concept is driving the massive investments forecasted in AI chips?

The 'scaling laws' concept that combining more data and compute leads to better AI model performance. This concept helped drive advances like GPT-3 and GPT-4.

How is Nvidia responding to increased competition in AI chips?

By aggressively increasing the pace of updates to their AI accelerator lineup, releasing new products as often as every 6 months.

How might vertical integration by tech giants like Microsoft impact Nvidia?

In the medium to long term, as tech giants develop more advanced custom AI chips, it could reduce Nvidia's market share. Nvidia aims to stay ahead by leading on performance and cost.

What evidence is there so far that AI products are driving real consumer demand?

There are some signs of promise like ChatGPT's reported $2 billion annualized revenue run rate, but it's still early. Microsoft's Copilot subscription will be an important test case.

Where else might AI have a major financial impact?

In advertising and automation, where AI could greatly boost revenues and efficiencies for major platforms like Google and Meta.

What does the author see as a disappointing possibility for AI?

That its biggest impact is automating and optimizing existing industries rather than enabling wholly new products and companies, making the rich richer rather than producing new wealth.

What approach does the author recommend for embedding AI capabilities?

Recommend embedding AI into existing successful products to maximize its impact, using AI as an enabling technology rather than a standalone consumer product.

What are the author's plans going forward regarding AI?

The author plans to make more frequent trips to the US and other regions to have interesting conversations with people in the AI space and keep abreast of the latest developments.

Outlines

00:00

🤔 The Trillion Dollar Question

Paragraph 1 discusses the recent news around Sam Altman raising massive investments for an AI chip venture, with figures ranging from billions to trillions of dollars over several years. It questions the feasibility of such a high investment level and compares it to the entire semiconductor industry revenue. The key theme is the uncertainty around the scale of investment needed to realize Altman's vision.

05:06

📈 Scaling Laws - The New Moore's Law?

Paragraph 2 introduces the concept of scaling laws that have driven advancements in AI like GPT. It notes the parallels between scaling laws and Moore's Law in driving industry roadmaps and investments. However, it also highlights arguments against the perpetual viability of scaling laws.

10:09

💸 Is the AI Boom Financially Sustainable?

Paragraph 3 questions whether the current AI boom, despite rapid progress, is financially sustainable in the long run. It examines early revenue data for ChatGPT and other potential indicator products. The conclusion is cautious optimism, noting AI's potential as an enabling technology, but uncertainty around transformative mainstream adoption.

Keywords

💡Scaling Laws

The Scaling Laws refer to the concept from a 2020 OpenAI paper that combining more data and compute leads to better AI model performance. This idea helped drive the development of large language models like GPT-3 and is a core rationale behind the massive investments being made in AI.

💡ChatGPT

ChatGPT is an AI chatbot launched by OpenAI in late 2022. It uses a large language model and has rapidly gained popularity. The Financial Times reported it already has a $2 billion annual revenue run rate, making it one of the fastest growing consumer tech products ever.

💡Nvidia

Nvidia is the leading supplier of AI/GPU chips. It faces competition from other chipmakers and tech giants developing custom AI chips. But Nvidia is responding aggressively with rapid new chip iterations, reminiscent of the GPU wars era.

💡Moore's Law

Moore's Law refers to the long-term trend of computing power doubling every 1-2 years. This drove massive investment in semiconductors to fuel industries like PCs and smartphones. The similarities between Moore's Law and Scaling Laws suggest AI could also fuel huge economic growth.

💡Microsoft

Microsoft is emerging as the tech giant most aggressively pursuing vertical integration in AI, with custom chips and data center gear. This could cut costs but also suggests AI industry growth is slowing if giants now focus on pricing.

💡ASICs

ASICs refers to Application-Specific Integrated Circuits - customized chips designed by large tech companies to save costs over general purpose chips like GPUs. The push towards ASICs highlights the tech giants' incentive to cut suppliers like Nvidia out.

💡Advertising

Some argue AI's economic impact could be most realized by supercharging digital advertising, which already generates tens of billions in revenue. AI-powered ads may be the "killer app" enriching the giants further.

💡Vertical Integration

Vertical integration refers to tech giants like Microsoft controlling more of the AI tech stack in-house, e.g. custom chips, data center gear. This saves costs but may signal slowing industry growth.

💡Consumer Demand

Despite rapid AI progress, consumer demand remains unclear beyond ChatGPT subscriptions. Historical examples suggest AI investment likely requires public enthusiasm to be sustainable long-term.

💡Killer App

It's debated if AI itself is the next "killer app" fueling broad economic growth, or merely an enabling technology making existing apps and giants richer. Lack of clear consumer use cases beyond ChatGPT feeds this skepticism.

Highlights

Sam Altman and his team have talked to TSMC, people at Abu Dhabi, and the US Government about a trillion dollar chip venture

The trillion dollar figure may be a negotiating tactic to set high expectations for future talks

The scaling laws concept is driving massive investments - more data and compute leads to better AI model performance

The scaling laws could have a similar rallying effect in AI to what Moore's Law had on the semiconductor industry

Nvidia is aggressively updating their AI accelerators to fend off startup competitors

Nvidia ships GPU designs before fully testing them using advanced modeling tools

Tech giants like Google and Microsoft pose the biggest threat to Nvidia through custom ASICs

Microsoft is pushing hard on vertical integration for inference chips and networking

It's unclear if the AI boom is financially sustainable without compelling consumer products

ChatGPT grew quickly, but few products have development and operating costs like AI

All eyes are on Microsoft's Copilot subscription to indicate consumer demand

The killer app for AI could just be improving ad targeting and sales

If AI mainly helps the wealthy get richer, it may not justify massive investments

AI could have more impact by supercharging existing products versus creating new ones

Ads enhanced by AI automation generate tens of billions in revenue already

Transcripts

For the past three weeks I have been  traveling through Japan and the United States.

This trip has been part sight-seeing  and part learning tour with the goal  

of understanding more about the AI and  AI chip boom landscape outside of Taiwan.

Now that I am finished with the big conversations,  I wanted to sit down and share a few thoughts.

## The Trillion

There’s been a lot of news about a  chip venture thing that Sam Altman  

of OpenAI is working on for the AI industry.

The news came slowly. First there was  the report - released at around the  

time he was temporarily ousted from  OpenAI - that he was raising money  

from Middle East investors for some chip  venture. Back then it was just "billions".

Since then, you have also gotten news that  Sam Altman and his team have talked to TSMC,  

people at Abu Dhabi, and the US Government.

Just as I was leaving Tokyo for San  Francisco, the WSJ broke the report with  

that eye catching $1-7 trillion figure that  has got everyone buzzing. It was one of the  

first things that I started asking people  about when I arrived in Silicon Valley.

Many people agreed with me that the  number is a bit too much. Perhaps  

it can be a negotiating tactic to set  expectations for future talks. Once you  

start talking trillions then hundreds of  billions no longer feels as substantial.

Then you have the Information adding a bit  of context to Altman's remarks, saying:

> But in reality, Altman privately has told  people that figure represents the sum total of  

investments that participants ... would need  to make, in everything from real estate and  

power for data centers to the manufacturing  of the chips, over some period of years.

The ecosystem concept makes a lot more sense.  Total semiconductor sales in 2023 were about $520  

billion. Total capital expenditures - according  to the trade group SEMI - were about $140 billion.

But even if we were to assume that this  trillion is spread out over five years  

and diluted with real estate expenses and power,  

that is still another COVID-like step  function upwards in capital expenditure.

The semiconductor industry is a conservative  one. Many of the people there are old-heads  

who have seen many a cycle of booms and busts  downstream. Sam isn’t really the first guy to  

knock on a foundry’s door saying that his  use case is going to change the world.

## Scaling Laws

The concept driving this investment is  something called the "scaling laws".

The name dates to a 2020 paper  posted by OpenAI titled "Scaling  

laws for neural language models". I am  not going to go over the fine details,  

but the gist is that if we combine more data and  compute, we get better results (i.e. less loss).

This was one of the core ideas that  helped OpenAI make the GPT-series a  

reality. Ilya Sutskever, one  of the company's cofounders,  

mentioned something like this in an appearance  on the No Priors podcast back in November:

> I was very fortunate in that I was able to  realize that the reason neural networks of  

the time weren't good is because they are too  small. So like if you tried to solve a vision  

task with a neural network with a thousand  neurons, what can it do? It can't do anything.

> It doesn't matter how good your  learning is and anything else.  

But if you have a much larger network  then it can do something unprecedented.

GPT-3 is big. GPT-4 was even bigger, performing  far better. And GPT-5, whenever it comes out, will  

be even bigger. So far there are no indications  that the scaling laws have broken apart.

I find the parallels with the semiconductor  

industry's Moore's Law very interesting.  The two "laws" do not say the same thing,  

but they might have similar effects  on their respective industries.

In the 1980s and 1990s, Moore's Law became  the rallying cry of the entire semiconductor  

industry - the metronome keeping time for every  company from Santa Clara to Tokyo to Hsinchu.

There is a chance that the Scaling Laws can make  a similar impact on the AI industry. A simple set  

of easily understandable rallying cries that  drive R&D roadmaps for whatever years to come.

There are a variety of arguments against  the Scaling Laws. For instance, people have  

commented that we have basically pulled all the  existing data across the entirety of the Internet.

But there are ways around such things, if the  money is there to make it happen. Physics problems  

have cropped up ahead of Moore's Law, and the  industry derived new engineering solutions:  

the High-K Metal Gate, FinFET, DUV Lithography,  Dry Etch, Ion Implantation, SEM, and so on.

The bigger question is whether  the money is there to continue  

driving this investment. More on that later.

For a way deeper review of the  technical issues behind scaling,  

I recommend Dwarkesh Patel's post - "Will scaling  work?" His podcast is quite excellent too.

## AI Chips & Nvidia

Where there are profits, there are competitors  coming out with ideas to take them.

There has been a lot of ink spilled on Nvidia  and their competition - particularly the  

multi-pronged attack on Nvidia's AI accelerator  profits. I don't want to add too much to this.

I do question whether the Nvidia fortress is going  

to be as assailable as it might at first  seem. Jensen is responding to the threat  

by aggressively ramping up the annual  updates to their accelerator lineup.

For more information on this, I refer you to  

the report from SemiAnalysis. It goes  into much more detail. Dylan's great.

But this relentless pace reminds me of the days  of old Nvidia during the Graphics Cards Wars,  

when Nvidia released a new product  every six months to the market. It  

worked for them then. Why not try  it again now for the AI Chip Wars?

Nvidia can maintain this rollout  speed because - as Jensen implied  

during his appearance at the Acquired  podcast - they ship before they test.

They use the latest computer software  design and emulation tools to model  

and ship new GPU designs to the market  without first fabbing a physical chip:

> The reason why we needed that emulator is  because if you figure out how much money that  

we have, if we taped out a chip and we got  it back from the fab and we started working  

on our software, by the time that we found  all the bugs because we did the software,  

then we taped out the chip again. We  would’ve been out of business already.

The Nvidia teams will try to tape  out "perfect chips" as Huang said,  

but this inevitably will cause problems  for customers. For instance, deployments  

with drivers and other affiliated software  that won't work all that well at first.

But the Nvidia brand can take a hit like that,  

whereas it is unlikely that the AI chip  startups will be able to. It takes several  

years to build a team and then get that  team good enough to ship a working product.

This high speed of iteration will be rough  on customers. Many have no choice but to buy  

what is available, but it stings to spend  tens of thousands of dollars on a machine,  

only to have a vastly better one  coming out just a short time later.

## Giants and Verticalization Nvidia is less concerned about the startups and

even the established silicon players  than they are about the tech giants.

The tech giants - Microsoft, Google, and the like  - are the ones driving the current investment  

spend in AI today. They are also the ones with the  most incentive to cut Nvidia out of their margin.

They would do this using custom-designed  chips or ASICs. Large companies have  

sought these since the good old days of  the computer - commissioning an ASIC to  

replace an entire board of discrete  electronic components to save money.

An example of this would be the Apple IIe,  

a third-generation version of the  Apple II PC powered by a custom ASIC.

A more modern example would be the Google TPU,  which right now is in its fourth iteration.

It makes up a big part of Google's compute  advantage over OpenAI and the other AI labs.

Microsoft seems to be the giant pushing hardest  on vertical integration. CEO Satya Nadella said  

in the Q2 2024 earnings call that most of  the usage in Azure AI services is coming from  

inference rather than training. Those activities  are easier to push to a custom-designed ASIC.

In addition, Microsoft has been working  on vertically integrating other parts  

of the database stack. For instance,  the Information's recent report that  

they are working on a network card  to shuttle data between servers.

This type of vertical integration implies either  that the scale of the AI database stack is so  

large and costly that every penny counts/will  count. Or that the growth in the industry has  

petered out, leaving players to compete on price.  These scenarios both feel weird to consider.

What does this vertical integration trend  mean for Nvidia? It will take some time  

for the other tech giants to ramp up their AI  chip designs. Apparently the first Google TPU  

was a very bare-bones product. So in the  short term, things will be as they are.

But as those chips get better in the medium  to long term, it makes sense that Nvidia  

push as hard as they can to always win  the performance-cost crown. Who knows.

## Financially Sustainable?

I want to move away from cost. I think the cost  benefits of vertical integration are significant  

enough to matter. Now, I want to move to  the other side of the financial equation.

One of the really big questions that I wanted to  answer coming to the United States was to learn  

more about whether or not this OpenAI  boom is Real. Real, with a capital R.

There has been so much progress in the AI  industry in the past year. For instance, the  

OpenAI text to video Sora model. The improvement  in the models' output quality has been impressive.

But even if the technology is amazing, it does  not seem like a hit product by itself. We need  

to embed them into compelling products. Are  there indications of such products breaking  

through into the public? Who is making  money from actual consumers with them?

The massive investments required by Moore's  Law and the semiconductor industry were  

driven by real demand from various downstream  industries. First the military, then consumer  

electronics like radios and calculators, then  the PC, smartphones, and then cloud computing.

These were all things that  ordinary people wanted. To me,  

it feels unlikely that the truly  large investments in AI will happen  

unless those ordinary consumers start  buying these services in a major way.

The Financial Times did report that ChatGPT hit a  

$2 billion run-rate revenue in December  2023. That is basically the only solid  

piece of news indicating that people are  paying real money for an LLM service.

Now, we should note that it is still early in the  

generative AI boom. ChatGPT  is a little over a year old.

But I also want to note that the  iPhone was first released in June 2007.

About two years later in 2009,  

Apple sold 20 million iPhones for like  $13 billion in revenue. Growth continued,  

with 40 million iPhones sold the year after that,  and then 72 million and 125 million after that.

Okay fine, the iPhone is the  greatest consumer technology  

product that history has ever seen. It  is a difficult example to live up to.

ChatGPT is still one of the fastest  growing consumer tech products in history,  

depending on what you think about  Threads. And people are still  

using it a lot - Altman recently said that  ChatGPT generates 100 billion words a day.

And Facebook took a while to produce revenue  - at first investing to build up the audience.  

Perhaps ChatGPT is doing the same Aggregator  approach, gathering users to monetize later.  

But few products cost as much to use. AI  probably has shorter a leash than we think.

Other than ChatGPT, the one product  that everyone in the industry seems  

to have their eye on is the Microsoft  Copilot subscription service. We want  

to see if this $20 a month offering  catches on with people and enterprises.

If it does, then we are really off to the races  ... the sky really is the limit ... we are so back  

... and so on, insert bombastic metaphor here. If  not, then we have to adjust some mental models.

## Conclusion

There is one last thing that I  should mention. Maybe where the  

LLM revolution will be most Real is when it  is embedded into the products we know today.

For instance, a few people I spoke  to insisted that I am overlooking the  

impact that AI automation will have  in supercharging advertising sales.

Like Google's Performance Max ads, which  use AI to automate the creation, deployment,  

and targeting of new ads. Ads like Performance Max  apparently generate tens of billions of dollars.

Or how when Apple disabled cookie tracking in  2021 with the App Tracking Transparency thing.  

Meta/Facebook at first said that this would  cost the company over $10 billion in sales.

But over time, Meta somehow managed to claw  back its targeting accuracies using AI,  

collecting data from its Conversions API.

So maybe the killer app for AI is just more ads.

But Ben Thompson and others have been saying for a  

while that AI is an enabling technology -  a technology for making the rich richer,  

rather than making a whole new class of rich  people like the PC or electrification was.

So for me, if this is all that it is,  

then it is a bit disappointing. But it  makes the AI boom nevertheless real.

Anyway, these are my reflections from my trip to  the United States. I hope to make more frequent  

trips in the future and have more interesting  conversations with people in the space.

If you are interested in having a chat,  

shoot me an email. Maybe I will be coming to  town again and can have that chat in person.