Google I/O 2024: Everything Revealed in 12 Minutes

Summary

Google IO has unveiled a plethora of AI advancements across their platforms. Project Astra, an AI assistance initiative, enhances information processing by encoding video frames into a timeline for efficient recall. Google's new generative video model, Vo, generates high-quality 1080p videos from various prompts, offering users creative control. The sixth generation of TPUs, Trillion, promises a 4.7x improvement in compute performance. Google Search has been revolutionized with AI, allowing users to ask complex questions and search with photos. The new Gemini model will offer a revamped AI overview experience, providing dynamic and organized search results. Gemini is also set to become the new AI assistant on Android, with context awareness and multimodality capabilities. The integration of AI into Android's OS aims to enhance the smartphone experience while maintaining privacy.

Takeaways

• 📈 **Gemini Model Usage**: Over 1.5 million developers utilize Gemini models for debugging, gaining insights, and developing AI applications.
• 🚀 **Project Astra**: An advancement in AI assistance that processes information faster by encoding video frames and combining them with speech input into a timeline for efficient recall.
• 📚 **Caching for Speed**: Introducing a cache between the server and database to improve system speed.
• 🎥 **VO Video Model**: A new generative video model that creates high-quality 1080p videos from text, image, and video prompts, offering creative control and various cinematic styles.
• 🧠 **Sixth Generation TPUs**: The introduction of Trillion TPU, offering a 4.7x improvement in compute performance per chip over the previous generation.
• 🔍 **Google Search Transformation**: The use of Gemini in Google Search has led to a new generative experience, allowing for more complex queries and innovative search methods.
• 🍽️ **AI Overviews**: A revamped search experience that clusters results and provides dynamic, whole-page experiences for categories like dining, recipes, movies, and more.
• 🤖 **Live Interaction with Gemini**: A new feature allowing real-time interaction with Gemini using Google's latest speech models, enabling more natural conversations.
• 📱 **Personalized AI with Gems**: The ability to create personalized AI experts, or 'gems,' for any topic, offering tailored assistance.
• 📱 **AI-Powered Android**: Android's multi-year journey to integrate AI more deeply, starting with AI-powered search, a new AI assistant, and on-device AI for fast, private experiences.
• 📚 **Educational Assistance**: The use of on-device AI for educational purposes, such as providing step-by-step instructions for homework problems directly on the device.

Q & A

What is the significance of Gemini models for developers?

Gemini models are crucial for developers as they are used across various tools to debug code, gain new insights, and build the next generation of AI applications.

What is Project Astra and how does it improve AI assistance?

Project Astra is an advancement in AI assistance that builds on the Gemini model. It developed agents that can process information faster by continuously encoding video frames, combining video and speech input into a timeline of events, and caching this for efficient recall.

How does adding a cache between the server and database improve the system's speed?

Adding a cache between the server and database can significantly improve speed by reducing the need to access the database for every request, thus speeding up data retrieval times.

What is the new generative video model announced at Google IO?

The new generative video model announced is called 'vo'. It creates high-quality 1080p videos from text, image, and video prompts, offering unprecedented creative control and the ability to capture details in various visual and cinematic styles.

What is the improvement in compute performance per chip offered by the sixth generation of TPUs, called Trillian?

Trillian, the sixth generation of TPUs, delivers a 4.7x improvement in compute performance per chip over the previous generation, making it the most efficient and performant TPU to date.

How has Gemini transformed Google Search?

Gemini has transformed Google Search by enabling a generative experience that allows users to search in entirely new ways, ask new types of questions, and even search with photos, leading to an increase in search usage and user satisfaction.

What is the new feature in Android that allows for AI-powered search at the user's fingertips?

The new feature in Android is an AI-powered search that provides step-by-step instructions and answers directly on the device, making it easier for users to get the information they need without having to switch between apps.

How does Gemini become a more helpful assistant on Android?

Gemini becomes a more helpful assistant on Android by becoming context-aware, which allows it to anticipate what the user is trying to do and provide more helpful suggestions at the moment.

What is the 'gems' feature in Gemini and how does it work?

The 'gems' feature in Gemini allows users to create personalized experts on any topic they want. Users can set up a gem by tapping to create it, writing their instructions once, and then coming back to it whenever they need it.

How does the new live experience with Gemini using Google's latest speech models enhance user interaction?

The new live experience with Gemini enhances user interaction by allowing Gemini to better understand the user and answer naturally. Users can even interrupt while Gemini is responding, and it will adapt to the user's speech patterns.

What is the significance of Android being the first mobile operating system to include a built-in on-device Foundation model?

The inclusion of a built-in on-device Foundation model in Android signifies a major step forward in integrating AI directly into the OS. This allows for faster experiences while also protecting user privacy by bringing the capabilities of models like Gemini directly to the user's device.

Outlines

00:00

🚀 Project Astra and AI Advancements at Google IO

The first paragraph introduces the audience to Google IO and highlights the widespread use of Gemini models by developers for various purposes, including debugging code and building AI applications. It also mentions the integration of Gemini's capabilities into Google's products such as search, photos, workspace, Android, and more. The speaker then presents Project Astra, which is built on the Gemini model to process information faster by encoding video frames and combining video and speech input. The system's efficiency is discussed, along with the announcement of a new generative video model called 'vo' that creates high-quality videos from various prompts. The paragraph concludes with the unveiling of the sixth generation of TPU, Trillion, and the introduction of new CPUs and GPUs, emphasizing Google's commitment to offering diverse and powerful hardware options for cloud customers.

05:04

🔍 Enhanced Search and Personalized AI with Gemini

The second paragraph delves into the transformation of Google search with the help of Gemini, where users are engaging with a new generative search experience that allows for more complex queries and even photo-based searches. The speaker announces the launch of an AI-driven search experience that will be available to users in the US, with plans for global expansion. The paragraph also covers a new live conversational experience with Gemini, which utilizes Google's latest speech models for better understanding and natural responses. Additionally, the concept of 'gems' is introduced, allowing users to create personalized AI experts on any topic. The paragraph concludes with a demonstration of how Gemini can be used to assist with tasks such as solving physics problems and understanding sports rules, showcasing its contextual awareness and ability to provide helpful suggestions.

10:05

📱 AI Integration in Android and the Future of Mobile Experiences

The third paragraph focuses on the integration of Google AI directly into the Android operating system, enhancing the smartphone experience by making Android the first mobile OS to include a built-in on-device Foundation model. This integration aims to bring the benefits of Gemini to users' pockets while protecting their privacy. The speaker discusses the upcoming expansion of capabilities with the latest model, Gemini Nano, which introduces multimodality, allowing the phone to understand the world through text, sound, and spoken language. The paragraph ends with a light-hearted moment where the speaker acknowledges the frequent mention of AI during the presentation and provides a humorous touch by counting the occurrences.

Keywords

💡Gemini models

Gemini models refer to a set of advanced AI tools utilized by over 1.5 million developers for debugging code, gaining insights, and developing AI applications. They are central to the theme of leveraging AI for innovation and efficiency, as mentioned in the context of using these models across Google's various products like search, photos, workspace, and Android.

💡Project Astra

Project Astra is an exciting new development in AI assistance that builds upon the Gemini model. It involves developing agents capable of processing information more rapidly by encoding video frames continuously, combining video and speech input into a timeline of events, and caching this information for efficient recall. This project is pivotal to the video's narrative on advancing AI capabilities for faster and more integrated processing of multimedia data.

💡TPUs (Tensor Processing Units)

TPUs, or Tensor Processing Units, are specialized hardware accelerators designed to speed up machine learning tasks. The announcement of the sixth generation of TPUs, named 'Trillion', highlights a significant 4.7x improvement in compute performance per chip. This keyword is key to understanding Google's commitment to enhancing AI processing capabilities and its role in the video's focus on the future of AI.

💡AI overviews

AI overviews represent a revamped search experience that uses AI to provide users with a more comprehensive and dynamic set of search results. This feature is set to roll out to users in the US and then globally. It is a prime example in the script that illustrates the integration of AI into everyday online activities, offering a more intuitive and personalized search experience.

💡Live using Google's latest speech models

This refers to an upcoming feature that will allow users to have in-depth, natural conversations with Gemini using voice commands. It signifies Google's progress in speech recognition and natural language processing, enabling more interactive and accessible AI experiences. The feature is a part of the broader theme of making AI more user-friendly and integrated into daily life.

💡Gems

Gems are customizable features within the Gemini app that allow users to create personal experts on any topic they desire. They are designed to be simple to set up and use, offering personalized AI assistance. Gems are an example of the video's emphasis on tailoring AI technology to individual needs and preferences.

💡Android with AI at the core

This phrase encapsulates Google's vision for the future of its mobile operating system, where AI is an integral part of the user experience. The integration of AI in Android aims to create new ways to access information, provide contextual assistance, and maintain privacy. It is a key component of the video's narrative on the transformational potential of AI in mobile technology.

💡Gemini Nano

Gemini Nano is the latest model of Google's AI technology that is set to expand the capabilities of Android devices with multimodality. This means the phone will understand the world not just through text but also through sight, sounds, and spoken language. It represents the video's focus on making AI more intuitive and responsive to various forms of human interaction.

💡Video FX

Video FX is an experimental tool mentioned in the script that will allow users to create high-quality 1080p videos from text, image, and video prompts using the new generative video model called 'vo'. This tool signifies the video's theme of providing creative control and enhancing the ability to generate content through AI.

💡Search generative experience

The search generative experience is a feature that has been tested and is planned for wider rollout. It involves using AI to answer queries in new ways, including handling more complex and longer queries, and even searching with photos. This experience is tied to the video's overarching theme of enhancing search capabilities with AI to improve user satisfaction and search efficiency.

💡Custom arm-based CPU

The custom arm-based CPU is Google's first in-house processor designed with industry-leading performance and energy efficiency. It represents a significant step in the video's discussion about the hardware advancements that support advanced AI functionalities and computational demands, highlighting Google's commitment to building end-to-end solutions for AI.

Highlights

Google IO welcomes over 1.5 million developers using Gemini models for debugging code, gaining insights, and building AI applications.

Gemini's capabilities are being integrated across Google's products like search, photos, workspace, Android, and more.

Project Astra is introduced, an advancement in AI assistance that processes information faster by encoding video frames and combining inputs into a timeline.

A new generative video model called 'vo' is announced, capable of creating high-quality 1080p videos from various prompts.

Sixth generation of TPUs, named Trillian, offers a 4.7x improvement in compute performance per chip.

Google is offering CPUs and GPUs, including the new Axion processors and Nvidia's Blackwell GPUs, for cloud customers.

Gemini has transformed Google search, enabling new ways to search with longer and more complex queries, including photo searches.

A fully revamped AI overview experience for search is being launched in the US with plans for global expansion.

Google's new search experience uses Gemini to uncover interesting angles and organize results into helpful clusters.

An AI overview feature provides instant troubleshooting steps for issues, like why a device might not be staying in place.

Live conversational experiences with Gemini using the latest speech models allow for natural interactions and real-time adjustments.

Customization of Gemini through 'gems' allows users to create personal experts on any topic.

Android is being reimagined with AI at its core, starting with AI-powered search, a new AI assistant, and on-device AI for fast, private experiences.

Circle the search feature on Android provides step-by-step instructions for solving problems, like physics word problems.

Gemini becomes context-aware on Android, offering more helpful suggestions in the moment.

Google AI is being integrated directly into the OS, starting with Android being the first mobile OS with a built-in on-device Foundation model.

Gemini Nano, with multimodality, will be expanded to understand the world through text, sights, sounds, and spoken language.

Google counted the number of times 'AI' was mentioned during the presentation as part of the theme of letting Google do the work.

Transcripts

welcome to Google IO it's great to have

all of you with us more than 1.5 million

developers use Gemini models across our

tools you're using it to debug code get

new insights and the build build the

next generation of AI

applications we've also been bringing

Gemini's breakthrough capabilities

across our products in powerful ways

we'll show examples today across search

photos workspace Android and more today

we have some exciting new progress to

share about the future of AI assistance

that we're calling project Astra

building on our Gemini model we

developed agents that can process

information Faster by continuously

encoding video frames combining the

video and speech input into a timeline

of events and caching this for efficient

recall tell me when you see something

that makes

sound I see a speaker which makes sound

do you remember where you saw my glasses

yes I do your glasses were on the desk

near a red

apple what can I add here to make this

system

faster adding a cach between the server

and database could improve

speed what does this remind you

of shringer cat today I'm excited to

announce our newest most capable

generative video model called

vo vo creates high quality 1080p videos

from text image and video prompts it can

capture the details of your instructions

in different Visual and cinematic Styles

you can prompt for things like aerial

shots of a landscape or a time lapse and

further edit your videos using

additional prompts you can use vo in our

new experimental tool called video FX

we're exploring features like

storyboarding and generating longer

scenes vo gives you unprecedented

creative control core technology is

Google deep mind's generative video

model that has been trained to convert

input text into output

video it looks good we are able to bring

ideas to life that were otherwise not

possible we can visualize things on a

time scale that's 10 or 100 times faster

than before today we are excited to

announce the sixth generation of tpus

called

trillion trillum delivers a 4.7x

Improvement in compute performance per

chip over the previous generation it's

our most efficient and performant TPU

today we'll make trillum available to

our Cloud customers in late

2024 alongside our tpus we are proud to

offer CPUs and gpus to support any

workload that includes the new Axion

processes we announced last month our

first custom arm-based CPU with

industry-leading performance and Energy

Efficiency we are also proud to be one

of the first Cloud providers to offer

envidia Cutting Edge Blackwell gpus

available in early 2025 one of the most

exciting Transformations with Gemini has

been in Google search in the past year

we answered billions of queries as part

of her search generative experience

people are using it to search in

entirely new ways and asking new types

of questions longer and more complex

queries even searching with photos and

getting back the best the web has to

offer we've been testing this experience

outside of labs and we are encouraged to

see not only an increase in search usage

but also an increase in user

satisfaction I'm excited to announce

that we will begin will'll begin

launching this fully revamped experience

AI overviews to everyone in the US this

week and we'll bring it to more

countries soon say you're heading to

Dallas to celebrate your anniversary and

you're looking for the perfect

restaurant what you get here breaks AI

out of the box and it brings it to the

whole

page our Gemini model uncovers the most

interesting angles for you to explore

and organizes these results into these

helpful

clusters like like you might never have

considered restaurants with live

music or ones with historic

charm our model even uses contextual

factors like the time of the year so

since it's warm in Dallas you can get

rooftop patios as an

idea and it pulls everything together

into a dynamic whole page

experience you'll start to see this new

AI organized search results page when

you look for inspiration starting with

dining and recipes and coming to movies

music books hotels shopping and more I'm

going to take a video and ask

Google why will does not stay in

place and in a near instant Google gives

me an AI overview I guess some reasons

this might be happening and steps I can

take to troubleshoot so looks like first

this is called a tonger very helpful and

it looks like it may be unbalanced and

there's some really helpful steps here

and I love that because I'm new to all

this I can check out this helpful link

from Audio Technica to learn even more

and this summer you can have an in-depth

conversation with Gemini using your

voice we're calling this new experience

live using Google's latest speech models

Gemini can better understand you and

answer naturally you can even interrupt

while Gemini is responding and it will

adapt to your speech

patterns and this is just the beginning

we're excited to bring the speed gains

and video understanding capabilities

from Project Astra to the Gemini app

when you go live you'll be able to open

your camera so Gemini can see what you

see and respond to your surroundings in

real

time now the way I use Gemini isn't the

way you use Gemini so we're rolling out

a new feature that lets you customize it

for your own needs and create personal

experts on any any topic you want we're

calling these gems they're really simple

to set up just tap to create a gem write

your instructions once and come back

whenever you need it we've embarked on a

multi-year journey to reimagine Android

with AI at the core and it starts with

three breakthroughs you'll see this

year first we're putting AI powered

search right at your fingertips creating

entirely new ways to get the answers you

need second Gemini is becoming your new

AI assistant on Android there to help

you any time and third we're harnessing

on device AI to unlock new experiences

that work as fast as you do while

keeping your sensitive data private one

thing we've heard from students is that

they're doing more of their schoolwork

directly on their phones and tablets so

we thought could Circle the search be

your perfect study

buddy let's say my son needs help with a

tricky physics word problem like this

one my first thought is oh boy it's been

a while since I've thought about

kinematics if he stumped on this

question instead of putting me on the

spot he can Circle the exact part he's

stuck on and get stepbystep

instructions right where he's already

doing the work now we're making Gemini

context aware so it can anticipate what

you're trying to do and provide more

helpful suggestions in the Moment In

other words to be a more helpful

assistant so let me show you how this

works and I have my shiny new pixel 8A

here to help

me so my friend Pete is asking if I want

to play pickle ball this weekend and I

know how to play tennis sort of I had to

say that for the demo uh but I'm new to

this pickle ball thing so I'm going to

reply and try to be funny and I'll say

uh is that like tennis but with uh

pickles um this would be actually a lot

funnier with a meme so let me bring up

Gemini to help with that and I'll say uh

create image of tennis with Pickles now

one you think you'll notice is that the

Gemini window now hovers in place above

the app so that I stay on the

flow okay so that generates some pretty

good images uh what's nice is I can then

drag and drop any of these directly into

the messages app below so like so and

now I can ask specific questions about

the video so for example uh what is is

kind type the two bounce rule because

that's something that I've heard about

but don't quite understand in the game

by the way this us signals like

YouTube's captions which means you can

use it on billions of videos so give it

a moment and there and get a nice

distinct answer the ball must B once on

each side of the Court uh after a serve

so instead of trolling through this

entire document I can pull up Gemini to

help and again Gemini anticipates what I

need and offers me an ask this PDF

option so if I tap on that Gemini now

ingests all of the rules to become a

pickle ball expert and that means I can

ask very esoteric questions like for

example are

spin uh

serves allowed and there you have it it

turns out nope spin serves are not

allowed so Gemini not only gives me a

clear answer to my question it also

shows me exactly where in the PDF to

learn more building Google AI directly

into the OS elevates the entire

smartphone experience and Android is the

first mobile operating system to include

a built-in on device Foundation model

this lets us bring Gemini goodness from

the data center right into your pocket

so the experience is faster while also

protecting your privacy starting with

pixel later this year we'll be expanding

what's possible with our latest model

Gemini Nano with

multimodality this means your phone can

understand the world the way you

understand it so not just through text

input but also through sites sounds and

spoken language before we wrap I have a

feeling that someone out there might be

counting how many times you have

mentioned AI today

[Applause]

and since the big theme today has been

letting Google do the work for you we

went ahead and counted so that you don't

have

[Applause]

to that might be a record in how many

times someone has said AI