Full Keynote: Satya Nadella at Microsoft Build 2024

Summary

The video script outlines the transformative impact of AI across various sectors, emphasizing the accessibility of expertise through personal AI assistants like GitHub Copilot. It showcases AI's potential in enhancing productivity, creativity, and learning, highlighting innovations from Microsoft, including AI-first PCs, Azure AI services, and tools like Copilot Studio for customizing AI workflows. The script also underscores the importance of security and the democratizing force of AI in creating more accessible and innovative experiences.

Takeaways

• 📊 The industry is moving towards a common vision of 'information at your fingertips', emphasizing accessibility and expertise through AI.
• 🌐 AI is becoming ubiquitous, aiding in various sectors such as farming, laboratories, boardrooms, and classrooms, highlighting its universal applicability.
• 🔧 GitHub Copilot is reported to save developers up to 50% of their time, allowing for increased efficiency and innovation.
• 🧑‍🏫 AI is being utilized as a personal tutor, helping individuals learn new skills, including complex subjects like banking.
• 🚜 Generative AI is being employed to improve productivity, with examples such as helping farmers analyze conditions of their ponds and boosting agricultural productivity.
• 🎨 AI is unlocking creativity, providing detailed descriptions that inspire artwork, showcasing its role in the arts and imagination.
• 👩‍🏫 Teachers are now able to create customized lesson plans with the help of AI, indicating a shift in educational approaches and tools.
• 🤖 The introduction of Copilot+PCs represents a new category of AI-first computers, designed to be the fastest for AI operations.
• 🛠️ Windows Copilot Runtime aims to be as transformative for AI as Win32 was for graphical user interfaces, offering a new layer for developers.
• 🌟 Microsoft is focusing on democratization and sustainability, expanding AI infrastructure globally while aiming for 100% renewable energy in data centers.
• 🔬 The tech stack is evolving at every level, from data center power management to edge computing, adapting to new workloads and use cases.

Q & A

What is the core vision presented for the industry in the script?

The core vision is 'information at your fingertips', which has evolved into 'access to expertise', indicating a shift towards making AI available for everyone, everywhere.

How does the speaker describe the impact of AI on productivity and innovation?

The speaker mentions that AI, such as GitHub Copilot, can save about 50% of time, allowing individuals to use that time for other innovative tasks.

What is the significance of the new era of AI mentioned in the script?

The new era of AI is significant because it represents a shift from mere access to information to providing access to expertise, which can be utilized across various sectors like farming, laboratories, boardrooms, and classrooms.

How does AI technology facilitate learning according to the script?

AI technology serves as a personal tutor, helping individuals learn new skills such as banking, loan applications, and saving money, with the potential to reimagine student learning worldwide.

What role does Generative AI play in enhancing productivity and insights?

Generative AI can learn from data to improve productivity, such as helping farmers improve productivity, and providing insights in near real-time through personal coaching.

How is AI contributing to creativity, as described in the script?

AI is unlocking creativity by providing detailed descriptions that fuel imagination, aiding in activities like painting artwork, and allowing teachers to create lesson plans tailored to specific needs.

What is the significance of the introduction of 'Copilot+PCs' in the script?

The introduction of 'Copilot+PCs' signifies the creation of the fastest AI-first PCs, which are part of the new platforms being developed to support AI applications and solutions.

What is the role of the 'Windows Copilot Runtime' in the tech stack?

The 'Windows Copilot Runtime' is introduced to make Windows the best platform for building AI applications, similar to how Win32 was pivotal for the graphical user interface.

How does the speaker describe the advancements in AI infrastructure?

The speaker describes the AI infrastructure advancements as comprehensive and scalable, with Azure being built as the world's computer, offering the most complete selection of AI accelerators and focusing on sustainability.

What is the significance of the partnership with NVIDIA and AMD in the AI infrastructure?

The partnership with NVIDIA and AMD is significant as it brings the latest AI accelerators to Azure, offering the best performance and cost for AI workloads, and enabling advanced AI capabilities like confidential compute on GPUs.

How does the script emphasize the importance of democratization in technology?

The script emphasizes democratization by showcasing how AI models developed in one part of the world can be used to improve lives in another, and by making AI capabilities more accessible to developers globally.

Outlines

00:00

🤖 AI Transformation and Accessibility

The speaker introduces the concept of 'information at your fingertips' and discusses the evolution of AI to become accessible expertise for everyone. They highlight the impact of AI on various sectors like farming, education, and business, and share personal experiences of time saved through GitHub Copilot. The transformative potential of AI in reimagining learning and productivity is emphasized, along with its ability to unlock creativity and enhance daily tasks.

05:02

🚀 Accelerating AI Capabilities

This paragraph delves into the rapid advancements in AI, comparing the scaling laws of Deep Neural Networks (DNNs) to Moore's Law. The speaker discusses the new natural user interface that supports multimodal input and output, the importance of memory in AI for context retention, and the new capabilities in reasoning and planning. They also reflect on the significant changes brought about by developers utilizing AI to improve lives globally, exemplified by a rural Indian farmer's use of GPT-3.5.

10:02

💻 Introducing Copilot+ PCs and Windows AI Integration

The speaker announces the introduction of Copilot+ PCs, highlighting their AI-first design and the integration of AI as a first-class namespace for Windows. They discuss the Windows Copilot Runtime and its significance for building AI applications, as well as the new Windows Copilot library that facilitates AI integration with local APIs. The paragraph also mentions the introduction of native PyTorch support and the WebNN framework through Windows DirectML.

15:04

🌐 Expanding AI Infrastructure and Sustainability

The speaker outlines the expansion of Microsoft's AI infrastructure, emphasizing the company's commitment to a sustainable, scalable, and comprehensive global presence. They discuss the optimization of power and efficiency, the use of AI accelerators, and collaborations with industry leaders like NVIDIA and AMD. The paragraph also touches on the integration of enterprise platforms like Omniverse Cloud and the strategic partnership with OpenAI.

20:04

🛠️ Democratizing AI with Diverse Models and Tools

This paragraph focuses on the democratization of AI through a broad selection of models and tools. The speaker mentions partnerships with various AI model providers and the integration of open-source models via Azure AI. They also discuss the introduction of small language models (SLMs) like the Phi-3 family and their cost-effectiveness and performance, as well as the new partnership with Khan Academy to enhance math tutoring.

25:04

🛡️ Prioritizing AI Safety and Customization

The speaker discusses the importance of AI safety and the new capabilities in Azure AI Studio for developing and safeguarding AI models responsibly. They announce the general availability of Azure AI Studio and introduce Azure AI custom models for training domain-specific models. The paragraph also covers the integration of AI with data platforms and the introduction of real-time intelligence in Microsoft Fabric.

30:05

🔧 Enhancing Developer Experience with GitHub Copilot Extensions

The speaker introduces GitHub Copilot Extensions, which allow for customization of GitHub Copilot with third-party services. They demonstrate how Copilot can assist with coding and non-coding tasks, such as gathering requirements and creating plans, by staying in the flow across the entire development process. The paragraph also showcases how developers can create extensions for any tool in their stack.

35:06

🤝 Empowering Teams with Team Copilot and Security

The speaker announces Team Copilot, an extension of the personal assistant concept to team collaboration. They discuss how Team Copilot can facilitate meetings, manage projects, and serve as a team assistant across various platforms. The paragraph also emphasizes the security measures underlying Microsoft's approach to AI, including the Secure Future Initiative and its core design principles.

40:06

🎨 AI for a More Accessible World

The speaker concludes with an inspiring example of how AI is being used to create audio descriptions for the visually impaired, making media and art more accessible. They highlight the impact of AI in strengthening culture and shared humanity by providing audio descriptions for art galleries and advertising campaigns, showcasing the power of AI to enhance inclusivity and experience.

Keywords

💡AI Transformation

AI Transformation refers to the shift in industries and processes due to the integration of artificial intelligence. In the video, it is the overarching theme where AI is depicted as a catalyst for change, enabling access to expertise and enhancing productivity across various sectors such as farming, education, and business.

💡GitHub Copilot

GitHub Copilot is an AI-powered development tool that assists in coding by predicting what a programmer intends to write next. The script mentions it as a time-saving tool, allowing developers to increase their efficiency by approximately 50%, which can then be channeled into other innovative endeavors.

💡Personal Assistant

A personal assistant in the context of the video represents AI systems that provide personalized support to individuals, saving time and enhancing productivity. The script illustrates this with examples such as helping to manage ponds or learn new skills like banking, showcasing the versatility of AI in personal use cases.

💡Generative AI

Generative AI is a subset of AI that can create new content, such as text, images, or music. The video emphasizes its role in learning from data to improve productivity, like helping farmers, and in unlocking creativity, as in the case of generating detailed descriptions for artwork creation.

💡Expertise at Your Fingertips

This concept in the video represents the ease of access to information and knowledge through AI, allowing individuals to make informed decisions quickly. It is exemplified by the script's mention of developers being able to optimize their work with AI and teachers creating lesson plans tailored to needs.

💡AI-first PCs

AI-first PCs, as introduced in the script, are personal computers designed with AI capabilities at their core. They are built to leverage AI for various tasks, signifying a new category of devices that are optimized for AI applications and experiences.

💡Azure

Azure, in the context of the video, refers to Microsoft's cloud computing service that provides a range of AI infrastructure and services. It is highlighted as a platform that supports AI applications, from training to inference, and is integral to the deployment and scaling of AI models.

💡Moore's Law

Moore's Law is a principle observed in the evolution of computing hardware, predicting the doubling of transistor count and computing power every couple of years. The video script refers to it as a historical benchmark for progress, comparing it to the current rapid scaling laws of deep neural networks (DNNs).

💡Distributed Synchronous Data Parallel Workloads

This term from the script describes a type of computing workload where data is processed in parallel across multiple distributed systems, synchronized to handle large-scale AI tasks. It is indicative of the changes in tech stack design influenced by AI and machine learning requirements.

💡Natural User Interface

A natural user interface in the video is an interface that supports multiple modes of interaction like text, speech, images, and video. It represents the evolution of how humans interact with machines, moving towards more intuitive and human-like communication methods, facilitated by advances in AI.

💡Democratization of AI

The democratization of AI as mentioned in the script refers to making AI technology accessible to everyone, regardless of their location or background. It is illustrated through examples of developers worldwide using AI to solve local problems and create impactful solutions.

Highlights

The vision of 'information at your fingertips' has evolved into 'access to expertise' in the new era of AI.

AI is transforming various sectors including farming, labs, business, and education by providing personalized assistance and tutoring.

GitHub Copilot is reported to save developers up to 50% of their time, enabling them to focus on innovation.

AI's potential to reimagine student learning worldwide through personalized tutoring is highlighted.

Generative AI is aiding farmers by learning from data to improve productivity.

AI is unlocking creativity, providing detailed descriptions that inspire artwork creation.

Teachers can now create lesson plans tailored to needs with AI providing expertise at their fingertips.

Microsoft Build conference marks a fundamental change in the tech industry, with every layer of the tech stack being reshaped by AI.

Breakthroughs in AI have been driven by scaling laws similar to Moore's Law, but with more rapid advancements every six months.

New natural user interfaces support multimodal input and output, including text, speech, images, and video.

Developers are leveraging AI capabilities to make a global impact, as demonstrated by the use of GPT-3.5 in rural India.

Microsoft introduced Copilot+PCs, the fastest AI-first PCs, and the Windows Copilot Runtime to support AI application development.

Azure's AI infrastructure is expanding globally, with a focus on sustainability and efficiency.

Microsoft is partnering with NVIDIA to offer Azure confidential compute on GPUs and integrate NVIDIA's enterprise platforms with Azure.

AMD's MI300X AI accelerator is now available in Azure, offering the best price-performance ratio for GPT-4 inference.

Microsoft Fabric is introducing Real-Time Intelligence, enabling actionable insights across the entire data estate.

GitHub Copilot Extensions allow for customization with third-party services, enhancing the developer experience.

Team Copilot is announced, extending assistance to team collaborations within Microsoft Teams and other platforms.

Copilot Studio enables the creation of custom AI agents that can work independently and manage tasks asynchronously.

Microsoft's Secure Future Initiative underpins the security of all AI and developer tools, ensuring they are secure by design, default, and operation.

The impact of AI in making the world more accessible, such as through audio descriptions for the visually impaired, is showcased.

Transcripts

I think that our industry

has to have a common vision.

It was a time that connected us

to incredible things.

My name for this vision is,

information at your fingertips.

And three decades later,

we find ourselves in a new era.

One where access to information

becomes access to expertise.

From the farm, to the lab,

from the boardroom, to the classroom,

this new generation of AI

is for everyone, everywhere.

Now, anyone can save time

with a personal assistant.

With GitHub Copilot,

I’m saving about 50% of time.

And that's time that I can use

to do other innovative things.

It allows me to find out

the condition of my ponds faster.

Anyone can access

a personal tutor to learn new skills.

We got to learn about banking:

How to apply for a loan,

how to save money.

We learned so much.

I think this technology

has the potential to completely reimagine

the way every single student learns in the world.

This is a new way to analyze

with a personal coach.

We're going to be able to have

not only productivity gains,

but insights served to us, near real-time.

Generative AI can learn from the data

to help improve the farmer productivity.

AI is unlocking creativity for us all.

Descriptions are so detailed,

in my imagination

I can paint the artwork.

Now teachers are free to create

lesson plans according to our needs.

With expertise at your fingertips.

You can build,

what matters.

Welcome to the age of AI transformation.

Good morning.

Good morning.

It's fantastic to be back here at Microsoft Build.

Welcome to everyone here and joining us on the web.

You know, developer conferences

are always most exciting, most fun

when there's these fundamental changes

that you can sense in the air.

You know, I've marked all my adult life

by coming to PDCs and Builds

for the last three decades.

I still remember,

you know, distinctly

the first time Win32 was discussed,

I guess it was ‘91,

.NET, Azure, right?

These are moments that I marked my life with.

And

it just feels like we're yet again

at a moment like that.

It's just that

the scale,

the scope is so much deeper,

so much broader this time around, right?

Every layer of this tech stack is changing,

you know, from everything from the power draw

and the cooling layer of the data center

to the NPUs at the Edge

are being shaped by these new workloads, right?

These distributed,

synchronous, data parallel workloads

are reshaping every

layer of the tech stack.

But if you think about

even going all the way back

to the beginning of modern

computing, say, 70 years ago

there have been two real dreams we've had.

First

is can computers understand us instead of us

having to understand computers?

And second,

in a world

where we have this ever increasing information

of people, places and things, right?

So, as you digitize more artifacts

from people, places and things

and you have more information,

can computers

help us

reason, plan

and act more effectively on all that information?

Those are the two dreams that we've had

for the last 70-plus years.

And here we are.

I think that we have real

breakthroughs on both fronts.

The core underlying force,

one of the questions I always ask myself is like,

“Okay, this is great.

This is like maybe the golden age of systems.

What's really driving it?”

I always come back to these scaling laws,

just like Moore's Law,

you know, helped drive the information revolution.

The scaling laws of DNNs

are really, along with the model architecture,

interesting ways to use data,

generate data,

are really driving this intelligence revolution.

You could say Moore's Law was probably,

you know, more stable in the sense

that it was scaling at maybe 15 months, 18 months.

We now have these things that are scaling

every six months or doubling every six months.

You know, what we have, though,

with the effect of these scaling

laws is a new natural user interface that's multimodal.

That means supports text,

speech, images, video as input and output.

We have memory

that retains important

context, recalls

both our personal knowledge and data

across our apps and devices.

We have new reasoning and planning capabilities

that helps us understand very complex context

and complete complex tasks.

While reducing the cognitive load on us.

But what stands out for me

as I look back at this past year

is how you all as developers have taken

all of these capabilities

and applied them, quite frankly,

to change the world around us.

I’ll always remember this moment in January 2023

when I met a rural Indian farmer

who was able to reason over some government

farm subsidies

that he had heard about on television

using GPT-3.5 and his voice.

It was remarkable right?

For me, it just brought home the power of all of this

because a frontier model

developed in the West Coast of the United States

just a few months

earlier was used by a developer in India

to directly improve the life of a rural Indian farmer.

The rate of diffusion

is unlike anything

I've seen in my professional career,

and it's just increasing.

In fact, earlier this month I was in Southeast Asia.

I was in Thailand where I met a developer

and I was having a great roundtable

and he was talking to me

about how he's using Phi-3 and GPT-4

and he was using Phi-3

to just optimize

all of the things that he was doing with RAG.

I mean, this is crazy I mean, this is unbelievable.

It had just launched a few weeks earlier

and I was there in Thailand, in Bangkok,

listening to a developer talk

about this technology as a real expert on it.

So it's just great to see the democratization force

that we love to talk about

but to witness it is just been something.

And this is, quite frankly,

the impact of why we are in this industry.

And it's what gives us,

I would say that deep meaning in our work.

So I want to start, though, with a very big thank you

to every one of you

who is really going about bringing

about this impact to the world.

Thank you all so very much.

You know, when I think about

what progress we've made

even since last time we were here at Build,

we built really three platforms.

The first is Microsoft Copilot,

which is your everyday AI companion.

It puts knowledge and expertise

at your fingertips, helps you act on it.

And we built the Copilot stack

so that you can build your AI applications

and solutions and experiences.

And just yesterday,

we introduced a new category of

Copilot+PCs, the fastest AI-first PCs ever built.

All three of these things are exciting platforms

but I want to start with Copilot+ PCs.

You know, we're exposing AI

as a first-class namespace for Windows.

This week

we are introducing the Windows Copilot Runtime

to make Windows the best platform

for you to be able to build your AI applications.

Yeah.

You know what Win32 was to graphical user interface,

we believe the Windows Copilot Runtime

will be for AI.

It starts with our Windows Copilot library,

a collection of these ready-to-use local APIs

that help you integrate into your new experiences

all of the AI capabilities that we shared yesterday.

Now, this includes

no code integrations for Studio Effects

things like creative filters,

teleprompter, voice focus, and much more.

But of course, if you want to access these models itself,

you can directly call them through APIs.

We have 40 plus models available

out of the box, including Phi-Silica

our newest member

of our small language

family model, which we can specific,

which we specifically designed to run

locally on your NPUs, on Copilot+ PCs

bringing that

lightning-fast local inference to the device.

You know, the other thing is the Copilot library

also makes it easy for you to incorporate RAG

inside of your applications on the,

on device data.

It gives you the right tools to build

a vector store within your app.

It enables you to do that

semantic search that you saw with Recall.

But now you can, in your own application, construct

these prompts using local data for RAG applications.

Now, I’m so thrilled to announce as well today

that we will be natively supporting

PyTorch and new WebNN framework

through Windows DirectML.

Native PyTorch support means

thousands of OSS models

will just walk out of the box on Windows,

making it easy for you to get started.

In fact, with WebNN, web developers

finally have a web-native machine learning framework

that gives them direct access

to both GPUs and NPUs

in fact, last night

I was playing with it, turning it on in Edge

and seeing the WebNN sample code running.

It's just so cool to see it

you know, now use even the NPUs.

Both PyTorch and WebNN are available

in Developer Preview today, let's take a look.

And these are just

one of the many announcements today.

We're introducing more than 50-plus

new products and partnerships

to create a new opportunity for you.

We’ve always been a platform company

and our goal is to build the most complete

end-to-end stack

from infrastructure,

to data, to tooling to the application extensibility

so that you can apply the power of this technology

to build your own applications.

And so today I want to highlight

our top news for this event

across every layer of this Copilot stack.

So let's dive right in with infrastructure.

You know, we have the most complete

scalable AI infrastructure

that meets your needs in this AI era.

We're building Azure as the world's computer.

We have the most comprehensive global

infrastructure with more than 60-plus

datacenter regions,

more than any other cloud provider.

Over the past year,

we have expanded our datacenter regions

and AI capacity from Japan

to Mexico, from Spain to Wisconsin.

We're making our best-in-class AI

infrastructure available everywhere

and we're doing this with a focus

on delivering on cloud services sustainability.

In fact,

we're on track to meet our goal

to have our data centers powered

by 100% renewable energy by next year.

Yeah.

You know,

we’re optimizing power

and efficiency across every layer of the stack

from the data center to the network.

Our latest data center

designs are purpose built for these AI workloads

so that we can effectively and responsibly use

every megawatt of power to drive down the cost of AI

and the power draw.

And we are incorporating

advanced data center cooling techniques

to fit the thermal profile of the workloads

and match it to the environment

in the location where it operates.

And the silicon layer,

we are dynamically

able to map workloads

to the best accelerated AI hardware

so that we have the best performance.

And our custom IO hardware and server

designs allow us to provide dramatically faster

networking, remote storage

and local storage throughput.

You know, this

end-to-end approach

is really helping us get to the unprecedented scale.

In fact, last November

we announced the most powerful

AI supercomputer in the cloud

for training.

Using just actually a very small fraction

of our cloud infrastructure.

And over the past six months

we've added 30 times that

supercomputing power to Azure.

Yeah, it's crazy to see the scale.

And of course we're not just scaling training our fleets,

we’re scaling our inference fleet

around the world, quadrupling the number of countries

where Azure AI services are available today

and it's great to see that.

At the heart of our AI infrastructure

are the world's most advanced AI accelerators, right?

We offer the most complete selection

of AI accelerators, including from NVIDIA and AMD,

as well as our own Azure Maia,

all dynamically optimized for the workloads.

That means whether you're using Microsoft Copilot

or building your own Copilot apps,

we ensure that you get

the best accelerator performance at the best cost.

For example, you know,

you see this in what has happened with GPT-4, right?

It's 12x cheaper

and 6x faster since it launched.

And that's,

you know,

the type of progress

you can continue to see,

how, you know, you continue to see the progress

as we evolve the system architecture.

It all starts, though,

with this very deep, deep partnership with NVIDIA,

which spans the entirety of the Copilot

stack across

both all of their hardware innovation

as well as their system software innovation.

Together, we offer Azure

confidential compute on GPUs to

really help you protect sensitive data

around the AI models end to end.

We're bringing

in fact the latest H200s

to Azure later this year,

and will be among the first cloud providers

to offer NVIDIA's

Blackwell GPUs in B100

as well as GB200 configurations.

And we are continuing

to work with them to train and optimize

both large language models

like GPT-4o, as well as small language

models like the Phi-3 family.

Now beyond the hardware,

we are bringing NVIDIA’s key

enterprise platform offerings to our cloud,

like the Omniverse Cloud

and DGX Cloud to Azure

with deep integration

with even the broader Microsoft Cloud.

For example,

NVIDIA recently announced

that their DGX Cloud integrates

natively with Microsoft Fabric.

That means you can train those models using

DGX Cloud with the full access through Fabric data.

And Omniverse APIs will be available

first on Azure for developers

to build their industrial AI solutions.

We're also working

with NVIDIA’s NIM industry

specific developer services

and making them fantastic on Azure.

So, a lot of exciting work with NVIDIA.

Now, coming to AMD,

I am really excited to share

that we are the first cloud

to deliver

general availability of VMs based on AMD’s MI300X

AI accelerator.

It's a big milestone for both AMD and Microsoft.

We've been working at it for a while

and it's great to see that today

as we speak, it offers the best price performance

on GPT-4 inference.

And we'll continue to move forward with Azure Maia.

In fact, our first clusters are live and soon

if you're using Copilot

or one of the Azure OpenAI services,

some of your prompts will be served

using Maia hardware.

Now beyond AI, our end to end systems

optimization also makes cloud-native apps

and the development of cloud-native

apps better, right?

Six months ago

is when we announced our first general purpose

ARM-based compute processor Microsoft Cobalt.

And today

I am really excited to announce

the public preview of Cobalt-based Vms.

You know,

Cobalt is being used for video processing

and permissions management in Microsoft 365,

helping power billion of conversations

on services like Microsoft Teams already.

And we are delivering that same ARM-based

performance and efficiencies to many customers,

in fact, including

Elastic, Mongo, Siemens, Snowflake and Teradata.

In our most recent benchmark

data and tests, our Cobalt 100 VMs delivered up to

40 percent better performance

than any other generally available ARM-based VMs.

So we are very very excited about Cobalt

getting into the market.

Now let's move up the stack to the foundation models.

With Azure AI,

we offer the broadest selection

of frontier and open source models,

including LLMs and SLMs,

so you can choose the model

that makes the most sense for your unique needs

and your application needs.

In fact,

more than 50,000 organizations use Azure AI today.

Yeah.

It's great momentum

and it all starts

though, with our most strategic

and most important partnership with OpenAI.

Just last week, OpenAI announced GPT-4o,

for all their latest multimodal model,

which was trained on Azure.

It's an absolute breakthrough.

It has text, audio, image and video as input and output.

It can respond and just have a humanlike conversation

that's fast and fluid.

It can even be interrupted mid-sentence.

GPT-4o is also the top performing model

on benchmarks across a variety of modalities

and it's always going to get you to the best answer.

It has state-of-the-art performance

at understanding the information

you actually provide in your prompt

because that's really what matters.

What OpenAI demoed last week,

I mean, that was just pretty magical stuff.

And yesterday

we showed how Copilot will leverage GPT-4o.

In the video

I'm just going to play, notice

that you can share

your screen or session as a prompt in Copilot

and have Copilot help you in

whatever it is that you're doing.

Let's roll the video.

Hey Copilot, how's it going?

Hey, I'm doing great.

And it looks like you're about to dive

into some Minecraft.

Ready to build, explore and maybe dodge a few mobs?

Yeah, I'm trying to figure out how to make a sword.

My son plays this game all the time

and I have no idea what I'm doing.

No worries. I'm here to help you impress your son.

I see you've got a crafting table in front of you,

which is a great start.

To craft a sword

you'll need some materials.

Can you open your inventory?

Just press “E” on your keyboard.

Yeah. Do I have what I need?

Let's see.

You've got some sticks.

Perfect for the swords handle.

But it looks like you're missing the material

for the blade.

You can use wood, stone, iron, gold or diamond.

Let's start by gathering some wood or stone.

Shall we head out and find some resources?

Yeah, that sounds great.

Oh my god, what's that?

Oh no, it's a Zombie!

Run! You need to get away from it fast.

Either sprint away or quickly

build a tower of blocks straight up

to get out of its reach.

If you can find a shelter

or dig into the side of a hill, do it now.

Just make sure it can't reach you.

Alright, I think I might be safe now.

That was a close one. Great job finding shelter.

It's really cool to see that.

You know,

the magic here is that

Copilot is your personal assistant

that's helping you with any task, helping

you stay, quite frankly, in the flow in the moment,

whether it's a game or a document or a line of code.

And as OpenAI innovates,

our promise is that we will bring all that innovation

to Azure too, in fact,

the same day that OpenAI

announced GPT-4o,

we made the model available

for testing on Azure OpenAI Service and today

I'm excited to say

that it’s generally available on Azure AI.

You know, what this means though

is that now we can have these groundbreaking apps

that all of you can build using this capability.

One of the coolest things is that now any app,

any website, can essentially be turned

into a full multi-modal,

full duplex conversational canvas.

Let's take a look.

Alright, let me ask the agent here.

Hey, Seth. How's it going?

What can I help you with today?

I am fantastic.

I'm sorry, I'm in such a hurry.

I'm going to go on an

overnight camping hike,

where we're camping -

it's going to be dark, cold.

Should I use these shoes?

It's the mountains.

It's probably going to be cold.

Hey, no worries.

Let me take a look.

Ah, those are the TrekStar hiking sandals.

While they're fantastic for summer hikes

and keeping your feet cool,

they might not be the best choice for cold -

Okay, I'm sorry to interrupt you.

Pick the best thing that I should use,

and can you just add it to my cart?

Just one product.

No worries.

Let's go with the TrekReady

hiking boots.

They're durable,

provide great ankle support,

and most importantly,

they'll keep your feet warm

in the cold mountains.

I'll add them to your cart now.

Ready for checkout?

I see that, amazing.

Thank you very much.

I hope to have a good time camping.

You’re welcome, Seth.

Enjoy camping and stay warm in those boots.

Good luck and have fun in the mountains.

While Seth was,

you know, Seth was in a hurry, but there you go.

GPT-4o kept up with him.

It's incredible.

I really want to thank the OpenAI

team for their partnership and

and really the responsible approach to innovation,

helping our industry move forward.

Sam will be here in fact, joining Kevin

in a little bit to talk a lot more about what's coming

because that's the exciting stuff,

how do you all sample what comes next.

We also,

I brought, we're bringing lots

and lots of other models

as well from Cohere and Databricks and Deci, Meta,

Mistral, Snowflake, all through Azure AI.

We want to support the broadest set of models

from every country, every language.

I'm excited to announce,

in fact, we're bringing models from Cohere,

G42, NTT DATA, Nixtla,

as well as many more, as models of services,

because that's the way

you can easily get to managed AI models.

And we all love open source, too.

In fact,

two years ago at Build,

we were the first to partner

with Hugging Face, making it simple

for you to access the leading open source library

with state-of-the-art language models

via Azure AI.

And today I'm really excited to announce

that we're expanding our partnership,

bringing more models from Hugging Face

with text generation inference,

with text embedding inference

directly into Azure AI Studio.

And, and we're not stopping there.

We are adding

not just large language models,

but we are also leading the small language revolution.

So small language model revolution,

you know, our Phi-3 family of SLMs

are the most capable and most cost effective.

They outperform models of the same size

or the next size up

even across

a variety of language

reasoning, coding, as well as math benchmarks.

If you think about it by performance

to parameter count ratio, it's truly best in class.

And today we're adding new models

to the Phi-3 family

to add even more flexibility

across that quality cost curve.

We're introducing Phi-3 Vision,

a 4.2 billion parameter

multimodal model with language

and vision capabilities.

It can be used to reason our real-world images so

generate insights and answer questions about images.

As you can see right here. Yeah.

And we're also making a 7 billion parameter

Phi-3 small in a 14 billion parameter

Phi-3 medium models available.

With Phi,

you can build apps that span the web,

your Android, iOS, Windows and the Edge.

They can take advantage of local hardware

when available and fall back on the cloud.

We're not simplifying really

all of what

we as developers have to do to support

multiple platforms using one AI model.

Now, it's just awesome

to see how many developers are already using

Phi0-3 to, you know, do incredible things.

From Amity Solutions, the Thai company

that I mentioned earlier,

the ITC, which is been

built a copilot for Indian farmers

to ask questions

about their crops.

Epic in health care

which is now using Phi to summarize complex

patient histories more quickly

and efficiently.

And out of the very,

very cool use cases in education.

Today, I'm very thrilled to announce

a new partnership with Khan Academy.

We'll be working together to use Phi-3

to make math tutoring more accessible.

And I'm also excited to share

that they'll be making Khanmigo

their AI assistant free to all US teachers.

Let's roll the video.

I felt like I was in a place in my teaching career

where I felt like I was kind of losing my sparkle.

And I would just feel really defeated

when I looked out on the classroom

and I would see students

that just didn't look engaged.

Teachers have an incredibly hard job

and what we think we can do

is leverage technology

to take some of the

stuff off of their plate,

to really actually humanize the classroom.

By some miracle, we became

a Khanmigo pilot school.

With new advances in generative AI,

we launched Khanmigo.

The point is to be that personalized tutor

for every student

and to be a teaching assistant for every teacher.

I started to build these more robust lessons

and I started to see my students engage.

We're working with Microsoft

on these Phi models

that are specifically tuned for math tutoring.

If we can make a small language model like Phi,

work really well in that use case,

then we would like to, kind of, shift the traffic to Phi

in those particular scenarios.

Using a small language model,

the cost is a lot lower.

We're really excited that Khanmigo,

and especially in the partnership with Microsoft,

being able to give these teacher tools

for free, to U.S. teachers

is going to make a dramatic impact

on U.S. education.

I think we're going to make them the innovators,

the questioners, isn't that really

just why you wake up every morning?

Right? Because that's our future,

our next generation.

And to me, that's everything.

You know, I’m super excited to see the impact

this all will have and what Khan Academy will do.

And Sal is going to, in fact,

join Kevin soon to share more.

And I'm really thankful for Teachers

like Melissa and everything that they do.

Thank you very much.

You know, of course,

it's about more than just models.

It's about the tools

you need to build these experiences.

With Azure AI Studio

we provide an end-to-end

tooling solution to develop and safeguard

the copilot apps you build.

We also provide tooling and guidance

to evaluate your AI models

and applications

for performance and quality,

which is one of the most important tasks

as you can imagine with all of these models.

And I'm excited to announce

that Azure AI Studio now is generally available.

It's an end to end

development environment to build, train,

and fine tune AI models – and do so responsibly.

It includes built-in support.

For what is perhaps the most important feature,

which is, in this age of AI,

which is AI Safety.

Azure AI Studio

includes the state of the art safety tooling.

You know,

to everything from detecting hallucinations

in model outputs, risk and safety monitoring.

It helps understand

which inputs and outputs are triggering

content filters. Prompt shields, by the way,

to detect and block these prompt injection attacks.

And so today

we are adding

new capabilities, including custom categories,

so that you can create these unique filters

for prompts and completions

with rapid

deployment options,

which I think is super important

as you deploy these models into the real world.

Even when an emerging threat is, you know, appears.

Beyond Azure AI Studio,

we recognize that there are advanced applications

where you need

much more customization

of these models for very specific use cases.

And today

I'm really excited to announce that

Azure AI custom models

will come, giving you the ability

to train a custom model

that's unique to your domain, to your data, that's

perhaps proprietary.

That's same builders and data scientists

who’ve been working with

Open AI, brought all the Phi

advances to you, will work

with all of you to be able

to build out these custom models.

The output will be domain specific.

It will be multitask

and multimodal,

best in class as defined by benchmarks,

including perhaps even specific language proficiency

that may be required.

Now, let's just roll up the stack to data.

Ultimately,

in order to train

fine-tune, ground your models,

you need your data to be in its best shape.

And to do so, we are building out the full data estate

right from operational stores to analytics in Azure.

We’ve also added

AI capabilities

to all of our operational stores,

whether it's Cosmos DB or SQL, or PostgreSQL.

At the core though,

of the Intelligent Data Platform.

Is Microsoft Fabric.

We now have over 11,000 customers,

including leaders in every industry who’re using Fabric.

It's fantastic to see the progress.

With Fabric,

you get everything you need in a single integrated

SAS platform.

It's deeply integrated at its most fundamental level

with compute and storage being unified.

Your experience is

unified, governance is unified, and more importantly,

the business model is unified.

And what's also great about Fabric

is that it works with data anywhere, right?

Not just on Azure,

but it can be on AWS or on GCP

or even in your on-premise data center.

And today we are taking the next step.

We're introducing Real-Time Intelligence in Fabric.

Customers today have

more and more of this real-time

data coming from your IoT systems,

your telemetry systems. In fact, cloud

applications themselves are generating lots of data,

but with Fabric, anyone can unlock

actionable insights across all of your data estate.

Let's take a look.

Introducing

real-time intelligence

in Microsoft Fabric,

an end-to-end solution

empowering you to get instant

actionable insights

on streaming data.

At its heart lies

a central place to discover,

manage, and consume event data

across your entire organization

with a rich governed experience.

Get started quickly

by bringing in data

from Microsoft sources

and across clouds with a variety

of out-of-the-box connectors.

Route the relevant data to

the right destination in Fabric

using a simple

drag-and-drop experience.

Explore insights on petabytes

of streaming data

with just a few clicks.

Elevate your analysis

by harnessing the intelligence

of Copilot in Microsoft Fabric

using simple natural language.

Make efficient business decisions

in the moment,

with real-time actionable insights

and respond to changing landscapes proactively.

Allow users to monitor

the data they care about,

detect changing patterns,

and set alerts or actions

that drive business value.

All your data, all your teams,

all in one place.

This is Microsoft Fabric.

And, we're making it

even easier to design, build and interoperate

with Fabric with your own applications, right?

And in fact, we're building out a new app platform

with Fabric Workload Development Kit

so that people like ESRI, for example, having,

you know, who have integrated

their spatial analytics with Fabric

so that customers can generate insights

from their own location

data using ESRI’s rich tools

and libraries, right on Fabric, right.

This is just exciting to see

As the first time,

you know,

where the analytics stack is really a first-class

app platform as well.

And beyond Fabric,

we are integrating the power of AI across

the entirety of the data stack.

There's no question that RAG is core to any

AI-powered application,

especially in the enterprise today.

And Azure AI Search makes it possible

to run RAG at any scale,

delivering very highly accurate responses

using the state of the art retrieval systems.

In fact, ChatGPT supports for GPTs, their Assistants

API, are all powered by Azure AI Search today.

And with built-in OneLake integration.

Azure AI Search will automatically

index your unstructured data too.

And it's also integrated into Azure

AI Studio to support

bringing your own embedding model, for example.

And so it's pretty incredible

to see Azure Search grow over the last year

into that very core developing service.

Now let's go up through developer tools.

Nearly 50 years after our

founding as a developer tools company,

here, we are once again

redefining software development, right?

GitHub Copilot

was the first, I would say, hit product

of this generative AI age.

And it's the most widely adopted AI

developer tool, 1.8 million subs

across 50,000 organizations, are using it.

And GitHub Copilot,

we're empowering

every developer on the planet to be able to access

programing languages and programing knowledge

in their own native language.

Think about that.

Any person can start programing,

whether it's in Hindi

or Brazilian Portuguese,

and then bring back

the joy of coding to their native language.

And with Copilot Workspace, staying in your flow

has never been easier.

We are an order of magnitude closer

to a world

where any person can go from idea to code

in an instant.

You start with an issue,

it creates a spec

based on its deep understanding of your codebase.

It then creates a plan

which you can execute to generate the code

across the full repo that is multiple files.

At every point in this process – from the issue, to spec,

to plan, to code, you are in control, you can edit it.

And that's really what is fundamentally

a new way of building software.

And we are looking forward

to making it much more broadly

available in the coming months.

And today, we are taking one more big leap forward.

You know, we are bridging the broader developer

tools and services ecosystem

with Copilot for the first time.

We are really thrilled to be announcing

GitHub Copilot Extensions.

Now you can

customize GitHub Copilot

with capabilities from third-party services,

whether it's Docker, Sentry, and many, many more.

And of course we have a new extension

for Azure too: GitHub Copilot for Azure.

You can instantly deploy to Azure

to get information about your Azure resources

just using natural language.

And what Copilot did for coding.

We are now doing for infra and ops.

To show you all this in action

here is Neha from our GitHub team.

Neha, take it away.

Thanks Satya.

GitHub Copilot gives you suggestions

in your favorite editor like here

where I'm writing unit tests.

Copilot is great, at meeting you

where you're at regardless of the language

you're most comfortable with.

So, let's ask for something simple,

like how to write a prime number test in Java?

But, let's converse in Spanish using my voice.

How to check if the given number,

is a prime number in java?

Look at that.

Thank you, Copilot.

Copilot is great at turning natural language

into code and back again.

But, what about beyond the code

with the new GitHub Copilot Extensions,

you can now bring the context

from your connected systems to you.

So, now I can ask Azure,

where my app is deployed.

I could ask

what my available Azure resources are

or I could diagnose issues with my environment.

And this isn't just for Azure.

As Satya announced,

any developer can now

create Extensions for GitHub Copilot,

and that includes any tool in your stack.

Include your in-house tools,

keeping you in the flow across your entire day.

Actually, 75% of a developer's day

is spent outside of coding:

gathering requirements,

writing specifications, and creating plans.

Let's show how GitHub Copilot can help with that.

Live, on stage, for the first time.

So typically,

my day starts by looking at GitHub issues.

Looks like we want to support a rich text

input for our product description.

Let's open Workspace and get some help with that.

Copilot interprets

the intent of the issue to see what's required.

And it then looks across the entire codebase,

and it proposes what changes should be made.

This specification is fully editable,

and the whole process is iterative,

but actually, this looks pretty good.

Copilot can now help us build a plan

on how to implement this change.

All right,

that's a great start,

but we must not forget about our documentation.

So let's edit the plan,

and have Copilot update our readme.

And then we can even get Copilot’s help

in starting to implement the code for us.

Now, this was just a simple example,

but in a large enterprise codebase,

there are tens of thousands of files,

and dozens of stakeholders involved.

And that means meetings. So many meetings.

Workspace helps you focus

on what you need to change.

And by the way, as a developer, I'm always in control.

I can see exactly what changes Copilot is proposing

and I can even get a live preview.

All right. Let's test out the input.

All right.

This looks great.

So, I can go back and I can edit my code,

in VS Code,

or I can submit these changes as a pull request

to share with my team.

GitHub Copilot,

Copilot Extensions, and Copilot Workspace

help you stay focused on solving problems

and keeping you in the flow.

Back to you, Satya.

Thank you so much, Neha!

I tell you GitHub, Copilot

and everything that that ecosystem is doing

is just bringing back a lot of fun

and a lot of joy back to coding.

And really the thing about staying in that flow,

is I think what we all have dreamt for,

and dreamt about, and it's coming back.

That brings us to the very top of the stack.

Microsoft Copilot.

We built Copilot so that you have the ability

to tap into the world’s knowledge

as well as the knowledge

inside of your organization and act on it.

Now, Copilot has had a remarkable impact.

It's democratizing expertise across organizations.

It's having a real cascading effect.

Right. In fact,

it reminds me

like of the very beginning of the PC era

where work, the work artifact and the workflow.

We're all changing.

And it's just,

you know, really having broad

enterprise business processes impact.

It's lowering, I always say this, it’s lowering

both the floor

and raising the ceiling at the same time,

for anything any one of us can do.

Since no two business processes are the same

with Copilot Studio, you now can extend Copilot

to be able to customize it,

for your business processes and workflows.

Today we're introducing

Copilot connectors in Copilot Studio,

so you can ground Copilot with data

from across the Graph, from Power Platform.

Fabric, Dataverse,

as well you now have all the third-party

connectors for SaaS applications

from Adobe, Atlassian, ServiceNow,

Snowflake and many, many more.

This makes the process of grounding Copilot

in first and third-party line of business data.

Just a wizard-like experience enabling you

to quickly incorporate your own organizational

knowledge in data.

We're also extending Copilot

beyond a personal assistant

to become a team assistant.

I'm thrilled today to announce Team Copilot.

You'll be able to invoke a Team Copilot

wherever you collaborate in Teams, right?

It can be in Teams,

it can be in Loop, it can be Planner and many,

many other places. I mean, think about it, right?

It can be your meeting facilitator,

when you're in Teams,

creating agendas, tracking time, taking notes for you.

Or a collaborator writing chats,

surfacing the most important information,

tracking action items, addressing unresolved issues.

And it can even be your project manager,

ensuring that every project that you're

working on as a team is running smoothly.

These capabilities will all come to you

all and be available in preview later this year.

And we're not stopping there.

With Copilot Studio,

anyone can build copilots

that have agent capabilities

and work on your behalf, and independently,

and proactively orchestrate tasks for you.

Simply provide your Copilot a job description

or choose from one of our pre-made templates

and equip it with the necessary knowledge

and actions,

and Copilot will work in the background

and act asynchronously for you.

That's I think, one of the key things

that's going to really change in the next year

where you're going to have Copilots plus agents

with this async behavior.

You can delegate authority to Copilots

to automate long running business processes.

Copilot can even ask for help

when it encounters situations

that he does not know much about and it can’t handle.

And to show you all of this, let's roll the video.

Redefine business processes

with Copilot Studio.

Create copilots that act as agents

working independently for you.

Simply describe what you want your copilot to do.

Easily configure your copilot with the details it needs

like instructions, triggers,

knowledge and actions.

Quickly test your copilot before you deploy,

and seamlessly publish across

multiple channels.

Watch it use memory for context,

reason over user input,

and manage long running tasks.

Copilot can learn from feedback to improve,

and you're always in control.

Put copilot to work for you.

with Copilot Studio.

You know all around

this stack

is perhaps

one of the most important things that we at

Microsoft are doing, which is wrapping it

with robust security.

You know, security underlies

our approach with Copilot, Copilot+PCs,

Copilot Stack.

We're committed to our Secure Future Initiative.

You can see,

you will see us make rapid progress

across each of the six pillars of SFI,

and the core design principles, right?

Which is secure by design,

secure by default and secure operations.

You'll hear about throughout this conference.

In fact, a lot more in Scott's keynote tomorrow,

how it underlies everything

that we build and everything that we do.

So, coming to the close, I want to sort of,

you know,

there are many announcements

that you will hear about at Build,

but I want to go back to

I think the core of what I think

why we chose to be in this industry

and why we come to work every day as developers,

which is the mission ultimately of

empowering every person and every organization.

At the end of the day, it's not about innovation,

that is only useful for a few.

It's about really being able to empower that everyone,

And it comes down to you

all as developers and builders of this new world.

For us, it's never,

never about celebrating tech for tech's sake.

It's about celebrating what we can do

with technology

to create magical experiences

that make a real difference in our countries,

in our companies, in our communities.

Already, this new generation of AI

is having an incredible impact,

thanks to all of you,

the passion you bring and the hard work you put in.

And I want to leave you with this one

unbelievable example of how

you are all building a more accessible world,

which means a lot to me using our platform and tools.

Thank you all so very much.

Enjoy the rest of Build.

Audio description is something

that enables me to be able

to watch a program or a film

and get as much out of it

as everybody else who is sighted.

A white car drives down a road.

Hands on a steering wheel.

I see art as a collective good,

I think everyone should be able to have access to art.

Audio description

really helps me get the full experience.

A portrait of a group of 17th century

civic guardsmen in Amsterdam.

The challenge, though, is that there are limited

amounts of audio descriptions

being incorporated across media and entertainment.

Tech and AI have the potential

to bring the blind and low vision community

into the fold.

So at WPP,

we really care

passionately about opening up access to content

to people in the way that they want to consume it.

The tool that I've made

is an application

that allows you to upload videos,

and on the other end, with GPT-4 with Vision and

Azure AI services

you get your video back

with spoken narrations over the top.

Kitchen scene with cat and Hellmann's mayonnaise.

This makes audio descriptions cheaper and faster.

Our goal is to be able to offer this product

as a service for all of our advertisement campaigns.

There are so many artworks in the Rijksmuseum,

there are almost a million.

To describe ourselves,

it would have taken hundreds of years.

With AI, we can do this in a matter of hours.

The subject is a male,

with a reddish beard and mustache,

visible brushstrokes that add texture and mood.

The first time I heard audio descriptions

it just brought me delight.

It was this opportunity of “Oh my gosh, I'm seen.”

Through the power of AI we’re able to do things

only dreamt about until recently.

When we strengthen

our access to culture,

we strengthen the culture itself,

connecting our shared humanity.