Full Keynote: Satya Nadella at Microsoft Build 2024
Summary
TLDRThe video script outlines the transformative impact of AI across various sectors, emphasizing the accessibility of expertise through personal AI assistants like GitHub Copilot. It showcases AI's potential in enhancing productivity, creativity, and learning, highlighting innovations from Microsoft, including AI-first PCs, Azure AI services, and tools like Copilot Studio for customizing AI workflows. The script also underscores the importance of security and the democratizing force of AI in creating more accessible and innovative experiences.
Takeaways
- π The industry is moving towards a common vision of 'information at your fingertips', emphasizing accessibility and expertise through AI.
- π AI is becoming ubiquitous, aiding in various sectors such as farming, laboratories, boardrooms, and classrooms, highlighting its universal applicability.
- π§ GitHub Copilot is reported to save developers up to 50% of their time, allowing for increased efficiency and innovation.
- π§βπ« AI is being utilized as a personal tutor, helping individuals learn new skills, including complex subjects like banking.
- π Generative AI is being employed to improve productivity, with examples such as helping farmers analyze conditions of their ponds and boosting agricultural productivity.
- π¨ AI is unlocking creativity, providing detailed descriptions that inspire artwork, showcasing its role in the arts and imagination.
- π©βπ« Teachers are now able to create customized lesson plans with the help of AI, indicating a shift in educational approaches and tools.
- π€ The introduction of Copilot+PCs represents a new category of AI-first computers, designed to be the fastest for AI operations.
- π οΈ Windows Copilot Runtime aims to be as transformative for AI as Win32 was for graphical user interfaces, offering a new layer for developers.
- π Microsoft is focusing on democratization and sustainability, expanding AI infrastructure globally while aiming for 100% renewable energy in data centers.
- π¬ The tech stack is evolving at every level, from data center power management to edge computing, adapting to new workloads and use cases.
Q & A
What is the core vision presented for the industry in the script?
-The core vision is 'information at your fingertips', which has evolved into 'access to expertise', indicating a shift towards making AI available for everyone, everywhere.
How does the speaker describe the impact of AI on productivity and innovation?
-The speaker mentions that AI, such as GitHub Copilot, can save about 50% of time, allowing individuals to use that time for other innovative tasks.
What is the significance of the new era of AI mentioned in the script?
-The new era of AI is significant because it represents a shift from mere access to information to providing access to expertise, which can be utilized across various sectors like farming, laboratories, boardrooms, and classrooms.
How does AI technology facilitate learning according to the script?
-AI technology serves as a personal tutor, helping individuals learn new skills such as banking, loan applications, and saving money, with the potential to reimagine student learning worldwide.
What role does Generative AI play in enhancing productivity and insights?
-Generative AI can learn from data to improve productivity, such as helping farmers improve productivity, and providing insights in near real-time through personal coaching.
How is AI contributing to creativity, as described in the script?
-AI is unlocking creativity by providing detailed descriptions that fuel imagination, aiding in activities like painting artwork, and allowing teachers to create lesson plans tailored to specific needs.
What is the significance of the introduction of 'Copilot+PCs' in the script?
-The introduction of 'Copilot+PCs' signifies the creation of the fastest AI-first PCs, which are part of the new platforms being developed to support AI applications and solutions.
What is the role of the 'Windows Copilot Runtime' in the tech stack?
-The 'Windows Copilot Runtime' is introduced to make Windows the best platform for building AI applications, similar to how Win32 was pivotal for the graphical user interface.
How does the speaker describe the advancements in AI infrastructure?
-The speaker describes the AI infrastructure advancements as comprehensive and scalable, with Azure being built as the world's computer, offering the most complete selection of AI accelerators and focusing on sustainability.
What is the significance of the partnership with NVIDIA and AMD in the AI infrastructure?
-The partnership with NVIDIA and AMD is significant as it brings the latest AI accelerators to Azure, offering the best performance and cost for AI workloads, and enabling advanced AI capabilities like confidential compute on GPUs.
How does the script emphasize the importance of democratization in technology?
-The script emphasizes democratization by showcasing how AI models developed in one part of the world can be used to improve lives in another, and by making AI capabilities more accessible to developers globally.
Outlines
π€ AI Transformation and Accessibility
The speaker introduces the concept of 'information at your fingertips' and discusses the evolution of AI to become accessible expertise for everyone. They highlight the impact of AI on various sectors like farming, education, and business, and share personal experiences of time saved through GitHub Copilot. The transformative potential of AI in reimagining learning and productivity is emphasized, along with its ability to unlock creativity and enhance daily tasks.
π Accelerating AI Capabilities
This paragraph delves into the rapid advancements in AI, comparing the scaling laws of Deep Neural Networks (DNNs) to Moore's Law. The speaker discusses the new natural user interface that supports multimodal input and output, the importance of memory in AI for context retention, and the new capabilities in reasoning and planning. They also reflect on the significant changes brought about by developers utilizing AI to improve lives globally, exemplified by a rural Indian farmer's use of GPT-3.5.
π» Introducing Copilot+ PCs and Windows AI Integration
The speaker announces the introduction of Copilot+ PCs, highlighting their AI-first design and the integration of AI as a first-class namespace for Windows. They discuss the Windows Copilot Runtime and its significance for building AI applications, as well as the new Windows Copilot library that facilitates AI integration with local APIs. The paragraph also mentions the introduction of native PyTorch support and the WebNN framework through Windows DirectML.
π Expanding AI Infrastructure and Sustainability
The speaker outlines the expansion of Microsoft's AI infrastructure, emphasizing the company's commitment to a sustainable, scalable, and comprehensive global presence. They discuss the optimization of power and efficiency, the use of AI accelerators, and collaborations with industry leaders like NVIDIA and AMD. The paragraph also touches on the integration of enterprise platforms like Omniverse Cloud and the strategic partnership with OpenAI.
π οΈ Democratizing AI with Diverse Models and Tools
This paragraph focuses on the democratization of AI through a broad selection of models and tools. The speaker mentions partnerships with various AI model providers and the integration of open-source models via Azure AI. They also discuss the introduction of small language models (SLMs) like the Phi-3 family and their cost-effectiveness and performance, as well as the new partnership with Khan Academy to enhance math tutoring.
π‘οΈ Prioritizing AI Safety and Customization
The speaker discusses the importance of AI safety and the new capabilities in Azure AI Studio for developing and safeguarding AI models responsibly. They announce the general availability of Azure AI Studio and introduce Azure AI custom models for training domain-specific models. The paragraph also covers the integration of AI with data platforms and the introduction of real-time intelligence in Microsoft Fabric.
π§ Enhancing Developer Experience with GitHub Copilot Extensions
The speaker introduces GitHub Copilot Extensions, which allow for customization of GitHub Copilot with third-party services. They demonstrate how Copilot can assist with coding and non-coding tasks, such as gathering requirements and creating plans, by staying in the flow across the entire development process. The paragraph also showcases how developers can create extensions for any tool in their stack.
π€ Empowering Teams with Team Copilot and Security
The speaker announces Team Copilot, an extension of the personal assistant concept to team collaboration. They discuss how Team Copilot can facilitate meetings, manage projects, and serve as a team assistant across various platforms. The paragraph also emphasizes the security measures underlying Microsoft's approach to AI, including the Secure Future Initiative and its core design principles.
π¨ AI for a More Accessible World
The speaker concludes with an inspiring example of how AI is being used to create audio descriptions for the visually impaired, making media and art more accessible. They highlight the impact of AI in strengthening culture and shared humanity by providing audio descriptions for art galleries and advertising campaigns, showcasing the power of AI to enhance inclusivity and experience.
Mindmap
Keywords
π‘AI Transformation
π‘GitHub Copilot
π‘Personal Assistant
π‘Generative AI
π‘Expertise at Your Fingertips
π‘AI-first PCs
π‘Azure
π‘Moore's Law
π‘Distributed Synchronous Data Parallel Workloads
π‘Natural User Interface
π‘Democratization of AI
Highlights
The vision of 'information at your fingertips' has evolved into 'access to expertise' in the new era of AI.
AI is transforming various sectors including farming, labs, business, and education by providing personalized assistance and tutoring.
GitHub Copilot is reported to save developers up to 50% of their time, enabling them to focus on innovation.
AI's potential to reimagine student learning worldwide through personalized tutoring is highlighted.
Generative AI is aiding farmers by learning from data to improve productivity.
AI is unlocking creativity, providing detailed descriptions that inspire artwork creation.
Teachers can now create lesson plans tailored to needs with AI providing expertise at their fingertips.
Microsoft Build conference marks a fundamental change in the tech industry, with every layer of the tech stack being reshaped by AI.
Breakthroughs in AI have been driven by scaling laws similar to Moore's Law, but with more rapid advancements every six months.
New natural user interfaces support multimodal input and output, including text, speech, images, and video.
Developers are leveraging AI capabilities to make a global impact, as demonstrated by the use of GPT-3.5 in rural India.
Microsoft introduced Copilot+PCs, the fastest AI-first PCs, and the Windows Copilot Runtime to support AI application development.
Azure's AI infrastructure is expanding globally, with a focus on sustainability and efficiency.
Microsoft is partnering with NVIDIA to offer Azure confidential compute on GPUs and integrate NVIDIA's enterprise platforms with Azure.
AMD's MI300X AI accelerator is now available in Azure, offering the best price-performance ratio for GPT-4 inference.
Microsoft Fabric is introducing Real-Time Intelligence, enabling actionable insights across the entire data estate.
GitHub Copilot Extensions allow for customization with third-party services, enhancing the developer experience.
Team Copilot is announced, extending assistance to team collaborations within Microsoft Teams and other platforms.
Copilot Studio enables the creation of custom AI agents that can work independently and manage tasks asynchronously.
Microsoft's Secure Future Initiative underpins the security of all AI and developer tools, ensuring they are secure by design, default, and operation.
The impact of AI in making the world more accessible, such as through audio descriptions for the visually impaired, is showcased.
Transcripts
I think that our industry
has to have a common vision.
It was a time that connected us
to incredible things.
My name for this vision is,
information at your fingertips.
And three decades later,
we find ourselves in a new era.
One where access to information
becomes access to expertise.
From the farm, to the lab,
from the boardroom, to the classroom,
this new generation of AI
is for everyone, everywhere.
Now, anyone can save time
with a personal assistant.
With GitHub Copilot,
Iβm saving about 50% of time.
And that's time that I can use
to do other innovative things.
It allows me to find out
the condition of my ponds faster.
Anyone can access
a personal tutor to learn new skills.
We got to learn about banking:
How to apply for a loan,
how to save money.
We learned so much.
I think this technology
has the potential to completely reimagine
the way every single student learns in the world.
This is a new way to analyze
with a personal coach.
We're going to be able to have
not only productivity gains,
but insights served to us, near real-time.
Generative AI can learn from the data
to help improve the farmer productivity.
AI is unlocking creativity for us all.
Descriptions are so detailed,
in my imagination
I can paint the artwork.
Now teachers are free to create
lesson plans according to our needs.
With expertise at your fingertips.
You can build,
what matters.
Welcome to the age of AI transformation.
Good morning.
Good morning.
It's fantastic to be back here at Microsoft Build.
Welcome to everyone here and joining us on the web.
You know, developer conferences
are always most exciting, most fun
when there's these fundamental changes
that you can sense in the air.
You know, I've marked all my adult life
by coming to PDCs and Builds
for the last three decades.
I still remember,
you know, distinctly
the first time Win32 was discussed,
I guess it was β91,
.NET, Azure, right?
These are moments that I marked my life with.
And
it just feels like we're yet again
at a moment like that.
It's just that
the scale,
the scope is so much deeper,
so much broader this time around, right?
Every layer of this tech stack is changing,
you know, from everything from the power draw
and the cooling layer of the data center
to the NPUs at the Edge
are being shaped by these new workloads, right?
These distributed,
synchronous, data parallel workloads
are reshaping every
layer of the tech stack.
But if you think about
even going all the way back
to the beginning of modern
computing, say, 70 years ago
there have been two real dreams we've had.
First
is can computers understand us instead of us
having to understand computers?
And second,
in a world
where we have this ever increasing information
of people, places and things, right?
So, as you digitize more artifacts
from people, places and things
and you have more information,
can computers
help us
reason, plan
and act more effectively on all that information?
Those are the two dreams that we've had
for the last 70-plus years.
And here we are.
I think that we have real
breakthroughs on both fronts.
The core underlying force,
one of the questions I always ask myself is like,
βOkay, this is great.
This is like maybe the golden age of systems.
What's really driving it?β
I always come back to these scaling laws,
just like Moore's Law,
you know, helped drive the information revolution.
The scaling laws of DNNs
are really, along with the model architecture,
interesting ways to use data,
generate data,
are really driving this intelligence revolution.
You could say Moore's Law was probably,
you know, more stable in the sense
that it was scaling at maybe 15 months, 18 months.
We now have these things that are scaling
every six months or doubling every six months.
You know, what we have, though,
with the effect of these scaling
laws is a new natural user interface that's multimodal.
That means supports text,
speech, images, video as input and output.
We have memory
that retains important
context, recalls
both our personal knowledge and data
across our apps and devices.
We have new reasoning and planning capabilities
that helps us understand very complex context
and complete complex tasks.
While reducing the cognitive load on us.
But what stands out for me
as I look back at this past year
is how you all as developers have taken
all of these capabilities
and applied them, quite frankly,
to change the world around us.
Iβll always remember this moment in January 2023
when I met a rural Indian farmer
who was able to reason over some government
farm subsidies
that he had heard about on television
using GPT-3.5 and his voice.
It was remarkable right?
For me, it just brought home the power of all of this
because a frontier model
developed in the West Coast of the United States
just a few months
earlier was used by a developer in India
to directly improve the life of a rural Indian farmer.
The rate of diffusion
is unlike anything
I've seen in my professional career,
and it's just increasing.
In fact, earlier this month I was in Southeast Asia.
I was in Thailand where I met a developer
and I was having a great roundtable
and he was talking to me
about how he's using Phi-3 and GPT-4
and he was using Phi-3
to just optimize
all of the things that he was doing with RAG.
I mean, this is crazy I mean, this is unbelievable.
It had just launched a few weeks earlier
and I was there in Thailand, in Bangkok,
listening to a developer talk
about this technology as a real expert on it.
So it's just great to see the democratization force
that we love to talk about
but to witness it is just been something.
And this is, quite frankly,
the impact of why we are in this industry.
And it's what gives us,
I would say that deep meaning in our work.
So I want to start, though, with a very big thank you
to every one of you
who is really going about bringing
about this impact to the world.
Thank you all so very much.
You know, when I think about
what progress we've made
even since last time we were here at Build,
we built really three platforms.
The first is Microsoft Copilot,
which is your everyday AI companion.
It puts knowledge and expertise
at your fingertips, helps you act on it.
And we built the Copilot stack
so that you can build your AI applications
and solutions and experiences.
And just yesterday,
we introduced a new category of
Copilot+PCs, the fastest AI-first PCs ever built.
All three of these things are exciting platforms
but I want to start with Copilot+ PCs.
You know, we're exposing AI
as a first-class namespace for Windows.
This week
we are introducing the Windows Copilot Runtime
to make Windows the best platform
for you to be able to build your AI applications.
Yeah.
You know what Win32 was to graphical user interface,
we believe the Windows Copilot Runtime
will be for AI.
It starts with our Windows Copilot library,
a collection of these ready-to-use local APIs
that help you integrate into your new experiences
all of the AI capabilities that we shared yesterday.
Now, this includes
no code integrations for Studio Effects
things like creative filters,
teleprompter, voice focus, and much more.
But of course, if you want to access these models itself,
you can directly call them through APIs.
We have 40 plus models available
out of the box, including Phi-Silica
our newest member
of our small language
family model, which we can specific,
which we specifically designed to run
locally on your NPUs, on Copilot+ PCs
bringing that
lightning-fast local inference to the device.
You know, the other thing is the Copilot library
also makes it easy for you to incorporate RAG
inside of your applications on the,
on device data.
It gives you the right tools to build
a vector store within your app.
It enables you to do that
semantic search that you saw with Recall.
But now you can, in your own application, construct
these prompts using local data for RAG applications.
Now, Iβm so thrilled to announce as well today
that we will be natively supporting
PyTorch and new WebNN framework
through Windows DirectML.
Native PyTorch support means
thousands of OSS models
will just walk out of the box on Windows,
making it easy for you to get started.
In fact, with WebNN, web developers
finally have a web-native machine learning framework
that gives them direct access
to both GPUs and NPUs
in fact, last night
I was playing with it, turning it on in Edge
and seeing the WebNN sample code running.
It's just so cool to see it
you know, now use even the NPUs.
Both PyTorch and WebNN are available
in Developer Preview today, let's take a look.
And these are just
one of the many announcements today.
We're introducing more than 50-plus
new products and partnerships
to create a new opportunity for you.
Weβve always been a platform company
and our goal is to build the most complete
end-to-end stack
from infrastructure,
to data, to tooling to the application extensibility
so that you can apply the power of this technology
to build your own applications.
And so today I want to highlight
our top news for this event
across every layer of this Copilot stack.
So let's dive right in with infrastructure.
You know, we have the most complete
scalable AI infrastructure
that meets your needs in this AI era.
We're building Azure as the world's computer.
We have the most comprehensive global
infrastructure with more than 60-plus
datacenter regions,
more than any other cloud provider.
Over the past year,
we have expanded our datacenter regions
and AI capacity from Japan
to Mexico, from Spain to Wisconsin.
We're making our best-in-class AI
infrastructure available everywhere
and we're doing this with a focus
on delivering on cloud services sustainability.
In fact,
we're on track to meet our goal
to have our data centers powered
by 100% renewable energy by next year.
Yeah.
You know,
weβre optimizing power
and efficiency across every layer of the stack
from the data center to the network.
Our latest data center
designs are purpose built for these AI workloads
so that we can effectively and responsibly use
every megawatt of power to drive down the cost of AI
and the power draw.
And we are incorporating
advanced data center cooling techniques
to fit the thermal profile of the workloads
and match it to the environment
in the location where it operates.
And the silicon layer,
we are dynamically
able to map workloads
to the best accelerated AI hardware
so that we have the best performance.
And our custom IO hardware and server
designs allow us to provide dramatically faster
networking, remote storage
and local storage throughput.
You know, this
end-to-end approach
is really helping us get to the unprecedented scale.
In fact, last November
we announced the most powerful
AI supercomputer in the cloud
for training.
Using just actually a very small fraction
of our cloud infrastructure.
And over the past six months
we've added 30 times that
supercomputing power to Azure.
Yeah, it's crazy to see the scale.
And of course we're not just scaling training our fleets,
weβre scaling our inference fleet
around the world, quadrupling the number of countries
where Azure AI services are available today
and it's great to see that.
At the heart of our AI infrastructure
are the world's most advanced AI accelerators, right?
We offer the most complete selection
of AI accelerators, including from NVIDIA and AMD,
as well as our own Azure Maia,
all dynamically optimized for the workloads.
That means whether you're using Microsoft Copilot
or building your own Copilot apps,
we ensure that you get
the best accelerator performance at the best cost.
For example, you know,
you see this in what has happened with GPT-4, right?
It's 12x cheaper
and 6x faster since it launched.
And that's,
you know,
the type of progress
you can continue to see,
how, you know, you continue to see the progress
as we evolve the system architecture.
It all starts, though,
with this very deep, deep partnership with NVIDIA,
which spans the entirety of the Copilot
stack across
both all of their hardware innovation
as well as their system software innovation.
Together, we offer Azure
confidential compute on GPUs to
really help you protect sensitive data
around the AI models end to end.
We're bringing
in fact the latest H200s
to Azure later this year,
and will be among the first cloud providers
to offer NVIDIA's
Blackwell GPUs in B100
as well as GB200 configurations.
And we are continuing
to work with them to train and optimize
both large language models
like GPT-4o, as well as small language
models like the Phi-3 family.
Now beyond the hardware,
we are bringing NVIDIAβs key
enterprise platform offerings to our cloud,
like the Omniverse Cloud
and DGX Cloud to Azure
with deep integration
with even the broader Microsoft Cloud.
For example,
NVIDIA recently announced
that their DGX Cloud integrates
natively with Microsoft Fabric.
That means you can train those models using
DGX Cloud with the full access through Fabric data.
And Omniverse APIs will be available
first on Azure for developers
to build their industrial AI solutions.
We're also working
with NVIDIAβs NIM industry
specific developer services
and making them fantastic on Azure.
So, a lot of exciting work with NVIDIA.
Now, coming to AMD,
I am really excited to share
that we are the first cloud
to deliver
general availability of VMs based on AMDβs MI300X
AI accelerator.
It's a big milestone for both AMD and Microsoft.
We've been working at it for a while
and it's great to see that today
as we speak, it offers the best price performance
on GPT-4 inference.
And we'll continue to move forward with Azure Maia.
In fact, our first clusters are live and soon
if you're using Copilot
or one of the Azure OpenAI services,
some of your prompts will be served
using Maia hardware.
Now beyond AI, our end to end systems
optimization also makes cloud-native apps
and the development of cloud-native
apps better, right?
Six months ago
is when we announced our first general purpose
ARM-based compute processor Microsoft Cobalt.
And today
I am really excited to announce
the public preview of Cobalt-based Vms.
You know,
Cobalt is being used for video processing
and permissions management in Microsoft 365,
helping power billion of conversations
on services like Microsoft Teams already.
And we are delivering that same ARM-based
performance and efficiencies to many customers,
in fact, including
Elastic, Mongo, Siemens, Snowflake and Teradata.
In our most recent benchmark
data and tests, our Cobalt 100 VMs delivered up to
40 percent better performance
than any other generally available ARM-based VMs.
So we are very very excited about Cobalt
getting into the market.
Now let's move up the stack to the foundation models.
With Azure AI,
we offer the broadest selection
of frontier and open source models,
including LLMs and SLMs,
so you can choose the model
that makes the most sense for your unique needs
and your application needs.
In fact,
more than 50,000 organizations use Azure AI today.
Yeah.
It's great momentum
and it all starts
though, with our most strategic
and most important partnership with OpenAI.
Just last week, OpenAI announced GPT-4o,
for all their latest multimodal model,
which was trained on Azure.
It's an absolute breakthrough.
It has text, audio, image and video as input and output.
It can respond and just have a humanlike conversation
that's fast and fluid.
It can even be interrupted mid-sentence.
GPT-4o is also the top performing model
on benchmarks across a variety of modalities
and it's always going to get you to the best answer.
It has state-of-the-art performance
at understanding the information
you actually provide in your prompt
because that's really what matters.
What OpenAI demoed last week,
I mean, that was just pretty magical stuff.
And yesterday
we showed how Copilot will leverage GPT-4o.
In the video
I'm just going to play, notice
that you can share
your screen or session as a prompt in Copilot
and have Copilot help you in
whatever it is that you're doing.
Let's roll the video.
Hey Copilot, how's it going?
Hey, I'm doing great.
And it looks like you're about to dive
into some Minecraft.
Ready to build, explore and maybe dodge a few mobs?
Yeah, I'm trying to figure out how to make a sword.
My son plays this game all the time
and I have no idea what I'm doing.
No worries. I'm here to help you impress your son.
I see you've got a crafting table in front of you,
which is a great start.
To craft a sword
you'll need some materials.
Can you open your inventory?
Just press βEβ on your keyboard.
Yeah. Do I have what I need?
Let's see.
You've got some sticks.
Perfect for the swords handle.
But it looks like you're missing the material
for the blade.
You can use wood, stone, iron, gold or diamond.
Let's start by gathering some wood or stone.
Shall we head out and find some resources?
Yeah, that sounds great.
Oh my god, what's that?
Oh no, it's a Zombie!
Run! You need to get away from it fast.
Either sprint away or quickly
build a tower of blocks straight up
to get out of its reach.
If you can find a shelter
or dig into the side of a hill, do it now.
Just make sure it can't reach you.
Alright, I think I might be safe now.
That was a close one. Great job finding shelter.
It's really cool to see that.
You know,
the magic here is that
Copilot is your personal assistant
that's helping you with any task, helping
you stay, quite frankly, in the flow in the moment,
whether it's a game or a document or a line of code.
And as OpenAI innovates,
our promise is that we will bring all that innovation
to Azure too, in fact,
the same day that OpenAI
announced GPT-4o,
we made the model available
for testing on Azure OpenAI Service and today
I'm excited to say
that itβs generally available on Azure AI.
You know, what this means though
is that now we can have these groundbreaking apps
that all of you can build using this capability.
One of the coolest things is that now any app,
any website, can essentially be turned
into a full multi-modal,
full duplex conversational canvas.
Let's take a look.
Alright, let me ask the agent here.
Hey, Seth. How's it going?
What can I help you with today?
I am fantastic.
I'm sorry, I'm in such a hurry.
I'm going to go on an
overnight camping hike,
where we're camping -
it's going to be dark, cold.
Should I use these shoes?
It's the mountains.
It's probably going to be cold.
Hey, no worries.
Let me take a look.
Ah, those are the TrekStar hiking sandals.
While they're fantastic for summer hikes
and keeping your feet cool,
they might not be the best choice for cold -
Okay, I'm sorry to interrupt you.
Pick the best thing that I should use,
and can you just add it to my cart?
Just one product.
No worries.
Let's go with the TrekReady
hiking boots.
They're durable,
provide great ankle support,
and most importantly,
they'll keep your feet warm
in the cold mountains.
I'll add them to your cart now.
Ready for checkout?
I see that, amazing.
Thank you very much.
I hope to have a good time camping.
Youβre welcome, Seth.
Enjoy camping and stay warm in those boots.
Good luck and have fun in the mountains.
While Seth was,
you know, Seth was in a hurry, but there you go.
GPT-4o kept up with him.
It's incredible.
I really want to thank the OpenAI
team for their partnership and
and really the responsible approach to innovation,
helping our industry move forward.
Sam will be here in fact, joining Kevin
in a little bit to talk a lot more about what's coming
because that's the exciting stuff,
how do you all sample what comes next.
We also,
I brought, we're bringing lots
and lots of other models
as well from Cohere and Databricks and Deci, Meta,
Mistral, Snowflake, all through Azure AI.
We want to support the broadest set of models
from every country, every language.
I'm excited to announce,
in fact, we're bringing models from Cohere,
G42, NTT DATA, Nixtla,
as well as many more, as models of services,
because that's the way
you can easily get to managed AI models.
And we all love open source, too.
In fact,
two years ago at Build,
we were the first to partner
with Hugging Face, making it simple
for you to access the leading open source library
with state-of-the-art language models
via Azure AI.
And today I'm really excited to announce
that we're expanding our partnership,
bringing more models from Hugging Face
with text generation inference,
with text embedding inference
directly into Azure AI Studio.
And, and we're not stopping there.
We are adding
not just large language models,
but we are also leading the small language revolution.
So small language model revolution,
you know, our Phi-3 family of SLMs
are the most capable and most cost effective.
They outperform models of the same size
or the next size up
even across
a variety of language
reasoning, coding, as well as math benchmarks.
If you think about it by performance
to parameter count ratio, it's truly best in class.
And today we're adding new models
to the Phi-3 family
to add even more flexibility
across that quality cost curve.
We're introducing Phi-3 Vision,
a 4.2 billion parameter
multimodal model with language
and vision capabilities.
It can be used to reason our real-world images so
generate insights and answer questions about images.
As you can see right here. Yeah.
And we're also making a 7 billion parameter
Phi-3 small in a 14 billion parameter
Phi-3 medium models available.
With Phi,
you can build apps that span the web,
your Android, iOS, Windows and the Edge.
They can take advantage of local hardware
when available and fall back on the cloud.
We're not simplifying really
all of what
we as developers have to do to support
multiple platforms using one AI model.
Now, it's just awesome
to see how many developers are already using
Phi0-3 to, you know, do incredible things.
From Amity Solutions, the Thai company
that I mentioned earlier,
the ITC, which is been
built a copilot for Indian farmers
to ask questions
about their crops.
Epic in health care
which is now using Phi to summarize complex
patient histories more quickly
and efficiently.
And out of the very,
very cool use cases in education.
Today, I'm very thrilled to announce
a new partnership with Khan Academy.
We'll be working together to use Phi-3
to make math tutoring more accessible.
And I'm also excited to share
that they'll be making Khanmigo
their AI assistant free to all US teachers.
Let's roll the video.
I felt like I was in a place in my teaching career
where I felt like I was kind of losing my sparkle.
And I would just feel really defeated
when I looked out on the classroom
and I would see students
that just didn't look engaged.
Teachers have an incredibly hard job
and what we think we can do
is leverage technology
to take some of the
stuff off of their plate,
to really actually humanize the classroom.
By some miracle, we became
a Khanmigo pilot school.
With new advances in generative AI,
we launched Khanmigo.
The point is to be that personalized tutor
for every student
and to be a teaching assistant for every teacher.
I started to build these more robust lessons
and I started to see my students engage.
We're working with Microsoft
on these Phi models
that are specifically tuned for math tutoring.
If we can make a small language model like Phi,
work really well in that use case,
then we would like to, kind of, shift the traffic to Phi
in those particular scenarios.
Using a small language model,
the cost is a lot lower.
We're really excited that Khanmigo,
and especially in the partnership with Microsoft,
being able to give these teacher tools
for free, to U.S. teachers
is going to make a dramatic impact
on U.S. education.
I think we're going to make them the innovators,
the questioners, isn't that really
just why you wake up every morning?
Right? Because that's our future,
our next generation.
And to me, that's everything.
You know, Iβm super excited to see the impact
this all will have and what Khan Academy will do.
And Sal is going to, in fact,
join Kevin soon to share more.
And I'm really thankful for Teachers
like Melissa and everything that they do.
Thank you very much.
You know, of course,
it's about more than just models.
It's about the tools
you need to build these experiences.
With Azure AI Studio
we provide an end-to-end
tooling solution to develop and safeguard
the copilot apps you build.
We also provide tooling and guidance
to evaluate your AI models
and applications
for performance and quality,
which is one of the most important tasks
as you can imagine with all of these models.
And I'm excited to announce
that Azure AI Studio now is generally available.
It's an end to end
development environment to build, train,
and fine tune AI models β and do so responsibly.
It includes built-in support.
For what is perhaps the most important feature,
which is, in this age of AI,
which is AI Safety.
Azure AI Studio
includes the state of the art safety tooling.
You know,
to everything from detecting hallucinations
in model outputs, risk and safety monitoring.
It helps understand
which inputs and outputs are triggering
content filters. Prompt shields, by the way,
to detect and block these prompt injection attacks.
And so today
we are adding
new capabilities, including custom categories,
so that you can create these unique filters
for prompts and completions
with rapid
deployment options,
which I think is super important
as you deploy these models into the real world.
Even when an emerging threat is, you know, appears.
Beyond Azure AI Studio,
we recognize that there are advanced applications
where you need
much more customization
of these models for very specific use cases.
And today
I'm really excited to announce that
Azure AI custom models
will come, giving you the ability
to train a custom model
that's unique to your domain, to your data, that's
perhaps proprietary.
That's same builders and data scientists
whoβve been working with
Open AI, brought all the Phi
advances to you, will work
with all of you to be able
to build out these custom models.
The output will be domain specific.
It will be multitask
and multimodal,
best in class as defined by benchmarks,
including perhaps even specific language proficiency
that may be required.
Now, let's just roll up the stack to data.
Ultimately,
in order to train
fine-tune, ground your models,
you need your data to be in its best shape.
And to do so, we are building out the full data estate
right from operational stores to analytics in Azure.
Weβve also added
AI capabilities
to all of our operational stores,
whether it's Cosmos DB or SQL, or PostgreSQL.
At the core though,
of the Intelligent Data Platform.
Is Microsoft Fabric.
We now have over 11,000 customers,
including leaders in every industry whoβre using Fabric.
It's fantastic to see the progress.
With Fabric,
you get everything you need in a single integrated
SAS platform.
It's deeply integrated at its most fundamental level
with compute and storage being unified.
Your experience is
unified, governance is unified, and more importantly,
the business model is unified.
And what's also great about Fabric
is that it works with data anywhere, right?
Not just on Azure,
but it can be on AWS or on GCP
or even in your on-premise data center.
And today we are taking the next step.
We're introducing Real-Time Intelligence in Fabric.
Customers today have
more and more of this real-time
data coming from your IoT systems,
your telemetry systems. In fact, cloud
applications themselves are generating lots of data,
but with Fabric, anyone can unlock
actionable insights across all of your data estate.
Let's take a look.
Introducing
real-time intelligence
in Microsoft Fabric,
an end-to-end solution
empowering you to get instant
actionable insights
on streaming data.
At its heart lies
a central place to discover,
manage, and consume event data
across your entire organization
with a rich governed experience.
Get started quickly
by bringing in data
from Microsoft sources
and across clouds with a variety
of out-of-the-box connectors.
Route the relevant data to
the right destination in Fabric
using a simple
drag-and-drop experience.
Explore insights on petabytes
of streaming data
with just a few clicks.
Elevate your analysis
by harnessing the intelligence
of Copilot in Microsoft Fabric
using simple natural language.
Make efficient business decisions
in the moment,
with real-time actionable insights
and respond to changing landscapes proactively.
Allow users to monitor
the data they care about,
detect changing patterns,
and set alerts or actions
that drive business value.
All your data, all your teams,
all in one place.
This is Microsoft Fabric.
And, we're making it
even easier to design, build and interoperate
with Fabric with your own applications, right?
And in fact, we're building out a new app platform
with Fabric Workload Development Kit
so that people like ESRI, for example, having,
you know, who have integrated
their spatial analytics with Fabric
so that customers can generate insights
from their own location
data using ESRIβs rich tools
and libraries, right on Fabric, right.
This is just exciting to see
As the first time,
you know,
where the analytics stack is really a first-class
app platform as well.
And beyond Fabric,
we are integrating the power of AI across
the entirety of the data stack.
There's no question that RAG is core to any
AI-powered application,
especially in the enterprise today.
And Azure AI Search makes it possible
to run RAG at any scale,
delivering very highly accurate responses
using the state of the art retrieval systems.
In fact, ChatGPT supports for GPTs, their Assistants
API, are all powered by Azure AI Search today.
And with built-in OneLake integration.
Azure AI Search will automatically
index your unstructured data too.
And it's also integrated into Azure
AI Studio to support
bringing your own embedding model, for example.
And so it's pretty incredible
to see Azure Search grow over the last year
into that very core developing service.
Now let's go up through developer tools.
Nearly 50 years after our
founding as a developer tools company,
here, we are once again
redefining software development, right?
GitHub Copilot
was the first, I would say, hit product
of this generative AI age.
And it's the most widely adopted AI
developer tool, 1.8 million subs
across 50,000 organizations, are using it.
And GitHub Copilot,
we're empowering
every developer on the planet to be able to access
programing languages and programing knowledge
in their own native language.
Think about that.
Any person can start programing,
whether it's in Hindi
or Brazilian Portuguese,
and then bring back
the joy of coding to their native language.
And with Copilot Workspace, staying in your flow
has never been easier.
We are an order of magnitude closer
to a world
where any person can go from idea to code
in an instant.
You start with an issue,
it creates a spec
based on its deep understanding of your codebase.
It then creates a plan
which you can execute to generate the code
across the full repo that is multiple files.
At every point in this process β from the issue, to spec,
to plan, to code, you are in control, you can edit it.
And that's really what is fundamentally
a new way of building software.
And we are looking forward
to making it much more broadly
available in the coming months.
And today, we are taking one more big leap forward.
You know, we are bridging the broader developer
tools and services ecosystem
with Copilot for the first time.
We are really thrilled to be announcing
GitHub Copilot Extensions.
Now you can
customize GitHub Copilot
with capabilities from third-party services,
whether it's Docker, Sentry, and many, many more.
And of course we have a new extension
for Azure too: GitHub Copilot for Azure.
You can instantly deploy to Azure
to get information about your Azure resources
just using natural language.
And what Copilot did for coding.
We are now doing for infra and ops.
To show you all this in action
here is Neha from our GitHub team.
Neha, take it away.
Thanks Satya.
GitHub Copilot gives you suggestions
in your favorite editor like here
where I'm writing unit tests.
Copilot is great, at meeting you
where you're at regardless of the language
you're most comfortable with.
So, let's ask for something simple,
like how to write a prime number test in Java?
But, let's converse in Spanish using my voice.
How to check if the given number,
is a prime number in java?
Look at that.
Thank you, Copilot.
Copilot is great at turning natural language
into code and back again.
But, what about beyond the code
with the new GitHub Copilot Extensions,
you can now bring the context
from your connected systems to you.
So, now I can ask Azure,
where my app is deployed.
I could ask
what my available Azure resources are
or I could diagnose issues with my environment.
And this isn't just for Azure.
As Satya announced,
any developer can now
create Extensions for GitHub Copilot,
and that includes any tool in your stack.
Include your in-house tools,
keeping you in the flow across your entire day.
Actually, 75% of a developer's day
is spent outside of coding:
gathering requirements,
writing specifications, and creating plans.
Let's show how GitHub Copilot can help with that.
Live, on stage, for the first time.
So typically,
my day starts by looking at GitHub issues.
Looks like we want to support a rich text
input for our product description.
Let's open Workspace and get some help with that.
Copilot interprets
the intent of the issue to see what's required.
And it then looks across the entire codebase,
and it proposes what changes should be made.
This specification is fully editable,
and the whole process is iterative,
but actually, this looks pretty good.
Copilot can now help us build a plan
on how to implement this change.
All right,
that's a great start,
but we must not forget about our documentation.
So let's edit the plan,
and have Copilot update our readme.
And then we can even get Copilotβs help
in starting to implement the code for us.
Now, this was just a simple example,
but in a large enterprise codebase,
there are tens of thousands of files,
and dozens of stakeholders involved.
And that means meetings. So many meetings.
Workspace helps you focus
on what you need to change.
And by the way, as a developer, I'm always in control.
I can see exactly what changes Copilot is proposing
and I can even get a live preview.
All right. Let's test out the input.
All right.
This looks great.
So, I can go back and I can edit my code,
in VS Code,
or I can submit these changes as a pull request
to share with my team.
GitHub Copilot,
Copilot Extensions, and Copilot Workspace
help you stay focused on solving problems
and keeping you in the flow.
Back to you, Satya.
Thank you so much, Neha!
I tell you GitHub, Copilot
and everything that that ecosystem is doing
is just bringing back a lot of fun
and a lot of joy back to coding.
And really the thing about staying in that flow,
is I think what we all have dreamt for,
and dreamt about, and it's coming back.
That brings us to the very top of the stack.
Microsoft Copilot.
We built Copilot so that you have the ability
to tap into the worldβs knowledge
as well as the knowledge
inside of your organization and act on it.
Now, Copilot has had a remarkable impact.
It's democratizing expertise across organizations.
It's having a real cascading effect.
Right. In fact,
it reminds me
like of the very beginning of the PC era
where work, the work artifact and the workflow.
We're all changing.
And it's just,
you know, really having broad
enterprise business processes impact.
It's lowering, I always say this, itβs lowering
both the floor
and raising the ceiling at the same time,
for anything any one of us can do.
Since no two business processes are the same
with Copilot Studio, you now can extend Copilot
to be able to customize it,
for your business processes and workflows.
Today we're introducing
Copilot connectors in Copilot Studio,
so you can ground Copilot with data
from across the Graph, from Power Platform.
Fabric, Dataverse,
as well you now have all the third-party
connectors for SaaS applications
from Adobe, Atlassian, ServiceNow,
Snowflake and many, many more.
This makes the process of grounding Copilot
in first and third-party line of business data.
Just a wizard-like experience enabling you
to quickly incorporate your own organizational
knowledge in data.
We're also extending Copilot
beyond a personal assistant
to become a team assistant.
I'm thrilled today to announce Team Copilot.
You'll be able to invoke a Team Copilot
wherever you collaborate in Teams, right?
It can be in Teams,
it can be in Loop, it can be Planner and many,
many other places. I mean, think about it, right?
It can be your meeting facilitator,
when you're in Teams,
creating agendas, tracking time, taking notes for you.
Or a collaborator writing chats,
surfacing the most important information,
tracking action items, addressing unresolved issues.
And it can even be your project manager,
ensuring that every project that you're
working on as a team is running smoothly.
These capabilities will all come to you
all and be available in preview later this year.
And we're not stopping there.
With Copilot Studio,
anyone can build copilots
that have agent capabilities
and work on your behalf, and independently,
and proactively orchestrate tasks for you.
Simply provide your Copilot a job description
or choose from one of our pre-made templates
and equip it with the necessary knowledge
and actions,
and Copilot will work in the background
and act asynchronously for you.
That's I think, one of the key things
that's going to really change in the next year
where you're going to have Copilots plus agents
with this async behavior.
You can delegate authority to Copilots
to automate long running business processes.
Copilot can even ask for help
when it encounters situations
that he does not know much about and it canβt handle.
And to show you all of this, let's roll the video.
Redefine business processes
with Copilot Studio.
Create copilots that act as agents
working independently for you.
Simply describe what you want your copilot to do.
Easily configure your copilot with the details it needs
like instructions, triggers,
knowledge and actions.
Quickly test your copilot before you deploy,
and seamlessly publish across
multiple channels.
Watch it use memory for context,
reason over user input,
and manage long running tasks.
Copilot can learn from feedback to improve,
and you're always in control.
Put copilot to work for you.
with Copilot Studio.
You know all around
this stack
is perhaps
one of the most important things that we at
Microsoft are doing, which is wrapping it
with robust security.
You know, security underlies
our approach with Copilot, Copilot+PCs,
Copilot Stack.
We're committed to our Secure Future Initiative.
You can see,
you will see us make rapid progress
across each of the six pillars of SFI,
and the core design principles, right?
Which is secure by design,
secure by default and secure operations.
You'll hear about throughout this conference.
In fact, a lot more in Scott's keynote tomorrow,
how it underlies everything
that we build and everything that we do.
So, coming to the close, I want to sort of,
you know,
there are many announcements
that you will hear about at Build,
but I want to go back to
I think the core of what I think
why we chose to be in this industry
and why we come to work every day as developers,
which is the mission ultimately of
empowering every person and every organization.
At the end of the day, it's not about innovation,
that is only useful for a few.
It's about really being able to empower that everyone,
And it comes down to you
all as developers and builders of this new world.
For us, it's never,
never about celebrating tech for tech's sake.
It's about celebrating what we can do
with technology
to create magical experiences
that make a real difference in our countries,
in our companies, in our communities.
Already, this new generation of AI
is having an incredible impact,
thanks to all of you,
the passion you bring and the hard work you put in.
And I want to leave you with this one
unbelievable example of how
you are all building a more accessible world,
which means a lot to me using our platform and tools.
Thank you all so very much.
Enjoy the rest of Build.
Audio description is something
that enables me to be able
to watch a program or a film
and get as much out of it
as everybody else who is sighted.
A white car drives down a road.
Hands on a steering wheel.
I see art as a collective good,
I think everyone should be able to have access to art.
Audio description
really helps me get the full experience.
A portrait of a group of 17th century
civic guardsmen in Amsterdam.
The challenge, though, is that there are limited
amounts of audio descriptions
being incorporated across media and entertainment.
Tech and AI have the potential
to bring the blind and low vision community
into the fold.
So at WPP,
we really care
passionately about opening up access to content
to people in the way that they want to consume it.
The tool that I've made
is an application
that allows you to upload videos,
and on the other end, with GPT-4 with Vision and
Azure AI services
you get your video back
with spoken narrations over the top.
Kitchen scene with cat and Hellmann's mayonnaise.
This makes audio descriptions cheaper and faster.
Our goal is to be able to offer this product
as a service for all of our advertisement campaigns.
There are so many artworks in the Rijksmuseum,
there are almost a million.
To describe ourselves,
it would have taken hundreds of years.
With AI, we can do this in a matter of hours.
The subject is a male,
with a reddish beard and mustache,
visible brushstrokes that add texture and mood.
The first time I heard audio descriptions
it just brought me delight.
It was this opportunity of βOh my gosh, I'm seen.β
Through the power of AI weβre able to do things
only dreamt about until recently.
When we strengthen
our access to culture,
we strengthen the culture itself,
connecting our shared humanity.
Browse More Related Video
![](https://i.ytimg.com/vi/jt6oq2CjaiA/hq720.jpg)
Transform productivity with AI experiences in Microsoft Fabric | OD24
![](https://i.ytimg.com/vi/vBCrJaLseQc/hq720.jpg)
90-Second Recap: Satya Nadella's Keynote at Microsoft Build 2024
![](https://i.ytimg.com/vi/XB-KFtfFiL0/hq720.jpg)
Foundation Models | Satya Nadella at Microsoft Build 2024
![](https://i.ytimg.com/vi/xBuRVG7ujQM/hq720.jpg?sqp=-oaymwEmCIAKENAF8quKqQMa8AEB-AH-CYAC0AWKAgwIABABGDggQih_MA8=&rs=AOn4CLBUdnPps1ULp5qkpAmywobp6iwd9w)
Get ready for the future of work with Microsoft Teams | Enterprise Connect 2024 Keynote
![](https://i.ytimg.com/vi/z1ycDvspv8U/hq720.jpg)
Intro to GitHub Copilot in Visual Studio
![](https://i.ytimg.com/vi/ylV2HGh22Bw/hq720.jpg)
8 Crazy AI Tools That Will Blow Your Mind π€― 2024
5.0 / 5 (0 votes)