Building for Accessibility with Azure

Summary

このビデオスクリプトでは、Azureエンジニアリング組織でアクセシビリティに特化しているElsa Leenが、開発者プラットフォームのアクセシビリティとAIの活用について語っています。Elsaは、開発者ツールのアクセシビリティ強化と、AIがアクセシビリティを向上させる方法について説明し、TeamsのライブキャプションやWindowsの音声認識機能など、マイクロソフトの製品におけるアクセシビリティ機能の例を紹介しています。さらに、AIが持つ自然言語処理能力がアクセシビリティに与える影響についても触れ、コピロットやSeeing AIなどのツールがどのように視覚障害者や学習障害者を支援するのかを解説しています。

Takeaways

• 😀 アクセシビリティとは、製品やサービスを障害を持つ人々も含めたすべての人に利用可能にすることです。
• 👥 障害を持つ人々は、恒久的な障害だけでなく、一時的または状況依存の障害も考慮に入れます。
• 🛠️ Microsoftは、開発者ツールのアクセシビリティを向上させる取り組みを行っており、Visual Studioなどがその一例です。
• 🌐 Azureエンジニアリング組織のElsaは、AIを使い障碍を抱える人々にとって開発プラットフォームをよりアクセス可能にし、AIがアクセシビリティを向上させる方法を探求しています。
• 🎥 Teamsなどのアプリでは、ライブキャプションや翻訳機能が利用でき、聴覚障害者や他の人々にとって重要な機能となっています。
• 📖 OfficeのディクテーションやRead Aloud機能は、音声をテキストに変換したり、画面の内容を読み上げたりするアクセシビリティツールです。
• 🤖 Co-pilotは、アクセシビリティを高めるためのテクノロジーであり、テキストの要約、編集、生成を支援します。
• 👓 Seeing AIは、視覚に障碍を抱える人々が周囲の世界を理解するのに役立つモバイルアプリです。
• 🔍 AIはマルチモーダルで、画像や音声を理解し、テキストに変換することができるため、さまざまなアクセシビリティのギャップを埋めることができます。
• 📈 AIはアクセシビリティを拡大し、自然言語処理を通じて個々のニーズに応じたパーソナライズされたアシスタントを提供します。
• 👂 アクセシビリティの改善は、ユーザーの包括性と使いやすさを高めることにもつながります。
• 🔗 アクセシビリティに関するフィードバックを得るためには、障碍を持つ人々とコミュニケーションを取り、彼らのニーズを理解することが重要です。

Q & A

このビデオでのエルサ・リーンの役割は何ですか？

エルサ・リーンはAzureエンジニアリング組織でデザインとアクセシビリティに焦点を当てて働いています。

アクセシビリティとは何ですか？

アクセシビリティとは、障害を持つ人々がデジタル製品やサービスを利用できるようにすることを指します。

エルサが紹介したMicrosoft製品のアクセシビリティ機能の一例は何ですか？

一例として、Teamsのライブキャプション機能があります。この機能は聴覚障害を持つ人々が会議に参加しやすくするためのものです。

一時的な障害の例を挙げてください。

一時的な障害の例として、エルサ自身が左耳の聴力が一時的に低下している状況が挙げられます。

Microsoftがアクセシビリティの重要性を強調する理由は何ですか？

Microsoftは、より多くの人々が製品やサービスを利用できるようにすることで、全ての人々にとってのインクルーシブな体験を提供することを目指しているからです。

Copilotのアクセシビリティへの影響は何ですか？

Copilotは、コンテンツの要約や編集を手助けすることで、特にディスレクシア（読み書き障害）を持つ人々にとって有益なアシスタント技術です。

Seeing AIとは何ですか？

Seeing AIは、視覚障害を持つ人々が周囲の世界をモバイルアプリを通じて理解できるようにするためのアプリです。

AIがアクセシビリティに与える主な利点は何ですか？

AIは、スケールでのアクセシビリティを容易にし、自然言語のサポートを提供することで、個々のニーズに合わせたパーソナライズを可能にします。

Microsoftの製品で使われている読み上げ機能の例を挙げてください。

例として、Microsoft EdgeやOfficeでの読み上げ機能があります。これにより、画面上のテキストを音声で読み上げることができます。

アクセシビリティ向上のためにエンジニアが考慮すべきことは何ですか？

エンジニアは、障害を持つコミュニティからのフィードバックを収集し、実際のユーザーのニーズに基づいて設計を行うべきです。

Outlines

00:00

😀 アクセス性の重要性とAIの役割

Elsa LeenはMicrosoftのAzureエンジニアリング組織で働く。彼女は開発者プラットフォームのアクセシビリティを確保し、AIを通じてアクセシビリティを向上させる方法を探求している。ビデオでは、開発者ツールのアクセシビリティを向上させる取り組みや、AIがアクセシビリティに与える恩恵について語っている。また、Microsoftのアクセシビリティの考え方や、製品やサービスを設計する際のアクセシビリティの重要性を説明している。

05:01

😉 一時的または状況依存の障害

ビデオでは一時的または状況依存の障害についても議論されている。例えば、雑音の多い環境での聴覚の障害や、車椅子での移動に関する議論がある。カーブカット（歩道の傾斜部分）の例を通じて、永久的な移動障害を持つ人々にとって非常に重要なインフラストラクチャであることが説明されている。しかし、カーブカットは車輪のスーツケースやベビーストローラー、自転車などを持つ人々にとっても非常に便利な機能であると指摘している。

10:02

🎥 AIを活用したアクセシビリティの向上

Microsoft製品におけるAIパワードなアクセシビリティツールについて紹介されている。Teamsでのライブキャプション、翻訳機能、Officeのディクテーション、Read Aloud機能などが挙げられる。これらは特定の障害を持つ人々だけでなく、一般のユーザーにも恩恵をもたらす機能である。また、CopilotというAIアシスタントがどのように障害を持つ人々にとって生産性を高める助けとなるかについても触れている。

15:03

👥 個人差に応じたアクセシビリティの重要性

アクセシビリティが個人差に応じてどのように重要であるかについて議論されている。AIが持つマルチモーダルな能力が、聴覚障害者や視覚障害者、学習障害者にとってどのように役立つかを説明している。また、AIがスケールでアクセシビリティを提供する能力についても触れており、USA Todayアプリのテキスト読み上げ機能など、実際の例を紹介している。

20:03

🤖 自然言語処理とアクセシビリティ

自然言語処理（NLP）がアクセシビリティに与える影響について語られている。CopilotのようなAIアシスタントが、自然言語を通じてどのようにアクセシビリティを向上させるかについて説明されている。また、自然言語を活用して個々のニーズに応じたアシスタントツールを作成できるようになる可能性についても触れている。

25:04

👓 Seeing AIアプリの紹介

Microsoft社内ハッカソンから生まれたSeeing AIアプリについて紹介されている。これは視覚に障害を持つ人々に周囲の世界を説明するモバイルアプリである。アプリはコンピュータービジョン技術を用いてメニューの説明などを行うことができ、最近のGPT 4の機能を活用して会話を通じてより詳細な情報を提供することができるようになっている。

30:05

🔍 アクセシビリティ向上のためのツールとアプローチ

アクセシビリティを向上させるためのツールやアプローチについて議論されている。Redditなどのオンラインコミュニティでのフィードバックや、製品レビューを通じて障害を持つユーザーの声を聞くことが重要であると強調している。また、アクセシビリティを向上させる取り組みは、ユーザーエクスペリエンス全体を向上させることにもつながると指摘している。

👋 アクセシビリティとユーザーフィードバック

アクセシビリティに関するユーザーフィードバックの重要性について話されている。製品のフィードバックチャネルをアクセシブルにし、障害を持つユーザーからのフィードバックを積極的に求めるべきだと主張している。また、ハッカソンの文脈では、オンラインコミュニティやフォーラムを活用して障害を持つユーザーとの対話を促すことが推奨されている。

Keywords

💡アクセシビリティ

アクセシビリティとは、製品やサービスが障害を持つ人々も含め、誰もが使用できるように設計されている状態を指します。このビデオでは、アクセシビリティが重要なテーマとして取り上げられており、開発者ツールが障害を持つ開発者にも使用できるようにすることが重要だと述べられています。また、AIを利用してアクセシビリティを向上させる方法についても議論されています。

💡Azure

Azureは、マイクロソフトが提供するクラウドコンピューティングサービスです。ビデオでは、Azureエンジニアリング組織のElsaがアクセシビリティに焦点を当てた話をしています。Azureは開発者にとって重要なプラットフォームであり、そのアクセシビリティの向上が議論されている点では、ビデオの核心に位置しています。

💡デザイン

デザインは、このビデオでは特にアクセシビリティに関連するコンテキストで使用されています。Elsaはデザインに焦点を当てており、製品やサービスが障害を持つ人々にとって使いやすいものになるようにデザインする重要性が強調されています。デザインはアクセシビリティを確保するための重要な要素です。

💡AI

AIは、ビデオ内で繰り返し強調されている人工知能の略です。AIがアクセシビリティを向上させるためのツールとして機能する例が多数紹介されています。例えば、音声認識や文字起こし、画像の説明、翻訳などが挙げられます。AIはアクセシビリティの分野で大きな進歩をもたらす可能性があると述べられています。

💡ハッカソン

ハッカソンとは、一定期間内に特定の問題を解決するために集まった人々が協力してソフトウェア開発を行うイベントです。ビデオでは、参加者がハッカソンに参加し、アクセシビリティに関する問題に取り組む様子が触れられています。ハッカソンはアイデアの創出と技術の革新を促進する場です。

💡Teams

Microsoft Teamsは、ビデオ会議やコミュニケーションを目的としたソフトウェアです。ビデオでは、Teamsにおけるライブキャプション機能が紹介されており、それがどのようにアクセシビリティを向上させるかが説明されています。ライブキャプションは聴覚障害者や他の人々にとって非常に有用な機能です。

💡ディクテーション

ディクテーションは、音声をテキストに変換するプロセスであり、ビデオではMicrosoft Officeの機能として紹介されています。ディクテーションは、手書きやタイピングが困難な障害を持つ人々にとって非常に役立つ機能です。ビデオではディクテーションがアクセシビリティを向上させる方法の1つとして挙げられています。

💡ライブキャプション

ライブキャプションは、ビデオや会議中の発言をリアルタイムで文字起こしし表示する機能です。ビデオ内で強調されており、TeamsやWindowsなどのMicrosoft製品に組み込まれていると説明されています。ライブキャプションはアクセシビリティを向上させるために重要な機能で、特に聴覚障害者にとって有用です。

💡Co-Pilot

Co-Pilotは、ビデオ内で紹介されているAIパワードなアシスタントです。Co-Pilotは、内容の要約、編集、生成を手助けし、アクセシビリティを向上させる役割を果たしています。ビデオではCo-Pilotがどのように障害を持つ人々にとって生産性を高めるツールとなるかが語られています。

💡Seeing AI

Seeing AIは、ビデオ内で紹介されているアプリで、視覚に障害を持つ人々が周囲の世界を理解するのに役立ちます。アプリは画像認識技術を用いてメニューやその他の文書を説明することができ、アクセシビリティを向上させるための創造的なソリューションの1つです。

Highlights

Elsa Leen from Azure engineering discusses the importance of accessibility in AI and developer platforms.

Developer tools like Visual Studio and VS Code are highlighted for their focus on accessibility features.

Accessibility is defined as inclusive design for over 1.3 billion people with disabilities worldwide.

The concept of 'disability tax' is introduced, referring to the extra time and effort required by individuals with disabilities.

Curb cuts are used as an analogy for universal design that benefits everyone, not just those with disabilities.

Live captions in Teams and Windows are showcased as an AI-powered accessibility tool.

Read Aloud feature in Microsoft Edge and Office is demonstrated, highlighting text-to-speech advancements.

Co-pilot is presented as a productivity tool that has been particularly beneficial for people with disabilities.

The video testimonial from Microsoft employees with disabilities about the impact of Co-pilot is shared.

AI's role in bridging gaps for individuals with disabilities through multimodal experiences is explained.

The Seeing AI app, developed by a blind engineer, uses AI for the blind and low vision community to describe the world around them.

Conversational UI in Seeing AI is demonstrated to show how it can understand and interact with visual content.

The mantra 'nothing about us, without us' is emphasized for engaging with the disability community in product development.

Tips for hackers on engaging with the disability community and testing assumptions during hackathons are provided.

The importance of good accessibility for overall good usability in product design is discussed.

Strategies for finding and involving people with disabilities in the development process are suggested.

Transcripts

[Music]

[Music]

hello happy Friday Innovation Challenge

hackathon hackers um thank you for

taking the time to join us today and

Elsa thank you for taking the time to

talk to us today I'm here with Elsa Leen

who is she works in the Azure

engineering organization and she is

focused on uh design and specifically uh

accessibility and so she's going to

share um some of what she presented at

build just last week um to feel free to

put questions in the chat and uh I hope

we just turn it over to you great yeah

so you know these people are trying to

make a recommendation system more

accessible

what what's your advice yeah well thanks

for having me Matthew I'm really excited

to be speaking to you all um like

Matthew said my job is focused on

accessibility so it's kind of two parts

one is I make sure our developer

platforms for building with AI are

accessible so that developers with

disabilities can build with AI and join

hackathons just like all of you um some

of which you know you may may be you as

well um and then the other part is kind

of jobs like this and talking about how

AI can be used to benefit access ility

and kind of you know encouraging folks

to think about using Azure in these ways

um so I can start the talk we're going

to keep it chill and Friday morning

casual so Matthew you can act as our

audience proxy and jump in with any

questions as we go and yeah I'll just

kind of start with explaining

accessibility in a little bit more

detail before getting into the AI stuff

yeah and the you know and just to the to

the point of you know making our

developer tools accessible you know

um Visual Studio has always been

really good at building an accessibility

features there's I know there's more in

there than I'm even aware of um and so I

think that just I just want to you know

tip of the hat to to the work that you

and your team have been doing for years

uh to make us leaders in that space

thanks Matthew well that's definitely

not me but I agree that you know I'm not

a developer myself but I know visual

studi has done a lot of work vs code all

of the rest of the development tools um

and you know uh like be being a Dev is a

great career right so it's important

that that's inclusive for everyone and a

lot of the kind of accessibility

features they build into those tools are

inclusive for everyone like think about

how adjustable all of the different code

and you know kind of display and color

settings are in visual studio and and

all of our other tools like that's

really useful for people with you know

different visual conditions whether it's

being color blind so like preferring a

different kind of type of color in their

um you know display or you know uh light

sensitive and things like that so yeah

totally um true so so I'm going to talk

about accessibility just a little bit to

start us off um so you know some of you

may already be familiar with this but

this is how we talk about accessibility

at Microsoft um firstly what is

accessibility um it's really about um

making sure that everything that we

build and design and you know at

Microsoft that's digital products but it

can also be you know services like

government services the built

environment right like buildings and you

know hospitals and things like that they

all need to be accessible to people with

disabilities right these are all the

things that you know make up a

fulfilling life and and um you know over

1.3 billion people in the world um have

disabilities and and many of those

benefit from assist of Technology um

when we talk about disability though you

know we often kind of talk about folks

with permanent disabilities right so

that might be somebody who is blind or

who is deaf or heart of hearing and

that's a long-term condition and we also

at Microsoft when we consider designing

our products consider that many of us in

our lives will experience temporary or

situational disabilities so by temporary

I mean um you know that might be

actually for me at the moment I have a a

pretty blocked ear on the left side and

and you know I don't think it's anything

long term but I need to get that cleared

out and that's something temporarily

that's impairing my hearing I hear much

much worse at the moment out of the left

ear than the right I

was I was talking to a fellow who um he

he wound up I think injuring both of his

hands right um a while back and for him

it was this great opportunity to figure

out what okay how would I use word how

would I get my job done um and it was a

real eye opener for him too so the

temporary disability piece is it's

something to you know that matters yeah

yeah yeah absolutely and voice is a

great great um solution for some of the

hand based stuff I feel like that can

kind of be an opportunity for the person

you know to to try out some of our voice

Technologies which are AI based and we

can talk about later and that might you

know help him be more efficient in the

long term right yeah and then the other

one is situational and so would um not

that that's slightly different to

Temporary Disability that would be

something like if we kind of extend the

hearing example that I was using if

you're in a really noisy bar or a noisy

environment um it can be difficult to

hear what's happening and that's a

situational uh disability or situational

impairment um so that's kind of the

spectrum and then when we think about

you know designing for and including

people all across that Spectrum not just

permanent disabilities um you know it

really makes our experiences more

inclusive for everyone and just that

little bit more sort of flexible and

accommodating and easier to use so I

have this picture on the screen of what

they call in America a curb cut um mat

are you familiar with these and have you

taken advantage of these for any reason

at any

point um I'm familiar with them the I've

not I I don't know if I've knowingly

consciously taken advantage of them oh

okay interesting well okay so what we

have up here called called carb Cuts

these are the basically little ramps

that are built into the sidewalk um that

are really really important for people

with wheelchairs to navigate our streets

safely right because otherwise they you

know if someone with the wheelchair

doesn't have a ramp they can't cross the

street and go safely from the sidewalk

onto the road and then back up again

right so these are really really

important for people um who have a

permanent Mobility disability um but I

can bet that you've probably use one at

some point Matthew because if you have

ever tried to navigate the sidewalk with

a wheelie suitcase or maybe even a

stroller or um for me sometimes a

bicycle if I'm along um they are

incredibly useful especially if you have

a heavy suitcase um not to have to lift

that up and kind of pull it along um so

carb cuts are a great example of

something I think most people really who

use the sidewalk use at some point or

another um so um we know I have a couple

more kind of technical examples here

these are all experiences and because I

work on the AI team these are all AI

powered experiences for more or less um

that uh make things just that little bit

more inclusive and accommodating for um

people who use our Microsoft software

and many of them are crucial

accessibility tools for people who have

disabilities um so one example here is

uh in teams um we've had live captions

for a while and those captions now have

translation which is really cool too um

but captions are so so important for our

uh deaf colleagues here at Microsoft and

any other company that uses Microsoft um

so that people can follow along and

actually participate in our meetings and

conversations and but they are a feature

that you know we know many people

benefit from who aren't necessarily deaf

or hard of hearing um so uh I when I did

this talk at build last week um our VP

of azure Services Marco um was was

talking about this with me on stage and

he um is not deaf himself but he uses

captions almost every day uh in his

meetings because he just finds it lowers

the cognitive load to follow along in a

meeting and have the captions while he's

also engaged in the conversation um so

that's one example you know in office

we've had dictation for a long time so

that's that voice enabled um version if

you'd prefer to put your information in

via voice um we have these lovely read

alive features in most of Microsoft

products whether it's Edge and office um

that can read everything out to you um

on the screen and there's much more and

and co-pilot which I will talk about in

a little bit co-pilot um wasn't a you

know a product that was designed

explicitly for people with disabilities

but it's something we found you know all

of the assistive capabilities that come

with co-pilot has been um you know an

incredible productivity boost for people

with disabilities too and so I'm going

to jump into a few examples actually now

so for the first one actually I could

have done this ear earlier but I will um

fire up live captions um so I talked

about captions in teams um but we also

have captions in Windows and I hope this

isn't too small um I did change the

setting to make them larger just before

this call but it looks like it reverted

it should be large text okay so uh let

me turn on microphone audio and now that

I start talking the live caption should

automatically pick me up um and this is

a really useful tool um for folks who

aren't necessarily using teams maybe you

have to use zoom for work or you know

you're watching a video on the internet

which hasn't been captioned because

sometimes unfortunately that happens um

now we have uh you know AI powered um

speech to text transcription live

captions in

Windows uh I'm getting distracted by my

own captions now turn that off but

that's an incredible accessibility tool

that is AI powered that's in Windows

since Windows 11 um that's a great stop

gap for people where captions are

missing in other products I did not know

about that one yeah and it's free right

if you have Windows part of Windows yeah

um so we we provide a lot of our

accessibility tools that way so then one

other one that I'll show while we're

here is read aloud and the you know

browser up here is probably really small

for folks who are following along but we

have this button um up here with an A

and sort of a few lines coming out of it

and that's called read alowed That's

office and also in Edge where if you're

reading you know this blog about

accessibility that our chief

accessibility officer wrote um you can

hit I'm not sharing audio am I Matthew

no do you want do you want to redo it do

you want to I'll R it right now I won't

talk so we don't get the

echo yeah perfect

um

so here and um if I hit that read aloud

button um us a month of access ability

Innovation May 30th 2024 Jenny leury

Microsoft so yeah you got the idea just

that little demo those are you know

quite natural nice voices now with our

um text to speech capabilities and Azure

they're a lot more natural than the the

sort of robotic um computer voices that

people might be used to hearing um you

know we've moved far past that and there

are a ton of different voice options

available I can choose my reading speed

and I can choose between all of these

different voices um including voices in

different languages this is kind of a

fun demo if I play

Eloise seems like Microsoft Chief

accessibility

officer um new this mon so and kind of a

silly demo because obviously you know

who wants to specifically hear um

someone speaking English in a French

accent but you know obviously if I had a

French page here you know they'd have a

proper voice that sounds French reading

the content um the you know we have we

have childlike voices and adult voices

um and you know many different things

that you can set here um and there are

features within Azure where folks can

train you know brands or companies can

train their own custom voices as well um

so yeah that's read AI I think the next

thing I was going to show while I have

video and then I'll let you chime in

again if you want Matthew I know you're

being very um careful not to cause an

echo is the co-pilot example so um I

have a video here that is

um folks from across Microsoft who have

disabilities talking about how co-pilot

has benefited them um so I'll play that

has a massive opportunity

to is it is it working did it

pause the

video is like me I'm going to show you

some of my favorite features that help

improve my productivity on the daily

basis Jeremy Curry and I'll just go into

co-pilot and

say summarize this email I'm able to

really use my voice through the

microphone and voice access it's built

in to co-pilot I say tell me the changes

in accessibility for the past month I'm

going to launch word describe what you'd

like to write including notes or an

outline and co-pilot can generate a

draft to help you get started in this

case I'm going to ask it to write about

the history of computer science history

of computer science

generate a draft one of the things that

is challenging when you have dyslexia is

writing so to help with that I will ask

a pilot to write a draft email I'm going

to ask it to still keep my tone because

I want it to sound like me and I wanted

to be short asking Mary to do a pread of

the document entering a prompt and is

now generating a template email which is

short and sweet which is a brilliant

productivity enhancement for me when I

think about General AI I think about

things that I've done in my 20 plus year

career in accessibility that we've not

been able to do before now and some of

these things are literally life changing

this process with copilot usually takes

an hour but this took 3 minutes it's

important to give me the time back to

work on the things that I love and

generative AI does that if you want to

make AI more accessible and better for

you use it and provide feedback I'm

excited about the possibility that it

will help bridge the disability divide

Microsoft so yeah uh let me switch back

to no

audio

um that actually a go ahead that's a

that's a good example because the I mean

Microsoft the you know this this event

The Innovation Challenge is about

inclusion diversity and inclusion among

our community of developers out there

building on our stuff among our

customers but a lot of what has driven

our commitment um you know to to this at

Microsoft has been our commitment to our

own people these are some really

brilliant people and we want everyone to

like work really hard and commit a lot

because it's Microsoft

but but yeah it's like we need we need

to find the best people in the world and

you know and if they need a little extra

you

know help to make working with us easier

we that's we focus on that yeah exactly

because you know the reality is like

lots of you know for people with

disabilities like you know thinking

about uh Christina who was the woman uh

sitting in the chair she uses her

computer with her feet because she can't

use her arms like things are just going

to take a lot longer computers were not

really designed for that right um you

know people people didn't think about

these things when they designed um most

laptop computers so things take longer

for her that's just loads of loss like

you said loss potential for Microsoft

right as a company and and for her she

you know probably has what you know what

we call the disability tax it takes

longer um to get things done than

someone who's able to use their hands to

work with the computer so you know

co-pilot makes all of us more efficient

but it really unlocks the potential of

our you know colleagues with

disabilities or other folks in the

disability Community who get to use some

of these tools um if I can interrupt

again one you talked about you know the

disability tax and things taking longer

um measuring how long things take um is

a great engineering

metric and um you know just a tip for

the hackers out there if if your hack

focuses on doing something faster that's

kind of a good thing to focus on very

measurable very focused um and you know

I just kind of leave that at that nice

yeah for Thought um so yeah like I guess

you know I sort of showed a couple of

different examples and kind of talked

about accessibility but I want to kind

of step back and think about you know

what what does the era of AI mean for

accessibility because it is a very

exciting time um there's two points that

I have so the first one is you know what

AI is really doing is it's making

accessibility at scale easier than ever

and you know so we have what what's

amazing about the AIS of today is

multimodal right you know you can

describe images or understand images you

can um you know bring voice to things or

turn voice into text right like some of

the examples that we showed um you know

the and and these really address kind of

what we call mismatches or gaps between

you know someone who's deaf you know

they just can't take in sound but they

can you know if if that Gap is bridged

for them you know they can interact with

the world like perfectly um so you know

what what AI is really doing is bridging

those gaps and it's doing it at a scale

that it can be hard to provide without

AI so one example I have on this screen

is a a picture of the USA Today app they

use our text to speech tools to have um

all of their articles automatically read

out loud just like we showed in in

Microsoft Edge right now in a natural

voice so that's a feature in their

mobile app and think about how many you

know articles USA Today produces per day

right it's not really feasible for them

unless they created like a whole other

you know team um of narrators and funded

that to provide that kind of inclusive

capability where people can choose to

either read or listen um you know it

would be a whole other Initiative for

them to start that up but with AI you

know much much more cheaply they're able

to basically provide a service they

wouldn't otherwise and bridge that Gap

and help help people who just you know

just prefer voice and and add that

experience the same with captioning

right you know with with Windows we're

able to bridge the gap for any audio um

transcribe that um and it's the same

with with uh image description you know

so um I think what we can see and and

that's easy that's one of the things

that I have in this slide as simple as

an API call you know at this point

there's kind of no excuse for adding

multimodal experiences into you know

engaging user experiences right like why

not add voice just so that people have

the option of hearing things rather than

reading them you know why not describe

things so there's a lot of a lot of

things you can do there and and AI is

really enabling that scale the second

actually you just think of um a tip for

the hackers too um one of the open data

sets that Microsoft makes available to

play with is the MSN News recommendation

system where you can go through and look

at interactions clicks views Etc that we

get from headlines um another another

thing that hackers can play with cool

yeah that's great yeah um and then my

second point about why you know this era

of generative AI is such a big deal for

accessibility is the natural language

component right and we kind of hinted at

that when we saw the co-pilot video you

know what llms are bringing to us or you

know multimodal models is natural

language assistance and you know

co-pilot is an assist of Technology

right it's a mainstream assist of

technology compared to something like a

screen reader that is focused on the

blind Community co-pilot is an assist of

technology for everyone and I think when

assisted tools are mainstream that's

much better for people with disabilities

too because they generally get more

funding they get more attention because

they're for a larger market um what

copilot can do is you know it helps you

summarize it helps you edit it helps you

generate content if you're someone with

dyslexia for example having that

summarization ability and able to you

know rephrase written content into

something um that might be easier for

you to read is an incredible tool and so

you know all of all of these things can

can make life again easier for everyone

but especially people with disabilities

um and what I think we're really getting

with natural language is the ability for

everyone to individually personalize and

adapt to their own needs right so if

it's someone with dyslexia their own

need will be around reading um but you

know it's it's really about being able

to use that natural language to make

like the assistant tool that you want so

I think that's really exciting and

especially because of how mainstream

it's going to be I would say ultimately

anything to do with natural language is

kind of an accessibility tool for all of

us yeah Sacha loves to say that the it's

the the easiest programming interface

ever yeah exactly yeah um so that's

really exciting I think really those are

so those are the two M oh I have a

couple of other examples uh here um of

innovation so this is an app called

seeing AI um and it

was share share again with audio um and

then kind of contextualize what we're

show so this is a product that was

created quite some years ago now um by a

blind engineer on the Microsoft team um

through a hackathon project and it's now

his full-time job and it has been for

you know over five years or so um and

it's an app called seeing AI um for the

Blind and low vision Community um to

basically describe uh the world around

you via a mobile app and we've you know

and this has been available for a while

with the computer vision Technologies of

you know five some years ago and there

with GPT 4 um visual and you know GPT 40

is coming um or is here uh you know but

this this is showing GPT 4 um there's

some really cool kind of conversational

abilities that blind users of cing AI

can now use to learn more so I think the

most interesting one is the one on the

right um which is around using cing a to

understand a menu and I think it really

shows how conversational UI is and that

kind of multimodal language plus vision

is so much more useful than just Vision

so here we go after scanning this menu

and seeing AI

what are the categories the categories

in the document are antipasti salads

pizza with subcategories red and white

specials and dessert what antipasti

dishes are there there are three

antipasti dishes mentioned in the

document one oven roasted asparagus with

Paprika ioli Red Onion Tomatoes arugula

and lemon zest two Penta with marinara

sauce granino homemade meatballs and

herbs three oven roasted castl vetrano

olives marinated with chilies garlic and

lemon how much does the asparagus anti

cost the asparagus antiac costs

$12 basically what's cool about that

video for me is

um you know it's one thing to use

computer vision and use OCR to just read

out an entire menu to someone who's

playing right like here's all the text

in this image but what's so much more

useful and how you know cited people

interact with written content too is

being able to see you know what what

sections are there you know if I you

know if I'm really into anti pasty can

you tell me more about that or like tell

me if something on this menu has cheese

because I really love cheese and I feel

like cheese today you know it's just a

much more intuitive way to interpret

your visual surroundings being able to

ask questions about it so um another

cool example that I have um yeah uh what

else do I have I see that we're running

running out of time so five minutes one

am I going to show us today uh let me

just uh see what else is in my little

example briefcase but any thoughts or or

questions

Matthew um you know I I guess the you

know one thing that I always come back

to with accessibility is that good

accessibility is generally good

usability yeah absolutely it requires

that you really think through how are

people interacting with your

system and how do you you know make that

better and it gets you thinking in these

really interesting ways that get make

they make me more creative when I try to

think that way I mean I've already you

I'm trying to think of like for people

that like are presented with a you know

wine list that's only only in French and

it's only French wines and they don't

know the difference between something

from one Village and another um it's not

really accessibility but like how do you

interact with that if you don't have a

map of France and the weather report and

you know everything that goes into it

maybe not a good example but yeah stop

with good accessibility is good

usability yeah if there's a gap how do

you make it easier for people like

that's and I think basically designing

um for the needs of accessibility is is

kind of helping us think a little bit

more creative right like if you design

um for sometimes we talk about if you

design for folks at the edges you know

who are more under represented um you

design a solution that better fits the

majority um so this is just a slide with

examples I mean I kind of tore you

through a loot of different things um

today um and there wasn't time to cover

everything but this is just showing you

know uh kind of a summary slide of some

more examples of you know a use case and

that can benefit accessibility who it

benefits and the Technologies of course

that underpin some of this and and you

know when I presented this build you

know we also talked about what customers

already use these because there are lots

of people already using these at scale

right um you know we show we we build

these things into our products you know

such as live captions and then we you

know we show that it can be done at the

scale of windows then and you know folks

like Paton and Swedish television are

more encouraged to also put it into

their own

products how um I guess I'll just have

kind of one last question for you you

know we encourage people to you know if

you're if you're for example if you're

focused on adding accessibility features

to an app

um you need to engage people that need

that

feature yeah need that Gap bridged um

how do you how do we find those people

how do we test with those folks how do

you how do you

gather

requirements when it's they're not your

requirements yeah that's such a good

point and I'm so glad that you brought

it up Matthew so in the disability

Community there's a a well-known slogan

or Mantra called nothing about us

without us and and that is absolutely

the case you know if you're going to and

there there are so many well- meaning

people in the world right you know

especially when when they're hacking and

things trying to come up with an

Innovative solution to a problem they

might come up with a solution to a

problem that they don't necessarily know

about you know they want to help there

are so many barriers for the disability

Community you know someone a hacker

might want to help um but if they aren't

grounded in the needs of let's say um

the deaf Community um they they will

they're likely to make and we all are

likely to make assumptions about you

know what deaf people might or might not

need or what de people might be able to

do um and and that can sometimes be

harmful right you know if you make an

assumption let's say for the blind

example that you know blind people need

help with everything whereas you know in

reality they're actually fairly

independent they only need help with

interpreting visual surrending so you

know sometimes you know making those

assumptions and building them into the

product can be you know offensive to

people or it can be building a solution

wasting time building a solution that

isn't necessary so it's so important um

sign language um is is another example

right you know there's a lot of folks

working on sign language AI sometimes

the computer scientists who work on that

and they see it just as a um you know

we've had folks who've seen it as just a

um created a solution around sign

language gloves for example right

because sign language is used primarily

you know a lot of sign language is done

with hands but it's also facial

expression so you know if you didn't

learn much about sign language and

thought that the problem was only you

know to do with detecting hand movements

you'd be missing a huge part of the

problem and a lot of this can be solved

by talking to people from that community

and you ask me how which rather than why

um so I guess what I have on the slide

is a little bit more geared towards you

know um folks working in organizations

because it's what I showed up build you

know I I talked about how you know if

you've got um you know if you work in a

large organization there are often

employee resource groups or you know if

you have a product in Market there are

customer feedback channels that you can

use what I said in build was you

probably have more customers with

disabilities then you realize you know

make sure your feedback channels are

accessible so they can give you feedback

and make sure you're looking for that

feedback um those things aren't

necessarily the case of course if you're

working on a hackathon um I would really

encourage were you going to say

something Matthew or should I keep no

keep going keep going you're you're more

important than me here yeah I mean

definitely the best best case is you

know working on a hack with a hack

teammate who has lived experience in the

problem you're solving right you're just

going to have a much much deeper

grounding in any problem by doing that

um the other thing is you know reaching

out online or or you know posting about

your work I would say in the context of

a hack probably the most realistic thing

in the time frame is you know to spend

time on places where your community

might um gather right you know if it's

like Reddit sometimes there there are

forums where people just talking about

their experiences you know in a

particular Disability Group and that'll

be really wide ranging and not just

focus on Tech Solutions as well so it's

a great way to learn you know really

more context of the lived experience um

I would say social you know social

forums um you know places where people

are talking on the internet about kind

of these sort of problems are a great

place to spend time and learn is

probably the most realistic thing in the

context of the hack and you know you can

even ask questions in some of these

groups I would say you know take care um

when you do things like that you know if

there's a group uh online let's say in

Reddit that is specifically for you know

people who are neurod Divergent to you

know share and exchange with each other

that could be more of a listening space

rather than somewhere to come in and say

you've come up with a solution for this

group that you're not part of um so I I

think you know just kind of using using

good judgment but really trying to

consciously lean into learning from

others and not assuming um that your

hack solves the problem that you think

it is if you're not from that group but

it's not to put everyone anyone off any

of this I think it's just you know

thinking about taking that time to learn

and kind of test your assumptions as you

build I think um in this this might be

bad advice but um another source of

information the a lot of times when I'm

doing um sort of like

competitive analysis or looking for or

how to improve a certain type of

software I'll do a search on especially

some place with lots of customer reviews

called Product name X sucks for blind

people or something like that yeah

that's really yeah find where people are

complaining and find what they're

complaining about and go solve those

problems yeah those are the

opportunities yeah just learn learn what

people have to say online right there's

a lot of information online about what

any given person or Community thinks

about something so yeah and you you

mentioned Reddit which made me think of

complaining so yeah

relation Elsa thank you so much for your

time today and um you know I look

forward to talking again we'll have to

find another another reason to do this I

look forward to hearing what everyone

comes up with especially for the

personalization and accessibility

challenge so cool thanks so much thank

you bye

[Music]