Building for Accessibility with Azure
Summary
TLDRこのビデオスクリプトでは、Azureエンジニアリング組織でアクセシビリティに特化しているElsa Leenが、開発者プラットフォームのアクセシビリティとAIの活用について語っています。Elsaは、開発者ツールのアクセシビリティ強化と、AIがアクセシビリティを向上させる方法について説明し、TeamsのライブキャプションやWindowsの音声認識機能など、マイクロソフトの製品におけるアクセシビリティ機能の例を紹介しています。さらに、AIが持つ自然言語処理能力がアクセシビリティに与える影響についても触れ、コピロットやSeeing AIなどのツールがどのように視覚障害者や学習障害者を支援するのかを解説しています。
Takeaways
- 😀 アクセシビリティとは、製品やサービスを障害を持つ人々も含めたすべての人に利用可能にすることです。
- 👥 障害を持つ人々は、恒久的な障害だけでなく、一時的または状況依存の障害も考慮に入れます。
- 🛠️ Microsoftは、開発者ツールのアクセシビリティを向上させる取り組みを行っており、Visual Studioなどがその一例です。
- 🌐 Azureエンジニアリング組織のElsaは、AIを使い障碍を抱える人々にとって開発プラットフォームをよりアクセス可能にし、AIがアクセシビリティを向上させる方法を探求しています。
- 🎥 Teamsなどのアプリでは、ライブキャプションや翻訳機能が利用でき、聴覚障害者や他の人々にとって重要な機能となっています。
- 📖 OfficeのディクテーションやRead Aloud機能は、音声をテキストに変換したり、画面の内容を読み上げたりするアクセシビリティツールです。
- 🤖 Co-pilotは、アクセシビリティを高めるためのテクノロジーであり、テキストの要約、編集、生成を支援します。
- 👓 Seeing AIは、視覚に障碍を抱える人々が周囲の世界を理解するのに役立つモバイルアプリです。
- 🔍 AIはマルチモーダルで、画像や音声を理解し、テキストに変換することができるため、さまざまなアクセシビリティのギャップを埋めることができます。
- 📈 AIはアクセシビリティを拡大し、自然言語処理を通じて個々のニーズに応じたパーソナライズされたアシスタントを提供します。
- 👂 アクセシビリティの改善は、ユーザーの包括性と使いやすさを高めることにもつながります。
- 🔗 アクセシビリティに関するフィードバックを得るためには、障碍を持つ人々とコミュニケーションを取り、彼らのニーズを理解することが重要です。
Q & A
このビデオでのエルサ・リーンの役割は何ですか?
-エルサ・リーンはAzureエンジニアリング組織でデザインとアクセシビリティに焦点を当てて働いています。
アクセシビリティとは何ですか?
-アクセシビリティとは、障害を持つ人々がデジタル製品やサービスを利用できるようにすることを指します。
エルサが紹介したMicrosoft製品のアクセシビリティ機能の一例は何ですか?
-一例として、Teamsのライブキャプション機能があります。この機能は聴覚障害を持つ人々が会議に参加しやすくするためのものです。
一時的な障害の例を挙げてください。
-一時的な障害の例として、エルサ自身が左耳の聴力が一時的に低下している状況が挙げられます。
Microsoftがアクセシビリティの重要性を強調する理由は何ですか?
-Microsoftは、より多くの人々が製品やサービスを利用できるようにすることで、全ての人々にとってのインクルーシブな体験を提供することを目指しているからです。
Copilotのアクセシビリティへの影響は何ですか?
-Copilotは、コンテンツの要約や編集を手助けすることで、特にディスレクシア(読み書き障害)を持つ人々にとって有益なアシスタント技術です。
Seeing AIとは何ですか?
-Seeing AIは、視覚障害を持つ人々が周囲の世界をモバイルアプリを通じて理解できるようにするためのアプリです。
AIがアクセシビリティに与える主な利点は何ですか?
-AIは、スケールでのアクセシビリティを容易にし、自然言語のサポートを提供することで、個々のニーズに合わせたパーソナライズを可能にします。
Microsoftの製品で使われている読み上げ機能の例を挙げてください。
-例として、Microsoft EdgeやOfficeでの読み上げ機能があります。これにより、画面上のテキストを音声で読み上げることができます。
アクセシビリティ向上のためにエンジニアが考慮すべきことは何ですか?
-エンジニアは、障害を持つコミュニティからのフィードバックを収集し、実際のユーザーのニーズに基づいて設計を行うべきです。
Outlines
😀 アクセス性の重要性とAIの役割
Elsa LeenはMicrosoftのAzureエンジニアリング組織で働く。彼女は開発者プラットフォームのアクセシビリティを確保し、AIを通じてアクセシビリティを向上させる方法を探求している。ビデオでは、開発者ツールのアクセシビリティを向上させる取り組みや、AIがアクセシビリティに与える恩恵について語っている。また、Microsoftのアクセシビリティの考え方や、製品やサービスを設計する際のアクセシビリティの重要性を説明している。
😉 一時的または状況依存の障害
ビデオでは一時的または状況依存の障害についても議論されている。例えば、雑音の多い環境での聴覚の障害や、車椅子での移動に関する議論がある。カーブカット(歩道の傾斜部分)の例を通じて、永久的な移動障害を持つ人々にとって非常に重要なインフラストラクチャであることが説明されている。しかし、カーブカットは車輪のスーツケースやベビーストローラー、自転車などを持つ人々にとっても非常に便利な機能であると指摘している。
🎥 AIを活用したアクセシビリティの向上
Microsoft製品におけるAIパワードなアクセシビリティツールについて紹介されている。Teamsでのライブキャプション、翻訳機能、Officeのディクテーション、Read Aloud機能などが挙げられる。これらは特定の障害を持つ人々だけでなく、一般のユーザーにも恩恵をもたらす機能である。また、CopilotというAIアシスタントがどのように障害を持つ人々にとって生産性を高める助けとなるかについても触れている。
👥 個人差に応じたアクセシビリティの重要性
アクセシビリティが個人差に応じてどのように重要であるかについて議論されている。AIが持つマルチモーダルな能力が、聴覚障害者や視覚障害者、学習障害者にとってどのように役立つかを説明している。また、AIがスケールでアクセシビリティを提供する能力についても触れており、USA Todayアプリのテキスト読み上げ機能など、実際の例を紹介している。
🤖 自然言語処理とアクセシビリティ
自然言語処理(NLP)がアクセシビリティに与える影響について語られている。CopilotのようなAIアシスタントが、自然言語を通じてどのようにアクセシビリティを向上させるかについて説明されている。また、自然言語を活用して個々のニーズに応じたアシスタントツールを作成できるようになる可能性についても触れている。
👓 Seeing AIアプリの紹介
Microsoft社内ハッカソンから生まれたSeeing AIアプリについて紹介されている。これは視覚に障害を持つ人々に周囲の世界を説明するモバイルアプリである。アプリはコンピュータービジョン技術を用いてメニューの説明などを行うことができ、最近のGPT 4の機能を活用して会話を通じてより詳細な情報を提供することができるようになっている。
🔍 アクセシビリティ向上のためのツールとアプローチ
アクセシビリティを向上させるためのツールやアプローチについて議論されている。Redditなどのオンラインコミュニティでのフィードバックや、製品レビューを通じて障害を持つユーザーの声を聞くことが重要であると強調している。また、アクセシビリティを向上させる取り組みは、ユーザーエクスペリエンス全体を向上させることにもつながると指摘している。
👋 アクセシビリティとユーザーフィードバック
アクセシビリティに関するユーザーフィードバックの重要性について話されている。製品のフィードバックチャネルをアクセシブルにし、障害を持つユーザーからのフィードバックを積極的に求めるべきだと主張している。また、ハッカソンの文脈では、オンラインコミュニティやフォーラムを活用して障害を持つユーザーとの対話を促すことが推奨されている。
Mindmap
Keywords
💡アクセシビリティ
💡Azure
💡デザイン
💡AI
💡ハッカソン
💡Teams
💡ディクテーション
💡ライブキャプション
💡Co-Pilot
💡Seeing AI
Highlights
Elsa Leen from Azure engineering discusses the importance of accessibility in AI and developer platforms.
Developer tools like Visual Studio and VS Code are highlighted for their focus on accessibility features.
Accessibility is defined as inclusive design for over 1.3 billion people with disabilities worldwide.
The concept of 'disability tax' is introduced, referring to the extra time and effort required by individuals with disabilities.
Curb cuts are used as an analogy for universal design that benefits everyone, not just those with disabilities.
Live captions in Teams and Windows are showcased as an AI-powered accessibility tool.
Read Aloud feature in Microsoft Edge and Office is demonstrated, highlighting text-to-speech advancements.
Co-pilot is presented as a productivity tool that has been particularly beneficial for people with disabilities.
The video testimonial from Microsoft employees with disabilities about the impact of Co-pilot is shared.
AI's role in bridging gaps for individuals with disabilities through multimodal experiences is explained.
The Seeing AI app, developed by a blind engineer, uses AI for the blind and low vision community to describe the world around them.
Conversational UI in Seeing AI is demonstrated to show how it can understand and interact with visual content.
The mantra 'nothing about us, without us' is emphasized for engaging with the disability community in product development.
Tips for hackers on engaging with the disability community and testing assumptions during hackathons are provided.
The importance of good accessibility for overall good usability in product design is discussed.
Strategies for finding and involving people with disabilities in the development process are suggested.
Transcripts
[Music]
[Music]
hello happy Friday Innovation Challenge
hackathon hackers um thank you for
taking the time to join us today and
Elsa thank you for taking the time to
talk to us today I'm here with Elsa Leen
who is she works in the Azure
engineering organization and she is
focused on uh design and specifically uh
accessibility and so she's going to
share um some of what she presented at
build just last week um to feel free to
put questions in the chat and uh I hope
we just turn it over to you great yeah
so you know these people are trying to
make a recommendation system more
accessible
what what's your advice yeah well thanks
for having me Matthew I'm really excited
to be speaking to you all um like
Matthew said my job is focused on
accessibility so it's kind of two parts
one is I make sure our developer
platforms for building with AI are
accessible so that developers with
disabilities can build with AI and join
hackathons just like all of you um some
of which you know you may may be you as
well um and then the other part is kind
of jobs like this and talking about how
AI can be used to benefit access ility
and kind of you know encouraging folks
to think about using Azure in these ways
um so I can start the talk we're going
to keep it chill and Friday morning
casual so Matthew you can act as our
audience proxy and jump in with any
questions as we go and yeah I'll just
kind of start with explaining
accessibility in a little bit more
detail before getting into the AI stuff
yeah and the you know and just to the to
the point of you know making our
developer tools accessible you know
um Visual Studio has always been
really good at building an accessibility
features there's I know there's more in
there than I'm even aware of um and so I
think that just I just want to you know
tip of the hat to to the work that you
and your team have been doing for years
uh to make us leaders in that space
thanks Matthew well that's definitely
not me but I agree that you know I'm not
a developer myself but I know visual
studi has done a lot of work vs code all
of the rest of the development tools um
and you know uh like be being a Dev is a
great career right so it's important
that that's inclusive for everyone and a
lot of the kind of accessibility
features they build into those tools are
inclusive for everyone like think about
how adjustable all of the different code
and you know kind of display and color
settings are in visual studio and and
all of our other tools like that's
really useful for people with you know
different visual conditions whether it's
being color blind so like preferring a
different kind of type of color in their
um you know display or you know uh light
sensitive and things like that so yeah
totally um true so so I'm going to talk
about accessibility just a little bit to
start us off um so you know some of you
may already be familiar with this but
this is how we talk about accessibility
at Microsoft um firstly what is
accessibility um it's really about um
making sure that everything that we
build and design and you know at
Microsoft that's digital products but it
can also be you know services like
government services the built
environment right like buildings and you
know hospitals and things like that they
all need to be accessible to people with
disabilities right these are all the
things that you know make up a
fulfilling life and and um you know over
1.3 billion people in the world um have
disabilities and and many of those
benefit from assist of Technology um
when we talk about disability though you
know we often kind of talk about folks
with permanent disabilities right so
that might be somebody who is blind or
who is deaf or heart of hearing and
that's a long-term condition and we also
at Microsoft when we consider designing
our products consider that many of us in
our lives will experience temporary or
situational disabilities so by temporary
I mean um you know that might be
actually for me at the moment I have a a
pretty blocked ear on the left side and
and you know I don't think it's anything
long term but I need to get that cleared
out and that's something temporarily
that's impairing my hearing I hear much
much worse at the moment out of the left
ear than the right I
was I was talking to a fellow who um he
he wound up I think injuring both of his
hands right um a while back and for him
it was this great opportunity to figure
out what okay how would I use word how
would I get my job done um and it was a
real eye opener for him too so the
temporary disability piece is it's
something to you know that matters yeah
yeah yeah absolutely and voice is a
great great um solution for some of the
hand based stuff I feel like that can
kind of be an opportunity for the person
you know to to try out some of our voice
Technologies which are AI based and we
can talk about later and that might you
know help him be more efficient in the
long term right yeah and then the other
one is situational and so would um not
that that's slightly different to
Temporary Disability that would be
something like if we kind of extend the
hearing example that I was using if
you're in a really noisy bar or a noisy
environment um it can be difficult to
hear what's happening and that's a
situational uh disability or situational
impairment um so that's kind of the
spectrum and then when we think about
you know designing for and including
people all across that Spectrum not just
permanent disabilities um you know it
really makes our experiences more
inclusive for everyone and just that
little bit more sort of flexible and
accommodating and easier to use so I
have this picture on the screen of what
they call in America a curb cut um mat
are you familiar with these and have you
taken advantage of these for any reason
at any
point um I'm familiar with them the I've
not I I don't know if I've knowingly
consciously taken advantage of them oh
okay interesting well okay so what we
have up here called called carb Cuts
these are the basically little ramps
that are built into the sidewalk um that
are really really important for people
with wheelchairs to navigate our streets
safely right because otherwise they you
know if someone with the wheelchair
doesn't have a ramp they can't cross the
street and go safely from the sidewalk
onto the road and then back up again
right so these are really really
important for people um who have a
permanent Mobility disability um but I
can bet that you've probably use one at
some point Matthew because if you have
ever tried to navigate the sidewalk with
a wheelie suitcase or maybe even a
stroller or um for me sometimes a
bicycle if I'm along um they are
incredibly useful especially if you have
a heavy suitcase um not to have to lift
that up and kind of pull it along um so
carb cuts are a great example of
something I think most people really who
use the sidewalk use at some point or
another um so um we know I have a couple
more kind of technical examples here
these are all experiences and because I
work on the AI team these are all AI
powered experiences for more or less um
that uh make things just that little bit
more inclusive and accommodating for um
people who use our Microsoft software
and many of them are crucial
accessibility tools for people who have
disabilities um so one example here is
uh in teams um we've had live captions
for a while and those captions now have
translation which is really cool too um
but captions are so so important for our
uh deaf colleagues here at Microsoft and
any other company that uses Microsoft um
so that people can follow along and
actually participate in our meetings and
conversations and but they are a feature
that you know we know many people
benefit from who aren't necessarily deaf
or hard of hearing um so uh I when I did
this talk at build last week um our VP
of azure Services Marco um was was
talking about this with me on stage and
he um is not deaf himself but he uses
captions almost every day uh in his
meetings because he just finds it lowers
the cognitive load to follow along in a
meeting and have the captions while he's
also engaged in the conversation um so
that's one example you know in office
we've had dictation for a long time so
that's that voice enabled um version if
you'd prefer to put your information in
via voice um we have these lovely read
alive features in most of Microsoft
products whether it's Edge and office um
that can read everything out to you um
on the screen and there's much more and
and co-pilot which I will talk about in
a little bit co-pilot um wasn't a you
know a product that was designed
explicitly for people with disabilities
but it's something we found you know all
of the assistive capabilities that come
with co-pilot has been um you know an
incredible productivity boost for people
with disabilities too and so I'm going
to jump into a few examples actually now
so for the first one actually I could
have done this ear earlier but I will um
fire up live captions um so I talked
about captions in teams um but we also
have captions in Windows and I hope this
isn't too small um I did change the
setting to make them larger just before
this call but it looks like it reverted
it should be large text okay so uh let
me turn on microphone audio and now that
I start talking the live caption should
automatically pick me up um and this is
a really useful tool um for folks who
aren't necessarily using teams maybe you
have to use zoom for work or you know
you're watching a video on the internet
which hasn't been captioned because
sometimes unfortunately that happens um
now we have uh you know AI powered um
speech to text transcription live
captions in
Windows uh I'm getting distracted by my
own captions now turn that off but
that's an incredible accessibility tool
that is AI powered that's in Windows
since Windows 11 um that's a great stop
gap for people where captions are
missing in other products I did not know
about that one yeah and it's free right
if you have Windows part of Windows yeah
um so we we provide a lot of our
accessibility tools that way so then one
other one that I'll show while we're
here is read aloud and the you know
browser up here is probably really small
for folks who are following along but we
have this button um up here with an A
and sort of a few lines coming out of it
and that's called read alowed That's
office and also in Edge where if you're
reading you know this blog about
accessibility that our chief
accessibility officer wrote um you can
hit I'm not sharing audio am I Matthew
no do you want do you want to redo it do
you want to I'll R it right now I won't
talk so we don't get the
echo yeah perfect
um
so here and um if I hit that read aloud
button um us a month of access ability
Innovation May 30th 2024 Jenny leury
Microsoft so yeah you got the idea just
that little demo those are you know
quite natural nice voices now with our
um text to speech capabilities and Azure
they're a lot more natural than the the
sort of robotic um computer voices that
people might be used to hearing um you
know we've moved far past that and there
are a ton of different voice options
available I can choose my reading speed
and I can choose between all of these
different voices um including voices in
different languages this is kind of a
fun demo if I play
Eloise seems like Microsoft Chief
accessibility
officer um new this mon so and kind of a
silly demo because obviously you know
who wants to specifically hear um
someone speaking English in a French
accent but you know obviously if I had a
French page here you know they'd have a
proper voice that sounds French reading
the content um the you know we have we
have childlike voices and adult voices
um and you know many different things
that you can set here um and there are
features within Azure where folks can
train you know brands or companies can
train their own custom voices as well um
so yeah that's read AI I think the next
thing I was going to show while I have
video and then I'll let you chime in
again if you want Matthew I know you're
being very um careful not to cause an
echo is the co-pilot example so um I
have a video here that is
um folks from across Microsoft who have
disabilities talking about how co-pilot
has benefited them um so I'll play that
has a massive opportunity
to is it is it working did it
pause the
video is like me I'm going to show you
some of my favorite features that help
improve my productivity on the daily
basis Jeremy Curry and I'll just go into
co-pilot and
say summarize this email I'm able to
really use my voice through the
microphone and voice access it's built
in to co-pilot I say tell me the changes
in accessibility for the past month I'm
going to launch word describe what you'd
like to write including notes or an
outline and co-pilot can generate a
draft to help you get started in this
case I'm going to ask it to write about
the history of computer science history
of computer science
generate a draft one of the things that
is challenging when you have dyslexia is
writing so to help with that I will ask
a pilot to write a draft email I'm going
to ask it to still keep my tone because
I want it to sound like me and I wanted
to be short asking Mary to do a pread of
the document entering a prompt and is
now generating a template email which is
short and sweet which is a brilliant
productivity enhancement for me when I
think about General AI I think about
things that I've done in my 20 plus year
career in accessibility that we've not
been able to do before now and some of
these things are literally life changing
this process with copilot usually takes
an hour but this took 3 minutes it's
important to give me the time back to
work on the things that I love and
generative AI does that if you want to
make AI more accessible and better for
you use it and provide feedback I'm
excited about the possibility that it
will help bridge the disability divide
Microsoft so yeah uh let me switch back
to no
audio
um that actually a go ahead that's a
that's a good example because the I mean
Microsoft the you know this this event
The Innovation Challenge is about
inclusion diversity and inclusion among
our community of developers out there
building on our stuff among our
customers but a lot of what has driven
our commitment um you know to to this at
Microsoft has been our commitment to our
own people these are some really
brilliant people and we want everyone to
like work really hard and commit a lot
because it's Microsoft
but but yeah it's like we need we need
to find the best people in the world and
you know and if they need a little extra
you
know help to make working with us easier
we that's we focus on that yeah exactly
because you know the reality is like
lots of you know for people with
disabilities like you know thinking
about uh Christina who was the woman uh
sitting in the chair she uses her
computer with her feet because she can't
use her arms like things are just going
to take a lot longer computers were not
really designed for that right um you
know people people didn't think about
these things when they designed um most
laptop computers so things take longer
for her that's just loads of loss like
you said loss potential for Microsoft
right as a company and and for her she
you know probably has what you know what
we call the disability tax it takes
longer um to get things done than
someone who's able to use their hands to
work with the computer so you know
co-pilot makes all of us more efficient
but it really unlocks the potential of
our you know colleagues with
disabilities or other folks in the
disability Community who get to use some
of these tools um if I can interrupt
again one you talked about you know the
disability tax and things taking longer
um measuring how long things take um is
a great engineering
metric and um you know just a tip for
the hackers out there if if your hack
focuses on doing something faster that's
kind of a good thing to focus on very
measurable very focused um and you know
I just kind of leave that at that nice
yeah for Thought um so yeah like I guess
you know I sort of showed a couple of
different examples and kind of talked
about accessibility but I want to kind
of step back and think about you know
what what does the era of AI mean for
accessibility because it is a very
exciting time um there's two points that
I have so the first one is you know what
AI is really doing is it's making
accessibility at scale easier than ever
and you know so we have what what's
amazing about the AIS of today is
multimodal right you know you can
describe images or understand images you
can um you know bring voice to things or
turn voice into text right like some of
the examples that we showed um you know
the and and these really address kind of
what we call mismatches or gaps between
you know someone who's deaf you know
they just can't take in sound but they
can you know if if that Gap is bridged
for them you know they can interact with
the world like perfectly um so you know
what what AI is really doing is bridging
those gaps and it's doing it at a scale
that it can be hard to provide without
AI so one example I have on this screen
is a a picture of the USA Today app they
use our text to speech tools to have um
all of their articles automatically read
out loud just like we showed in in
Microsoft Edge right now in a natural
voice so that's a feature in their
mobile app and think about how many you
know articles USA Today produces per day
right it's not really feasible for them
unless they created like a whole other
you know team um of narrators and funded
that to provide that kind of inclusive
capability where people can choose to
either read or listen um you know it
would be a whole other Initiative for
them to start that up but with AI you
know much much more cheaply they're able
to basically provide a service they
wouldn't otherwise and bridge that Gap
and help help people who just you know
just prefer voice and and add that
experience the same with captioning
right you know with with Windows we're
able to bridge the gap for any audio um
transcribe that um and it's the same
with with uh image description you know
so um I think what we can see and and
that's easy that's one of the things
that I have in this slide as simple as
an API call you know at this point
there's kind of no excuse for adding
multimodal experiences into you know
engaging user experiences right like why
not add voice just so that people have
the option of hearing things rather than
reading them you know why not describe
things so there's a lot of a lot of
things you can do there and and AI is
really enabling that scale the second
actually you just think of um a tip for
the hackers too um one of the open data
sets that Microsoft makes available to
play with is the MSN News recommendation
system where you can go through and look
at interactions clicks views Etc that we
get from headlines um another another
thing that hackers can play with cool
yeah that's great yeah um and then my
second point about why you know this era
of generative AI is such a big deal for
accessibility is the natural language
component right and we kind of hinted at
that when we saw the co-pilot video you
know what llms are bringing to us or you
know multimodal models is natural
language assistance and you know
co-pilot is an assist of Technology
right it's a mainstream assist of
technology compared to something like a
screen reader that is focused on the
blind Community co-pilot is an assist of
technology for everyone and I think when
assisted tools are mainstream that's
much better for people with disabilities
too because they generally get more
funding they get more attention because
they're for a larger market um what
copilot can do is you know it helps you
summarize it helps you edit it helps you
generate content if you're someone with
dyslexia for example having that
summarization ability and able to you
know rephrase written content into
something um that might be easier for
you to read is an incredible tool and so
you know all of all of these things can
can make life again easier for everyone
but especially people with disabilities
um and what I think we're really getting
with natural language is the ability for
everyone to individually personalize and
adapt to their own needs right so if
it's someone with dyslexia their own
need will be around reading um but you
know it's it's really about being able
to use that natural language to make
like the assistant tool that you want so
I think that's really exciting and
especially because of how mainstream
it's going to be I would say ultimately
anything to do with natural language is
kind of an accessibility tool for all of
us yeah Sacha loves to say that the it's
the the easiest programming interface
ever yeah exactly yeah um so that's
really exciting I think really those are
so those are the two M oh I have a
couple of other examples uh here um of
innovation so this is an app called
seeing AI um and it
was share share again with audio um and
then kind of contextualize what we're
show so this is a product that was
created quite some years ago now um by a
blind engineer on the Microsoft team um
through a hackathon project and it's now
his full-time job and it has been for
you know over five years or so um and
it's an app called seeing AI um for the
Blind and low vision Community um to
basically describe uh the world around
you via a mobile app and we've you know
and this has been available for a while
with the computer vision Technologies of
you know five some years ago and there
with GPT 4 um visual and you know GPT 40
is coming um or is here uh you know but
this this is showing GPT 4 um there's
some really cool kind of conversational
abilities that blind users of cing AI
can now use to learn more so I think the
most interesting one is the one on the
right um which is around using cing a to
understand a menu and I think it really
shows how conversational UI is and that
kind of multimodal language plus vision
is so much more useful than just Vision
so here we go after scanning this menu
and seeing AI
what are the categories the categories
in the document are antipasti salads
pizza with subcategories red and white
specials and dessert what antipasti
dishes are there there are three
antipasti dishes mentioned in the
document one oven roasted asparagus with
Paprika ioli Red Onion Tomatoes arugula
and lemon zest two Penta with marinara
sauce granino homemade meatballs and
herbs three oven roasted castl vetrano
olives marinated with chilies garlic and
lemon how much does the asparagus anti
cost the asparagus antiac costs
$12 basically what's cool about that
video for me is
um you know it's one thing to use
computer vision and use OCR to just read
out an entire menu to someone who's
playing right like here's all the text
in this image but what's so much more
useful and how you know cited people
interact with written content too is
being able to see you know what what
sections are there you know if I you
know if I'm really into anti pasty can
you tell me more about that or like tell
me if something on this menu has cheese
because I really love cheese and I feel
like cheese today you know it's just a
much more intuitive way to interpret
your visual surroundings being able to
ask questions about it so um another
cool example that I have um yeah uh what
else do I have I see that we're running
running out of time so five minutes one
am I going to show us today uh let me
just uh see what else is in my little
example briefcase but any thoughts or or
questions
Matthew um you know I I guess the you
know one thing that I always come back
to with accessibility is that good
accessibility is generally good
usability yeah absolutely it requires
that you really think through how are
people interacting with your
system and how do you you know make that
better and it gets you thinking in these
really interesting ways that get make
they make me more creative when I try to
think that way I mean I've already you
I'm trying to think of like for people
that like are presented with a you know
wine list that's only only in French and
it's only French wines and they don't
know the difference between something
from one Village and another um it's not
really accessibility but like how do you
interact with that if you don't have a
map of France and the weather report and
you know everything that goes into it
maybe not a good example but yeah stop
with good accessibility is good
usability yeah if there's a gap how do
you make it easier for people like
that's and I think basically designing
um for the needs of accessibility is is
kind of helping us think a little bit
more creative right like if you design
um for sometimes we talk about if you
design for folks at the edges you know
who are more under represented um you
design a solution that better fits the
majority um so this is just a slide with
examples I mean I kind of tore you
through a loot of different things um
today um and there wasn't time to cover
everything but this is just showing you
know uh kind of a summary slide of some
more examples of you know a use case and
that can benefit accessibility who it
benefits and the Technologies of course
that underpin some of this and and you
know when I presented this build you
know we also talked about what customers
already use these because there are lots
of people already using these at scale
right um you know we show we we build
these things into our products you know
such as live captions and then we you
know we show that it can be done at the
scale of windows then and you know folks
like Paton and Swedish television are
more encouraged to also put it into
their own
products how um I guess I'll just have
kind of one last question for you you
know we encourage people to you know if
you're if you're for example if you're
focused on adding accessibility features
to an app
um you need to engage people that need
that
feature yeah need that Gap bridged um
how do you how do we find those people
how do we test with those folks how do
you how do you
gather
requirements when it's they're not your
requirements yeah that's such a good
point and I'm so glad that you brought
it up Matthew so in the disability
Community there's a a well-known slogan
or Mantra called nothing about us
without us and and that is absolutely
the case you know if you're going to and
there there are so many well- meaning
people in the world right you know
especially when when they're hacking and
things trying to come up with an
Innovative solution to a problem they
might come up with a solution to a
problem that they don't necessarily know
about you know they want to help there
are so many barriers for the disability
Community you know someone a hacker
might want to help um but if they aren't
grounded in the needs of let's say um
the deaf Community um they they will
they're likely to make and we all are
likely to make assumptions about you
know what deaf people might or might not
need or what de people might be able to
do um and and that can sometimes be
harmful right you know if you make an
assumption let's say for the blind
example that you know blind people need
help with everything whereas you know in
reality they're actually fairly
independent they only need help with
interpreting visual surrending so you
know sometimes you know making those
assumptions and building them into the
product can be you know offensive to
people or it can be building a solution
wasting time building a solution that
isn't necessary so it's so important um
sign language um is is another example
right you know there's a lot of folks
working on sign language AI sometimes
the computer scientists who work on that
and they see it just as a um you know
we've had folks who've seen it as just a
um created a solution around sign
language gloves for example right
because sign language is used primarily
you know a lot of sign language is done
with hands but it's also facial
expression so you know if you didn't
learn much about sign language and
thought that the problem was only you
know to do with detecting hand movements
you'd be missing a huge part of the
problem and a lot of this can be solved
by talking to people from that community
and you ask me how which rather than why
um so I guess what I have on the slide
is a little bit more geared towards you
know um folks working in organizations
because it's what I showed up build you
know I I talked about how you know if
you've got um you know if you work in a
large organization there are often
employee resource groups or you know if
you have a product in Market there are
customer feedback channels that you can
use what I said in build was you
probably have more customers with
disabilities then you realize you know
make sure your feedback channels are
accessible so they can give you feedback
and make sure you're looking for that
feedback um those things aren't
necessarily the case of course if you're
working on a hackathon um I would really
encourage were you going to say
something Matthew or should I keep no
keep going keep going you're you're more
important than me here yeah I mean
definitely the best best case is you
know working on a hack with a hack
teammate who has lived experience in the
problem you're solving right you're just
going to have a much much deeper
grounding in any problem by doing that
um the other thing is you know reaching
out online or or you know posting about
your work I would say in the context of
a hack probably the most realistic thing
in the time frame is you know to spend
time on places where your community
might um gather right you know if it's
like Reddit sometimes there there are
forums where people just talking about
their experiences you know in a
particular Disability Group and that'll
be really wide ranging and not just
focus on Tech Solutions as well so it's
a great way to learn you know really
more context of the lived experience um
I would say social you know social
forums um you know places where people
are talking on the internet about kind
of these sort of problems are a great
place to spend time and learn is
probably the most realistic thing in the
context of the hack and you know you can
even ask questions in some of these
groups I would say you know take care um
when you do things like that you know if
there's a group uh online let's say in
Reddit that is specifically for you know
people who are neurod Divergent to you
know share and exchange with each other
that could be more of a listening space
rather than somewhere to come in and say
you've come up with a solution for this
group that you're not part of um so I I
think you know just kind of using using
good judgment but really trying to
consciously lean into learning from
others and not assuming um that your
hack solves the problem that you think
it is if you're not from that group but
it's not to put everyone anyone off any
of this I think it's just you know
thinking about taking that time to learn
and kind of test your assumptions as you
build I think um in this this might be
bad advice but um another source of
information the a lot of times when I'm
doing um sort of like
competitive analysis or looking for or
how to improve a certain type of
software I'll do a search on especially
some place with lots of customer reviews
called Product name X sucks for blind
people or something like that yeah
that's really yeah find where people are
complaining and find what they're
complaining about and go solve those
problems yeah those are the
opportunities yeah just learn learn what
people have to say online right there's
a lot of information online about what
any given person or Community thinks
about something so yeah and you you
mentioned Reddit which made me think of
complaining so yeah
relation Elsa thank you so much for your
time today and um you know I look
forward to talking again we'll have to
find another another reason to do this I
look forward to hearing what everyone
comes up with especially for the
personalization and accessibility
challenge so cool thanks so much thank
you bye
[Music]
5.0 / 5 (0 votes)