Adopting AI: Ensuring Business Readiness

Summary

The webinar 'Adopting AI: Ensuring Business Readiness' discusses the importance of artificial intelligence in organizational futures. It explores挑选合适的问题用AI解决, launching AI initiatives, and the need for oversight and understanding of AI risks. Industry experts from diverse fields, including healthcare, share insights on AI use cases, challenges, and strategies for successful AI integration, emphasizing the need for robust data infrastructure, a skilled workforce, and compliance with evolving regulations.

Takeaways

• 🤖 AI algorithms are abstract and probabilistic, making them complex and imprecise by nature.
• 🔍 'Explainable AI' techniques can shed light on AI algorithms' predictions but cannot determine fairness or justice.
• 🧠 AI systems are adaptive and learn from data, but lack creativity and may revert to known patterns in unforeseen circumstances.
• 📈 AI systems are typically deployed at scale, which can magnify small errors and require robust feedback loops.
• 👀 AI algorithms are impressionable and know only what they've been exposed to during training and production.
• 🚫 AI algorithms can inadvertently pick up and reinforce biases present in the training data.
• 🛠️ It's crucial to define operating conditions for AI systems and engineer safety controls into their processes.
• 💡 AI solutions must be adopted and scaled effectively, with a focus on value creation and strategic alignment.
• 🌐 AI and machine learning in healthcare hold great promise for predictive analytics, personalized care, and efficiency improvements.
• 🧬 Healthcare data is vast and complex, requiring advanced NLP technologies to unlock valuable insights.
• 🔄 The success of AI in healthcare relies on a collaborative effort between data scientists, clinicians, and end-users.

Q & A

What are the six characteristics of AI algorithms that require additional business due diligence?

The six characteristics are: 1) AI algorithms are abstract with complex inner workings; 2) They are probabilistic systems with imprecise outputs; 3) They are adaptive and respond to changes in data input; 4) They are not creative and revert to known patterns in unforeseen circumstances; 5) They are typically deployed at scale, which can magnify small errors; 6) They are impressionable and learn from the data they are exposed to during training and production.

How can organizations ensure their AI solutions deliver the intended outcomes?

Organizations can ensure intended outcomes by establishing robust feedback loops, clearly defining operating conditions, engineering safety controls, monitoring actual usage alongside intended use, and educating users on how the application is meant to be used.

What are some challenges in adopting AI solutions in healthcare?

Challenges include technical issues such as interpretability, bias, and drift, as well as non-technical issues like clinical validation, workflow integration, privacy concerns, data governance, and deciding whether to build or buy AI solutions.

What is the importance of explainable AI in business applications?

Explainable AI is crucial as it provides insights into which factors most influence an algorithm's predictions. This helps in making informed decisions about whether those factors are fair, just, and aligned with the business's strategic goals.

How can businesses mitigate the inherent risks of AI?

Businesses can mitigate risks by implementing robust safety controls, investing in explainable AI technologies, ensuring proper oversight, and maintaining a feedback loop to correct errors and improve the system over time.

What are some best practices for adopting AI solutions at scale?

Best practices include starting with a clear understanding of business strategy, defining and finding value in AI applications, ensuring the basics are done brilliantly, focusing on people capability and culture, and proceeding responsibly while building trust.

What is the role of AI in enhancing healthcare delivery systems?

AI can enhance healthcare delivery by providing predictive analytics for disease progression and readmission risks, personalized preventative care, natural language processing for unstructured data, and image recognition for various medical imaging needs.

How can companies balance the need for speed in AI adoption with the slower process of cultural change?

Companies can balance speed with cultural change by prioritizing education and awareness about AI across the organization, leveraging existing talent, and fostering partnerships between industry and academia to develop a skilled workforce.

What are some strategies for finding the right AI talent?

Strategies include looking within the organization for potential talent, partnering with academic institutions, investing in internal training programs, and considering a mix of STEM and SHAPE (sociology, humanities, ethics, sustainability) skills.

How can organizations prepare for and comply with international, regional, and national regulations regarding AI?

Organizations should have a dedicated team or individual to understand and monitor regulatory changes, engage with regulators, anticipate future compliance needs, and integrate legal and ethical considerations into AI development and deployment.

What are some indicators of data-science maturity within an organization?

Indicators of data-science maturity include the ability to build and implement AI solutions that improve quality, service, cost, and provider sustainability, as well as the capacity to communicate effectively with end-users and stakeholders.

Outlines

00:00

🌟 Introduction and Setting the Stage for AI Readiness

The webinar begins with Abbie Lundberg introducing the topic of AI adoption in businesses and its growing importance. Abbie outlines the webinar's agenda, which includes discussions on selecting the right AI problems to solve, ensuring business readiness for AI initiatives, understanding AI risks, and sharing expertise across industries with a focus on healthcare. The panel includes Kimberly Nevala from SAS, Fernando Lucini from Accenture Applied Intelligence, and Dr. Tad Funahashi from Kaiser Permanente, each bringing their unique perspectives on AI's strategic value and practical implementation.

05:00

🤖 AI Algorithm Characteristics and Business Diligence

Kimberly Nevala starts the discussion by highlighting six key characteristics of AI algorithms that require additional business diligence. These include the abstract nature of AI logic, the probabilistic nature of AI predictions, the adaptability of AI systems to data inputs, the magnification of small errors at scale, the impressionability of AI systems to data, and the incautious nature of AI algorithms. She emphasizes the need for explainable AI, robust safety controls, and the importance of understanding and mitigating AI's inherent risks.

10:02

📈 Business Practices for AI Readiness and Scaling

Fernando Lucini presents insights from Accenture's research on AI and business readiness, highlighting the belief among executives that scaling AI is crucial for growth. He discusses the challenges in scaling AI and the risks of not leveraging AI effectively. Fernando categorizes companies into three groups based on their AI maturity: proof-of-concept factories, strategic scalers, and industrial growers. He emphasizes the importance of affordability and accessibility of AI, defining and finding value in AI initiatives, and the need for a clear strategy and multidisciplinary teams to ensure successful AI adoption and scaling.

15:03

🏥 Applying AI in Healthcare: Challenges and Opportunities

Dr. Tad Funahashi shares the practitioner's view on AI in healthcare, discussing the use of AI and machine learning in various clinical applications. He talks about the potential of AI in predictive analytics, preventative care, and unlocking valuable information from unstructured data. Tad also addresses the technical challenges in AI, such as interpretability, bias, outlier cases, and the drift of medical practices over time. He stresses the importance of clinical validation, workflow integration, privacy, data governance, and the decision-making process in implementing AI solutions in healthcare.

20:05

🛠️ Implementation and Execution of AI in Healthcare

Tad continues the discussion on the hard parts of implementing AI in healthcare, focusing on clinical validation, workflow integration, privacy, data governance, and the build vs. buy dilemma. He emphasizes the need for a collaborative approach between data scientists and medical professionals to ensure the AI models are accurate, timely, and actionable. Tad also discusses the importance of protecting sensitive healthcare information and the challenges of integrating AI models into existing healthcare systems while complying with regulatory requirements.

25:07

🌐 Finding and Developing AI Talent

The panelists discuss the challenges of finding AI talent and suggest looking within the organization and partnering with academia to cultivate AI skills. They emphasize the importance of education and creating career paths within the company to develop a skilled workforce in AI. Fernando suggests creating unicorns within the company rather than finding them and stresses the need for a diverse set of skills, including those that understand the business context and can communicate effectively with end-users.

30:09

📜 Navigating AI Regulations and Compliance

The panelists address the need for organizations to understand and comply with international, regional, and national regulations regarding AI. They suggest looking inward to find individuals passionate about regulatory compliance and ethics, and outward to regulatory bodies for guidance. Kimberly and Fernando highlight the importance of being proactive and planning for future regulations, such as those requiring audits and assessments of AI systems for potential harms.

35:12

📊 Assessing and Monitoring Data Science Maturity

Tad shares his approach to assessing and monitoring the maturity of data science within an organization, emphasizing the importance of measurable outcomes and improvements in quality, patient service, and cost. He discusses the need for data scientists to understand the needs of end-users and the importance of communication between engineers and clinicians. Tad suggests that while there are many metrics to consider, focusing on the impact on healthcare providers and patients is key.

Keywords

💡AI algorithms

AI algorithms refer to the set of rules and computations that artificial intelligence systems use to make sense of data, make predictions, and solve problems. In the context of the video, these algorithms are highlighted as being abstract and complex, requiring additional business diligence due to their probabilistic nature and the inherent risks they pose.

💡Business readiness

Business readiness refers to a company's preparedness to adopt and integrate new technologies, such as AI, into its operations. It involves ensuring that the organization has the necessary infrastructure, oversight, and cultural adaptability to effectively use AI to achieve its goals.

💡Data governance

Data governance is the set of processes, policies, and standards that organizations implement to manage and protect the quality and availability of their data. It is crucial in AI initiatives to ensure that the data used by AI algorithms is accurate, reliable, and compliant with regulations.

💡Health care delivery system

The health care delivery system refers to the organization and management of health care services, including the way health care is provided, accessed, and paid for. In the context of AI, it involves leveraging technology to improve patient care, streamline operations, and enhance the overall quality and efficiency of health services.

💡Explainable AI

Explainable AI is a subfield of AI focused on creating systems whose actions can be easily understood by humans. It aims to make the decision-making process of AI algorithms transparent, allowing users to comprehend why specific predictions or actions were taken.

💡Adaptive systems

Adaptive systems are those that can change their behavior based on the data they receive or the environment they operate in. In the context of AI, it means the system can learn and improve over time by adapting to new data inputs.

💡Artificial intelligence and digital transformation

Artificial intelligence and digital transformation refer to the use of AI and other digital technologies to fundamentally change the way organizations operate and deliver value to their customers. It involves leveraging data and automation to enhance processes, innovate, and improve decision-making.

💡Machine learning engineering

Machine learning engineering is the process of designing, building, and maintaining systems that can learn from and make predictions or decisions based on data. It involves the application of statistical models and algorithms to enable computers to 'learn' from data without being explicitly programmed for every decision.

💡Bias in AI

Bias in AI refers to the tendency of AI systems to produce prejudiced or unfair outcomes due to imbalanced training data or flawed algorithms. This can lead to discrimination against certain groups or the reinforcement of existing inequalities.

💡Strategic value

Strategic value refers to the long-term benefits and advantages that an organization can gain from a particular action or initiative, aligning with its overall mission and goals. In the context of AI, it involves understanding how AI can contribute to the strategic objectives of the business.

Highlights

AI will be crucial for most organizations' futures, and understanding how to pick the right problems to solve with AI is essential.

Launching an AI initiative requires business readiness, including having a basic understanding of AI among users and appropriate oversight.

AI algorithms are abstract and probabilistic, with outputs that are predictions and inherently imprecise.

Explainable AI techniques can provide information on influential factors in algorithms' predictions but cannot determine fairness or justice.

AI systems are adaptive and respond to changes in data, but they are not creative and may revert to previously known patterns in unanticipated changes.

AI systems are typically deployed at scale, meaning small errors can magnify and become self-reinforcing without proper feedback loops.

AI algorithms are impressionable and their worldview is based solely on the data they are exposed to during training and production.

AI algorithms can make mistakes and are incautious; they require clearly defined operating conditions and robust safety controls.

Users' understanding of AI applications is critical for AI solutions to deliver intended outcomes, and monitoring their actual use is essential.

AI solutions are becoming more affordable and accessible, with tools like OpenAI's GPT-3 available as APIs for natural language processing.

Defining and finding value in AI initiatives is crucial, with a focus on identifying problems that can bring 10 times the value or savings.

Scaling value in AI requires doing the basics brilliantly, such as having strong data infrastructure and methodology.

AI is everyone's problem in a firm, and successful organizations ensure the right talent mix and alignment between strategy setters and workers.

Proceeding responsibly and building trust in AI involves creating a responsibility framework and ensuring transparency in AI's use.

Health care has a great deal of enthusiasm for AI and machine learning, with vast amounts of electronic health records offering potential for improved predictions and personalized care.

Technical challenges in AI for health care include interpretability, bias, outlier detection, and keeping up with changes in medical practice.

Clinical validation, workflow integration, privacy, data governance, and deciding whether to build or buy AI solutions are critical for successful implementation.

AI and machine learning in health care are part of a larger effort to increase operational efficiency and quality, requiring timely data access and clinical interventions.

Organizations must ensure their AI solutions comply with international, regional, and national regulations, and staying proactive in understanding and adhering to these regulations is key.

Assessing and monitoring data-science maturity in an organization involves measuring real-world outcomes, the capability of data scientists, and their understanding of end users' needs.

Transcripts

- [Abbie] Hello, and welcome to our webinar.

"Adopting AI: Ensuring Business Readiness."

I'm Abbie Lundberg. I'm a business technology researcher

and writer and president of Lundberg Media.

I'll be moderating today's discussion.

One way or another, artificial intelligence

will be important to most organizations' futures.

In Part One of this series,

we explored how to pick the right problems to solve with AI.

In the second installment, we'll examine

what it takes to launch an AI initiative

from a business readiness standpoint.

This includes making sure critical

business enables are in place,

including a basic level of understanding

of AI among users and appropriate oversight

of AI initiatives and business processes.

It also requires understanding

and mitigating the inherent risks of AI.

Our speakers today will discuss these issues and more.

They'll help you determine

if your organization is ready for AI,

sharing their expertise across a range of industries

with a special deep dive into AI use cases and

challenges in the sector that affects us all: health care.

Kimberly Nevala will start things off.

Kimberly is a strategic advisor at SAS

and an expert in the areas of advanced analytics,

information governance, and data-driven culture.

She helps clients understand

both the strategic value and the practical realities

of artificial intelligence and digital transformation.

Kimberly will be followed by Fernando Lucini,

managing director and global lead for data science

and machine learning engineering

at Accenture Applied Intelligence.

Fernando has spent more than 20 years

creating technologies to automate

and understand text, speech, and video data

and integrating these technologies

into business solutions for Fortune 100 companies

across a wide range of industries.

Dr. Tad Funahashi will provide the practitioner's view.

Tad is a practicing orthopedic surgeon

and the chief innovation officer

for Kaiser Permanente, Southern California.

He leads a team of physicians, consultants,

designers, data scientists,

and engineers who work together

across Kaiser Permanente to envision

and build the health-care delivery system of the future.

Welcome to you all.

And Kimberly, I'll turn it over to you.

- [Kimberly] Thank you, Abbie. All right.

So I'm going to kick things off today

by quickly reviewing six characteristics

of AI algorithms that require us

to apply additional business due diligence

as we design, deploy, and maintain

these systems in the world today.

So the first -- and you're probably

well aware of this -- is

that AI algorithms are abstract.

Unlike rule-based systems, the logic --

in which it's fairly easy to follow the logic

to get from A to Z, the inner workings

of AI algorithms can be almost in comprehensively complex.

And these are probabilistic systems.

So their outputs are predictions

which are imprecise by nature.

Now, techniques to be able to shine a light

on the inner workings of AI algorithms, known

as "explainable AI," are rapidly evolving.

but it's important to note that

while these techniques can give us information

about which factors most influence

in algorithms' predictions,

they cannot make the decision for us

whether those factors are, in fact,

right, fair, or just. In addition --

and this is important to know --

because AI systems are adaptive,

they respond to changes

in the data that, the data input they receive.

In other words: They've learned --

and this is both the good news

and the bad news. The solutions are very smart,

but they are not creative.

So if the behaviors or the environment

change in unanticipated ways,

the solution is not going to come up

with a novel offering or response.

It's going to revert to the best-fitting

previously known pattern.

And this is why we saw so many analytics

and AI models initially fail

when COVID came on the scene.

It's also important to note

that AI systems are typically deployed at scale.

And what this means is

that small errors can quickly become magnified,

and in fact, become self-reinforcing over time.

If we don't have really good feedback loops

that tell the algorithm when it's making the right choices

and the wrong or a suboptimal choice,

it is going to assume that the choices

that it makes are correct.

And that will then inform its future choices

and so on and so forth.

And you see where this leads;

you can see this in your day-to-day life

and your social-media feed --

for instance, when they seem like

they very quickly become hyper-focused

on a single theme or topic.

So it's important, then, as we think

about that, to know that these systems

are also highly impressionable.

And by that, what I mean by that,

is that they only know what they see.

So the data that an algorithm is exposed to

while it's in training,

and while it's in production, is its entire worldview.

It has no insight. It's completely blind to data

or factors not reflected in that information.

And it is really, really good at picking out,

sometimes just strange correlations

or really thin correlations in that data,

even if those correlations are spurious

or they don't reflect our desired future state.

And this is where we see things

like hiring algorithms that become biased against women

even though gender isn't actually an explicit data.

Now, in addition to being impressionable,

AI algorithms are a little like teenagers --

and I don't want to anthropomorphize here --

but they're incautious.

They're going to make mistakes,

but because they're not self-aware,

again, they're not going to know unless we tell them.

They also don't apply any level

of independent discretion or tact.

So it's critically important

that we clearly define the operating conditions

for these systems and then engineer

robust safety controls and resiliency into these processes.

And particularly, this is particularly important

because AI is increasingly just interwoven

into the fabric of our core business processes.

And this means that it becomes increasingly difficult

for us to always fully understand the downstream impacts

and implications of these systems.

It can also become easy...

It is also...I'm sorry, it's also easy

to become over-reliant or overly trusting

in the information being provided.

So if your self-parking car has never made a mistake,

you let your guard down,

even though it may...make a mistake in the future.

And as soon as you let your guard down,

the engagement model that that system

was designed to operate in has changed.

So ensuring that our users not only understand

how an application is intended to be used,

but also watching how they actually use it

is mission-critical to making sure

that AI solutions deliver their intended outcomes.

So there you go: Six characteristics of AI systems

that require increased business due diligence.

Now, with that said, I want to turn things

over to Fernando to talk about six,

I believe, best practices

for ensuring your AI solutions are adopted

and are to be adopted at scale, Fernando.

- [Fernando] Lovely, thank you very much for that.

I wanted to give you a perspective

as I'm part of Accenture,

which, of course, is a service company.

So we're great observers --

as we are having to live with these things,

we're also great observers.

So let me give you my six lessons

of what we see are good practices people have

when they do it, when they have that

business readiness, right?

But first, if somebody can move me

to the first slide -- we're going to "top

and tail" with this particular slide,

what I want you to tell you here.

So you understand that when we think about AI,

we have to think about everything that goes with it.

From the beginning to the end of the journey,

it never is just about the model.

It never is just about, you know,

the data, it's always about that journey

all the way from idea

to how you make this work in production.

So keep that in mind; we'll come back to this

but I just want you to --

and you'll have this slide later

so you can do your mental modeling --

but it's important to think of it as a complete problem.

Otherwise, business readiness becomes incomplete

and [inaudible].

So for the first -- first, I wanted to give you a couple

of bits of data from our latest research.

We've done research for around 1,500 companies,

C-level executives, asking them

about AI and their business-readiness.

And a couple of things stood out for me.

First of all, you see there's some interesting things:

that 84% of the executives that we interviewed

believed that they wouldn't achieve their growth objectives

unless they scale AI.

Okay, this was not a leading question.

So it's interesting that it comes like that. Second fact:

that 76% thought they will struggle to scale.

Okay, so "I need to scale to be successful,

but I'm going to struggle to scale" -- that's interesting.

And the last one, which is that 75%

believed they would be risking going out

of business if they didn't, if they didn't get the value.

By the way, to me, this tells me a bunch of things

as a practitioner and as a chief data scientist,

which is that there's a high level

of where we have to do in educating our executives

because it's, I mean, this is --

the numbers are almost like scare=level numbers.

The other [Thing] is, there's a lot of work to do

across the organization,

so they understand, you know,

how do we get to these, to this value?

But I thought it was interesting for you to see this.

And, Kimberly, you can move me to the next,

the second interesting piece that came out

of this research, which I hope

helps your thinking, is this:

We can really clearly see these customers

in three categories. The category of, the,

where most customers were,

is this first proof-of-concept factory category.

And I've given you the characteristics of each there --

and this is per the research,

as opposed to just my point of view --

where 80 to 85% were effectively getting,

not really seeing the value of the work in AI,

because it was stuck in the proof of concept,

never achieving production.

And there were a bunch of reasons why --

that misalignment with the CEO; not really,

you know, having a lot of labs,

but not a lot of abilities

to do the route to [inaudible] and so on and so forth.

Interestingly, the second category

was this, "strategic scalers." Whereas only 10

to 15% where they had figured out,

they had great connection to the strategy,

to the CEO, to work for the people

who are actually delivering on the ground.

They had a clear strategy

of what was important -- part

of the first session in this series;

we talked about that. Without the multidisciplinary teams,

all the seats, you see there.

And, finally, the third category,

which was the "industrial growers," a very small amount.

And so those folks, these guys were beyond readiness.

These were now real, you know, fast mode, right?

I always say it's a little bit,

AI is a little bit like building a car.

They don't, you've got to get the wheels on quick

so you can do miles, right?

So these guys had, you know,

the best car, they were rolling out.

They had a lot of miles behind them.

So I think the challenge for us

in business readiness is that line in between --

between the 85%, that Category One and that Category Two.

So let's talk about these five lessons

as they relate to this.

The first lesson is that this,

the AI is is affordable and easily accessible.

And this for me is, by the way,

in case you haven't read the book,

I quote Dr. Kai-Fu Lee's book about

China and AI in China. It's an incredible book.

Please read it. But in the first few chapters

he describes very clearly that China's doing great

in terms of using AI because they are using --

not necessarily because they're doing

fundamental research in AI,

but because they are great users of that technology.

And we can talk about, we'll talk about

the moral and the other responsible type

of positions later on this.

But as it relates to this -- and Kimberly,

if you can move into the next one --

an example of this, for example,

is this is a little fun thing

from OpenAI and their product called GPT-3,

which, in case you guys haven't played around with it,

is effectively a great breakthrough

in natural language processing.

But the point is that, you know,

it's something that you can use as an API.

It's readily available.

It can interpret and understand texts

in ways that it can even do simple arithmetic.

And those examples are put on the screen,

real examples where I can ask it to do a sequence,

one, three, five, seven,

and then we'll finish it as a Fibonacci sequence,

one, one, three and so on, and it will finish it.

So, and it will do this

on the basis of its understanding of texts, pure text.

So when in a world where we have all

of these tools available, like GPT-3,

likes the APS for [inaudible]

like. There is no single data scientist

in the world that builds or writes, line

by line, a model for our regression anymore.

They take it down from the internet and

it's done. It's very important

that if I taught business readiness

we have the muscle of use,

and accessibility is important.

Second one -- thank you, Kimberly -- is the definition

and finding a value. People that are doing

very well are very good at defining and finding value.

And let me give you a couple of examples.

Kimberly, if you can move me to the first one,

the first, the next slide: The research tells us

that the people that are doing great defining --

so this...by the way,

by "defining" and "finding," I mean,

"What is it that we need to do?

What's important, what's meaningful?

What's going to change the strategy?

What's going to be material to me, executing the strategy

of the company that happens

to have AI if it doesn't accelerate?"

We found that the people

that were doing it very well

tended to have a great return.

And the way I tend to use this,

as a rule of thumb that I want you to think of --

if you move into the next, Kimberly -- is this idea

of 10 times...it's critical

that as you approach problems in AI

and you think about the average,

a bank may have three, four, 500 things of AI

initiatives that they want to do that have AI in them.

Telecom companies might have a thousand

in that long, long list of strategic things

they want to do, where AI is a key part.

Let's think 10 times. What is the thing

that's going to bring us 10 times value 10 times saving?

Not a marginal benefit or a marginal saving.

Why? Because very rarely do we see people

actually approach problems little by little

and actually get there.

Given that there's so much technology available,

it's almost better to take a problem

that actually is more material and can actually

be real, a real part of your strategy

and that in itself, if it washes its own face,

as we say in the UK, if it has a good business case,

it will bring all the value behind.

So 10 times, not 10%, keep that.

Thank you, Kimberly.

Yes, the next slide.

So the next lesson is this scaling of value.

So if we look at the first one, we know we have to be,

we have to -- it's affordable, easily accessible.

So let's get using No. 2,

where we exercise the muscle

of defining what's valuable

and making sure we're doing that.

Third one is: Just do more of it.

How do we do more of this?

And this is a complicated one.

If you move onto the next slide, Kimberly, thank you.

There's many lessons here.

I'm going to give you two

or three because of the, so you can have context.

No. 1 is this idea of "do the basics brilliantly."

There's -- it's very, very rare that you can,

you can build a racing car or you can build

any kind of advanced technology without having the basics --

well, the wheels turning correctly,

the brakes working, all these things

that need to be true.

So for AI to work very well

and scale very well, you have to set yourself up

with great data infrastructure,

a great data science methodology.

That is, [one] that is geared towards getting you

to from the beginning to end,

a great route that allows you

to take that project and make it into production.

A great focus on the kind of technology

they need to scale certain things.

So think of the basics, and within this same sphere,

without moving from this slide,

you also have to think about simple --

really simple, sounds simple,

but are quite complicated,

but they are in scale-wise governance

and other things that are outside the normal realm.

Maybe they're not in mobile

outside of the realms doing this.

So do the basics brilliantly -- buy, build,

borrow strategies.

So all the kind of things that allow you

to have a great base.

So -- and, Kimberly, if you can move me,

if you can move me to the next, please.

So, next lesson is this idea of people, capability,

and culture, which sounds like the,

you know, the most obvious thing, right?

But the truth is, if you move onto the next one,

the truth is, we sometimes, we many times, get this wrong.

So what do we mean by this?

We have to bring people along for the journey.

AI is everybody's problem in a firm.

So the firms that do very well

tend to have the following characteristics.

They tend to have the right talent mix.

So I always joke that data scientists don't build products.

They are software engineers,

they are behavioral engineers,

all or an entire family of people building products.

So let's have that, let's have those families

of people doing our objectives, right?

So look at how are you doing this.

Your organization: Make sure

that the distance between the C-suite

and the people that are setting up

the strategy for the company and the people

that are doing the work is a small distance, [that]

it's quite, very well-aligned.

We see that as well.

And I say my favorite, you know,

that we used it for: buy before you build.

So don't try to, don't try to build everything yourself.

And if you look at some of the stats that

which were interesting, I'll give you one of those.

One of the ones that I like,

which is the idea of this employees

that are fully understand AI at scale,

there's a, quite a large distance

between the POC group and the scales, right?

If you can observe that where

the more we educate the firm

about their role in the AI lifecycle,

the more that we see that we become

a strategic scale-up, which is the place you want to be

because that's where

we're actually getting things done

and scaling them up.

So people can do better.

And the last one, which is this idea

of proceed responsibly and build trust,

I mean, this is an incredibly important topic.

I'm sure others, other people in this room

will agree, but what I want to,

if you move me to the next one,

just to introduce a few concepts, Kimberly,

I think what's important is several things:

that we have to create,

a responsibility framework that adapts

to our company, our citizens, our society

and that we're very open and transparent about how this is.

This is Category #1.

Let's make sure we're clear

about what it is that we're going to do

with this wonderful technology.

Secondly, we have to put the technological aspects

in place that allow us to have the guardrails to act safely.

There's going to be a lot of people

that are going to be working on AI

in a single company or a single government.

So to think that one individual

or many working together

are always going to get it right is not really fair.

So we really have to think

about the professionalization of AI

and how that professionalization

leads to doing these things responsibly.

And by the way, [inaudible]

and I mean, everybody's well understood,

you know, guard rails, methods,

all of this technical implementations

which connect with some of the numbers

you see there. I specifically like the 60%

of respondents reported that they still wanted

to have a manual override.

Of course you would. And again,

we're not trying to make that number change

but we're trying to do with building responsibility,

and understand the role of everybody.

So we can all be comfortable

about whatever the number is,

that it's actually doing the right thing

for our company, our government, our citizenship

and the world and the world at large,

which -- Kimberly, can move me to the last one,

which, funnily enough, as you can see,

it brings us back to the roadmap.

So it becomes clear that if you,

as you're going through this roadmap,

you're going from ideation to [questions about]:

"Do I have the value on the strategy?

Do I have the right people?

You know, am I setting up myself correctly

in governance to do this?

And am I actually then getting the value,

realizing the value, throughout this?"

These five lessons will bear you out,

which is: Get using. Make sure you're getting,

you're doing the valuable things,

the things that matter.

Make sure you're setting yourself up

to do more of those things.

And make sure your people

are coming with you on the journey.

And make sure you're watching for whether

it's within the boundaries,

that you've set yourself up morally

and ethically to continue that work.

And then within that, I will give you the challenge

before I hand off to Tad to take your own company

or your own government

through this framework. Start it,

go from start and see if you actually have, today,

the ability to go uninterrupted

to the end of that journey

and see where the journey is interrupted.

Think about why that's interrupted

and whether you can actually

make that continuous to give you that scale.

Kimberly, I think at this point,

I'm handing over to my friend, Tad.

- [Tad] All right, Kimberly. A really great framework

in terms of how to apply AI in the business sense.

What I thought we would do today is to take some

of those learnings that we've applied

to health care and share with you

some of the lessons that we've learned.

So the next slide, please.

So we'll touch on a little bit about

the use of AI and machine learning

in the area of health care.

We'll give you some clinical examples

where we've applied it,

and the challenges that we found in that,

and then some guides, which I think Kimberly

and Fernando have done a really good job

of already, in terms of addressing the challenges

as certainly applies to health care.

Next slide. So there's no doubt

that there's a great deal of enthusiasm

in the use of AI and machine learning and health care.

For example, in our institution,

we've collected electronic records

for almost two decades, amounting

to practically 44 petabytes of health care data.

Just to give you a relative scale of what that is,

it would be roughly a physician that's been in practice

for 170,000 to 180,000 years with perfect recollection

and near-perfect pattern recognition.

So if you could have such a doctor serving you

or taking care of you, there'll be a great advantage, right?

So we want a lot of stability.

We still be able to do [inaudible],

make that those...better make their predictions,

treat illnesses more precisely,

and really be a much more personalized type

of care that we could deliver for you,

if we could really unlock that data.

And we're also beginning

to look at data in the uses of machine learning

and AI and wearables. Next slide.

So the main areas that we're really looking at,

about patients of AI in health care

in our institution currently,

is: One, in predictive analytics,

with all of this experience in terms

of what's happened thus far, we can tap into that

and guess what might happen again in the future.

So looking at things like who's most likely

to progress in the disease,

who is most likely to be readmitted

after a hospitalization or for that matter,

who is most likely to be admitted

even if they haven't been admitted before.

Cause I think very few of you out there think

to yourselves, I have a week off next week.

I think I'll go to the hospital, right?

So if you could prevent you

from having to go to the hospital

it is something that we believe is much more

of a desired outcome for all of our members,

personal device can and risk stratification

is a key part of our use of data.

And then evidently it's well known

for preventative care, preventative care.

Thus far, you could say that we've used

crude instruments such as gender, age,

ethnicity, and whatnot to try to look at it,

people that are higher risk for certain diseases.

But if you could leverage that data

and really tailor the prevention

or even a tighter group you can really focus

on those groups and really have much better health outcomes.

Now, a lot of our health records are,

as you might imagine locked up

in our unstructured data that is in the notes of doctors,

in the notes of nurses and whatnot.

There's always been this joke

about doctors' [hand]writing, but it used

to not be electronic.

And nobody could read that writing, right?

Very hard to unlock. Well, even in the computer,

if it's natural, if it's unstructured,

we need natural language processing

and whatnot to really extract that information out of there.

And we're really beginning to develop

some advanced NLP technologies,

to be able to really unlock the value

in the critical knowledge.

That's locked in the unstructured portions

of our electronic records.

And, finally, I think all of you have heard

about image recognition use in health care,

such as retinal imaging,

dermatologic imaging, pathology imaging,

and then radiology imaging,

and we are also looking at all of those areas.

Next slide. So first, we need to talk

a little bit about the technical challenges,

so to speak, of doing modeling

for AI, machine learning, and health.

And I think some of these were covered previously very well

by our previous speakers: interpretability

or explainability, bias out by our edge cases,

and drift. Let's look a little bit more carefully

at each one of these. Next slide.

So in terms of interpretability:

Doctors and healthcare workers are famous

for not believing about anything

that's sent to them, right?

So there's a new study that comes out

and it's a definitive, randomized, double-blinded study

and they go, "Ha! Let me look

at that to see if it's really right."

And, in fact, that's one

of the reasons that any kind of new discovery in health care

takes about 17 years before it becomes the norm.

Well, imagine if you gave somebody a blinding insight

about what's about to happen,

but it came out of a black box, so doctors...

So we have to figure out a way

to make the data that comes out,

a little bit more interpretable,

and sometimes, interestingly, rather

than creating tremendously complex models,

we sometimes want to simplify the model,

so it's a little bit more explainable

and interpretable to the physicians. Now bias

is something that's innate in health care, right?

Most bias can be good, but some bias can be bad.

That is, we know that certain populations

that are higher-risk, that it's innate bias

in the statistics, right?

But if the bias is nefarious --

meaning that it doesn't improve the outcome

of those people -- then it's a negative type of bias.

And it could be anything from ethnic groups

and socioeconomic groups and whatnot.

We need to carefully analyze

that to make sure that the historic practices

of the past that had innate biases

that we didn't detect aren't replicated

in the future. Outlier cases, in fact,

are the ones that we stay up at night [about].

The reason you want to come see a doctor, instead

of Doctor Google, is that most of the time,

you're going to be fine with whatever it is

that's common. However, there are certain cases

that are atypical, the red herrings,

And when everybody is in my office

and I'm examining them,

I'm looking not only at what they most likely have,

but what they could possibly have.

And, unfortunately, right now

AI is not that good at detecting those.

It's more of a statistical norm, rather

than looking for outliers.

And as you know, health care practice

changes over the course of time.

We don't practice the way we did a hundred years ago,

50 years ago, or for that matter, even five, 10 years ago.

But remember: AI looks at the empiric data

that is available from our past medical records

and replicates that. I think

Kimberly talked about this earlier.

And so as practice begins to drift,

we need to make sure that our recommendations

that come out of the AI systems

are also updated and continuously improved.

Next slide. Now my data scientist

and engineer tells me all those problems

that we just talked about are the easy part.

The hard part, in actuality, is

in the implementation and execution.

That is clinical validation,

workflow integration, privacy, data governance

and MIP, and a decision

as to whether to build or buy, to create these.

Let's take it a little bit deeper.

Look at these in terms of clinical validation.

We need to make sure that the data scientists

and the doctors work hand in hand closely

so that the predictions that that machine is making

in fact makes sense.

No. 1. And No. 2 is [it's] accurate, right?

So that it's not making wrong predictions.

And then, ultimately in that process,

so that the physicians begin to develop a trust

in that data that's coming, ultimately AI,

or artificial intelligence,

in health care will most likely

become augmented intelligence.

That is a pinging of things

that we should be thinking about

or broadening our diagnostic diagnosis

in clinical arenas that AI can help us do.

Perhaps one of the most important

is workflow integration.

If the data isn't given to us timely,

if the data isn't relevant,

if it's not actionable or interpretable,

and we don't trust the data, and it's not a part

of our decision-making process

that we typically do in taking care of our patients,

all the fancy arithmetic

and the predictions are not that helpful.

But we need to make sure

that the productions of the model are presented

to the physicians in a usable manner

that's timely, that's meaningful.

It goes without saying that health care information

is incredibly sensitive.

So privacy issues are critical for us.

If we ever do share data with external companies,

and just to get into the "build or buy,"

that protection of that data,

what happens to the data after it leaves,

so to speak, our walls? The governance

of that data, ultimately even the intellectual property

that's associated with it, can really be challenges

in terms of how best to utilize that data,

because sometimes there's great engineering resources

outside, so to speak, our walls of Kaiser Permanente.

And yet, how do we leverage that expertise

without putting any of the privacy

or ownership issues at risk

in terms of build or buy?

There are models that are built

for a variety of hospital systems and

health care systems. And yet geography,

financial incentives, a variety of other things

can have slightly different impact on the data.

And so that data in the models that are built

on that are not necessarily directly transferable.

So we need to pay attention to that.

And, yeah, if you say, we want to make sure

that we build it, that's customized for our database.

It requires tremendous CPU, GPU, engineers' time,

and it can be extremely expensive to build.

And it obviously to continue to keep it updated.

So that's another area

that we need to keep attention to. Next slide.

And so if we really began to think

about the application of AI

or augmented intelligence for health care

and machine learning, it really is only one part

of the creation towards increased operational efficiency

and quality that we look at in health care.

And so it's a tool that can be potentially helpful,

but an amazingly accurate tool that's 99% accurate.

It isn't really that helpful

if you don't have timely access to the data,

if there are clinical interventions

that make any difference, you can have

If you know when it's going to rain,

but you don't have an umbrella ...

So that doesn't help you, right?

But the same thing in health care:

We know something's going to happen

but there's nothing we can do about it.

It can be sometimes frustrating

and we need to make sure

that when we do get the data insights

that there are supporting processes

and workflows that allow the intervention to happen.

We need to make sure that there are trained resources

to make that happen.

And then there's appropriate infrastructure

for not only scaling it, but also in terms

of making sure that the model is interoperable

across our entire enterprise.

And for example, we do that.

We want to make sure we have metrics that look

at both the performance of it,

from making sure that there's no bias,

that the quality is there,

in fact, it has good impact on our costs,

and ultimately that it's serving our members

in a much more efficient manner.

The next slide. So if we were to put this all together,

we kind of take a step-wise approach to that.

So as we look at applying AI,

or any large models like this -- First,

we want to make sure that we're not doing it

just because it's fun, but it's actually

solving a specific clinical problem

that we typically couldn't solve as well,

or could begin, like Fernando said,

really creating a different type of model for health care.

And once we do that, we want to make sure

that the engineers and the physicians

work side by side all along,

that the physicians have faith

in what the engineers and the models are outputting,

and that the engineers truly understand

what they're solving for.

Of course, since we get so much data,

we need to make sure that the data quality is good,

that we adjudicate data that may be coming

from slightly different sources,

that the quality of the data is at the highest level

that it can be, and then once we build

that model and steps one, two, three

are all about the model building

and the outputs are objective, right?

And that unwanted bias is not programmed into that model.

And then in the lower three, four, five,

and six are really about execution.

We need to make sure

that once we do put that in there,

that there is value in terms of quality outcomes,

service outcomes, cost, whatever it might be,

that it is in fact something

that is a value to the organization.

And as we rolled out these things,

we need to make sure that its performance,

even though it may have been good to begin with,

isn't beginning to decay over the course of time.

And I think the last goes without saying,

and certainly in health care, regulatory requirements

and oversight of all of these models

are absolutely essential to make sure

that we are compliant.

So with that let me now turn it back over

to Kimberly and Abbie, I believe,

who're going to jump us into the Q and A section of this.

Thank you, everybody, for your attention

and I look forward to our discussion.

- [Abbie] Great. Thank you for those great presentations.

Really useful information.

And we do have some great questions coming

in from the audience, and we will continue

to take your questions for the remainder of the hour

so you can keep submitting those.

To kick things off,

so there's -- there, we've gotten questions

around a couple of different areas

that I wanted to start with, the state of AI right now.

And so this question was how mature

are AI solutions now, can you find one

for any application, and how much

are customization or tailored solutions required?

So, you know, can you go out and just buy something

for a problem that you're trying to solve?

Or how much do you really need

to be customizing and doing your own work on that

or hiring somebody to do your work on that?

And I guess that's a question

for all of you. Kimberly, you want to start?

- [Kimberly] Yeah, absolutely, and I think Fernando

probably has some good experience in this area.

Well, I think that in a lot of ways,

AI is very ubiquitous in that,

a lot of the solutions that you buy

at the shop today, and a lot of cases

may have these advanced analytics capabilities

built into them. But as Tad said,

I think in a lot of cases,

what you're looking to do is to buy some

of the core capabilities,

whether that is computer vision capabilities,

natural language processing,

and then customize those

to the specific use case that is there.

So today I think the build versus buy,

there's probably still a lot more customization

that happens even when we're buying a solution

that's packaged with artificial intelligence.

It's really important in those cases

that you ensure the environment and the operating...

whether that's sort of the customer,

the way it interacts with your customers

or your consumers, your users,

the operating environment that you are deploying

that actually matches the environment it was trained in --

i.e, that it reflects the data that it was trained on.

And there's really no sort of fast track

through that part of the process,

I don't believe. But in a lot of cases.

I think most organizations find that in,

especially around core business processes,

whether that's in sales or customer experiences

with these areas, there are packaged solutions

that you can begin with validated

and customized to your needs.

That being said, there are also a lot

of off-the-shelf now packages

around that are the data science packages.

So you can get the math,

if you will, you can find organizations

that these algorithms that come out of the box,

that you can apply to your data

that come packaged with interpretability

and explainability and some of these components.

The trick again is really matching those solutions

so that you're finding for the right problem

with the sort of right hammer.

And that's probably the bigger issue right now --

not so much finding an available tool,

but the features in that the tools

that are available actually

match the problem you're trying to solve.

Fernando, your thoughts.

- [Fernando] Yeah, I started a different perspective,

which is that, if you think about it,

today, if you're a startup and you don't do some AI,

you're probably not going to get any money.

So there's tools for anything you can think of,

there's AI's in or AI's for most of you can do.

I do a little test that may make you smile,

which is to think about from the moment you wake up

to the moment you go to sleep,

write down where you feel AI

has been part of your life.

If you know a little bit about AI,

like it's my world, the list is enormous.

It's enormous. You give it to my mom and nothing, obviously.

So our perception of this.

So what I will tell you is there's tools for everything.

There's things that you can use really easily

but doing what Tad has described is mega-difficult.

So there's a lot of things available, but

it doesn't mean you're actually going to get what you need.

It's almost an excess, an excess of stuff,

but still the people that didn't do very well

still tend to be super-advanced in putting it,

which is what you're saying, Kimberly, putting it together

and making sure it starts a purpose

and in the context of a company

and in the context of all these other things.

So, and that's why I have that first lesson,

which is availability.

I mean, it's there; doesn't mean it's easy, right?

But there, absolutely, totally present.

- [Tad] Yeah, I would agree.

I think the number of tools that are available

in the area of AI is I would make it -- I can go

an auto shop or a Home Depot, right?

There's tons of tools out there.

And you can build just about anything

with all the tools that are out there.

And, in fact, we do leverage many of those tools.

At the end of the day, though,

you have to build those tools

to build something that you need

that's unique to your particular business, right?

And in order to do that, just having

a collection of tools doesn't do it.

You have to put it all together.

And not only after you put it all together,

you have to have somebody that's willing to drive it,

so to speak, right? The end user.

So that not only are the ... the tools important,

the user interface and the method

by which the output of those tools are given

to the end user is really critical, right?

So if you think about driving a car

and we didn't have those simple tools

that allowed you to steer and accelerate and brake,

A car could be incredibly hard to drive.

And if it's incredibly hard to drive, people

aren't going to use it, right?

And if you think about it, though,

all of the components of that car --

incredibly complex; in fact, many of them leverage AI now.

And I think it's not that dissimilar in

most industries and certainly not dissimilar in health care.

- [Abbie] Great, thanks. You know,

we've got a lot of questions coming in

that are very appropriately,

because we were talking about business readiness --

these questions are around the people part of doing AI.

So the first one is: Who should be on the AI team?

Who do you need on your AI team?

And Tad, if you want to start on that one

that would be great.

- [Tad] Yeah, fair.

So, first of all, the basics, right?

You need the people that understand the computer --

the engineers, the data scientists --

and those are just table stakes.

Without that, there is no data science, right?

Having said that, though, just them in a vacuum

doesn't do it for us.

We actually need the end users, for a variety of reasons.

No. 1, we need to know

and understand exactly what their problem is

and what the type of solution would best help their problem.

Right? So we need the end users.

Sometimes, though, the interpretation of that

by engineers lacks a set, so to speak.

So human-centered design people

are really critical not only to translate

what the end users are saying that they need,

but also to create the interface back

from the engineers to those users.

So it becomes very understandable, right?

And interestingly, also keeping our team --

of course, both consultants

that can manage the project,

but also classic epidemiologists.

And the statisticians that can begin to

bridge that puts that, that, that gap

for lack of a better word, between traditional statistics

that we are very comfortable with in machine learning.

So that physicians begin to understand

what's really happening.

And even with that, the interpretability,

or the explainability can be difficult.

But without it, that trust, you know,

what's happened, can be lacking.

So certainly in our field,

that's also another critical component.

- [Abbie] Fernando, you talked about the importance of --

I think your slide was people, capabilities,

and culture, and you talked about --

so when you think about that culture piece,

I was really thinking about the fact

that, you know, changing culture takes time, right?

And you also talked about the importance

of moving quickly when you're moving forward with AI.

So how do you balance the importance

of moving quickly with the fact

that culture does take time to change?

Like where do you put your focus

to sort of create, get past that dynamic?

- [Fernando] Yeah, the simple answer for me is education.

It's one of those moments where --

and by the way, COVID has an interesting effect on this,

which is that before COVID...many CEOs

were thinking, "Oh, my God!

Where's all this money that I spent on AI

that doesn't seem to have produced any value?"

And suddenly COVID hits,

and it's been an acceleration

through survival for many that needed it.

And I'm sure, Tad, you've seen a lot of this

in the health care industry.

So AI has become an accelerator

to that culture point because it was survival.

I don't know which one, if it's continuing,

going to continue to burn this bright

and help push the -- obviously,

we want this pandemic to go away, thank you.

But, but apart from that,

is it going to be have a tail

that continues to help us with this culture?

Abbie, we have the same problem with data,

in that we talk about data-driven culture,

and the truth is data-driven culture

is a culture of self-service of,

"I can serve myself, I can do things.

I am a creator of data and a user of data."

AI is a bit more complicated,

because you don't wake up in the morning

and decide you're going to create a neural net,

and you figured out the mathematics of that.

But if you follow Tad's great way of putting it,

you can learn what your role is in it.

So I think education across the widest possible base

of companies and governments on the possibilities of AI

not just the use cases, but a real notional understanding

of what it is -- not to understand the mathematics

but the notions will help remove

the humanizing view that we have of this technology,

of these machines, and help us with,

"Ah! I understand more or less

how these things kind of work, so I can

understand why this is going to work a particular way.

I've got this notional view of this."

And that really helps accelerate these things.

And, about education...

if you look at the winners, the people that do really well,

it's all education; it's a hundred percent education.

And I can give you a fun way to think about this:

...the people we don't talk about a lot:

which is, if you put five data scientists

in a room for three hours and you come back,

they've done five models in five different machines

in five different languages

that will never work together.

If you put five software engineers,

you come back five, three hours later,

there'll be no code written

but they'll have worked out how to work together.

They will have worked out

how to share; everything will be sorted.

We need -- there's a set of people

called ML engineers, who are the performance

and scaling engineering of models,

who today are probably fighting,

you know, to throw my hat out to those

that career, is probably the people

that we need the most today to make that translation.

They're the people that are going to take those two rooms

and remove the partition, so they can be together.

And we need a lot of those.

- [Abbie] It sounds like the last people you talked about,

those are people that you need

to have on your team, inside the company.

but there's other roles that maybe

you've got to go out and get

that have have particular expertise

around the data science

and the real technical parts of doing AI.

So this question is from someone in the audience;

the question is: Where do you find

the people to create AI solutions?

And they're thinking in terms of

you know, companies, countries,

talent, marketplaces. Where should people

go to look to get that AI talent?

Kimberly, you're on mute.

- [Kimberly] So let me start again, but I'll start.

And then I'll go to Fernando and Tad.

So one of the things I think that is becoming increasingly

interesting is partnerships between industry and academia.

And that could be whether you're utilizing

friends working with folks that are in college programs,

and even PhD and advanced programs, to come in

and work on your projects

as part of their educational bit.

At the end of last year, I had hosted

a panel with the head of basically machine learning,

AI deployment for Uber.

And one of the things that they did

that was really interesting,

they essentially have a process by

which folks who are actively

pursuing the research aspect,

which is really important in academia,

are part-time working also for Uber

and looking at the operational application

of these components and balancing that, sort of,

you're getting this intersection

of industry experts and business arguments

that probably works in the real world

and not sucking away all the talent

from the R&D side and that sort of basic science

and research that's also required.

And I think more and more,

we're going to start to see those kinds

of -- maybe not in that way --

but those kinds of interactions

and partnerships between things like academia,

some of the research organizations,

and commercial organizations as well.

So I think that's one area to look at.

I also don't think -- Don't underestimate

the people who are interested and willing

within your organization today

to actually be trained up

and to come into these areas to work.

I spoke to Michael Kanaan, who was chair of AI

for the Air Force, and they did actually

an internal survey and realized that they had

a lot of folks that were engineers

and not necessarily computer engineers,

but who had a lot of the skills and talents

and interests in this area.

And so being able to make the move

on training internally available

really resulted in dividends for them

in terms of being able to build

a skilled workforce over time.

So those are some, maybe not traditional, ways

of thinking about how do we develop

and build or buy what can be very scarce

and expensive resources.

- [Tad] I would just jump in. Really good comments,

Kimberly, and to help the questioner

look for that talent:

Don't underestimate the talent

that you actually have in your own organization.

Data is a very interesting view source, right?

And you actually need a data maven

in your organization that truly understands

the where, that context, and the availability of the data.

You can have the most advanced software

and data science in the world come into your organization.

And we've certainly done that, right?

And we say, okay, we will say, "Okay,

go ahead and hop into our 44 petabytes of data."

But they don't know the context of data,

the availability of data,

the adjudication of the data,

the quality of the data, all of that.

So you need somebody in your company that really understands

to really leverage the data well.

And then on the flip side,

or the way on the other side,

you also need people that understand

the power of the data and its applicability

in your organization.

Those two sources will come

from within your organization.

And then, so to speak, the toolmakers, right?

The people that really understand

the building of the model,

that you can either hire them in-house --

or, as Kimberly said, I'm sure

Fernando will say -- you can outsource that.

But they have to be people that can communicate well

with your data mavens

and people that really understand it,

and the end users. Otherwise, you have this aggregated,

this team that does not create for you,

anything that's particularly effective

for your organization.

- [Fernando] Can I quickly come in on that, Abbie?

Is that all right?

- [Abbie] Yeah, before you do --

- [Fernando] Because that's from a different --

- [Abbie] Before you do, I just want to ask Tad

if you could mute your microphone

when you're not speaking; we're getting

a little bit of feedback.

Go ahead, Fernando.

- [Fernando] Thanks, Abbie. This is interesting for me.

So we talk in Accenture about making unicorns,

not finding unicorns. But the reason we do is

because we've done a study

and a thought process of what is the future of data science?

Where is the data science going to be in three years?

Not for everybody, for us,

as it relates to us, very inward.

And the truth is what came out of that

was that if you think three years

in the future, it's very unlikely

that the world is going to be full

of generic data scientists.

Remember, data scientist is a profession.

It's a profession like Doctor Tad, right, is a doctor.

This is his profession.

You don't wake up in the morning

and decide you're a data scientist.

So the best thing we can do

is create those careers in companies,

so we can be proud of having a career track,

which creates you these wonderful people

that makes your industry problem.

And, you know, and I'm actually,

to be honest, sad: Teaching hospitals

are like the best example in the world of this.

I mean, there's no better example.

So you almost need like your teaching hospital

of AI within firms to create careers for people.

And that's what's going to set you up,

because if in three years we think

we're just going to be able to pick

these unicorns that understand my industry

and understand this, and we're going to be in big trouble.

So if I were to give anybody advice --

I did this for a bunch of CEOs in the

last few weeks -- is create a career, create careers.

If you create careers, you're winning,

then you don't need anybody.

You don't need me, you don't need anybody.

you just need to create that career.

- [Kimberly] And I'll take it,

I'll extend that a little further.

One of the things that we talk about --

and I think whether you're building

the sort of internal curriculum

and career paths or borrowing

and leveraging external -- we talk a lot about STEM.

So science, technology, engineering, math --

absolutely critical and core

to what we're talking about here.

But you're also now starting to hear people

talk about, I think the new acronym I've seen is SHAPE,

which is about, you know, sociology,

humanities, ethics, sustainability.

So that somewhat the softer side of this --

which is actually the harder piece of this

in terms of adoption -- really thinking more critically

about how people want to interact

with the solutions, what the impact

of these solutions are -- not just on users

in terms of how they work

and how it impacts our jobs,

but also on the populations

are being applied to. I think, Tad,

you spoke to some examples of this in health care,

where we really have to make sure

that it's not just the users

that understand the solution

and are bought into how it's being applied,

but also the populations we're using it

on -- and hopefully for and not against.

And so as you look to develop

these curriculum skills, make sure

that you're not just building

these bastions of STEM that don't also incorporate

and develop those skills associated with SHAPE, or

however you want to call that out.

- [Abbie] We have time for one or two more quick questions.

This one is a little bit different.

When you're thinking about business readiness,

what do organizations have to do

to understand and comply

with the international, regional,

and national regulations and regulatory bodies

regarding the use of AI?

- [Fernando] I can have a go, if you want.

- [Kimberly] Yeah, please.

- [Fernando] I'm sure, Kim, it's an interesting one,

because every company I know

has a body that actually reads and understands

the regulations for them and every...

single one of those departments

will tell you that nobody seems to really pay attention

in depth to those. So GDPR is a wonderful example.

If you think in Europe about protecting your data,

even further because of AI, and you think,

"Oh, my God! Have you read GDPR?"

There's no chance that we're going to misuse data --

no chance, no chance, not possible.

So there, so my suggestion is,

please look at -- inside your company,

you will find somebody who cares deeply

about how to use technology like this.

Otherwise, the regulators around the world,

are doing a good job of trying

to explain what their position is,

even if their position is very young

and very instilled thinking,

and still moving. Go and look;

it's very easy to find, even on Google,

the regulators' pages for how they believe

AI should be used.

So either look inward and you'll get great advice or

look to the regulators, they'll have great advice as well.

- [Kimberly] Yeah, and some of this is, you know,

a lot of the regulations today,

Fernando referenced GDPR,

which has provisions that then also apply

to AI or what they call automated decision systems

and automated systems.

There are a lot of emerging regulations.

So there's a lot of principles today.

There's a lot of fundamental foundational statements now

about what we want to do.

There aren't necessarily outside

of what have been traditional

legal-regulatory concerns,

around things like data privacy

and security that are specific to AI,

but they are coming.

So the other thing to be aware of here is: Don't assume.

You have to start looking forward a little bit

and try to start to project

and predict a bit of where this is going to go

and build these things again in advance.

Some of the new EU regulations,

for instance, will require companies

that are using AI-driven systems, right?

AI-enabled systems to actually

audit those systems and perform

some assessment and due diligence

of potential harms, for instance,

and inventory and make that inventory available.

None of these are set in stone today,

but having folks, as Fernando said,

that are passionate about it,

working in that space today,

whether it's compliance or ethics, will be cool.

But also starting to look forward

and plan proactively will be very important

I understand that a lot of times,

legal and regulatory compliance

tends to come after harm

has been done or, you know, ill has occurred.

And so hopefully, we can run some lessons here

and start to get, as organizations,

in front of that with or without

the mandate of regulation,

which I think is required and is coming.

- [Abbie] Tad, I'm going to -- I mean, yes, Tad,

I'm going to throw this one to you.

What would be the, what is a good practice

to assess and monitor data-science maturity

in your organization?

I assume that's something that keeps changing.

And do you have a process in place

for sort of making sure that the things

that you're trying to do,

you have that maturity to be able to accomplish

what you're trying to do?

- [Tad] Well, first let me take myself off mute.

Wow! That is a really complicated question, right?

Not that easily answered. But, you know,

working backwards, I think we said in our presentation

you want to make sure that the solutions

that are being built have some pragmatic

and real-world measurable outcomes and improvement, right?

Otherwise, it's all an academic exercise, right?

So we do look at the, make sure that any application there

has actually had a positive impact

on quality, on patient service, cost,

providers sustainability.

Provider sustainability is a significant issue

for us right now, especially having coming out

of that COVID surge, which really

put a great strain on our providers.

So we do look at those as key metrics

for any type of not only AI, but almost any type

of [inaudible] process that we put into place.

Of course, the understanding,

so to speak the, the capability

of that data scientist to build the models

that we're looking for is also important.

But assuming that we hired the right people,

with the right qualifications, the other part

that's really key is for that data scientists,

to be able in the health care environment,

to understand, truly understand

the needs of the end users in terms of physicians.

And we put them in front of physicians

and physician groups as you saw earlier.

And if they can't communicate,

the most brilliant data scientist

or the engineer that can build amazing things for you,

but doesn't understand that the needs

or the language of the clinicians

that they're building these tools for,

are not going to be as effective.

So just in brief, a couple of key things

that we look at. Obviously, there's multitude

of other metrics that we need to look at,

but I would say those are some of the key things.

- [Abbie] Great. And I wish we had time to get that answer

from everybody, but we are just about out of time.

So, thank you, Kimberly, Fernando, and Tad

for this really interesting and useful talk

and thank you in the audience for your great questions.

I wish we had time for more of them.

And a final thank you to SAS for sponsoring this webinar.

We hope you'll all join us for the next in the series

which is on April 22nd, where we'll cover

how to ensure the technical readiness

of your organization for AI.

Thank you so much.