AI for Business: #5 How to do AI Experiments?

Summary

This episode of the AI for Business course delves into the importance of conducting proof of concept (PoC) experiments for AI projects. It outlines the necessity of testing AI's viability, gathering stakeholder feedback, and planning for production. The script guides through 10 key elements for a successful PoC, using an industrial visual inspection case study. It emphasizes setting clear success criteria, managing data, selecting tools, and assessing the outcomes to inform production planning, ultimately aiming to minimize risk and investment before full-scale implementation.

Takeaways

• 🧐 Proof of Concept (PoC) is essential for AI projects to manage uncertainty and assess the feasibility of an idea before significant investment.
• 🔍 PoCs serve three main purposes: testing the AI approach, gathering early feedback from stakeholders, and aiding in production planning.
• 📝 Planning a PoC involves considering 10 key elements: problem definition, hypothesis, scope, success criteria, data, modeling and tools, infrastructure, deliverables, team, and timeline.
• 🔑 Success criteria for a PoC should be clear and realistic, focusing on improvements over the current state, such as quality, speed, or cost efficiency.
• 📈 Data is crucial for PoCs, requiring a well-defined dataset to train and evaluate the AI model, with annotations where necessary.
• 🛠️ Modeling and tools define the techniques and technologies used in the PoC, which may include cloud AI services or custom deep learning frameworks.
• 💻 Infrastructure needs for a PoC might include powerful hardware for model training, especially when using deep learning frameworks like PyTorch.
• 📑 Deliverables from a PoC should include code, documentation, knowledge transfer, a prototype application, and recommendations for scaling to production.
• 👥 A balanced team for a PoC should consist of data scientists, domain experts, and project managers to cover AI, data engineering, and domain knowledge aspects.
• ⏱ The duration of a PoC should be concise, typically ranging from 3 to 8 weeks, to quickly test the idea with minimal investment.
• 🤔 Post-PoC critical questions include assessing if AI is the right solution, expected accuracy boost, reasons for subpar results, and whether to use off-the-shelf models or build from scratch for production.

Q & A

What is the primary purpose of a Proof of Concept (PoC) in AI projects?

The primary purpose of a PoC in AI projects is to test the feasibility of an AI solution, gather early feedback from stakeholders, and assist in production planning by providing insights into the complexity, time, effort, and cost involved in building a full-fledged production system.

Why are PoCs especially critical for AI projects compared to other types of projects?

PoCs are especially critical for AI projects due to the inherent uncertainty and the need to validate whether AI is the right approach to solving the problem, the availability of required data and skills, and to ensure the outcome serves the business needs effectively.

What are the three main reasons for conducting a PoC?

The three main reasons for conducting a PoC are testing to answer essential questions quickly, feedback to ensure the solution meets stakeholders' needs and increases adoption, and production planning to understand the resources required for a full-scale implementation.

Can you explain the importance of setting clear success criteria for a PoC?

Setting clear success criteria for a PoC is essential as it provides measurable goals to assess the PoC's effectiveness against the current state of affairs. It helps determine if the PoC has shown significant improvement in terms of quality, speed, cost, or other business metrics.

What should be the focus of the scope when planning an AI PoC?

The scope of an AI PoC should be limited and focused on building the model rather than a full production system. It should concentrate on a specific subset of products, geography, or a narrowed-down problem to limit variables and complexity, making the project more manageable and easier to test.

Why is it important to have a team with diverse skills for a PoC?

A diverse team is important for a PoC as it ensures coverage of AI and data engineering, domain knowledge, and project management aspects. This interdisciplinary approach helps in effectively addressing the technical and practical challenges that may arise during the PoC.

What are some examples of deliverables one might expect from a PoC?

Deliverables from a PoC may include a trained AI model, a data pipeline for preprocessing and transforming data, a code repository with source code, a detailed report on model development and evaluation, a prototype application for testing the model, and recommendations for scaling to a production system.

How long should a typical PoC take to complete?

A typical PoC should ideally take between 3 to 8 weeks, focusing on testing the idea quickly with minimal investment rather than spending excessive time on it.

What are some critical questions to ask after completing a PoC?

Critical questions post-PoC include assessing if AI is the right solution for the problem, estimating the expected accuracy boost for a production system, identifying reasons for suboptimal results and deciding on further investment, choosing between off-the-shelf models or building from scratch, and estimating the total expected cost for a production system.

What is the significance of the infrastructure element in planning an AI PoC?

The infrastructure element defines the hardware, storage, and compute resources required for the PoC. It's crucial for training the model and, if necessary, for deploying the model in a prototype application. The choice between cloud services and on-premise solutions will impact the infrastructure needs.

How does the outcome of a PoC help in planning for a production system?

The outcome of a PoC provides insights into the feasibility, potential challenges, and the expected performance of the AI solution. It helps in making informed decisions about the resources, budget, and timeline required for a production system, and in building a business case for the investment.

Outlines

00:00

🚀 Introduction to AI Proof of Concept

The first paragraph introduces the concept of a Proof of Concept (PoC) in AI projects, emphasizing its importance due to the inherent uncertainty in AI endeavors. It sets the stage for the AI for Business course's fifth episode, which focuses on the initial phase of bringing AI ideas to life through PoC experiments. The paragraph outlines the necessity of PoCs for testing AI feasibility, gathering early feedback, and aiding in production planning. It also introduces the 10 elements to consider when planning an AI PoC, using the example of automating industrial visual inspection with deep learning.

05:01

🔍 PoC Planning: Elements and Execution

This paragraph delves into the specifics of planning an AI PoC, discussing the 10 elements in detail. It explains the significance of defining a problem hypothesis, setting limited scope, establishing success criteria, and gathering data. The paragraph further elaborates on the choice of modeling techniques and tools, infrastructure needs, deliverables, team composition, and the timeline for the PoC. Using the industrial visual inspection example, it illustrates how to apply these elements to develop a computer vision model to detect defects in products.

10:02

📈 Assessing PoC Outcomes and Next Steps

The third paragraph focuses on evaluating the outcomes of a PoC and determining the subsequent steps. It highlights the importance of understanding whether AI is the optimal solution, estimating the potential accuracy improvement for a production system, and identifying reasons for subpar PoC results. The paragraph also addresses the decision-making process regarding whether to conduct another round of PoC or to move to production, considering factors like data sufficiency, team skills, and problem relevance.

15:04

🛠 Decision Making Post-PoC: Tools and Strategies

This paragraph discusses the critical decisions to be made after a PoC, such as whether to use off-the-shelf models or build a system from scratch. It explores the advantages and disadvantages of leveraging existing AI services versus custom development. The paragraph also emphasizes the importance of assessing the total expected cost for a production system based on the PoC findings, including infrastructure, software engineering, and ongoing maintenance. It guides on building a business case for the production system by evaluating ROI and making informed decisions.

20:05

🌟 Wrapping Up and Future Outlook

The final paragraph wraps up the discussion on PoCs, summarizing their role in validating AI ideas and planning for production systems. It previews the next episode's topic, which will cover building AI capacity, including team building, skills assessment, and the considerations of in-house development versus outsourcing. The paragraph concludes with an invitation to share the episode and an acknowledgment of the audience's time, promising further insights in the upcoming episodes.

Keywords

💡Proof of Concept (PoC)

A Proof of Concept (PoC) is an initial demonstration of a concept or idea to evaluate its feasibility and potential value. In the context of the video, a PoC is critical for AI projects as it allows businesses to test whether AI is the right approach to a problem, gather early feedback, and plan for production. The script mentions that PoCs are especially important for AI due to their inherent uncertainty and the need to minimize risk before significant investment.

💡AI Projects

AI Projects refer to initiatives that involve the application of artificial intelligence to solve complex problems or improve business processes. The video discusses the challenges of initiating AI projects due to their uncertain nature and how a PoC can be a valuable first step in the project lifecycle, as seen in the exploration of automating industrial visual inspection with AI.

💡Uncertainty

Uncertainty in the video script refers to the unpredictability and risk associated with AI projects, which can make it difficult to determine the value of pursuing a particular AI idea. The concept is integral to the discussion on why PoCs are necessary, as they help mitigate this uncertainty by providing a low-risk way to test AI hypotheses.

💡Stakeholders

Stakeholders are individuals or groups who have an interest in the outcome of a project. In the video, the importance of getting early feedback from stakeholders is emphasized to ensure the AI solution meets their needs and to increase their buy-in and adoption of the solution.

💡Production Planning

Production planning involves the process of preparing for the full-scale implementation of a project or system. The script discusses how PoCs aid in production planning by providing insights into the complexity, time, effort, and cost involved in building a full-fledged production system, which is essential for businesses looking to implement AI capabilities.

💡Deep Learning

Deep Learning is a subset of machine learning that uses artificial neural networks with multiple layers to model and solve complex problems. The video script uses the example of automating industrial visual inspection, where deep learning is employed to develop a model capable of detecting defects in products or parts.

💡Computer Vision

Computer Vision is a field of AI that enables computers to interpret and understand visual information from the world, such as images and videos. In the script, a PoC for computer vision is used to develop a deep learning model to detect defects in industrial products, illustrating the application of this technology in real-world scenarios.

💡Success Criteria

Success criteria are the predefined metrics or conditions that determine whether a project or experiment has achieved its objectives. The video emphasizes the importance of setting clear success criteria for a PoC, such as improving quality, speed, or cost, and provides an example where the criteria include a 70% accuracy rate and the ability to inspect 100 images in under a minute.

💡Data Sets

Data sets are collections of data that are used for training and testing AI models. The script mentions the use of a data set of 5,000 annotated images for the PoC in industrial visual inspection, highlighting the importance of having a robust and relevant data set for the success of an AI project.

💡Infrastructure

Infrastructure in the context of the video refers to the hardware, storage, and compute resources required to support the development and operation of an AI model. The script discusses the need for infrastructure when using cloud AI services like Amazon Lookout for Vision or when training custom models with frameworks like PyTorch.

💡ROI (Return on Investment)

ROI is a measure used to evaluate the efficiency of an investment or compare the profitability of different investments. The video script discusses assessing the ROI of AI projects after a PoC to determine if the potential benefits justify the costs, emphasizing the importance of understanding the financial implications of pursuing AI solutions.

💡ETL (Extract, Transform, Load)

ETL refers to the process of extracting data from various sources, transforming it to fit operational needs, and loading it into a target system for further use. In the script, ETL is mentioned as a component of production planning, where it is necessary to automate the flow of data into and out of the AI model within a production system.

Highlights

AI projects often have uncertainty, making it challenging to determine the viability of ideas without significant investment.

Proof of concept (POC) experiments can help mitigate risks before substantial investments in AI projects.

Proof of concept experiments are critical for testing, obtaining feedback, and planning production.

Testing during a POC answers essential questions quickly, such as the suitability of AI for the problem and data requirements.

Early feedback from stakeholders during a POC ensures the AI solution meets their needs and increases adoption.

POCs aid in production planning by providing insights into complexity, time, effort, and cost for building full-fledged systems.

Ten elements to consider in an AI POC include the problem, hypothesis, scope, success criteria, data, modeling and tools, infrastructure, deliverables, team, and time.

Setting clear success criteria is crucial for a POC, focusing on significant improvement over the current state in terms of quality, speed, cost, or other metrics.

Realistic target model accuracy for a POC should be set lower than the desired production system accuracy to assess feasibility quickly.

Data required for the POC includes specific datasets with annotations, such as 5,000 images of industrial products with defect labels.

POCs can utilize cloud AI services, such as Amazon Lookout for Vision, or custom model training using frameworks like PyTorch.

The deliverables from a POC should include the trained model, code, documentation, knowledge transfer, and recommendations for scaling to production.

It is important to assess whether AI is the right solution for the problem, as traditional techniques might sometimes be more efficient.

Evaluating the expected accuracy boost from POC to production helps decide if further investment is justified.

Identifying reasons for poor POC results—insufficient data, skills, or working on the wrong problem—is crucial before deciding on further rounds or moving to production.

Deciding between off-the-shelf models and custom-built systems for production depends on the flexibility and control needed versus technical skill and time required.

The total expected cost for a production system should be estimated based on POC insights, including data pipelines, software engineering, infrastructure, and ongoing maintenance.

Transcripts

AI projects by Design can have a lot of

uncertainty which makes it really

challenging to detm mind whether a

certain idea is worth pursuing or not

however there is a way to drisk AI ideas

before making significant Investments

doing a proof of concept or a PC for

short welcome to the fifth episode of

the AI for business course your ultimate

non-technical intro to the world of AI

and how to applies in the real world in

previous episodes we explored the

fundamentals of AI looked at over 80

different machine learning use cases

across various domains and learned how

to select our first AI projects today

we'll Kickstart the initial phase of

bringing these ideas to life through

proof of concept

experiments we'll be exploring many

aspects including why PCS are especially

critical for AI projects how to plan and

execute them and set the right success

criteria we'll explore different p see

plans for real life use cases and most

importantly how to use the outcome to

plan for production so with that said

let's Dive Right In so why do we need

poc's three main reasons testing

feedback and production

planning the first reason testing they

allow you to answer essential questions

quickly such as whether AI is the right

approach to solving the problem if you

have the required data and skills

whether the outcome is serving the

business and how much it will cost for

our production system feedback getting

early feedback from stakeholders who

will be using the AI solution is

essential this ensures that the solution

is addressing their needs and

requirements increasing their Buy in and

Adoption of the solution the third

reason is production planning PC's help

with production Planning by giving a

sense of the complexity time effort and

cost involved in building a full-fledged

production system this is crucial

information for any business looking to

implement AI as a real production

capability not just as a lab experiment

there are 10 elements to take into

consideration when planning an AI proof

of concept those are the problem

hypothesis scope success criteria data

modeling and tools infrastructure

deliverables team and finally

time let's explore each element by using

an example use case automating

industrial visual inspection in this use

case we'll be using deep learning to

automate the detection of defects in

products or parts instead of heavily

relying on manual inspection by subject

matter experts let's start with a

problem which should be a pain to solve

or a value to bring in this example the

problem is the manual inspection of

Industrial Products which is timec

consuming resource intensive and leads

to increased costs and reduced

efficiency the hypothesis is how you

think the POC will contribute to the

solution in this case AI can help

automate the inspection process process

by detecting defects in Industrial

Products leading to faster inspections

and reduced inspection costs the

hypothesis is that AI will enable the

team to conduct more inspections faster

and free of the time of workers

ultimately reducing the number of

inspectors

needed scope defines the nature of work

to be done it should be limited as much

as possible and really focused on

building the model not a full production

system examples of limited scope include

focusing on a certain subset of products

instead of every product that you

have making the POC scope on a certain

geography or a subset of problems

instead of the big problem you're trying

to solve and so on by narrowing down the

scope of a puc you can really limit the

amount of variables and complexity of

the overall project which would make it

easier to manage and

test in this example the PC will focus

on developing a computer vision deep

learning model to detect four types of

defects in images of IND products the

model will be trained on a data set of

5,000 adapated images and the evaluation

will be done on a test data set and new

images if you'd like to know more about

training or label data sets for AI

models I recommend you watch the first

episode of this course setting clear

success criteria is essential for a

successful PC the experiment needs to

show a significant improvement over the

current state of things whether this is

going to be measured in terms of quality

speed cost or something else success

criteria could be based on specific

business metrics such as for example

higher click-through rate for a use case

like personalized recommendations as

well as the model's accuracy which could

be expressed in different metrics such

as Precision recall intersection of a

union mean average Precision rmsse or

something else depending on the model

type and the specific problem you're

trying to solve it's important to set

realistic Target Model accuracy for the

proof of concept stage while high

accuracy is generally desirable for a

production system that's not really the

focus of this stage let's say for your

real production system you would love to

have like 90 95% model accuracy for a

proof of concept you can Target

something like 75% accuracy because it's

understood that by time with more data

and better modeling you're going to

reach that 95% right but for the sake of

the experiment that you'd like to run

real quick to assess the feasibility you

might want to set lower accuracy

objective in our visual inspection

example the success criteria include an

accuracy of 70% Precision an 80% recall

and the model's ability to inspect 100

images in under 1 minute The Fifth

Element is data this is basically the

data sets required to do the PC in this

example the data sets include 5,000

images of Industrial Products with

annotations indicating the presence or

absence of defects modeling and tools

Define the modeling technique and any

tools services or libraries expected to

be used in this example the PC will use

Amazon lookout for vision as the primary

Cloud AI service optimized especially

for defect detection however if the

accuracy is not satisfactory the PC may

also explore using Amazon recognition or

even custom model training using a deep

learning framework like pytorch the

infrastructure element defines the

hardware storage and compute required

for the PC in this example when using

AWS lookout for vision no dedicated

infrastructure is needed users can

leverage the service directly everything

from training to inference is managed

within the service itself however if

we're going to train a custom model

using a deep learning framework for

example like pytorch a powerful on

premise or a cloud virtual machine will

be required this machine should have

enough processing power memory and

storage to handle the size and

complexity of the data set and the

training process of the model for

example if you're training a deep

learning model you're going to need the

machine with a powerful GPU the

infrastructure will also need to provide

a way to deploy the model in case a

prototype application will need to

interface with it one example of a

virtual machine from Amazon is the

Amazon ec2 P3 2x large instance which is

optimized for high performance Computing

workloads such as machine learning this

instance type includes clud a single

Nvidia v00 GPU which provides

significant acceleration for compute

intensive training tasks in addition it

features 8 V CPUs 61 GB of memory and

1.5 tbte of NVM SSD storage we can

consider this or a similar on Perm

machine if we're going to adopt a custom

model development methodology versus a

cloud-hosted AI service like Amazon

recognition or lookout for vision the

deliverable section highlights the

specific outputs that you're going to

get from the puc beyond the model itself

so it's understood that you're going to

get a trained model ideally with a good

accuracy perfect but what exactly do you

need beyond that so that you can

continue to build on top of that PC and

build a production system ideally here

are some things that are typically

received after the PC is done one code

you need the code files for all the

stages of the PC whether it's sourcing

the data cleaning it processing it you

know any feature engineering work that

was done in case you're using classical

machine learning algorithms and so forth

so you need those source code files so

that your team or other you know

Consultants can eventually build on top

of that work two documentation you need

documentation for all the pieces in the

proof of concept how do they talk to

each other and stuff like that and you

need a summary of the outcome and how

that outcome was achieved

number three you probably need some sort

of knowledge transfer in many cases you

have some consultant or an AI company

helping you do a prototype or a proof of

concept and then you have your own data

science team right if you would like

your team to carry on the work you need

some sort of knowledge transfer from

that consultant or external company to

your team in case you don't have a team

that's not probably going to be required

if the same company is going to still

build a production system you might also

need a prototype application that

interfac with the model so here's the

thing once you are done with the PC you

would like to test that you know maybe

with new data see how it works there are

different ways that you can test that

pilot right whether you have a data

science team or someone who understands

python so you can just run scripts in a

notebook for example or in a python

environment right or if you're not able

to do that or you need an easier way to

interface and test the model you're

probably going to need some sort of a

web interface or a mobile application or

a desktop application some sort of

um a graphical user interface that you

know can integrate the model and you can

use it as a way to test it and you know

provide it with some data and get some

output for example so a prototype

application might be required and then

finally and the most important thing

here some sort of recommendations on

what to do to scale that pilot or

experiment to our production system this

is one of the most important things of

our proof of concept that you learn a

lot in that experiment and you know the

kind of recommendations on what to do

next are really critical to be provided

so to summarize you need code resources

documentation of the system and a

summary of the results some sort of

knowledge transfer if it's required a

prototype application if you require

some sort of a dedicated app to

interface with the model and test it and

finally some recommendations on what to

do next and things to consider if you

would like to scale that to our

production system in our example for the

industrial visual inspection use case

the primary deliverable is a computer

vision Model A convolution Neal Network

in that case that can accurately detect

defects in Industrial Products

deliverables will also include the

following a data pipeline that can

efficiently pre-process and transform

row images into a format suitable for

training the model the data pipeline can

also be used to update the model with

new data in the future a code repository

that contains the source code for the

data pipeline model training and

evaluation scripts and any other tools

developed during the PC a report

detailing the develop and evaluation of

the model including a description of the

data set used for training the model

architecture on hyperparameters and the

evaluation metrics the report should

also include a discussion of any

challenges or limitations encountered

during the PC and recommendations for

further

improvements the PCU will also result in

a prototype application in our case a

web- based interface for uploading

images and receiving defect detection

results or an integration with an

existing inspection system

the team should ideally include

resources covering the AI and data

engineering piece the domain knowledge

piece and the project management aspect

in our example use case the team

actually consists of a data scientist an

inspection engineer and a project

manager finally the PC should ideally

take somewhere between 3 to 8 weeks more

or less again the objective is to test

the idea quickly with the minimal

investment instead of like spending a

lot of time on it um in our case for the

visual inspection use case we're going

to take about one and a half month for

the proof of concept I highly recommend

you include these 10 elements when

planning for your proof of concept

experiments whether you're doing it

in-house or Outsourcing to a contractor

these 10 elements will help you

structure the planning of your proof of

concept for better familiarity with this

tool I have included four different

examples for POC plans one of which is a

generative AI use case in the video

description so check those out you're

going to find the detail proof of

concept plans for these different use

cases and use these as examples to

understand the methodology better and

apply it in your future experiments

throughout my work with more than 100

customer that I have helped a lot of

them pursue successful PC's I found that

there are some critical questions to

address after the puc stage I'm going to

go through these questions one question

at a time right now and provide some

Reflections on how you should think

about it the first question is is AI the

right solution to solve that that

problem or not AI is amazing it is so

powerful it can solve so many problems

but guess what sometimes it's just not

the right solution sometimes more

traditional techniques could be more

efficient in solving that problem versus

AI after trying machine learning

techniques in the proof of concept you

need to really assess the ROI is it

significant are we talking 2x 3x 10x

something like that again speed quality

time customer experience on any of these

metrics or the return isn't really that

significant

again sometimes more traditional

techniques could be more efficient and

if that's the case don't get to fixate

on AI and try to brainstorm and think

about other use cases where AI could

really bring significant return on

investment the second question is what

is the expected level of accuracy boost

we can get for the model for a real

production system versus the proof of

concept proof of concept act as good

reality checks for what we can

potentially achieve if we invest more

for a production system

let's say you're getting 70% accuracy

for your model maybe with more data and

better modeling techniques and more

iterations we can achieve you know like

85% or maybe 90% accuracy the question

is would that be enough for your for

your use case or not sometimes that's

enough sometimes it's not depending on

the use case for self-driving cars for

example you can't freely go with that

you have to hit like

99.999% or something for your perception

systems because it's really vital for

those cars to see what's on the road

right well but for other use cases let's

say automating visual Quality Inspection

uh for industrial use cases maybe for

your use case you can just go with 80%

accuracy right it can save you a lot of

time so depending on the levels of

accuracies you're getting for your model

for the PC you can start you know

estimating what level of boost you can

get for a production system and have the

decision whether this is going to be you

know enough or not and decide to invest

more in that experiment or not based on

that the third question is if you aren't

getting good results from your PC what

are the reasons for that and is it worth

doing a second round of a PC or moving

to a production system is it worth

allocating Dev investment or not and

what kind of changes we need to do to

ensure better outcomes if you're get if

you're not getting good results there

are usually three main reasons

insufficient data insufficient skills or

working on the wrong problem let's take

this one by one and have a closer look

at it insufficient data sometimes you

either need like different data sets or

more from the data that you have already

you know sometimes the data that you

have that you're working with simply

does not have enough predictive power

for you to use and find relationships

and stuff like that and make predictions

let's say for example you're trying to

forecast retail store demand or retail

demand in general and you're using only

historical sales data for that if you're

not getting good accuracy probably need

to add more data sets like for example

weather and promotions and historical

marketing campaigns and maybe economic

indicators and stuff like that now you

start adding these data sets you might

get better accuracy sometimes you you're

only working with one or two years of

data maybe you need like five or six

years of data historically to solve that

problem the question is in either case

Is it feasible to get these data sets

whether it's more from the data that you

have or different data sets if the

answer is yes that's usually a good sign

then maybe it's worth having another

round of the puc getting more data and

seeing if you can get more results the

second reason could be insufficient

skills let's say you're trying to solve

a computer vision problem you know the

very example we have at hand automating

visual Quality Inspection and you're not

getting good results maybe it's not

about the data Maybe it's about the team

skills maybe you have a team inhouse who

have great experience in you know

Predictive Analytics dealing with

tabular data to make predictions about

you know demand or predict analytics for

other use cases but they haven't worked

on deep learning computer vision

problems before and they're still

learning and they might not be familiar

with the latest architectures the latest

tips and tricks to optimize the

hyperparameters uh the latest tips and

tricks to achieve better accuracies and

stuff like that so maybe that's the

reason in that case it's not a ship

stopper there are other routes that you

can proceed with you can hire a

consultant who have been working on

these problems before you can consider

partnering with uh a technology company

who have been solving these problems

before right uh and see but if you don't

have access to these and at the same

time you don't have a meaningful way to

upgrade the skills of your team you need

to make a decision here because it's not

expected that you're going to get

amazing results if you're not able to

improve either of those the third

possible reason for not getting great

results is as we mentioned in the

previous points maybe you're just

working on the wrong problem maybe

you're trying to solve a problem that

could be really solved with traditional

Noni techniques and we trying to force

AI to solve it right and I find this

happening a lot of times by the way some

customers that have been working with

they are really excited about AI they

want to they have some mandate to use it

to solve problems so they Rush uh into

using it to solve specific problems that

honestly like other tools could could

solve it more efficiently so if that's

the case then again you need to consider

those other tools and move on to other

use cases the fourth question is a

really important one are you going to be

using off-the-shelf models or services

or tools to build the production system

after the PC or you're going to need to

build it from scratch that's a very

important question the PC will give you

a very good chance to explore the

different tools and services in the

market you know the different Cloud AI

Services the pre-train models the open

source tools and stuff like that and see

if you can leverage any of these when

building your real production system if

you have tried a lot of these tools and

you see that it's not providing you know

decent results maybe you need to

consider building a system fromt scratch

that has its own advantages right you

have a lot of control into crafting

exactly the workflows that you want um

you can control so many different things

but at the same times it comes with a

cost it requires probably high technical

skills and it will probably take longer

time right on the other side leveraging

offthe shelf tools Solutions and Cloud

AI Services can save you a lot of time

you know we can proceed with using it

while having less AI experience inhouse

but it might not give you the

flexibility

of building things from scratch there is

no right or wrong here you have to weigh

the pros and cons of each but you need

to understand based on the effort that

you have put in the proof of concept you

know what are the available Solutions

out there that you can leverage and if

they are enough or not answering this

question will help you identify the

complexity of a production system and

the expected time and budget for it

which are very important to assess

before committing a specific budget or

an investment or team effort to solve

that problem question number five what

is the total expected cost for a

production system in the light of what

we have seen in the proof of concept

experiment here's the thing for the

proof of concept usually we just focus

on the science aspects of the problem we

focus on building a good model with good

accuracy to test the hypothesis uh of

our problem right for a production

system you're going to need to scale

this across the Enterprise and for that

to happen you're going to need to move

from the limited scope from the PC to a

fully fledged scope you're going to need

for example to take care of different

items each comes with that owns cost for

example the ETL or data pipelines the

how are you going to get input data from

different systems integrate it process

it clean it and pass it to the model and

then you know how are you going to get

data or predictions out of the model and

feed it to different information

products in order to do that you're

going to need software engineering to

integrate the model with different

systems in your company you're going to

need this flow to happen in an automated

fashion right in in a PC usually you run

the model in a jupyter notebook for

example or run a python script in a real

production system that's not going to

work you're going to need this flow to

be automated right so there is a

software engineering cost infrastructure

cost ETL cost uh there is ongoing

maintenance and support sometimes the

model results change by time like they

degradate for example and different

types of drifts could happen so there's

going to need be a need to potentially

continuously retrain the model redeploy

to production stuff like that so you're

just going to need to take care of these

different items assess the cost

associated with each and build a

business case for the kind of return

that you're expecting from a production

system with more data and more

iterations and the kind of cost you're

looking at for this production system

how is the ROI looking like is it

justifiable and if yes then definitely

proceed if not then maybe you reconsider

cutting some parts of that production

system pushing some parts to a later

stage and focusing on the core aspects

Instead at the end of the day the proof

of concept should should give you a good

idea of how much cost potentially our

production system would look like and

that will help you in taking uh the

decision and building the business case

to justify these Investments these were

just some examples for how the puc can

help you ask the right questions get the

needed answers to plan for the next

stage building a production system I

hope that today's episode was useful for

you I hope that you have learned how to

take your idea to action and start

planning for that experiment and what

are the main elements to take into

consideration when doing these

experiments and for the next episode

we're going to be looking into a very

interesting subject building AI capacity

we're going to talk about different

things we're going to talk about

building AI teams what kind of skills

and resources and talent you need to

have based on your stage based on what

you're trying to do there is no you know

black and white here there is no one

right answer there are different flavors

of this uh based on what you're trying

to do and we're also going to be looking

at the option of Outsourcing and compare

these two different uh options together

building AI teams versus Outsourcing

each one has its own pros and cons we're

going to be exploring that and we're

going to be looking at so many different

scenarios for what kind of AI team you

might need to have in house and some

tips and tricks on how to hire the best

AI talent and retain them if you like

today's episode I really appreciate if

you share it with others who might

benefit from it and you know like the

video on social media and on YouTube and

with that said see you next episode and

thank you for your time today

[Music]