#1 Machine Learning Engineering for Production (MLOps) Specialization [Course 1, Week 1, Lesson 1]

Summary

This video introduces the practical challenges of deploying machine learning models in production environments. Andrew Ng discusses the importance of moving beyond just training models to understanding the full lifecycle of machine learning projects. He explains the significance of production deployments, using real-world examples like automated visual defect inspection in manufacturing. Ng highlights common challenges like data drift and the 'proof of concept to production gap,' emphasizing that deploying machine learning systems requires much more than just model training. The course focuses on practical skills for building production-ready systems that create real value.

Takeaways

• 🚀 Deploying machine learning models into production maximizes their value and is essential for many roles in machine learning.
• 👨‍🏫 The course is led by Andrew Ng and Robert Crowe, providing practical, hands-on skills for deploying machine learning models.
• 📱 Example: A computer vision model inspecting smartphones for defects shows how machine learning models are applied in real-world manufacturing.
• 🏭 Deployment of machine learning in factories often involves edge devices to avoid downtime when internet access is unavailable.
• 🌩️ Prediction servers can be deployed in the cloud or at the edge, depending on the application needs and infrastructure limitations.
• ⚠️ Real-world challenges like concept drift or data drift, where the production environment differs from training data, can impact model performance.
• 💻 Machine learning engineers need to handle various problems, including data shifts, to ensure models perform well in production.
• 🔧 Moving from a proof of concept to production involves more than just the machine learning code—much of the work is in other system components.
• 📊 Beyond model training, data collection, verification, feature extraction, and system monitoring are critical for a full production deployment.
• 📆 A systematic approach to managing the life cycle of machine learning projects is crucial for successful deployment in real-world environments.

Q & A

What is the main focus of the course 'Machine Learning Engineering for Production'?

The course focuses on teaching learners how to not only train machine learning models but also deploy them in production environments. It covers practical, hands-on skills and techniques for managing the full life cycle of machine learning projects.

Why is it important to deploy machine learning models in production?

Deploying machine learning models in production is crucial for creating real-world value. While training models is important, the full potential of machine learning is realized when these models are put into production and can be used to make automated decisions.

What is an edge device in the context of machine learning deployment?

An edge device is a device that operates at the site where data is generated, such as a factory in the example of smartphone manufacturing. It can run machine learning models to make real-time decisions, even if the internet connection to the cloud is unavailable.

What is the role of the prediction server in a machine learning production environment?

The prediction server receives API calls with input data (like images of smartphones) and processes the data using a machine learning model to return a decision, such as whether a phone is defective or not. This enables real-time decision-making in production.

What is concept drift or data drift in machine learning deployment?

Concept drift or data drift refers to changes in the data distribution over time, such as differences in lighting conditions that affect the quality of images in production. These changes can cause machine learning models to perform poorly if not addressed.

Why can it take months to go from proof of concept (PoC) to production deployment of a machine learning model?

Moving from PoC to production can take months because there is a significant amount of work involved beyond just training the model. This includes setting up data pipelines, API interfaces, and monitoring systems, among other components, to ensure a reliable production environment.

What is the proof of concept (PoC) to production gap in machine learning?

The PoC to production gap refers to the difference between having a working machine learning model in a development environment (like a Jupyter notebook) and deploying it in a real-world production system. This gap exists due to the complexity of integrating the model into a larger system.

How much of the overall code in a machine learning project is typically dedicated to the machine learning model itself?

In many machine learning projects, only 5-10% (or even less) of the overall code is dedicated to the machine learning model itself. The majority of the code focuses on other aspects of the system, such as data management, monitoring, and deployment.

What are some components beyond the machine learning model that are necessary for a successful deployment?

Components beyond the machine learning model include data collection, data verification, feature extraction, monitoring, and analysis tools to ensure the model works effectively in production and can adapt to real-world changes.

Why is it important to consider the full life cycle of a machine learning project?

Considering the full life cycle of a machine learning project is essential because the model's performance in production depends on more than just training. It includes deployment, monitoring, and adapting to changes over time, ensuring long-term success and value.

Outlines

00:00

👋 Introduction to Machine Learning Engineering for Production

In this opening, Andrew Ng introduces the specialization 'Machine Learning Engineering for Production.' He highlights how learning to deploy machine learning models into production is crucial for creating real-world value. This skill set is essential not only for maximizing the usefulness of models but also for passing job interviews, as many interviewers ask about deployment experience. Andrew also introduces Robert Crowe, a Google expert who will teach parts of the specialization. By the end of the course, learners will understand the full lifecycle of a machine learning project, from training to deployment.

05:00

📱 Example: Defect Detection in Smartphones Using Computer Vision

Andrew presents a practical example where computer vision is used in a factory to detect defects in smartphones, such as scratches. He explains the concept of edge devices and their role in processing images locally within a factory for quality control purposes. The example demonstrates the need for deploying machine learning models in real-world environments, where images are captured, analyzed, and decisions are made using a prediction server. Andrew emphasizes that building the model is only one part of the process—deploying it with API interfaces and other software is equally critical to ensuring its operational success.

⚙️ Practical Deployment Challenges: Concept Drift and Real-World Issues

Andrew discusses the challenges of deploying machine learning models in production, using concept drift as an example. In a factory setting, changes in lighting or other factors can cause the model’s performance to degrade, leading to issues like data drift. These challenges are often unexpected and require machine learning engineers to adapt their models to new conditions. Andrew underscores the importance of not only developing a good model in a controlled environment, such as a Jupyter notebook, but also ensuring that the model performs well in unpredictable production settings. This problem-solving mindset is key to making machine learning systems valuable in practice.

🛠️ Beyond Machine Learning Code: The Complexity of Deployment

This paragraph addresses the complexity of deploying machine learning systems in production. Andrew points out that the machine learning code itself often represents only a small fraction—about 5 to 10 percent—of the entire system. The rest includes components like data management, monitoring, and feature extraction, which are essential for running a successful production deployment. Andrew introduces a diagram adapted from a research paper to illustrate the various components beyond the model. He notes that transitioning from a proof-of-concept model to full production often involves significant additional work, which is why this specialization will teach learners how to manage the entire lifecycle of a machine learning project effectively.

Keywords

💡Production Deployment

Production deployment refers to the process of taking a machine learning model from the development phase and implementing it in a real-world environment where it can generate value. In the video, it's emphasized that building a machine learning model is just the first step, and putting it into production is critical for practical use, like defect detection in manufacturing.

💡Edge Device

An edge device is hardware that operates at the 'edge' of a network, such as a device within a factory that performs localized computing without relying on cloud services. The video explains that in manufacturing, edge devices are often used for tasks like inspecting smartphones for defects, as they can continue working even if the internet connection goes down.

💡Prediction Server

A prediction server is a system that receives input data, runs it through a trained machine learning model, and returns predictions. In the video, the prediction server is a key component in the defect detection system, receiving images of phones and deciding whether they are defective. It can operate in the cloud or at the edge.

💡API

API (Application Programming Interface) is a set of protocols for building and interacting with software applications. In this context, the video describes how an API is used to send images of phones from an edge device to a prediction server for defect detection, showing how APIs facilitate communication between components of a machine learning system.

💡Data Drift

Data drift, also known as concept drift, occurs when the data in a production environment differs from the data used during the training of the model. The video highlights that this is a common challenge in machine learning deployment, such as when lighting conditions change in a factory, which can affect the model's performance in detecting defects.

💡Proof of Concept (POC)

A proof of concept (POC) is an early model of a machine learning project that demonstrates the feasibility of the approach. The video describes how even when a POC works in a controlled environment (like Jupyter Notebook), much more effort is required to translate it into a reliable production system.

💡Automated Visual Defect Inspection

Automated visual defect inspection is the process of using computer vision algorithms to detect defects in products, such as smartphones on a manufacturing line. The video provides an example of this system, where images are analyzed to identify scratches or cracks, illustrating a practical application of machine learning.

💡Model Lifecycle

The model lifecycle refers to the entire process of developing, training, deploying, and maintaining a machine learning model. In the video, Andrew Ng stresses the importance of understanding the full lifecycle, from model training to production deployment, and managing changes in data or production environments.

💡Data Verification

Data verification is the process of ensuring that the data used in machine learning models is accurate and reliable. The video mentions data verification as part of the broader set of tasks required for a production deployment, such as verifying data integrity before sending it to a model for predictions.

💡Feature Extraction

Feature extraction involves selecting and transforming raw data into a set of features that can be used by a machine learning model. The video references feature extraction as one of the essential components that need to be built and managed in addition to the machine learning code when deploying models in production.

Highlights

Training a machine learning model is only the beginning; putting it into production is key to maximizing value.

Many interviewers will ask if you have experience deploying machine learning algorithms.

The specialization teaches hands-on skills to deploy machine learning models, including managing the entire life cycle.

An example of machine learning in production: inspecting smartphones on a manufacturing line for defects using computer vision.

Edge devices in factories can use inspection software to detect scratches on phones, making real-time decisions.

Edge deployments are common in manufacturing to ensure operations continue even if internet access is lost.

Concept drift or data drift occurs when the data collected during training differs from real-world deployment data.

Machine learning engineers should take responsibility for adjusting models to real-world data conditions.

Success in a development environment, like a Jupyter notebook, is only the beginning; practical deployment requires more work.

A significant challenge in production deployment is dealing with data distribution changes, requiring engineers to adapt models.

Only 5-10% of the code in a machine learning system is related to the model; the majority supports other functions like data collection and monitoring.

A gap exists between proof-of-concept models and real-world production deployment.

Transcripts

hi and welcome to machine learning

engineering for production a lot of

learners have asked me hey andrew i've

learned to train a machine learning

model now what do i do

machine learning models are great but

unless you know how to put them into

production it's hard to get them to

create the maximum amount of possible

value

or for those of you that may be looking

for position in machine learning many

interviewers will ask have you ever

deployed a machine learning algorithm

in this full course specialization the

first course taught by me the second

third and fourth courses taught by

robert crowe who's an expert at this

from google

we hope to share with you the practical

hands-on skills and techniques you need

to not just build a machine learning

model but also to put them into

production and so by the end of this

first course and by the end of this

specialization i hope you have a good

sense of the entire life cycle of

machine learning project from training

model to put into production and really

how to manage the entire machine

learning project let's jump in

let's start with an example

let's say you're using computer vision

to inspect phones coming off the

manufacturing line to see if there are

defects on them so this film shown on

the left doesn't have any scratches on

it but if there was a scratch or crack

or something

a computer vision algorithm

would hopefully be able to find

this type of scratch or defect

and maybe put the bounding box around it

as part of quality control

if you get a data set of scratch phones

you can train a computer vision

algorithm maybe in your network to

detect these types of defects but what

do you now need to do in order to put

this into production deployment

this would be an example of how you

could deploy a system like this you

might have an edge device by edge device

i mean a device that is living inside

the factory that is manufacturing these

smartphones and that edge device would

have a piece of inspection software

whose job it is to take a picture of the

phone see if there's a scratch and then

make a decision on whether this phone is

acceptable or not this is actually

commonly done in factories this is

called automated

visual defect inspection what the

inspection software does is it will

control camera that will take a picture

of the smartphone as it rolls off the

manufacturing line

and it then has to make an api call to

pass this picture

to a prediction server

and the job of the prediction server is

to accept these api calls you know

receive an image make a decision as to

whether or not this phone is effective

and return this prediction and then the

inspection software can make the

appropriate control decision whether to

let the stone move on in the

manufacturing line or whether to shove

it to a side because you know was

defective and not acceptable

so after you have trained a learning

algorithm maybe trained in your network

to take as input x pictures of phones

and map them to why

predictions about whether the phone is

defective or not

you still have to take this

machine learning model

puts it in a prediction server set up

api interfaces and really write all of

the rest of the software in order to

deploy this learning algorithm into

production

this prediction server

is sometimes in the cloud

and sometimes the prediction server is

actually at the edge as well in fact in

manufacturing we use edge deployments a

lot because you can't have your factory

go down every time your internet access

goes down

but

cloud deployments with prediction server

is a server in the cloud is also used

for many applications

let's say you write all the software

what could possibly go wrong

it turns out that just because you've

trained a learning algorithm that does

well on your test set which is to be

celebrated it's great when you do well

when you hold a test set

unfortunately reaching that milestone

doesn't mean you're done there can still

be quite a lot of work and challenges

ahead

to get a valuable production deployment

running

for example

let's say your training set has images

that look like this

there's a

good phone on the left the one in the

middle has a big scratch across it and

you've trained your learning algorithm

to recognize that phones like this on

the left are okay

meaning that no defects and

maybe draw bounding boxes around

scratches or other defects it finds in

phones

when you deploy it in the factory you

may find that

the real-life production deployment

gives you back images like this

much darker ones because the lighting

factory

because the lighting conditions in the

factory have changed for some reason

compared to the time when the training

set was collected

this problem is sometimes called concept

drift or data drift you learn more about

these terms later in this week

but this is one example of the many

practical problems that we as machine

learning engineers should step up to

solve

if we want to make sure that we don't

just do well on the hold out test set

but that our systems actually create

value

in a practical

production deployment environment i've

worked on quite a few projects where my

machine learning team and i would

successfully know a proof of concept and

by that i mean we train a model in

jupiter notebook and it will work great

and we will celebrate that you know you

should celebrate it when you have a

learning algorithm work well in a

jupiter notebook or in a development

environment

but it turns out that sometimes i'll see

many projects where that success which

is a great success

to the practical deployment is still

maybe another six months of work

and this is just one of many of the

practical things that a machine learning

team has to

watch out for and handle in order to

actually deploy these systems

some machine learning engineers will say

is not a machine learning problem to

access these problems you know the data

set changes some machine engineers think

well is that a machine learning problem

my point of view is that our job is to

make these things work um and so if the

data set has changed is i think of it as

my responsibility when i work on a

project to step in and do what i can to

adjust the data distribution as it is

rather than as i wish it is

so this specialization will teach you

about a lot of these important practical

things for building machine learning

systems that work not just in the lab

not just in the jupyter notebook but in

a production deployment environment

a second challenge of deploying machine

learning models in production is that it

takes a lot more than machine learning

code

over the last decade there's been a lot

of attention on machine learning models

so your neural network or

other algorithm

that learns a function

mapping from some input to some output

and there's been

amazing progress in machine learning

models

but it turns out that if you look at a

machine learning system in production

if this little orange rectangle

represents the machine learning code the

machine learning model code

then this is all the code you need for

the entire machine learning project

i feel like for many machine learning

projects

maybe only five to ten percent maybe

even less of the code is machine

learning code and and i think this is

one of the reasons why when you have a

proof of concept model working

maybe in jupiter notebook

it can still be a lot of work to go from

that initial proof of concept to the

production deployment so sometimes

people refer to the poc

or the proof of concept to production

gap

and a lot of that gap is sometimes just

the sheer amount of work it is to also

write all of this code out here

beyond the

initial

machine learning model code so what is

all this other stuff this is a diagram

that i've adapted from a paper by d

scully and others beyond the machine

learning code there are also many

components especially components for

managing the data such as data

collection data verification feature

extraction

and after you are serving it how to

monitor the system well monitor the data

comes back help you analyze it but there

are often many other components that

need to be built

to enable a working production

deployment so in this course you learn

what are all of these other pieces of

software needed for a valuable

production deployment

but rather than looking at all of these

complex pieces one of the most useful

frameworks i found for organizing the

workflow of a machine learning project

is to systematically plan out the life

cycle of a machine learning project

let's go to the next video to dive in to

what is the full life cycle of a machine

learning project and i hope this

framework will be very useful for all of

your machine learning projects that you

plan to deploy in the future let's go to

the next video