Getting Started with the What-if Tool | Introducing the What-If Tool

Summary

The video script introduces the What-If Tool, a visual interface for understanding machine learning models and datasets. It's compatible with various platforms and supports TensorFlow models, with extensions for others. The tool features three tabs: data point editor, performance and fairness, and features analysis. It allows users to manipulate data, visualize model predictions, and assess model fairness and performance. The script also demonstrates using the tool with Cloud AI Platform and comparing different models.

Takeaways

• 🛠️ The What-If Tool is a visual interface designed to help users understand their datasets and the output of their machine learning models.
• 📈 It can be used with minimal code on various platforms, including Jupyter Notebooks, Kollab, TensorBoard, and Cloud AI Platform Notebooks.
• 🔍 Analysis with the What-If Tool is beneficial during training data collection, model creation, and post-training evaluation.
• 🌐 The tool supports TensorFlow models natively and can be adapted to work with models from other frameworks with a few lines of code.
• 🏠 An example given in the script involves a TensorFlow model trained on the Iowa Housing dataset, predicting house prices as above or below $160,000.
• 📊 The What-If Tool features three tabs: Data Point Editor, Performance and Fairness, and Features, each offering different insights into the model's performance and data.
• 🔧 The Data Point Editor allows users to inspect and modify individual data points, create custom visualizations, and see model predictions.
• 📉 The Performance and Fairness tab provides common evaluation metrics and tools to slice data and optimize for fairness.
• 📋 The Features tab displays the balance of the dataset and the range of values for each feature, which can be loaded even before model training.
• ☁️ The tool can be used with models deployed on Cloud AI Platform, utilizing the Wit Config Builder for customization.
• 📝 The script demonstrates how to compare the performance of different models using the What-If Tool by adding a line in the Wit Config Builder.
• 📚 The What-If Tool is versatile, supporting both classification and regression models, and can be used for analyzing image and text models as well.

Q & A

What is the purpose of the What-If Tool?

The What-If Tool is a visual interface designed to help users understand their datasets and the output of their machine learning models.

Who are James and Sara in the context of the video?

James is an engineer on the What-If Tool, and Sara is a developer advocate on Cloud AI Platform.

On which platforms can the What-If Tool be run?

The What-If Tool can be run on various platforms including Jupyter Notebooks, Kollab, TensorBoard, and Cloud AI Platform Notebooks.

What are the different phases of machine learning where the What-If Tool can be helpful?

The What-If Tool can be helpful during training data collection, model creation, and post-training evaluation.

What type of model is used in the example provided in the script?

In the example, a TensorFlow binary classification model is used, which predicts whether a house is worth more or less than $160,000.

How can the What-If Tool be instantiated in an example?

The What-If Tool can be instantiated with a few lines of code by passing it a set of test data points along with the ground truth labels for those examples.

What are the three tabs with different features in the What-If Tool?

The three tabs are the data point editor, Performance and Fairness, and Features.

What can be done in the data point editor tab of the What-If Tool?

In the data point editor, users can see the model's prediction for each data point, inspect individual data points, change feature values, create custom visualizations, and more.

What is displayed in the Performance and Fairness tab of the What-If Tool?

The Performance and Fairness tab shows common evaluation metrics for the model and allows users to slice data by different features and apply strategies to optimize for fairness.

What information can be found in the Features tab of the What-If Tool?

The Features tab shows how balanced the dataset is, displaying the range of values for each feature sent to the What-If Tool.

How can the What-If Tool be used to compare models?

The What-If Tool can compare models by adding a line to the Wit Config Builder to specify which models to compare, allowing users to evaluate their performance using various capabilities of the tool.

What additional capabilities of the What-If Tool are mentioned for future videos?

Future videos will explore more about what can be done in the data point editor and Performance and Fairness tabs of the What-If Tool.

Are there any limitations on the types of models the What-If Tool supports?

The What-If Tool supports TensorFlow models out of the box and can support models built with any other framework with just a few lines of code. It can be used for both classification and regression models.

How can users find more demos of the What-If Tool?

Users can find more demos of the What-If Tool in the links provided below the video, which show how it works with image and text models.

Outlines

00:00

🔍 Introduction to the What-If Tool

James and Sara introduce themselves and the What-If Tool, a visual interface designed to enhance understanding of machine learning models and datasets. They discuss its utility across various platforms like Jupyter Notebooks, Kollab, TensorBoard, and Cloud AI Platform Notebooks. The tool is beneficial during different stages of the machine learning process, including data collection, model creation, and post-training evaluation. Sara also mentions the tool's compatibility with TensorFlow models and its extendibility to other frameworks with minimal additional code.

📊 Exploring the What-If Tool's Features

The presenters delve into the What-If Tool's functionality, starting with an example using a TensorFlow model trained on the Iowa Housing dataset. They demonstrate how to instantiate the tool with test data points and ground truth labels. The tool's interface is explored through its three tabs: the data point editor for model prediction inspection and feature value adjustments, the Performance and Fairness tab for evaluation metrics and fairness optimization strategies, and the Features tab for dataset balance analysis. The tool's versatility for both classification and regression models is highlighted.

🌐 Using the What-If Tool with Cloud AI Platform

Sara showcases how to utilize the What-If Tool with models deployed on Cloud AI Platform. She walks through opening an AI Platform Notebook instance to analyze an XGBoost model, using the Wit Config Builder with the set AI Platform model method, which requires the Google Cloud project ID, model name, and version. The Wit Config Builder is highlighted for its customization capabilities for different use cases.

📊 Comparing Models with the What-If Tool

James and Sara discuss the ability to compare models using the What-If Tool, especially when they are trained on the same dataset. Sara explains the process of adding a configuration line to compare the performance of different models within the tool. They emphasize the tool's applicability to various model types, including classification and regression models.

🎥 Conclusion and Further Exploration

In conclusion, James and Sara recap the introduction to the What-If Tool across different platforms and model types. They encourage viewers to stay tuned for more detailed videos on the tool's capabilities in the data point editor and Performance and Fairness tabs. Sara also directs viewers to additional demos linked below to see the tool in action with image and text models, wrapping up the session with thanks for watching.

Keywords

💡What-If Tool

The What-If Tool is a visual interface designed to help users understand their datasets and the output of their machine learning models. It is central to the video's theme as it is the main subject being introduced and discussed. In the script, it is mentioned as a tool that can be run with minimal code from various platforms and is useful during different stages of the machine learning process, including training data collection, model creation, and post-training evaluation.

💡Engineer

In the context of the video, James is introduced as an engineer on the What-If Tool, indicating his role in developing or working with the technical aspects of the tool. The term 'engineer' is used to establish his expertise and to lend credibility to the information he provides about the What-If Tool throughout the video.

💡Developer Advocate

Sara is introduced as a developer advocate on Cloud AI, Platform. This term refers to a role that typically involves promoting and supporting the use of a particular technology or platform. In the video, Sara's role as a developer advocate is relevant as she demonstrates and advocates for the use of the What-If Tool on various platforms.

💡Machine Learning Models

Machine learning models are algorithms that improve their performance on tasks without being explicitly programmed. In the video, the What-If Tool is presented as a way to better understand these models by analyzing their predictions and outputs. The script mentions using the tool during the creation and evaluation of machine learning models, emphasizing its importance in the machine learning workflow.

💡Jupyter Notebooks

Jupyter Notebooks are web-based interactive computational notebooks that allow users to create and share documents containing live code, equations, visualizations, and narrative text. In the script, Jupyter Notebooks are listed as one of the platforms where the What-If Tool can be run, indicating its versatility and ease of use across different development environments.

💡Kollab

Kollab, mentioned in the script, is likely a reference to a collaborative platform or tool, although it's not a widely recognized term in the context of the video. It could be a specific platform or a typo for another platform. The script suggests that the What-If Tool can be used with minimal code on Kollab, indicating its compatibility with various environments for machine learning model analysis.

💡TensorBoard

TensorBoard is a visualization tool integrated with TensorFlow that allows users to visualize and analyze the TensorFlow computation graph, as well as runtime metrics of the model during training and inference. In the video, TensorBoard is listed among the platforms that support the What-If Tool, showcasing its integration with popular machine learning tools.

💡Cloud AI Platform Notebooks

Cloud AI Platform Notebooks is a managed service that provides pre-configured JupyterLab environments for machine learning development. In the script, it is mentioned as one of the platforms where the What-If Tool can be utilized, highlighting the tool's cloud compatibility and its ability to support users working in cloud-based environments.

💡Binary Classification

Binary classification is a type of machine learning problem where the goal is to classify instances into one of two classes. In the script, an example is given of a TensorFlow model trained on the Iowa Housing dataset, which is a binary classification model predicting whether a house is worth more or less than $160,000. This example is used to illustrate how the What-If Tool can be applied to understand and analyze specific types of machine learning models.

💡Data Point Editor

The Data Point Editor is one of the three tabs in the What-If Tool, which allows users to inspect individual data points, change feature values, and create custom visualizations. In the script, it is described as the initial view users are presented with when using the tool, emphasizing its role in the hands-on analysis of data points and model predictions.

💡Performance and Fairness

The Performance and Fairness tab is another feature of the What-If Tool mentioned in the script. It displays common evaluation metrics for the model and allows users to slice data by different features and apply strategies to optimize for fairness. This tab is highlighted as a way to assess the model's performance and ensure fairness in its predictions, which is crucial for responsible machine learning model development.

💡Features Tab

The Features tab in the What-If Tool provides insights into the balance of the dataset by showing the range of values for each feature. It can be used even before a model is trained, as mentioned in the script, to understand the distribution of data. This tab is essential for evaluating the dataset's characteristics and ensuring that the model is trained on a representative and balanced set of features.

💡Wit Config Builder

The Wit Config Builder is a component mentioned in the script that allows users to create an instance of the What-If Tool and customize it for their specific use case. It is used to configure the tool for analyzing models, particularly when working with models deployed on Cloud AI Platform, as demonstrated in the video.

💡XGBoost Model

XGBoost is an optimized distributed gradient boosting library designed to be highly efficient with速度快, flexible, and portable. In the script, Sara demonstrates using the What-If Tool to analyze an XGBoost model deployed on AI Platform, showing the tool's capability to work with different types of models beyond just TensorFlow.

💡Regression Models

Regression models are a type of predictive modeling technique that estimates the relationship between variables. The script mentions that the What-If Tool can be used for both classification models and regression models, indicating its versatility in supporting different types of machine learning models for analysis and understanding.

Highlights

James Wexler and Sara Robinson introduce the What-If Tool, a visual interface for understanding machine learning models and datasets.

The tool can be used on various platforms including Jupyter Notebooks, Kollab, TensorBoard, and Cloud AI Platform Notebooks.

Analysis with the What-If Tool is beneficial during training data collection, model creation, and post-training evaluation.

Sara Robinson demonstrates the What-If Tool's support for TensorFlow models and its adaptability to other frameworks with minimal code.

The What-If Tool can be utilized with models deployed on Cloud AI Platform.

An example showcases the tool's use with a TensorFlow model trained on the Iowa Housing dataset from Kaggle.

The What-If Tool's instantiation requires only a few lines of code and a set of test data points with their ground truth labels.

The Data Point Editor tab allows for model prediction inspection, individual data point analysis, and feature value changes.

Custom visualizations can be created in the Data Point Editor to better understand model predictions.

The Performance and Fairness tab displays common evaluation metrics and allows data slicing by different features for fairness optimization.

The Features tab provides insights into the balance of the dataset and the range of values for each feature.

The What-If Tool can be used for both classification and regression models.

Sara Robinson shows how to use the What-If Tool for models deployed on Cloud AI Platform using an AI Platform Notebook instance.

The Wit Config Builder is used to customize the What-If Tool for specific use cases, including setting AI Platform model methods.

Models trained on the same dataset can be compared using the What-If Tool to analyze their performance.

The What-If Tool offers capabilities to compare models and evaluate their performance and fairness.

Upcoming videos will explore the What-If Tool's features in the data point editor and Performance and Fairness tabs.

Demos of the What-If Tool with image and text models are available in the provided links.

Transcripts

JAMES WEXLER: I'm James, an engineer on the What-If Tool.

SARA ROBINSON: I'm Sara, a developer advocate on Cloud AI

Platform.

JAMES WEXLER: We're going to give you

an introduction into how you can use the What-If Tool to better

understand your machine learning models.

SARA ROBINSON: We'll also show you

how to run it on various platforms and model types.

[MUSIC PLAYING]

So James, can you tell everyone what the What-If Tool is

and when they should use it?

JAMES WEXLER: The What-If Tool is a visual interface designed

to help you understand your data sets

and the output of your machine learning models.

You can run with minimal code from many different platforms,

including Jupyter Notebooks, Kollab, TensorBoard, and Cloud

AI Platform Notebooks.

Analysis using the What-If Tool can

be helpful during training data collection, model creation,

and post-training evaluation.

Sara, you've tried the What-If Tool on a bunch platforms.

What are the ways you can use it?

SARA ROBINSON: The What-If Tool supports TensorFlow models out

of the box and can support models

built with any other framework with just a few lines of code.

You can also use the What-If Tool

with your models deployed on Cloud AI Platform.

JAMES WEXLER: Let's start with an example in Kollab.

Here we've trained a TensorFlow model on this Iowa Housing

data set from Kaggle.

It's a binary classification model

that predicts whether a house is worth more

or less than $160,000.

We can instantiate the What-If Tool with a few lines of code

by passing it a set of test data points

along with the ground truth labels for those examples.

The What-If Tool has three tabs with different features.

And the initial view we're dropped into

is the data point editor.

Here we can see our model's prediction for each data point

we've sent to the What-If Tool.

The blue points indicate houses our model

classified as above $160,000.

In the data point editor, we can also

inspect individual data points, change feature values,

create custom visualizations, and much more.

SARA ROBINSON: The next tab, Performance and Fairness,

shows a couple of common evaluation

metrics for our model.

It also lets us slice data by different features

and apply various strategies to optimize for fairness.

The final tab, Features, shows us

how balanced our data set is.

For each feature, we can see the range of values for the data

we've sent to the What-If Tool.

You can load the Features tab with only a data set even

before you've trained a model.

JAMES WEXLER: That's awesome.

Do you want to show how you can use the What-If Tool for models

deployed on Cloud AI Platform?

SARA ROBINSON: Yes.

Here are a few models I've deployed on AI Platform.

I'll open up an AI Platform Notebook instance here

to analyze this XGBoost model.

Then I create an instance of the Wit Config Builder, this time

using the set AI Platform model method,

passing in my Google Cloud project ID, model name,

and model version.

The Wit Config Builder contains more methods

to customize the What-If Tool for your use case.

JAMES WEXLER: I noticed two of your models

are trained on the same data set.

Can you show us how we can use the What-If

Tool to compare them?

SARA ROBINSON: Sure.

All I need to do is add a line to my Wit Config Builder

to tell it which model I want to compare.

Now you can compare the performance

of both models using various capabilities in the What-If

Tool.

I should also mention that you can

use the What-If Tool for both classification models

and regression models.

JAMES WEXLER: So to recap, we've introduced you

to using the What-If Tool on various platforms and model

types.

Stay tuned for more videos to learn

what you can do in the data point editor

and Performance and Fairness tabs of the tool.

SARA ROBINSON: You can also check out more What-If Tool

demos in the links below to see how it works

with image and text models.

Thanks for watching.

[MUSIC PLAYING]