Building an Object Detection App with Tensorflow.JS and React.JS in 15 Minutes | COCO SSD

Summary

This video tutorial guides viewers through building a real-time object detection app using React and TensorFlow.js. The host introduces a pre-built template on GitHub to expedite the setup, then demonstrates how to integrate the COCO SSD model from TensorFlow.js for object detection. The tutorial covers setting up the React app, capturing webcam images, processing detections, and rendering them on-screen. It concludes with a live demo showcasing the app's ability to detect and highlight objects in real-time, with an option to switch to a 'party mode' for a more dynamic visual effect.

Takeaways

• 😀 The video is a tutorial on building a real-time object detection app using React and TensorFlow.js.
• 🛠️ A computer vision template is provided on GitHub to help kickstart the development of the object detection app.
• 📚 The tutorial covers three main topics: accessing the computer vision template, setting up and coding with COCO SSD, and making real-time detections with the app and webcam.
• 🔍 The app uses a pre-built TensorFlow.js model that utilizes COCO SSD for real-time object detection.
• 💻 The tutorial guides through setting up a React app, capturing images from a webcam, and rendering detections to the screen.
• 🔑 The 'tensorflow-models/coco-ssd' package is a key dependency for the pre-trained COCO SSD model in the project.
• 🎨 The 'drawRect' function is created to visually represent the detected objects by drawing rectangles and text on the canvas.
• 🔄 The app starts with importing the model, loading the network, making detections, and then drawing the results on the canvas.
• 🎉 The tutorial also suggests enhancing the app with a 'party mode' feature that changes the color of the detections for a more dynamic effect.
• 🔍 The detections are logged in the console, showing details like bounding box coordinates, class of the object, and the confidence score.
• 📈 The tutorial concludes with a demonstration of the app detecting various objects in real-time using the webcam.

Q & A

What is the main topic of the video?

The main topic of the video is building a real-time object detection app using React JS and TensorFlow.js.

What are the three key things the video covers?

The video covers accessing the computer vision template, setting up and coding with COCO SSD, and making real-time detections using the app and webcam.

What is the purpose of the GitHub template mentioned in the video?

The GitHub template is designed to kickstart the development of a real-time object detection app, providing a foundation to build upon.

Which pre-built model is used for object detection in the video?

The video uses the pre-built TensorFlow.js model that utilizes COCO SSD for object detection.

How does the video guide the setup of the React JS app?

The video guides the setup by cloning the React computer vision template from GitHub and using the 'create-react-app' library.

What is the role of the 'tensorflow-models/coco-ssd' package in the project?

The 'tensorflow-models/coco-ssd' package provides the pre-trained COCO SSD model from TensorFlow.js, which is used for making object detections.

How does the video describe the process of capturing images from the webcam?

The video describes the process as streaming whatever is in the webcam's frame to the TensorFlow.js model for object detection.

What is the purpose of the 'drawRect' function in the utilities.js file?

The 'drawRect' function is used to draw rectangles and text on the canvas, representing the detected objects from the webcam feed.

How does the video demonstrate updating the app.js file?

The video demonstrates updating the app.js file by importing the required model, loading the network, making detections, and using the 'drawRect' function to visualize the results.

What additional feature does the video suggest for enhancing the app?

The video suggests updating the drawing function to change the color of the detections dynamically, creating a 'party mode' effect.

How can viewers access the code and resources mentioned in the video?

Viewers can access the code and resources by cloning the provided GitHub repository or downloading the code from the custom object detection React JS TensorFlow repo link provided in the video description.

Outlines

00:00

🛠️ Building a Real-Time Object Detection App with React and TensorFlow.js

The video script introduces a project to build a real-time object detection application using React and TensorFlow.js. The presenter outlines the process, starting with setting up a React app using a template available on GitHub, which accelerates the development. The app will utilize the COCO SSD model from TensorFlow.js for object detection. The script details the steps to clone the repository, set up the project in an IDE like VS Code, and install necessary dependencies, particularly the pre-trained COCO SSD model. The presenter also mentions an alternative for those who prefer to download and run the code without coding.

05:01

🔍 Setting Up and Coding with COCO SSD for Object Detection

This paragraph delves into the technical setup for the object detection app. It covers the installation of packages via npm install and the subsequent steps to update the app.js file. The presenter guides the audience through importing the COCO SSD model, loading the network, and setting up the detection function. The script explains how to use the webcam to capture images and pass them to the TensorFlow.js model for detection. It also includes the initial steps to start the React app and observe the detection results in the console.

10:02

🎨 Developing the Drawing Utility for Visual Detections

The script continues with the development of a drawing utility function named 'drawRect', which is designed to visualize the detection results on the webcam canvas. It explains how to create and style the function to draw rectangles and text based on the detection predictions. The function iterates over each detection, extracting relevant variables such as class, score, and bounding box coordinates. The presenter also demonstrates how to integrate this function into the app.js file to display the detections on the screen in real-time.

15:04

🥳 Enhancing Detections with 'Party Mode' and Finalizing the App

The final paragraph describes the enhancement of the detection app with a 'party mode' feature, which changes the color of the detection boxes for a more dynamic visual effect. The presenter shows how to modify the drawing function to implement this feature. After updating the function, the script concludes with running the app to see the real-time detections with the new visual style. The video ends with a summary of the steps taken and an invitation for viewers to share their thoughts and experiences with building their own real-time object detection apps.

Keywords

💡Real-time object detection

Real-time object detection refers to the process of identifying and locating objects within an image or video frame in real-time, as the video is being captured or streamed. In the video's context, it is the main theme where the creator is building an app to perform this task using TensorFlow.js and React.js. The script mentions setting up a React app to capture images from a webcam and using a TensorFlow.js model to detect objects, showcasing this concept in action.

💡React.js

React.js is a popular JavaScript library used for building user interfaces, particularly single-page applications. It is mentioned in the script as the foundational technology for the object detection app. The creator uses 'create-react-app' to set up the project, emphasizing its role in facilitating the development process.

💡TensorFlow.js

TensorFlow.js is a JavaScript library for training and deploying machine learning models in the browser or on Node.js. The script discusses leveraging TensorFlow.js for its object detection capabilities, specifically using a pre-built model that employs the COCO SSD algorithm for real-time detection.

💡COCO SSD

COCO SSD (Single Shot MultiBox Detector) is an algorithm used for object detection that is part of the TensorFlow.js models. It is highlighted in the script as the method the creator uses to perform real-time object detection within the app, allowing for the identification of various objects in the webcam's frame.

💡Webcam

A webcam is a digital camera that captures images or video streams and is used in the script to feed the real-time object detection app. The video mentions capturing images from the webcam to stream to the TensorFlow.js model, which then processes these images for object detection.

💡Computer vision

Computer vision is an interdisciplinary field that deals with how computers can gain high-level understanding from digital images or videos. In the script, the creator is building an app that utilizes computer vision techniques to detect objects in real-time, as evidenced by the use of a template and TensorFlow.js for this purpose.

💡Model

In the context of machine learning and the script, a model refers to a trained algorithm that makes predictions or decisions based on input data. The creator imports and loads a pre-trained COCO SSD model from TensorFlow.js to perform object detection within the app.

💡Detections

Detections in the script refer to the output of the object detection process, where the model identifies and locates objects within the video frame. The creator discusses making detections using the app and logs these in the console, which are then used to draw bounding boxes around detected objects.

💡Bounding box

A bounding box is a rectangular box that outlines an object within an image or video frame, used to indicate the location of the object. The script describes the creation of a 'drawRect' function to draw these bounding boxes around detected objects on the webcam canvas.

💡Canvas

In web development, the canvas refers to an HTML element used to draw graphics via scripting (usually JavaScript). The script mentions using a canvas to render the detections by drawing rectangles and text onto it, visualizing the object detection results.

💡Utilities.js

Utilities.js is a JavaScript file created in the script to contain helper functions for the app, such as 'drawRect' for drawing bounding boxes on the canvas. This file is an example of organizing code into modules to handle specific tasks within the app, enhancing maintainability and readability.

Highlights

Building a real-time object detection app using ReactJS and TensorFlow.js.

Utilizing a pre-built COCO SSD model from TensorFlow.js for object detection.

Accessing a computer vision template on GitHub for a head start on development.

Setting up and coding with COCO SSD to leverage TensorFlow.js's capabilities.

Making real-time detections using the app and webcam to identify objects in the frame.

The React app setup includes capturing images from the webcam to feed into the TensorFlow.js model.

Rendering detections to the screen to visualize different objects captured by the webcam.

Cloning the React computer vision template for a faster development process.

Using the 'create-react-app' library to streamline the app creation process.

Importing the pre-trained COCO SSD model to facilitate real-time object detection.

Installing dependencies via npm to set up the necessary packages for the app.

Updating the app.js file to integrate the TensorFlow.js model for detections.

Creating a 'drawRect' function to visually represent detections on the canvas.

Adjusting the drawing function to add a 'party mode' effect to detections.

Starting the React app to see real-time object detections in action.

Using console logs to monitor the output and performance of the model's detections.

The ability to detect various objects such as people, furniture, and other items.

Engaging with the audience by encouraging feedback on the app and its features.

Inviting viewers to subscribe and enable notifications for new video releases.

Transcripts

i spy something beginning with

um pete

[Music]

what's happening guys in today's video

we're going to be building our very own

real-time object detection app we're

going to be using refjs and

tensorflow.js to do this

and in order to speed us along the way

we're going to be taking a look at the

real-time object detection template that

i've set up that you're going to be able

to access

let's take a deep look as to what we're

going to be going through so in today's

video we're going to be covering

three key things so first up we're going

to be accessing our computer vision

template so this is a template that i've

set up for you on

github that's going to allow you to

kickstart your journey in terms of

building

your real-time object detection app

we're also going to be setting up and

coding with

coco ssd so for this we're going to

leverage the

pre-built tensorflow js model so this is

going to allow us to make

detections and then last but not least

we're going to make real-time detections

using our app and our webcam so we'll

actually be able to detect

different objects within our frame let's

take a look as to how this is all going

to work

so first up we're going to set up our

react js app so this is

all included inside of the template so

it's pretty easy to set up

then what we're going to do is capture

images from our webcams our webcam is

going to stream whatever's in that

particular frame

to our tensorflow.js model and then

we're going to render those detections

to the screen so you'll be able to see

all the different objects that have been

captured ready to do it

let's get to it alrighty so in order to

build our real-time object detection

app using react and tensorflow.js we're

going to be leveraging

a couple of things that are going to

help us along the way so first and

foremost i've gone and set up this

react computer vision template that's

going to allow us to get up to speed

a whole heap faster so we're going to be

able to clone this down

and build our app from there we've also

got another reaper which is if you don't

want to actually go on ahead and code

and you just want to download the code

and run it from the get go you can

actually download this link so this is

called

the custom object detection react js

tensorflow repo

bit of a mouthful i know but everything

that you need to actually run this

without writing a single line of code is

there

in terms of how we've actually gone and

built it we're using react

and specifically we've gone and used the

create react

app library to go and do that and then

we're also going to be leveraging

tensorflow.js and if we go and take a

look at the tfjs models

we're specifically going to be using the

object detection model

so this model actually uses coco ssd to

allow us to go and perform

real-time object detection enough

webbing let's actually get started so

what we're actually going to do is we're

going to clone this repo so the react

computer vision template so we can copy

that

and we're going to open up a new command

prompt

and in order to clone this repo we're

just going to go into our d

drive or a drive that you want to clone

this into and we're just going to type

in git

clone and then the name of our

repository so if we just minimize that

for a second

so basically what we've written is git

clone and then the link to our react

computer vision template

now all the links that i just mentioned

i'm going to make available in the

description below so if you want to pick

those up by all means

just grab them you'll be able to get

started super quick

so let's go on ahead and clone this

awesome so that's now cloned

now if we open up our d drive you should

have a

cloned repository so let's go ahead and

open that up

so you can see in fact we've now got a

folder called react

computer vision template and this

contains all the code we need to get

started

now in order to go and build up from

this we're going to open it up inside of

an

integrated development environment or a

coding environment so in this case we're

going to be using vs code

so what we'll do is we'll first up go

into that directory and then we can open

it up using the code dot command this is

only if you're using vs code

so let's cd into it and then we can type

in code and then dot and this will open

it up inside of vs code

and i'll just bring it onto the right

screen and there you go

so inside of this folder we've got a

bunch of stuff

so namely we've got a css file so

app.css

we've got an app.js file and this is

where we're going to be doing

the majority of our work we've also got

an index.s

or css file and an index.js file

now as i mentioned the majority of our

work's going to be inside of our app.js

file

but before we actually go on and start

making some updates to our code what

we're going to do is just make sure we

install our dependencies

so if you select package.json let's make

this a little bit bigger

you're going to be able to see all the

dependencies that we've got within

our application now in this case the

library that

is most important is or the package that

is most important

is the one named tensorflow dash models

forward slash coco

dash ssd so this is the pre-trained coco

ssd model that we're going to be able to

leverage from tensorflow.js

now i've also got tensorflow.js we've

also got react

we can delete this fingerpose one out

because that's from a previous code set

and in order to go and install all of

these all we need to do is open up a new

terminal so i can just open one up by

hitting control and the squiggly bracket

i never know what it's called

on a mac it's going to be commands

wiggly bracket and then

to go and install this stuff we're just

going to type in npm install

and let that run so this should take a

couple of minutes to run but once it's

done

you're going to see a node underscore

modules folder pop up

so we'll be right back in a second

alrighty so you can see that all of our

packages have installed and we're back

at our regular command line

now what we're going to go and do is

start making our updates to our app.js

file

so if you actually take a look at this

so let's just make this a little bit

smaller

you can see that we've got a bunch of to

do's now i've specifically called this

out

because it's basically going to walk you

through the steps that you need to

update in order to use this but likewise

if you wanted to use this for other use

cases you could do that as well

here what we're going to do is we're

going to go through steps one two

three four and five and then by the time

we've got through each one of those

steps we should

effectively have a real-time object

detection

reactor so the first thing that we need

to do is import the required

model here now the model that we're

actually going to need is actually

coming out of our coco ssd package

so let's go on ahead and import that

model first up

alrighty so that's our first dependency

imported so the line that we've just

written there is import

star as coco ssd from at tensorflow dash

models

forward slash coco ssd so this basically

is allowing us to download our

pre-trained tensorflow.js model

the next thing that we need to do is

actually import our drawing utility

but we're going to hold off on that

until the end because the last thing

that we want to do is draw

now the next thing that we would need to

do is actually go on ahead and load our

network into our model so let's go ahead

and do that okay so that's our model

loaded

so in order to do that we've created a

new variable called net

and then we've made this because our

function is asynchronous we're just

waiting for that to load

so here we're actually loading our coco

ssd model and this is what we imported

right up here and then we're using the

load method to actually go on ahead and

bring it in

now if we go to step 4 the next thing

that we want to do is actually start

making some detection so really quickly

we're already up to making some

detections so let's go ahead and start

doing that

so once we get these detections we're

going to be able to start our

application and actually log

these out so we'll actually be able to

see how model is actually performing

alrighty so that's the line of code that

we need to actually make our detections

so basically what we're doing here is

we're creating a new variable called obj

or

object and we're using our network that

we defined

up here so we're actually passing it

through to this detect function

so we're using that network and we're

passing through our video so this is the

video from our webcam

and we're passing that to our detect

function so ideally we should be able to

detect a bunch of objects

then what we're doing is we're console

logging that out so we should be able to

see

the output of each one of these

detections in our console

now what we need to do is actually start

up our app so this is going to allow us

to see our objects and whether or not

our model is actually performing well so

to start our app we just need to type in

npm

start and this is going to start our

react app and

open up a new browser so you can see

it's opened up a new browser and it's

gone directly to localhost 3000 so by

default

this is where our react app is going to

start

so let's give that a couple of seconds

while it compiles and then ideally we

should be able to see a couple of

detections in our console

all right so you can see that we've got

our camera showing up on the screen

now if we go and inspect our console

and if we open that up you can see we're

getting a bunch of detections here and

if we open this up

you can see that in fact we're making

detection so we've got a couple of

things showing here so we've got an

array and inside of

our array we've got a b box this

represents

our bounding box and each one of these

represents a specific thing

so this is our x coordinate this is our

y coordinate and this is our box width

and our box height

we can also see our different classes so

in this case you can see the class as

person because it's detecting me

and we can also see our score but at the

moment we're not actually drawing

anything to the screen

so let's stop our app and actually

finish our drawing function so we'll

actually be able to see our results

okay so the last two things that we

needed to do were update our drawing

utility

and bring it in right up here

so let's go on ahead and do that now to

start building our drawing utility we're

just going to right click on source

create a new file and call that

utilities dot js

and in here we're going to define a

function that's going to allow us to

draw to the screen

so this function is going to be called

draw rect short for draw rectangle

and it's going to allow us to pass our

predictions to this function

and draw them to our actual webcam

screen or to our webcam canvas

so let's go ahead and start doing that

okay so we're going to

start setting up our function and

remember our function is going to be

called

draw rect

alrighty so to our draw rest function

we're going to be passing out detections

and if you actually take a look this is

actually going to be our object variable

here

and we're also going to pass through our

canvas so this is going to be our canvas

that we've already had predefined

what we're then going to do is loop

through each one of our detections using

the 4h function

so let's do that

so the first thing that we're actually

doing within our draw rect function

is we're going and grabbing our x

variable our y variable our width and

our height so remember when we're

console logging out our predictions we

were able to see our class and our text

these are exactly the same variables

here we're just extracting them out now

we're also extracting our class so in

this case it was going to be person from

our last prediction

now the next thing that we want to do is

set up some styling and actually go on

ahead and draw our rectangles

so let's power through that

okay so we've finished up our code now

what we've gone and done

is we've first upset our styling so

there we've created a new variable

called color and this is going to hold

the color of our box as well as the

color of our text

and we've also gone and set this should

actually be stroke style so let's go and

change that

and we've set that to color we've also

set our fill style to color

and we've also set our font so in this

case it's going to be 18 pixels

and aerial then what we've gone and done

is we've gone and drawn

our rectangles and text so we've used

our canvas and we've

commenced our path we've then used the

fill text method

and to that we've passed the text that

we've extracted from our prediction

as well as the x and y coordinates in

terms of their placement

and then we're drawing our rectangle as

well so to that we're passing our x

variable our y variable our width and

our height

so this should ideally draw a rectangle

around each one of our predictions

and then last but not least we're

drawing our stroke so this is going to

actually apply it to the screen so we

can save this now

and then all we need to do is bring this

into our app.js file and

run it for each prediction so let's go

ahead and do that

so right up here on step two we can

bring in our drawrect function

so that's our drawrect function imported

now what we need to do is

just run it down here so in step five we

just need to use our drawrect function

and pass through our object so these

were all about individual predictions

and pass through our canvas

so if we save that now ideally what

should happen is if we go back

to our browser we should be able to see

our

detection so you can see it's right up

in the top screen but you can see that

we're actually drawing our box

now if we adjust our camera a little bit

and take this down you can see that

we're making a whole bunch of detection

so we're detecting myself as a person

the couch we're also detecting the chair

pretty cool right now if we got some

other stuff i don't actually have my

phone on me but maybe if we tried out

our bottle

it's saying cup bottle there you go so

you can see that we're doing we're

creating real time detections

using our webcam and using our pre-built

tensorflow.js model now what we could do

as well if we wanted to make this a

little bit fancier or go

and make detections in party mode we

could do that as well

so let's go ahead and do that

so now what we need to do is we

basically just need to update

our drawing function so what we can do

here is just go back into utilities.js

and then rather than just automatically

setting our color to green we're going

to do a little bit of magic to make this

a little bit more

party mode so let's go ahead and define

this so rather than our color just being

green

we're going to change it to pound sign

and then plus

and if we save it now and go back to our

app

you can see that our detections are now

appearing in party modes our person

is flashing and if we go and use our

bottle

you can see we're getting detections in

party mode

and that about wraps it up so we've gone

through quite a fair bit now if we

go back and take a look at our app we

went through our importing of our model

we

created our drawrect function and

imported it there we loaded up our

neural network

then we made our detections and we also

drew it

to our canvas and that about wraps it up

thanks so much for tuning in guys

hopefully you found this video useful if

you did

be sure to give it a thumbs up hit

subscribe and tick that bell so you get

notified of when i'm releasing new

videos

let me know what you think of my mo and

also let me know how you went about

building your real-time object detection

app

thanks again for tuning in peace