Building an Object Detection App with Tensorflow.JS and React.JS in 15 Minutes | COCO SSD
Summary
TLDRThis video tutorial guides viewers through building a real-time object detection app using React and TensorFlow.js. The host introduces a pre-built template on GitHub to expedite the setup, then demonstrates how to integrate the COCO SSD model from TensorFlow.js for object detection. The tutorial covers setting up the React app, capturing webcam images, processing detections, and rendering them on-screen. It concludes with a live demo showcasing the app's ability to detect and highlight objects in real-time, with an option to switch to a 'party mode' for a more dynamic visual effect.
Takeaways
- 😀 The video is a tutorial on building a real-time object detection app using React and TensorFlow.js.
- 🛠️ A computer vision template is provided on GitHub to help kickstart the development of the object detection app.
- 📚 The tutorial covers three main topics: accessing the computer vision template, setting up and coding with COCO SSD, and making real-time detections with the app and webcam.
- 🔍 The app uses a pre-built TensorFlow.js model that utilizes COCO SSD for real-time object detection.
- 💻 The tutorial guides through setting up a React app, capturing images from a webcam, and rendering detections to the screen.
- 🔑 The 'tensorflow-models/coco-ssd' package is a key dependency for the pre-trained COCO SSD model in the project.
- 🎨 The 'drawRect' function is created to visually represent the detected objects by drawing rectangles and text on the canvas.
- 🔄 The app starts with importing the model, loading the network, making detections, and then drawing the results on the canvas.
- 🎉 The tutorial also suggests enhancing the app with a 'party mode' feature that changes the color of the detections for a more dynamic effect.
- 🔍 The detections are logged in the console, showing details like bounding box coordinates, class of the object, and the confidence score.
- 📈 The tutorial concludes with a demonstration of the app detecting various objects in real-time using the webcam.
Q & A
What is the main topic of the video?
-The main topic of the video is building a real-time object detection app using React JS and TensorFlow.js.
What are the three key things the video covers?
-The video covers accessing the computer vision template, setting up and coding with COCO SSD, and making real-time detections using the app and webcam.
What is the purpose of the GitHub template mentioned in the video?
-The GitHub template is designed to kickstart the development of a real-time object detection app, providing a foundation to build upon.
Which pre-built model is used for object detection in the video?
-The video uses the pre-built TensorFlow.js model that utilizes COCO SSD for object detection.
How does the video guide the setup of the React JS app?
-The video guides the setup by cloning the React computer vision template from GitHub and using the 'create-react-app' library.
What is the role of the 'tensorflow-models/coco-ssd' package in the project?
-The 'tensorflow-models/coco-ssd' package provides the pre-trained COCO SSD model from TensorFlow.js, which is used for making object detections.
How does the video describe the process of capturing images from the webcam?
-The video describes the process as streaming whatever is in the webcam's frame to the TensorFlow.js model for object detection.
What is the purpose of the 'drawRect' function in the utilities.js file?
-The 'drawRect' function is used to draw rectangles and text on the canvas, representing the detected objects from the webcam feed.
How does the video demonstrate updating the app.js file?
-The video demonstrates updating the app.js file by importing the required model, loading the network, making detections, and using the 'drawRect' function to visualize the results.
What additional feature does the video suggest for enhancing the app?
-The video suggests updating the drawing function to change the color of the detections dynamically, creating a 'party mode' effect.
How can viewers access the code and resources mentioned in the video?
-Viewers can access the code and resources by cloning the provided GitHub repository or downloading the code from the custom object detection React JS TensorFlow repo link provided in the video description.
Outlines
🛠️ Building a Real-Time Object Detection App with React and TensorFlow.js
The video script introduces a project to build a real-time object detection application using React and TensorFlow.js. The presenter outlines the process, starting with setting up a React app using a template available on GitHub, which accelerates the development. The app will utilize the COCO SSD model from TensorFlow.js for object detection. The script details the steps to clone the repository, set up the project in an IDE like VS Code, and install necessary dependencies, particularly the pre-trained COCO SSD model. The presenter also mentions an alternative for those who prefer to download and run the code without coding.
🔍 Setting Up and Coding with COCO SSD for Object Detection
This paragraph delves into the technical setup for the object detection app. It covers the installation of packages via npm install and the subsequent steps to update the app.js file. The presenter guides the audience through importing the COCO SSD model, loading the network, and setting up the detection function. The script explains how to use the webcam to capture images and pass them to the TensorFlow.js model for detection. It also includes the initial steps to start the React app and observe the detection results in the console.
🎨 Developing the Drawing Utility for Visual Detections
The script continues with the development of a drawing utility function named 'drawRect', which is designed to visualize the detection results on the webcam canvas. It explains how to create and style the function to draw rectangles and text based on the detection predictions. The function iterates over each detection, extracting relevant variables such as class, score, and bounding box coordinates. The presenter also demonstrates how to integrate this function into the app.js file to display the detections on the screen in real-time.
🥳 Enhancing Detections with 'Party Mode' and Finalizing the App
The final paragraph describes the enhancement of the detection app with a 'party mode' feature, which changes the color of the detection boxes for a more dynamic visual effect. The presenter shows how to modify the drawing function to implement this feature. After updating the function, the script concludes with running the app to see the real-time detections with the new visual style. The video ends with a summary of the steps taken and an invitation for viewers to share their thoughts and experiences with building their own real-time object detection apps.
Mindmap
Keywords
💡Real-time object detection
💡React.js
💡TensorFlow.js
💡COCO SSD
💡Webcam
💡Computer vision
💡Model
💡Detections
💡Bounding box
💡Canvas
💡Utilities.js
Highlights
Building a real-time object detection app using ReactJS and TensorFlow.js.
Utilizing a pre-built COCO SSD model from TensorFlow.js for object detection.
Accessing a computer vision template on GitHub for a head start on development.
Setting up and coding with COCO SSD to leverage TensorFlow.js's capabilities.
Making real-time detections using the app and webcam to identify objects in the frame.
The React app setup includes capturing images from the webcam to feed into the TensorFlow.js model.
Rendering detections to the screen to visualize different objects captured by the webcam.
Cloning the React computer vision template for a faster development process.
Using the 'create-react-app' library to streamline the app creation process.
Importing the pre-trained COCO SSD model to facilitate real-time object detection.
Installing dependencies via npm to set up the necessary packages for the app.
Updating the app.js file to integrate the TensorFlow.js model for detections.
Creating a 'drawRect' function to visually represent detections on the canvas.
Adjusting the drawing function to add a 'party mode' effect to detections.
Starting the React app to see real-time object detections in action.
Using console logs to monitor the output and performance of the model's detections.
The ability to detect various objects such as people, furniture, and other items.
Engaging with the audience by encouraging feedback on the app and its features.
Inviting viewers to subscribe and enable notifications for new video releases.
Transcripts
i spy something beginning with
um pete
[Music]
what's happening guys in today's video
we're going to be building our very own
real-time object detection app we're
going to be using refjs and
tensorflow.js to do this
and in order to speed us along the way
we're going to be taking a look at the
real-time object detection template that
i've set up that you're going to be able
to access
let's take a deep look as to what we're
going to be going through so in today's
video we're going to be covering
three key things so first up we're going
to be accessing our computer vision
template so this is a template that i've
set up for you on
github that's going to allow you to
kickstart your journey in terms of
building
your real-time object detection app
we're also going to be setting up and
coding with
coco ssd so for this we're going to
leverage the
pre-built tensorflow js model so this is
going to allow us to make
detections and then last but not least
we're going to make real-time detections
using our app and our webcam so we'll
actually be able to detect
different objects within our frame let's
take a look as to how this is all going
to work
so first up we're going to set up our
react js app so this is
all included inside of the template so
it's pretty easy to set up
then what we're going to do is capture
images from our webcams our webcam is
going to stream whatever's in that
particular frame
to our tensorflow.js model and then
we're going to render those detections
to the screen so you'll be able to see
all the different objects that have been
captured ready to do it
let's get to it alrighty so in order to
build our real-time object detection
app using react and tensorflow.js we're
going to be leveraging
a couple of things that are going to
help us along the way so first and
foremost i've gone and set up this
react computer vision template that's
going to allow us to get up to speed
a whole heap faster so we're going to be
able to clone this down
and build our app from there we've also
got another reaper which is if you don't
want to actually go on ahead and code
and you just want to download the code
and run it from the get go you can
actually download this link so this is
called
the custom object detection react js
tensorflow repo
bit of a mouthful i know but everything
that you need to actually run this
without writing a single line of code is
there
in terms of how we've actually gone and
built it we're using react
and specifically we've gone and used the
create react
app library to go and do that and then
we're also going to be leveraging
tensorflow.js and if we go and take a
look at the tfjs models
we're specifically going to be using the
object detection model
so this model actually uses coco ssd to
allow us to go and perform
real-time object detection enough
webbing let's actually get started so
what we're actually going to do is we're
going to clone this repo so the react
computer vision template so we can copy
that
and we're going to open up a new command
prompt
and in order to clone this repo we're
just going to go into our d
drive or a drive that you want to clone
this into and we're just going to type
in git
clone and then the name of our
repository so if we just minimize that
for a second
so basically what we've written is git
clone and then the link to our react
computer vision template
now all the links that i just mentioned
i'm going to make available in the
description below so if you want to pick
those up by all means
just grab them you'll be able to get
started super quick
so let's go on ahead and clone this
awesome so that's now cloned
now if we open up our d drive you should
have a
cloned repository so let's go ahead and
open that up
so you can see in fact we've now got a
folder called react
computer vision template and this
contains all the code we need to get
started
now in order to go and build up from
this we're going to open it up inside of
an
integrated development environment or a
coding environment so in this case we're
going to be using vs code
so what we'll do is we'll first up go
into that directory and then we can open
it up using the code dot command this is
only if you're using vs code
so let's cd into it and then we can type
in code and then dot and this will open
it up inside of vs code
and i'll just bring it onto the right
screen and there you go
so inside of this folder we've got a
bunch of stuff
so namely we've got a css file so
app.css
we've got an app.js file and this is
where we're going to be doing
the majority of our work we've also got
an index.s
or css file and an index.js file
now as i mentioned the majority of our
work's going to be inside of our app.js
file
but before we actually go on and start
making some updates to our code what
we're going to do is just make sure we
install our dependencies
so if you select package.json let's make
this a little bit bigger
you're going to be able to see all the
dependencies that we've got within
our application now in this case the
library that
is most important is or the package that
is most important
is the one named tensorflow dash models
forward slash coco
dash ssd so this is the pre-trained coco
ssd model that we're going to be able to
leverage from tensorflow.js
now i've also got tensorflow.js we've
also got react
we can delete this fingerpose one out
because that's from a previous code set
and in order to go and install all of
these all we need to do is open up a new
terminal so i can just open one up by
hitting control and the squiggly bracket
i never know what it's called
on a mac it's going to be commands
wiggly bracket and then
to go and install this stuff we're just
going to type in npm install
and let that run so this should take a
couple of minutes to run but once it's
done
you're going to see a node underscore
modules folder pop up
so we'll be right back in a second
alrighty so you can see that all of our
packages have installed and we're back
at our regular command line
now what we're going to go and do is
start making our updates to our app.js
file
so if you actually take a look at this
so let's just make this a little bit
smaller
you can see that we've got a bunch of to
do's now i've specifically called this
out
because it's basically going to walk you
through the steps that you need to
update in order to use this but likewise
if you wanted to use this for other use
cases you could do that as well
here what we're going to do is we're
going to go through steps one two
three four and five and then by the time
we've got through each one of those
steps we should
effectively have a real-time object
detection
reactor so the first thing that we need
to do is import the required
model here now the model that we're
actually going to need is actually
coming out of our coco ssd package
so let's go on ahead and import that
model first up
alrighty so that's our first dependency
imported so the line that we've just
written there is import
star as coco ssd from at tensorflow dash
models
forward slash coco ssd so this basically
is allowing us to download our
pre-trained tensorflow.js model
the next thing that we need to do is
actually import our drawing utility
but we're going to hold off on that
until the end because the last thing
that we want to do is draw
now the next thing that we would need to
do is actually go on ahead and load our
network into our model so let's go ahead
and do that okay so that's our model
loaded
so in order to do that we've created a
new variable called net
and then we've made this because our
function is asynchronous we're just
waiting for that to load
so here we're actually loading our coco
ssd model and this is what we imported
right up here and then we're using the
load method to actually go on ahead and
bring it in
now if we go to step 4 the next thing
that we want to do is actually start
making some detection so really quickly
we're already up to making some
detections so let's go ahead and start
doing that
so once we get these detections we're
going to be able to start our
application and actually log
these out so we'll actually be able to
see how model is actually performing
alrighty so that's the line of code that
we need to actually make our detections
so basically what we're doing here is
we're creating a new variable called obj
or
object and we're using our network that
we defined
up here so we're actually passing it
through to this detect function
so we're using that network and we're
passing through our video so this is the
video from our webcam
and we're passing that to our detect
function so ideally we should be able to
detect a bunch of objects
then what we're doing is we're console
logging that out so we should be able to
see
the output of each one of these
detections in our console
now what we need to do is actually start
up our app so this is going to allow us
to see our objects and whether or not
our model is actually performing well so
to start our app we just need to type in
npm
start and this is going to start our
react app and
open up a new browser so you can see
it's opened up a new browser and it's
gone directly to localhost 3000 so by
default
this is where our react app is going to
start
so let's give that a couple of seconds
while it compiles and then ideally we
should be able to see a couple of
detections in our console
all right so you can see that we've got
our camera showing up on the screen
now if we go and inspect our console
and if we open that up you can see we're
getting a bunch of detections here and
if we open this up
you can see that in fact we're making
detection so we've got a couple of
things showing here so we've got an
array and inside of
our array we've got a b box this
represents
our bounding box and each one of these
represents a specific thing
so this is our x coordinate this is our
y coordinate and this is our box width
and our box height
we can also see our different classes so
in this case you can see the class as
person because it's detecting me
and we can also see our score but at the
moment we're not actually drawing
anything to the screen
so let's stop our app and actually
finish our drawing function so we'll
actually be able to see our results
okay so the last two things that we
needed to do were update our drawing
utility
and bring it in right up here
so let's go on ahead and do that now to
start building our drawing utility we're
just going to right click on source
create a new file and call that
utilities dot js
and in here we're going to define a
function that's going to allow us to
draw to the screen
so this function is going to be called
draw rect short for draw rectangle
and it's going to allow us to pass our
predictions to this function
and draw them to our actual webcam
screen or to our webcam canvas
so let's go ahead and start doing that
okay so we're going to
start setting up our function and
remember our function is going to be
called
draw rect
alrighty so to our draw rest function
we're going to be passing out detections
and if you actually take a look this is
actually going to be our object variable
here
and we're also going to pass through our
canvas so this is going to be our canvas
that we've already had predefined
what we're then going to do is loop
through each one of our detections using
the 4h function
so let's do that
so the first thing that we're actually
doing within our draw rect function
is we're going and grabbing our x
variable our y variable our width and
our height so remember when we're
console logging out our predictions we
were able to see our class and our text
these are exactly the same variables
here we're just extracting them out now
we're also extracting our class so in
this case it was going to be person from
our last prediction
now the next thing that we want to do is
set up some styling and actually go on
ahead and draw our rectangles
so let's power through that
okay so we've finished up our code now
what we've gone and done
is we've first upset our styling so
there we've created a new variable
called color and this is going to hold
the color of our box as well as the
color of our text
and we've also gone and set this should
actually be stroke style so let's go and
change that
and we've set that to color we've also
set our fill style to color
and we've also set our font so in this
case it's going to be 18 pixels
and aerial then what we've gone and done
is we've gone and drawn
our rectangles and text so we've used
our canvas and we've
commenced our path we've then used the
fill text method
and to that we've passed the text that
we've extracted from our prediction
as well as the x and y coordinates in
terms of their placement
and then we're drawing our rectangle as
well so to that we're passing our x
variable our y variable our width and
our height
so this should ideally draw a rectangle
around each one of our predictions
and then last but not least we're
drawing our stroke so this is going to
actually apply it to the screen so we
can save this now
and then all we need to do is bring this
into our app.js file and
run it for each prediction so let's go
ahead and do that
so right up here on step two we can
bring in our drawrect function
so that's our drawrect function imported
now what we need to do is
just run it down here so in step five we
just need to use our drawrect function
and pass through our object so these
were all about individual predictions
and pass through our canvas
so if we save that now ideally what
should happen is if we go back
to our browser we should be able to see
our
detection so you can see it's right up
in the top screen but you can see that
we're actually drawing our box
now if we adjust our camera a little bit
and take this down you can see that
we're making a whole bunch of detection
so we're detecting myself as a person
the couch we're also detecting the chair
pretty cool right now if we got some
other stuff i don't actually have my
phone on me but maybe if we tried out
our bottle
it's saying cup bottle there you go so
you can see that we're doing we're
creating real time detections
using our webcam and using our pre-built
tensorflow.js model now what we could do
as well if we wanted to make this a
little bit fancier or go
and make detections in party mode we
could do that as well
so let's go ahead and do that
so now what we need to do is we
basically just need to update
our drawing function so what we can do
here is just go back into utilities.js
and then rather than just automatically
setting our color to green we're going
to do a little bit of magic to make this
a little bit more
party mode so let's go ahead and define
this so rather than our color just being
green
we're going to change it to pound sign
and then plus
and if we save it now and go back to our
app
you can see that our detections are now
appearing in party modes our person
is flashing and if we go and use our
bottle
you can see we're getting detections in
party mode
and that about wraps it up so we've gone
through quite a fair bit now if we
go back and take a look at our app we
went through our importing of our model
we
created our drawrect function and
imported it there we loaded up our
neural network
then we made our detections and we also
drew it
to our canvas and that about wraps it up
thanks so much for tuning in guys
hopefully you found this video useful if
you did
be sure to give it a thumbs up hit
subscribe and tick that bell so you get
notified of when i'm releasing new
videos
let me know what you think of my mo and
also let me know how you went about
building your real-time object detection
app
thanks again for tuning in peace
Посмотреть больше похожих видео
Tensorflow Lite with Object Detection on Raspberry Pi!
Build a 🔥 Chat App with FastAPI and React JS (👾 Discord Clone)
Getting Started with React Native: Install Node.js, Create Your First App, and Pair with Expo Go
Is Redis the ONLY database you need? // Fullstack app from scratch with Next.js & Redis
YOLOv8: How to Train for Object Detection on a Custom Dataset
tensorflow custom object detection model | raspberry pi 4 tensorflow custom object detection
5.0 / 5 (0 votes)