Importing Development Data | Lecture 92 | Node.JS ๐ฅ
Summary
TLDRIn this instructional video, the presenter takes a break from API development to demonstrate a script that imports tour data from a JSON file into a MongoDB database. The script operates independently from the express application and includes steps to connect to the database, read the JSON file, and utilize the tour model for data import. Additionally, the video covers command-line interaction, allowing users to run the script with options to import or delete data. The presenter addresses potential validation errors and concludes with a successful data import, setting the stage for future API enhancements.
Takeaways
- ๐ ๏ธ The script's purpose is to import tour data from a JSON file into a MongoDB database, separate from the main Express application.
- ๐ The script is created in a 'data' folder and named 'importDevData.js', emphasizing its standalone nature.
- ๐ Mongoose is required for database interactions, and environment variables are needed for database connection details.
- ๐ The script requires the file system module 'fs' to read the JSON file and the 'tour' model to interact with the database.
- ๐ The JSON file is read synchronously with a specified encoding of 'utf eight'.
- ๐ An async function is used to import data into the database, utilizing 'tour.create' to handle an array of objects.
- ๐๏ธ There's a function to delete all data from the MongoDB collection using 'deleteMany', a method provided by Mongoose.
- ๐ The script can be run with command-line options to either import or delete data, demonstrated by checking 'process.argv'.
- ๐ The script includes error handling with try-catch blocks to log issues during data import or deletion.
- ๐ซ Validation errors are encountered due to incomplete data in the JSON file, highlighting the importance of data integrity.
- ๐ The script exits the process after completing its task using 'process.exit', ensuring the script doesn't continue running unnecessarily.
Q & A
What is the purpose of the script mentioned in the video?
-The script's purpose is to import tour data from a JSON file into a MongoDB database, which is a standalone task separate from the main Express application.
Why was the decision made to include the script-building exercise in the course?
-The script-building exercise was included because it was considered a nice little exercise that adds value to the course, despite the possibility of just providing the script in the starter files.
What does the script need to perform its task?
-The script requires mongoose for database interaction, the .env package for environment variables, the file system module to read the JSON file, and the tour model to write the data into the database.
How does the script handle reading the JSON file?
-The script uses the synchronous version of `fs.readFile` to read the JSON file named 'toursSimple.json' and specifies the encoding as 'utf eight'.
What is the significance of the 'importData' function in the script?
-The 'importData' function is an asynchronous function that imports data into the database by using the 'tour.create' method with an array of JavaScript objects obtained from parsing the JSON file.
How can the script handle the deletion of existing data in the database?
-The script includes a function to delete all data from the database using the 'delete many' function of the tour model, which removes all documents in the tours collection.
What is the role of 'process.argv' in the script?
-'process.argv' is used to interact with the command line, allowing the script to accept arguments that determine whether to import or delete data from the database.
Why is the 'process.exit' function used in the script?
-The 'process.exit' function is used to stop the script execution after completing the import or delete operation, as it is a small standalone script and not part of a larger application.
What issue occurred when the script was first run to import data?
-The script encountered validation errors because the JSON file contained some tours with incomplete data, specifically missing 'group size' and other required fields.
How can the script be improved to avoid importing incomplete data?
-The script can be improved by adding checks to ensure that all required fields are present in the JSON data before attempting to import it into the database.
What is the final step in the script after successfully importing the data?
-The final step is to log a success message to the console indicating that the data has been successfully loaded into the database and then exit the process using 'process.exit'.
Outlines
๐ ๏ธ Building a Data Import Script for MongoDB
The script's purpose is to import tour data from a JSON file into a MongoDB database, operating independently from the main Express application. The author considers whether to include this as an exercise or provide it pre-built but decides to include it for educational value. The process involves creating a script in the data folder named 'importDevData.js', setting up necessary modules including mongoose for database connection, the 'fs' module for file system access, and the tour model for data insertion. The script reads a JSON file synchronously and includes error handling. It demonstrates the use of 'tour.create' to insert data into the database, converting JSON data to JavaScript objects with 'JSON.parse'.
๐๏ธ Deleting Existing Data and Command Line Interaction
The script also features a function to delete all existing data from the database using 'deleteMany', which removes all documents in the collection. The author discusses the use of 'process.argv' to interact with the command line, allowing the script to be run with different options to either import or delete data. The script is tested by running it with node and observing the output of 'console.log(process.argv)'. Adjustments are made to correctly specify file paths using the 'dirname' variable, and the script is enhanced to handle command line arguments for importing or deleting data.
๐ง Refining the Script with Command Line Options
The script is refined to check command line arguments and execute the appropriate function based on the user's input. If the '--import' option is provided, the import function is executed; if the '--delete' option is used, the delete function is triggered. The author demonstrates running the script with the delete option, which successfully removes all data from the database, and then attempts to run it with the import option. However, an error occurs due to validation errors related to missing 'group size' in the tour data. The author investigates and identifies that the issue stems from incomplete data entries added in a previous section of the course.
๐ Successful Data Import and Wrapping Up
After addressing the data validation issues by removing incomplete tour entries, the author successfully imports the data into the database using the script with the import option. The script includes 'process.exit' to terminate the process after completing the operation. The author concludes the tutorial by expressing satisfaction with the script's functionality and looks forward to the next video, where the focus will be on enhancing the API with new features using the imported data.
Mindmap
Keywords
๐กAPI
๐กJSON
๐กMongoDB
๐กMongoose
๐กExpress
๐กEnvironment Variables
๐กFile System Module
๐กAsynchronous Function
๐กTry-Catch Block
๐กCommand Line
๐กProcess.argv
Highlights
Building a script to import tour data from a JSON file into MongoDB database.
The script is independent of the express application and will be run separately.
Using mongoose for database connection and environment variables from .env package.
Accessing the file system module to read the JSON file.
Importing the tour model to write tours data into the database.
Reading the JSON file synchronously with fs.readFile.
Creating an async function to import data into the database using tour.create.
Using JSON.parse to convert JSON data into JavaScript objects.
Creating a function to delete all data from the collection using deleteMany.
Mongoose's deleteMany method is similar to MongoDB's native function.
Interacting with the command line to run functions based on user input.
Using process.argv to access command line arguments.
Creating a command line application to import or delete data based on options.
Using process.exit to stop the application after running the desired function.
Handling validation errors due to incomplete tour data.
Ensuring that the script exits properly after importing or deleting data.
Finalizing the script with a successful data import into MongoDB.
Transcripts
Let's now take a small break
from building our API
and build a fun little script
that will import the tour data from our JSON file
into the MongoDB database.
And I wasn't really sure
if I should include this video in the course
or if I should just come with the script
already in the course starter files.
But I thought I thought it was a nice little exercise
and so I included it anyways.
So, anyway, let's now get started.
So, basically we're gonna create a script
that will simply load the data from the JSON file
as I just said into the database.
And this script is completely independent
of the rest of our express application.
And so we'll run this completely separately
from the comment line just to import everything once.
Okay, so let me actually create the script
right here in the data folder.
So new file,
import,
dev data.js.
All right.
So, what are we gonna need for this?
We will need mongoose of course
so let's start with that.
Or actually, let's just go ahead into the server.js file
and copy this stuff
because why writing it all over again.
So let's copy everything
and then delete what we don't need.
So we don't need our express application
we also need the .env package
because we need our environment variables
in order to be able to connect to the database again, okay.
And we need to connect to the database in this script
again because it runs completely independent
from the express application.
It's only gonna run once in the beginning.
Next up, we need access to the file system module
because of course we want to read the JSON file.
So require fs
and finally we also need access to the tour model
because the tour model is
where we want to write the tours to, right?
So, tour equals
and now let's find the path there
so from the place where we are right now
we need to go up one level.
And what's going on here with these quotes?
All right, so one level up
and we're in dev data.
So we need another level up
so that we're in the main folder.
And from there we go in to models
and into the tour model.
Okay and that should be it for setup.
Now let's start by reading the file.
So, read JSON file
and that should be fairly simple.
So the tours are at fs.readfile
and we can use the synchronous version of course.
And let's simply say tours simple.json, okay.
And then,
also the encoding.
So, file encoding, utf eight.
Give it a save.
And so now we can write the actual function
that is gonna import the data into the database.
So import data into database.
Okay, just like this.
And so let's create this function.
Import data, import data,
yep.
And that's gonna be and async function
which does not need any arguments
and so let's again use a try catch block here.
And here I'm simply gonna log it to the console
if there is some error,
just to know what's going on in that case.
And now here,
what we will do is very simple.
We are simply gonna await,
tour.create.
So we already used tour.create
and we pass then an object back then right?
But the create method can also accept an array of objects.
And in that case
it will then simply create a new document
for each of the objects in the array.
So, very simple,
all we have to do is to specify our tours data here, right?
And actually it's not 100% correct
because remember that this is JSON.
And so we need to first convert it actually
into a JavaScript object using json.parse.
Okay and so now we actually have
an array of JavaScript objects
that we can now pass into the create method, okay.
And if that was successful,
then the next line is gonna be executed
and so here we can say data successfully
loaded, all right.
And this should already do the job.
Now what about the data that is already in the database?
We can also create an easy way
to basically delete all of that data at the same time.
And so let's simply go ahead and do that as well.
So, delete all data
from collection, let's say.
And this weird yellow color that you see here
actually comes from an extension that I have installed here.
And so to get rid of that,
I'm simply gonna write database again, okay.
So delete data
and again this is gonna be an async function
without any arguments.
And let me actually copy this code here.
So, delete it
and now about the deleting itself
we can use the delete many function.
Okay and actually I showed you this one
back in the intro to MongoDB, right?
Where we could use delete many
and then simply pass in nothing in there
and that would then delete
all of the documents in a certain collection, right?
And so mongoose basically implemented
the same function here on the model, okay.
So in this case,
the tour model has access to this delete many method
which will then do exactly the same
as delete many does in native MongoDB, right?
So, remember that mongoose is just a,
like a layer of abstraction on top of MongoDB.
Which is why it doesn't use the exact same functions
but it still gives us access to some similar ones
or to ones that actually have the same name.
So delete many actually has the same name
as the native MongoDB function, all right?
So again, what this is gonna do is to simply go ahead
and delete all the documents in the tours collection.
So, we have our two functions here
but if we now actually run this file
then nothing will happen.
And that's because
we're not calling any of these functions anywhere, right?
Now we could go ahead
and simply write something like import data here
and then simply call the function here
but I wanted to make this a little bit more fun.
So let's now actually learn a tiny little bit
about interacting with the command line, okay.
And so I'm actually gonna go ahead
and run this file without calling any of these functions.
But instead I'm gonna log to the console
process.argv,
okay.
Just so we can see what process.argv actually is
so that we can then use it.
All right.
Let me open up here another terminal
and then I will use node,
go into dev data
then into data
and then in there import dev data.
And we get some errors here.
Let's see where.
Ah, yeah so it's because of this,
of this file name.
So I guess we should specify the path to there basically.
So let's run this one again
and again we have this error.
And yeah, of course I get this error.
That's a stupid one.
Remember how I told you
that this dot here is always relative from the folder
where the node application was actually started.
And so that's the home folder.
And so we're basically looking for this file here
in the home folder, okay.
So what I should use instead is the dir name,
variable that is available to us everywhere.
So that goes like this.
All right.
Give it another save
and clear up the console and run it again.
And so now it works.
And so here
is the result of this console.log that we have down here,
So process.argv
and basically that is an array
of these two arguments of running this node process.
So, this here is basically
where the node command is located.
So this equivalent to this node
and then the second one,
so this path to this file is actually this here, okay.
So let's quit this here
and let's add kind of an option here.
So I'm gonna write,
dash, dash import
and so I'm sure you have seen something like this
many times before.
For example, when we save a package as a dev dependency
we do it like this.
Save dev, and so we use the same
kind of format for specifying options.
Okay so, dash dash
and then whatever string we put here.
And so I choose to basically specify
the import option like this.
And so you see that now the third argument
is dash dash import, all right?
And so that means that we can now go ahead
and basically use this data here
in order to write
a very simple command line application basically
which will import the data when we specify this option
and will delete the data
when we specify the delete option, all right?
So, let's actually do that.
So, if
process.argv
and it's an array
and we want the third.
So zero, one, two.
So if that element is equal to import
well, then we want to run import data.
Right?
If...
Process.argv two
is equal,
to delete then we want to run
delete data.
And that's it.
So, that should actually give us
the result that we're looking for.
Let's finish this here.
And so now let's run the command here
with delete in order to delete all the data
that we have in the database.
So let's try that out.
It's doing something
and data successfully deleted.
So let's take a look at that now.
And if we run now this get all tours route
then indeed we have zero results.
So, all our tours are now gone.
So it worked.
Now this process here is basically still running.
And so let's quickly fix that,
which is kind of easy.
So, that's a new one we haven't used yet.
Which is process.exit,
All right?
Now this process.exit is kind of an aggressive way
of stopping an application
but in this case it's no problem
because it's really just a very small script
that we're running here and not a real application, right?
Let's just copy the same thing here
into our import data function.
And so now I'm gonna quit it.
And just to show that it works
I'm gonna run it again
so data successfully deleted
and then it exited the process.
All right.
And so now it's time to actually run the function
that we were interested in in the first place.
So with the import flag, basically.
So the import option.
So let's run that
and let's see if it actually works.
And it didn't.
So why is that?
So it tells us here a tour must have a group size.
So where's that coming from?
And we see a lot of validation errors here.
So something must have gone wrong here.
So yeah, we have the cover image,
we have the tour description.
We have the price.
Well, that's weird.
Max group size,
let me actually check if anything happened here.
Actually we have nine tours here
and I think nine is actually all we have.
And so to me it kind of looks like it actually did work.
But let's actually take a quick look at our data here.
So, just to figure out why this actually happens.
So tour simple here
and so we can already see,
actually the problem that is happening.
So we have all these tours
so the original ones.
But then from the last section,
we have these three here
that we kind of added using our file based API.
So, remember that?
So back then we only specified the name, duration
and difficulty and nothing else.
And so right now
our script is trying to import these three tours.
But of course, we're not interested in them at all
and so let's go ahead and save this here, okay.
Then quit this process
and so actually this process.exit
can be outside of the try catch block
and be simply here by the end of the function.
So that no matter if there's an error or not
it will always just exit the process.
All right.
So let's delete everything.
All right, clear the console again.
Now import,
and yeah.
So data successfully loaded.
As indeed here we are again.
So now it's working 100%,
we have our data that we can start working with now
and so, yeah.
Our work with this one here is done.
So, a nice little function,
or a nice little script actually.
I hope that everything made sense to you
and yeah that it was kind of a fun exercise for you as well.
Anyway, see you in the next video
where we are finally starting to then use all this data
and to improve our API
by implementing a couple of nice features.
5.0 / 5 (0 votes)