Importing Development Data | Lecture 92 | Node.JS 🔥

Summary

In this instructional video, the presenter takes a break from API development to demonstrate a script that imports tour data from a JSON file into a MongoDB database. The script operates independently from the express application and includes steps to connect to the database, read the JSON file, and utilize the tour model for data import. Additionally, the video covers command-line interaction, allowing users to run the script with options to import or delete data. The presenter addresses potential validation errors and concludes with a successful data import, setting the stage for future API enhancements.

Takeaways

• 🛠️ The script's purpose is to import tour data from a JSON file into a MongoDB database, separate from the main Express application.
• 📝 The script is created in a 'data' folder and named 'importDevData.js', emphasizing its standalone nature.
• 📚 Mongoose is required for database interactions, and environment variables are needed for database connection details.
• 🔌 The script requires the file system module 'fs' to read the JSON file and the 'tour' model to interact with the database.
• 📖 The JSON file is read synchronously with a specified encoding of 'utf eight'.
• 🔍 An async function is used to import data into the database, utilizing 'tour.create' to handle an array of objects.
• 🗑️ There's a function to delete all data from the MongoDB collection using 'deleteMany', a method provided by Mongoose.
• 📋 The script can be run with command-line options to either import or delete data, demonstrated by checking 'process.argv'.
• 🔄 The script includes error handling with try-catch blocks to log issues during data import or deletion.
• 🚫 Validation errors are encountered due to incomplete data in the JSON file, highlighting the importance of data integrity.
• 🔚 The script exits the process after completing its task using 'process.exit', ensuring the script doesn't continue running unnecessarily.

Q & A

What is the purpose of the script mentioned in the video?

The script's purpose is to import tour data from a JSON file into a MongoDB database, which is a standalone task separate from the main Express application.

Why was the decision made to include the script-building exercise in the course?

The script-building exercise was included because it was considered a nice little exercise that adds value to the course, despite the possibility of just providing the script in the starter files.

What does the script need to perform its task?

The script requires mongoose for database interaction, the .env package for environment variables, the file system module to read the JSON file, and the tour model to write the data into the database.

How does the script handle reading the JSON file?

The script uses the synchronous version of `fs.readFile` to read the JSON file named 'toursSimple.json' and specifies the encoding as 'utf eight'.

What is the significance of the 'importData' function in the script?

The 'importData' function is an asynchronous function that imports data into the database by using the 'tour.create' method with an array of JavaScript objects obtained from parsing the JSON file.

How can the script handle the deletion of existing data in the database?

The script includes a function to delete all data from the database using the 'delete many' function of the tour model, which removes all documents in the tours collection.

What is the role of 'process.argv' in the script?

'process.argv' is used to interact with the command line, allowing the script to accept arguments that determine whether to import or delete data from the database.

Why is the 'process.exit' function used in the script?

The 'process.exit' function is used to stop the script execution after completing the import or delete operation, as it is a small standalone script and not part of a larger application.

What issue occurred when the script was first run to import data?

The script encountered validation errors because the JSON file contained some tours with incomplete data, specifically missing 'group size' and other required fields.

How can the script be improved to avoid importing incomplete data?

The script can be improved by adding checks to ensure that all required fields are present in the JSON data before attempting to import it into the database.

What is the final step in the script after successfully importing the data?

The final step is to log a success message to the console indicating that the data has been successfully loaded into the database and then exit the process using 'process.exit'.

Outlines

00:00

🛠️ Building a Data Import Script for MongoDB

The script's purpose is to import tour data from a JSON file into a MongoDB database, operating independently from the main Express application. The author considers whether to include this as an exercise or provide it pre-built but decides to include it for educational value. The process involves creating a script in the data folder named 'importDevData.js', setting up necessary modules including mongoose for database connection, the 'fs' module for file system access, and the tour model for data insertion. The script reads a JSON file synchronously and includes error handling. It demonstrates the use of 'tour.create' to insert data into the database, converting JSON data to JavaScript objects with 'JSON.parse'.

05:01

🗑️ Deleting Existing Data and Command Line Interaction

The script also features a function to delete all existing data from the database using 'deleteMany', which removes all documents in the collection. The author discusses the use of 'process.argv' to interact with the command line, allowing the script to be run with different options to either import or delete data. The script is tested by running it with node and observing the output of 'console.log(process.argv)'. Adjustments are made to correctly specify file paths using the 'dirname' variable, and the script is enhanced to handle command line arguments for importing or deleting data.

10:03

🔧 Refining the Script with Command Line Options

The script is refined to check command line arguments and execute the appropriate function based on the user's input. If the '--import' option is provided, the import function is executed; if the '--delete' option is used, the delete function is triggered. The author demonstrates running the script with the delete option, which successfully removes all data from the database, and then attempts to run it with the import option. However, an error occurs due to validation errors related to missing 'group size' in the tour data. The author investigates and identifies that the issue stems from incomplete data entries added in a previous section of the course.

15:04

🎉 Successful Data Import and Wrapping Up

After addressing the data validation issues by removing incomplete tour entries, the author successfully imports the data into the database using the script with the import option. The script includes 'process.exit' to terminate the process after completing the operation. The author concludes the tutorial by expressing satisfaction with the script's functionality and looks forward to the next video, where the focus will be on enhancing the API with new features using the imported data.

Keywords

💡API

API stands for Application Programming Interface, which is a set of rules and protocols for building and interacting with software applications. In the context of the video, the script is being developed to interact with an API, specifically for importing tour data into a MongoDB database. The script is part of a larger project to build an Express application, which is a type of server-side software framework that can be used to create APIs.

💡JSON

JSON, or JavaScript Object Notation, is a lightweight data-interchange format that is easy for humans to read and write, and for machines to parse and generate. In the video, the script is designed to import data from a JSON file, which contains the tour information that needs to be loaded into the MongoDB database.

💡MongoDB

MongoDB is a popular NoSQL database that stores data in a flexible, JSON-like format. The script mentioned in the video is used to interact with a MongoDB database, specifically for the purpose of importing tour data from a JSON file into the database.

💡Mongoose

Mongoose is a MongoDB object modeling tool designed to work in an asynchronous environment. It includes built-in methods for creating, querying, updating, and deleting data in MongoDB. In the video, Mongoose is used to connect to the MongoDB database and to create or delete documents within the 'tours' collection.

💡Express

Express is a minimal and flexible Node.js web application framework that provides a robust set of features for web and mobile applications. Although not the main focus of the script in the video, the script is part of a larger Express application, and the process of importing data is independent of the Express application's main functionality.

💡Environment Variables

Environment variables are a set of dynamic values, set outside the application, that can affect the way running processes will behave on a computer. In the video, the script requires access to environment variables to connect to the database, which is a common practice for managing sensitive information like database credentials.

💡File System Module

The file system module in Node.js provides an API for interacting with the file system. In the script, the file system module is used to read the JSON file containing the tour data that needs to be imported into the MongoDB database.

💡Asynchronous Function

An asynchronous function is a function that can operate independently of the flow of the application, allowing other code to run while waiting for the function to complete. In the video, the script uses asynchronous functions to handle database operations, such as creating documents or deleting data, without blocking the execution of the script.

💡Try-Catch Block

A try-catch block is used in JavaScript to handle errors by allowing developers to write code that attempts to execute a block of code (the 'try' block) and, if an error occurs, handle it in the 'catch' block. In the script, a try-catch block is used to log errors to the console during the data import process.

💡Command Line

The command line is a text-based interface for interacting with a computer's operating system. In the video, the script is run from the command line, and the script's functionality is controlled by command line arguments, such as specifying the 'import' or 'delete' options.

💡Process.argv

Process.argv is an array in Node.js that contains the command-line arguments passed to the script. In the video, process.argv is used to determine which action the script should perform based on the arguments provided when the script is run from the command line.

Highlights

Building a script to import tour data from a JSON file into MongoDB database.

The script is independent of the express application and will be run separately.

Using mongoose for database connection and environment variables from .env package.

Accessing the file system module to read the JSON file.

Importing the tour model to write tours data into the database.

Reading the JSON file synchronously with fs.readFile.

Creating an async function to import data into the database using tour.create.

Using JSON.parse to convert JSON data into JavaScript objects.

Creating a function to delete all data from the collection using deleteMany.

Mongoose's deleteMany method is similar to MongoDB's native function.

Interacting with the command line to run functions based on user input.

Using process.argv to access command line arguments.

Creating a command line application to import or delete data based on options.

Using process.exit to stop the application after running the desired function.

Handling validation errors due to incomplete tour data.

Ensuring that the script exits properly after importing or deleting data.

Finalizing the script with a successful data import into MongoDB.

Transcripts

Let's now take a small break

from building our API

and build a fun little script

that will import the tour data from our JSON file

into the MongoDB database.

And I wasn't really sure

if I should include this video in the course

or if I should just come with the script

already in the course starter files.

But I thought I thought it was a nice little exercise

and so I included it anyways.

So, anyway, let's now get started.

So, basically we're gonna create a script

that will simply load the data from the JSON file

as I just said into the database.

And this script is completely independent

of the rest of our express application.

And so we'll run this completely separately

from the comment line just to import everything once.

Okay, so let me actually create the script

right here in the data folder.

So new file,

import,

dev data.js.

All right.

So, what are we gonna need for this?

We will need mongoose of course

so let's start with that.

Or actually, let's just go ahead into the server.js file

and copy this stuff

because why writing it all over again.

So let's copy everything

and then delete what we don't need.

So we don't need our express application

we also need the .env package

because we need our environment variables

in order to be able to connect to the database again, okay.

And we need to connect to the database in this script

again because it runs completely independent

from the express application.

It's only gonna run once in the beginning.

Next up, we need access to the file system module

because of course we want to read the JSON file.

So require fs

and finally we also need access to the tour model

because the tour model is

where we want to write the tours to, right?

So, tour equals

and now let's find the path there

so from the place where we are right now

we need to go up one level.

And what's going on here with these quotes?

All right, so one level up

and we're in dev data.

So we need another level up

so that we're in the main folder.

And from there we go in to models

and into the tour model.

Okay and that should be it for setup.

Now let's start by reading the file.

So, read JSON file

and that should be fairly simple.

So the tours are at fs.readfile

and we can use the synchronous version of course.

And let's simply say tours simple.json, okay.

And then,

also the encoding.

So, file encoding, utf eight.

Give it a save.

And so now we can write the actual function

that is gonna import the data into the database.

So import data into database.

Okay, just like this.

And so let's create this function.

Import data, import data,

yep.

And that's gonna be and async function

which does not need any arguments

and so let's again use a try catch block here.

And here I'm simply gonna log it to the console

if there is some error,

just to know what's going on in that case.

And now here,

what we will do is very simple.

We are simply gonna await,

tour.create.

So we already used tour.create

and we pass then an object back then right?

But the create method can also accept an array of objects.

And in that case

it will then simply create a new document

for each of the objects in the array.

So, very simple,

all we have to do is to specify our tours data here, right?

And actually it's not 100% correct

because remember that this is JSON.

And so we need to first convert it actually

into a JavaScript object using json.parse.

Okay and so now we actually have

an array of JavaScript objects

that we can now pass into the create method, okay.

And if that was successful,

then the next line is gonna be executed

and so here we can say data successfully

loaded, all right.

And this should already do the job.

Now what about the data that is already in the database?

We can also create an easy way

to basically delete all of that data at the same time.

And so let's simply go ahead and do that as well.

So, delete all data

from collection, let's say.

And this weird yellow color that you see here

actually comes from an extension that I have installed here.

And so to get rid of that,

I'm simply gonna write database again, okay.

So delete data

and again this is gonna be an async function

without any arguments.

And let me actually copy this code here.

So, delete it

and now about the deleting itself

we can use the delete many function.

Okay and actually I showed you this one

back in the intro to MongoDB, right?

Where we could use delete many

and then simply pass in nothing in there

and that would then delete

all of the documents in a certain collection, right?

And so mongoose basically implemented

the same function here on the model, okay.

So in this case,

the tour model has access to this delete many method

which will then do exactly the same

as delete many does in native MongoDB, right?

So, remember that mongoose is just a,

like a layer of abstraction on top of MongoDB.

Which is why it doesn't use the exact same functions

but it still gives us access to some similar ones

or to ones that actually have the same name.

So delete many actually has the same name

as the native MongoDB function, all right?

So again, what this is gonna do is to simply go ahead

and delete all the documents in the tours collection.

So, we have our two functions here

but if we now actually run this file

then nothing will happen.

And that's because

we're not calling any of these functions anywhere, right?

Now we could go ahead

and simply write something like import data here

and then simply call the function here

but I wanted to make this a little bit more fun.

So let's now actually learn a tiny little bit

about interacting with the command line, okay.

And so I'm actually gonna go ahead

and run this file without calling any of these functions.

But instead I'm gonna log to the console

process.argv,

okay.

Just so we can see what process.argv actually is

so that we can then use it.

All right.

Let me open up here another terminal

and then I will use node,

go into dev data

then into data

and then in there import dev data.

And we get some errors here.

Let's see where.

Ah, yeah so it's because of this,

of this file name.

So I guess we should specify the path to there basically.

So let's run this one again

and again we have this error.

And yeah, of course I get this error.

That's a stupid one.

Remember how I told you

that this dot here is always relative from the folder

where the node application was actually started.

And so that's the home folder.

And so we're basically looking for this file here

in the home folder, okay.

So what I should use instead is the dir name,

variable that is available to us everywhere.

So that goes like this.

All right.

Give it another save

and clear up the console and run it again.

And so now it works.

And so here

is the result of this console.log that we have down here,

So process.argv

and basically that is an array

of these two arguments of running this node process.

So, this here is basically

where the node command is located.

So this equivalent to this node

and then the second one,

so this path to this file is actually this here, okay.

So let's quit this here

and let's add kind of an option here.

So I'm gonna write,

dash, dash import

and so I'm sure you have seen something like this

many times before.

For example, when we save a package as a dev dependency

we do it like this.

Save dev, and so we use the same

kind of format for specifying options.

Okay so, dash dash

and then whatever string we put here.

And so I choose to basically specify

the import option like this.

And so you see that now the third argument

is dash dash import, all right?

And so that means that we can now go ahead

and basically use this data here

in order to write

a very simple command line application basically

which will import the data when we specify this option

and will delete the data

when we specify the delete option, all right?

So, let's actually do that.

So, if

process.argv

and it's an array

and we want the third.

So zero, one, two.

So if that element is equal to import

well, then we want to run import data.

Right?

If...

Process.argv two

is equal,

to delete then we want to run

delete data.

And that's it.

So, that should actually give us

the result that we're looking for.

Let's finish this here.

And so now let's run the command here

with delete in order to delete all the data

that we have in the database.

So let's try that out.

It's doing something

and data successfully deleted.

So let's take a look at that now.

And if we run now this get all tours route

then indeed we have zero results.

So, all our tours are now gone.

So it worked.

Now this process here is basically still running.

And so let's quickly fix that,

which is kind of easy.

So, that's a new one we haven't used yet.

Which is process.exit,

All right?

Now this process.exit is kind of an aggressive way

of stopping an application

but in this case it's no problem

because it's really just a very small script

that we're running here and not a real application, right?

Let's just copy the same thing here

into our import data function.

And so now I'm gonna quit it.

And just to show that it works

I'm gonna run it again

so data successfully deleted

and then it exited the process.

All right.

And so now it's time to actually run the function

that we were interested in in the first place.

So with the import flag, basically.

So the import option.

So let's run that

and let's see if it actually works.

And it didn't.

So why is that?

So it tells us here a tour must have a group size.

So where's that coming from?

And we see a lot of validation errors here.

So something must have gone wrong here.

So yeah, we have the cover image,

we have the tour description.

We have the price.

Well, that's weird.

Max group size,

let me actually check if anything happened here.

Actually we have nine tours here

and I think nine is actually all we have.

And so to me it kind of looks like it actually did work.

But let's actually take a quick look at our data here.

So, just to figure out why this actually happens.

So tour simple here

and so we can already see,

actually the problem that is happening.

So we have all these tours

so the original ones.

But then from the last section,

we have these three here

that we kind of added using our file based API.

So, remember that?

So back then we only specified the name, duration

and difficulty and nothing else.

And so right now

our script is trying to import these three tours.

But of course, we're not interested in them at all

and so let's go ahead and save this here, okay.

Then quit this process

and so actually this process.exit

can be outside of the try catch block

and be simply here by the end of the function.

So that no matter if there's an error or not

it will always just exit the process.

All right.

So let's delete everything.

All right, clear the console again.

Now import,

and yeah.

So data successfully loaded.

As indeed here we are again.

So now it's working 100%,

we have our data that we can start working with now

and so, yeah.

Our work with this one here is done.

So, a nice little function,

or a nice little script actually.

I hope that everything made sense to you

and yeah that it was kind of a fun exercise for you as well.

Anyway, see you in the next video

where we are finally starting to then use all this data

and to improve our API

by implementing a couple of nice features.