Web Scraping Tutorial | Data Scraping from Websites to Excel | Web Scraper Chorme Extension

Summary

In this tutorial video, Rafi demonstrates how to use a free Google Chrome extension called 'Web Scraper' to extract data from multiple web pages automatically. He provides a step-by-step guide on scraping information from the Yellow Pages business directory, focusing on car insurance service providers in New York City. The data collected includes business names, phone numbers, addresses, websites, and email addresses. The video also covers navigating pagination and setting up selectors for efficient data extraction.

Takeaways

• 💻 The video demonstrates how to scrape data from websites using a free Google Chrome extension called Web Scraper.
• 🏢 The target data source is the Yellow Pages business directory, specifically gathering information about car insurance service providers in New York City and State.
• 📊 The scraping process involves collecting details like business name, phone number, address, website, and email address from multiple pages.
• 🔄 Web Scraper automates the process by moving from one page to the next after scraping 30 results per page.
• 🛠️ To begin, users need to install the Web Scraper extension from the Chrome Web Store and then reload the target website.
• 🖱️ Using 'Inspect Element,' users can identify and create a sitemap with selectors to scrape specific information from the webpage.
• 🌐 The tutorial includes selecting business listings, extracting information like name, phone number, and website, and handling multi-page navigation for continuous scraping.
• 📈 The tool allows users to adjust scraping intervals to avoid hitting website restrictions or being blocked.
• 📥 After completing the scraping process, users can export the gathered data into a CSV file for further use and cleaning.
• 🔧 The video emphasizes the importance of cleaning data post-extraction, such as removing unnecessary text (e.g., 'mailto:') from email addresses.

Q & A

What is the main topic of the video?

The main topic of the video is how to scrape data from websites using a free Google Chrome extension called Web Scraper.

What specific information is the presenter going to extract from the Yellow Pages business directory?

The presenter is going to extract car insurance service providers' information from New York City and State, including business profiles' names, phone numbers, addresses, website addresses, and email addresses.

How does the tool handle pagination on the website?

The tool automatically visits subsequent pages after completing the data extraction from the first page, continuing to scrape data from each page.

What is the name of the Google Chrome extension used in the video?

The Google Chrome extension used in the video is called 'Web Scraper'.

How does one install the Web Scraper extension on Google Chrome?

To install the Web Scraper extension, one needs to visit the extension page, click on 'Add to Chrome', and then confirm by clicking 'Add extension'.

What is a sitemap in the context of web scraping with the Web Scraper extension?

A sitemap in the context of web scraping with the Web Scraper extension is a configuration that defines how the tool navigates and extracts data from a website.

How does the presenter select the data points to be scraped from each business listing?

The presenter selects data points by clicking on 'Add new selector', choosing the type (text or link), and then selecting the specific elements on the webpage such as business name, phone number, address, website, and email.

What is the purpose of setting a delay between requests when scraping?

Setting a delay between requests prevents the scraper from being blocked by the website due to too many rapid requests, as most websites have limitations on the number of accesses per user per day.

How can one export the scraped data from the Web Scraper extension?

The scraped data can be exported by clicking on the 'Export data' button and then choosing 'Export data as CSV' to download the data into an Excel document.

What is the final format of the scraped data as mentioned in the video?

The final format of the scraped data is a CSV file containing the information such as business or person's name, phone number, address, website, and email.

How does the presenter clean the extracted email addresses in the CSV file?

The presenter cleans the extracted email addresses by using the 'Find and Replace' feature in Excel to remove the 'mailto:' prefix from each email address.