How to Scrape LinkedIn Without Paid APIs Using n8n

Summary

In this video, the creator demonstrates how to scrape LinkedIn profiles for lead generation without paying for APIs. By using Google’s Custom Search API, the script searches for specific profiles (like CEOs in the automotive industry) and collects key details—name, title, LinkedIn link, and a profile snippet—before adding them to Google Sheets. The process includes handling pagination to gather multiple pages of results and managing API rate limits. The method is free to use within API limits, with the potential to scale by upgrading the API tier. This tutorial is aimed at automating LinkedIn lead collection for outreach.

Takeaways

• 😀 You can scrape LinkedIn profiles for lead generation without paying for APIs by using Google’s Custom Search API.
• 😀 The method involves setting up a custom search engine in Google to target specific profiles, such as CEOs in a given industry and location.
• 😀 Google’s Custom Search API returns only 10 results per request, but you can handle pagination by adjusting the start index to collect more results.
• 😀 By using an automation tool like Make.com, you can automate the entire process, pulling LinkedIn profiles and saving them directly to a Google Sheet.
• 😀 The free tier of Google’s Custom Search API provides 100 requests per day, meaning you can gather up to 1,000 leads daily (10 results per request).
• 😀 It’s important to manage API rate limits by adding delays (e.g., 5 seconds) to avoid exceeding Google’s daily usage limits.
• 😀 You can adjust your search parameters to focus on very specific niches, and the script will automatically stop when there are no more results to fetch.
• 😀 To set up your Google API credentials, you need to create a project in Google Cloud Console, enable the Custom Search API, and generate an API key.
• 😀 After setting up the API and automation, you can test the workflow to ensure leads are properly pulled and added to your Google Sheet.
• 😀 This scraping method is scalable, and you can increase the number of results by upgrading your Google API tier, but the free plan gives you enough to generate a significant volume of leads.

Q & A

What is the main purpose of the script in the video?

The script demonstrates how to scrape LinkedIn profiles using a Google search API to generate leads without paying for any APIs. It explains how to automate the process of collecting leads' names, titles, profile links, and snippets from LinkedIn, which can be stored in a Google Sheet.

How does the Google search API help in scraping LinkedIn profiles?

The Google search API is used to perform targeted searches on LinkedIn by specifying criteria like job title and industry. This allows the script to extract relevant LinkedIn profiles based on the search results and collect specific data such as names, titles, and profile links.

What is the role of the Google Sheets integration in the automation process?

The integration with Google Sheets allows the script to store the scraped LinkedIn leads in a structured format. It adds the data—such as names, titles, profile links, and snippets—into a spreadsheet, making it easy for users to track and manage the leads.

How does the script manage pagination when retrieving LinkedIn leads?

The script manages pagination by using a dynamic 'start' index that increments with each API call. This ensures that after retrieving 10 results, the script fetches the next 10 results until all the relevant leads are collected, up to a maximum of 100 results per search query.

What are the key parameters used in the Google search API call?

The key parameters used in the API call include the API key, the custom search engine ID (CX value), the search query (e.g., CEOs in the automotive industry), and the 'start' value which determines which page of results to fetch.

What are the limitations of the free tier of the Google search API?

The free tier of the Google search API allows up to 100 requests per day, with each request returning 10 search results. This means users can retrieve a maximum of 1,000 LinkedIn leads per day without incurring any cost.

How does the script handle API rate limits and delays?

To prevent hitting API rate limits, the script includes a delay of 5 seconds between requests. This helps avoid hitting the Google search API and Google Sheets API limits, ensuring smooth operation during the lead collection process.

What is the significance of the 'start index' in the pagination process?

The 'start index' determines where the search results begin in each API call. It starts at 1 for the first API request and increments by 10 with each subsequent request to fetch the next page of results, ensuring that the script retrieves the next set of LinkedIn profiles.

What credentials are needed to set up the Google search API?

To set up the Google search API, users need to create a project in the Google Cloud Console, enable the Custom Search API, generate an API key, and create a programmable search engine. The API key and the search engine's CX value are then used to authenticate and make requests to the Google search API.

How can users scale up their lead generation process beyond the free tier of the Google search API?

Users can scale up by upgrading to a paid tier of the Google search API, which increases the number of requests and results they can generate daily. This allows for more extensive lead generation without hitting daily limits.