How I get Tweet data for FREE in 2024 as a data scientist

Summary

In this tutorial, the presenter demonstrates how to scrape Twitter (now X) data in 2024, even after Elon Musk's changes to the platform. Using the `twkit` package, the script shows how to authenticate, retrieve tweets based on queries, and handle rate-limiting issues. The tutorial emphasizes creating secondary accounts, saving cookies for repeated access, and using delays to simulate human-like behavior and avoid bans. Finally, it covers saving tweet data into CSV files and handling complex queries, offering a free alternative to paid scraping services.

Takeaways

• 😀 **Install TKit**: The script uses the `TKit` package for scraping Twitter data, and it's installed using the command `pip install TKit`.
• 😀 **Set Up Authentication**: A secondary Twitter account is recommended to avoid risking your primary account. Authentication is done with credentials stored in a `config.ini` file.
• 😀 **Use Cookies to Persist Login**: After logging in with credentials, cookies are saved to avoid frequent logins and to keep the session active.
• 😀 **Simple Data Scraping**: Initially, tweets are scraped using a simple query (`chat GPT`), which returns a basic set of data like tweet text, user, retweets, and likes.
• 😀 **Handling Rate Limits**: The script includes error handling to pause and wait when rate limits are hit, ensuring the account doesn't get banned.
• 😀 **Simulate Human Behavior**: Random delays (between 5-10 seconds) are added before making requests to mimic human behavior and avoid triggering bans from Twitter.
• 😀 **Store Data in CSV**: Instead of printing the tweet data, it’s saved into a CSV file for easier analysis and storage.
• 😀 **Pagination Handling**: When fetching tweets, the script ensures to collect more data by handling pagination (next set of results) until the desired number of tweets is collected.
• 😀 **Rate Limit Handling**: The script manages rate limits by checking the reset time and waits until it’s safe to continue scraping, avoiding account bans.
• 😀 **Advanced Query Creation**: Users can craft complex queries using Twitter's advanced search syntax to filter tweets by user, language, date, and more, enabling refined data collection.
• 😀 **Practical Use Case**: The script enables users to freely scrape data from Twitter in 2024 without needing to pay for expensive API access, though caution is needed to avoid getting banned.

Q & A

Why is it more difficult to scrape Twitter data in 2024?

In 2024, scraping Twitter data has become more difficult due to Twitter's new policies under Elon Musk's ownership. Twitter has actively tried to limit scraping through various measures, including making it harder to access data for free.

What is the purpose of using the T-kit package in this script?

The T-kit package is used to interact with Twitter's website and retrieve tweet data. It helps simplify the scraping process and provides methods for handling rate limit errors and authentication.

What precautions should be taken when scraping Twitter data?

It is recommended to use a secondary Twitter account for scraping to avoid risking your primary account. Additionally, managing rate limits and implementing delays between requests can help prevent account bans.

How does the script handle authentication and login?

The script authenticates using a username, email, and password. After logging in, it saves session cookies in a JSON file. For future requests, these cookies are loaded to avoid repeated logins.

What are the benefits of saving cookies for future sessions?

Saving cookies allows the script to bypass repeated login attempts, which reduces the chances of being flagged by Twitter for suspicious activity. It ensures smooth access to Twitter data for subsequent scraping sessions.

How does the script manage rate limit exceptions from Twitter?

The script handles rate limit exceptions by detecting the `TooManyRequests` error. It calculates the time to wait until the rate limit is reset and pauses the script using `time.sleep()` to avoid hitting the API too frequently.

Why is the `min_tweets` variable important in the script?

The `min_tweets` variable sets the minimum number of tweets the script should scrape. It ensures that the script continues scraping until the desired number of tweets is reached, even if the first search results are insufficient.

What does the `random.randint()` function do in the script?

The `random.randint()` function generates a random number between a specified range (e.g., 5 to 10 seconds) and is used to introduce delays between scraping requests. This simulates more human-like behavior to avoid getting banned.

How does the script ensure that the scraping process doesn't get flagged by Twitter?

To avoid being flagged by Twitter, the script introduces random delays between requests and handles rate limits gracefully. It also scrapes data in a controlled manner, making requests intermittently rather than all at once.

How can users modify the query for more complex searches on Twitter?

Users can modify the query by using Twitter's advanced search feature to define specific parameters like keywords, user accounts, date ranges, and tweet types. The generated query can then be copied and pasted into the script for more refined searches.