Scraping Dark Web Sites with Python

Summary

In this video, the speaker demonstrates how to automate interactions with websites on the dark web using Tor and Python. They show how to install and configure Tor on a Kali Linux virtual machine, use the torify command to tunnel traffic, and access onion sites with curl. The video also explains creating a Python script with the requests-tor library to scrape dark web data. Additionally, they highlight tools like Flare for monitoring cyber threats and the dark web. The speaker aims to educate viewers on tracking cybercrime and automating data collection from the dark web.

Takeaways

• 🌐 The video discusses automating interactions with websites on the dark web using Tor and .onion addresses.
• 🛠️ The presenter demonstrates installing Tor on a Kali Linux virtual machine and using the 'torify' command to tunnel traffic through the Tor network.
• 🔒 The importance of configuring the Tor control port for secure communication with the Tor service is highlighted, including enabling authentication methods.
• 📝 The script shows how to modify the Tor configuration file to enable the control port and set it up for cookie authentication.
• 🔄 The presenter explains how to restart the Tor service after configuration changes and verify the new IP address through Tor.
• 🕵️‍♂️ The video mentions using Tor for threat intelligence gathering, tracking cybercrime, and understanding the activities of threat actors on the dark web.
• 🛑 The use of the 'requests-unixsocket' library in Python is introduced to automate HTTP requests through Tor.
• 🤖 An example Python script is provided to demonstrate how to scrape content from .onion websites using Tor.
• 🔎 The video showcases the use of tools like Flare for cyber threat intelligence and attack surface management, emphasizing the value of tracking threat actors and ransomware groups.
• 📈 The presenter discusses the potential for using Tor to scrape and monitor changes on dark web marketplaces, forums, and leak sites for intelligence purposes.
• 🔗 The script concludes with a mention of various resources and libraries for further exploration of Tor usage in Python and command-line tools.

Q & A

What is the main purpose of the video?

The main purpose of the video is to demonstrate how to automate interactions with websites on the dark web using tools like Tor, Curl, and Python.

Why does the speaker use a Kali Linux virtual machine?

The speaker uses a Kali Linux virtual machine because it is a popular environment for cybersecurity and penetration testing, providing necessary tools for the demonstration.

What is the command to install Tor as a service in Kali Linux?

The command to install Tor as a service in Kali Linux is `sudo apt install tor`.

What is the purpose of the 'torify' command?

The 'torify' command is used to wrap other commands and tunnel their traffic through the Tor network.

Why does the speaker modify the Tor configuration file?

The speaker modifies the Tor configuration file to enable the control port and authentication, which is necessary for tunneling traffic and automating interactions with Tor.

How can you verify that your IP address is routed through Tor using Curl?

You can verify that your IP address is routed through Tor using Curl by running the command `torify curl ifconfig.me` to see the IP address that Curl reports.

What are the two main ports used by Tor and what are their purposes?

The two main ports used by Tor are 9050 (for the Socks proxy) and 9051 (for the control port). The Socks proxy port is used for routing traffic through Tor, and the control port is used for configuration and management of the Tor service.

What Python library does the speaker install to make requests through Tor?

The speaker installs the `requests[socks]` library in Python to make requests through Tor.

How does the speaker automate accessing a dark web URL in Python?

The speaker automates accessing a dark web URL in Python by using the `requests` library with the Tor proxy settings, making a GET request to the URL through the Tor network.

What kind of information can be gathered from dark web scraping according to the speaker?

Information that can be gathered from dark web scraping includes threat intelligence, cyber crime activities, ransomware updates, leaked credentials, personal identifiable information (PII), and other cyber threats.

What tool does the speaker mention for tracking cyber threats and managing attack surfaces?

The speaker mentions 'Flare' as a tool for tracking cyber threats and managing attack surfaces, providing visibility into various threats and vulnerabilities.