Find endpoints in the blink of an eye! GoSpider - Hacker Tools

Summary

In this hacker tools video, the presenter introduces Go Spider, a powerful web crawling tool designed to discover endpoints, subdomains, and other resources on a website. It efficiently scans web pages, identifies links, and can even recursively crawl through found files. The tool offers customization options like setting user agents, cookies, headers, and managing request speed to comply with platform rules. Advanced features include utilizing third-party archives and filtering results based on file length or extensions, making it an essential tool for initial target enumeration in cybersecurity assessments.

Takeaways

• 🕷️ Go Spider is a tool designed for web scraping and crawling web pages to discover various endpoints, subdomains, and other assets.
• 🔍 It operates by requesting a web page and then searching for links, JavaScript files, directories, subdomains, and endpoints, presenting a comprehensive map of the target's web presence.
• 🔄 The tool can recursively crawl through discovered files to uncover even more links and resources, creating a detailed web of the application's structure.
• ⚙️ Basic usage involves running Go Spider with the '-s' option to specify a URL, '-o' for output file, and '-c' for setting the number of concurrent requests.
• 🔑 'Big bounty parameters' like '-u' for user-agent and '-h' for custom headers can help comply with platform rules during scanning.
• 🚀 The tool can be configured for speed with parameters like '-threads' for setting the number of threads and '-concurrent' for concurrency level.
• 🛑 To avoid overwhelming targets, use '-k' or '-delay' to set a delay between requests, ensuring you stay within acceptable request limits.
• 🗂️ Advanced features include crawling JavaScript files with '-js', including subdomains with '-subs', and utilizing sitemaps and robots.txt with '-sitemap' and '-robots' respectively.
• 🔎 Go Spider can integrate with third-party archives like Common Crawl and Wayback Machine using '-h' or '-other-source' to find URLs from historical data.
• ⛔️ Use '-blacklist' with regex to exclude specific results or '-whitelist' to focus only on desired outcomes.
• 📊 Filtering options like '-l' or '-length' allow for the exclusion of certain file types or HTTP status codes to refine the scan results.

Q & A

What is Go Spider and what does it do?

Go Spider is a tool that spiders web pages to crawl them and extract information such as endpoints, subdomains, and other links. It can also recursively crawl the discovered files to create a comprehensive web of the application's structure.

How does Go Spider perform a basic scan?

To perform a basic scan, Go Spider is run with the '-s' option to input a URL, '-o' for output to specify an output file, and '-c' for concurrency to set the number of concurrent requests.

What are 'big bounty parameters' in Go Spider and why are they important?

'Big bounty parameters' refer to options like user-agent, cookies, and headers that can be set with Go Spider to adhere to the rules of a platform and ensure ethical hacking practices.

How can Go Spider be configured to respect the speed limits of a target platform?

Go Spider allows setting the number of threads with '-d', concurrency with '-c', and delay between requests with '-k' to control the speed and avoid overwhelming the target platform.

What additional features does Go Spider offer beyond basic crawling?

Go Spider can find JavaScript files, include subdomains, crawl sitemaps, and utilize third-party archives like Common Crawl and Wayback Machine for more extensive data collection.

How can Go Spider be used to filter out unwanted results during a scan?

Go Spider provides options like '-h' for excluding specific sources, '-l' to view file lengths, and '-f' to filter out specific lengths or extensions to refine the scan results.

What is the purpose of the '-blacklist' option in Go Spider?

The '-blacklist' option allows users to supply a regex pattern to exclude results that match it, helping to focus on relevant data during a scan.

Can Go Spider handle multiple URLs at once?

Yes, Go Spider can handle multiple URLs by using the '-s' option with a file that contains multiple links, allowing for batch processing of URLs.

How does Go Spider help in the initial enumeration of targets?

Go Spider assists in the initial enumeration by mapping out the target's web structure, identifying running services, and providing a comprehensive overview of what's present on the target's web pages.

What is the recommended next step after using Go Spider for initial scanning?

After the initial scan, the recommended next step is to analyze the results, identify important targets, and proceed with more focused and in-depth security testing.

Outlines

00:00

🕷️ Introduction to Go Spider

This paragraph introduces Go Spider, a tool designed for web crawling and data extraction. It explains how Go Spider can spider web pages to discover endpoints, subdomains, and other valuable information. The tool is capable of recursively crawling through files and directories to create a comprehensive map of a web application. The video demonstrates a basic scan using Go Spider with the '-s' option for specifying a URL, '-o' for output file, and '-c' for setting the number of concurrent requests. The tool's ability to quickly find a wealth of information is highlighted, emphasizing its power in web scanning.

05:01

🛠️ Advanced Features and Filtering with Go Spider

The second paragraph delves into the advanced features of Go Spider, including parameters that can be set to comply with platform rules, such as user-agent and cookie settings. It discusses the importance of controlling the speed of requests to avoid exceeding limits set by the target website. The paragraph also covers additional functionalities like crawling JavaScript files, sitemaps, and using third-party archives for enhanced data retrieval. The tool offers filtering options to refine results, such as excluding certain file extensions or HTTP response lengths, which is particularly useful for avoiding false positives like custom 404 pages. The video concludes by encouraging viewers to explore the tool for initial target enumeration and suggests viewers to provide feedback on future tool coverage.

Keywords

💡Go Spider

Go Spider is a web crawling tool that is central to the video's theme. It is designed to spider web pages, meaning it can traverse and index content from the web. The tool is used to discover endpoints, subdomains, and other resources linked within a web page, which is crucial for initial reconnaissance in cybersecurity assessments. The script mentions that Go Spider can be used to 'crawl web pages and to get targets endpoints, subdomains everything out of it,' showcasing its capability to create a comprehensive map of a website's structure.

💡Endpoints

Endpoints in the context of the video refer to the specific URLs or routes within a web application that can be accessed. These are important for security testing as they might reveal potential vulnerabilities. The script explains that Go Spider can identify endpoints by 'searching through that page and look for links to... other endpoints,' which is a critical step in assessing the attack surface of a web application.

💡Subdomains

Subdomains are subsets of a primary domain, such as 'blog.example.com' where 'blog' is the subdomain of 'example.com'. They are significant in web crawling as they can host different services or applications that might have distinct security configurations. The video script highlights Go Spider's ability to find subdomains, which is part of the broader process of identifying all accessible points within a web property.

💡Concurrent Requests

Concurrent requests are multiple HTTP requests sent to a server at the same time. In the video, the script discusses setting the concurrency level with the 'dash c' option, which controls how many requests Go Spider can make simultaneously. This is important for efficiency but also must be managed to avoid overwhelming the server or violating the terms of engagement with the target website.

💡User Agent

A user agent is a string that allows network protocols peers to identify the application, operating system, and version of the requesting software. In the script, it is mentioned that Go Spider can be configured with a user agent using the 'dash u' or '--user-agent' parameter. This is often necessary to comply with the rules of the platform being scanned or to bypass certain security measures that might block requests without a recognizable user agent.

💡Cookies

Cookies are small pieces of data stored on a user's computer by the web browser while browsing a website. In the context of the video, setting specific cookies is discussed as a way to customize the requests made by Go Spider using the 'cookie' parameter. This can be important for accessing pages that require authentication or maintaining a session during the crawling process.

💡Headers

HTTP headers are key/value pairs sent in an HTTP request or response that provide additional information about the request or response. The video script refers to setting specific headers with the 'dash capital h' or '--header' parameters. This can be used to mimic legitimate traffic or to provide additional information that might be required to access certain resources on the web server.

💡Blacklist

A blacklist is a list of entries that are blocked or excluded from a set of resources. In the video, the script mentions using a 'blacklist' to exclude certain URLs or patterns that match a regular expression provided by the user. This feature is useful for focusing the crawl on relevant resources and avoiding unnecessary data collection.

💡Whitelist

A whitelist, the opposite of a blacklist, is a list of entities that are allowed access or are considered safe. The script briefly touches on the concept of whitelisting, where Go Spider can be configured to only include specific resources during the crawl. This is beneficial for narrowing down the scope of the crawl to the most relevant parts of the website.

💡Sitemap.xml

A sitemap.xml file is an XML file that lists URLs for a site and allows webmasters to include additional information about each URL for search engines. In the video script, it is mentioned that Go Spider can crawl a sitemap.xml file if found. This is particularly useful for discovering all the URLs of a website in a structured manner.

💡Robots.txt

Robots.txt is a file that tells web crawlers which areas of a website should not be processed or scanned. The script refers to the option to crawl the robots.txt file with 'dash dash robots'. This file can provide insights into the parts of the website that the owners do not want to be indexed, which might be relevant for ethical web crawling practices.

Highlights

Go Spider is a tool for spidering web pages to crawl and gather information such as endpoints and subdomains.

It can request a page and search for links, JavaScript files, directories, subdomains, and endpoints.

The tool can also crawl the findings to create a web of the application's files and links.

Go Spider can be run with simple commands to perform scans and gather results instantly.

The '-s' option allows input of a URL to scan, while 'capital S' can take a file with multiple links.

The '-o' option is used for outputting results to a file.

The '-c' option sets the level of concurrency for the scan.

Parameters like 'big bounty parameters' can be set to adhere to platform rules, such as user-agent with '-u'.

The '-cookie' and '-header' options allow setting specific cookies and headers for the scan.

The '-d' and '-c' options control the number of threads and concurrency of the scan.

The '-k' option sets the delay between requests to match domains.

Go Spider can find JavaScript files, include subdomains, and crawl sitemaps and robots.txt files.

The '-h' or '--other-source' option uses third-party archives like Common Crawl and Wayback Machine to find URLs.

The '-r' or '--include' option allows crawling of web pages from archives like archive.org.

The '--blacklist' option can be used to blacklist specific items using regex patterns.

The '--whitelist' option can be used to only allow specific items during the scan.

The '-l' or '--length' option can show the length of each file found, which can be useful for filtering.

The '--filter-length' option can be used to filter out specific file lengths or HTTP status codes.

The tool provides extensive filtering options to refine the scan results.

Go Spider is useful for initial enumeration of targets to understand what they are running.

The video concludes with a call to action for viewers to comment on tools they would like to see covered in the future.

Transcripts

this

is go spider

and that's what we're gonna talk about

in today's hacker tools video

[Music]

go spider is a really cool tool that

allows us to spider web pages to crawl

web pages and to get targets endpoints

subdomains everything out of it so the

idea here is that this tool is going to

request a page and then it's going to

search through that page and look for

links to javascript files other

directories other sub domains other

endpoints anything and it's going to

show that all to us now additionally to

that you can even set it that it's also

going to crawl those findings that we

have so it finds a file and it also

crawls that file for more files and that

way you really create a web of this

application where you have all the files

and all the links between them and you

can be sure that you have almost

everything

that can be found on that web page

without having to go through it all

manually but let's take a look at how

this works by running it and doing a

very simple scan here

so what are we going to do we're going

to run go spider and then we're going to

supply the dash s option the dash

lowercase s option allows us to input a

url to

any page that we want to scan

now you can also use capital s and then

supply a file that holds multiple links

that you always can

following that argument we're gonna use

the dash o for output and we're gonna

supply an output file and then we're

gonna use this dash c lastly which is c

standing for a concurrency how many

concurrent requests are we gonna run

and that is how we can perform a simple

scan and if i press enter here

we're gonna see that we get a lot of

results instantly it finds subdomains

urls javascript forms uh links anything

we can think of it is going to find it

and that's obviously very very powerful

with that simple scan out of the way we

can also look at some more features that

this tool has because this was just the

most simple way to go but there are

plenty more features that we can use

and first of all i want to talk a bit

about

some

parameters that i call

big bounty parameters because they can

help us and help the companies and help

us adhere to the rules of a platform for

example your platform may have a rule

that you have to set a user agent to be

for example your integrity email address

you can do that with the dash u or the

dash dash

user-agent parameter

we can also set specific cookies with

the cookie and specific headers with the

dash capital h or dash dash header

parameters

next up we also have to talk about speed

because this tool can make a lot of

requests and can go very fast but you

have to make sure that you adhere to the

rules of the program that you're hacking

and that you don't go over that

threshold of that many

requests per second

you can set

the amount of threats you want to use

with dash d or dash dash threads you can

set the concurrency with dash c or dash

dash concurrent as we saw in the example

and then you can set your delay

between new requests and matching

domains to dash k or dash dash delay

now with that out of the way let's look

at some of them some more features that

this tool has

because it can not only find files you

can also file in javascript files as we

as we've shown with dash dash js

it can include subdomains which with

dash dash subs it can also crawl site

maps so if it finds a sitemap.xml file

it can also crawl that if you supply

dash sitemap and the robots.txt file as

well with dash dash robots

now

following that we can do some really

cool stuff and that is done with dash h

or dash dash other dash source

and this is gonna use third third-party

archives such as common crawl um

wayback machine virustotal all that's

all those uh

already big databases of files in the

from the past it's going to use them to

also find urls um

and then you can also use dash r or dash

dash include other domain or other

source rather and what that's gonna do

it's gonna also then crawl those found

um web pages from archive.org or for

from common crawl and also keep on

crawling them so you know that you have

found everything

as you've seen in the example this

generates a ton of output so we need a

means of blacklisting this and with dash

dash blacklist we can black blacklist

specific um

things so we can supply a regex and it's

then gonna blacklist everything that

matches that regex we can also just

whitelist things if we only want

specific things

we can also choose to view the length of

every of our of every file that we get

with dash l or dash dash length and then

we can filter out specific things with

dash dash filter dash length

now this could be extremely useful if

for example this

website has a custom 404 page that just

returns a 200 so the crawler is going to

give you for everything it crawls um

this 200 but you don't want that

obviously so you could remove that from

the results with this filter length and

then you can also do more filtering and

find all of these

flags in the dash help page for this

tool but you can even filter out certain

extensions that you don't want for

example i can think of pngs stuff that's

not really interesting that you don't

want to see well you can also filter

them out

now that was it for this tool go spider

i think it's a really interesting tool

to get some

some first enumeration of your targets

to know

what they are running what's going on

there

and then from there you can obviously

pick the targets that are important and

start hacking

now i hope you enjoyed this video i hope

you like it if you liked it of course

comment down below what tools you would

like to see us cover in the future so

that was it for me have a good day take

care

[Music]