Find endpoints in the blink of an eye! GoSpider - Hacker Tools
Summary
TLDRIn this hacker tools video, the presenter introduces Go Spider, a powerful web crawling tool designed to discover endpoints, subdomains, and other resources on a website. It efficiently scans web pages, identifies links, and can even recursively crawl through found files. The tool offers customization options like setting user agents, cookies, headers, and managing request speed to comply with platform rules. Advanced features include utilizing third-party archives and filtering results based on file length or extensions, making it an essential tool for initial target enumeration in cybersecurity assessments.
Takeaways
- 🕷️ Go Spider is a tool designed for web scraping and crawling web pages to discover various endpoints, subdomains, and other assets.
- 🔍 It operates by requesting a web page and then searching for links, JavaScript files, directories, subdomains, and endpoints, presenting a comprehensive map of the target's web presence.
- 🔄 The tool can recursively crawl through discovered files to uncover even more links and resources, creating a detailed web of the application's structure.
- ⚙️ Basic usage involves running Go Spider with the '-s' option to specify a URL, '-o' for output file, and '-c' for setting the number of concurrent requests.
- 🔑 'Big bounty parameters' like '-u' for user-agent and '-h' for custom headers can help comply with platform rules during scanning.
- 🚀 The tool can be configured for speed with parameters like '-threads' for setting the number of threads and '-concurrent' for concurrency level.
- 🛑 To avoid overwhelming targets, use '-k' or '-delay' to set a delay between requests, ensuring you stay within acceptable request limits.
- 🗂️ Advanced features include crawling JavaScript files with '-js', including subdomains with '-subs', and utilizing sitemaps and robots.txt with '-sitemap' and '-robots' respectively.
- 🔎 Go Spider can integrate with third-party archives like Common Crawl and Wayback Machine using '-h' or '-other-source' to find URLs from historical data.
- ⛔️ Use '-blacklist' with regex to exclude specific results or '-whitelist' to focus only on desired outcomes.
- 📊 Filtering options like '-l' or '-length' allow for the exclusion of certain file types or HTTP status codes to refine the scan results.
Q & A
What is Go Spider and what does it do?
-Go Spider is a tool that spiders web pages to crawl them and extract information such as endpoints, subdomains, and other links. It can also recursively crawl the discovered files to create a comprehensive web of the application's structure.
How does Go Spider perform a basic scan?
-To perform a basic scan, Go Spider is run with the '-s' option to input a URL, '-o' for output to specify an output file, and '-c' for concurrency to set the number of concurrent requests.
What are 'big bounty parameters' in Go Spider and why are they important?
-'Big bounty parameters' refer to options like user-agent, cookies, and headers that can be set with Go Spider to adhere to the rules of a platform and ensure ethical hacking practices.
How can Go Spider be configured to respect the speed limits of a target platform?
-Go Spider allows setting the number of threads with '-d', concurrency with '-c', and delay between requests with '-k' to control the speed and avoid overwhelming the target platform.
What additional features does Go Spider offer beyond basic crawling?
-Go Spider can find JavaScript files, include subdomains, crawl sitemaps, and utilize third-party archives like Common Crawl and Wayback Machine for more extensive data collection.
How can Go Spider be used to filter out unwanted results during a scan?
-Go Spider provides options like '-h' for excluding specific sources, '-l' to view file lengths, and '-f' to filter out specific lengths or extensions to refine the scan results.
What is the purpose of the '-blacklist' option in Go Spider?
-The '-blacklist' option allows users to supply a regex pattern to exclude results that match it, helping to focus on relevant data during a scan.
Can Go Spider handle multiple URLs at once?
-Yes, Go Spider can handle multiple URLs by using the '-s' option with a file that contains multiple links, allowing for batch processing of URLs.
How does Go Spider help in the initial enumeration of targets?
-Go Spider assists in the initial enumeration by mapping out the target's web structure, identifying running services, and providing a comprehensive overview of what's present on the target's web pages.
What is the recommended next step after using Go Spider for initial scanning?
-After the initial scan, the recommended next step is to analyze the results, identify important targets, and proceed with more focused and in-depth security testing.
Outlines
🕷️ Introduction to Go Spider
This paragraph introduces Go Spider, a tool designed for web crawling and data extraction. It explains how Go Spider can spider web pages to discover endpoints, subdomains, and other valuable information. The tool is capable of recursively crawling through files and directories to create a comprehensive map of a web application. The video demonstrates a basic scan using Go Spider with the '-s' option for specifying a URL, '-o' for output file, and '-c' for setting the number of concurrent requests. The tool's ability to quickly find a wealth of information is highlighted, emphasizing its power in web scanning.
🛠️ Advanced Features and Filtering with Go Spider
The second paragraph delves into the advanced features of Go Spider, including parameters that can be set to comply with platform rules, such as user-agent and cookie settings. It discusses the importance of controlling the speed of requests to avoid exceeding limits set by the target website. The paragraph also covers additional functionalities like crawling JavaScript files, sitemaps, and using third-party archives for enhanced data retrieval. The tool offers filtering options to refine results, such as excluding certain file extensions or HTTP response lengths, which is particularly useful for avoiding false positives like custom 404 pages. The video concludes by encouraging viewers to explore the tool for initial target enumeration and suggests viewers to provide feedback on future tool coverage.
Mindmap
Keywords
💡Go Spider
💡Endpoints
💡Subdomains
💡Concurrent Requests
💡User Agent
💡Cookies
💡Headers
💡Blacklist
💡Whitelist
💡Sitemap.xml
💡Robots.txt
Highlights
Go Spider is a tool for spidering web pages to crawl and gather information such as endpoints and subdomains.
It can request a page and search for links, JavaScript files, directories, subdomains, and endpoints.
The tool can also crawl the findings to create a web of the application's files and links.
Go Spider can be run with simple commands to perform scans and gather results instantly.
The '-s' option allows input of a URL to scan, while 'capital S' can take a file with multiple links.
The '-o' option is used for outputting results to a file.
The '-c' option sets the level of concurrency for the scan.
Parameters like 'big bounty parameters' can be set to adhere to platform rules, such as user-agent with '-u'.
The '-cookie' and '-header' options allow setting specific cookies and headers for the scan.
The '-d' and '-c' options control the number of threads and concurrency of the scan.
The '-k' option sets the delay between requests to match domains.
Go Spider can find JavaScript files, include subdomains, and crawl sitemaps and robots.txt files.
The '-h' or '--other-source' option uses third-party archives like Common Crawl and Wayback Machine to find URLs.
The '-r' or '--include' option allows crawling of web pages from archives like archive.org.
The '--blacklist' option can be used to blacklist specific items using regex patterns.
The '--whitelist' option can be used to only allow specific items during the scan.
The '-l' or '--length' option can show the length of each file found, which can be useful for filtering.
The '--filter-length' option can be used to filter out specific file lengths or HTTP status codes.
The tool provides extensive filtering options to refine the scan results.
Go Spider is useful for initial enumeration of targets to understand what they are running.
The video concludes with a call to action for viewers to comment on tools they would like to see covered in the future.
Transcripts
this
is go spider
and that's what we're gonna talk about
in today's hacker tools video
[Music]
go spider is a really cool tool that
allows us to spider web pages to crawl
web pages and to get targets endpoints
subdomains everything out of it so the
idea here is that this tool is going to
request a page and then it's going to
search through that page and look for
links to javascript files other
directories other sub domains other
endpoints anything and it's going to
show that all to us now additionally to
that you can even set it that it's also
going to crawl those findings that we
have so it finds a file and it also
crawls that file for more files and that
way you really create a web of this
application where you have all the files
and all the links between them and you
can be sure that you have almost
everything
that can be found on that web page
without having to go through it all
manually but let's take a look at how
this works by running it and doing a
very simple scan here
so what are we going to do we're going
to run go spider and then we're going to
supply the dash s option the dash
lowercase s option allows us to input a
url to
any page that we want to scan
now you can also use capital s and then
supply a file that holds multiple links
that you always can
following that argument we're gonna use
the dash o for output and we're gonna
supply an output file and then we're
gonna use this dash c lastly which is c
standing for a concurrency how many
concurrent requests are we gonna run
and that is how we can perform a simple
scan and if i press enter here
we're gonna see that we get a lot of
results instantly it finds subdomains
urls javascript forms uh links anything
we can think of it is going to find it
and that's obviously very very powerful
with that simple scan out of the way we
can also look at some more features that
this tool has because this was just the
most simple way to go but there are
plenty more features that we can use
and first of all i want to talk a bit
about
some
parameters that i call
big bounty parameters because they can
help us and help the companies and help
us adhere to the rules of a platform for
example your platform may have a rule
that you have to set a user agent to be
for example your integrity email address
you can do that with the dash u or the
dash dash
user-agent parameter
we can also set specific cookies with
the cookie and specific headers with the
dash capital h or dash dash header
parameters
next up we also have to talk about speed
because this tool can make a lot of
requests and can go very fast but you
have to make sure that you adhere to the
rules of the program that you're hacking
and that you don't go over that
threshold of that many
requests per second
you can set
the amount of threats you want to use
with dash d or dash dash threads you can
set the concurrency with dash c or dash
dash concurrent as we saw in the example
and then you can set your delay
between new requests and matching
domains to dash k or dash dash delay
now with that out of the way let's look
at some of them some more features that
this tool has
because it can not only find files you
can also file in javascript files as we
as we've shown with dash dash js
it can include subdomains which with
dash dash subs it can also crawl site
maps so if it finds a sitemap.xml file
it can also crawl that if you supply
dash sitemap and the robots.txt file as
well with dash dash robots
now
following that we can do some really
cool stuff and that is done with dash h
or dash dash other dash source
and this is gonna use third third-party
archives such as common crawl um
wayback machine virustotal all that's
all those uh
already big databases of files in the
from the past it's going to use them to
also find urls um
and then you can also use dash r or dash
dash include other domain or other
source rather and what that's gonna do
it's gonna also then crawl those found
um web pages from archive.org or for
from common crawl and also keep on
crawling them so you know that you have
found everything
as you've seen in the example this
generates a ton of output so we need a
means of blacklisting this and with dash
dash blacklist we can black blacklist
specific um
things so we can supply a regex and it's
then gonna blacklist everything that
matches that regex we can also just
whitelist things if we only want
specific things
we can also choose to view the length of
every of our of every file that we get
with dash l or dash dash length and then
we can filter out specific things with
dash dash filter dash length
now this could be extremely useful if
for example this
website has a custom 404 page that just
returns a 200 so the crawler is going to
give you for everything it crawls um
this 200 but you don't want that
obviously so you could remove that from
the results with this filter length and
then you can also do more filtering and
find all of these
flags in the dash help page for this
tool but you can even filter out certain
extensions that you don't want for
example i can think of pngs stuff that's
not really interesting that you don't
want to see well you can also filter
them out
now that was it for this tool go spider
i think it's a really interesting tool
to get some
some first enumeration of your targets
to know
what they are running what's going on
there
and then from there you can obviously
pick the targets that are important and
start hacking
now i hope you enjoyed this video i hope
you like it if you liked it of course
comment down below what tools you would
like to see us cover in the future so
that was it for me have a good day take
care
[Music]
関連動画をさらに表示
5.0 / 5 (0 votes)