How To Crawl Behind A Login (Authentication) - Screaming Frog SEO Spider

Summary

The video script offers a quickfire guide on how to authenticate and crawl websites that are behind a login. It explains two types of authentication: basic/digest, where the browser prompts for a username and password, and web form authentication, which involves entering credentials on a login page. The guide demonstrates how to use Screaming Frog's SEO Spider to crawl these sites by entering authentication details and navigating any pop-ups. It also addresses the need to ignore robots.txt for development or staging sites and provides tips on handling JavaScript rendering and responsible crawling practices, including using the exclude function to prevent unintended actions like logging out or data deletion.

Takeaways

• 🔒 **Basic and Digest Authentication**: If your browser prompts for a username and password, it's likely using basic or digest authentication.
• 🚀 **Starting the Crawl**: To crawl a site with basic or digest authentication, simply input the URL and your credentials in the SEO Spider, then click start.
• 🔍 **Crawling Behind a Login**: Useful for accessing internal portals or development websites that require authentication.
• 🛠️ **Handling Robots.txt**: If a site is blocked by robots.txt, you can ignore it by adjusting the configuration to ignore robots.txt settings.
• 📝 **Web Form Authentication**: For login screens within a page, you must enter credentials into the inbuilt browser to replicate the login process.
• 🔗 **Crawling Logged-In Areas**: Once logged in through the inbuilt browser, you can start crawling the URL within the logged-in area.
• ⚙️ **JavaScript Rendering**: Some sites require JavaScript to render links and content correctly, which can be enabled in the spider rendering settings.
• 💡 **Powerful Tool**: The SEO Spider is a powerful tool that can click every link on a page, so use it responsibly to avoid unintended actions.
• 🚫 **Avoid Admin Crawling**: Do not crawl with admin rights if not necessary, to prevent accidental changes or deletions.
• ⛔ **Exclude Sensitive URLs**: Use the exclude function to prevent crawling of sensitive URLs, such as logout links or pages that perform actions.
• 📈 **SEO Spider's Capabilities**: The SEO Spider can be a valuable tool for SEO professionals, developers, and website owners to access and crawl authenticated areas of a website.

Q & A

What is the topic of the video guide?

The video guide is about authentication and crawling websites that are behind a login.

What are the two forms of authentication mentioned in the script?

The two forms of authentication mentioned are basic and digest authentication.

How does basic or digest authentication typically work?

Basic or digest authentication usually involves a browser pop-up requesting a username and password when you visit the website.

What should you do if a website is blocked by robots.txt during crawling?

To crawl a URL that is blocked by robots.txt, you need to go to the configuration, find the robots.txt settings, and click on 'ignore robots.txt'.

What is web form authentication?

Web form authentication is when the login screen is part of the webpage itself, and you need to enter your credentials into the page to receive cookies.

How can you replicate web form authentication in the SEO Spider?

You can replicate web form authentication by entering the required details into the inbuilt Chrome browser within the SEO Spider and logging in.

Why might a website require JavaScript rendering to be enabled for crawling?

Some websites require JavaScript rendering to be enabled to properly crawl links and content that are contained within the DOM rather than just in the raw HTML.

What is the potential risk of crawling a website with admin privileges?

Crawling a website with admin privileges could unintentionally create posts, install plugins, or even delete data if the SEO Spider clicks on certain links.

What is the purpose of the exclude function in the SEO Spider's configuration?

The exclude function allows you to specify a list of URLs that you do not want to be crawled, which can be useful for avoiding sensitive or logout URLs.

Why should the powerful feature of the SEO Spider be used responsibly?

The SEO Spider should be used responsibly because it can potentially perform actions like logging out or modifying data on the website, which could have unintended consequences.

What is the final advice given in the video guide regarding crawling websites?

The final advice is to not crawl as an admin with admin rights and to use the exclude function to prevent crawling sensitive URLs.

What does the '200 status' mentioned in the script indicate?

A '200 status' indicates that the request to the server was successfully received, understood, and accepted.

What is the role of the SEO Spider in the context of this video guide?

The SEO Spider is a tool used for crawling websites, particularly those that require authentication to access, and it helps in analyzing and managing the crawling process.