Webscraping with AutoHotkey-101.5 Getting the text from page

Summary

In this tutorial, the speaker demonstrates how to perform web scraping using AutoHotKey (AHK) with Internet Explorer. The video walks through setting up a script to retrieve data from a webpage, focusing on troubleshooting object pointers, ensuring the right page is targeted, and correctly extracting data using properties like `innerText`, `value`, and `outerHTML`. Practical examples are provided, highlighting the importance of using the right property for different types of data, especially when working with form fields or list items. The tutorial is aimed at helping users debug and fine-tune their AHK web scraping scripts.

Takeaways

• 😀 Ensure your AutoHotkey script correctly targets an open IE window by obtaining a pointer to it using a specific function.
• 😀 Check if the pointer variable (`PWB`) is a valid object with the `is object` command to avoid issues with invalid pointers.
• 😀 Use `locationURL` to verify the script is interacting with the correct page, especially when multiple IE windows are open.
• 😀 Troubleshoot common issues by checking whether the `PWB` pointer is valid. A `0` indicates a problem with the pointer.
• 😀 When extracting data from an HTML element, you can use properties like `innerText`, `outerHTML`, and `value`, depending on the context.
• 😀 Use `innerText` to grab the visible text within an element, but `outerHTML` includes the HTML tags surrounding the content.
• 😀 Be aware that `value` is typically used for input fields (like textboxes), and should be used to retrieve the value entered by the user.
• 😀 Always ensure you're using the correct property for the element you're interacting with (e.g., `value` for input fields, `innerText` for general text).
• 😀 Remember that `outerHTML` can exclude the tag when extracting content, and `innerHTML` will include the tags but strip the element itself.
• 😀 Debug by verifying your targeted element and checking that you're pulling data from the right page. This will prevent unnecessary troubleshooting.
• 😀 It’s crucial to test each part of your script step-by-step, especially when extracting data from web pages with different structures or multiple windows open.

Q & A

What is the first step in setting up the web scraping script in AutoHotkey?

The first step is to get a pointer to an open Internet Explorer window using a custom function. This allows the script to interact with the window and scrape data from it.

How does the script verify that the pointer to the IE window is valid?

The script uses the `IsObject()` function to verify if the pointer is valid. If the pointer is valid, it will return '1' (true), and if not, it will return '0' (false).

What should you do if the `IsObject()` function returns '0'?

If `IsObject()` returns '0', it means the pointer is not valid. You should check if the pointer has been correctly initialized and assigned to ensure the object is valid.

How can you check which page the script is scraping from when there are multiple IE windows open?

You can check the current URL by using `PWB.locationURL`. This will show the URL of the last active IE window. If the script grabs the wrong page, it might be due to targeting the wrong window.

What is the difference between `innerText`, `outerHTML`, and `value` when scraping web elements?

`innerText` retrieves the visible text content inside an HTML element. `outerHTML` grabs the full HTML of the element, including the tags. `value` is typically used for form elements, like input fields, to retrieve the value the user has entered.

How does the script identify which list item to scrape from an ordered list?

The script identifies list items by their index within the ordered list. It uses the tag name `<li>` and specifies the index (e.g., `li5` for the 6th item due to zero-based indexing).

Why does the script use `outerHTML` and when should it be used?

`outerHTML` is used when you want to capture the entire HTML code for an element, including its tags. This can be useful if you need the structure around the content, but it's not always necessary if you only need the visible text.

What happens if you try to retrieve the `innerText` from an input field, and why?

If you try to retrieve `innerText` from an input field, it will return nothing or be blank because input elements do not have visible text content. Instead, you should use the `value` property to get the data entered by the user.

What is the purpose of using the learner tool in the script?

The learner tool is used to help identify which property of an HTML element (such as `innerText`, `outerHTML`, or `value`) you should target to extract the correct data from the page.

What common mistake might cause confusion when scraping form elements like input fields?

A common mistake is confusing the properties `innerText` and `value`. While `innerText` is for visible text in HTML elements, form elements like input fields require the `value` property to capture user input.