PyAutoGUI - Locate anything on your screen | Simple Pyautogui project

Summary

In this tutorial, the creator demonstrates the most crucial function of the 'PyAutoGUI' package used for automating tasks on your computer. By using functions like 'locateOnScreen', 'moveTo', and 'locateCenterOnScreen', users can automate tasks such as navigating through GUIs, pressing hotkeys, and clicking on specific areas of the screen. As an example, the video automates subscribing to a YouTube channel. The creator also shows how to fine-tune the automation using confidence levels for more accurate results. Viewers are encouraged to follow along and try automating the task themselves.

Takeaways

• 💻 The video introduces the PiAutoGUI package for Python, emphasizing its usefulness in automation projects.
• 📦 The 'locateOnScreen' function finds the location of an image on the screen, returning its coordinates and size.
• 📸 To use 'locateOnScreen', capture a screenshot of the desired area (e.g., with the Snipping Tool) and pass the image to the function.
• 🖱️ The 'center' function can return the center coordinates of the located image, useful for precise mouse movements.
• 🎯 The 'moveTo' function moves the mouse to the specified coordinates on the screen.
• 🔗 The 'locateCenterOnScreen' function combines 'locateOnScreen' and 'center' for more efficient workflows.
• 🔑 To open a new browser tab, the 'hotkey' function is used, simulating keyboard shortcuts (e.g., Ctrl+T).
• 📝 The 'write' function types text (like URLs or search queries) into the targeted location.
• 🤖 Confidence levels can be adjusted in image recognition (using the 'confidence' attribute) to handle pixel mismatches.
• 📽️ The script automates subscribing to a YouTube channel by combining several PiAutoGUI functions, including image recognition and keyboard/mouse interactions.

Q & A

What is the most important function in the pyautogui package discussed in the video?

The most important function in the pyautogui package discussed in the video is 'locateOnScreen', which is used to find the location or coordinates of any part on the screen using an image as a parameter.

How does the 'locateOnScreen' function work?

The 'locateOnScreen' function works by taking an image as a parameter and returning the full location of that image on the screen, including its coordinates (x and y), width, and height.

What is the purpose of the 'center' function in pyautogui?

The 'center' function is used to find the center coordinates of a part on the screen. It is often used after 'locateOnScreen' to get the exact center point for mouse movement.

How can you move the mouse to a specific location using pyautogui?

To move the mouse to a specific location, you can use the 'moveTo' function in pyautogui, passing the coordinates (x and y) as arguments.

What is the 'locateCenterOnScreen' function and how does it simplify the automation process?

The 'locateCenterOnScreen' function is a combination of 'locateOnScreen' and 'center' functions. It returns the center location of an image on the screen in one operation, simplifying the process of locating and moving to screen elements.

What is the role of the 'hotkey' function in the automation project described in the video?

The 'hotkey' function is used to simulate keyboard shortcuts. In the video, it is used to press 'Ctrl+T' to open a new tab and 'Enter' to navigate to the typed URL.

How does the 'prompt' function assist in taking user inputs during automation?

The 'prompt' function in pyautogui shows a pop-up with a text box to take user inputs. It is used in the video to get the channel name for subscribing to a YouTube channel without manually typing it in the code.

What is the purpose of the 'confidence' attribute when locating images on the screen?

The 'confidence' attribute is used to set the threshold for image matching. A higher value means a more precise match is required. It is important for accurately locating elements on the screen, especially when there might be slight variations in the image.

Why is it necessary to install the 'opencv-python' package when using the 'confidence' attribute?

The 'opencv-python' package is required for image processing tasks, such as adjusting the confidence level for image matching. It provides the necessary tools to handle image recognition with varying degrees of accuracy.

What is the final project demonstrated in the video using pyautogui?

The final project demonstrated in the video is an automated process to subscribe to a YouTube channel. It involves opening a browser, navigating to YouTube, searching for the channel, and subscribing to it using pyautogui functions.

Outlines

00:00

💻 Introduction to pi auto gui

The speaker begins by introducing the 'pi auto gui' package in Python, emphasizing its importance in automation projects. They demonstrate how to set up a new Python file in VS Code and install the package using 'pip install pi auto gui'. The key function 'locate on screen' is introduced, which finds the location of an image on the screen. An example is given where the speaker captures the 'edit menu' using a snipping tool, saves it, and uses the function to get its coordinates. The 'center' function is also explained, which finds the center coordinates of an image. The 'move to' function is briefly mentioned, which moves the mouse cursor to a specified location.

05:01

🔍 Advanced Automation with pi auto gui

The speaker continues by showing how to automate browser actions using 'pi auto gui'. They discuss the 'hotkey' function to simulate key presses and the 'write' function to input text into a field. A problem arises where the actions are performed in the terminal instead of the browser, which leads to the introduction of the 'prompt' function. This function allows the user to input data through a pop-up, which is used to get the channel name for subscription. The speaker also explains the use of the 'locate center on screen' function with a 'confidence' attribute, which requires the installation of 'opencv-python'. The speaker demonstrates how to automate the process of subscribing to a YouTube channel by locating and interacting with various UI elements.

10:06

🎯 Completing Automation with Confidence Adjustment

The final paragraph details the completion of the YouTube subscription automation. The speaker captures images of the channel logo and subscribe button, and uses these to locate and interact with them in the script. They mention the importance of the 'confidence' attribute in the 'locate center on screen' function, which affects the accuracy of image recognition. After adjusting the confidence level, the automation successfully opens YouTube, searches for the channel, and subscribes without manual intervention. The speaker concludes by encouraging viewers to follow along, subscribe to their channel, and provide feedback in the comments.

Keywords

💡Pi Auto GUI

Pi Auto GUI is a Python package designed for automating computer GUIs (Graphical User Interfaces). In the video, the presenter uses this package to demonstrate how to automate tasks on a computer. It's a core tool for the tutorial, as it allows the creation of scripts that can interact with the graphical interface of other programs, such as opening menus, clicking buttons, and typing text.

💡Locate On Screen

The 'locate on screen' function is a crucial feature of the Pi Auto GUI package. It is used to find the location or coordinates of a specific element on the screen by matching it with an image file. In the video, this function is used to identify the position of the 'Edit' menu, which is then used to move the mouse cursor to that location, showcasing the automation of a common GUI interaction.

💡Center

The 'center' function is another utility within the Pi Auto GUI package that calculates the center coordinates of an element once its location has been identified using 'locate on screen'. This is particularly useful for mouse movement automation, as it allows the script to move the cursor not just to the element's location but to its precise center. In the script, this function is applied to ensure accurate mouse positioning.

💡Move To

The 'move to' function is used to move the mouse cursor to a specific set of coordinates. In the context of the video, after determining the center coordinates of the 'Edit' menu using the 'center' function, the 'move to' function is employed to simulate the act of moving the mouse to that point, effectively automating the navigation to that menu.

💡Locate Center On Screen

This function combines the 'locate on screen' and 'center' functions into a single operation. It directly returns the center coordinates of an image when passed as a parameter, streamlining the process of locating and centering on an element. The video script uses this function to simplify the automation process by obtaining the center coordinates in one step.

💡Hotkey

A 'hotkey' is a keyboard shortcut that can be used to perform certain actions quickly. In the video, the presenter uses the 'hotkey' function to simulate pressing 'Ctrl + T' to open a new tab in a browser, which is an example of automating keyboard input using the Pi Auto GUI package.

💡Write Function

The 'write' function is used to simulate typing text into a field or application. In the video, it's employed to write the URL of YouTube into the browser's address bar, demonstrating how text input can be automated using the Pi Auto GUI package.

💡Prompt

The 'prompt' function is used to create a pop-up that can take user input. In the video, it's utilized to get the name of the YouTube channel to subscribe to from the user, showing how the package can interact with the user to gather information needed for the automation process.

💡Confidence

The 'confidence' attribute in the 'locate center on screen' function refers to the level of certainty required for the image recognition to be successful. A higher confidence level means the function will only return a match if the image closely resembles the one provided. In the video, adjusting the confidence level is discussed as a method to improve the accuracy of the image recognition process.

💡OpenCV Python

OpenCV Python is a library used for computer vision tasks, including image recognition. In the video, it's mentioned as a requirement for using the 'confidence' attribute effectively. The presenter installs this package to enhance the image recognition capabilities of the Pi Auto GUI package, which is crucial for accurately locating elements on the screen.

💡Automation Project

An 'automation project' refers to a task or series of tasks that are automated using software. In the video, the presenter creates a simple automation project that involves subscribing to a YouTube channel automatically. This project serves as a practical example of how the various functions of the Pi Auto GUI package can be combined to achieve a complex automation goal.

Highlights

Introduction to the most important function of the PyAutoGUI package.

Demonstration of creating a new Python file in VS Code for the project.

Explanation of installing the PyAutoGUI package using pip.

Importing the PyAutoGUI package in the Python script.

Description of the 'locateOnScreen' function to find screen coordinates.

Tutorial on capturing an image of the 'Edit' menu using the Snipping Tool.

Using the 'locateOnScreen' function to find the location of the 'Edit' menu.

Introduction to the 'center' function to get the center coordinates of an image.

Explanation of the 'moveTo' function to move the mouse cursor.

Combining 'locateOnScreen' and 'center' into the 'locateCenterOnScreen' function.

Creating a simple project to automate subscribing to a YouTube channel.

Using the 'hotkey' function to press 'Ctrl+T' for opening a new tab.

Utilizing the 'write' function to enter the YouTube URL into the search bar.

Explanation of the 'prompt' function to minimize the editor and run code in the background.

Capturing the YouTube search box and subscribe button using the Snipping Tool.

Using 'locateCenterOnScreen' to find the search box and subscribe button coordinates.

Automating the process of clicking, typing the channel name, and pressing enter.

Introduction to the 'confidence' attribute for more accurate image recognition.

Installation of the 'opencv-python' package for image recognition.

Adjusting the 'confidence' level to handle pixel mismatch issues.

Final demonstration of the automated YouTube channel subscription process.

Encouragement for viewers to follow the tutorial and subscribe to the presenter's channel.

Transcripts

hello coders

in this video i am going to show you the

most important function of pi auto gui

package

this function is so important that you

will use it in almost every automation

project that you create and i will also

be creating a simple project using this

function

so let's get started

i am inside my vs code i'll create a new

file

set the language to python

and save the file on my desktop

now i'll come here

those who have not seen my previous

video let me tell you i am using a

python package named pi auto gui which

is used to automate computer guise

so you will have to install it like this

type pip install pi auto gui

i have already installed this package

so now coming to the code

i'll import the package

now

if you want to get the location or

coordinates of any part on the screen

then there is a function named locate on

screen

this function takes an image as a

parameter and returns the full location

of that image on the screen

for example

if i want to get the location of this

edit menu

then what i will do is

i'll search for snipping tool on my

computer like this open it

click on new

and then capture this edit menu area

and save it as a png file

or any image type file

now i will pass the name of this image

file over here

let me print the response

i'll come here

and run the code

you can see it has printed the full

location it says from left it's at 116

and from top it's at 6.

these are the coordinates x and y

where x is 1 1 6 and y is 6

then it gives the width and height of

that image

but we just require the coordinates that

is

the x and y value so in order to do that

there is another function named center

let me show you

i'll pass the response to the center

function

and it will return to me the center

coordinates of that part

let me run this

and there you go it has given me the

exact center coordinates

now i'll move to that location using the

move to function

but before that i'll have to take the

coordinates inside a variable like this

and then pass it to the move to function

by the way move to function i have

explained it in detail and there are

more functions that i have explained in

my previous video so do check that out

what move to function does is whatever

the coordinates that you pass to this

function it is going to move your mouse

to that particular location or to that

particular coordinate that's it

okay

let's run this

did you see that my mouse automatically

came to this edit menu

so

you had to perform two operations in

order to get the location of a part on

the screen but you can also do this

using one operation that is by using a

function named locate center on screen

this function is the combination of

locate on screen and center functions

here you need to pass the image file

name and it will return the center

location of that image i'll just print

the response

and now run the code

you got the same result let me just

write the comments for you just to

understand

so you can use only this function and

your work is done so coders

this was the most important function of

buy auto gui package which you will

require in almost every automation

project

now is the time to create a simple

project using this function and other

functions that i showed you in my

previous video

the project that i'll create is

subscribing a youtube channel

automatically i have written down the

steps already so the first step is to

open a new tab on the browser see i know

that shortcut for new tab is control

plus t so to press ctrl and t i will use

the hotkey function

and i'll pass

ctrl

and

t

that's it the next step is to search

youtube.com

to do that i'll use the write function

and pass the youtube url

so this will write the url in the search

bar

and then i'll press enter using the

hotkey function now we'll just run this

much and see

but there is one problem you'll see it

when i run the code

let's see i'll keep the browser open

behind the editor like this

i'm running the code and here is the

problem you can see that it typed

control plus ta and the url inside the

terminal itself

so we need to find a way to minimize our

editor and run the code in the

background

in order to do this there are some

functions in pi to gui package which can

take inputs from you we will use one of

those functions

so the function i will use is

prompt

this function shows a pop-up and a text

box to take inputs

the attribute i'll pass is text

which will store whatever i'll type in

the text box

and title

to give a heading for the popup i'll

type

enter the channel name

and this will return the text that i

will enter so i'll store it in this

variable

i'll call it

channel underscore name

you can give whatever name you want i'll

write the comment

so let me show you the channel that i

want to subscribe

by the way i am not doing any paid

promotions

i've been following this channel

so the channel name is

the english scholar online camp

if you want to learn grammar and

punctuation

improve your vocabulary get some useful

tips and free resources

then you can go ahead and check this

channel and subscribe it

one thing i love is that most of the

videos on this channel are short ones

okay so now i'll go back to the code

here i'll print the channel name now

i'll run the code again

and here the popup is

it says enter the channel name

i'll type

the english

scholar

online camp

ok

did you see that

my code was running in the background

and we were able to do what we wanted

now the next step is to type the enter

channel name in the search box

so to locate the search box

i'll capture this search

using the snipping tool

and i'll save it

i'll also capture the subscribe button

oh but for that first i need to

unsubscribe

okay now you can see the subscribe

button

now i will capture it using the snipping

tool again

and i'll save it

going back to the code

so use our important function again

to locate the search box and also locate

the subscribe button on the screen

locate center on screen

the file name

once i locate it i'll move to that

location so i'll use move to function

then i need to click on the search box

so i'll use the click method

after clicking i need to type the

channel name

so i'll use the write function and then

pass the channel name variable to that

function i'll just write the comment

okay then again to press enter

pi auto gui

dot

hotkey

enter i forgot one thing i also need to

click after i press ok on the pop-up

so i will have to add a click function

there also

here okay

now i want to show you one attribute for

the locate center on screen function

that is confidence

i'll type confidence

equal to

0.9

0.9 means 90 percent so 1 will be 100

percent this means that even if there is

a 90 match

then still it should consider

and to use this confidence attribute i

need to install a package called opencv

python

so i'll type pip install

opencv python

done

now let's run this code

did you see that i didn't touch my mouse

or keyboard it automatically opened

youtube and searched for the channel

so now

the next step would be to open the

channel

to do that i need to click on the

channel logo

so now you know what i'm going to do

open snipping tool

capture the logo

and save it

now come back to the code

i'll just copy paste this over here

and

change the file name

that's it

i'll put the necessary sleep of one

second for the next and final step

again i'll copy paste

change the file name

this image is the subscribe button that

we captured earlier

and that's it

now let's run it

oops

it did not search the logo this might

happen sometime due to pixel mismatch so

here the confidence was 90 percent still

it did not work so now i'd reduce it to

80 percent and try again

in such cases you'll have to decrease or

increase the confidence and keep trying

that's why the confidence attribute is

very important

now i think it should work let's see

bingo it worked i did not touch my mouse

or keyboard it automatically opened

youtube search for the channel name

then opened the channel

and subscribed it

so by this i have completed almost all

the functions of the pi auto gui package

and showed you how you can automate

anything on your computer using this

package

now i want you to follow this video step

by step

and then using your code

subscribe to my channel

and let me know in the comment section

if you learned something from this video

then please do not forget to subscribe

to my channel in order to be notified

about my latest videos thank you so much

for watching see you in my next video