PyAutoGUI - Locate anything on your screen | Simple Pyautogui project
Summary
TLDRIn this tutorial, the creator demonstrates the most crucial function of the 'PyAutoGUI' package used for automating tasks on your computer. By using functions like 'locateOnScreen', 'moveTo', and 'locateCenterOnScreen', users can automate tasks such as navigating through GUIs, pressing hotkeys, and clicking on specific areas of the screen. As an example, the video automates subscribing to a YouTube channel. The creator also shows how to fine-tune the automation using confidence levels for more accurate results. Viewers are encouraged to follow along and try automating the task themselves.
Takeaways
- 💻 The video introduces the PiAutoGUI package for Python, emphasizing its usefulness in automation projects.
- 📦 The 'locateOnScreen' function finds the location of an image on the screen, returning its coordinates and size.
- 📸 To use 'locateOnScreen', capture a screenshot of the desired area (e.g., with the Snipping Tool) and pass the image to the function.
- 🖱️ The 'center' function can return the center coordinates of the located image, useful for precise mouse movements.
- 🎯 The 'moveTo' function moves the mouse to the specified coordinates on the screen.
- 🔗 The 'locateCenterOnScreen' function combines 'locateOnScreen' and 'center' for more efficient workflows.
- 🔑 To open a new browser tab, the 'hotkey' function is used, simulating keyboard shortcuts (e.g., Ctrl+T).
- 📝 The 'write' function types text (like URLs or search queries) into the targeted location.
- 🤖 Confidence levels can be adjusted in image recognition (using the 'confidence' attribute) to handle pixel mismatches.
- 📽️ The script automates subscribing to a YouTube channel by combining several PiAutoGUI functions, including image recognition and keyboard/mouse interactions.
Q & A
What is the most important function in the pyautogui package discussed in the video?
-The most important function in the pyautogui package discussed in the video is 'locateOnScreen', which is used to find the location or coordinates of any part on the screen using an image as a parameter.
How does the 'locateOnScreen' function work?
-The 'locateOnScreen' function works by taking an image as a parameter and returning the full location of that image on the screen, including its coordinates (x and y), width, and height.
What is the purpose of the 'center' function in pyautogui?
-The 'center' function is used to find the center coordinates of a part on the screen. It is often used after 'locateOnScreen' to get the exact center point for mouse movement.
How can you move the mouse to a specific location using pyautogui?
-To move the mouse to a specific location, you can use the 'moveTo' function in pyautogui, passing the coordinates (x and y) as arguments.
What is the 'locateCenterOnScreen' function and how does it simplify the automation process?
-The 'locateCenterOnScreen' function is a combination of 'locateOnScreen' and 'center' functions. It returns the center location of an image on the screen in one operation, simplifying the process of locating and moving to screen elements.
What is the role of the 'hotkey' function in the automation project described in the video?
-The 'hotkey' function is used to simulate keyboard shortcuts. In the video, it is used to press 'Ctrl+T' to open a new tab and 'Enter' to navigate to the typed URL.
How does the 'prompt' function assist in taking user inputs during automation?
-The 'prompt' function in pyautogui shows a pop-up with a text box to take user inputs. It is used in the video to get the channel name for subscribing to a YouTube channel without manually typing it in the code.
What is the purpose of the 'confidence' attribute when locating images on the screen?
-The 'confidence' attribute is used to set the threshold for image matching. A higher value means a more precise match is required. It is important for accurately locating elements on the screen, especially when there might be slight variations in the image.
Why is it necessary to install the 'opencv-python' package when using the 'confidence' attribute?
-The 'opencv-python' package is required for image processing tasks, such as adjusting the confidence level for image matching. It provides the necessary tools to handle image recognition with varying degrees of accuracy.
What is the final project demonstrated in the video using pyautogui?
-The final project demonstrated in the video is an automated process to subscribe to a YouTube channel. It involves opening a browser, navigating to YouTube, searching for the channel, and subscribing to it using pyautogui functions.
Outlines
💻 Introduction to pi auto gui
The speaker begins by introducing the 'pi auto gui' package in Python, emphasizing its importance in automation projects. They demonstrate how to set up a new Python file in VS Code and install the package using 'pip install pi auto gui'. The key function 'locate on screen' is introduced, which finds the location of an image on the screen. An example is given where the speaker captures the 'edit menu' using a snipping tool, saves it, and uses the function to get its coordinates. The 'center' function is also explained, which finds the center coordinates of an image. The 'move to' function is briefly mentioned, which moves the mouse cursor to a specified location.
🔍 Advanced Automation with pi auto gui
The speaker continues by showing how to automate browser actions using 'pi auto gui'. They discuss the 'hotkey' function to simulate key presses and the 'write' function to input text into a field. A problem arises where the actions are performed in the terminal instead of the browser, which leads to the introduction of the 'prompt' function. This function allows the user to input data through a pop-up, which is used to get the channel name for subscription. The speaker also explains the use of the 'locate center on screen' function with a 'confidence' attribute, which requires the installation of 'opencv-python'. The speaker demonstrates how to automate the process of subscribing to a YouTube channel by locating and interacting with various UI elements.
🎯 Completing Automation with Confidence Adjustment
The final paragraph details the completion of the YouTube subscription automation. The speaker captures images of the channel logo and subscribe button, and uses these to locate and interact with them in the script. They mention the importance of the 'confidence' attribute in the 'locate center on screen' function, which affects the accuracy of image recognition. After adjusting the confidence level, the automation successfully opens YouTube, searches for the channel, and subscribes without manual intervention. The speaker concludes by encouraging viewers to follow along, subscribe to their channel, and provide feedback in the comments.
Mindmap
Keywords
💡Pi Auto GUI
💡Locate On Screen
💡Center
💡Move To
💡Locate Center On Screen
💡Hotkey
💡Write Function
💡Prompt
💡Confidence
💡OpenCV Python
💡Automation Project
Highlights
Introduction to the most important function of the PyAutoGUI package.
Demonstration of creating a new Python file in VS Code for the project.
Explanation of installing the PyAutoGUI package using pip.
Importing the PyAutoGUI package in the Python script.
Description of the 'locateOnScreen' function to find screen coordinates.
Tutorial on capturing an image of the 'Edit' menu using the Snipping Tool.
Using the 'locateOnScreen' function to find the location of the 'Edit' menu.
Introduction to the 'center' function to get the center coordinates of an image.
Explanation of the 'moveTo' function to move the mouse cursor.
Combining 'locateOnScreen' and 'center' into the 'locateCenterOnScreen' function.
Creating a simple project to automate subscribing to a YouTube channel.
Using the 'hotkey' function to press 'Ctrl+T' for opening a new tab.
Utilizing the 'write' function to enter the YouTube URL into the search bar.
Explanation of the 'prompt' function to minimize the editor and run code in the background.
Capturing the YouTube search box and subscribe button using the Snipping Tool.
Using 'locateCenterOnScreen' to find the search box and subscribe button coordinates.
Automating the process of clicking, typing the channel name, and pressing enter.
Introduction to the 'confidence' attribute for more accurate image recognition.
Installation of the 'opencv-python' package for image recognition.
Adjusting the 'confidence' level to handle pixel mismatch issues.
Final demonstration of the automated YouTube channel subscription process.
Encouragement for viewers to follow the tutorial and subscribe to the presenter's channel.
Transcripts
hello coders
in this video i am going to show you the
most important function of pi auto gui
package
this function is so important that you
will use it in almost every automation
project that you create and i will also
be creating a simple project using this
function
so let's get started
i am inside my vs code i'll create a new
file
set the language to python
and save the file on my desktop
now i'll come here
those who have not seen my previous
video let me tell you i am using a
python package named pi auto gui which
is used to automate computer guise
so you will have to install it like this
type pip install pi auto gui
i have already installed this package
so now coming to the code
i'll import the package
now
if you want to get the location or
coordinates of any part on the screen
then there is a function named locate on
screen
this function takes an image as a
parameter and returns the full location
of that image on the screen
for example
if i want to get the location of this
edit menu
then what i will do is
i'll search for snipping tool on my
computer like this open it
click on new
and then capture this edit menu area
and save it as a png file
or any image type file
now i will pass the name of this image
file over here
let me print the response
i'll come here
and run the code
you can see it has printed the full
location it says from left it's at 116
and from top it's at 6.
these are the coordinates x and y
where x is 1 1 6 and y is 6
then it gives the width and height of
that image
but we just require the coordinates that
is
the x and y value so in order to do that
there is another function named center
let me show you
i'll pass the response to the center
function
and it will return to me the center
coordinates of that part
let me run this
and there you go it has given me the
exact center coordinates
now i'll move to that location using the
move to function
but before that i'll have to take the
coordinates inside a variable like this
and then pass it to the move to function
by the way move to function i have
explained it in detail and there are
more functions that i have explained in
my previous video so do check that out
what move to function does is whatever
the coordinates that you pass to this
function it is going to move your mouse
to that particular location or to that
particular coordinate that's it
okay
let's run this
did you see that my mouse automatically
came to this edit menu
so
you had to perform two operations in
order to get the location of a part on
the screen but you can also do this
using one operation that is by using a
function named locate center on screen
this function is the combination of
locate on screen and center functions
here you need to pass the image file
name and it will return the center
location of that image i'll just print
the response
and now run the code
you got the same result let me just
write the comments for you just to
understand
so you can use only this function and
your work is done so coders
this was the most important function of
buy auto gui package which you will
require in almost every automation
project
now is the time to create a simple
project using this function and other
functions that i showed you in my
previous video
the project that i'll create is
subscribing a youtube channel
automatically i have written down the
steps already so the first step is to
open a new tab on the browser see i know
that shortcut for new tab is control
plus t so to press ctrl and t i will use
the hotkey function
and i'll pass
ctrl
and
t
that's it the next step is to search
youtube.com
to do that i'll use the write function
and pass the youtube url
so this will write the url in the search
bar
and then i'll press enter using the
hotkey function now we'll just run this
much and see
but there is one problem you'll see it
when i run the code
let's see i'll keep the browser open
behind the editor like this
i'm running the code and here is the
problem you can see that it typed
control plus ta and the url inside the
terminal itself
so we need to find a way to minimize our
editor and run the code in the
background
in order to do this there are some
functions in pi to gui package which can
take inputs from you we will use one of
those functions
so the function i will use is
prompt
this function shows a pop-up and a text
box to take inputs
the attribute i'll pass is text
which will store whatever i'll type in
the text box
and title
to give a heading for the popup i'll
type
enter the channel name
and this will return the text that i
will enter so i'll store it in this
variable
i'll call it
channel underscore name
you can give whatever name you want i'll
write the comment
so let me show you the channel that i
want to subscribe
by the way i am not doing any paid
promotions
i've been following this channel
so the channel name is
the english scholar online camp
if you want to learn grammar and
punctuation
improve your vocabulary get some useful
tips and free resources
then you can go ahead and check this
channel and subscribe it
one thing i love is that most of the
videos on this channel are short ones
okay so now i'll go back to the code
here i'll print the channel name now
i'll run the code again
and here the popup is
it says enter the channel name
i'll type
the english
scholar
online camp
ok
did you see that
my code was running in the background
and we were able to do what we wanted
now the next step is to type the enter
channel name in the search box
so to locate the search box
i'll capture this search
using the snipping tool
and i'll save it
i'll also capture the subscribe button
oh but for that first i need to
unsubscribe
okay now you can see the subscribe
button
now i will capture it using the snipping
tool again
and i'll save it
going back to the code
so use our important function again
to locate the search box and also locate
the subscribe button on the screen
locate center on screen
the file name
once i locate it i'll move to that
location so i'll use move to function
then i need to click on the search box
so i'll use the click method
after clicking i need to type the
channel name
so i'll use the write function and then
pass the channel name variable to that
function i'll just write the comment
okay then again to press enter
pi auto gui
dot
hotkey
enter i forgot one thing i also need to
click after i press ok on the pop-up
so i will have to add a click function
there also
here okay
now i want to show you one attribute for
the locate center on screen function
that is confidence
i'll type confidence
equal to
0.9
0.9 means 90 percent so 1 will be 100
percent this means that even if there is
a 90 match
then still it should consider
and to use this confidence attribute i
need to install a package called opencv
python
so i'll type pip install
opencv python
done
now let's run this code
did you see that i didn't touch my mouse
or keyboard it automatically opened
youtube and searched for the channel
so now
the next step would be to open the
channel
to do that i need to click on the
channel logo
so now you know what i'm going to do
open snipping tool
capture the logo
and save it
now come back to the code
i'll just copy paste this over here
and
change the file name
that's it
i'll put the necessary sleep of one
second for the next and final step
again i'll copy paste
change the file name
this image is the subscribe button that
we captured earlier
and that's it
now let's run it
oops
it did not search the logo this might
happen sometime due to pixel mismatch so
here the confidence was 90 percent still
it did not work so now i'd reduce it to
80 percent and try again
in such cases you'll have to decrease or
increase the confidence and keep trying
that's why the confidence attribute is
very important
now i think it should work let's see
bingo it worked i did not touch my mouse
or keyboard it automatically opened
youtube search for the channel name
then opened the channel
and subscribed it
so by this i have completed almost all
the functions of the pi auto gui package
and showed you how you can automate
anything on your computer using this
package
now i want you to follow this video step
by step
and then using your code
subscribe to my channel
and let me know in the comment section
if you learned something from this video
then please do not forget to subscribe
to my channel in order to be notified
about my latest videos thank you so much
for watching see you in my next video
Voir Plus de Vidéos Connexes
PyautoGUI: Three Great Uses
What Is Robotic Process Automation (RPA)? | Introduction To RPA | RPA Training | Simplilearn
How I Automated My Workflow with AI Agent Teams (NO-CODE)
Custom GPT for Business to Automate ANY repetitive task!
Automate PDF Invoices Data Transfer to Google Sheets with ChatGPT & Zapier | Tutorial
Smart click | Find Image | If Image Found - Automatic Mouse and Keyboard Membuat Robot/Script|PART 7
5.0 / 5 (0 votes)