PyautoGUI: Three Great Uses
Summary
TLDRIn this 'Bald Guy DIY' tutorial, viewers are introduced to three innovative ways to utilize Python's 'Pi Auto GUI' library for computer automation. The host demonstrates how to automate mouse clicks, keyboard inputs, and screen captures, showcasing the library's capabilities to simplify repetitive online tasks. By following along with the script, viewers can learn to automate web interface interactions, such as navigating YouTube, with precision and ease, all through a user-friendly Python interface.
Takeaways
- 😀 The video introduces three cool things you can do with the 'Pi Auto GUI' library in Python to automate computer tasks.
- 📚 The speaker has been using Python for about a year and a half and finds it an incredible experience due to its ease of learning and vast library ecosystem.
- 🛠️ Python can be used for various tasks including spreadsheet automation, time calculations, web scraping, GUI development, and even controlling microcontrollers.
- 🔍 The 'Pi Auto GUI' library allows you to control the computer using keyboard and mouse inputs for automation.
- 💻 The demonstration uses VS Code and assumes Python is already installed; the library can be installed via 'pip install pyautogui'.
- 🖱️ The first script shown finds the location of the mouse cursor, which is essential for clicking actions in automation scripts.
- 🔒 To use mouse click functionality on a Mac, you may need to grant permissions to the script through system preferences.
- ⏱️ Adding a delay in scripts can be necessary to allow time for switching between windows or waiting for web pages to load.
- 🔑 The 'pyautogui.write' function can be used to simulate keyboard input, such as typing in a search bar.
- 🔑 The 'pyautogui.press' function can emulate key presses, including special keys like 'Enter'.
- 📸 'Pi Auto GUI' also includes screenshot functionality, allowing you to capture and work with images of the screen.
- 🔍 The library can locate the center of an image on the screen and perform actions like clicking based on that reference.
Q & A
What is the main focus of the 'Bald Guy DIY' video?
-The video focuses on demonstrating three cool things that can be done with the 'Pi Auto GUI' library in Python to automate various tasks on a computer.
How long has the presenter been experimenting with Python as a coding language?
-The presenter has been experimenting with Python for about a year and a half.
Why is Python considered an incredible experience according to the presenter?
-Python is considered incredible because it is easy to learn, especially for those with coding background, and it has a vast number of libraries that can be used for various tasks with little experience.
What are some of the tasks the presenter has automated using Python?
-The presenter has automated tasks such as spreadsheets, time calculations, web scraping, graphical user interfaces, and controlling microcontrollers.
What is the first script the presenter shows in the video for?
-The first script is to find the location of the mouse cursor, which is necessary for clicking and double-clicking actions in automation.
Why is it necessary to install 'Pi ObjC Core Module' when running the first script?
-The 'Pi ObjC Core Module' is a dependency required for 'Pi Auto GUI' to function properly, especially for accessing mouse coordinates.
What is the solution to the error encountered when trying to install the 'Pi ObjC Core Module'?
-The solution is to follow the instructions provided in the documentation for the respective operating system, which includes running a specific command in the terminal.
How does the presenter handle the permission issue to click the mouse from an automated script on a Mac?
-The presenter goes to System Preferences, unlocks the settings, grants permission to VS Code for mouse click functionality, and then locks the settings again.
What function is used in 'Pi Auto GUI' to simulate typing text on the screen?
-The 'piautogui.write' function is used to simulate typing text on the screen.
What is the purpose of adding a delay (using 'sleep' function) in the script?
-The delay is added to give the user enough time to switch over to the desired window or page before the script performs the mouse click action.
How can 'Pi Auto GUI' be used to take a screenshot of the screen?
-The 'piautogui.screenshot' function is used, where the user specifies a file name for the screenshot image to be saved.
What additional step is required to use the screenshot functionality on a Mac?
-On a Mac, the user needs to grant permission to VS Code to take screenshots, which is done through the System Preferences.
How does the presenter locate the center of an image on the screen using 'Pi Auto GUI'?
-The presenter uses 'piautogui.locateCenterOnScreen' function to find the center of a specified image on the screen.
What is the presenter's suggestion for using the 'Pi Auto GUI' library?
-The presenter suggests exploring the library, looking through the documentation, and using the provided functions to automate simple tasks or create more personalized tasks.
How can viewers provide feedback or suggest content for future videos?
-Viewers can give the video a thumbs up, subscribe to the channel, send an email to the presenter, or leave a comment with their suggestions.
Outlines
😀 Introduction to Python Automation with Pi Auto GUI
The speaker introduces the video by welcoming viewers to 'Bald Guy DIY' and expressing enthusiasm for Python as a coding language, highlighting its ease of learning and the vast array of libraries available for various tasks. The speaker has been experimenting with Python for about a year and a half, automating spreadsheets, performing time calculations, web scraping, and creating graphical user interfaces. They emphasize the continuous growth of Python's capabilities due to new libraries and the potential for community contributions. The video's focus is on demonstrating three cool automation tasks using the 'Pi Auto GUI' library, which allows for computer control via keyboard and mouse inputs. The speaker uses VS Code for demonstration and guides viewers through installing 'Pi Auto GUI', troubleshooting installation issues, and showcasing basic mouse position tracking.
🛠️ Automating Mouse Clicks and Keyboard Inputs
The speaker proceeds to demonstrate how to automate mouse clicks using 'Pi Auto GUI' by first obtaining the mouse's current coordinates and then using these to execute a click at a specific location. They encounter a permission issue on a Mac, which is resolved by adjusting system preferences to allow script control of the mouse. The video also covers adding delays to accommodate window-switching and using the 'write' function to simulate keyboard input into fields like the YouTube search bar. The speaker further explains how to press keys programmatically, such as the 'enter' key, to complete actions like submitting a search query. They encourage viewers to consult the 'Pi Auto GUI' documentation for a comprehensive list of functions and parameters, noting that most tasks rely on specifying x and y coordinates for mouse actions.
📸 Utilizing Screenshot and Image Recognition Features
In the final part of the video, the speaker explores the screenshot functionality of 'Pi Auto GUI', which allows users to capture and save images of their screen. They discuss the necessity of granting permissions on a Mac to perform this action and demonstrate taking a screenshot of the search results for 'Bald Guy DIY'. The speaker then introduces the concept of using image recognition to automate clicks on specific elements within a screenshot, such as a logo. They guide viewers through cropping an image to focus on a particular element, saving it, and then using 'Pi Auto GUI' to locate and click on that element within the screenshot. The video concludes with the speaker encouraging viewers to explore the library's documentation, experiment with its features, and apply these automation techniques to their own projects. They invite feedback and suggestions for future video topics and remind viewers to subscribe for weekly DIY content.
Mindmap
Keywords
💡Python
💡Automation
💡Pi Auto Gui
💡Mouse Cursor
💡Coordinates
💡Script
💡Permissions
💡Delay
💡Screenshot
💡Documentation
💡Hotkey
Highlights
Introduction to using 'pi auto gui' in Python for automating computer tasks.
Python's ease of learning and vast library support make it ideal for beginners and experienced coders alike.
Demonstration of automating spreadsheets, time calculations, web scraping, and GUIs using Python.
The potential to control microcontrollers with Python through platforms like Raspberry Pi or Micro Python.
Explanation of how to install 'pi auto gui' using pip in a Python environment.
The necessity of installing additional libraries for 'pi auto gui' on a Mac.
Creating a script to find the location of the mouse cursor using 'pi auto gui'.
How to handle errors related to permissions for mouse click automation on a Mac.
Demonstration of clicking at specific coordinates on the screen using 'pi auto gui'.
Use of the 'sleep' function to delay actions and allow for window switching.
Writing text into input fields using 'pi auto gui' and handling keyboard controls.
Pressing keys programmatically with 'pi auto gui' to enhance automation tasks.
Using 'pi auto gui' to take screenshots and save them as image files.
Handling permissions for screenshot functionality on different operating systems.
Using image recognition to automate clicking on specific UI elements.
Combining text input, key presses, and image recognition for complex automation tasks.
Encouragement to explore 'pi auto gui' documentation for a wide range of automation capabilities.
Invitation for viewers to subscribe and provide feedback for future video content.
Transcripts
welcome to bald guy diy in this video
i'm going to show you three cool things
that you can do with pi
auto gui in python in order to automate
a lot of tasks
on your computer
[Music]
[Applause]
it's been about a year and a half now
since i started playing with python as a
coding language
and i must tell you it is a really
incredible experience
not only because python is pretty easy
to learn if you have any coding
background in any other language or
even if you're just starting fresh but
it's also incredible how many libraries
exist for it and different things that
you can start playing with
very quickly without having a lot of
experience in them i've done all sorts
of things
including automation of spreadsheets
different time calculations web scraping
i've used it for graphical user
interfaces as you've seen some of my
videos before
and even putting them out to things like
the raspberry pi or even micro python
to control microcontrollers as well so
there's just an incredible amount of
things that you can do
and the list is always growing because
there's always new libraries being
created
if there isn't something already
existing for python there probably is
something already in development
or with a little bit of time and effort
you could create your own modules
that could benefit the whole community
one of the things that i like to
automate
is web interfaces to get through some of
those login processes
and functionalities that you do over and
over again in the same website
in this video i'm going to show you
three different things that are pretty
cool that you can do with a library
called
pi auto gui which lets you control your
computer the way that you would through
a keyboard or mouse
in order to automate a lot of processes
so without further ado
let's get into it for the sake of this
demonstration i'm using vs
code and i'm assuming you already have
python installed you're just going to
need to do pip
install pi auto gui and let that run
through the installation process
i'm using a mac to demonstrate here
today but on a windows machine it's even
easier because there's not usually any
other additional libraries you need to
install
on the first script i'm going to show
you how to find the location of your
mouse
cursor because you're going to need to
be able to do that in order to
click and double click so as we look
through mouse functions i'm just going
to create a little true loop
after importing piyo gui and in that
true loop i'm going to print
pi auto gui dot position which is the
location
of the mouse at any given time now by
using the true loop it's going to
constantly refresh
and constantly print an updated position
when i try to run that you see here it
says you must first install the pi
objc core module and if you try to do
that from the terminal prompt here
you will find there's some problems with
it it says that it's already installed
but yet it'll still give you the same
error over and over again
so the solution is to click on the link
there that it refers you to the
documentation
and then scroll down here there's
instructions that can be used for
windows
and linux and also mac os which is what
i'm using here so i'm going to find the
command that it
suggests here and going to copy that
over and open a terminal window
and then just paste it in there
and as you can see again it's going to
run through the installation process
and there's quite a bit pyro gui is
doing a lot of
conversion to other commands and so it
does take a
minute or so to install and then when we
run it again
now you can see in this bottom left hand
corner that as i move the mouse around
you get new x and y coordinates as you
move it around the screen
now this is good for the entire desktop
area so depending on your resolution
it will be different so now if i was to
try to find
a particular spot like a search box on
youtube
i'd simply have to put my mouse in that
spot and then
option tab over or on a windows machine
alt tab over
and you'd be able to see the current x
and y coordinates of your mouse position
now that we have those coordinates it's
easy to create a little script that will
actually click the mouse at that
location
you will also need to stop that mouse
position script from running because
it'll just run forever because of the
while true loop so you just need to
click the kill button which is the trash
can
and then it will stop the script and let
you know that you're back to the command
prompt ready to run again
on my new script i'm going to import pi
auto gui again and now i'm just going to
use pi auto gui dot click
and put in the coordinates that we found
now the coordinates we had was 6 13 and
93
but i'm just going to round them to the
nearest 10 which should be just fine
now if i was to test it out you can see
that once again we get an error
and that's telling us that we don't yet
have the permission to click the mouse
from an automated script so if you click
the preferences and then click the lock
here to unlock it
on a mac you'll find that you'll now
have access to give vs code permission
in order to be able to use the mouse
click function
so after giving it the permission and
then locking it up again i can close
that
re-run it
and now you can see it does click at the
top where the search bar was but as you
can tell
if i was trying to do this with the
youtube page that i had previously
loaded
by the time i ran this file and then
clicked it would be too late
because it would have already moved the
mouse to that position and clicked
so i'm going to add some delay that
gives me time to switch over the windows
i'm simply going to from the time
library import the sleep function
and then i'm going to add five seconds
of sleep which will be giving me enough
time to switch over to the youtube page
so if we run this again and then i do my
command tab or on windows
alt tab it gives it time to pop over and
then click the mouse
in the search bar as you can see there
to do double click right click and lots
of other little things so if you consult
the documentation you'll find all of the
different parameters that are required
for those things
mostly though they work on the same idea
of x and y coordinates
so let's add the ability to type
something in that search bar
well all we need to do is use the
piotogui.write function
and then within quotations put what we
want to write out in a shameless plug
for my channel let's just put in bald
guy diy
and then run it again and see what we
get that's really all it takes to do
keyboard control and on the mac you
don't have to do anything extra for that
it doesn't generate any error messages
so we'll simply tab back to our youtube
page
and we'll let it click in the box it
also types in our search term
you can of course click it manually or
you could find the location of the mouse
on the search button and click into the
same function we already explored
now if you look through the
documentation on the pi auto gui
reference pages you'll see here a lot of
different functions things that you can
do
there's a function called press which
emulates pressing a key
and if you choose the keyboard key you
can see all of the keys that are
available to be pressed
and also what to call them in the case
of the enter button if we want to be
able to press
enter after our search criteria it's
just enter within quotes all lowercase
to make the functionality of our search
a little bit better i'm just going to do
a pile of gui dot press
and that's going to press the enter key
when we're done
now you can see it types in there and
presses enter which brings us to the
results
of my channel there are a bunch of other
things that you can do there's hotkey
functionality so you can program
control and shift together or control
shift escape or
as their as the example is here or even
do other sequences where you would hold
one key down
press some other keys and then let that
key up as you can see here
that's using the key down function and
then while holding that key down
you're gonna do key presses and then at
the end you'll do a key up which is
simulating releasing
that key the documentation gives you so
many different cases
you can find pretty much anything you'd
need the last thing i want to show off
is how you can use the screenshot
functionality in order to take snapshots
of your screen
and then locate different elements of it
so the first thing i'm going to do is
the piotogui.screenshot function
and then all you need to do is specify a
file name
and that's going to be what you want to
call the screenshot image that you're
taking
in this case i'm just going to call it
screen.png it's going to save in the
same folder as your python
script now you can see immediately it
also gives you one more error message
saying that you don't have the rights to
do that yet
so just like before we're going to
select it and give permission for vs
code to be able to do that
on the mac that's required on windows
it's not
and then we're going to run it again
after reloading vs code because it tells
you there you need to quit it first
after running that script you can see it
creates a screenshot on my desktop
called screen.png and now i can open
that up and what i'm left with here is
actually an
image of what i had not the actual vs
code interface
this image can be edited so if we were
to take our previous look here
at our youtube page that we loaded and
searched for bald guy diy
we can now add a screenshot function
here and
save that image to a png file so i'm
going to call it bald guy diy channel
dot png
i'm going to add a couple more seconds
of sleep here in order to let the search
results load after enter is pressed now
if we run it
and clear that up again we're going to
see after five seconds
it types in the name and then it does a
search and then with the search results
it creates a screenshot now if we were
to open that in an image editor i'm
using here
which is a free software and then select
the part of it that we want to recognize
we can now give pyo auto gui an image to
look for
something to reference i'm going to crop
the whole image to the selection that i
made
and then export just that as a new file
which is going to be the clue that the
software looks for
here i'm going to just call it
boldguylogo.png save it to the desktop
as well
and once the exports done i can now
reference it in the code and it'll be
able to find it
in order to do this we're going to use
something called pi auto gui dot
locate center on screen which is going
to load locate the center of this image
and you create two new variables x and y
and make them equal to the location of
that logo once pi auto gui can find it
it's going to save those values of the x
and y
and now all we need to do is use them
with the click function
in order to click on the location of
that logo right in the middle now if we
were to run it and load our youtube page
again
after clearing the box five seconds
later of course it will type in
the name of the channel it'll hit enter
when the results load it now
also clicks on the image which you can
tell it worked because it loads my first
video in queue
there are so many other functions in the
documentation but this is a great start
and i hope you'll check it out so there
you have
three simple ideas but also incredibly
powerful
to automate your computer with pi auto
gui
as usual the ideas i've shared are
building blocks so that you can take
them
and turn them into more creative and
personalized tasks
that you can do on your own computer so
play with this library
look through the documentation and see
all of the features that are available
and i'm sure you'll find some use for it
on even simple little things that you
need automated
in the future if you like this kind of
content please give the video a thumbs
up
and subscribe to the channel as i post a
new video every weekend
on all sorts of diy interests if you
want to send me an email
my information is in the description
below or leave a comment
tell me what kind of videos you'd like
to see in the next weeks and months
and i'll use them as inspiration for
future content
until next time in all your diy projects
hopefully those that you can automate a
little don't be afraid
to be balder
関連動画をさらに表示
PyAutoGUI - Locate anything on your screen | Simple Pyautogui project
User Operating System Interface
Smart click | Find Image | If Image Found - Automatic Mouse and Keyboard Membuat Robot/Script|PART 7
Tic Tac Toe Game In Python | Python Project for Beginners
Optimization: Do this to your UI in Unity & Thing to be careful about...
Edit Any PDF With Accuracy With This Free Tool 🔥✅🤯 #pc #trickblower #shorts
5.0 / 5 (0 votes)