PyautoGUI: Three Great Uses

Summary

In this 'Bald Guy DIY' tutorial, viewers are introduced to three innovative ways to utilize Python's 'Pi Auto GUI' library for computer automation. The host demonstrates how to automate mouse clicks, keyboard inputs, and screen captures, showcasing the library's capabilities to simplify repetitive online tasks. By following along with the script, viewers can learn to automate web interface interactions, such as navigating YouTube, with precision and ease, all through a user-friendly Python interface.

Takeaways

• 😀 The video introduces three cool things you can do with the 'Pi Auto GUI' library in Python to automate computer tasks.
• 📚 The speaker has been using Python for about a year and a half and finds it an incredible experience due to its ease of learning and vast library ecosystem.
• 🛠️ Python can be used for various tasks including spreadsheet automation, time calculations, web scraping, GUI development, and even controlling microcontrollers.
• 🔍 The 'Pi Auto GUI' library allows you to control the computer using keyboard and mouse inputs for automation.
• 💻 The demonstration uses VS Code and assumes Python is already installed; the library can be installed via 'pip install pyautogui'.
• 🖱️ The first script shown finds the location of the mouse cursor, which is essential for clicking actions in automation scripts.
• 🔒 To use mouse click functionality on a Mac, you may need to grant permissions to the script through system preferences.
• ⏱️ Adding a delay in scripts can be necessary to allow time for switching between windows or waiting for web pages to load.
• 🔑 The 'pyautogui.write' function can be used to simulate keyboard input, such as typing in a search bar.
• 🔑 The 'pyautogui.press' function can emulate key presses, including special keys like 'Enter'.
• 📸 'Pi Auto GUI' also includes screenshot functionality, allowing you to capture and work with images of the screen.
• 🔍 The library can locate the center of an image on the screen and perform actions like clicking based on that reference.

Q & A

What is the main focus of the 'Bald Guy DIY' video?

The video focuses on demonstrating three cool things that can be done with the 'Pi Auto GUI' library in Python to automate various tasks on a computer.

How long has the presenter been experimenting with Python as a coding language?

The presenter has been experimenting with Python for about a year and a half.

Why is Python considered an incredible experience according to the presenter?

Python is considered incredible because it is easy to learn, especially for those with coding background, and it has a vast number of libraries that can be used for various tasks with little experience.

What are some of the tasks the presenter has automated using Python?

The presenter has automated tasks such as spreadsheets, time calculations, web scraping, graphical user interfaces, and controlling microcontrollers.

What is the first script the presenter shows in the video for?

The first script is to find the location of the mouse cursor, which is necessary for clicking and double-clicking actions in automation.

Why is it necessary to install 'Pi ObjC Core Module' when running the first script?

The 'Pi ObjC Core Module' is a dependency required for 'Pi Auto GUI' to function properly, especially for accessing mouse coordinates.

What is the solution to the error encountered when trying to install the 'Pi ObjC Core Module'?

The solution is to follow the instructions provided in the documentation for the respective operating system, which includes running a specific command in the terminal.

How does the presenter handle the permission issue to click the mouse from an automated script on a Mac?

The presenter goes to System Preferences, unlocks the settings, grants permission to VS Code for mouse click functionality, and then locks the settings again.

What function is used in 'Pi Auto GUI' to simulate typing text on the screen?

The 'piautogui.write' function is used to simulate typing text on the screen.

What is the purpose of adding a delay (using 'sleep' function) in the script?

The delay is added to give the user enough time to switch over to the desired window or page before the script performs the mouse click action.

How can 'Pi Auto GUI' be used to take a screenshot of the screen?

The 'piautogui.screenshot' function is used, where the user specifies a file name for the screenshot image to be saved.

What additional step is required to use the screenshot functionality on a Mac?

On a Mac, the user needs to grant permission to VS Code to take screenshots, which is done through the System Preferences.

How does the presenter locate the center of an image on the screen using 'Pi Auto GUI'?

The presenter uses 'piautogui.locateCenterOnScreen' function to find the center of a specified image on the screen.

What is the presenter's suggestion for using the 'Pi Auto GUI' library?

The presenter suggests exploring the library, looking through the documentation, and using the provided functions to automate simple tasks or create more personalized tasks.

How can viewers provide feedback or suggest content for future videos?

Viewers can give the video a thumbs up, subscribe to the channel, send an email to the presenter, or leave a comment with their suggestions.

Outlines

00:00

😀 Introduction to Python Automation with Pi Auto GUI

The speaker introduces the video by welcoming viewers to 'Bald Guy DIY' and expressing enthusiasm for Python as a coding language, highlighting its ease of learning and the vast array of libraries available for various tasks. The speaker has been experimenting with Python for about a year and a half, automating spreadsheets, performing time calculations, web scraping, and creating graphical user interfaces. They emphasize the continuous growth of Python's capabilities due to new libraries and the potential for community contributions. The video's focus is on demonstrating three cool automation tasks using the 'Pi Auto GUI' library, which allows for computer control via keyboard and mouse inputs. The speaker uses VS Code for demonstration and guides viewers through installing 'Pi Auto GUI', troubleshooting installation issues, and showcasing basic mouse position tracking.

05:01

🛠️ Automating Mouse Clicks and Keyboard Inputs

The speaker proceeds to demonstrate how to automate mouse clicks using 'Pi Auto GUI' by first obtaining the mouse's current coordinates and then using these to execute a click at a specific location. They encounter a permission issue on a Mac, which is resolved by adjusting system preferences to allow script control of the mouse. The video also covers adding delays to accommodate window-switching and using the 'write' function to simulate keyboard input into fields like the YouTube search bar. The speaker further explains how to press keys programmatically, such as the 'enter' key, to complete actions like submitting a search query. They encourage viewers to consult the 'Pi Auto GUI' documentation for a comprehensive list of functions and parameters, noting that most tasks rely on specifying x and y coordinates for mouse actions.

10:02

📸 Utilizing Screenshot and Image Recognition Features

In the final part of the video, the speaker explores the screenshot functionality of 'Pi Auto GUI', which allows users to capture and save images of their screen. They discuss the necessity of granting permissions on a Mac to perform this action and demonstrate taking a screenshot of the search results for 'Bald Guy DIY'. The speaker then introduces the concept of using image recognition to automate clicks on specific elements within a screenshot, such as a logo. They guide viewers through cropping an image to focus on a particular element, saving it, and then using 'Pi Auto GUI' to locate and click on that element within the screenshot. The video concludes with the speaker encouraging viewers to explore the library's documentation, experiment with its features, and apply these automation techniques to their own projects. They invite feedback and suggestions for future video topics and remind viewers to subscribe for weekly DIY content.

Keywords

💡Python

Python is a high-level, interpreted programming language known for its readability and efficiency. In the context of the video, it is the coding language used for automation tasks. The script mentions that Python is 'pretty easy to learn' and has a vast library ecosystem, which makes it suitable for beginners and experienced programmers alike.

💡Automation

Automation refers to the process of making a task or system operate automatically. The video's theme revolves around automating computer tasks using Python. Examples include automating web interfaces, spreadsheets, and even controlling microcontrollers with Python scripts, showcasing the broad applications of automation.

💡Pi Auto Gui

Pi Auto Gui is a Python library specifically mentioned in the script for automating GUI interactions. It allows the control of a computer through keyboard and mouse inputs programmatically. The video demonstrates using Pi Auto Gui to automate tasks such as clicking, typing, and taking screenshots.

💡Mouse Cursor

The mouse cursor's location is a critical aspect of GUI automation. The video script explains how to find the cursor's position using Pi Auto Gui, which is essential for tasks like clicking at specific coordinates on the screen.

💡Coordinates

In the context of GUI automation, coordinates refer to the x and y positions on the screen. The script uses the term to describe how to determine the location for mouse clicks, which is fundamental for automating interactions with specific elements on a webpage or application.

💡Script

A script in this video refers to a Python program created to automate tasks. The script is used to demonstrate how to automate mouse clicks, keyboard inputs, and other GUI interactions using Pi Auto Gui.

💡Permissions

Permissions in the script refer to the access rights required for a script to perform certain actions, such as clicking the mouse or taking screenshots. The video explains how to grant these permissions to the Python script in order to execute automation tasks successfully.

💡Delay

Delay in the script is used to introduce a pause in the execution of the automation script. It is implemented to allow the user time to switch windows or for the webpage to load before the script proceeds with the next action, such as clicking the mouse.

💡Screenshot

A screenshot in the video is a digital image captured by the script using Pi Auto Gui. It is used to capture the current state of the screen or specific elements for further processing or analysis, such as locating a particular logo for automated clicking.

💡Documentation

Documentation in the script refers to the official guide or reference material for Pi Auto Gui. It is mentioned as a resource for understanding the different functions and parameters available for use in automation scripts, which is crucial for expanding the capabilities of the automation tasks.

💡Hotkey

A hotkey is a keyboard shortcut that performs a specific action when pressed. The script mentions the 'press' function of Pi Auto Gui, which can emulate hotkeys, such as pressing 'Enter' after typing a search term, to enhance the automation process.

Highlights

Introduction to using 'pi auto gui' in Python for automating computer tasks.

Python's ease of learning and vast library support make it ideal for beginners and experienced coders alike.

Demonstration of automating spreadsheets, time calculations, web scraping, and GUIs using Python.

The potential to control microcontrollers with Python through platforms like Raspberry Pi or Micro Python.

Explanation of how to install 'pi auto gui' using pip in a Python environment.

The necessity of installing additional libraries for 'pi auto gui' on a Mac.

Creating a script to find the location of the mouse cursor using 'pi auto gui'.

How to handle errors related to permissions for mouse click automation on a Mac.

Demonstration of clicking at specific coordinates on the screen using 'pi auto gui'.

Use of the 'sleep' function to delay actions and allow for window switching.

Writing text into input fields using 'pi auto gui' and handling keyboard controls.

Pressing keys programmatically with 'pi auto gui' to enhance automation tasks.

Using 'pi auto gui' to take screenshots and save them as image files.

Handling permissions for screenshot functionality on different operating systems.

Using image recognition to automate clicking on specific UI elements.

Combining text input, key presses, and image recognition for complex automation tasks.

Encouragement to explore 'pi auto gui' documentation for a wide range of automation capabilities.

Invitation for viewers to subscribe and provide feedback for future video content.

Transcripts

welcome to bald guy diy in this video

i'm going to show you three cool things

that you can do with pi

auto gui in python in order to automate

a lot of tasks

on your computer

[Music]

[Applause]

it's been about a year and a half now

since i started playing with python as a

coding language

and i must tell you it is a really

incredible experience

not only because python is pretty easy

to learn if you have any coding

background in any other language or

even if you're just starting fresh but

it's also incredible how many libraries

exist for it and different things that

you can start playing with

very quickly without having a lot of

experience in them i've done all sorts

of things

including automation of spreadsheets

different time calculations web scraping

i've used it for graphical user

interfaces as you've seen some of my

videos before

and even putting them out to things like

the raspberry pi or even micro python

to control microcontrollers as well so

there's just an incredible amount of

things that you can do

and the list is always growing because

there's always new libraries being

created

if there isn't something already

existing for python there probably is

something already in development

or with a little bit of time and effort

you could create your own modules

that could benefit the whole community

one of the things that i like to

automate

is web interfaces to get through some of

those login processes

and functionalities that you do over and

over again in the same website

in this video i'm going to show you

three different things that are pretty

cool that you can do with a library

called

pi auto gui which lets you control your

computer the way that you would through

a keyboard or mouse

in order to automate a lot of processes

so without further ado

let's get into it for the sake of this

demonstration i'm using vs

code and i'm assuming you already have

python installed you're just going to

need to do pip

install pi auto gui and let that run

through the installation process

i'm using a mac to demonstrate here

today but on a windows machine it's even

easier because there's not usually any

other additional libraries you need to

install

on the first script i'm going to show

you how to find the location of your

mouse

cursor because you're going to need to

be able to do that in order to

click and double click so as we look

through mouse functions i'm just going

to create a little true loop

after importing piyo gui and in that

true loop i'm going to print

pi auto gui dot position which is the

location

of the mouse at any given time now by

using the true loop it's going to

constantly refresh

and constantly print an updated position

when i try to run that you see here it

says you must first install the pi

objc core module and if you try to do

that from the terminal prompt here

you will find there's some problems with

it it says that it's already installed

but yet it'll still give you the same

error over and over again

so the solution is to click on the link

there that it refers you to the

documentation

and then scroll down here there's

instructions that can be used for

windows

and linux and also mac os which is what

i'm using here so i'm going to find the

command that it

suggests here and going to copy that

over and open a terminal window

and then just paste it in there

and as you can see again it's going to

run through the installation process

and there's quite a bit pyro gui is

doing a lot of

conversion to other commands and so it

does take a

minute or so to install and then when we

run it again

now you can see in this bottom left hand

corner that as i move the mouse around

you get new x and y coordinates as you

move it around the screen

now this is good for the entire desktop

area so depending on your resolution

it will be different so now if i was to

try to find

a particular spot like a search box on

youtube

i'd simply have to put my mouse in that

spot and then

option tab over or on a windows machine

alt tab over

and you'd be able to see the current x

and y coordinates of your mouse position

now that we have those coordinates it's

easy to create a little script that will

actually click the mouse at that

location

you will also need to stop that mouse

position script from running because

it'll just run forever because of the

while true loop so you just need to

click the kill button which is the trash

can

and then it will stop the script and let

you know that you're back to the command

prompt ready to run again

on my new script i'm going to import pi

auto gui again and now i'm just going to

use pi auto gui dot click

and put in the coordinates that we found

now the coordinates we had was 6 13 and

93

but i'm just going to round them to the

nearest 10 which should be just fine

now if i was to test it out you can see

that once again we get an error

and that's telling us that we don't yet

have the permission to click the mouse

from an automated script so if you click

the preferences and then click the lock

here to unlock it

on a mac you'll find that you'll now

have access to give vs code permission

in order to be able to use the mouse

click function

so after giving it the permission and

then locking it up again i can close

that

re-run it

and now you can see it does click at the

top where the search bar was but as you

can tell

if i was trying to do this with the

youtube page that i had previously

loaded

by the time i ran this file and then

clicked it would be too late

because it would have already moved the

mouse to that position and clicked

so i'm going to add some delay that

gives me time to switch over the windows

i'm simply going to from the time

library import the sleep function

and then i'm going to add five seconds

of sleep which will be giving me enough

time to switch over to the youtube page

so if we run this again and then i do my

command tab or on windows

alt tab it gives it time to pop over and

then click the mouse

in the search bar as you can see there

to do double click right click and lots

of other little things so if you consult

the documentation you'll find all of the

different parameters that are required

for those things

mostly though they work on the same idea

of x and y coordinates

so let's add the ability to type

something in that search bar

well all we need to do is use the

piotogui.write function

and then within quotations put what we

want to write out in a shameless plug

for my channel let's just put in bald

guy diy

and then run it again and see what we

get that's really all it takes to do

keyboard control and on the mac you

don't have to do anything extra for that

it doesn't generate any error messages

so we'll simply tab back to our youtube

page

and we'll let it click in the box it

also types in our search term

you can of course click it manually or

you could find the location of the mouse

on the search button and click into the

same function we already explored

now if you look through the

documentation on the pi auto gui

reference pages you'll see here a lot of

different functions things that you can

do

there's a function called press which

emulates pressing a key

and if you choose the keyboard key you

can see all of the keys that are

available to be pressed

and also what to call them in the case

of the enter button if we want to be

able to press

enter after our search criteria it's

just enter within quotes all lowercase

to make the functionality of our search

a little bit better i'm just going to do

a pile of gui dot press

and that's going to press the enter key

when we're done

now you can see it types in there and

presses enter which brings us to the

results

of my channel there are a bunch of other

things that you can do there's hotkey

functionality so you can program

control and shift together or control

shift escape or

as their as the example is here or even

do other sequences where you would hold

one key down

press some other keys and then let that

key up as you can see here

that's using the key down function and

then while holding that key down

you're gonna do key presses and then at

the end you'll do a key up which is

simulating releasing

that key the documentation gives you so

many different cases

you can find pretty much anything you'd

need the last thing i want to show off

is how you can use the screenshot

functionality in order to take snapshots

of your screen

and then locate different elements of it

so the first thing i'm going to do is

the piotogui.screenshot function

and then all you need to do is specify a

file name

and that's going to be what you want to

call the screenshot image that you're

taking

in this case i'm just going to call it

screen.png it's going to save in the

same folder as your python

script now you can see immediately it

also gives you one more error message

saying that you don't have the rights to

do that yet

so just like before we're going to

select it and give permission for vs

code to be able to do that

on the mac that's required on windows

it's not

and then we're going to run it again

after reloading vs code because it tells

you there you need to quit it first

after running that script you can see it

creates a screenshot on my desktop

called screen.png and now i can open

that up and what i'm left with here is

actually an

image of what i had not the actual vs

code interface

this image can be edited so if we were

to take our previous look here

at our youtube page that we loaded and

searched for bald guy diy

we can now add a screenshot function

here and

save that image to a png file so i'm

going to call it bald guy diy channel

dot png

i'm going to add a couple more seconds

of sleep here in order to let the search

results load after enter is pressed now

if we run it

and clear that up again we're going to

see after five seconds

it types in the name and then it does a

search and then with the search results

it creates a screenshot now if we were

to open that in an image editor i'm

using here

which is a free software and then select

the part of it that we want to recognize

we can now give pyo auto gui an image to

look for

something to reference i'm going to crop

the whole image to the selection that i

made

and then export just that as a new file

which is going to be the clue that the

software looks for

here i'm going to just call it

boldguylogo.png save it to the desktop

as well

and once the exports done i can now

reference it in the code and it'll be

able to find it

in order to do this we're going to use

something called pi auto gui dot

locate center on screen which is going

to load locate the center of this image

and you create two new variables x and y

and make them equal to the location of

that logo once pi auto gui can find it

it's going to save those values of the x

and y

and now all we need to do is use them

with the click function

in order to click on the location of

that logo right in the middle now if we

were to run it and load our youtube page

again

after clearing the box five seconds

later of course it will type in

the name of the channel it'll hit enter

when the results load it now

also clicks on the image which you can

tell it worked because it loads my first

video in queue

there are so many other functions in the

documentation but this is a great start

and i hope you'll check it out so there

you have

three simple ideas but also incredibly

powerful

to automate your computer with pi auto

gui

as usual the ideas i've shared are

building blocks so that you can take

them

and turn them into more creative and

personalized tasks

that you can do on your own computer so

play with this library

look through the documentation and see

all of the features that are available

and i'm sure you'll find some use for it

on even simple little things that you

need automated

in the future if you like this kind of

content please give the video a thumbs

up

and subscribe to the channel as i post a

new video every weekend

on all sorts of diy interests if you

want to send me an email

my information is in the description

below or leave a comment

tell me what kind of videos you'd like

to see in the next weeks and months

and i'll use them as inspiration for

future content

until next time in all your diy projects

hopefully those that you can automate a

little don't be afraid

to be balder