Manipulate the screen without using a mouse or keyboard(PyAutoGUI)

English pages

Hello, I’m Yuki (@engineerblog_Yu), a student engineer.

Suddenly, does anyone want to become a psychic?

I really want to be a psychic.

I used to love to surprise my friends with magic tricks.

I wish I could be a psychic too, right? (laugh).

Here is a good news for everyone who wants to become a psychic.

Actually, anyone can become a psychic by reading this article!

You can operate a PC without even touching it!

So this time, I would like to introduce how to operate a PC using Python without using a mouse or keyboard.

This article is

Those who want to become psychic
Those who want to automate routine daily tasks
Those who are a pain in the ass

This article is especially recommended for such people.

Using the PyAutoGUI library introduced in this article, you can perform Excel and other PC operations by simply executing a code without having to touch the PC.

If you keep the code for operations that are used many times, the next time you want to perform the same operation, you can simply run the program to perform the same operation in an instant, which is very convenient.

If you want to work on Python projects or shorten your work with Python, this is a good book to take a look at.

(I am using JupiterNotebook to handle PyAutoGUI this time.)


Instalation of PyAutoGUI

!pip install pyautogui

Configuration for using PyAutoGUI

※If PyAutoGUI works well, you can skip this section.

System Preferences→Security and Privacy→Accessibility→Unlock→Check iTerm→Lock

to allow iTerm.

Now I would like to show you how to use iTerm.


import pyautogui as pgui
import time

Obtains the current position of the mouse


It outputs the current position of the mouse as Point(x=1929,y=20).

Obtain screen size


Size(width=1440,height=900) and outputs the screen size.

Click on the specified location

Executing the code here will automatically perform the operation of clicking on the position (x,y)=(190,12) for 1 second.,y=12,duration=1)

Let’s use it in conjunction with to get the current mouse position and then set the x and y coordinates to click on that position.

Take a screenshot


This one is a bit more complicated, but the code is to take a screenshot of a horizontal area of 100 and a vertical area of 200 from the location (0,100).


text entry

Let’s use pgui.write() after the text is ready to be input with and other methods above.


You can specify how many seconds to type a character after the text content.

The above code is an operation in which a character is typed every 0.3 seconds after the text content.

Automate keystrokes


For example,


You can also make multiple selections like this.


When the process is difficult to see


If it is difficult to see the process, you can insert a sleep in the TIME library to provide time between processes.

At the end

This is how to automate the process using PyAutoGUI.

In this article, we introduced PyAutoGUI, which can automate both mouse and keyboard.

The content is interesting and a dream come true for those who want to do simple daily tasks such as Excel without doing anything else.

If you are interested, you can take a course on Udemy.


Thank you for reading!