# Manipulate the screen without using a mouse or keyboard(PyAutoGUI)

スポンサーリンク

Hello, I’m Yuki (@engineerblog_Yu), a student engineer.

Suddenly, does anyone want to become a psychic?

I really want to be a psychic.

I used to love to surprise my friends with magic tricks.

I wish I could be a psychic too, right? (laugh).

Here is a good news for everyone who wants to become a psychic.

You can operate a PC without even touching it!

So this time, I would like to introduce how to operate a PC using Python without using a mouse or keyboard.

Those who want to become psychic
Those who want to automate routine daily tasks
Those who are a pain in the ass

Using the PyAutoGUI library introduced in this article, you can perform Excel and other PC operations by simply executing a code without having to touch the PC.

If you keep the code for operations that are used many times, the next time you want to perform the same operation, you can simply run the program to perform the same operation in an instant, which is very convenient.

If you want to work on Python projects or shorten your work with Python, this is a good book to take a look at.

(I am using JupiterNotebook to handle PyAutoGUI this time.)

スポンサーリンク

## Instalation of PyAutoGUI

!pip install pyautogui

## Configuration for using PyAutoGUI

※If PyAutoGUI works well, you can skip this section.

System Preferences→Security and Privacy→Accessibility→Unlock→Check iTerm→Lock

to allow iTerm.

Now I would like to show you how to use iTerm.

## import

import pyautogui as pgui
import time

## Obtains the current position of the mouse

pgui.position()

It outputs the current position of the mouse as Point(x=1929,y=20).

## Obtain screen size

pgui.size()

Size(width=1440,height=900) and outputs the screen size.

## Click on the specified location

Executing the code here will automatically perform the operation of clicking on the position (x,y)=(190,12) for 1 second.

pgui.click(x=190,y=12,duration=1)

Let’s use it in conjunction with pgui.click() to get the current mouse position and then set the x and y coordinates to click on that position.

## Take a screenshot

pgui.screenshot('image.png')

This one is a bit more complicated, but the code is to take a screenshot of a horizontal area of 100 and a vertical area of 200 from the location (0,100).

pgui.screenshot('image.png',region=(0,100,100,200))

## text entry

Let’s use pgui.write() after the text is ready to be input with pgui.click() and other methods above.

pgui.write('contents',0.3)

You can specify how many seconds to type a character after the text content.

The above code is an operation in which a character is typed every 0.3 seconds after the text content.

## Automate keystrokes

pgui.hotkey('key')

For example,

pgui.hotkey('return')

You can also make multiple selections like this.

pgui.hotkey('option','down')

## When the process is difficult to see

time.sleep(0.8)

If it is difficult to see the process, you can insert a sleep in the TIME library to provide time between processes.

## At the end

This is how to automate the process using PyAutoGUI.

In this article, we introduced PyAutoGUI, which can automate both mouse and keyboard.

The content is interesting and a dream come true for those who want to do simple daily tasks such as Excel without doing anything else.

If you are interested, you can take a course on Udemy.