【Python(Automation)】How to do web scraping using Selenium

English pages

Hi, I’m Yuki (@engineerblog_Yu), a student engineer.

Are you interested in getting a scraping project in Python?

In this article, I would like to explain Selenium for those who want to do scraping in Python.

With Selenium, you can automatically open a browser and collect information on a site.

If you keep the code for operations that are used many times, the next time you want to perform the same operation, you can simply run the program and perform the operation in an instant, which is very convenient.

If you want to work on Python projects or shorten your workload with Python, this is a good book to take a look at.

Scraping in particular is rich in projects, so if you want to get a project in Python, this is a good place to start.


Basic Scraping Flow

The basic flow of scraping is

1, Check the HTML information of the website from the verification.

2, Store the desired information as a list using find_elements_by~.

3, Make a table using Pandas

4, Output as a CSV file or Excel file

The following is a list of the steps.


from selenium import webdriver

How to open browser

This code will automatically open Google Chrome.

browser = webdriver.Chrome()

Open the web site

Put the URL of the website you want to open in the brackets of browser.get() and execute it.


Some websites may prohibit scraping, so please confirm this on your own.

Obtaining the id of a website

elem = browser.find_element_by_id('id')

An id is an HTML tag on a website.

This may be difficult because it requires knowledge of HTML as well as knowledge of Python.

If you are using Google Chrome, you can view the HTML information of a website by right-clicking and selecting “Verify”.

Please type in the id tag that contains the information you want to extract from the HTML information as a code.

(If there are multiple id tags with the same name, the information of the first id tag will be stored in the elem.)

If you want to store the information of multiple id tags with the same name as an array, type

elems = browser.find.elements_by_id('id')

Let’s give it as follows.

I think we can make the array store as many id tags as elms[0],elms[1],,,,and so on.

Get the class of the website

If you want to get the class in the same way, give it this way.

elem = browser.find.element_by_class_name('class')
elems = browser.find.elements_by_class_name('class')

Output element as text

After storing the information in the elem as shown above, you can use the text method to output the text.


If you execute this code, the text will be stored in the array named values in order.

You can use the append method to put the text into the array named values in order.

values = []

for elem in elems:
    value = elem.text

How to make a list into a table

import pandas as pd

df = pd.DataFrame()

Output as CSV or Excel file


At the end

In this article, we introduced Selenium, which is used for web scraping.

If you are interested, you can take a course on Udemy.


Thank you for reading!