Web Scraping with Selenium

To understand the basics of HTML.
To explore the web page structure and usage of developer tools.
To make HTTP requests and get HTML responses.
To get specific structured information using beautifulsoup.

Scroll down to load more content when reached at the end of the page.
Click the next button to see the next page of available offers on an e-commerce website.
Click the button to view complete details of a comment or user profile to apply scraping.

Google Shopping Insights loads the data at runtime so any attempt to extract data using requests package will be responded to with an empty response.

Selenium : To download selenium package, execute the below pip command in terminal:

1pip install selenium 

Selenium Drivers: Web drivers enable python to control the browser via OS-level interactions. Web drivers use the browser's built-in support for the automation process so, in order to control the browser, the web-driver must be installed and should be accessible via the PATH variable of the operating system (only required in case of manual installation).

Safari 10 on OS X El Capitan and macOS Sierra have built-in support for the automation driver. This guide contains snippets to interact with popular web-drivers, though Safari is being used as a default browser throughout this guide.

Other browsers like UC, Netscape etc., cannot be used for automation. The Selenium-RC (remote-control) tool can control browsers via injecting its own JavaScript code and can be used for UI testing.

Import Dependencies and Create Driver Instance: The initial step is to create an object of webdriver for particular browser by importing it from selenium module as:

1from selenium import webdriver                    # Import module 
2from selenium.webdriver.common.keys import Keys   # For keyboard keys 
3import time                                       # Waiting function  
4URL = 'https://shopping.thinkwithgoogle.com'      # Define URL 
5browser = webdriver.Safari()                       # Create driver object means open the browser 

By default, the automation control is disabled in safari and it needs to be enabled for automation environment otherwise it will raise SessionNotCreatedException. So, enable the Develop option under the advanced settings in Safari preferences.

Apply Wait and Search : Follow the steps to apply search:

Open the link.
Wait for the website to load.
Find the id of the search element using its 'subjectInput' id.

There's also a hidden input tag which is not required. So, use find_elements to get the list of all elements with matched searched criteria and use the index to access it.

Provide input using send_keys.
Wait for the result to appear.
To press enter and proceed.

1browser.get('https://shopping.thinkwithgoogle.com')       # 1 
2time.sleep(2)                                             # 2 
3search = browser.find_elements_by_id('subjectInput')[1]   # 3 
4# find_elements will give us the list of all elements with id as subjectInput 
5search.send_keys('Google Pixel 3')                        # 4 
6time.sleep(2)                                             # 5 
7search.send_keys(Keys.ENTER)                              # 6 

1# Firefox 
2firefoxBrowser = webdriver.Firefox(executable_path=FIREFOX_GOCKO_DRIVER_PATH)  # Chrome 
3chromeBrowser = webdriver.Chrome(executable_path=CHROMEDRIVER_PATH) 

Download Files: In order to download files, locate Download all data div tag using si-button-data download-all class value.

1time.sleep(2) # Wait for button to appear 
2browser.find_element_by_class_name('si-button-data download-all').click() 

Extract Data from Elements: To download interest data by location, first locate the ul list tag by its class value and then fetch data of all list items as:

1data = browser.find_element_by_class_name('content content-breakpoint-gt-md') 
2dataList = data.find_elements_by_tag_name('li') 
3for item in dataList: 
4    text = item.text 
5    print(text) 

find_element_by_id: Use id to find an element.
find_element_by_name: Use name to find an element.
find_element_by_xpath: Use xpath to find an elements.
find_element_by_link_text: Use text value of a link to find element.
find_element_by_partial_link_text: Find element by matching some part of a hyper link text(anchor tag).
find_element_by_tag_name: Use tag name to find an element.
find_element_by_class_name: Use value of class attribute to find an element.
find_element_by_css_selector: Use CSS selector for id, class to find element. Or use find_element with BY locater as:

1search = browser.find_element(By.ID,'subjectInput')[1]

1searchList = browser.find_elements(By.ID,'subjectInput')[1] 

/ : Select node from the root. /html/body/div[1] will find the first div.

//: Select node from the current node. //form[1] will find the first form element.
[@attributename='value']: Use the syntax to find the node with the required value of the attribute.

1<html> 
2 <body> 
3   <div class = "content-login"> 
4     <form id="loginForm"> 
5         <div> 
6            <input type="text" name="Email" value="Email Address:"> 
7            <input type="password" name="Password"value="Password:"> 
8         </div> 
9        <button type="submit">Submit</button> 
10     </form> 
11   </div> 
12 </body> 
13<html> 

Read more about XPath to combine multiple attributes or use supported function.

1from selenium import webdriver 
2from selenium.webdriver.firefox.options import Options  
3options = Options() 
4# options.headless = True  # older webdriver versions 
5options.set_headless(True) # newer webdriver versions  
6# Firefox 
7firefoxBrowser = webdriver.Firefox(options=options, executable_path=FIREFOX_GOCKO_DRIVER_PATH)  
8# Chrome 
9chromeBrowser = webdriver.Chrome(CHROMEDRIVER_PATH, chrome_options=options) 

At the time of writing this guide, Headless mode is not supported by Safari.

To optimize the code, use WebDriverWait instead of time.sleep. It will automatically check and proceed when searched element is visible:

1driver = webdriver.Safari() 
2browser.get('https://shopping.thinkwithgoogle.com') 
3try: # proceed if element is found within 3 seconds otherwise will raise TimeoutException 
4    element = WebDriverWait(browser, 3).until(EC.presence_of_element_located((By.ID, 
5'Id_Of_Element'))) 
6except TimeoutException: 
7    print("Time out!") 

Use ActionChains for interactions like mouse movement like click, hold, hover, drag-drop and TouchActions for touch interactions like double-tap, long-press, flick, scroll as:

1# perform hover 
2from selenium import webdriver 
3from selenium.webdriver.common.action_chains import ActionChains 
4from selenium.webdriver.common.touch_actions import TouchActions 
5browser = webdriver.Firefox() 
6browser.get(URLString) 
7element_to_hover_over = browser.find_element_by_id("anyID") 
8ActionChains(browser).move_to_element(element_to_hover_over).perform()  
9scroll_up_arrow = browser.find_element_by_id("scroll_up") 
10TouchActions(browser).long_press(scroll_up_arrow) 

At the time of writing this guide, ActionChains and TouchActions are not supported by Safari.

Selenium offers many other navigational functions like:
Back and Forward: To navigate the history
Switch: To navigate between frames and alert dialog using Switch(browser).alert(),Switch(browser)window(window_Object_2), Switch(browser).frame(frame_Object_2) methods of Switch class.

driver.switch_to methods has been deprecated so instead use Switch class function.
The code is available on github for demonstration and practice. It also contains few more use-cases and optimized code.