Scraping Your First Web Page with Python

By Janani Ravi
This course covers the important tools for retrieving web content using HTTP libraries such as Requests, Httplib2 and Urllib, as well as powerful technologies for web parsing. These include Beautiful Soup, which is a popular library, and Scrapy, which is a powerful, production-grade framework.
Play course overview
Course info
Level
Beginner
Updated
Nov 5, 2019
Duration
2h 39m
Table of contents
Course Overview
Course Overview 2m
Getting Started with Web Scraping
Module Overview 1m Prerequisites and Course Outline 1m HTTP Requests and Responses 6m Web Scraping 2m HTTP Client Libraries 4m Making GET Requests Using httplib2 7m Making OPTIONS, POST, PUT Requests with httplib2 4m Handling Redirects with httplib2 4m Making HTTP Requests and Parsing URLs Using urllib 7m GET and POST Requests Using the Requests Library 5m Handling Redirects with the Requests Library 3m Module Summary 1m
Working with the Parse Tree in BeautifulSoup
Module Overview 1m The HTML Parse Tree 4m Beautiful Soup for HTML Parsing 2m Introducing Beautiful Soup 5m Extracting Specific Page Elements 6m Filtering Elements Using Find and Find All 7m Searching and Filtering Using Custom Functions 3m Extracting Links from a Page 6m Using a Soup Strainer to Parse a Subset of a Document 4m Module Summary 1m
Selecting Elements Using the Scrapy Shell
Module Overview 1m Parsing Web Content 2m Introducing Scrapy 4m Getting Started with Scrapy 4m Introducing the Scrapy Shell 4m Selecting Elements Using CSS Selectors 7m Advanced Selections Using CSS Selectors 5m Selecting Elements Using XPath Selectors 7m Module Summary 1m
Scraping Web Sites Using Scrapy Spiders
Module Overview 1m How Scrapy Works 3m Creating Your First Custom Spider 7m Writing Scraped Contents to a File 2m Exploring Items Using the Scrapy Shell 4m Using Items to Store Extracted Content 4m Using Item Loaders and Input and Output Processors for Scraped Data 7m Using Pipelines to Transform Scraped Data 5m Module Summary 1m
Description
Course info
Level
Beginner
Updated
Nov 5, 2019
Duration
2h 39m
Description

Web scraping is an important technique that is widely used as the first step in many workflows in data mining, information retrieval, and text-based machine learning. In this course, Scraping your First Web Page with Python, you will gain the ability to apply different scraping techniques including Beautiful Soup, and Scrapy. First, you will learn and use various HTTP client libraries such as Requests, httplib2, and urllib to download HTML content. Next, you will discover how Beautiful Soup is an extremely popular Python library that does better than regex in important ways. You will see how Beautiful Soup fixes up badly formed HTML, and constructs a nice parse tree that can be traversed and queried. Finally, you will add to your toolkit the knowledge of Scrapy, which is a full-fledged web scraping framework that combines the steps of retrieving and parsing web content and does so at production-scale. When you’re finished with this course, you will have the skills and knowledge to identify the relative strengths and use-cases of different web retrieval and scraping technologies such as regular expressions, Beautiful Soup, and Scrapy.

About the author
Janani Ravi
About the author

A problem solver at heart, Janani has a Masters degree from Stanford and worked for 7+ years at Google. She was one of the original engineers on Google Docs and holds 4 patents for its real-time collaborative editing framework.

More from the author
Building Statistical Summaries with R
Advanced
3h 3m
Jan 1, 2020
Understanding Statistical Models and Mathematical Models
Beginner
2h 35m
Dec 17, 2019
Interpreting Data Using Descriptive Statistics with Python
Intermediate
2h 21m
Nov 8, 2019
More courses by Janani Ravi
Section Introduction Transcripts
Section Introduction Transcripts

Course Overview
Hi. My name is Janani Ravi, and welcome to this course on Scraping Your First Web Page with Python. A little about myself, I have a master's degree in Electrical Engineering from Stanford and have worked at companies such as Microsoft, Google, and Flipkart. At Google I was one of the first engineers working on real-time collaborative editing in Google Docs, and I hold four patents for its underlying technologies. I currently work on my own startup, Loonycorn, a studio for high-quality video content. Web scraping is an important technique that is widely used as the first step in many work flows in data mining, information retrieval, and text-based machine learning. In this course, you will gain the ability to apply different scraping techniques, including Beautiful Soup and Scrapy. First, you will learn and use various HTTP client libraries, such as requests, httplib2, and urllib to download HTML content. Next, you will discover how Beautiful Soup is an extremely popular Python library that does better than regular expressions in important ways. You will see how Beautiful Soup fixes a badly formed HTML and constructs a nice parse tree that can be traversed and queried. Finally, you'll add to your toolkit the knowledge of Scrapy, which is a full-fledged web scraping framework that combines the steps of retrieving and parsing web content and does so at production scale. When you're finished with this course, you will have the skills and knowledge to identify the relative strengths and use cases of different web retrieval and scraping technologies such as regular expressions, Beautiful Soup, and Scrapy.

You're the smartest person in the room

PROVE IT

Your team won't just close the skills gap

THEY'LL HURDLE IT