All courses
Resource hub
Featured resource
Tech Upskilling Playbook 2025
Tech Upskilling Playbook

Build future-ready tech teams and hit key business milestones with seven proven plays from industry leaders.

Learn more
Hamburger Icon
  • Course
    • Libraries: If you want this course, consider one of these libraries.
    • Core Tech

Python: Scalable Web Scraping and API Workflows

Learn to build scalable web scraping workflows in Python. This course will teach you how to efficiently fetch, parse, and process web data using Requests, BeautifulSoup, and Celery, enabling you to handle large-scale scraping challenges with ease.

Kevin Cunningham - Pluralsight course - Python: Scalable Web Scraping and API Workflows
by Kevin Cunningham
Get started

What you'll learn

Handling web scraping at scale can be challenging. In this course, Python: Scalable Web Scraping and API Workflows, you’ll learn to build efficient, maintainable web scraping workflows using Requests, BeautifulSoup, and Celery. First, you’ll explore how to fetch web data efficiently using the Requests library while handling headers, sessions, and rate limits. Next, you’ll discover how to parse and extract meaningful information from HTML with BeautifulSoup, handling messy or inconsistent data structures. Finally, you’ll learn how to scale your web scraping tasks using Celery and Redis to process jobs asynchronously and improve performance. When you’re finished with this course, you’ll have the skills and knowledge of scalable web scraping and API integration needed to handle large-scale data extraction efficiently, avoid common pitfalls, and integrate your scraping workflows into production-ready applications.

Table of contents

About the author

Kevin Cunningham - Pluralsight course - Python: Scalable Web Scraping and API Workflows
Kevin Cunningham

Kevin is a Developer Educator, speaker and writer based in beautiful Northern Ireland. He specializes in web development across a range of languages and frameworks.

More Courses by Kevin