How to use AWS Glue crawlers with Amazon Athena

As a data professional, you deal with immense amounts of data from a variety of sources. This can make data management and analysis a challenge. Luckily, two AWS services can help: AWS Glue and Amazon Athena. 

When you integrate these services, you unlock the full potential of data discovery, cataloging, and querying within the AWS ecosystem. Let’s take a look at how they can streamline your data analytics workflow.

• What is Amazon Athena?
• What is AWS Glue?
• What is AWS Glue crawler?
• What is the Glue Data Catalog?
• Why use Amazon Athena and AWS Glue?
  • 4 main Amazon Athena use cases
  • 3 key AWS Glue use cases
• Getting started with AWS Glue: How to get data from AWS Glue to Amazon Athena
• Streamline data analytics with AWS Glue crawlers and Amazon Athena

Amazon Athena is a serverless, interactive data analytics service from an open-source framework. Because it supports open-table and file formats, you can quickly and easily perform in-browser SQL searches of your data. Querying data, even big data, becomes something you can do instantly.

For a deeper dive into this service, check out our article Amazon Athena, Explained.

AWS Glue is a serverless managed service that allows you to discover, prepare, move, and integrate data from multiple sources. As a data integration service, AWS Glue gives you the power to manage data in a centralized location without having to manage infrastructure.

Glue crawlers are automated data discovery tools that scan a data source to classify, group, and catalog the data within it automatically. It then creates new or updates existing tables in your AWS Glue Data Catalog.

The AWS Glue Data Catalog is an index of your data’s location, schema, and runtime metrics. You need this information to create and monitor your extract, transform, and load (ETL) jobs. 

Now that we’ve covered the basics of Amazon Athena, AWS Glue, and AWS Glue crawlers, let’s talk about them in a little more depth.

Amazon Athena provides a simplified, flexible way to analyze petabytes of data right where they live. For example, Athena can analyze data or build applications from an Amazon Simple Storage Service (S3) data lake and 30 data sources, including on-premises data sources or other cloud systems using SQL or Python. 

There are four main Amazon Athena use cases:

1. Run queries on S3, on-premises data centers, or on other clouds 

2. Prepare data for machine learning models

3. Use machine learning models in SQL queries or Python to simplify complex tasks, such as anomaly detection, customer cohort analysis, and sales predictions

4. Perform multicloud analytics (like querying data in Azure Synapse Analytics and then visualizing the results with Amazon QuickSight)

Now that we’ve covered Amazon Athena, let's talk about AWS Glue. You can do a few different things with AWS Glue. 

First, you can use AWS Glue data integration engines, which allow you to get data from a few different sources. This includes Amazon S3, Amazon DynamoDB, and Amazon RDS, as well as databases running on Amazon EC2 (which integrates with AWS Glue studio) and AWS Glue for Ray, Python Shell, and Apache Spark. 

Once the data is interfaced and filtered so it can interact with places to load or create data, this list expands to include data from places like Amazon Redshift, data lakes, and data warehouses.

You can also use AWS Glue to run your ETL jobs. These jobs allow you to segregate customer data, protect customer data in transit and at rest, and access customer data only as needed in response to customer requests. When provisioning an ETL job, all you need to do is provide input data sources and output data targets in your virtual private cloud.

The final way you can use AWS Glue is through a data catalog to quickly discover and search multiple AWS datasets without moving the data. Once the data is cataloged, it’s immediately available for search and query using Amazon Athena, Amazon EMR, and Amazon Redshift Spectrums.

So, how can you get data from AWS Glue into Amazon Athena? Follow these steps:

1. Start by uploading data to a data source. The most popular option is an S3 bucket, but DynamoDB tables and Amazon RedShift are also options. 

2. Select your data source and create a classifier if necessary. A classifier reads the data and generates a schema if it recognizes the format. You can create custom classifiers to see different data types. 

3. Create a crawler. 

4. Set up a name for the crawler, then choose your data sources and add any custom classifiers to make sure AWS Glue recognizes the data correctly.

5. Set up an Identity and Access Management (IAM) role to make sure the crawler can run the processes correctly.

6. Create a database that will hold the data set. Set when and how often the crawler works to keep your data fresh and up to date.

7. Run the crawler. This process can take a while depending on how big the dataset is. Once the crawler has successfully run, you’ll see changes to tables in the database.

Now that you’ve completed this process, you can jump over to Amazon Athena and run the queries you need to filter the data and get the results you’re looking for.

I hope this cleared some things up about AWS Glue crawlers and Amazon Athena. If you’re looking to learn more about Amazon Athena and how it works with AWS Glue and other services, you can check out my new course Amazon Athena Deep Dive.

Our lab, Transform Data Using AWS Glue and Amazon Athena, can also give you some hands-on experience with these services. Sign up for a 10-day free trial with no commitments to kickstart your learning journey.