Data Engineer

Data Pipeline with Amazon Athena you own this product

This project is part of the liveProject series Become a Data Engineer with AWS
prerequisites
basic AWS cloud computing • advanced shell scripting • basic Python
skills learned
create a batch ETL pipeline from RDS (MySQL) to AWS S3 • extract and process data using AWS Lambda • run SQL queries programmatically in the cloud • create data environment configurations using YAML files • test AWS Lambda locally • create an optimized ICEBERG table in Athena
Mike Shakhomirov
1 week · 5-7 hours per week · BEGINNER
Included with a Manning Online subscription
try now

pro $24.99 per month

  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose one free eBook per month to keep
  • exclusive 50% discount on all purchases

lite $19.99 per month

  • access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

Look inside

Congratulations! You’ve just been hired as a data engineer for a mobile game development studio. The company’s modern data platform architecture includes an Amazon Athena data lake and an AWS Redshift data warehouse solution. Your task is to enable batch processing of revenue transaction data by creating an end-to-end data pipeline, connecting various data sources—including user engagement events, stage controls, and public chat messaging—to the lake house solution. Using AWS CloudFormation, you’ll provision the resources required for the data pipeline. You’ll connect a MySQL data source to the AWS S3 Data Lake and transform data in the data lake using Amazon Athena. You’ll wrap up the project by creating a dynamic analytics dashboard with AWS QuickSight. When you’re done, you’ll have built a batch-processing data pipeline, start to finish, using Amazon Athena.

This project is designed for learning purposes and is not a complete, production-ready application or solution.

book resources

When you start your liveProject, you get full access to the following books for 90 days.

project author

Mike Shakhomirov

Mike Shakhomirov is the head of data engineering at The World's Online Festival. He has an MBA as well as a diploma in big data and social analytics from MIT, and he’s a Google Cloud Certified Professional Data Engineer. Passionate, enthusiastic, and digitally focused, he loves the challenges that the diverse gamut of digital marketing can offer. Mike is an official writer for publications including Towards Data Science and The Startup, and he’s the author of more than 50 published articles on topics such as data engineering, machine learning, and AI in digital marketing. You can find him on LinkedIn and Medium.

prerequisites

This liveProject is for intermediate Python programmers who are interested in learning how to create batch-processing data pipelines with Amazon Athena. To begin these liveProjects you’ll need to be familiar with the following:

TOOLS
  • Intermediate Python skills and knowledge
  • AWS account
  • Basic cloud computing skills
  • Basic knowledge of the MySQL database
  • Basic knowledge of serverless infrastructure
  • Advanced shell scripting
TECHNIQUES
  • Build basic REST APIs

you will learn

In this liveProject, you’ll learn how to create a batch-processing data pipeline with Amazon Athena and Lambda.

  • Create a batch ETL pipeline from RDS (MySQL) to AWS S3
  • Extract and process data using AWS Lambda
  • Run SQL queries programmatically in the cloud
  • Create data environment configurations using YAML files
  • Test AWS Lambda locally
  • Build a data pipeline in AWS Data Lake connecting a relational database and Athena
  • Create an optimized ICEBERG table in Athena and add it to data pipeline
  • Use AWS QuickSight for BI Dashboard design
  • Provision the resources required for your data pipeline using AWS CloudFormation templates

features

Self-paced
You choose the schedule and decide how much time to invest as you build your project.
Project roadmap
Each project is divided into several achievable steps.
Get Help
While within the liveProject platform, get help from other participants and our expert mentors.
Compare with others
For each step, compare your deliverable to the solutions by the author and other participants.
book resources
Get full access to select books for 90 days. Permanent access to excerpts from Manning products are also included, as well as references to other resources.

choose your plan

pro

monthly
annual
$24.99
$249.99
only $20.83 per month
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • Data Pipeline with Amazon Athena project for free

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • Data Pipeline with Amazon Athena project for free