Congratulations! You’ve just been hired as a data engineer for a mobile game development studio. The company’s modern data platform architecture includes an Amazon Athena data lake and an AWS Redshift data warehouse solution. Your task is to enable batch processing of revenue transaction data by creating an end-to-end data pipeline, connecting various data sources—including user engagement events, stage controls, and public chat messaging—to the lake house solution. Using AWS CloudFormation, you’ll provision the resources required for the data pipeline. You’ll connect a MySQL data source to the AWS S3 Data Lake and transform data in the data lake using Amazon Athena. You’ll wrap up the project by creating a dynamic analytics dashboard with AWS QuickSight. When you’re done, you’ll have built a batch-processing data pipeline, start to finish, using Amazon Athena.

As a data engineer for a mobile game development studio, your task is to create a data streaming pipeline that collects and processes large streams of data records in real-time for lightning-fast analytics. Your company’s modern data platform architecture includes an Amazon Athena data lake and an AWS Redshift data warehouse solution. To store files, you’ll create an AWS S3 bucket, and you’ll create an AWS Kinesis delivery stream by using the boto3 library to connect to AWS Kinesis endpoints and send event data to the service. You’ll provision AWS Redshift resources and connect them to your AWS Kinesis Data Stream to analyze user behavior data to understand the user's journey inside the app. When you’re done, you’ll have a simple yet reliable data streaming pipeline that prevents resource shortages and transforms data in real-time—while it’s still relevant—ensuring more accurate data.

Help your company’s messenger application provide better product recommendations for its customers. As a data engineer at the company, your task is to create a machine learning (ML) pipeline using the Amazon Personalize service. You’ll use CloudFormation templates to create a repository for the required AWS infrastructure resources, and AWS Glue to transform the raw user engagement data. Using Amazon Personalize, you’ll import a dataset and create and train the Amazon Personalize ML model for your users’ recommendations. To complete the project, you’ll create a workflow to train your Amazon Personalize recommendation solution using AWS Step Functions and user engagement events. When you’re done, you’ll have designed an ML pipeline using the Amazon Personalize API that provides product recommendations that suit your users best.