Step into the role of a developer at Piper Data Concepts (PDC), a company that specializes in helping Fortune 1000 companies improve their workflows. Your task is to review the 15-year-old workflow architecture of one of your clients, Trade Data Systems. You’ll identify issues and bottlenecks, then determine what’s needed to transform its workflow into a more modern, responsive architecture. To accomplish this, you’ll set up a development environment with Docker using Kafka, Python, and Postgres. As you go, you’ll deploy a Kafka cluster and write Python code using the Faust library to seamlessly process pre-defined business events.

Put on your platform architect hat! You’re a member of the development team at Piper Data Concepts (PDC), and your client is looking to modernize its workflow. An existing benchmarked development environment, made up of Kafka, Python, and Postgres, is at your disposal. Now it’s time to start conceptualizing the new and improved workflow. You’ll use Kafka to create an event-driven data pipeline, review and understand business requirements, use Python Poetry to package the project, write Python code using the Faust library to communicate with Kafka, and store the consumed data in a PostgreSQL database.

As a member of the development team at Piper Data Concepts, you’ll carry out the final steps of a workflow-improvement project: enabling your client’s staff to gather workflow process information in real-time. Several prototypes have been built, and the client’s workflow is more resilient than ever. You’ll write Python code that consumes messages from Kafka and prepares them for storing in the database, create Postgres queries to access the aggregated data, and build reports in CSV files to be read by visualization tools—and ultimately, your client’s staff. When you’re done, your client’s modern system will provide a feedback loop, enable external API access to status updates, and be ready for more specialized services to be plugged in later, with no code changes.