Data Engineer’s Lunch #86: Building Real-Time Applications at Scale

As the demand for real-time data processing continues to grow, so too do the challenges associated with building production-ready applications that can handle large volumes of data and handle it quickly. In this talk, we will explore common problems faced when building real-time applications at scale, with a focus on a specific use case: detecting and responding to cyclist crashes. Using telemetry data collected from a fitness app, we’ll demonstrate how we used a combination of Apache Kafka and Python-based microservices running on Kubernetes to build a pipeline for processing and analyzing this data in real-time. We’ll also discuss how we used machine learning techniques to build a model for detecting collisions and how we implemented notifications to alert family members of a crash. Our ultimate goal is to help you navigate the challenges that come with building data-intensive, real-time applications that use ML models. By showcasing a real-world example, we aim to provide practical solutions and insights that you can apply to your own projects.

Key Takeaways:

• An understanding of the common challenges faced when building real-time applications at scale
• Strategies for using Apache Kafka and Python-based microservices to process and analyze data in real-time
• Tips for implementing machine learning models in a real-time application
• Best practices for responding to and handling critical events in a real-time application