Event-Driven Architectures using Apache Kafka

Abstract

Event-Driven architectures are a very useful approach to decoupling inter-service communication while enabling unique communication patterns. The specific resiliency, availability, and performance requirements of such architectures, warrants the usage of specialized Message-Broker services. Moreover, new real-time requirements of many event processing tasks, such as metrics and analytics, requires a new approach compared to legacy batch processing. Apache Kafka is an open-source event streaming platform that can accommodate both use cases. This thesis performs a succinct presentation of Apache Kafka’s main architectural model, while focusing on its concrete resiliency and availability guarantees and providing guidelines for achieving desired operational characteristics. We also document and further enhance a mathematical model that permits the estimation and concrete sizing of cluster infrastructure, as well as express the link between topic sizing, maximum Event-Driven architectures are a very useful approach to decoupling inter-service communication while enabling unique communication patterns. The specific resiliency, availability, and performance requirements of such architectures, warrants the usage of specialized Message-Broker services. Moreover, new real-time requirements of many event processing tasks, such as metrics and analytics, requires a new approach compared to legacy batch processing. Apache Kafka is an open-source event streaming platform that can accommodate both use cases. This thesis performs a succinct presentation of Apache Kafka’s main architectural model, while focusing on its concrete resiliency and availability guarantees and providing guidelines for achieving desired operational characteristics. We also document and further enhance a mathematical model that permits the estimation and concrete sizing of cluster infrastructure, as well as express the link between topic sizing, maximum unavailability and end-to-end latency.