Research Proposal: Reliability Evaluation of the Apache Kafka Streaming System

Apache Kafka is a distributed messaging system with high throughput, high scalability and low latency. It has been widely adopted in enterprise and due to its widespread integration into enterprise-level infrastructures, the research on the reliability of Kafka consumers has become an increasingly important issue. The application scenarios vary from tracking user profiles on a website, server log monitoring, to online bank transfer and online reservation. The main purpose of this research is to evaluate the reliability of Kafka in different application scenarios. Kafka is highly configurable and provides many options to manage reliability strategies. In this research we test the impacts of an kinds of configuration parameters on the reliability of Kafka, including retry strategies and replications of partitions for fault tolerance. The tradeoffs between performance and reliability is another portion of our research, which help users of Kafka using it in an appropriate way.