Kafka

Apache Kafka is a distributed (message broker) system designed for large-scale data flows in real- time. It provides high reliability and fault tolerance, making it ideal for connecting sensors or external data sources to the application infrastructure. In this paper, Kafka is used as a source of data stream inputs, a repository of data generated by the algorithm training and prediction processes, and as a data repository. The connection to the Kafka broker is established through the Python library confluent_kafka, which is the official Confluent client[15] for al- eration with Kafka. This library was chosen because of its extensive user manual and its simplicity in implementation.