Μελέτη επίδοσης συστημάτων επεξεργασίας ροών δεδομένων

Abstract

Over the last few years Big data technology has gained an increasing interest . As a result of the IOT and social media development, a huge volume of streaming data is generated at an unprecedented rate. This volume of streaming data needs to be handled in a timely fashion. Thus, big data processing needs to take place in real-time .The need for scalable and efficient stream analysis has led to the development of many open-source streaming data processing systems (SDPSs) with highly diverging capabilities and performance characteristics. The aim of this thesis is to evaluate the performance of two widely used SDPSs in detail, namely Apache Spark, and Apache Flink. Our evaluation focuses in particular on measuring the throughput and latency of windowed operations, which are the basic type of operations in stream analytics. For this benchmark, we produce streaming big data , which vary depending on the use-cases. This thesis can help the users to choose the appropriate processing engine depending on the application , the data type and the available resources.