High-Performance Data Analytics with Ray

Abstract

The increasing demand for the development of efficient processing and analyzing tools, can be attributed to the significant role that Big Data analysis and Machine Learning have acquired in our days across various fields. In order for complex computations to be effectively performed, and vast amounts of data to be handled adequately and lead to the extraction of meaningful insights, the need for distributed computing frameworks with such capabilities has emerged. Through the development frameworks, such as Ray, the computational demands of Big Data analytics and ML tasks are addressed, due to a certain set of capabilities they provide.

Ray is enriched with APIs that parallelize Python code, and can therefore be characterized as a powerful tool regarding distributed computing. The objective of this thesis is to analyze Ray's performance on various applications, delving into ETL operations, graph processing, distributed ML training, and hyperparameter tuning. Apache Spark is another similar framework. Being widely used today and recognized for its powerful data-processing properties and extensive library support, Spark's performance is used as a point of reference in this work. The experiment was carried out on a cluster setup, taking into consideration various parameters including the time of execution, CPU time, as well as memory usage throughout different data sizes and node configurations. According to the results, it was demonstrated that Spark is superior to Ray regarding ETL and graph operations since it comprises a more mature ecosystem and exhibits efficient memory usage. It was nevertheless observed that Ray was outperforming Spark as far as ML training and hyperparameter tuning were involved, which showcases its significant parallel processing capabilities.