Optimizing Data Analysis Through Offline Large Language Models and Scalable Data Management Techniques

Abstract

The importance of data across various sectors demands innovative approaches to data management and analytics. This PhD thesis investigates the integration of offline large language models (LLMs) for automated code generation, aiming to streamline data analysis processes, and thus enhance the scalability and efficiency of data management systems. By leveraging offline LLMs, the proposed approach empowers users to perform data analyses without extensive programming skills, thereby democratizing data analytics. The research delves into the architecture and implementation of scalable data management systems that can efficiently handle datasets of several volumes. Based on an efficient data management platform, the capabilities of offline LLMs to generate analytical code are examined, showcasing how these models can transform user queries into executable scripts that facilitate data manipulation and interpretation. Through experiments and case studies, the practical applications and benefits of the proposed study are showcased. The results highlight the potential of offline Large Language Models in Data Science and Analysis. This thesis contributes to the field by presenting a study that integrates AI-driven code generation with robust data management practices, ultimately paving the way for more efficient and user-friendly data analytics solutions.