Flurm - Dynamic Flux Deployment Through Slurm Job Submission with Co-Scheduling Capabilities

Abstract

High Performance Computing (HPC) systems are essential for solving complex computational problems across various scientific and engineering domains. Efficient resource management and job scheduling are critical components that directly impact the performance and utilization of these systems. While Slurm is a widely adopted open-source workload/resource manager that provides basic scheduling capabilities, it often lacks the flexibility required for fine-grained resource allocation and dynamic job scheduling. Flux is a next-generation resource management framework designed to address these limitations by enabling more flexible and efficient resource allocation strategies. This thesis presents the integration of the Flux Framework into an existing Slurm cluster at the user level, which we refer to as Flurm, enabling Flux to be launched and managed from within a Slurm allocation. It further demonstrates how Flux can be customized to provide fine-grained control over a variety of hardware resources (e.g. sockets, cores) in accommodation with Flux's graph-based scheduler - Fluxion. Thereby, Flux was modified to support advanced resource allocation features such as co-execution, colocation, and flexible resource sharing within the Slurm environment. Finally, a series of experiments are conducted to assess any overhead introduced by the integration and the scalability of the proposed solution.