Flux Student Internship Project

The Research Computing Platform (RCP) is based in the Computational Biology theme. We are a group of software engineers with a broad experience in different research fields and computational sciences. Our objective is to support WEHI’s strategic plans by investing and building new capabilities in computational research and data analytics. We collaborate with researchers to help them get their computational research running on WEHI infrastructure or, where appropriate on external resources such as commercial and research clouds, and local and national facilities. \

At Research Computing Platform our mission is to create a collaborative, multi-disciplinary platform that supports and advocates for WEHI researchers and their computational research. This internship project focuses on evaluating the integration of the Flux HPC scheduler with the existing Slurm scheduler at a mid-size High-Performance Computing (HPC) facility dedicated to life sciences.

The project’s objective is to assess the features, compatibility, performance, and potential benefits of utilizing Flux as a complementary scheduler to Slurm.

The Research Data Engineer Intern role will:

be responsible for adapting existing Slurm pipelines to evaluate their performance on Flux, enabling a direct comparison of the two schedulers.
work under the guidance of a Research Computing Engineer to understand the existing Slurm-based workflows and pipelines used in the life science computations. They will then adapt these pipelines to run on the Flux scheduler, ensuring compatibility and functionality.
conduct thorough performance evaluations, benchmarking the adapted pipelines on both Flux and Slurm. This evaluation will involve analysing ease of use, resource utilization, job completion times, and system efficiency.
collaborate with staff and researchers to gather feedback and insights on the performance and user experience of the adapted pipelines on Flux.

To excel in this internship project, the ideal candidate should be a quick learner, capable of swiftly grasping new concepts and technologies. The candidate should be comfortable with shell scripting and the Linux command line, and be experienced with HPC, as these skills will be essential for adapting the existing pipelines.

Strong analytical, problem-solving, and communication skills are necessary to document findings and provide recommendations based on the evaluation.

Benefits for students whilst undertaking the internship include:

The student will learn Intermediate to advanced skills in Flux. They will also be able to learn intermediate to advanced knowledge of working with HPCs and the software infrastructure that surrounds them. They will get experience with agile development methodologies that is needed in research environments. Learning about how to work within a research environment and with a HPC support team.