Award-winning HyperShell software developed by RCAC helps simplify “many task” computing
RCAC staff members have developed a piece of software, known as HyperShell, that helps divide and conquer large volumes of discrete tasks to assist with scheduling and managing what’s known as “many task computing.” HyperShell has been dubbed “the ultimate workflow automation tool.”
In traditional high-performance computing, there’s just one big job for the supercomputer to tackle. But more often these days, researchers are doing what’s known as “high-throughput computing” or “many-task computing” where the supercomputer is performing a very large number of small, independent tasks, such as analyzing tens of thousands of pieces of data in a dataset.
Traditional HPC job schedulers like Slurm are not well-suited to managing this kind of computing, so RCAC lead research data scientist Geoffrey Lentner saw the need for a piece of software to help facilitate this kind of workflow, and HyperShell was born.
HyperShell, which is written in Python, is useful for researchers doing large volumes of data analysis and processing, such as in fields like bioinformatics, climate modeling, math and statistics optimization and agriculture.
It’s also useful for training AI models since the powerful RCAC GPUs are too large for many jobs and Slurm doesn’t allow for subdividing them. HyperShell allows a user to stack multiple models on a GPU and fill it up with multiple smaller jobs.
Several tools offer similar functionality to HyperShell but not all together in a single tool with the user ergonomics provided. Novel design elements include but are not limited to (1) cross-platform, (2) client-server design, (3) staggered launch for large scales, (4) persistent hosting of the server, and optionally (5) a database in-the-loop for persisting task metadata and automated retries.
Lentner and former RCAC staff member Lev Gorenstein presented an early version of HyperShell at the 2022 Practice and Experience in Advanced Research Computing (PEARC) conference, and won the best poster award for their poster about it.
Learn more about HyperShell here, or contact rcac-help@purdue.edu for more information.