Skip to main content
Have a request for an upcoming news/science story? Submit a Request

Workflow Automation Tools for Many-Task Computing


Thank you for registering for RCAC's Workflow Automation Tools for Many-Task Computing seminar. The session details are below.

Historically, high-performance computing (HPC), has primarily involved tightly coupled simulations executed in a synchronous fashion across nodes. More recently, other paradigms within research computing have become common where the workload is not a single large coupled task, but instead very many modest tasks with little to no dependency on each other. These could be data processing and analysis tasks, machine-learning experiments, bioinformatics tasks, or parameter searches in a calculation.

In these many-task, high-throughput computing (HTC) workloads, researchers often attempt some form of parallelization within their application language (e.g., Python, R, or MATLAB ®). This is tedious, fraught with difficulty for most users, lacking flexibility, and is a distraction from the research concerns of the project. It would be better to use a distinct workflow automation tool to manage the many individual tasks.

This workshop outlines this paradigm on a traditional SLURM cluster along with several solutions. Different tools and techniques are discussed in an escalating fashion covering various features and pitfalls. Finally, a demonstration of the hyper-shell utility will showcase the level of sophistication possible both for individuals and research groups in managing task execution. ( Previous Linux / Unix command-line and Cluster experience is required.

Topics Overview:

  • Intro to concepts in high-throughput computing.
  • How the scheduler helps and where it falls short.
  • Overview of various workflow automation tools.
  • Overview of the hyper-shell utility.

Date: April 13, 2023 2:00pm - 4:00pm EDT Location: Online URL:

Originally posted: