Skip to main content
Have a request for an upcoming news/science story? Submit a Request

Workflow Automation Tools for Many-Task Computing

Hello!

Thank you for registering for RCAC's Workflow Automation Tools for Many-Task Computing seminar. The session details are below.

Historically, high-performance computing (HPC), has primarily involved tightly coupled simulations executed in a synchronous fashion across nodes. More recently, other paradigms within research computing have become common where the workload is not a single large coupled task, but instead very many modest tasks with little to no dependency on each other. These could be data processing and analysis tasks, machine-learning experiments, bioinformatics tasks, or parameter searches in a calculation.

In these many-task, high-throughput computing (HTC) workloads, researchers often attempt some form of parallelization within their application language (e.g., Python, R, or MATLAB ®). This is tedious, fraught with difficulty for most users, lacking flexibility, and is a distraction from the research concerns of the project. It would be better to use a distinct workflow automation tool to manage the many individual tasks.

This workshop outlines this paradigm on a traditional SLURM cluster along with several solutions. Different tools and techniques are discussed in an escalating fashion covering various features and pitfalls. Finally, a demonstration of the hyper-shell utility will showcase the level of sophistication possible both for individuals and research groups in managing task execution. (hyper-shell.readthedocs.io) Previous Linux / Unix command-line and Cluster experience is required.

Topics Overview:

  • Intro to concepts in high-throughput computing.
  • How the scheduler helps and where it falls short.
  • Overview of various workflow automation tools.
  • Overview of the hyper-shell utility.

Date: April 13, 2023 2:00pm - 4:00pm EDT Location: Online URL: https://teams.microsoft.com/l/meetup-join/19%3af0b1a8abffa74ed2be59b79a8b6105a4%40thread.tacv2/1673473818839?context=%7b%22Tid%22%3a%224130bd39-7c53-419c-b1e5-8758d6d63f21%22%2c%22Oid%22%3a%22df08b697-0a9e-4cc0-a1ed-018adfba12cc%22%7d

Originally posted: