Unscheduled Data Depot outage on multiple clusters

UPDATE: August 13, 2021  9:20am

As of 9:20am, the issue with Data Depot mounts was resolved and Bell, Brown, Gilbreth, Halstead, Scholar, and Workbench clusters have been returned to normal service. Job queues have been enabled and job scheduling has been resumed. We apologize for the disruption of service. Please report any issues to rcac-help@purdue.edu.

ORIGINAL: August 13, 2021 12:30am - 9:20am EDT

The Bell, Brown, Gilbreth, Halstead, Scholar, and Workbench clusters began experiencing issues with mounting old Data Depot filesystem around 12:30am. Multiple nodes are flagged offline by an automatic check, and bioinformatics application suite (residing on the old Data Depot hardware while being migrated) is currently unavailable.

Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed.

We will provide an update by 2pm.

Originally posted: August 13, 2021 9:07am EDT