Unscheduled Data Depot outage on multiple clusters
UPDATE: August 13, 2021 9:20am
As of 9:20am, the issue with Data Depot mounts was resolved and Bell, Brown, Gilbreth, Halstead, Scholar, and Workbench clusters have been returned to normal service. Job queues have been enabled and job scheduling has been resumed. We apologize for the disruption of service. Please report any issues to email@example.com.
ORIGINAL: August 13, 2021 12:30am - 9:20am EDT
The Bell, Brown, Gilbreth, Halstead, Scholar, and Workbench clusters began experiencing issues with mounting old Data Depot filesystem around 12:30am. Multiple nodes are flagged offline by an automatic check, and bioinformatics application suite (residing on the old Data Depot hardware while being migrated) is currently unavailable.
Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed.
We will provide an update by 2pm.