Unscheduled Gilbreth cluster outage
Link to update at April 6, 2022 11:35pm EDT UPDATE:
As of 11:35pm EDT, the Gilbreth cluster scratch filesystem has recovered, and the cluster has been returned to normal service. Job queues have been re-enabled and job scheduling has been resumed.
Please note that scratch filesystem was completely unmounted and remounted on the cluster as part of the resolution process. Users with jobs that were either based in scratch and/or actively reading or writing into scratch at the time of the outage are advised to check their outputs for consistency and completeness.
We apologize for the disruption of service. Please report any issues to rcac-help@purdue.edu.
Link to original posting ORIGINAL:
The Gilbreth cluster began experiencing issues with its scratch filesystem around 7:00pm EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed.
We will provide an update by noon tomorrow (April 7th).