Unscheduled scratch outage on Rice, Snyder, and Hammer
The scratch filesystem serving Hammer, Rice, and Snyder is currently unavailable.
Both currently running jobs and attempts to access files in scratch will block until the filesystem is back online.
Job scheduling on Hammer, Rice, and Snyder has been paused while storage engineers address the issue.
UPDATE: In error, the Snyder cluster was not initially included in this outage but was also affected.
FINAL UPDATE: At this time, scratch for Rice, Snyder, and Hammer are back in operation.
Around 3:30pm, a server failure occurred and the file system recovered. However, some clients did not properly handle the failure like normal. These clients have been found and will be manually fixed by administrators.