Unscheduled scratch outage on Carter
There was an issue with the cluster's gateway switches, causing infiniband traffic to be incapable of IP over infiniband. This also caused an instability in the lustre scratch servers, which required that they be rebooted.
Jobs that were using scratch during this period may have been impacted.
Scheduling has been started back up.
Original Message:
The scratch filesystem serving Carter is currently unavailable.
Both currently running jobs and attempts to access files in scratch will block until the filesystem is back online.
Job scheduling on Carter has been paused while storage engineers address the issue.