Unscheduled Storage Outage
The Isilon filesystem was restored to normal service and all affected clusters had it remounted as quickly as was sustainable by the filesystem. This process was completed by Wednesday, May 18th, 2016 at 12:15am EDT. All clusters other than Conte (which was undergoing a distinct maintenance) have returned to normal operations.
If you see any problems with your jobs from this event, please let us know, although it is unlikely there is any way to recover any work from jobs that suffered a failure.
As of Tuesday, May 17th, 2016 at 5:30pm EDT, Carter, Hammer, Hansen, Hathi, Peregrine1, Radon, Rice, Scholar, and Snyder are unavailable due to a loss of Isilon home directory storage. Most processes on these systems will block until this storage is restored.
Our storage engineers have been notified, though there is not yet an estimate for return to service.