Outages and Maintenance
-
Unscheduled Outage on Data Depot
The Depot filesystem checks have all completed cleanly and the Depot has been fully returned to normal operations. All queues on all clusters are scheduling new jobs again. Any existing jobs which had been waiting for Depot access may also resume....
-
Unscheduled outage on Rice and Snyder
As of 9:15 PM, the Snyder and Rice clusters have been brought back into service after cooling was brought back online. Front-ends are operational and scheduling has been resumed. Original Message: At about 7:30 pm Wednesday, 17 February, 2016, the fr...
-
Unscheduled scratch outage on Carter
There was an issue with the cluster's gateway switches, causing infiniband traffic to be incapable of IP over infiniband. This also caused an instability in the lustre scratch servers, which required that they be rebooted. Jobs that were using scratc...
-
The scheduler issue has been resolved, and Conte has been returned to normal operations as of Wednesday, February 10th, 2016 at 9:30pm EST. Update: February 10, 2016 7:04pm There was a minor issue discovered with the newly upgraded scheduler which i...
-
The cause of this turned out to be a power loss to Carter's scratch filesystem and portions of the Data Depot, which has been restored now. Carter nodes are returning to normal operations now. Original Message: As of Thursday, February 4th, 2016 at...
-
Unscheduled Outage in Math Data Center
Most of the impact of this turned out to be to the Depot storage system, which has now been restored to normal operations. All the other affected systems are showing a return to normal operations now. Original Message: As of Thursday, February 4th,...
-
Hathi & WinHPC Power Maintenance
The Hathi and WinHPC clusters will be unavailable beginning at Thursday, February 4th, 2016 at 6:00am EST, for scheduled maintenance to the power feed. Both clusters will return to full production by Thursday, February 4th, 2016 at 5:00pm EST. During...
-
Fortress will be unavailable from 8:00am to 9:00am Wednesday, 3 February, 2016 for routine maintenance.
-
The underlying issues affecting Carter are resolved and job scheduling has been resumed. Many individual nodes remain offline for corrective action, and these will be returning to service gradually as engineers are able to fix them. In the interim,...
-
Rice and Snyder Cluster Maintenance
As of 10:40pm, the Snyder cluster was returned to normal service in the POD. This concludes this maintenance. Update: February 5, 2016 8:54pm As of 8:25 pm, Friday, 5 Feb 2016, the Rice cluster maintenance has completed and the system is returning...
-
Unscheduled scratch outage on Hammer
The Hammer scratch filesystem has now returned to normal operations. Original Message: During the maintenance of the Rice and Snyder clusters this week, it became necessary to shut down the scratch filesystem which these clusters currently share with...
-
Carter has been returned to normal operation. Update: January 20, 2016 3:26pm: We are doing return to service testing now and expect Carter to return to production by 7:00pm. Update: January 20, 2016 12:00pm: Work is being wrapped up on Carter and...
-
January 7, 2016, 6pm The Fortress move has completed and has been returned to production. Original Due to a failure in the notice system, the earlier attempts to notify of this work which were sent on Dec 7th and Jan 3rd were not delivered. The Fortr...
-
Unscheduled Home Filesystem Outage
As of 12:46, December 2, the home filesystem serving Conte, Hammer, Hansen, Hathi, Peregrine1, Radon, Rice, and Snyder was restored to normal operation. All queues have been re-enabled. As of Wednesday, December 2nd, 2015 at 12:00pm EST, Conte, Hamm...
-
Unscheduled scratch outage on Rice, Hammer, and Snyder
The scratch filesystem serving Hammer, Rice, and Snyder has been restored to normal operations, and all queues have been re-enabled. Original Message: The scratch filesystem serving Hammer, Rice, and Snyder is partially unavailable. Both currently ru...
-
Carter has been return to normal operations. All queues have been enabled. Update: December 2, 2015 12:15pm Carter is mostly ready to return to service, but the site-wide home filesystem has suffered a failure which is preventing this from being co...
-
Unscheduled scratch outage on Conte
The scratch filesystem has been restored to full service and all queues have been restarted. Original Message: The scratch filesystem serving Conte is currently unavailable. Both currently running jobs and attempts to access files in scratch will bl...
-
Update - 9:20pm Conte has been returned to full production as of 9:15pm. During the failure earlier today, the internal tracking of jobs within the scheduler on Conte was corrupted. Unfortunately, this resulted in all running and pending jobs being...
-
The Fortress Archive service, Fortress, will be unavailable starting Wednesday, November 4th, 2015 at 6:00am EST for regular maintenance and will return at Wednesday, November 4th, 2015 at 8:00am EST. During this time, access via HSI, HTAR, Globus of...
-
November 3, 2015 6:15pm The maintenance for Radon is completed and the cluster has been returned to production. Original The Radon cluster will be unavailable beginning at Tuesday, November 3, 2015 from 7:00am - 7:00pm EST, for scheduled maintenance....