Outages
-
Rice scheduling temporarily stopped
The Rice cluster is currently experiencing a vendor bug in its scratch filesystem. To prevent filesystem instability, job scheduling has been paused while engineers are diagnosing the issue. Currently running jobs will not be affected (only starting...
-
Unscheduled data center outage
Several clusters have experienced network connectivity and/or power issues around 10:00am EDT. Engineers are working on assessing and analyzing the situation. Job scheduling on affected clusters has been paused while this issue is being addressed. We...
-
Unscheduled Hammer outage - Closed
Update: The issues with the Hammer cluster has been resolved and the cluster is back in production. This outage is closed. Original: The Hammer cluster began experiencing issues around 10:10am EDT. Engineers are currently diagnosing the issue and are...
-
Work continues on bringing Hammer back to normal operation. Engineers have identified the source of the problem and are currently working to find a solution. We will provide another update by 3pm today.
-
Scheduling of Jobs Paused on Brown
Starting at 2:45pm EDT, new jobs are not being scheduled on the Brown cluster in order to lighten the load. The current hot weather outside is overtaxing Brown cooling systems. Existing jobs will continue to run as planned. As soon as the cooling sys...
-
Unscheduled research homes outage affecting all clusters
All clusters began experiencing issues with the home file system around 12:30pm EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed. We will provide a...
-
Unscheduled power outage affecting Brown and Hammer
The Brown and Hammer clusters experienced a partial power outage overnight which caused them to operate at a reduced capacity. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this...
-
Unscheduled Rice scratch outage
The Rice cluster began experiencing issues with the scratch filesystem around 4:40pm EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed. We will prov...
-
Data Depot suffered a system failure around 5:15pm EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed. We will provide an update by 9pm or earlier as...
-
The Fortress Archive began experiencing issues with an internal database around 4:30pm. Engineers are currently working to remove the affected database from the system to mitigate the issue. Fortress should return to normal function once this is remo...
-
The Gilbreth cluster began experiencing issues with its scratch filesystem around 11:30am EST. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed. We will...
-
Unscheduled Outage to Multiple Systems
Hammer, Scholar, Snyder, WCERES, WSC Hadoop, and Data Depot began experiencing issues with networking around 10:00am EST. Engineers are currently diagnosing the issue and are working to identify a fix. We will provide an update by 1:00pm if the issue...
-
Unscheduled Data Depot outage on the clusters
The Brown, Gilbreth, Halstead, Hammer, Rice, Scholar, Snyder, and Workbench clusters began experiencing issues with connection to Data Depot filesystem around 5:00pm EDT on Friday, April 17th, 2020. Engineers are currently diagnosing the issue and ar...
-
Running Jobs on Community Clusters While Data Depot is Unavailable
Since Friday, April 17, the Research Data Depot filesystem has been unavailable on community cluster systems due to an ongoing filesystem verification. While we don't believe there is any danger of data loss, the filesystem verification will continu...
-
Unscheduled Data Depot windows network drive outage
Since Friday, April 17, the Research Data Depot filesystem has been unavailable on community cluster systems, but remained available through other means of access (such as Windows Network Drive). Around 9:00am EDT on Tuesday, April 21st, 2020, the D...
-
Engineering Computing Network (ECN) has reported an outage on the software license servers for ITaP Research Computing systems that are hosted by ECN. ITaP Research Computing cluster job scheduling is not affected by the outage, but licenses for soft...
-
The Weber cluster began experiencing issues around 10:00am EDT. Engineers are currently diagnosing the issue and are working to identify a fix. We will provide an update by 2:00pm.
-
The Data Depot storage system began experiencing issues with No space left on device error messages around 10:30am EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling on community clusters has been paus...
-
Unscheduled Brown scratch outage
The Brown cluster began experiencing issues with its scratch filesystem around 4:30pm EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed. We will pro...
-
Unscheduled Home Directory Outage
The Brown, Gilbreth, Halstead, Rice, Scholar, Snyder, and Workbench clusters began experiencing issues with intermittently slow home directories access around 2:30pm EDT. The issue has been traced to a high load on one of the filesystem's back-end se...