Outages
-
The Fortress Archive began experiencing issues with an internal database around 4:30pm. Engineers are currently working to remove the affected database from the system to mitigate the issue. Fortress should return to normal function once this is remo...
-
Data Depot suffered a system failure around 5:15pm EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed. We will provide an update by 9pm or earlier as...
-
Unscheduled Rice scratch outage
The Rice cluster began experiencing issues with the scratch filesystem around 4:40pm EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed. We will prov...
-
Unscheduled power outage affecting Brown and Hammer
The Brown and Hammer clusters experienced a partial power outage overnight which caused them to operate at a reduced capacity. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this...
-
Unscheduled research homes outage affecting all clusters
All clusters began experiencing issues with the home file system around 12:30pm EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed. We will provide a...
-
Scheduling of Jobs Paused on Brown
Starting at 2:45pm EDT, new jobs are not being scheduled on the Brown cluster in order to lighten the load. The current hot weather outside is overtaxing Brown cooling systems. Existing jobs will continue to run as planned. As soon as the cooling sys...
-
Work continues on bringing Hammer back to normal operation. Engineers have identified the source of the problem and are currently working to find a solution. We will provide another update by 3pm today.
-
Unscheduled Hammer outage - Closed
Update: The issues with the Hammer cluster has been resolved and the cluster is back in production. This outage is closed. Original: The Hammer cluster began experiencing issues around 10:10am EDT. Engineers are currently diagnosing the issue and are...
-
Unscheduled data center outage
Several clusters have experienced network connectivity and/or power issues around 10:00am EDT. Engineers are working on assessing and analyzing the situation. Job scheduling on affected clusters has been paused while this issue is being addressed. We...
-
Rice scheduling temporarily stopped
The Rice cluster is currently experiencing a vendor bug in its scratch filesystem. To prevent filesystem instability, job scheduling has been paused while engineers are diagnosing the issue. Currently running jobs will not be affected (only starting...
-
Github is currently offline and is not responding. Engineers are currently working on bringing Github back up. We will provide another update later today or as soon as it is back online.
-
Campus power outage affecting multiple clusters
At approximately, 8:30am EDT, the Brown, Hammer, Rice, and Snyder clusters became unavailable due to a campus power outage. While power has been restored, engineers are currently working returning the clusters to service. Job scheduling has been paus...
-
The Fortress tape archive began experiencing issues with Globus and HSI access around 2:30pm EDT on Sunday, March 10th, 2019. Engineers are currently diagnosing the issue and are working with the vendor to identify a fix. We will provide an update by...
-
Home Filesystem / Slowdown / Login Issue on all Clusters
All clusters began experiencing issues with logins and a general slowdown around 3:10pm EST. This has been identified as being due to an issue on the /home/ filesystem. Engineers are continuing to examine this and are working to alleviate the issue r...
-
Fortress began experiencing issues around 11:15am EST. Engineers are currently diagnosing the issue and are working to identify a fix. We will provide an update by 4:00pm.
-
Unscheduled Brown and BrownGPU outage
The Brown and BrownGPU cluster began experiencing issues with the scratch filesystem around 8:00am EST. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed...
-
Halstead and Brown unscheduled outage
Halstead, HalsteadGPU, Brown, and BrownGPU went offline during a campus power event around 8:40 am this morning. Engineers are working to bring the compute nodes and the scratch system back online. Other systems are back online at this time. Job sche...
-
A power event around 8:40 am this morning affected all Research Computing resources, along with networking, and other resources around campus. Networking is offline making all resources inaccessible. Networking is slowly coming back online around 9:3...
-
The Workbench cluster began experiencing issues with the server which manages its Thinlinc virtual desktop around 10:30pm EST. Currently, no new Thinlinc sessions will be able to be started. Engineers are currently diagnosing the issue and are worki...
-
Unscheduled Fortress Archive Outage
The Fortress Archive was undergoing routine monthly maintenance at 8:00am EST when it suffered a failure of the internal database. Engineers are working with the vendor to repair the database and restore service as soon as possible. Until then, any a...