Outages
-
Work continues on bringing Hammer back to normal operation. Engineers have identified the source of the problem and are currently working to find a solution. We will provide another update by 3pm today.
-
Unscheduled Hammer outage - Closed
Update: The issues with the Hammer cluster has been resolved and the cluster is back in production. This outage is closed. Original: The Hammer cluster began experiencing issues around 10:10am EDT. Engineers are currently diagnosing the issue and are...
-
Unscheduled data center outage
Several clusters have experienced network connectivity and/or power issues around 10:00am EDT. Engineers are working on assessing and analyzing the situation. Job scheduling on affected clusters has been paused while this issue is being addressed. We...
-
Rice scheduling temporarily stopped
The Rice cluster is currently experiencing a vendor bug in its scratch filesystem. To prevent filesystem instability, job scheduling has been paused while engineers are diagnosing the issue. Currently running jobs will not be affected (only starting...
-
Github is currently offline and is not responding. Engineers are currently working on bringing Github back up. We will provide another update later today or as soon as it is back online.
-
Campus power outage affecting multiple clusters
At approximately, 8:30am EDT, the Brown, Hammer, Rice, and Snyder clusters became unavailable due to a campus power outage. While power has been restored, engineers are currently working returning the clusters to service. Job scheduling has been paus...
-
The Fortress tape archive began experiencing issues with Globus and HSI access around 2:30pm EDT on Sunday, March 10th, 2019. Engineers are currently diagnosing the issue and are working with the vendor to identify a fix. We will provide an update by...
-
Home Filesystem / Slowdown / Login Issue on all Clusters
All clusters began experiencing issues with logins and a general slowdown around 3:10pm EST. This has been identified as being due to an issue on the /home/ filesystem. Engineers are continuing to examine this and are working to alleviate the issue r...
-
Fortress began experiencing issues around 11:15am EST. Engineers are currently diagnosing the issue and are working to identify a fix. We will provide an update by 4:00pm.
-
Unscheduled Brown and BrownGPU outage
The Brown and BrownGPU cluster began experiencing issues with the scratch filesystem around 8:00am EST. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed...
-
Halstead and Brown unscheduled outage
Halstead, HalsteadGPU, Brown, and BrownGPU went offline during a campus power event around 8:40 am this morning. Engineers are working to bring the compute nodes and the scratch system back online. Other systems are back online at this time. Job sche...
-
A power event around 8:40 am this morning affected all Research Computing resources, along with networking, and other resources around campus. Networking is offline making all resources inaccessible. Networking is slowly coming back online around 9:3...
-
The Workbench cluster began experiencing issues with the server which manages its Thinlinc virtual desktop around 10:30pm EST. Currently, no new Thinlinc sessions will be able to be started. Engineers are currently diagnosing the issue and are worki...
-
Unscheduled Fortress Archive Outage
The Fortress Archive was undergoing routine monthly maintenance at 8:00am EST when it suffered a failure of the internal database. Engineers are working with the vendor to repair the database and restore service as soon as possible. Until then, any a...
-
Unscheduled Rice scratch outage
The Rice cluster began experiencing issues with its scratch system around 4:30pm EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed. We will provide...
-
The Data Depot central servers lost their connection to Depot and all rebooted around 11:45am EDT. Servers are already returning, but engineers are watching the situation closely. Scheduling on clusters is being paused. Cluster jobs might be able...
-
The Data Workbench and WSC Hadoop cluster began experiencing issues with connectivity around 2:00pm EDT. Engineers are currently diagnosing the issue and are working to identify a fix. We will provide an update by 3 pm.
-
Scratch unavailable on Snyder cluster
As of approximately 2:30pm EDT, the Snyder cluster is currently experiencing issues with its scratch filesystem. Engineers are currently diagnosing the issue and are working to identify a fix. Attempts to access scratch will likely fail. Job scheduli...
-
The Rice cluster is currently experiencing issues with the scratch filesystem. Engineers have identified the problem and are in the process of applying the fix. Job scheduling has been paused while this issue is being addressed. We will provide an up...
-
Unscheduled network outage on Brown, Rice, Snyder and Hammer
The Brown, Hammer, Rice, and Snyder clusters began experiencing issues with scratch filesystems and network connectivity around 2:30am EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused...