Outages
-
The Weber cluster began experiencing issues with expired VPN certificate around 10:00am EST. Engineers are currently diagnosing the issue and are working to identify a fix. We will provide an update by 5pm.
-
The Bell cluster began experiencing issues with high load and sluggish performance on the scratch filesystem around 1:20pm EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this...
-
Unscheduled multiple clusters and Data Depot outage
The Bell, Brown, Gilbreth, Halstead, Hammer, Scholar, Workbench clusters and Data Depot began experiencing issues with intermittent high load on the Data Depot servers around 4:30pm EDT. Engineers are currently diagnosing the issue and are working to...
-
Unscheduled Brown and Hammer outage
The Brown and Hammer clusters began experiencing issues with cooling due to problems at the Physical Facilities' chiller plant around 4:40pm EDT. To avoid overheating, job scheduling has been paused while this issue is being addressed. We will provid...
-
Unscheduled multiple clusters and Data Depot outage
The Bell, Brown, Gilbreth, Halstead, Hammer, Scholar, Workbench clusters and Data Depot servers began experiencing issues with Data Depot mounting on Wednesday, September 29th, 2021 around 4:40pm EDT. Engineers are currently diagnosing the issue and...
-
Unscheduled Bell, Brown, Gilbreth, Halstead, Hammer, Scholar, and Data Depot outage
The Bell, Brown, Gilbreth, Halstead, Hammer, Scholar, and Data Depot cluster began experiencing issues with Data Depot mounting around 7:00am EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been...
-
Unscheduled Brown, Halstead, Hammer Gilbreth, and Workbench outage
The Brown, Gilbreth, Halstead, Hammer, and Workbench clusters began experiencing issues with home mounts around Thursday, September 16th, 2021 at 11:00am EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job schedul...
-
Unscheduled Data Depot and community clusters outage
At about 9:30am EDT, Data Depot servers started experiencing a ramping high load. Coupled with an ongoing scaling issues with the metadata subsystem, this caused Data Depot to become increasingly unresponsive for both community clusters and network d...
-
Unscheduled Data Depot outage on multiple clusters
The Bell, Brown, Gilbreth, Halstead, Scholar, and Workbench clusters began experiencing issues with mounting old Data Depot filesystem around 12:30am EDT. Multiple nodes are flagged offline by an automatic check, and bioinformatics application suite...
-
Unscheduled Brown and Hammer outage
The Brown and Hammer cluster began experiencing issues with cooling in the POD data center around 5:40pm EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being add...
-
Unscheduled Brown, Hammer and Weber outage
The Brown, Hammer, and Weber clusters began experiencing issues with cooling in the POD data center around 11:00am EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is...
-
The Brown cluster began experiencing issues with cooling around 9:00pm EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed. We will provide an update...
-
Scheduling Paused on Multiple Clusters
At about 4:00 pm today (Wednesday, 21 July, 2021) System Engineers found an issue with the schedulers on the Bell, Brown, Gilbreth, Halstead, and Scholar clusters. Job scheduling has been paused while this is being investigated. Symptoms of this pro...
-
The Gilbreth cluster began experiencing issues with its scratch file system around 5:00pm EDT on Thursday, July 1st, 2021. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue...
-
The Bell cluster began experiencing issues with its home and scratch directories filesystem around 12:40pm EDT. Problems manifest as hanging new logins and unresponsive established sessions. Engineers are currently diagnosing the issue and are workin...
-
Intermittent Access Failures on Data Depot
As of Thursday, June 17th, 2021 at 11:00am EDT, users of community clusters may experience intermittent "permission denied" errors while trying to access their files on Data Depot. Errors may come and go, and may appear on both login and c...
-
The Fortress tape archive began experiencing load-induced issues around 1:00pm EDT. Problems manifest as various errors and timeouts while trying to access Fortress or transfer data. Engineers are currently diagnosing the issue and are working to ide...
-
Unscheduled outage on multiple clusters
Due to problems with cooling system in the MATH datacenter, the CMS, Bell, Brown, Gilbreth, Halstead, WCERES, and WSC Hadoop clusters began experiencing issues around 4:00pm EDT. Multiple front-end, compute and storage services are affected. Engineer...
-
ANSYS Fluent software unavailable on Bell
We have received multiple reports about ANSYS Fluent software on Bell cluster being unavailable. We are currently diagnosing the issue and are working to identify a fix. We will provide an update by 6pm tonight.
-
The Workbench cluster began experiencing issues with its network uplink around 6:30pm EST. Engineers are currently diagnosing the issue and are working to identify a fix. We will provide an update by 10 pm.