Outages and Maintenance
-
The Weber cluster began experiencing issues with weber-sftp subsystem around 2:00pm EST. The problem affects ingress/egress path to the cluster. Engineers are currently diagnosing the issue and are working to identify a fix. We will provide an upda...
-
Fortress Archive Monthly Maintenance
The Fortress Archive will be unavailable Wednesday, January 5, 2022 from 8:30am - 12:00pm EST for scheduled monthly maintenance (first Wednesday of every month). During this time, Fortress will receive normal software and hardware updates. Any transf...
-
The Scholar cluster will be unavailable January 4, 2022 8:00am - January 5, 2022 6:00pm EST for scheduled maintenance. The cluster will return to full production by Wednesday, January 5th, 2022 at 6:00pm EST. During this time, Scholar will have the...
-
The Bell cluster began experiencing issues with its scratch filesystem around 6:30pm EST. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed. We will prov...
-
The Weber cluster began experiencing issues with expired VPN certificate around 10:00am EST. Engineers are currently diagnosing the issue and are working to identify a fix. We will provide an update by 5pm.
-
Fortress Archive Monthly Maintenance
The Fortress Archive will be unavailable Wednesday, December 1, 2021 from 8:30am - 12:00pm EST for scheduled monthly maintenance (first Wednesday of every month). During this time, Fortress will receive normal software and hardware updates. Any trans...
-
Fortress Archive Monthly Maintenance
The Fortress Archive will be unavailable Wednesday, November 3, 2021 from 8:30am - 12:00pm EDT for scheduled monthly maintenance (first Wednesday of every month). During this time, Fortress will receive normal software and hardware updates. Any trans...
-
The Bell cluster began experiencing issues with high load and sluggish performance on the scratch filesystem around 1:20pm EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this...
-
The Weber cluster will be unavailable Wednesday, October 20, 2021 from 8:00am - 8:00pm EDT for scheduled maintenance. The cluster will return to full production by Wednesday, October 20th, 2021 at 8:00pm EDT. During this time, Weber will be expanded...
-
Unscheduled multiple clusters and Data Depot outage
The Bell, Brown, Gilbreth, Halstead, Hammer, Scholar, Workbench clusters and Data Depot began experiencing issues with intermittent high load on the Data Depot servers around 4:30pm EDT. Engineers are currently diagnosing the issue and are working to...
-
Unscheduled Brown and Hammer outage
The Brown and Hammer clusters began experiencing issues with cooling due to problems at the Physical Facilities' chiller plant around 4:40pm EDT. To avoid overheating, job scheduling has been paused while this issue is being addressed. We will provid...
-
Unscheduled multiple clusters and Data Depot outage
The Bell, Brown, Gilbreth, Halstead, Hammer, Scholar, Workbench clusters and Data Depot servers began experiencing issues with Data Depot mounting on Wednesday, September 29th, 2021 around 4:40pm EDT. Engineers are currently diagnosing the issue and...
-
Unscheduled Bell, Brown, Gilbreth, Halstead, Hammer, Scholar, and Data Depot outage
The Bell, Brown, Gilbreth, Halstead, Hammer, Scholar, and Data Depot cluster began experiencing issues with Data Depot mounting around 7:00am EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been...
-
Unscheduled Brown, Halstead, Hammer Gilbreth, and Workbench outage
The Brown, Gilbreth, Halstead, Hammer, and Workbench clusters began experiencing issues with home mounts around Thursday, September 16th, 2021 at 11:00am EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job schedul...
-
Fortress Archive Monthly Maintenance
The Fortress Archive will be unavailable Wednesday, September 1, 2021 from 8:30am - 12:00pm EDT for scheduled monthly maintenance (first Wednesday of every month). During this time, Fortress will receive normal software and hardware updates. Any tran...
-
Unscheduled Data Depot and community clusters outage
At about 9:30am EDT, Data Depot servers started experiencing a ramping high load. Coupled with an ongoing scaling issues with the metadata subsystem, this caused Data Depot to become increasingly unresponsive for both community clusters and network d...
-
Unscheduled Data Depot outage on multiple clusters
The Bell, Brown, Gilbreth, Halstead, Scholar, and Workbench clusters began experiencing issues with mounting old Data Depot filesystem around 12:30am EDT. Multiple nodes are flagged offline by an automatic check, and bioinformatics application suite...
-
Fortress Archive Monthly Maintenance
The Fortress Archive will be unavailable Wednesday, August 4, 2021 from 8:30am - 12:00pm EDT for scheduled monthly maintenance (first Wednesday of every month). During this time, Fortress will receive normal software and hardware updates. Any transfe...
-
Unscheduled Brown and Hammer outage
The Brown and Hammer cluster began experiencing issues with cooling in the POD data center around 5:40pm EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being add...
-
Unscheduled Brown, Hammer and Weber outage
The Brown, Hammer, and Weber clusters began experiencing issues with cooling in the POD data center around 11:00am EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is...