Campus power outage affecting multiple clusters
Link to update at October 13, 2024 12:30pm EDT UPDATE:
As of 12:30pm, work continues on restoring the affected community clusters to full capacity. At this point, the affected community clusters are in a stable but degraded state and scheduling has been resumed. Jobs are currently running on the nodes which have been restored following the power outage, and engineers are continuing to work on bringing the rest of the affected nodes online.
If you had jobs running around 8:45am when the power interruption occurred, those jobs were terminated, so you may need to check the state of running jobs that were dependent upon work that was interrupted.
We will provide another update when the clusters are returned to full service.
Link to original posting ORIGINAL:
Shortly before 9:00AM eastern time, many RCAC clusters experienced a power interruption which interrupted some work due to a campus power outage.
UPDATE: Engineers have arrived on campus and found additional impacts from today's power interruption. Scheduling on all clusters has been paused while efforts to restore full service are undertaken.