Unscheduled outage on Carter
February 4, 2016 8:00am – 10:30am
The cause of this turned out to be a power loss to Carter's scratch filesystem and portions of the Data Depot, which has been restored now. Carter nodes are returning to normal operations now.
As of Thursday, February 4th, 2016 at 8:00am, all nodes in Carter nodes are again offline. During unrelated power work in the Math datacenter, a breaker was tripped that resulted in core network routers losing power. This in turn has caused all the Carter nodes to go offline again.
Our engineers are assessing the situation now and we will post an update as soon as we know more.