Unscheduled multiple clusters and Data Depot outage
Link to update at October 20, 2021 6:11pm EDT UPDATE:
The engineers have worked with the vendor to successfully identify and isolate the root cause, and have put an appropriate mitigation in place. As of 6:10pm, the load on the Data Depot servers has returned to normal, and the filesystem performance is back to its expected parameters. We will continue monitoring the filesystem to ensure its availability and performance.
We apologize for the disruption of service. Please report any issues to rcac-help@purdue.edu.
Link to update at October 20, 2021 12:20pm EDT UPDATE:
The load on Data Depot servers has spiked overnight and continues to be very high, with a significant impact on the overall filesystem performance. Engineers are working with the vendor on identifying the problem.
We will provide an update by 6pm.
Link to update at October 19, 2021 9:49pm EDT UPDATE:
As of 9:49pm EDT, the Data Depot load has stabilized, offending processes terminated and Bell, Brown, Gilbreth, Halstead, Hammer, Scholar, Workbench clusters have been returned to normal service. We apologize for the disruption of service. Please report any issues to rcac-help@purdue.edu.
Link to original posting ORIGINAL:
The Bell, Brown, Gilbreth, Halstead, Hammer, Scholar, Workbench clusters and Data Depot began experiencing issues with intermittent high load on the Data Depot servers around 4:30pm EDT. Engineers are currently diagnosing the issue and are working to identify a fix.
We will provide an update by 8pm.