Outages
-
Degraded performance of several systems
We have seen a significant wave of these events this morning, September 21. For the most part, this wave seems to have been linked to a storage problem that has been resolved. However, we are implementing new monitoring and response procedures toda...
-
Unscheduled Outage on Data Depot
UPDATE As of 5:30 pm. Friday, 5 August, 2016, we believe the problem affecting access to the Data Depot has been corrected. Thank you for your patience, and I apologize for the disruption this caused. Original Message Access to the Data Depot is curr...
-
Self-service management web tool outage
As of 3:20 pm, the self-service tool is back in action. An issue with the database backing authentication was discovered and repaired. Original message The self-service management tool (user management) is experiencing issues with authentication. Att...
-
Unscheduled Outage on Data Depot
As of 7:30 pm, all methods for connecting to Data Depot have been restored to working order. All connections with Samba (Network Drive mappings: datadepot.rcac.purdue.edu, samba.rcac.purdue.edu) are working normally again. More Rice and Snyder nodes...
-
Engineering Computing Network (ECN) will be performing scheduled maintenance this weekend on several ECN server resulting in their unavailability for a short time. Some ECN services will be affected, including several software license servers for ITa...
-
The underlying storage has been fixed, and all these clusters have been returned to normal operations as of 10:00pm EDT. As of Tuesday, June 7th, 2016 at 4:10pm EDT, Conte, Hansen, Hathi, and Radon are unavailable due to a loss of Isilon home direct...
-
Networking to and from campus, and around large parts of campus are down. Many services are unreachable at the moment. We will provide updates as they become available.
-
Networking to and from campus, and around large parts of campus are down. Many services are unreachable at the moment. We will provide updates as they become available.
-
Network issues affecting Snyder cluster
The problem is now RESOLVED after the reboot of a router. ======= The network serving Snyder is currently experiencing issues. Attempts to log in to the cluster or access network filesystems will experience high latency and delays. Systems engineers...
-
The Isilon filesystem was restored to normal service and all affected clusters had it remounted as quickly as was sustainable by the filesystem. This process was completed by Wednesday, May 18th, 2016 at 12:15am EDT. All clusters other than Conte (...
-
Unscheduled Scratch Outage on Carter
UPDATE: The issue with Carter's scratch filesystem has been resolved. The filesystem is now available. Job scheduling on the cluster has been resumed. The scratch filesystem serving Carter is currently unavailable. Job scheduling on Carter has bee...
-
Unscheduled Scheduling Outage on Hansen
Job scheduling on Hansen has returned to normal. This concludes the outage. Original Message: Hansen is not currently scheduling any new jobs. A filesystem used for a specific project serving Hansen became unavailable on Hansen around 5:00am. This...
-
Unscheduled Scratch Outage on Carter
The scratch storage on Carter and Scholar has been returned to normal operations. The rebuild process will be continuing in the background, so we will be watching for any degradation in the storage performance. All queues have been re-activated. T...
-
Standby queues paused on Hansen Cluster
Update As of 5:20pm the standby and standby-c queues have been started and their jobs are being scheduled for execution. The standby queues on Hansen have been paused temporarily in order to address a slow scheduling issue. We expect to have them ba...
-
Unscheduled outage on Peregrine-1
Outage RESOLVED A misconfiguration that caused an unneeded IB driver to be loaded was fixed. Peregrine-1 is back online. Job scheduling is on. Original Message: The Peregrine-1 cluster is currently offline due to problems with the cluster nodes' op...
-
Unscheduled outage for Peregrine1
As of Monday, March 7th, 2016 at 12:30pm EST, the Peregrine1 cluster is unavailable due to a failed network switch in its datacenter. This switch is currently in the process of being replaced. Estimated time to complete this work and bring the clu...
-
ECN services outage - ITaP Research Computing systems impacted
Engineering Computing Network (ECN) will be performing staged patching and reboots of all of ECN's RedHat Linux workstations and servers to protect against a serious vulnerability in glibc system library. A significant number of ECN services will be...
-
Unscheduled Outage on Data Depot
The Depot filesystem checks have all completed cleanly and the Depot has been fully returned to normal operations. All queues on all clusters are scheduling new jobs again. Any existing jobs which had been waiting for Depot access may also resume....
-
Unscheduled outage on Rice and Snyder
As of 9:15 PM, the Snyder and Rice clusters have been brought back into service after cooling was brought back online. Front-ends are operational and scheduling has been resumed. Original Message: At about 7:30 pm Wednesday, 17 February, 2016, the fr...
-
Unscheduled scratch outage on Carter
There was an issue with the cluster's gateway switches, causing infiniband traffic to be incapable of IP over infiniband. This also caused an instability in the lustre scratch servers, which required that they be rebooted. Jobs that were using scratc...