Outages and Maintenance
-
Unscheduled Outage on Data Depot
As of 7:30 pm, all methods for connecting to Data Depot have been restored to working order. All connections with Samba (Network Drive mappings: datadepot.rcac.purdue.edu, samba.rcac.purdue.edu) are working normally again. More Rice and Snyder nodes...
-
Engineering Computing Network (ECN) will be performing scheduled maintenance this weekend on several ECN server resulting in their unavailability for a short time. Some ECN services will be affected, including several software license servers for ITa...
-
The maintenance is complete and github.rcac is back in production, now at version 2.6.4. There are several bug fixes and security patches in this version, but no major feature updates. Additionally, an infrastructure change has been made to the way...
-
The maintenance is complete and github.rcac is back in production, now at version 2.6.4. There are several bug fixes and security patches in this version, but no major feature updates. Additionally, an infrastructure change has been made to the way...
-
The underlying storage has been fixed, and all these clusters have been returned to normal operations as of 10:00pm EDT. As of Tuesday, June 7th, 2016 at 4:10pm EDT, Conte, Hansen, Hathi, and Radon are unavailable due to a loss of Isilon home direct...
-
The Depot maintenance has been completed successfully. Depot is now back to normal operations. Other compute cluster outages running concurrent with this however, are still in progress. This maintenance window next week has been reduced as much as...
-
Carter and Scholar are back online for use as of 6:25am, though they will be operating with many nodes still offline. Staff will be working through Wednesday to steadily increase the number of nodes available. This concludes the POD cluster mainten...
-
An issue was discovered shortly after Conte, Hansen, Hathi, and Radon were brought back online with the /group path on several front-ends and nodes. Any scripts or jobs that rely on the /group path may have had issues immediately following return to...
-
Networking to and from campus, and around large parts of campus are down. Many services are unreachable at the moment. We will provide updates as they become available.
-
Networking to and from campus, and around large parts of campus are down. Many services are unreachable at the moment. We will provide updates as they become available.
-
Network issues affecting Snyder cluster
The problem is now RESOLVED after the reboot of a router. ======= The network serving Snyder is currently experiencing issues. Attempts to log in to the cluster or access network filesystems will experience high latency and delays. Systems engineers...
-
The Isilon filesystem was restored to normal service and all affected clusters had it remounted as quickly as was sustainable by the filesystem. This process was completed by Wednesday, May 18th, 2016 at 12:15am EDT. All clusters other than Conte (...
-
Conte has been returned to normal operations as of Wednesday, May 18th, 2016 at 5:55am EDT. All upgrades were completed, though a small number of nodes which require more attention to be fully ready for jobs remain offline for now and will be return...
-
Unscheduled Scratch Outage on Carter
UPDATE: The issue with Carter's scratch filesystem has been resolved. The filesystem is now available. Job scheduling on the cluster has been resumed. The scratch filesystem serving Carter is currently unavailable. Job scheduling on Carter has bee...
-
Unscheduled Scheduling Outage on Hansen
Job scheduling on Hansen has returned to normal. This concludes the outage. Original Message: Hansen is not currently scheduling any new jobs. A filesystem used for a specific project serving Hansen became unavailable on Hansen around 5:00am. This...
-
Unscheduled Scratch Outage on Carter
The scratch storage on Carter and Scholar has been returned to normal operations. The rebuild process will be continuing in the background, so we will be watching for any degradation in the storage performance. All queues have been re-activated. T...
-
Standby queues paused on Hansen Cluster
Update As of 5:20pm the standby and standby-c queues have been started and their jobs are being scheduled for execution. The standby queues on Hansen have been paused temporarily in order to address a slow scheduling issue. We expect to have them ba...
-
Unscheduled outage on Peregrine-1
Outage RESOLVED A misconfiguration that caused an unneeded IB driver to be loaded was fixed. Peregrine-1 is back online. Job scheduling is on. Original Message: The Peregrine-1 cluster is currently offline due to problems with the cluster nodes' op...
-
Unscheduled outage for Peregrine1
As of Monday, March 7th, 2016 at 12:30pm EST, the Peregrine1 cluster is unavailable due to a failed network switch in its datacenter. This switch is currently in the process of being replaced. Estimated time to complete this work and bring the clu...
-
ECN services outage - ITaP Research Computing systems impacted
Engineering Computing Network (ECN) will be performing staged patching and reboots of all of ECN's RedHat Linux workstations and servers to protect against a serious vulnerability in glibc system library. A significant number of ECN services will be...