Outages
-
UPDATE As of 7:50 pm, Wednesday, 14 December 2016, this issue is completely resolved. UPDATE As of about 6:00 pm another problem has been found in the EXRC scheduler code. We will update this news item once we have more details. Original Item The EXR...
-
Update: Engineers were able to isolate the problem and restart the necessary systems. The Data Depot should be available again. Halstead users should double check their running work. A portion of the systems serving the Research Data Depot have suffe...
-
Job scheduling paused on Radon
Job scheduling was paused on Radon between 6 pm and 7 pm this evening. Node monitoring processes marked most nodes offline around 6 pm, preventing new jobs from starting. System engineers cleared the fault in the node monitoring, and nodes came back...
-
This issue has been resolved. Original Message: A portion of the systems serving the Research Data Depot have suffered a failure. Some systems using Depot have been affected, particularly Carter, Snyder, and systems accessing Depot over NFS. Some sys...
-
GitHub issues - Internal Server Error and Web Interface Not Updating
The issue with the GitHub web interface was resolved late yesterday evening. The website is now reflecting changes made to git repositories as normal. Please let us know at rcac-help@purdue.edu if you see any continuing issues. The sporadic "Int...
-
Measures taken within the first two hours of this problem seem to have resolved the issue. Original Message: A portion of the systems serving the Research Data Depot have suffered a failure. Some systems using Depot have been affected, particularly...
-
Unscheduled Scratch Outage on Carter
UPDATE As of about 6:30 pm, the new scratch system was brought back online, and scheduling has been restarted on Carter. Original Message The new scratch filesystem serving Carter that was just activated on Tuesday night is currently unavailable. Bot...
-
Unscheduled scratch outage on Carter
UPDATE: ITaP engineers have implemented a temporary solution so that work may continue on Carter until the scheduled upcoming maintenance window on Tuesday. Any jobs running which were using the scratch space have been stopped in order to allow for t...
-
Degraded performance of several systems
We have seen a significant wave of these events this morning, September 21. For the most part, this wave seems to have been linked to a storage problem that has been resolved. However, we are implementing new monitoring and response procedures toda...
-
Unscheduled Outage on Data Depot
UPDATE As of 5:30 pm. Friday, 5 August, 2016, we believe the problem affecting access to the Data Depot has been corrected. Thank you for your patience, and I apologize for the disruption this caused. Original Message Access to the Data Depot is curr...
-
Self-service management web tool outage
As of 3:20 pm, the self-service tool is back in action. An issue with the database backing authentication was discovered and repaired. Original message The self-service management tool (user management) is experiencing issues with authentication. Att...
-
Unscheduled Outage on Data Depot
As of 7:30 pm, all methods for connecting to Data Depot have been restored to working order. All connections with Samba (Network Drive mappings: datadepot.rcac.purdue.edu, samba.rcac.purdue.edu) are working normally again. More Rice and Snyder nodes...
-
Engineering Computing Network (ECN) will be performing scheduled maintenance this weekend on several ECN server resulting in their unavailability for a short time. Some ECN services will be affected, including several software license servers for ITa...
-
The underlying storage has been fixed, and all these clusters have been returned to normal operations as of 10:00pm EDT. As of Tuesday, June 7th, 2016 at 4:10pm EDT, Conte, Hansen, Hathi, and Radon are unavailable due to a loss of Isilon home direct...
-
Networking to and from campus, and around large parts of campus are down. Many services are unreachable at the moment. We will provide updates as they become available.
-
Networking to and from campus, and around large parts of campus are down. Many services are unreachable at the moment. We will provide updates as they become available.
-
Network issues affecting Snyder cluster
The problem is now RESOLVED after the reboot of a router. ======= The network serving Snyder is currently experiencing issues. Attempts to log in to the cluster or access network filesystems will experience high latency and delays. Systems engineers...
-
The Isilon filesystem was restored to normal service and all affected clusters had it remounted as quickly as was sustainable by the filesystem. This process was completed by Wednesday, May 18th, 2016 at 12:15am EDT. All clusters other than Conte (...
-
Unscheduled Scratch Outage on Carter
UPDATE: The issue with Carter's scratch filesystem has been resolved. The filesystem is now available. Job scheduling on the cluster has been resumed. The scratch filesystem serving Carter is currently unavailable. Job scheduling on Carter has bee...
-
Unscheduled Scheduling Outage on Hansen
Job scheduling on Hansen has returned to normal. This concludes the outage. Original Message: Hansen is not currently scheduling any new jobs. A filesystem used for a specific project serving Hansen became unavailable on Hansen around 5:00am. This...