Outages
-
Unscheduled Storage Permissions Outage
Data Depot and other group-restricted spaces began experiencing issues with file permissions around 5pm. Users will notice a "permission denied" error when attempting to access spaces they normally would be able to read based on their unix...
-
Data Depot began experiencing issues with file permissions around 8pm this evening that has since resolved itself. Users will have noticed a "permission denied" error when attempted to access spaces they normally would be able to read based...
-
The Negishi Scratch file system began experiencing issues around 11am EST today (December 16). The issue manifests as a "No space left on device" warning on new file creation. Reading and writing existing files is not affected. Engineers ar...
-
Campus power outage affecting multiple clusters
Shortly before 9:00AM eastern time, many RCAC clusters experienced a power interruption which interrupted some work due to a campus power outage. UPDATE: Engineers have arrived on campus and found additional impacts from today's power interruption. S...
-
Starting around 3:00PM eastern time, Bell's scratch file system began to experience service interruptions and periods of unresponsiveness to user requests. These events were caused by faulty hardware and engineers have temporarily mitigated the probl...
-
The Bell cluster began experiencing issues with the /scratch filesystem. Engineers are currently diagnosing the issue and are working to identify a fix for some files being unavailable. We will provide an update by 6pm.
-
Fortress began experiencing issues with communications around 3:00AM. Engineers are currently diagnosing the issue and are working to identify a fix. We will provide an update by 12:00PM.
-
Unscheduled Globus service outage
The Globus connection began experiencing issues on Data Depot endpoint around 11:00am EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Please use other data transfer methods introduced in the user guide. We will pr...
-
The Bell cluster began experiencing issues with the /scratch filesystem at around 5pm today. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed. We will p...
-
The Scholar cluster began experiencing issues with adding new students from the courses onto the cluster. Engineers are currently diagnosing the issue and are working to identify a fix. The cluster is functioning normally so all the active Scholar us...
-
The Negishi cluster began experiencing issues with SLURM accounting around 8:00am EDT. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed. We will provide...
-
The Scholar custer down from approximately 4:30PM onwards on Wednesday, March 27, 2024. Engineers have solved the issue and Scholar should return to service on Wednesday, March 27th at 8:30PM. If you submitted a job during this time frame, be sure to...
-
Clusters Bell, Geddes and Gilbreth experiencing outages as of 4PM Tuesday March 12th. Engineers are working to resolve the problem. Access to these clusters is impacted and running jobs may have stopped. We will provide an update Tuesday March 12th a...
-
The Negishi cluster began experiencing issues around 2PM, and engineers isolated the problem with Negishi. In order to resolve the issue, the scheduler had to be restarted. If you had jobs running on Negishi, please check them to ensure they are stil...
-
Unscheduled Anvil outage - Partial
The Anvil cluster began experiencing issues with Slurm Scheduling this past week. Engineers are currently diagnosing the root cause and are working to identify a fix. Scheduling is still enabled at this time. You may experience periodic SLURM outage...
-
The Geddes cluster began experiencing networking issues around 9:00am EST. Engineers have identified the issue and are working to bring Geddes back to service. Deployments may be in a degraded state during this time. We will provide an update by 3:00...
-
Gilbreth is experiencing scheduling issues and jobs have been paused while RCAC works to resolve this issue. Running jobs have also been impacted, so you will need to resubmit your job if it was running when the job scheduling resumes. We will provid...
-
Fortress began experiencing issues with its tape library around 5:00PM. Engineers are currently diagnosing the issue and are working to identify a fix. Vendor support has been contacted and diagnostics have been uploaded. During this time tape access...
-
Multiple clusters have been powered off in MATH G109 datacenter due to a water issue in the building. Affected systems are Bell, Brown, Geddes, Gilbreth and Negishi. We will provide an update by 5:00 PM today.
-
Scheduling Paused on Negish Cluster
At about noon today (Tuesday 12 September), we discovered an issue with the scheduler database related to the power outage last Sunday. Scheduling on Negishi has been paused to allow for work on correcting the database problem. We will have an upda...