Impact to you Attempts to read files that are on the affected storage may result in error messages

Our IT team is actively working with the vendor to restore service as quickly as possible. We will send an update as soon as more information is available.

During this maintenance window, several clusters will experience a temporary outage so that hardware can be safely powered down while facility work is performed:

• Gautschi, Gilbreth, Negishi, Bell, and Anvil cluster nodes will be powered down.

• The Gilbreth’s legacy V100 GPUs, that are well past their lifetime, will be decommissioned.

• Hammer (Math nodes) and Geddes: A subset of nodes will be powered down but the services will be available, unless communicated separately.

How does this maintenance impact you?

• Clusters listed in this message won’t be available to run jobs during the maintenance.

• Any jobs requesting a walltime which would take them past the start of the maintenance will not start and will remain in the queue until after the maintenance is completed.

• Users can continue to access their data.

• GenAI studio will remain available. This maintenance will position Purdue to support growing computational needs. Users should see long‑term benefits in system reliability and our ability to support future computing and AI resources.

If you have questions about how this outage will affect your work or need support, please contact rcac-help@purdue.edu.

Impact To You: At this time, you may notice latency or brief disruptions when accessing certain on-campus or external resources, especially during peak usage periods.

We appreciate your patience while we work to fully resolve the underlying problem and restore normal network performance. We will provide an update by 5:00PM EST today or sooner.

System monitoring has identified an issue where heavy job activity was overloading the Data Depot filesystem used by the clusters and Globus.

You may see the following impacts:

• Globus transfers to and from Depot collections may fail, stall, or run much more slowly than usual.
• Globus logins may be slow or occasionally fail when accessing Depot endpoints.
• Jobs on RCAC clusters that read from or write to Depot may experience slow file access, delayed directory listings, or timeouts.

Our engineers are investigating the high load from a large number of concurrent jobs and are working to reduce the impact on Depot, Globus, and cluster workloads. Existing jobs will continue to run, but any that are heavily Depot‑I/O‑bound may run more slowly or see I/O errors until performance improves. We will provide another update by 5:00PM EST or sooner if the issue is resolved.

We will provide an update by 2:00PM EST today.

What’s New on the dashboard?

• CPU/GPU Usage: Monitor your group usages and remaining available cores on Negishi.
• Disk Usage: Monitor your storage utilization across Negishi’s file systems.
• Job Queue: View and manage your running and queued jobs on Negishi.
• News Feed: Stay updated with the latest Negishi news, outages and announcements.
• Partition Status: Monitor the current state of partitions/queues on Negishi.
• My Jobs Page: Re-designed page to show detailed job information for your jobs and jobs in your group(s) as well as job management.
• Performance Metrics Page: Analyze your job performance and resource utilization patterns over time.

What will impact you?

• All Slurm jobs on Negishi (including jobs that have already submitted through Open Ondemand before this maintenance) will continue and NOT be impacted.
• All functions related to Open Ondemand including login will be unavailable during the maintenance.

Negishi Open Ondemand service will return to full production by Tuesday, January 20 at 5:00pm EDT, 2026.

Please submit a ticket through RCAC Help Desk rcac-help@purdue.edu if you have any questions or suggestions.

During this maintenance window, several clusters will experience a temporary outage so that hardware can be safely powered down while facility work is performed:

• Gautschi, Gilbreth, Negishi, Bell, and Anvil cluster nodes will be powered down.

• The Gilbreth’s legacy V100 GPUs, that are well past their lifetime, will be decommissioned.

• Hammer (Math nodes) and Geddes: A subset of nodes will be powered down but the services will be available, unless communicated separately.

How does this maintenance impact you?

• Clusters listed in this message won’t be available to run jobs during the maintenance.

• Any jobs requesting a walltime which would take them past the start of the maintenance will not start and will remain in the queue until after the maintenance is completed.

• Users can continue to access their data.

• GenAI studio will remain available. This maintenance will position Purdue to support growing computational needs. Users should see long‑term benefits in system reliability and our ability to support future computing and AI resources.

If you have questions about how this outage will affect your work or need support, please contact rcac-help@purdue.edu.

Research Computing staff members will monitor the status of all computing and data resources in an effort to ensure continuous availability.

Research Computing staff members will monitor the ticketing system throughout the holiday period and answer critical issues and problems. Non-critical user issues and questions will be addressed beginning January 5th, 2026. There will also be no coffee hour consultations during this break.

Scratch file purging (on community clusters with scratch space) will continue as normal during the break, so be sure to archive your files in scratch storage. This does not apply to Data Depot or home directories -- only scratch storage.

Have a wonderful break, everyone, and we look forward to great things in the new year!

We will provide an update by 9pm.

The Data Depot file system has returned to full service and scheduling has resumed on all clusters.

The Data Depot storage system began experience issues starting around 9am EDT this morning. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed.

We will provide an update by 12pm (noon).

Data Depot functionality has been restored.

The Data Depot file system began experiencing issues with writes around 2:30pm EDT. The data migration process currently ongoing from Data Depot 2 to Data Depot 3 ran into an unexpected problem. Engineers have identified the problem and are correcting it. Users may have seen "no space left on device" for approximately 30 minutes. Job scheduling has been paused while this issue is being addressed.

We will provide an update by 5 PM.

Currently, Yang and his team focus on semiconductor materials and metals such as copper that are used in semiconductor packaging to study how cracking and deformation happens at the atomic level in critical areas.

Yang and his research group also use the Gilbreth cluster’s GPUs to train machine learning models to predict material behavior and properties. This means in the future they won’t have to run time-consuming simulations because the trained machine learning model will be able to make fast predictions about material behavior and failure.

Yang says there have been no obstacles to using the clusters as a faculty member at the Indianapolis campus, and he’s been able to access the clusters remotely without any difficulties.

He says tapping into RCAC resources has also connected him to faculty in West Lafayette he might not otherwise have met.

“I’ve gotten a lot of connection opportunities, and chances to collaborate with faculty in West Lafayette that are more focused on the experiment side, so we can have that connection between the computational simulation and the experimental science. So that’s been a big benefit as well.”

To learn more about Negishi, Gilbreth and other RCAC resources, contact rcac-help@purdue.edu and visit the RCAC website.

To facilitate the data center renovation project, the clusters will be operating at limited capacity between Thursday, August 14th, 2025 at 5:00am EDT and Thursday, August 14th, 2025 at 8:00pm EDT.

Jobs that request a wall time that will take them past the beginning of the maintenance will remain in queue until after the maintenance is complete.

What to Expect During Maintenance:

1. You can log in to the cluster front-ends and access your files stored on Bell and Negishi.
2. Jobs running at the start of maintenance will be preempted and automatically resubmitted to their respective queues.
3. New jobs submitted during the maintenance window will be queued, but they will not begin running until after the maintenance is complete.

We appreciate your understanding and cooperation. If you have any questions or concerns, please contact us at rcac-help@purdue.edu.

• All jobs on the cluster will be required to explicitly specify a partition and an account (i.e. your group's name) at submission time. You can find the names of the available partitions and accounts from the showpartitions and slist commands respectively. Any job that does not specify an account and a partition will be rejected at submission time.
• The output of slist and the default output of squeue will be modified to reflect the new scheduler design.
• All "shared accounts" such as standby, highmem, etc. that represent resources outside of your typical "group accounts" will continue to exist but will require a different request syntax.
  • Standby will become a Quality of Service (QoS) and jobs that previously ran under the "standby" account, will now be submitted to your "group account" and be tagged with the standby QoS. I.e. if your job previously used the -A standby option, you would now use -A mylab -q standby
  • The highmem and gpu shared accounts will become partitions and jobs that previously ran under these accounts will now be submitted to your "group account" and should be submitted to the appropriate partition, i.e., -A highmem will become -A mylab -p highmem
  • Groups with access to the interactive account will now submit to their "group account" and the interactive partition. i.e. -A interactive will become -A mylab -p interactive

How will this affect you:

1. You will need to change your jobscripts and your method of invocation to include the required options outlined above.
2. If you have any scripts or tooling that rely on the current output of slist or squeue, those scripts will need to be modified to use the new formatted output.

You can prepare for this maintenance by reviewing the new Slurm organization in our user guide's Queues Page.

If you have any questions about these upcoming changes, please reach out to us at rcac-help@purdue.edu.

The Negishi cluster is scheduled for maintenance and will be unavailable on Tuesday, July 29th, 2025 at 8:00am EDT until July 30th, 2025 at 5:00pm EDT.

What is being upgraded?

During this maintenance, Negishi will have its scheduler's configurations updated to allow for the use of more modern features within Slurm, its ZFS and lustre storage systems will be updated, and maintenance will be performed on the cooling loop serving its compute nodes.

How does this affect you?

• The Negishi cluster will be unavailable during the maintenance window.
• Slurm jobs that are still queued when this maintenance begins on Tuesday, July 29th, 2025 at 8:00am EDT will be deleted.
• A reservation will be created that will prevent jobs from starting if their end time would take the job past the start of maintenance. Because pending jobs will be deleted, these jobs will never run.
• The Slurm options required for job submission will change during this maintenance. See our related news posting.
• The output of slist and the default output of squeue will be modified to be more useful under the new design.
• The available options for creating jobs through Open OnDemand will change to accomodate the new options.

How can you prepare for these changes?

In order to minimize disruption in researcher workflows, we have updated Negishi's User Guide page on Job Submission to describe the new method of job submission and this should be reviewed by users before the maintenance.

If you have questions about this upgrade or need help from our support staff, please reach out to us at rcac-help@purdue.edu.

Thank you!

OoD and desktop of Negishi and gilbreth may affect due to node unexpectedly rebooted.

Our engineer is working on the back up right now.

Update on 3:46 pm: Negishi and Gilbreth Nodes are back to normal.

Update: 3:00PM, EDT: Systems returning but queues are still paused on all clusters except for Bell and Hammer. Some clusters are waiting for full cooling to come fully back online. Next update at 4:00PM, EDT, or when all systems are returned to service.

Update: 2:30PM, EDT: Systems are steadily coming back online. Queues are still paused. Next update at 3:00PM, EDT.

The Rosen Center is currently experiencing an outage on all clusters due to a widespread power outage event on the Purdue West Lafayette Campus.

Currently engineers are bringing all services back up. Scheduling on all clusters will be resumed once services have stabilized.

We expect services to back shortly. We will update on the situation at 2:30PM, EDT.

We know how important timely access is to your research and truly appreciate your patience as this gets sorted out. If you have an urgent need or specific concerns, please don’t hesitate to contact us at rcac-help@purdue.edu.

Thanks again for your understanding.

During this time, Negishi will have its operating system patched, and upgrades will be made to SMB network drive connectivity. Filesystems will remain available on the cluster, but existing network drive connections may be disrupted. Login sessions via ThinLinc, SSH, and Open OnDemand will terminate as the servers reboot.

Any Slurm jobs submitted during the maintenance will not start and will remain in the queue until after the maintenance is completed.

What Users Can Expect

• You can log into the clusters and submit jobs during the maintenance, though new jobs will not start.
• Login nodes will reboot, terminating existing sessions, but you will be able to log back in without waiting for the end of the maintenance window.
• While you will have login node access, the ability to check job status or submit new jobs will be briefly interrupted while the scheduler reboots.
• While you will have access to the Open OnDemand interface through Gateway, you may experience a brief interruption when it is rebooted. If that happens, simply reload the web page. If you are running an interactive job, you will be able to reconnect to it.
• The Slurm scheduler will reboot, during which time new Slurm queries and job submissions will be interrupted. Any Slurm jobs running before the maintenance will continue to run to completion under the existing OS.