As part of the ongoing effort to upgrade data center capacity for future computing needs, maintenance is scheduled from Wednesday, April 22 at 6 AM through Thursday, April 23 at 5 PM. This will affect all RCAC systems and the Research Network.

During this window, all RCAC systems and research network services will be unavailable, including:

• All Computing Clusters including Bell, Negishi, Gautschi, Scholar, Rowdy, Gilbreth, Hammer, Anvil.
• All Data Storage Systems including Data Depot, Fortress, Anvil Ceph storage, scratch and home storage on clusters, Research Network and ScienceDMZ
• Gateway services including Hubzero, GenAI Studio, Anvil GPT
• www.rcac.purdue.edu
• Geddes
• Globus

How does this maintenance impact you?

• Any Slurm jobs requesting a walltime that would extend past the start of the maintenance will not start and will remain in the queue until after maintenance is complete.
• All active sessions and jobs running on affected systems will be preempted at the start of the outage any queued jobs will not begin until services are restored.
• Access to login nodes, storage systems, and web portals will be unavailable throughout the downtime.
• Automated data workflows that rely on affected systems (e.g., rsync, data pipelines, archive processes) will not function until systems are back online.
• Globus transfers may time out during the maintenance window.

To prepare for this maintenance, we suggest to:

• Download any needed data or scripts before 6:00 AM on April 22.
• Prepare instrumentation devices for the Data Depot to be unavailable and

We appreciate your understanding and cooperation as we complete these necessary upgrades to improve reliability and performance across RCAC infrastructure. For assistance, questions, or concerns contact rcac-help@purdue.edu.

Best regards, Rosen Center for Advanced Computing (RCAC) / Purdue IT

Due to the outage, some jobs on Gilbreth did not requeue automatically. Users should check the status of any jobs that were running early this morning and resubmit them if needed.

During this maintenance window, several clusters will experience a temporary outage so that hardware can be safely powered down while facility work is performed:

• Gautschi, Gilbreth, Negishi, Bell, and Anvil cluster nodes will be powered down.

• The Gilbreth’s legacy V100 GPUs, that are well past their lifetime, will be decommissioned.

• Hammer (Math nodes) and Geddes: A subset of nodes will be powered down but the services will be available, unless communicated separately.

How does this maintenance impact you?

• Clusters listed in this message won’t be available to run jobs during the maintenance.

• Any jobs requesting a walltime which would take them past the start of the maintenance will not start and will remain in the queue until after the maintenance is completed.

• Users can continue to access their data.

• GenAI studio will remain available. This maintenance will position Purdue to support growing computational needs. Users should see long‑term benefits in system reliability and our ability to support future computing and AI resources.

If you have questions about how this outage will affect your work or need support, please contact rcac-help@purdue.edu.

Impact To You: At this time, you may notice latency or brief disruptions when accessing certain on-campus or external resources, especially during peak usage periods.

We appreciate your patience while we work to fully resolve the underlying problem and restore normal network performance. We will provide an update by 5:00PM EST today or sooner.

System monitoring has identified an issue where heavy job activity was overloading the Data Depot filesystem used by the clusters and Globus.

You may see the following impacts:

• Globus transfers to and from Depot collections may fail, stall, or run much more slowly than usual.
• Globus logins may be slow or occasionally fail when accessing Depot endpoints.
• Jobs on RCAC clusters that read from or write to Depot may experience slow file access, delayed directory listings, or timeouts.

Our engineers are investigating the high load from a large number of concurrent jobs and are working to reduce the impact on Depot, Globus, and cluster workloads. Existing jobs will continue to run, but any that are heavily Depot‑I/O‑bound may run more slowly or see I/O errors until performance improves. We will provide another update by 5:00PM EST or sooner if the issue is resolved.

During this maintenance window, several clusters will experience a temporary outage so that hardware can be safely powered down while facility work is performed:

• Gautschi, Gilbreth, Negishi, Bell, and Anvil cluster nodes will be powered down.

• The Gilbreth’s legacy V100 GPUs, that are well past their lifetime, will be decommissioned.

• Hammer (Math nodes) and Geddes: A subset of nodes will be powered down but the services will be available, unless communicated separately.

How does this maintenance impact you?

• Clusters listed in this message won’t be available to run jobs during the maintenance.

• Any jobs requesting a walltime which would take them past the start of the maintenance will not start and will remain in the queue until after the maintenance is completed.

• Users can continue to access their data.

• GenAI studio will remain available. This maintenance will position Purdue to support growing computational needs. Users should see long‑term benefits in system reliability and our ability to support future computing and AI resources.

If you have questions about how this outage will affect your work or need support, please contact rcac-help@purdue.edu.

Note: Deployments running inside Geddes will remain available during this migration. Again, no downtime is expected for services deployed in Geddes. This outage only impacts access to the Rancher web interface.

Users can still interact with and manage their Geddes deployments via the command line with kubectl by updating their KUBECONFIG to use: current-context: "geddes-kube00" instead of: current-context: "geddes"

Changes to Geddes membership via the RCAC web portal will also not be available during this time.

Reach out to rcac-help@purdue.edu if you have any questions.

Research Computing staff members will monitor the status of all computing and data resources in an effort to ensure continuous availability.

Research Computing staff members will monitor the ticketing system throughout the holiday period and answer critical issues and problems. Non-critical user issues and questions will be addressed beginning January 5th, 2026. There will also be no coffee hour consultations during this break.

Scratch file purging (on community clusters with scratch space) will continue as normal during the break, so be sure to archive your files in scratch storage. This does not apply to Data Depot or home directories -- only scratch storage.

Have a wonderful break, everyone, and we look forward to great things in the new year!

We will provide an update by 11 AM.

To facilitate the data center renovation project, the composable cloud platform will undergo a maintenance between Thursday, August 14th, 2025 at 5:00am EDT and Thursday, August 14th, 2025 at 8:00pm EDT.

What to Expect during Maintenance:

During the maintenance, the websites and services hosted on the Geddes cluster will be unavailable. We would like to kindly ask the managers and/or principal investigators of services hosted on Geddes to notify their students and other general users as necessary and appropriate, e.g. by posting a banner on their service's website, or through a mailing list (if available).

Post Maintenance:

Upon completion of the maintenance, the services are expected to resume automatically. Regardless, we recommend verifying that websites and services are running normally after the maintenance and notifying us if there are any issues.

If you have constraints around making your service available during this window, have any questions or concerns about the maintenance, or need support please do not hesitate to reach out to rcac-help@purdue.edu...

Thank you!

Update: 3:00PM, EDT: Systems returning but queues are still paused on all clusters except for Bell and Hammer. Some clusters are waiting for full cooling to come fully back online. Next update at 4:00PM, EDT, or when all systems are returned to service.

Update: 2:30PM, EDT: Systems are steadily coming back online. Queues are still paused. Next update at 3:00PM, EDT.

The Rosen Center is currently experiencing an outage on all clusters due to a widespread power outage event on the Purdue West Lafayette Campus.

Currently engineers are bringing all services back up. Scheduling on all clusters will be resumed once services have stabilized.

We expect services to back shortly. We will update on the situation at 2:30PM, EDT.

We know how important timely access is to your research and truly appreciate your patience as this gets sorted out. If you have an urgent need or specific concerns, please don’t hesitate to contact us at rcac-help@purdue.edu.

Thanks again for your understanding.

We are actively monitoring the situation and working to resolve the disruptions as quickly as possible. During this time, you may encounter intermittent connectivity problems or degraded performance.

We appreciate your patience and understanding. If you encounter problems that require immediate attention, please contact rcac-help@purdue.edu. For Anvil users, please submit a ticket through ACCESS Help Desk at https://support.access-ci.org/help-ticket.

This is a reminder that the RCAC license server will be unavailable on today from 11:00 AM to 11:30 AM EST for scheduled maintenance. Licenses hosted by RCAC will be unavailable for checkout for the duration of the maintenance, and Slurm job scheduling will be paused.

Original Message:

The RCAC license server will be unavailable on Thursday, February 20th, 2025 from 11:00 AM to 11:30 AM EST for scheduled maintenance. During this time, the server will migrate to an upgraded host with security enhancements.

Licenses hosted by RCAC will be unavailable for checkout for the duration of the maintenance, and Slurm job scheduling will be paused. Licenses hosted outside of RCAC will not be affected. Jobs started before 11:00 AM will not be affected unless a license must be checked out in the middle of job execution. Jobs may still be submitted to Slurm during the maintenance, but will not be scheduled until after its completion. After this maintenance, RCAC license server access will require a connection to a local Purdue network or to the Purdue VPN (webvpn.purdue.edu).

If you have any questions or comments regarding this migration, please reach out to rcac-help@purdue.edu.

Research Computing staff members will monitor the status of all computing and data resources in an effort to ensure continuous availability.

Research Computing staff members will monitor the ticketing system throughout the holiday period and answer critical issues and problems. Non-critical user issues and questions will be addressed beginning Thursday, January 2nd, 2025. There will also be no coffee hour consultations during this break.

Scratch file purging (on community clusters with scratch space) will continue as normal during the break, so be sure to archive your files in scratch storage or check for any upcoming purge warning emails during the break. This does not apply to Data Depot or home directories -- only scratch storage.

Have a wonderful break, everyone, and we look forward to great things in the new year!

However, due to reduced staff availability during this period we will be canceling our regularly scheduled Coffee-Hour consultations session for that week. This includes both the Monday in-person consulting hours at the Convergence Center as well as the Tuesday virtual sessions.

UPDATE: Engineers have arrived on campus and found additional impacts from today's power interruption. Scheduling on all clusters has been paused while efforts to restore full service are undertaken.

Traditionally, high-performance resources (HPC) on university campuses have offered multiple tiers of storage to accommodate various workflows, but these options are provided by either POSIX-based storage systems or robotic tape libraries. With the advent of cloud-native technologies like Kubernetes, these traditional storage options are not sufficient for researchers who compute via the cloud. Titled “CC* Data Storage: Software Defined Storage for Composable and HPC Workflows,” this CC* project was aimed at closing the gap between campus and cloud storage through the deployment of a central, shared, multi-petabyte Ceph distributed storage system. The project held three major goals: to enhance campus shared storage capabilities, to support Science Domains via innovative storage infrastructure, and to enable education and workforce development by engaging with undergraduate students in the deployment and operation of the storage infrastructure.

“Five years ago,” says Erik Gough, a Senior Research Scientist for RCAC, “we saw the rise of Kubernetes and cloud computing in the campus environment, and some of the emerging scientific use cases didn’t fit that well onto our batch HPC systems. Our traditional storage resources didn’t work, so we needed to have software defined storage for that system. To address the problem, we deployed a small Ceph cluster inside Geddes that people could use. Through this grant, we can significantly expand our storage capacity and support multiple science domains across campus.”

The CC* grant was awarded in fall of 2022. Erik Gough is the Principle Investigator (PI) for the project, with Elizabett Hillery and Di Qi serving as Co-Pis. Wen-wen Tung also served as a former Co-PI. Together, the group not only expanded RCAC’s data storage options, but fostered education and workforce development. Undergraduate students enrolled in one of Tung’s courses were brought into the project, helping to build and deploy the new Ceph storage system with RCAC’s Geddes cluster. They were able to gain real-world experience in the field of high-performance computing.

Thanks to the CC* award and the hard work of both the PI team and the students, RCAC now offers new storage options, including Data Depot Block and Data Depot Object Storage:

• "Depot Object" - S3 compatible object storage for programmatic access to data across RCAC resources and a cold storage tier
• "Depot Block" - Block storage for virtualization on HPC resources and composable systems

If you would like to learn more about the available storage resources, please visit: https://www.rcac.purdue.edu/storage

The CC* award is funded under NSF award number 2232872. To learn more about High-Performance Computing, please visit our “Why HPC?” page. To stay up-to-date on all RCAC projects and updates, please visit our “News” page.

Written by: Jonathan Poole, poole43@purdue.edu