What’s happening: Our engineers identified a problem with the Gautschi scratch filesystem that requires an emergency upgrade and repair. The procedure must be done with the filesystem offline and is expected to take 8-10 hours or more. To avoid any risk of further damage, we have stopped new jobs from starting on Gautschi as of Monday afternoon, November 2.

Timeline

Now: No new jobs will start.

Tuesday, November 3, 8:00 AM: Remaining non-interactive batch jobs will be requeued, and interactive jobs (including OnDemand sessions) will be ended. Scratch will be unmounted and work will begin.

Estimated completion: Late Tuesday, though it may run longer.

What's Changing

The following changes will take effect on Monday, November 2, 2026.

1. Proportional GPU Limits: The per-account limit of 24 GPUs is being reduced to double the number of GPUs purchased or 2 GPUs, whichever is larger.
2. Reduced Max Walltimes: The current two-week max walltime for normal QOS jobs is being reduced to 4 days.
3. New burst QOS: A new quality of service will be created called burst with the following policies:
   1. Jobs using the burst QOS can use up to 8 GPUs at once for up to 8 hours at a time.
   2. Jobs with this QOS will have the highest priority on the cluster for more immediate start time, however they will only be able to preempt preemptible jobs. Normal QOS jobs will remain unaffected.
   3. Only one job from each account on the cluster can use this QOS at a time.
   4. Jobs using this QOS will be charged at 2x the rate for the GPU hours consumed by their job.
4. New Preemptible Features:
   1. Preemptible jobs will have a GraceTime value of 10 minutes which will allow preemptible jobs a guaranteed 10 minutes to clean-up and checkpoint their work before being interrupted.

Why are New Limits Needed?

As of October 1st, Gautschi has been in production for 19 months, and over that time, we have seen utilization continue to ramp up. Due to Gautschi's allocation method of allowing groups to purchase resources based on GPU hours rather than limiting them to the number of GPUs purchased, groups have been able to use many more GPUs in parallel than purchased. This is by design as we'd like for groups to be able to use the massively parallel capabilities of Gautschi, but while an overly generous limit of 24 H100s may have been suitable for a fledgling cluster with few users, it's become clear that it's impacting fair access to resources and consequently wait times for current users.

Below, we'll provide some of the data that are informing these limit changes as well as provide some discussion on the expected outcomes of each change.

Enforcing Proportional GPU Allocation

Over the past nine months, we've watched the total utilization of the AI partition creep towards 100% while simultaneously watching the percentage of jobs using the "preemptible" QOS fall. This indicates that we are at 100% utilization while there are always "normal" QOS jobs that are waiting to start which has resulted in longer and longer wait times for "normal" QOS jobs and rendered the "preemptible" QOS obsolete due to essentially never having idle cycles.

In large part, the reason for this is because we have allowed all groups on Gautschi to use up to 24 GPUs at once regardless of the size of their purchase into Gautschi. This is distinctly different from how our other community clusters operate. On those clusters, the amount of resources purchased by a group directly affects the priority of their jobs on that cluster through the use of Slurm's "Fairshare" feature by increasing the group's respective "share" of the cluster. For reasons explained in this blog, Gautschi cannot use "Fairshare". This presents a challenge for groups who want to capacity plan for their usage on Gautschi.

In order to address this, we'll be reducing the maximum number of GPUs that can be used by a group under the normal QOS from a flat 24 GPUs per group to a limit proportional to the size of a group's purchase. By scaling proportionally, this will allow us to schedule more fairly while guaranteeing capacity to those who choose to invest in more resources within Gautschi. To enable proportional allocation, we will set each group's limit of concurrent H100s under the "normal" QOS to be twice the number of GPUs purchased or two GPUs, whichever is larger.

When reviewing the average usage of all 97 groups on Gautschi since the start of the semester, 78.3% of groups would not notice this limit, and the new configuration would ensure larger availability of resources for all users, including the current largest consumers of Gautschi.

Why was "2x" the purchase chosen as the scaling factor?

Initially, we thought to simply reduce the number of GPUs each group can use from 24 to 16, but upon looking at the data, we saw that this would not practically change anything. We next considered allowing groups to use up to 4x the number of GPUs they purchased, and again we saw that this would only increase the availability of GPUs to serve normal jobs by a little more than 5%. Finally, when looking at a factor of 2x we saw that this would greatly improve the availability of GPUs to "preemptible" jobs and still allow groups who need all eight GPUs on a Gautschi node the ability to be able to purchase more GPUs and expect that there be idle GPUs for them to run on.

How to request larger quantities of GPUs?

We're introducing a new QOS called burst that will, in exchange for using twice the GPU hours as a normal QOS job, allow groups to use up to eight GPUs for a maximum walltime of eight hours. This will allow users to leverage the fast GPU-to-GPU connection provided in 8-GPU nodes. These jobs will have the highest priority of any job on the cluster. While a burst job cannot preempt a normal QOS job, it will always start before a normal QOS job. Because of the higher resource limits and priority, each group will only be allowed to have one burst job running and one accruing priority at a time. If you expect to need large numbers of GPUs frequently, consider purchasing enough to cover up to half of your regular GPU usage.

We hope that in addition to meeting the resource needs of groups who only need whole node jobs occasionally that this will also be a tool for groups who are willing to pay for more "interactive" access as this has been a request from many PIs.

Reducing Max Walltime per Job

Another limit that we are reducing in this tuning pass is the maximum walltime for each job under the normal QOS. There are a couple of reasons for this, but it essentially boils down to the fact that, empirically, users are requesting many, many more hours/days than their jobs require, and this makes it harder for Slurm to give estimates on start time to users.

Reducing the maximum walltime should allow Slurm to be more efficient in backfilling users' jobs. For a more in-depth discussion as to why that is the case, see the discussion on backfill scheduling in this blog. We recommend that users continue to implement and leverage checkpointing as a best practice to make efficient use of shorter walltimes and preemptible jobs.

New Preemptible Feature for Enabling Checkpointing

As discussed above, in recent months the preemptible QOS has been rendered unusable by the fact that there is always an eligible "normal" QOS job ready to preempt any preemptible job that starts. We expect under these new limits on Gautschi, the resources available to run preemptible jobs without interruption will increase considerably, and we encourage groups, particularly those with multi-GPU needs, to take advantage of these resources by submitting jobs under this quality of service.

To help make this transition smoother for groups that haven't used the preemptible QOS before (and for those that have), we will be adding a GraceTime of 10 minutes to preemptible jobs. What this means is that once your job has been selected to be interrupted by a higher priority job, it will be "told" through a Unix signal, that it has 10 minutes to gracefully shut down. We hope that this will provide an easy mechanism for users to checkpoint their work.

As the renovation and capacity improvement project in the MATH datacenter moves into its final stages, a planned maintenance outage is required to continue bringing the newly installed power and cooling infrastructure online. This work is part of a broader renovation that will result in a 32% increase in floor space, a 60% increase in usable power, and a two-fold increase in cooling capacity — all crucial facility ingredients needed to continue providing world-class high-performance computing resources to Purdue faculty and the nation.

**Outage Window: **Saturday, September 19, 2026, 12:00 AM – Sunday, September 20, 2026, 6:00 PM

What will remain online

The following resources are not affected by this power work and will remain available throughout the maintenance window:

• Data Depot and Data Depot Object
• ScienceDMZ
• Scholar
• Fortress
• HubZero
• Hammer
• Weber

What will go offline

The following resources, including their associated Data Depot storage and Data Depot Object allocations, will be unavailable during the outage window and will return to service gradually once the power work is complete:

• GenAI Studio
• Geddes
• Anvil
• Gautschi
• Gilbreth
• Bell
• Negishi
• Rowdy
• Rossmann

How does this affect you?

Jobs on the affected clusters will not run, and new jobs will not start during this window. Jobs that are expected to not complete before the maintenance begins will not start until after the maintenance window ends. Resources listed above as remaining online will continue to operate normally throughout.

Questions

If you have questions about this outage or need help from our support staff, please contact rcac-help@purdue.edu.

Best, RCAC

As of 3:30PM on August 16th 2026, all clusters with the exception of Gilbreth have been brought back online.

As the renovation and capacity improvement project in the MATH datacenter moves into its final stages, two planned maintenance outages are required to begin to utilize the newly installed power and cooling infrastructure. The MATH renovation will result in a 32% increase in floor space, a 60% increase in usable power, and a two-fold increase in cooling capacity, all of which are crucial facility ingredients needed to continue to provide world-class high-performance computing resources to Purdue faculty and the nation.

Here you can see the project’s progress with newly added floor space to accommodate our next HPC investment, though the new showcase window.

The first power outage is planned for Saturday, August 15, 2026, 12 AM through Sunday August 16, 2026, 6 PM. All clusters, storage, and gateway services such as GenAI Studio will be unavailable during this window and will return to service gradually once the power work is complete:

Data Depot, Geddes. and GenAI studio will return online as soon as power is available Sunday evening and will be ready for the work week.

All other HPC resources will return to service throughout the day and evening starting Monday.

Gilbreth cluster is scheduled for regular maintenance, including operating system patches, on August 18 and will return to service by 8 PM on that day.

The second power outage is planned for Saturday, September 19, 2026, 12 AM through Sunday, September 20, 2026, 6 PM. Additional details for the September maintenance will be shared at a later date.

How does this affect you?

All clusters, storage systems, and gateways will be unavailable during the power outage. Jobs will not run and new jobs will not start during this window. Jobs that are expected to not complete by the start of the maintenance will not start until after the maintenance window.

Questions

If you have questions about this outage or need help from our support staff, please contact rcac-help@purdue.edu.

We will provide another update by 9:00 AM tomorrow.

To protect running jobs, compute nodes will be set to drain starting at 3:00am EDT on June 24. Jobs that cannot complete before 8:00am will remain pending until after the maintenance window.

No user action is required, but you may wish to adjust any time‑sensitive workflows or job submissions to avoid the maintenance period. If you have questions, please contact the RCAC help desk at rcac-help@purdue.edu.

Yanran Wu is a third-year PhD student in Computer Science at Purdue University. Wu attended GHC 25 with assistance from a WHPC scholarship. GHC, eponymously named after pioneering computer scientist Grace Hopper, is the world’s largest gathering of women in tech, bringing together thousands of women from across the globe to learn, network, and celebrate their achievements. We sat down with Wu afterwards to discuss the conference and learn about her experiences whilst there:

Can you describe your overall experience at the conference?

I had a very exciting and inspiring experience at the Grace Hopper Celebration. GHC is very different from academic conferences, and it felt like a truly unique event where women from across the tech industry come together. Everyone was very willing to share helpful information and their life experiences, which created a warm and supportive community. The conference made me feel motivated to push my research forward in ways that connect with real industry needs, and it also encouraged me to support other women in tech in the future.

What was your biggest takeaway from the conference?

My biggest takeaway on research and technology is that there are still clear gaps between current AI applications and what industry teams actually need. Through conversations with engineers, I learned that some engineers hope to develop custom AI agents that work with their internal data, but they face challenges such as low inference efficiency and the high cost of fine-tuning. These discussions helped me think more about how my research can contribute to solving these problems.

On a personal level, I learned a great deal from senior professionals who shared how they made important career choices and navigated different transitions in their lives. Their experiences showed me that each person follows a path that fits their own goals and circumstances. It encouraged me to focus on my long-term development instead of feeling pressure from external expectations.

How did support from Purdue WHPC make this experience possible for you, and how do you feel about receiving the support?

The support from Purdue WHPC made it possible for me to attend the conference, since I would not have been able to cover the travel costs on my own. It allowed me to participate fully and focus on learning, networking, and exploring new opportunities. I feel very grateful for this support, and I also see it as recognition that encourages me to stay involved in the WHPC community and continue growing as a researcher.

Would you recommend the WHPC scholarship program to other Purdue students and why?

Yes, I would definitely recommend the WHPC scholarship program to other Purdue students. It provides valuable support that makes it possible to attend important conferences and gain experiences that can shape both academic and career goals.

Did you have any prior involvement with Research Computing or WHPC?

I first became involved with Research Computing through the Gautschi Early User Program and have continued to use the Gautschi cluster since then. The advanced H100 GPUs on Gautschi allowed me to scale up my experiments and conduct detailed profiling and benchmarking for my research on sustainable AI systems and large language model serving. I am very grateful for the timely help from the RCAC staff, including Ryan [DeRue] and Haniye [Kashgarani], who supported several technical needs for my research project and made it possible for us to run our studies smoothly.

Purdue Women in High-Performance Computing Program

Purdue Women in HPC is part of a broader engagement initiative by the Rosen Center for Advanced Computing (RCAC) and is led by women staffers affiliated with RCAC. WHPC is a dedicated community encompassing undergraduate, graduate, staff, and faculty men and women who are interested in exposing women to high-performance computing and encouraging their pursuit of research and careers in HPC and other technology fields.

Within the WHPC initiative, RCAC provides partial travel assistance to undergraduate and graduate students, empowering their participation in various national conferences that align with WHPC’s objectives. The WHPC travel scholarship sends students to numerous conferences each year, most recently including the Grace Hopper Celebration, Society of Women Engineers, and Game Developers Conference (GDC) Festival of Gaming.

For full eligibility requirements, and to apply for this scholarship, please visit the WHPC Scholarship page.

In addition to the scholarship program, Purdue WHPC organizes a range of activities, including regular meetings to discuss technical HPC-related issues of interest, opportunities to network with the WHPC community, a mentorship program, workshops, and exposure to external resources and opportunities.

If you would like to attend and/or are interested in future participation, please contact us at whpc@purdue.edu. You may also subscribe to our WHPC List Serve.

Written by: Jonathan Poole, poole43@purdue.edu

Following good data practices makes scratch more performant and more useful for all users. By keeping master copies of important data in backed‑up storage, staging only the inputs and outputs you need for active jobs into scratch, and promptly moving valuable results back to a persistent location when your work completes, you help keep scratch fast, responsive, and available for high‑throughput workloads across the system.

To support this, we’ve updated our scratch documentation with clearer guidance and policies, including:

• An emphasis on scratch as temporary, high‑performance workspace, not long‑term storage.
• Clearer explanation of purge behavior and age limits (including that files may be removed at any time after they become eligible).
• Removal of legacy “purge warning email” language; users should no longer expect email notifications before files are purged.
• Explicit guidance that using tools like touch or other methods to defeat purge mechanisms is a violation of Acceptable Research Resource Use.
• Practical tips and examples for staging data into scratch, monitoring file ages with tools such as purgelist , and moving important results back to Data Depot or Fortress.

Scratch filesystems are engineered for capacity and performance and are not protected by backup technology; some types of failures can result in permanent data loss. If losing a file in scratch would significantly impact your research, that file should have a current copy in a more durable storage location such as Data Depot (for active data) or Fortress (for long‑term archival).

You can review the updated scratch storage guidance here: https://www.rcac.purdue.edu/policies/scratchpurge

If you have questions about how to adapt your workflows or where to store particular data, please contact us at rcac-help@purdue.edu and we’ll be happy to help.

Users may see failures connecting via Windows/Samba and jobs are affected because all clusters are currently paused while recovery work proceeds. Our engineers are working with the storage platform to restore service; we will share further updates as we learn more.

**Issue: **Some users may not see Microsoft Authenticator push prompts unless they manually open the app. This can cause sign-ins to time out.

Workaround: If you do not see a push notification, open the Microsoft Authenticator app and approve the pending request. Also, please verify that your default sign-in method is set to ‘App based Authentication – notification’ You can follow the steps in this Knowledge Base article to check and update your settings: https://service.purdue.edu/TDClient/32/Purdue/KB/ArticleDet?ID=1788

Still Need Help? - Contact us at rcac-help@purdue.edu

Beginning May 11, 2026, RCAC will move from Duo Mobile to Microsoft Authenticator for logging in to Purdue’s research computing and supercomputing systems.

To ensure uninterrupted access, please complete the steps below before May 11.

What you need to do:

Enroll in Microsoft Multi-Factor Authentication (MFA) using the Microsoft Authenticator app (if you have not already)

Set your default sign-in method in Microsoft MFA to your preferred method

After May 11, Duo Mobile will no longer work for accessing RCAC systems.

Please contact rcac-help@purdue.edu If you have any questions or need help.