Purdue University

Anvil, Purdue’s most powerful supercomputer, is hammering out progress as it forges ahead into the second year of operations. Funded by a $10 million grant from the National Science Foundation (NSF), Anvil began early user operations in November 2021 and entered production operations in February 2022. Within its first year, Anvil amassed an impressive resume, having helped more than 2000 researchers push the boundaries of scientific exploration in a variety of fields, including artificial intelligence, biomedicine, climatology, and Covid-19 epidemiology.

Anvil at a Glance—Year 1 of Operations

Anvil has had a significant impact on scientific research and student development thus far in its operations. With nearly 2,800 total users in year one, of which more than 1,300 were students, Anvil is not only helping meet the growing need for high-performance computing (HPC) within the realms of research, but also actively assisting with the development of cyberinfrastructure professionals of tomorrow. Anvil allowed users access to 390 million CPU hours and 607 thousand GPU hours in its first year alone, supporting research across 42 diverse scientific domains. Nine journal publications were produced thanks to Anvil usage in 2022, with seven more currently in production. Aside from the supercomputer itself, the Anvil team has been hard at work promoting the benefits of HPC and ensuring the nation has a workforce trained in the use, operation, and support of advanced cyberinfrastructure. In its first year, the Anvil team participated in 56 outreach events and conducted 7 training sessions. These training sessions were designed to deliver working knowledge of HPC systems and teach users how to get the most out of their research time on Anvil. The team also provided hands-on training to students through initiatives such as the Anvil Summer REU program, which allowed the students to gain much-needed knowledge and experience in the field of HPC.

So what exactly is Anvil?

Anvil is a supercomputer deployed by Purdue’s Rosen Center for Advanced Computing (RCAC) in partnership with Dell and AMD. The system was created to significantly increase the computing capacity available to users of the NSF’s Advanced Cyberinfrastructure Coordination Ecosystem: Services and Support (ACCESS), a program that serves tens of thousands of researchers across the United States. The system consists of 1,000 Dell compute nodes, each with two 64-core third-generation AMD EPYC processors, and will make available over 1 billion CPU core hours to ACCESS every year. Anvil’s nodes are interconnected with 100 Gbps Nvidia Quantum HDR InfiniBand. The supercomputer ecosystem also includes 32 large memory nodes, with 1 TB of RAM per node, and 16 GPU nodes, each with four NVIDIA A100 Tensor Core GPUs. These GPU nodes provide an additional 1.5 PF of single-precision performance to support machine learning and artificial intelligence applications.

Powerful Performance Unleashed

Because of its configuration, Anvil is able to reach a peak processing speed of 5.3 petaFLOPS, making it one of the most powerful academic supercomputers in the US. When it debuted, the Anvil supercomputer was listed as number 143 on the Top500 list of the world’s most powerful supercomputers. This advanced processing speed and power has allowed researchers to save hours of time on computations and simulations.

“Because of Anvil’s state-of-the-art GPUs, we’ve gained four times better performance compared to our previous benchmarks,” says Daipayan Sarkar, a postdoc working with Josh Vermaas, an assistant professor at Michigan State University’s DOE Plant Research Laboratory. Vermaas’s team utilized Anvil to perform molecular dynamics simulations. These simulations helped the team study molecular movement, such as how carbon dioxide passes through the plant plasma membrane during photosynthesis.

Yinglong Miao, an assistant professor at the Center for Computational Biology and Department of Molecular Biosciences at the University of Kansas, is another researcher who used Anvil to perform molecular dynamics simulations much faster than ever before. “Using CPUs, this work could take months, if not years,” says Miao. “With the GPUs on Anvil, we can run these simulations much faster. Instead of needing months, we just need a couple of weeks.”

Anvil’s processing power and configuration also allow researchers to consolidate their workload to fewer nodes, providing improved workflow speed and reliability.. Aakash Kumar, a Postdoctoral Associate at the Energy Sciences Institute at Yale University, found that on Anvil, he could complete his medium-sized calculations using one or two nodes, whereas on HPC systems he has used in the past, he needed to use many more. “There [on other HPC systems], in my experience, you tend to use more nodes, which is fine, but what happens is, when you use more nodes, there is a communication issue between the machines.” These communications issues led to what Kumar described as a bottleneck, slowing down the calculations and the research. But because Anvil has such a large number of powerful cores per node, he could avoid this communication bottleneck and complete all of his calculations using a smaller number of nodes with a significant speedup.

Anvil also showcased its effectiveness in the world of artificial intelligence and machine learning. According to one user, Anvil helped “achieve the fastest training of agents I have seen, cutting months of training time on conventional HPC resources to hours on Anvil’s GPU nodes.” Another researcher noted that “Anvil GPUs allowed us to train our neural network model for RNA structure prediction with nearly 100M parameters using PyTorch,” while a third said that “it was the first time I was able to run deep learning jobs with 40GB memory GPUs and multiple ones.” The Anvil team was excited to receive such good feedback. As the world’s focus shifts more and more towards AI capabilities, the team feels confident that the Anvil system is well-equipped to handle the vast amount of data and computations that come with such research.

Training and Education Impact

Aside from enabling groundbreaking research across multiple fields of science, Anvil is being used as a tool to develop the future workforce in computing. From professional training and workshops to hands-on learning experiences for students, Anvil is helping to forge the next generation of researchers and cyberinfrastructure professionals.

One major training and educational impact made by Anvil involved supporting the 2022 NIH BigCare Workshop. The BigCare Workshop was a National Cancer Institute-funded biomedical data analysis workshop designed to train cancer researchers on how to visualize, analyze, manage, and integrate large amounts of data in cancer studies. A mix of lectures and hands-on activities were provided to 50 participants for two weeks using Anvil’s interactive computing and Biocontainers. Thanks to Anvil, significantly more HPC resources were available compared to prior years, allowing the organizers to host more participants than initially planned. Overall, the 2022 BigCare workshop was a huge success, and Anvil has been chosen to support it again in 2023.

Anvil also supported roughly 1,300 students in a national data science experiential learning and research program known as The Data Mine. The goal of The Data Mine is to foster faculty-industry partnerships and enable the adoption of cutting-edge technologies. The course introduces students of all levels and majors to concepts of data science and coding skills for research. The students then partner with outside companies for a year to work on real-world analytic problems. Anvil provided 1 million CPU hours for the program and allowed the students to manage extensive research datasets, thanks to the supercomputer’s large capacity.

Anvil and its support team also trained students on the skills necessary to enter the research computing and data workforce itself. Through the Anvil Summer Research Experience for Undergraduates (REU) program, student interns were provided 1-on-1 mentorship designed to teach them the knowledge and skills necessary to build and support advanced research computing systems and scientific applications. And the results of the program have been outstanding thus far.

Nrushad Joshi, an undergraduate student at Indiana University, recently accepted an offer to intern at Oak Ridge National Laboratory as part of the Department of Energy’s Science Undergraduate Laboratory Internships (SULI) program. According to Joshi, he “received this offer thanks to the skills I gained at the Anvil summer REU internship and student cluster competition.”

Joshi was a novice HPC user when entering the Anvil REU program. After spending 12 weeks working on Anvil under the mentorship and guidance of the Anvil team, he gained enough knowledge to secure a spot in the highly competitive SULI program. Not only that, but working on Anvil may have changed Joshi’s entire career path: “After the Anvil summer internship, my career plans changed. I wanted to go into robotics before, and now I want to work in high-performance computing.”

Lucas Snyder is another undergraduate student at Indiana University who found the Anvil REU program to be invaluable. Snyder was offered a position as an HPC intern at the NASA Center for Climate Simulation (NCCS), a department that focuses on weather and climate prediction to enhance NASA’s capabilities in Earth Science. After accepting the position, Snyder emailed his Anvil REU mentor, Erik Gough, a Lead Computational Scientist at the Rosen Center for Advanced Computing, to inform him of the good news:

“Working at NASA has been a dream of mine for years, and I wanted to thank you for the huge role you played in making that a reality for me. Everything I learned on the job at RCAC and during my time with the SCC team has meant so much to me! I can also tell you with absolute certainty that I would not have this opportunity if you’d not taken a chance on me last summer and hadn’t been such an awesome mentor.”

Looking to the Future

Anvil has already proven to be a huge success, providing benefits to students and researchers alike, and the Anvil team is showing no signs of slowing down. Big goals have been set for the upcoming year, all of which will lead Anvil to better performance and easier accessibility for users of any background.

One notable goal of the team is to broaden Anvil adoption nationwide, focusing on identifying and engaging with underrepresented communities, non-traditional HPC users, and industry partners. Making the barrier to entry for HPC as low as possible is a major aspiration of the project.

“Anvil not only provides users with the performance they need, as evidenced by its place among the world’s most powerful supercomputers, it was also built with a focus on usability and easy access for those in traditionally under-served domains,” says Carol Song, chief scientist for the Rosen Center for Advanced Computing and project director for Anvil.

“Providing low-barrier access to HPC for underrepresented or non-traditional user groups iscrucial to enabling widespread adoption of HPC resources throughout the research community,” says Song. “But that’s only half of the process. We have to reach out broadly to these groups—many of them will not come to us. If they don’t know what HPC is, or that it can be used in their field, why should we expect them to want to use Anvil? Or any supercomputer, for that matter? We must go to them, and we must show them how beneficial Anvil can be.”

With this in mind, the Anvil team has created a stout outreach and communications plan. From K-12 educational opportunities to sector- and field-specific marketing materials, the goal is to get the word out—HPC is for everybody.

Aside from the outreach initiatives, another goal for 2023 is to continue providing quality services and support to Anvil users through application improvements, expanded user training, and cloud and composable innovations. With things such as a larger software catalog and asynchronous training modules, the Anvil team will provide everything a user needs to easily start working on the supercomputer.

Having such a substantial list of goals for the upcoming year is certainly daunting, but according to Preston Smith, Executive Director for the Rosen Center for Advanced Computing and co-PI on the Anvil project, the Anvil team is excited to rise to the challenge.

“We have some pretty big aspirations for 2023,” says Smith, “adding new innovations to Anvil’s ecosystem, and finding new ways for them to impact the nation’s science and engineering research.”

And these big aspirations will require hard work and dedication on every level if the team wants to achieve all it has set out to do. But Smith knows this and isn’t shying away from it:

“You can’t forge the future of computing without solving hard problems. And that’s exactly what we intend to do.”

Anvil is funded under NSF award number 2005632. The Anvil executive team includes Carol Song (PI), Preston Smith (Co-PI), Rajesh Kalyanam (Co-PI), and Arman Pazouki (Director of Scientific Applications). Researchers may request access to Anvil via the ACCESS allocations process.