Radon has served the Purdue research community for a great many years and has undergone radical changes during that time. However, during that time the landscape of research computing options at Purdue has also changed radically.

The Community Clusters now serve a variety of different research need profiles from traditional HPC to high-memory applications to GPU-enabled code, each of which are expanded or replaced every year. Scholar is available for instructional use of research tools within courses. Data Workbench, one of our newest offerings, serves those with smaller scale interactive needs. Radon's mission has been subsumed and is better served by these newer resources.

If you still have need for research computing resources, please contact us at rcac-help@purdue.edu to discuss your options, or you may explore them yourself on this website. Faculty and staff may directly purchase access to any of the currently available resources from the "Purchase" menu on our website as well, by simply selecting what they are looking for and providing a Purdue account number.

We look forward to continuing the development of Research Computing services to meet your evolving needs. Thank you for your support!

You should change your mapped network drives to connect to: \\scratch.my_cluster_name_here.rcac.purdue.edu\my_cluster_name_here

Or for home directory access: \\home.rcac.purdue.edu\home

The old server, samba.rcac.purdue.edu, will be retired at the end of this month, so please change your connections as soon as possible to avoid any disruption in service. This change will allow us to better maintain this type of service during cluster maintenance and upgrades in the future. More information on this is available in the cluster user guides:

• Brown
• Halstead
• HalsteadGPU
• Rice
• Snyder
• Conte
• Scholar
• Radon

Contact us if you have any questions. Thank you!

Many nodes in these clusters temporarily lost access to Depot. Jobs accessing files on Depot may have paused or terminated depending on the severity of the issue on each node and the nature of the job being run.

The issue is already being corrected and access restored, but some job failures or delays may be seen.

During this time, Research Computing services will continue to be available, but all staff will be on leave. Critical system outages will be dealt with should they occur during this time. Other support needs will be handled when the University re-opens on January 3rd. There will also be no coffee hour consultations during this break.

Scratch file purging (only on community clusters with scratch space) will continue as normal during the break, so be sure to archive your files in scratch storage or check for any upcoming purge warning email during the break. This does not apply to Data Depot or home directories--only scratch storage.

Have a wonderful break, everyone, and we look forward to great things in the new year!

During this time, some of the power feeding Hathi, Radon, and WSC Hadoop will be upgraded.

Any PBS jobs which request a walltime which would take them past Friday, November 3rd, 2017 at 5:00pm EDT will not start and will remain in the queue until after the maintenance is completed.

We will update this posting as we know more.

Effective August 28, 2017, all scratch storage systems will begin purging files which have not been accessed (for either reading or writing) in 60 days or more on a weekly basis.

This is a change from the previous policy of 90 days. This change is necessary in order to maintain a minimal level of free space on our scratch storage systems and ensure they remain effective for everyone.

Any files which are in active use by jobs on the clusters will not be purged, nor will you need to take any steps to ensure they are not. Please refrain from explicitly taking actions to exempt files from purging on a long-term basis, as this impedes our ability to offer the generous spaces we do for all users. It may also endanger your research, because there are no backups or recovery options for lost files in scratch storage spaces.

As soon as you are no longer actively computing against data, we strongly recommend copying that data to either a Data Depot space or the Fortress Archive, where your data will be safe for any future needs. There is information on how to access both of these systems on our website, and you may contact us at rcac-help@purdue.edu at any time with questions. If you need personal assistance, our staff also hold Coffee Break Consultation .

Thank you!

-- Previous message: After consulting with vendor support, we have determined that Radon has experienced a failure in its network hardware. Parts and and a vendor engineer will be on-site Tuesday, June 27 to repair. Radon will remain unavailable until that time.

-- Previous message:

The outage is being extended until 11:59am. There have been continued issues bringing Radon back into production due to an issue with its connectivity. At the moment, it cannot be said whether this is due to a networking problem or an issue within the Radon internal server configuration.

As previously stated, all other resources are back online and in production.

Update by 11:59a.

Thank you for your continued patience.

We are extending the outage by two hours due to issues encountered with bringing the Radon cluster back into production.

All other resources are back in online and serving at this time.

Will update by 2:00am with status.

ORIGINAL OUTAGE NOTICE:

The clusters, as well as some restricted access highly-specialized resources will be unavailable beginning at Friday, June 23rd, 2017 at 5:00pm EDT, for scheduled power maintenance. The clusters will return to full production by Tuesday, June 27th, 2017 at 2:00pm EDT.

During this time, power work will be performed in the Math data center.

Any PBS jobs which request a walltime on Radon which would take them past Friday, June 23rd, 2017 at 5:00pm EDT will not start and will remain in the queue until after the maintenance is completed. The other systems affected do not use schedulers which are directly aware of this upcoming maintenance.

These packages are being removed at the system-level because these types of libraries should be managed at the "module" level. These packages were inadvertently installed but have not been tuned and optimized for use on the clusters. Some software compilations may have accidentally picked up these packages, rather than the "module" package, or may have picked up some combination of system-level packages and module packages. Anyone with software linked against these libraries may have had issues with the software mixing versions.

Many scientific applications are very sensitive to the compilers used to build software and its libraries, so by removing these packages at the system-level, and by the "module" command's enforcement of only loading one of a package at the time, you can be sure your code is being compiled with exactly the same version of the library and not scrambled with other versions that were not asked for.

There were a couple of our own module installations that fell victim to this scrambling of libraries which we have fixed. If your software does not use netcdf or hdf5, or your software does and you are loading a netcdf or hdf5 module, you will not be impacted. If you see an error such as:

error while loading shared libraries: [libnetcdf.so](http://libnetcdf.so).6: cannot open shared object file: No such file or directory

You will need to ensure you are loading a netcdf or hdf5 module and you may need to recompile your code to ensure you are picking up the library version from the module.

Please let us know at rcac-help@purdue.edu if you have any questions.

Original Message

Some core servers which operate the Research Data Depot have suffered a failure. Engineers are working to correct this now. To prevent any new jobs from starting while Depot is not functioning normally, scheduling on all major clusters has been paused.

We will update this as we learn more about the nature of the issue.

Original Message

On May 9, 2017 - December 31, 2017 the clusters will complete the change to hierarchy modules. When the clusters switched to hierarchy modules on February 1st, the old module names remained supported with a warning message in place. On May 9, 2017 - December 31, 2017 , support for this translation of old module names will end.

You should check PBS job output for warning messages about old module style names. Any 'module load' commands showing this warning will fail after May 9, 2017 - December 31, 2017 . The warning message will tell you what you should change your 'module load' commands to.

You can switch over early with the command:

touch ~/.hierarchy

This will disable the automatic translation and will put you in and environment equivalent to that of after May 9, 2017 - December 31, 2017 .

For more information about hierarchy module see the original news article. If you have any questions about this please contact us at rcac-help@purdue.edu.

Potential impact on users:

• Currently running batch jobs will NOT be impacted.
• For each front-end server, users will be given a 48-hour notice to save their work and exit from any currently running interactive jobs. Interactive jobs still running at the time of reboot will be terminated.
• Users may experience slightly longer scheduling delays during initial hours of the updating process.

Old style module load commands should still continue to function until the end of the Spring semester, however you will be given a warning message and a suggestion on how to correct your scripts. Once you believe you have made all the necessary changes you may turn off the old-name translation by:

$ touch ~/.hierarchy

Then, by logging out and back in, the automatic translation will be disabled. This will allow you to be sure that your scripts are correctly updated before the translation is turned off system wide at the end of the semester. We will send further details and timing on turning off translations in the coming weeks.

Original Message

On January 25th, 2017 the Radon cluster's software stack will be converted to an hierarchical configuration. This new software has been in use on Halstead since it was brought online. This change will help prevent errors and configuration problems due to version mismatches or conflicting software, and will allow for ITaP to provide a more robust software stack.

This change may require you to make small modifications to your job scripts. The module command will attempt to automatically translate these for you to help ease into the transition but an automatic translation may not always be possible, and this functionality will only be offered for a limited time.

To further aid in this transition, a Radon node has been converted to the new configuration for you to log in and explore the new configuration. In addition, a small tool has been developed that will scan your shell startup scripts and specified job scripts to identify potential changes that may need to be made.

From a Radon front-end machine, type the command:

$ ssh radon-modules

Once on the test node, you can use "module spider" and "module avail" to explore the new hierarchical configuration. Once on this test node you can run the module checker script.

To check shell start up scripts:

$ modulecheck

To check a job script(s):

$ modulecheck myjobscript.sub

The tool will attempt to also scan any other scripts that the specified scripts source or include. If the tool is unable to find a sourced file, it will warn you that it was unable to check that file.

The output of this command will give you a summary of the files it found, whether it found any issues and will offer suggestions on how to update your scripts. Please do not try to run your jobs on this test node.

Additional information and explanations about the module hierarchy can be found in the user guide.

If you have any questions or concerns please contact us at rcac-help@purdue.edu. The rest of the ITaP Research Computing clusters will follow in conversion a week later on February 1st, 2017.

No impact to existing jobs is suspected, but new jobs would not have started during this time. If you see any issues, please let us know at rcac-help@purdue.edu

A complete fix of the vulnerability will require an upgrade of the Operating System on the cluster nodes. We are currently working on scheduling a downtime for the clusters to perform this upgrade.

Update: September 27, 2016 11:55pm

All systems other than Conte have been successfully returned to normal operations with the new home directory filesystem. Work continues at this point on Conte to ensure the Phi accelerators are properly reconfigured.

Carter has also been given a new scratch filesystem during this maintenance. This should alleviate some of the problems with the previous scratch filesystem on Carter. For more details, please see the Carter-specific announcement on this topic: New Carter Scratch Filesystem

Reminder:

This is a reminder of the Home Filesystem Maintenance taking place next week on Tuesday, September 27th.

Details below.

Original Message:

All of the research clusters (Carter, Coates, Hansen, Radon, and Rossmann) as well as some other minor systems will be unavailable beginning at Tuesday, September 27th, 2016 at 7:00am EDT, for scheduled maintenance. All clusters other than Conte will return to full production by 11:59pm.

Conte will return to partial capacity by that time, but will not return to full production until the following day. Many Conte nodes will remain offline and gradually be returned to service over the following 12-24 hours to allow for power reconfiguration in the data center. Please see the separate article on Conte: Conte Cluster Maintenance.

During the large all-systems maintenance Tuesday, the /home filesystem used by all Research Computing systems will be replaced by a new filesystem. The new filesystem will be based on DDN's GRIDScalar technology and running on new hardware dedicated exclusively to Research Computing home directories.

All files on the existing /home filesystem will be migrated to the new system during the maintenance window and prior to any of the clusters returning to service.

In the coming weeks, any jobs which request a walltime which would take them past Tuesday, September 27th, 2016 at 7:00am EDT will not start and will remain in the queue until after the maintenance is completed.

These upgrades will provide the best performance for the new and existing clusters and will provide a consistent Intel version stack across all of our clusters. The new software stack is currently available on the clusters for testing and upgrade. ITaP research computing staff recommends testing out the new compilers and upgrading prior to September 27th.

WHAT WILL BE THE IMPACT TO INTEL COMPILERS?

We will be upgrading the default Intel version from 13.1.1.163 to 16.0.1.150. The current default has been around for several years, and many researchers are already switching to the latest versions of Intel compilers. The 13.1.1.163 version will remain available on the current clusters for a period of time to give researchers time to finish up projects and upgrade to the latest. Any software dependent on the default version of Intel 13.1.1.163 will also have it's default upgraded.

WHAT WILL BE THE IMPACT TO MPI LIBRARIES?

We will be upgrading the default version of OpenMPI from 1.6.3 to 1.8.1. This new versions offers stability and performance enhancements and some new features. Version 1.8.1 has been available for some time and many researchers have already moved to 1.8.1.

We will be upgrading the default version of IMPI from 4.1.1.036 to 5.1.2.150. This new versions offers stability and performance enhancements and some new features. Version 5.1.2.150 has been available for some time and many researchers have already moved to 5.1.2.150.

Any software dependent on one of these default MPI versions will also have it's default upgraded appropriately.

It is recommended that you upgrade to these new libraries, however, if you need to continue using the old default versions you may do so by switching your "module load" to the specific version. The Intel 13 stack will remain available for those who require it. These new compilers offer bug fixes and enhanced performance and stability. Users are encouraged to send in any experiences with these new compilers to help us evaluate the direction of new compilers on RCAC systems.

WHAT OTHER SOFTWARE WILL BE IMPACTED?

There will be several changes to other miscellaneous software. Older versions of some software will be removed in favor of newer versions. Default versions of a few software will be updated to the latest version. In most cases, these older versions are being infrequently used so most should not be impacted by these changes.

If any software you are using will be impacted by these changes you will see a notice message being printed to your session or in your job output files when loading an affected module. This notice will provide recommendations on the latest version.

HOW DO I KNOW IF MY WORKFLOW WILL BE IMPACTED?

Whenever a module that will be impacted is loaded a notice is printed to your screen or job output log. Please take a look at your job output over the next couple of weeks and make note of any changes being advertised. You may continue using these modules as-is until September 27th to allow time to make any changes necessary. Users are encouraged to make any changes necessary beforehand to avoid disruption when changes are made.

WHAT IF AN IMPACTED MODULE IS REQUIRED BY MY RESEARCH?

We understand some users may not be able to change compilers or MPI libraries in the middle of a research project. Modules involved in a default version update will continue to be available, however, you will need to update your job scripts to request the specific version of the module. If you are already loading specific versions no changes are necessary.

If a version of software you depend on is being completely removed and you are unable to upgrade, please contact us at rcac-help@purdue.edu. We will help you transition to a newer version if possible, or provide you with a copy of the old software version.

WHY ARE YOU CHANGING THE SOFTWARE STACK?

ITaP aims to provide a software stack that allows for optimal use (performance and stability) of the clusters. This necessitates periodic updates to the stack as compilers, libraries, and software are improved over time. By removing older modules from the main stack we help ensure the selection is simple and easy for users to find the best compilers and libraries to use. If no modules were removed the selection would become difficult to navigate as well become difficult for ITaP staff to manage. Any major changes will be coordinated with scheduled maintenance periods to minimize impact.

If you have any questions or concerns with the upcoming changes please contact rcac-help@purdue.edu

Original Message:

System, Network, Storage, and Support staff are working to diagnose and correct issues that have been seen recently within ITaP's Research Computing systems.

Symptoms being reported involve an apparent complete freeze of open sessions, the inability to open new login sessions, difficulties using text editors, and disruptions in file access. In cases we have seen, these events seem to last for about 3-5 minutes, then clear up. However, there may be ongoing effects on jobs running on the Research Clusters, including job failure due to the storage access disruption.

We are examining log files and monitoring processes actively, and are working to correlate the timing of these events across our systems, and expect to identify a fundamental cause that we can then correct. At this time, however, we do not have an estimated time for a fix.

Please follow this news item for further information.

Thank you for your patience.

Update

Due to unanticipated conflicts between the upgraded scheduler and our network configuration, the outage for Radon has been extended, while we work to eliminate the conflict.

We will have a further update no later than 7:00 pm.

Original Message

The cluster will be unavailable beginning at Tuesday, August 16th, 2016 at 8:00am EDT, for scheduled maintenance. The cluster will return to full production by Tuesday, August 16th, 2016 at 6:00pm EDT.

During this time, will have the operating system patched, the PBS resource management system upgraded, and maintenance performed on the Lustre scratch filesystem.

Any PBS jobs which request a walltime which would take them past Tuesday, August 16th, 2016 at 8:00am EDT will not start and will remain in the queue until after the maintenance is completed.