Update: September 27, 2016 11:55pm

All systems other than Conte have been successfully returned to normal operations with the new home directory filesystem. Work continues at this point on Conte to ensure the Phi accelerators are properly reconfigured.

Carter has also been given a new scratch filesystem during this maintenance. This should alleviate some of the problems with the previous scratch filesystem on Carter. For more details, please see the Carter-specific announcement on this topic: New Carter Scratch Filesystem

Reminder:

This is a reminder of the Home Filesystem Maintenance taking place next week on Tuesday, September 27th.

Details below.

Original Message:

All of the research clusters () as well as some other minor systems will be unavailable beginning at Tuesday, September 27th, 2016 at 7:00am EDT, for scheduled maintenance. All clusters other than Conte will return to full production by 11:59pm.

Conte will return to partial capacity by that time, but will not return to full production until the following day. Many Conte nodes will remain offline and gradually be returned to service over the following 12-24 hours to allow for power reconfiguration in the data center. Please see the separate article on Conte: Conte Cluster Maintenance.

During the large all-systems maintenance Tuesday, the /home filesystem used by all Research Computing systems will be replaced by a new filesystem. The new filesystem will be based on DDN's GRIDScalar technology and running on new hardware dedicated exclusively to Research Computing home directories.

All files on the existing /home filesystem will be migrated to the new system during the maintenance window and prior to any of the clusters returning to service.

In the coming weeks, any jobs which request a walltime which would take them past Tuesday, September 27th, 2016 at 7:00am EDT will not start and will remain in the queue until after the maintenance is completed.

These upgrades will provide the best performance for the new and existing clusters and will provide a consistent Intel version stack across all of our clusters. The new software stack is currently available on the clusters for testing and upgrade. ITaP research computing staff recommends testing out the new compilers and upgrading prior to September 27th.

WHAT WILL BE THE IMPACT TO INTEL COMPILERS?

We will be upgrading the default Intel version from 13.1.1.163 to 16.0.1.150. The current default has been around for several years, and many researchers are already switching to the latest versions of Intel compilers. The 13.1.1.163 version will remain available on the current clusters for a period of time to give researchers time to finish up projects and upgrade to the latest. Any software dependent on the default version of Intel 13.1.1.163 will also have it's default upgraded.

WHAT WILL BE THE IMPACT TO MPI LIBRARIES?

We will be upgrading the default version of OpenMPI from 1.6.3 to 1.8.1. This new versions offers stability and performance enhancements and some new features. Version 1.8.1 has been available for some time and many researchers have already moved to 1.8.1.

We will be upgrading the default version of IMPI from 4.1.1.036 to 5.1.2.150. This new versions offers stability and performance enhancements and some new features. Version 5.1.2.150 has been available for some time and many researchers have already moved to 5.1.2.150.

Any software dependent on one of these default MPI versions will also have it's default upgraded appropriately.

It is recommended that you upgrade to these new libraries, however, if you need to continue using the old default versions you may do so by switching your "module load" to the specific version. The Intel 13 stack will remain available for those who require it. These new compilers offer bug fixes and enhanced performance and stability. Users are encouraged to send in any experiences with these new compilers to help us evaluate the direction of new compilers on RCAC systems.

WHAT OTHER SOFTWARE WILL BE IMPACTED?

There will be several changes to other miscellaneous software. Older versions of some software will be removed in favor of newer versions. Default versions of a few software will be updated to the latest version. In most cases, these older versions are being infrequently used so most should not be impacted by these changes.

If any software you are using will be impacted by these changes you will see a notice message being printed to your session or in your job output files when loading an affected module. This notice will provide recommendations on the latest version.

HOW DO I KNOW IF MY WORKFLOW WILL BE IMPACTED?

Whenever a module that will be impacted is loaded a notice is printed to your screen or job output log. Please take a look at your job output over the next couple of weeks and make note of any changes being advertised. You may continue using these modules as-is until September 27th to allow time to make any changes necessary. Users are encouraged to make any changes necessary beforehand to avoid disruption when changes are made.

WHAT IF AN IMPACTED MODULE IS REQUIRED BY MY RESEARCH?

We understand some users may not be able to change compilers or MPI libraries in the middle of a research project. Modules involved in a default version update will continue to be available, however, you will need to update your job scripts to request the specific version of the module. If you are already loading specific versions no changes are necessary.

If a version of software you depend on is being completely removed and you are unable to upgrade, please contact us at rcac-help@purdue.edu. We will help you transition to a newer version if possible, or provide you with a copy of the old software version.

WHY ARE YOU CHANGING THE SOFTWARE STACK?

ITaP aims to provide a software stack that allows for optimal use (performance and stability) of the clusters. This necessitates periodic updates to the stack as compilers, libraries, and software are improved over time. By removing older modules from the main stack we help ensure the selection is simple and easy for users to find the best compilers and libraries to use. If no modules were removed the selection would become difficult to navigate as well become difficult for ITaP staff to manage. Any major changes will be coordinated with scheduled maintenance periods to minimize impact.

If you have any questions or concerns with the upcoming changes please contact rcac-help@purdue.edu

Original Message:

System, Network, Storage, and Support staff are working to diagnose and correct issues that have been seen recently within ITaP's Research Computing systems.

Symptoms being reported involve an apparent complete freeze of open sessions, the inability to open new login sessions, difficulties using text editors, and disruptions in file access. In cases we have seen, these events seem to last for about 3-5 minutes, then clear up. However, there may be ongoing effects on jobs running on the Research Clusters, including job failure due to the storage access disruption.

We are examining log files and monitoring processes actively, and are working to correlate the timing of these events across our systems, and expect to identify a fundamental cause that we can then correct. At this time, however, we do not have an estimated time for a fix.

Please follow this news item for further information.

Carter and Scholar are still being worked on. We will issue another update by 6:00am if not already in service.

The Rice, Hammer, and Peregrine1 clusters have been returned to normal operations as of 1:40am. Work continues on Carter and Scholar, and we will issue an update on those systems by 3:00am if not already in service.

The Snyder cluster has been returned to normal operations as of 12:00am. Work continues on the the other clusters listed here.

The work continues on these clusters, although progress was substantially delayed by the concurrent storage systems failure (Unscheduled Storage Outage). We will post an update by 2:00am or sooner as clusters return to service.

The clusters will be unavailable beginning at Tuesday, June 7th, 2016 at 5:30am EDT, for scheduled maintenance. The clusters will return to full production by Tuesday, June 7th, 2016 at 10:00pm.

During this time, maintenance will be performed on the cooling systems used by these clusters. This maintenance period will also allow critical high-availability fixes to be made to the Research Data Depot while client clusters are offline.

Any PBS jobs which request a walltime which would take them past Tuesday, June 7th, 2016 at 5:30am EDT will not start and will remain in the queue until after the maintenance is completed.

If you see any problems with your jobs from this event, please let us know, although it is unlikely there is any way to recover any work from jobs that suffered a failure.

As of Tuesday, May 17th, 2016 at 5:30pm EDT, are unavailable due to a loss of Isilon home directory storage. Most processes on these systems will block until this storage is restored.

Our storage engineers have been notified, though there is not yet an estimate for return to service.

Alerts for new users of Data Depot are being set up automatically as of May 6, 2016 - December 31, 2016 , and alerts for existing Data Depot users will be phased in over the coming weeks. These alerts can be modified or disabled at any time if so desired.

Alerts can be configured to notify you when your quota reaches a user-defined level (either an absolute value or a percentage). You can also create usage reports that will email you a usage report on a user-defined schedule.

The new tool is accessible from the My Quota page. If you have any questions, feedback, or encounter any problems with this new tool please contact us at rcac-help@purdue.edu.

Original Message:

The Peregrine-1 cluster is currently offline due to problems with the cluster nodes' operating system software. This failure occurred gradually as nodes completed jobs, so there was no loss of jobs due to the outage, although no new jobs are able to run at the moment.

Engineers are investigating the issue and hope to return the nodes to normal function. However, there is currently no estimate for return to service.

UPDATE: Peregrine1's network switch has been replaced, and the cluster is now functional.

A significant number of ECN services will be affected, including several software license servers for ITaP Research Computing systems that are hosted by ECN. License servers are expected to be rebooted around 6:30am EST on Tuesday, March 1st, 2016. Exact duration of the outage is unknown, but it is not expected to be long.

ITaP Research Computing cluster job scheduling is not affected by the outage, but licenses for software like Matlab, Ansys/Fluent, CFD++, Sentaurus, Comsol, Abaqus, PowerFlow, and PowerAcoustics will be unavailable during the outage period, which may lead to license-controlled software refusing to work and jobs exiting with error conditions.

Users of Matlab are encouraged to always submit jobs that explicitly request license tokens available from the job scheduler. These are specified using the gres attribute in your job submission command. For example, to request a single Matlab license:

$ qsub -l nodes=1:ppn=1,walltime=01:00:00,gres=MATLAB+1 myjob.sub

This way the job is guaranteed to only start execution when the necessary license is available. More examples for various Matlab toolboxes are available in the user guides.

Any other jobs using ECN licensed software that start during this downtime will not be able to check out a license and may result in jobs exiting with errors. As well, any software that requires a constant connection to the ECN licensing servers will stop during this time.

Further information regarding affected ECN services may be found in the ECN announcement.

If you are unsure if your software will be affected or have any other concerns please contact us at rcac-help@purdue.edu.

Original:

On January 6, 2016 - December 31, 2016 the environment module system on will be upgraded to Lmod. This new system has been in use on the Rice and Snyder clusters since they were put into production. It has also been available as an opt-in service on the other clusters for several months.

What is Lmod?

Lmod is a Lua based environment modules system developed by TACC (Texas Advanced Computing Center). This system is used on TACC clusters such as Stampede and Lonestar. Lmod is also used at several other universities and centers. Lmod offers all the same functionality as the current environment modules system and uses the same commands and syntax. However, Lmod offers new and unique functionality that can be leveraged to offer an easier to use and understand software environment on ITaP clusters.

Will I have to change anything?

No. Lmod offers exactly the same functionality as the current module system. No changes to your job submission scripts will be necessary - all the old commands will still work as they have before. It does, however, offer several new features and new functionality.

What are the new features of Lmod?

At the moment Lmod is mostly a one-for-one replacement of the current environment modules system. In the future, ITaP will expand on the functionality of Lmod to improve usability of our software stack.

One of the most exciting features of Lmod is the spider command. This command allows you to discover software available on the clusters that you might not otherwise find.

For example, finding the bioinformatics software bowtie2 is not easy. This software is not managed by ITaP and is not in the normal software availability. Nonetheless, it is available to the general cluster public if you happen to know how to find it. Lmod makes finding such software much easier:

$ module spider bowtie2

----------------------------------------------------------------------------
  bowtie2:
----------------------------------------------------------------------------
     Versions:
....
        bowtie2/2.2.3

----------------------------------------------------------------------------
  To find detailed information about bowtie2 please enter the full name.
  For example:

     $ module spider bowtie2/2.2.3
----------------------------------------------------------------------------

$ module spider bowtie2/2.2.3

----------------------------------------------------------------------------
  bowtie2: bowtie2/2.2.3
----------------------------------------------------------------------------

    This module can only be loaded through the following modules:

      bioinfo

With this command you can discover that this software is available by first loading the bioinfo module.

One of the other features lmod offers is the ability to prevent software conflicts. Loading multiple MPI libraries or compilers, for example, may result in conflicting commands and libraries being used. Lmod will not allow the accidental loading of multiple, conflicting software simultaneously. For example:

$ module load intel
$ module load pgi

Lmod has detected the following error: You can only have one compiler module loaded at a time.
You already have intel loaded.
To correct the situation, please enter the following command:

  module swap intel pgi/11.8-0

Please submit a consulting ticket if you require additional assistance.

As of Wednesday, December 2nd, 2015 at 12:00pm EST, are all unavailable due to a failure of the home directory filesystem served by Isilon.

Engineers are working on the issue now, but there is currently no estimate for return to service. We will update this post as we learn more.

The work affecting scratch filesystems has been completed and the clusters are back in full production.

Original

The cluster will be unavailable beginning at Tuesday, September 22, 2015 from 8:00am - 10:00am EDT, for scheduled maintenance. The cluster will return to full production by Tuesday, September 22nd, 2015 at 10:00am EDT.

During this time, Carter's scratch will be de-coupled from Hansen/Peregrine1. To ensure are not impacted by the separation, scheduling will be paused and access to scratch files will be unavailable.

The scratch filesystem will be unavailable for 1-2 hours. Any running job using scratch will block on I/O and appear to hang. Jobs will resume normally once access is restored.

Any PBS jobs which request a walltime which would take them past Tuesday, September 22, 2015 from 8:00am - 10:00am EDT will not start and will remain in the queue until after the maintenance is completed.

During this time, will be relocated from Calumet to the West Lafayette campus.

Any PBS jobs which request a walltime which would take them past August 17, 2015 8:00am - August 19, 2015 6:00pm EDT will not start and will remain in the queue until after the maintenance is completed. Any job that has not started will remain in queue and resume normally after the move.

For Research Computing users this means that software packages licensed through ECN servers will not be able to check out licenses. Affected software packages include:

Abaqus        Agilent       Altair        Altera        AMPL
Ansoft        Ansys         Autonomie     Cadence       Cadiq
COMSOL        Coventor      EDEM          Foundry       GridGen
GTSuite       Houdini       ImagineLab    LSDyna        Maple
Matlab        MSC           OriginPro     Pixar         PowerFlow
Simic         Synopsys      Tecplot       UGS

We apologize for the disruption.

As of 4:45 pm Tuesday, May 19, all the work noted below has been completed and both Hansen and Peregrine-1 have been returned to full service.

Thanks for your patience.

=-=-=

The Hansen and Peregrine-1 clusters will be unavailable beginning at 8:00 am on Tuesday, May 19, 2015, for scheduled maintenance. The clusters will return to full production by 6:00 pm the same day.

During this time, Peregrine1 and Hansen will have the operating system patched, and the PBS resource management system upgraded.

Any PBS jobs which request a walltime which would take them past 8:00 am on May 19 will not start and will remain in the queue until after the maintenance is completed.

Original message:

The storage servers powering the Research Data Depot will undergo maintenance on Thursday, February 26, 2015 from 10:00am - 4:00pm EST to install important security patches.

Research Data Depot servers are highly redundant, and this operation should have no visible impact to users of the filesystem.

Additionally, during this time scratch storage servers will be rebooted to apply the same update. Access to Lustre scratch may pause briefly while backend servers are restarting.

ITaP Research Computing cluster job scheduling is not affected by the outage, but licenses for software like Matlab, Ansys/Fluent, Sentaurus, Comsol, Abaqus, PowerFlow, and PowerAcoustics will be unavailable during the outage period, which may lead to license-controlled software refusing to work and jobs exiting with error conditions.

Users of Matlab are encouraged to submit jobs requesting license tokens available from the job scheduler. These are specified using the gres attribute in your job submission command. For example, to request a single Matlab license:

$ qsub -l nodes=1:ppn=1,walltime=01:00:00,gres=MATLAB+1 myjob.sub

This way the job is guaranteed to only start execution when the necessary license is available. More examples for various Matlab toolboxes are available in the user guides.

Any other jobs using ECN licensed software that start during this downtime will not be able to check out a license and may result in jobs exiting with errors. As well, any software that requires a constant connection to the ECN licensing servers will stop during this time.

Further information regarding affected ECN services may be found on the ECN website.

If you are unsure if your software will be affected or have any other concerns please contact us at rcac-help@purdue.edu.

New Intel 15 Compiler Suite

The largest of the software stack changes will be the addition of the latest Intel 15 compiler suite, version 15.0.1.133, and its associated common libraries.

No changes will be made to the default versions of common libraries or compilers, nor will there be any changes to any of the currently supported common libraries and compilers. As a reminder, a table of these supported compilers and libraries is included towards the end of this article.

Evaluation of new environment modules system

A new environment modules system, called Lmod, is available for evaluation on ITaP Research Computing clusters. ITaP Researching Computing staff are evaluating Lmod as a replacement for the current environment modules system and are looking for feedback from cluster users. Lmod is used in the same way as the current system but offers much more on top of existing functionality. Lmod is available as an opt-in to all current cluster users.

Anaconda, a new Python distribution

The default Python distribution will be replaced by Anaconda, a distribution of Python that includes over 200 popular Python packages used throughout scientific computing. Some of these packages include h5py, mpi4py, netcdf, numpy, scikit, and scipy.

New Intel 15 Compiler Suite

The largest of the software stack changes will be the addition of the latest Intel 15 compiler suite, version 15.0.1.133, and its associated common libraries. This additional will make the following modules available for use:

• intel/15.0.1.133
• boost/1.56.0_intel-15.0.1.133
• fftw/3.3.4_intel-15.0.1.133_openmpi-1.8.1
• hdf5/1.8.7_intel-15.0.1.133
• hdf5/1.8.13_intel-15.0.1.133
• mpich2/1.4.1p1_intel-15.0.1.133 (Radon, Rossmann, Hansen)
• mvapich2/1.9_intel-15.0.1.133
• netcdf/3.6.3_intel-15.0.1.133
• netcdf/4.1.1_intel-15.0.1.133
• openmpi/1.8.1_intel-15.0.1.133
• python/2.7.8_intel-15.0.1.133

No changes will be made to the default versions of common libraries or compilers, nor will there be any changes to any of the currently supported common libraries and compilers. As a reminder, a table of these supported compilers and libraries is included towards the end of this article.

Miscellaneous updates

Several miscellaneous software packages will also be replaced with the latest version. If you are using one of the affected module a notice message will be printed when loading it. Be sure to check your job output for any such messages and update your scripts as necessary.

Common Libraries and Long Term Support

We have chosen an installation of Intel and GCC to offer long-term support on. We expect to keep these versions of compilers and their associated common libraries for a few years. This software stack has proven to be stable and reliable. If you are looking for modules to use and not have to change them over the course of several years use these modules. Other versions, including newer ones, may come and go as we aim to offer a stable long-term stack and offer an up to date stack that undergoes more frequent updates.

These compilers are Intel 13.1.1.163 and GCC 4.7.2. These are combined with the following common libraries:

• fftw/3.3.4
• hdf5/1.8.7
• netcdf/3.6.3
• netcdf/4.1.1
• openmpi/1.8.1

In addition to the Long Term Support compilers, there are several other compilers the above libraries are combined with. The full list of supported compilers include:

• gcc/4.7.2
• intel/13.1.1.163
• intel/14.0.2.144
• intel/15.0.1.133

There are a few other compilers available on the clusters however these are not combined with the common libraries.

The following common software are built with the latest Intel compiler (at the time of installation):

• boost
• python