RCAC: News and Information

LustreC filesystem unavailable May 12 - May 13, 2013 - Resolved

rcac@purdue.edu — Mon, 13 May 2013 23:20:00 PST

Update: May 13, 2013 11:00pm: LustreC has been returned to service. Carter, Hansen, and Peregrine1 are back in production with queues enabled.
Update: May 13, 2013 3:00pm: storage engineers are continuing to work with vendor support to return LustreC to production.
The current estimate for LustreC's return to production is 8:00pm EDT (May 13).
Update: storage engineers have received a potential solution from vendor support, and are working on implementation.
The current estimate for LustreC's return to production is 9:00pm EDT.
Additionally, all jobs currently running in the queue will receive a 24-hour walltime extension.

Original Message:
As of the morning of 5/12/2013, the LustreC filesystem serving the Carter, Hansen, and Peregrine1 cluster is unavailable. ITaP engineers are working with vendor support to return the filesystem to production.
In the meantime, all queues on Carter, Hansen, and Peregrine1 are disabled.
Running jobs will block waiting on I/O until the filesystem returns to production, and interactive access to LustreC is currently not available.
The outage is currently scheduled until 4pm.

Unscheduled Fortress Outage - April 29, 2013

rcac@purdue.edu — Mon, 29 Apr 2013 13:11:00 PST

Resolved:
As of about 4:45pm ET, the connectivity issue affecting the Fortress archive has been resolved. The HPSS archive is back in full production. If you encounter any issues, please contact us at rcac-help@purdue.edu
Update:
ITaP Storage Engineers continue working with the vendor to resolve the connectivity issues with the HPSS archive. The next update will be sent out by 6pm this evening.
Original notice:
The Fortress archive is currently experiencing connectivity issues. ITaP Storage Engineers are investigating the problem and are working to bring the HPSS archive back in full production.
Our apologies for the inconvenience. We will provide an update by 3pm this afternoon. If you have any questions please contact us at rcac-help@purdue.edu

Network outage affecting Peregrine1 cluster - 4/24/2013

rcac@purdue.edu — Tue, 16 Apr 2013 21:01:00 PST

On April 24, 2013, network engineers will be relocating fiber optics that connnect the Peregrine1 cluster to infrastructure in West Lafayette. This outage is scheduled for 12:00am throuh 5:00am.
This will leave Peregrine1 unable to run jobs
Any PBS jobs which request a walltime which would take them past the 12:00am on 4/24/2013 will not start and will remain in the queue until after the maintenance is completed.

Scheduling paused on Carter cluster - 4/16/13

rcac@purdue.edu — Tue, 16 Apr 2013 20:14:00 PST

Update: 8:12pm
Scheduling on Carter has been resumed, and Carter is back in full production.
Original Message:
Beginning the morning of April 16, a number of compute nodes on the Carter cluster are experiencing a connectivity issue.
While ITaP engineers are troubleshooting the issue, the scheduling of new jobs on Carter has been paused.
Further updates on this issue will be posted by 4:00pm.

Unscheduled Outage to LustreC - Resolved

rcac@purdue.edu — Thu, 14 Mar 2013 11:42:00 PST

Update:
ITaP engineers have corrected the issue affecting the LustreC filesystem. The system is back in production. Job scheduling on Carter, Hansen and Peregrine-1 has been restarted. As always, thank you for your patience. If you encounter any issues, please contact us at rcac-help@purdue.edu
Original:
As of about 10:45am an issue was found on the LustreC filesystem that is making it unavailable for use. ITaP engineers and the vendor are on site working on restoring the system to production. While the filesystem is unavailable, job scheduling has been stopped on Carter, Hansen and Peregrine 1. We apologize for the inconvenience. An update will be sent around noon today.

Fortress tape library upgrade, March 28-29, 2013 (Updated 4:25pm 3/29)

rcac@purdue.edu — Thu, 28 Mar 2013 08:40:00 PST

Updated, 3/29/2013

During the datacenter migration the core server for the Fortress HPSS system experienced a failure with its boot device. Vendor engineers are working on the issue, and as a result, the estimate to Fortress's return to production has been revised to Saturday, March 30 at 12:00 noon.

Please contact us at rcac-help@purdue.edu if you have questions or concerns.

Original message:

Each year, ITaP partners with Purdue faculty to construct a new community cluster system.

Planning is already underway for the next system, and to make the data center ready for the next generation of HPC hardware at Purdue, some equipment must be physically moved to prepare for this project.

As part of this work, the ADIC Scalar 10K tape library that serves the Fortress HPSS archive system is going to be upgraded, and the server hardware driving Fortress will be relocated to a different data center facility.

This physical transition is scheduled to for 8am Thursday though 5pm Friday, March 28-29 2013.
During this time, the Fortress archive will be completely unavailable.
We suggest copying your data to another location if you will need it during the planned downtime.
- If you're unsure what your options are here then please consult the Fortress user guide at http://www.rcac.purdue.edu/userinfo/resources/fortress/userguide.cfm or contact ITaP at rcac-help@purdue.edu

The new tape library will have an increased capacity from 1.3 PB to 6 PB uncompressed, and uses newer LTO-6 tape technology that should provide a 1.5x speedup when reading data. Additionally, it provides redundant mechanical arms that will deliver additional speed when loading and mounting tapes, as well as providing redundancy for these moving parts.

Thank you for your patience during this upgrade.

Please contact ITaP at rcac-help@purdue.edu if you have any questions or concerns about this upgrade.

samba.rcac.purdue.edu maintenance outage

rcac@purdue.edu — Fri, 08 Mar 2013 09:34:00 PST

On Tuesday, 12 March 2013, the samba.rcac.purdue.edu host will be offline for about 2 hours between 8:00 and noon for maintenance.
This will not affect any running jobs or new job submission, but will mean that people who use Samba to map their home or scratch space to their personal workstation will be unable to do so during the outage.
During this maintenance, the host will be updated to a newer configuration control system.

As always please email rcac-help@purdue.edu with any questions or concerns

Scratch Filesystem Problem

rcac@purdue.edu — Fri, 22 Feb 2013 10:05:00 PST

As of 9:00am, are seeing a problem with the LustreC scratch filesystem that serves Carter, Hansen, and Peregrine1. To prevent any more jobs from running into this, we have temporarily suspended scheduling of new jobs, though you may still submit to queues. Our engineers are working to resolve this as soon as possible.10:00am UPDATE: The problem was triggered to a temporary glitch in the networking andit has been fixed now. All systems have been return to normal operation. We are sorry for any inconvenience this may have caused.

Scheduled Maintenance - WinHPC Cluster (Mar 12, 2013)

rcac@purdue.edu — Fri, 01 Mar 2013 16:03:00 PST

Scheduled Maintenance - WinHPC cluster (Mar 12, 2013)

During the March 12 to March 14 maintenance window, the WinHPC cluster will be unavailable due to upgrades to the electrical service in the MATH data center.

WinHPC will be shut down at 8:00 am Tuesday, 12 March, 2013, and is expected to return to service by 6:00 pm the same day.

As is typical for maintenance periods, any jobs submitted to the queues whose walltime requests would carry them past the start of the outage (8:00 am Tuesday, 12 March) will be held until the cluster returns to service.

Please send any questions or concerns about this maintenance outage to rcac-help@purdue.edu

Chilled water outage in MATH - Jan. 28, 2013

rcac@purdue.edu — Mon, 28 Jan 2013 16:58:00 PST

Update:
As of about 11:00 am, the problem with the chilled water has been corrected, and scheduling has resumed on all RCAC clusters. Thank you for your patience. If you encounter any issues or have questions, please contact us at rcac-help@purdue.edu

Scheduler outage:
The temperature of the chilled water supply in the data center has exceeded its optimum, which affects the cooling for most of our research clusters. Physical plant and RCAC personnel are evaluating the problem and we expect more information by 11:00am.
In the meantime, we have temporarily stopped scheduling on the following clusters: Coates, Hansen, Rossmann, Carter, and Radon. Currently running jobs will continue to run, but newly submitted jobs will wait in the queue until we are sure the hardware temperatures will not exceed workable limits.

Steele Scheduler Outage

rcac@purdue.edu — Thu, 24 Jan 2013 15:48:00 PST

Update:
As of approximately 3:00pm the issue affecting the scheduler on Steele has been resolved. Our engineers discovered a problem with the Torque server that was causing the errors and failures you experienced. The issue has been resolved and the server is back online.
PBS commands should be available again. Any job that was running prior to the service interruption should not be affected.
As always, thank you for your patience. If you encounter any issues or have any questions, please contact us at rcac-help@purdue.edu.

Original:
The scheduler on Steele is currently unavailable due to an issue with the server. Our engineers are working to resolve this as soon as possible. During this outage, new job submissions will not be possible and other PBS commands will be unresponsive or fail.
We will send an update and more information by 3PM (EST)

chilled water outage in MATH - 1/10/2013

rcac@purdue.edu — Thu, 10 Jan 2013 15:48:00 PST

Campus chilled water serving the MATH data center is experiencing above-normal temperatures, and as a precaution, scheduling on the Coates, Rossmann, Hansen, Carter, and Radon clusters has been stopped. Steele is not affected.
There should be no impact on currently running jobs.
The next update on this issue will be at approximately 1:00pm.

Coates: Unexpected Power Outage in MATH

rcac@purdue.edu — Tue, 08 Jan 2013 15:49:00 PST

Update: Noon, 1/8/13

The power issue in MATH has been resolved. Power has been restored to the nodes in the Coates-A subcluster affected by the outage. ITaP engineers have verified that the Coates-A subcluster is operating correctly, and have restarted job scheduling.

As always, thank you for your patience. If you have any questions or encounter any issues please contact us at rcac-help@purdue.edu

Issue:

Due to an unexpected power issue in MATH, a large portion of the Coates-A subcluster is currently offline. To prevent job loss, systems engineers have stopped scheduling on queues that contain Coates-A nodes, including the standby queue. Jobs currently running on the remaining operational nodes should not be affected.

System engineers are working to bring these systems back online and electricians are working to determine the cause of the power outage.

We will send out another update before around 2pm or sooner if the power situation changes.

Thank you for your patience during this outage.

Scheduling paused on ITaP research clusters - 12/1/2012

rcac@purdue.edu — Sat, 01 Dec 2012 15:50:00 PST

During scheduled network maintenance on network equipment connecting storage to ITaP clusters, all scheduling will be paused from 4-6pm.
Running jobs will continue to execute, and new jobs may be submitted to PBS queues, but no new jobs will start until the network maintenance is complete.
Please contact us at rcac-help@purdue.edu if you have questions about this maintenance.

Software Stack Changes during 12/29/12 - 1/4/2013 Scheduled Maintenance

rcac@purdue.edu — Thu, 29 Nov 2012 11:14:00 PST

During the New Years' weekend holiday, all ITaP HPC resources will be unavailable due to a scheduled upgrade of research home directories.

While the systems are down they will also receive several updates to the software stack and modules. These updates include changes to available compilers, OpenMPI 1.6, and Matlab.
WHAT WILL BE THE IMPACT TO OPENMPI?

We will be upgrading our OpenMPI 1.6 installations to the latest 1.6.3 version. This new version offers many performance and stability improvements for RCAC's RDMA and InfiniBand technologies. It is highly recommended that if you are using 1.6 that you upgrade to 1.6.3. We will be removing the 1.6 modules, however, if you need to continue using 1.6 the modules will be available via the "unsupported" module archive.

WHAT COMPILERS WILL BE IMPACTED BY THESE UPDATES?
Several old compilers are being removed:

Intel compilers prior to 12.0.084
PGI compilers prior to 11.8-0
GCC compilers prior to 4.4.0
Libraries that depend on the above compilers

These compilers are being removed to make room for a new set of compilers. We will be introducing the following compilers:

Intel 13.0.1.117
PGI 12.10-0
Popular MPIs and libraries built with these compilers

These new compilers offer bug fixes and enhanced performance and stability. Users are encouraged to send in any experiences with these new compilers to help us evaluate the direction of new compilers on RCAC systems. The default compilers will remain unchanged until these new compilers can be evaluated further.

The default GCC compiler version will be upgraded on all systems as follows.

Carter, Peregrine1: GCC 4.4.5 to 4.7.2
All other systems: GCC 4.4.0 to 4.7.2
Popular MPIs and libraries will be built with GCC 4.7.2

This will provide a unified GCC software stack across all of our systems. If you need to continue using GCC 4.4.0 you may do so by explicitly loading "gcc/4.4.0".

WHAT OTHER SOFTWARE WILL BE IMPACTED?
There will be a few other miscellaneous upgrades including upgrading the default matlab module to R2012a and removing a couple old versions of Matlab. If you need to continue using the old default R2010a, you may do so by explicitly loading "matlab/R2010a", though it is recommended that you upgrade to R2012a.

HOW DO I KNOW IF MY WORKFLOW WILL BE IMPACTED?
Whenever a module that will be impacted is loaded a notice is printed to your screen or job output log. Please take a look at your job output over the next week or so and make note of any changes being advertised. You may continue using these modules until December 29th to allow time to make any changes necessary. Users are encouraged to make any changes necessary to avoid disruption when systems are returned to service.

WHAT IF AN IMPACTED MODULE IS REQUIRED BY MY RESEARCH?
We understand some users may not be able to change compilers or MPI libraries in the middle of a research project. These modules will be available in the "unsupported" archive. You may use the archive by adding "module load unsupported" to your job scripts (prior to loading any other modules).

WHY ARE YOU CHANGING THE SOFTWARE STACK?
RCAC aims to provide a software stack that allows for optimal use (performance and stability) of the clusters. This necessitates periodic updates to the stack as compilers, libraries, and software are improved over time. By removing older modules from the main stack we help ensure the selection is simple and easy for users to find the best compilers and libraries to use. If no modules were removed the selection would become difficult for users to understand and navigate as well become difficult for RCAC staff to manage. Any major changes will be coordinated with scheduled maintenance periods to minimize impact.

Please send any questions or concerns with these changes to rcac-help@purdue.edu.

For further information on the maintenance scheduled for 12/29/12 – 1/4/2013 please visit:

http://www.rcac.purdue.edu/news/detail.cfm?newsId=559

Scheduled Maintenance - RCAC home directory upgrades (12/29/12 - 1/4/2013)

rcac@purdue.edu — Fri, 04 Jan 2013 19:16:00 PST

Update - 7:00pm, 1/4/2013:

All community clusters (Steele, Coates, Rossmann, Hansen, Carter, and Peregrine1) are back in production.
Radon is currently not in production, as ITaP engineers are addressing issues encountered during the upgrade.
The Condor Pool is operating at reduced capacity.

During the New Years' weekend holiday, all ITaP HPC resources will be unavailable due to a scheduled upgrade of research home directories.

During this downtime, systems will also receive OS updates, network updates, upgrades to the PBS batch system software, and updates to the software stack.

WHAT IS THE IMPACT OF THIS UPGRADE?

From 8:00am, Saturday December 29 through 5pm on January 4, 2013, all research clusters will be unavailable for login or running jobs. This includes the Condor pool, Radon, Moffett, Peregrine 1, Steele, Coates, Rossmann, Hansen, and Carter.

Additionally, any PBS jobs which request a walltime which would take them past the starting maintenance dates will not start and will remain in the queue until after the maintenance is completed.

Access to the Fortress archive system will not be interrupted.

WHY ARE HOME DIRECTORIES BEING MOVED?

After 5 years of service, the NAS system currently providing research home directories is being retired. Beginning January 3, 2013, ITaP research computing home directories will leverage the new Extensible Storage Facility hardware deployed for the new Boiler Backpack service [1][2].

WHAT ARE THE BENEFITS OF THE NEW HOME DIRECTORY SERVICE?

This upgrade will provide a number of benefits to research customers:

Home directories will no longer have a limit on the number of files.
Users will be able to use a self-service tool to select the allocation of their 100 GB of Boiler Backpack storage between CIFS, cloud storage, and Research homes.
- For example, you may opt to allocate 50G to home directories, 25G to CIFS, and 25G to cloud storage, or any other combination.
- More information on this self-service capability will be provided in future announcements.

While we regret the need for days without the clusters being available, we are excited about the new research capabilities that this new service will provide going forward.

Please send any questions or concerns about this upgrade to rcac-help@purdue.edu.

[1] http://www.purdue.edu/newsroom/general/2012/120627McCartneyEMC.html

[2] http://www.purdue.edu/boilerbackpack/

Scheduled Maintenance for Radon Cluster - 27 November 2012

rcac@purdue.edu — Tue, 27 Nov 2012 12:43:00 PST

UPDATE - 12:40 pm 27 Nov 2012:
The update went smoothly, with no delays or problems, and Radon has returned to service as of 12:30 pm, 5 and a half hours earlier than expected. Please let us know at rcac-help@purdue.edu if you see any problems with the system, or have any questions about this upgrade.

The Radon compute cluster will be unavailable from 8:00 am to 6:00 pm Tuesday, 27 November, 2012. During this outage, we will be installing the latest update to its batch scheduling software. This upgrade should provide enhanced reliability and better scalability of batch jobs.

System	Going Offline	Return to Service
Radon	8:00am, 27 Nov.	6:00pm, 27 Nov.

Any PBS jobs which request a walltime which would take them past the starting maintenance time will not start and will remain in the queue until after the maintenance is completed.
Please send any questions or concerns to rcac-help@purdue.edu.

Scheduled Maintenance - October 2012

rcac@purdue.edu — Tue, 09 Oct 2012 11:44:00 PST

UPDATE: 9 October, 2012
The Coates and Rossmann Clusters have both returned to production, and their maintenance is completed, as of 11:30 am, Tuesday 9 October, 2012

The Coates and Rossmann clusters will go down for scheduled maintenance at 8:00 am Monday, 8 October, and return to service by 6:00 pm Tuesday, 9 October.

Planned Maintenance Timeline
System	Going Offline	Return to Service
Coates	Mon, Oct 8, 8:00am	Tue, Oct 9, 6:00pm
Rossmann	Mon, Oct 8, 8:00am	Tue, Oct 9, 6:00pm

During the maintenance window scheduled for October 8-9, the Coates and Rossmann clusters' resource manager and scheduling software will be upgraded from PBS Pro to Torque and Moab, an upgrade which has already been completed on the other research clusters.
Full details on the batch system conversion may be found at:
http://www.rcac.purdue.edu/news/detail.cfm?newsId=529
Please contact us at rcac-help@purdue.edu with any questions about either the system maintenance or the Torque conversion.

ADIC Scalar 10k tape library maintenance - 9/7/2012

rcac@purdue.edu — Thu, 06 Sep 2012 19:26:00 PST

From 8:00am-12am on Friday, September 7, 2012, the ADIC Scalar 10K library serving the Fortress archive will be unavailable while emergency preventative maintenance is performed.

Fortress will still be able to write files into HPSS, and files already on the cache will remain accessible, but reading or writing to files requiring tape access will not be possible.

Unscheduled Power outage in Math Datacenter

rcac@purdue.edu — Tue, 21 Aug 2012 23:15:00 PST

Update: 10:00pm Tuesday
As of 8:30pm Tuesday 21 August 2012, the LustreB filesystem has been returned to full service. Our storage engineers with assistance of the vendor have verified that the system is stable. If you encounter any issues, please contact us at rcac-help@purdue.edu
Update: 6:00pm Tuesday
Work is continuing on the LustreB fileservers, but all Steele queues have been re-enabled. Jobs that refer to any directories or files within the /scratch/lustreB tree will probably fail and should not be submitted.
Update: 10:00am Tuesday
As of 10:00 am Tuesday 21 August 2012, the LustreB file servers are up, but the vendor is doing further work on them to ensure they will remain stable before returning them to service. Queues on Steele whose owners do not use LustreB are running and accepting new jobs; the 'standby' queue and owners whose users require LustreB remain stopped pending the return of the LustreB storage. The next update will be at 5:00 pm today.
Update: 11:30pm Monday
As of 11:30 pm Monday 20 August 2012, the LustreB file servers are still being worked on. Queues on Steele whose owners do not use LustreB are running and accepting new jobs; the 'standby' queue is still stopped pending the return of the LustreB storage. The next update will be at 10:00 am tomorrow.
Update: 6:20pm Monday
As of 6:20 pm Monday, 20 August 2012, Coates, Rossmann, Hansen, and Carter have been returned to full service. All jobs running when the power went out will need to be resubmitted, as the systems were forced to reboot by the interruption.
Some queues on Steele, including standby, are currently stopped while storage engineers bring the Lustre B filesystem back online. We will have another update prior to 9pm regarding Steele.
Update: 5:30pm Monday
As of 5:20 pm Monday, 20 August 2012, the Radon cluster and Moffett SiCortex system have been returned to normal service. All jobs running when the power went out will need to be resubmitted, as the systems were forced to reboot by the interruption.

Monday afternoon, 20 August 2012, the MATH building suffered a brief unexpected power outage. This has had significant impact on all of RCAC's research computing resources. System staff are bringing the clusters back on line at this time.
All Lustre scratch systems were affected; we don't believe any files or data were lost, but jobs using that scratch space were killed, and will need to be resubmitted when the clusters resume full production status. Jobs on Steele that did not use Lustre file systems continued to run, and are probably unaffected, but new jobs will not be scheduled until the system is fully back in service.
Systems known to be affected are Carter, Coates, Hansen, Moffett, Radon, and WinHPC.

Scheduled Maintenance - August 2012

rcac@purdue.edu — Tue, 17 Jul 2012 12:54:00 PST

In August 2012, some RCAC systems will be down for maintenance for up to three days in order to accommodate electrical service and chilled water upgrades in the Math building and OS and scheduler upgrades on the systems.

Planned Maintenance Timeline
System	Going Offline	Return to Service
Carter	Tue, Aug 14, 8:00am	Tue, Aug 14, 6:00pm
Hansen	Tue, Aug 14, 8:00am	Thu, Aug 16, 12:00 noon
Radon	Tue, Aug 14, 8:00am	Thu, Aug 16, 12:00 noon
Moffett	Tue, Aug 14, 8:00am	Thu, Aug 16, 12:00 noon
WinHPC	Tue, Aug 14, 8:00am	Thu, Aug 16, 12:00 noon
Fortress	Tue, Aug 14, 8:00am	Tue, Aug 14, 6:00pm

In addition, the plumbing and electrical work will result in reduced capacity for Coates and Rossmann, from Tue, Aug 14, 8:00am to Thu, Aug 16, 12:00 noon. During this time, the systems will be completely operational but some nodes, served by the power and water lines being worked on, will be offline which may cause some jobs to have an increased waiting time in the queue before they start. We don't expect this to have a large impact on overall throughput.
Any PBS jobs which request a walltime which would take them past the starting maintenance dates will not start and will remain in the queue until after the maintenance is completed.
Please send any questions or concerns to rcac-help@purdue.edu.

Scheduler Outage on Steele

rcac@purdue.edu — Fri, 25 May 2012 17:19:00 PST

Update:
As of approximately 5:00 pm, the issue affecting the scheduler on Steele has been resolved.
Thank you for your patience. Please report any issues to rcac-help@purdue.edu

The scheduler on Steele is currently unavailable.
Our enginerers are working to resolve the issue. We will send an update and more information by 5:00 pm (EDT).
We apologize for the inconvenience and thank you for your patience.

Batch system and module changes on RCAC systems

rcac@purdue.edu — Mon, 23 Jul 2012 09:58:00 PST

Beginning with the new Carter cluster, RCAC users will note some differences in the PBS batch system and the module names available for use. This article aims to outline the reasons for these changes and describe some of the details.
Why has ITaP changed the PBS software used on Carter, Steele, Hansen, and Radon?
TORQUE and Moab provide a number of stability and reliability improvements over PBS Pro and many new features:

Improved scalability for large clusters like Carter
Node health monitoring to prevent job failures
Advanced scheduling capabilities, such as providing fair-share scheduling within the standby queue
GUI interface to job submission and monitoring
More resilient communication protocols to make the cluster more reliable
Cpuset and NUMA capabilities to more effectively use today's NUMA architecture systems
Full integration with and knowledge of 3rd-party software licenses
Graphics processing unit (GPU) support

How does TORQUE differ from PBS Pro as used on other community clusters?
There are some differences that a user will experience when interacting with Torque vs PBS Pro. This table outlines the key differences in qsub options between the two systems. Please use this as a guide in converting submission scripts from PBS Pro to TORQUE:

PBS Pro	TORQUE

Any requests using the original PBS Pro syntax of "nodes=" and "ppn=" also work in TORQUE
-l nodes=N:ppn=P	-l nodes=N:ppn=P

N total cores on any number of nodes, adjust the processes per node to fit whatever is available now
-l select=N	-l procs=N
-l select=N,place=free	-l procs=N
-l select=N:ncpus=1,place=free	-l procs=N

N total cores on exactly N nodes, one process per node
-l select=N,place=scatter	-l nodes=N
-l select=N:ncpus=1,place=scatter	-l nodes=N

N total cores on exactly one node
-l select=1:ncpus=N	-l nodes=1:ppn=N

(N*P) total cores on exactly N nodes, with P MPI processes per node
-l select=N:ncpus=P:mpiprocs=P,place=scatter	-l nodes=N:ppn=P

N total cores on exactly N nodes, one MPI processes per node, with exclusive access to the entire node
-l select=N:mpiprocs=1,place=excl:scatter	-n -l nodes=N
-l select=N:ncpus=1:mpiprocs=1,place=excl:scatter	-n -l nodes=N

You may test your PBS Pro submission scripts for compatibility with the TORQUE version of "qsub" using the "check_torque_syntax" command.

Usage: check_torque_syntax [qsub options] filename

In addition, TORQUE offers the "checkjob" utility, which can tell you more information about the status of one of your jobs, including reasons it may not be able to start.

Why are module names different from other community clusters?

Module names on previous clusters took a variety of different formats both in name and version number. For example, openmpi-1.4.4-intel64/12.1, petsc-3.1-p8-openmpi-1.5.4-intel64/12.0.084, hdf5/1.8.0-gcc3, hdf5-1.8.5/gcc-4.1.2 are all formats that appear on community clusters.
Modules were rarely removed or defaults changed, making maintenance of the software stacks on the clusters extremely complicated.
As we transition to TORQUE, all modules will be given a consistent naming scheme: packagename/packageversion_compiler-compilerversion. For example: Open MPI 1.4.4, built with Intel 12.1 would be named: openmpi/1.4.4_intel-12.1
With a standard in place, RCAC application staff can better manage the installed software: providing default versions, informing when the default will change, and providing advance warning when a software module is to be retired. Additionally, this allows RCAC to provide an automatically-updated online software catalog already under development.
Major changes to the software stack and defaults will be performed only at scheduled downtimes.
Additionally, Carter will see a much smaller set of compiler and MPI versions. Only current default versions of each compiler and MPI will be installed, along with the previous default.
Old versions of software will be archived to an unsupported module path which can be accessed by "module load unsupported". For example, if your code can only build and run with an old version of the PGI compiler that has been deprecated, you can still access that module in the "unsupported" path.
Existing community clusters will begin to transition to this scheme through the spring and summer of 2012.

Some module tips:

"module load devel" will load a convenient set of the recommended compiler/MPI/math library combination for your cluster. In Carter's case, "devel" will load OpenMPI 1.4.4, Intel 12.1, and Intel MKL.
You can always execute "module load openmpi", and load the recommended OpenMPI built with the recommended compiler for the cluster you are on,and similarly for other parallel libraries.
All modules are 64-bit. The "bitness" of a module will no longer appear in its name.

Please contact us at rcac-help@purdue.edu if you have any questions about the tranisiotn to TORQUE, the module changes, or any other concerns.

Scheduled Maintenance - May 2012

rcac@purdue.edu — Fri, 13 Apr 2012 17:03:00 PST

In May 2012, all RCAC systems will each be down for maintenance for up to three days in order to accommodate electrical service work in the Math building and storage systems maintenance. Some systems will also be receiving OS and scheduler upgrades.

Planned Maintenance Timeline
System	Going Offline	Return to Service
Carter	Tue, May 15, 8:00am	Thu, May 17, 12:00pm
Hansen	Tue, May 15, 6:00am	Thu, May 17, 5:00pm
Rossmann	Tue, May 15, 6:00am	Thu, May 17, 5:00pm
Coates	Tue, May 15, 6:00am	Thu, May 17, 5:00pm
Steele	Tue, May 15, 6:00am	Thu, May 17, 5:00pm
Miner	Tue, May 15, 8:00am	Wed, May 16, 12:00pm
BoilerGrid	Tue, May 15, 6:00am	Thu, May 17, 12:00pm
Radon	Tue, May 15, 8:00am	Wed, May 16, 5:00pm
Moffett	Tue, May 15, 8:00am	Wed, May 16, 12:00pm
WinHPC	Tue, May 15, 6:00am	Thu, May 17, 12:00pm
Fortress	Wed, May 16, 8:00am	Wed, May 16, 5:00pm

Any PBS jobs which request a walltime which would take them past the starting maintenance dates will not start and will remain in the queue until after the maintenance is completed.
Please send any questions or concerns to rcac-help@purdue.edu.

Unscheduled HPSS outage

rcac@purdue.edu — Wed, 11 Apr 2012 14:53:00 PST

Update - April 11, 2012 240pm
At around 240pm, ITaP engineers have restored communications between
the HPSS system and the tape library. Access to Fortress from Samba,
HSI/HTAR and other methods has been restored.

I apologize for the inconvenience this has caused. Please email us at
rcac-help@purdue.edu if you continue to experience any problems with
Fortress.

Thank you again for your patience.

Original - April 11, 2012 130pm
At around 130pm, the HPSS archive system lost communications with the
tape library. This is not related to the earlier Samba outage. While
Samba has been returned to service, access to files on Fortress via
Samba, HSI/HTAR, or any other methods will not be available. ITaP
engineers are investigating and working to resolve this issue. There
is currently no estimate for return to service - we will provide
another update by 5pm this evening.
Thank you for your patience.

Unscheduled Samba Outage

rcac@purdue.edu — Wed, 11 Apr 2012 14:01:00 PST

Update : 1:45pm
As of As of 1:45pm this afternoon, systems staff have completed patching the samba servers used to access storage systems. You should now be able to connect to samba.rcac.purdue.edu for samba access to home and scratch directories and fortress-smb.rcac.purdue.edu for samba access to fortress. If you have used data.rcac.purdue.edu for samba access please switch to samba.rcac.purdue.edu.
Thank you for your patience during this downtime
Original:
As of 11am this morning, April 11th, systems staff determined that the samba servers used to access home, scratch, and fortress storage have a critical vulnerability. As a result we are working on getting these servers patched as soon as possible. You may experience issues connecting to samba.rcac.purdue.edu and fortress-smb.rcac.purdue.edu for the next few hours.
Another notice will be sent at 2pm or earlier. Sorry for the inconvenience and thanks for your patience.

Partial outage affecting some Coates queues

rcac@purdue.edu — Tue, 10 Apr 2012 18:52:00 PST

Update - 6:45 pm Tuesday, 10 April 2012
ITaP engineers have found and repaired the network issue that was affecting Coates nodes type B, C and E. Job scheduling has been resumed for all queues. If you encounter any problems, please report them to rcac-help@purdue.edu
Thank you for your patience.

5:45 pm Tuesday, 10 April 2012
Some Coates compute nodes have lost connection to the LustreA scratch file storage system. Affected nodes are in the type B, C, and E classes; type A and type D nodes are not affected.
System engineers are working on the problem. Until it is corrected, scheduling has been stopped on all queues on the affected nodes
New jobs on the affected queues will be accepted but held until scheduling is restarted. Currently running jobs will probably stall, and then continue once the problem has been solved.
The next update is scheduled for 9:00 pm tonight.

Unscheduled outage to MATH datacenter - 3/30/2012 (Updated)

rcac@purdue.edu — Mon, 02 Apr 2012 00:21:00 PST

Update - 9:30pm, 4/1/2012:

As of about 9:30pm, Sunday, 1 April, ITaP systems staff have returned Hansen to production status, and job scheduling is re-enabled.

The scratch filesystem on Hansen has been restored with no apparent loss of files; if you do encounter any problems, please let us know at rcac-help@purdue.edu.

Thank you for your patience during this outage.

Update - 5:00 pm, 4/1/2012:

The LustreC filesystem is still unavailable. ITaP engineers continue to work with the vendor to bring the system back online. Job scheduling remains disabled on Hansen.

We will update the status on Hansen storage at 9:00 pm, 4/1/2012

Update - 2:45pm, 4/1/2012:

The LustreA filesystem has been restored and Coates and Rossmann are back in production mode, with jobs running.

Storage engineers are still working on LustreC, so job scheduling on Hansen remains disabled.

We will update the status on LustreC and Hansen at 5:00pm 4/1/2012

Update - 12:00am, 4/1/2012:

ITaP engineers continue to work with storage vendors to bring LustreA and LustreC filesystems back online.

Job scheduling on Coates, Rossmann, and Hansen remains disabled.

The next update on the status of Coates, Rossmann, and Hansen storage will come by 12:00 noon, 4/1/2012.

Update - 4pm, 3/31/2012:

LustreB has been returned to production on Steele.

LustreA (Rossmann, Coates), and LustreC (Hansen) are still unavailable, as ITaP engineers are working carefully with vendor assistance to repair the filesystems while avoiding data corruption.

The current goal for return to production is 9pm.

Update - 3:30am, 3/31/2012:

Radon - In production

Moffett - In production

Steele - In production

Hansen - Available for login, job starting is disabled. LustreC unavailable - time to repair unknown.

Rossmann - Available for login, job starting is disabled. LustreA unavailable - time to repair unknown.

Coates - Available for login, job starting is disabled. LustreA unavailable - time to repair unknown.

ITaP storage engineers have escalated Lustre issues to the storage vendor. We will send an update on the cluster status by 12 noon, 3/31/2012.

Original Message:

At approximately 11:10 PM, March 30, 2012, part of the West Lafayette campus experienced an unexpected interruption to electrical service.

The part of campus affected included the MATH datacenter housing Coates, Rossmann, and Hansen clusters.

The Steele cluster is not affected.

Systems engineers are currently assessing the extent of this unscheduled outage.

At this time, there is no estimate of a return to production.

Please contact us at rcac-help@purdue.edu for any questions regarding this outage.

PBS unavailable on Rossmann cluster - 3/22/2012

rcac@purdue.edu — Thu, 22 Mar 2012 22:30:00 PST

Due to a network issue, the server running the PBS software for Rossmann is unavailable.

While the server is unavailable, attempts to use PBS commands ("qsub", "qstat", "pbsnodes") will fail with error messages like:

qstat: cannot connect to server rossmann-adm.rcac.purdue.edu (errno=113)

Even though PBS is unavailable, frontends can be accessed interactively.

Systems engineers are working to correct the problem. There is currently no estimate for a return to production.

Unscheduled outage to Rossmann cluster - March 15-16

rcac@purdue.edu — Tue, 20 Mar 2012 16:07:00 PST

At approximately 10:50pm, Thursday, March 15, the power distribution to large portions of the Rossmann cluster failed. These feeds also power the login nodes for the cluster, which, while unavailable, renders Rossmann unavailable for use.
Power was restored to the Rossmann racks in the datacenter this morning and ITaP engineers returned Rossmann to service at 9:30am.

Around 2:00pm, the problem that took approximately half the nodes and the front-ends offline last night has recurred, and Rossmann again became unreachable, though many jobs continued to run.
As of 2:50pm, half of Rossmann's nodes are still down due to the power failure, but one frontend has been restored to service, so it should be possible to monitor any active jobs and some jobs may still be able to run effectively, though queue waits will be longer than usual.

Engineers are continuing to work on restoring power to the remainder of Rossmann, and we will post an update by 5:00pm if not sooner.
As of 5:00 we are still running with about half the Rossmann compute nodes down. The front-end login hosts are up but we have seen intermittent problems causing poor interactive response, and network engineers continue to work on this issue. Our current plan is to issue another update tomorrow (Saturday) about 5:00. Until then we will try to maintain Rossmann in its current state.
New jobs can still be submitted, and running jobs can be monitored, but the queue wait time may be increased due to the reduced node count, and responsiveness of the login hosts may be poor at times.
On Sunday, 18 March, the system administrators implemented a workaround for the interactive response problem.
On Monday, 19 March, the power distribution issue was fixed.
As of Tuesday, 20 March, Rossmann is operating at full capacity again.

System Maintenance - Spring Break 2012

rcac@purdue.edu — Tue, 21 Feb 2012 17:23:00 PST

During the week of spring break, 2012, the Steele, Coates, and Rossmann clusters will each be down for maintenance for one day to install OS patches and update the PBS batch software to version 11.1.
Additionally, the Radon cluster will be unavailable on March 15 for installation of additional compute nodes.
Maintenance Schedule, by System:

Steele: 8am to 5pm on Tuesday, March 13
Coates: 8am to 5pm on Wednesday, March 14
Rossmann: 8am to 5pm on Thursday, March 15
Radon: 8am to 5pm on Thursday, March 15

Any PBS jobs whose walltime request will take them past these dates will not start and will remain in the queue until after the maintenance is completed.
Please send any questions or concerns to rcac-help@purdue.edu.

Lustre unavailable on Hansen cluster

rcac@purdue.edu — Tue, 07 Feb 2012 21:55:00 PST

Update:
As of 9:45pm, Lustre is back in production and scheduling has resumed on Hansen.
Original Notice:
As of approximately 8:00pm February 7, an issue was found the Lustre filesystem on Hansen making the filesystem unavailable for use. ITaP engineers are on site working on restoring the system to production, and estimate a return to service prior to midnight.
Already running jobs will block until the filesystem returns.
While the filesystem is unavailable, no new PBS jobs will be started on Hansen.
Please report any issues to rcac-help@purdue.edu.

Regular account expiration starting on ITaP research clusters

rcac@purdue.edu — Tue, 07 Feb 2012 13:20:00 PST

In February, 2012, many accounts belonging to former Purdue students and staff will be retired.

For most computing services at Purdue, your Purdue Career Account and @purdue.edu email will remain active as long as you remain affiliated with Purdue University. Since some services are based on your specific affiliation, certain services will be removed immediately.

Students no longer registered and taking courses in a given semester, and faculty/staff no longer at Purdue will lose access to Purdue computing resources after their affiliation ends. This removal process takes place twice a year - at Spring and October breaks.

Accounts for December graduates are deleted during Spring Break and accounts for May and August graduates are deleted during October Break. Faculty and staff accounts are also purged on the same bi-annual schedule.

What does this mean for your group members on the Community Clusters?

Following students' graduation or departure of research faculty or postdocs, they will automatically lose access to clusters on the schedule described above. All files in home directories will be safely archived.

If you have a business need for former students, postdocs, or collaborators to retain their access to research clusters, all you need to do is use the Request for Privilege process (R4P) to make this possible.

Please contact your business office or departmental IT support for assistance with this process.

Coates Scheduling unavailable - 1/30/2012

rcac@purdue.edu — Mon, 30 Jan 2012 11:31:00 PST

This morning, the PBS system on Coates developed an issue with the storage holding its internal state.

While systems engineers are working on recovering it from backup, any new job submissions will not be possible, nor will you be able to query job status with "qstat".

Engineers are also investigating any potential impact to currently-running jobs, which is currently unknown.

"qsub" or "qstat" will result in messages such as:

-bash-3.2$ qstat -a

Connection refused

qstat: cannot connect to server coates-adm.rcac.purdue.edu (errno=111)

We regret this inconvenience and will send an update at 12:00 noon. If you have any further questions, please contact us at rcac-help@purdue.edu

Fortress: ADIC Scalar 10k tape robot unavailable (1/4/2012)

rcac@purdue.edu — Mon, 09 Jan 2012 15:47:00 PST

Update - 1/9/2012

The repairs to the ADIC tape library have been completed and Fortress' tape functionality is back in operation.

Update - 1/6/2012

Following work today by vendor engineers, the latest estimate for the ADIC tape robot's return to service is 5PM Monday.

Original Notification

The ADIC Scalar 10K library serving the Fortress archive is currently offline with a mechanical failure. Support has been called, but the earliest estimate for repair is 5pm Friday, January 6. 2012.

Fortress is still able to write files into HPSS, and files already on the cache will still be accessible, but access to files requiring to be read from tape will not be possible.

Miner Downtime Scheduled for Jan 4

rcac@purdue.edu — Tue, 20 Dec 2011 16:53:00 PST

The Miner cluster will be taken down for one hour on January 4, 2012 from 10:00am - 11:00am Central Time (time at Purdue Calumet). This is to allow network engineers to diagnose some intermittent network problems with a core router between Calumet and West Lafayette.
If you have any questions or concerns, please contact rcac-help@purdue.edu.

Hansen: unscheduled outage to Lustre scratch (12/6/2011)

rcac@purdue.edu — Tue, 06 Dec 2011 22:13:00 PST

Update
The error condition on the Lustre filesystem has been cleared, and Hansen is back in production and accepting new jobs.

Jobs already running should have resumed at the point where they were blocked waiting when the Lustre error occurred.

This afternoon (December 6, 2011) the LustreC scratch filesystem serving the Hansen cluster has suffered a multiple disk failure. As a result, scheduling has been stopped on Hansen.
This issue has been escalated to the storage vendor. We currently have no estimate for when LustreC will return to service.

Fortress: ADIC Scalar 10k tape robot unavailable (12/1/2011)

rcac@purdue.edu — Fri, 02 Dec 2011 16:21:00 PST

Update 12/2/11 (4:15pm)

The tape robot has been returned to service and Fortress is back in production. Please contact us at rcac-help@purdue.edu if you encounter further issues.

Update 12/2/11:

The ADIC Scalar 10K robot is temporarily down again with a different error. Engineers are investigating and currently estimate that it will be unavailable until 12:00 noon. We will update you on the tape robot's status at noon.

Original message:

The ADIC Scalar 10K Robot serving the Fortress archive is currently offline with a hardware failure. Support has been called, but earliest availablility for repair is estimated at Friday, December 2 at noon.

Fortress is still able to write files into HPSS, and files already on the cache will still be accessible, but access to files requiring to be read from tape will not be possible.

Update:

As of 5:00pm, Thursday, Dec 1, the ADIC tape robot has been repaired and returned to service.

Hansen MPI Software Changes - December 16, 2011

rcac@purdue.edu — Tue, 22 Nov 2011 16:54:00 PST

As a follow up to our earlier tip describing the optimal MPI for computations on Hansen, we are undertaking an effort to reduce the number of maintained MPI implementations.

This will make it easier for you, the user, to identify and use the best MPI without the potential confusion caused by unnecessary software installations.

On December 16, 2011, the following MPI installations will be REMOVED from Hansen:

All versions of MPICH-1 (such as mpich-1.2.7p1-gcc64)
All versions of OpenMPI older than 1.4.4 or 1.5.4
All versions of MPICH-2 older than 1.4.0

If your research uses an MPI version that will be removed, please re-compile your program with OpenMPI 1.4.4, or MVAPICH2.

If this will not work for you - such as if your software CANNOT use OpenMPI 1.4.4 - please let us know at rcac-help@purdue.edu, and we will make every effort to accommodate you or provide a solution.

Unscheduled LustreA Outage

rcac@purdue.edu — Tue, 15 Nov 2011 11:04:00 PST

The LustreA scratch filesystem, used by Rossmann and Coates, suffered an unknown failure sometime in the early morning of November 15, 2011. LustreA was returned to normal operation at about 10:30am. Any jobs on those systems run overnight before then that use scratch storage may have either stalled or failed.

Please contact rcac-help@purdue.edu if you have any questions. Thank you.

Hansen Network Maintenance - 8:00AM, November 17, 2011.

rcac@purdue.edu — Mon, 30 Jan 2012 10:01:00 PST

Updated 11/30/11:

Network engineers have identified the cause of the network issue in question, and have applied a workaround, which has restored the Hansen network to full functionality.

The next maintenance window to address to address the root cause is scheduled for next Monday, (December 5, 2011) from 8:00am to 11:00am.

Thank you for you continued understanding. If you require any assistance or additional resources while job run times are limited to one week, please contact us at rcac-help@purdue.edu.

Updated 11/21/11:

Last Thursday, ITaP network engineers performed some troubleshooting on this network issue, and have identified new steps that can be taken towards resolving of this issue.

Therefore, another maintenance window has been scheduled for next Monday, November 28, 2011, from 8:00-11:00am.

Any jobs still running will be requeued, and any new job whose requested walltime would take it past this maintenance will remain in queue until the maintenance is complete.

Thank you for your patience while engineers continue to troubleshoot this issue. If you have any questions or concerns, please contact us at rcac-help@purdue.edu.

Original Message:

During these first weeks of run-time since Hansen has gone production, an intermittent issue with Hansen's network has ben discovered, for certain use cases.

While many of you have not encountered this issue, in order to ensure that Hansen continues to function in an optimal manner, ITaP network engineers have planned maintenance to troubleshoot the issue next Thursday, November 17, 2011, beginning at 8:00 AM.

This will result in a 2 hour outage to Hansen.

Any jobs still running will be requeued, and any new job whose requested walltime would take it past this maintenance will remain in queue until the maintenance is complete.

While we hope that this single maintenance will resolve the issue, we have, for the time being, changed Hansen's maximum wall time for a job from 30 to 7 days, which will allow network engineers a shorter troubleshooting cycle to perform further work, should it be necessary.

When the issue is resolved, the maximum run time will return to 30 days.

Thank you for your continued understanding. As always, if you encounter issues or have further questions, please don't hesitate to contact us at rcac-help@purdue.edu

Optimizing MPI performance on Hansen

rcac@purdue.edu — Thu, 03 Nov 2011 09:06:00 PST

To provide an optimal computing experience on the new Hansen cluster, ITaP would like to offer some tips on utilizing Hansen most effectively.

Hansen utilizes Mellanox 10 Gbps Ethernet adapters for its network interconnect. In addition to handling Hansen's TCP/IP traffic, these adapters can use a special protocol, named "RoCE" (pronounced "Rocky"), to provide hardware acceleration for MPI communication. In practice, enabling RoCE increases available bandwidth by up to 50%, and decreases latency by a factor of three.

You can read more about RoCE here:

http://www.mellanox.com/related-docs/prod_software/ConnectX-2_RDMA_RoCE.pdf

To fully take advantage of RoCE, MPI applications should be recompiled using OpenMPI 1.4.4, or MVAPICH-2.

You can list all compiler versions OpenMPI 1.4.4 by using the command:

module avail openmpi-1.4.4

To use the PGI version of OpenMPI 1.4.4:

module load openmpi-1.4.4-pgi64

The following graphs illustrate the latency, bandwidth, and message rate improvements achieved simply by using a RoCE-enabled OpenMPI vs the TCP/IP based MPICH-2 1.4.0. (Of course, actual results with your application may vary).

Please give OpenMPI 1.4.4 a try with your MPI application and see if it can help improve your time to solution.

If you have an questions or issues using OpenMPI 1.4.4, please don't hesitate to contact us at rcac-help@purdue.edu.

Unscheduled Steele outage - October 11, 2011

rcac@purdue.edu — Tue, 11 Oct 2011 15:26:00 PST

At approximately 11:30 am, October 11, 2011, there was an unplanned outage to the cooling plant serving the Steele

cluster, and to prevent damage, systems engineers were forced to shutdown Steele's compute nodes.

All running jobs have returned to the queue.

Physical facilities and ITaP staff are currently working on restoring cooling. It estimated the Steele will return to production at approximately 3:00pm.

Fortress archive upgrade to HPSS Oct 7-14

rcac@purdue.edu — Sat, 22 Oct 2011 08:59:00 PST

UPDATE: Archive Conversion moved to Oct. 7-14
Information Technology at Purdue (ITaP) is upgrading the research computing archival storage system Fortress. Currently based upon EMC's DiskXtender (DXUL), Fortress is being upgraded to new, more powerful software: the High Performance Storage System (HPSS).

Data currently in the DiskXtender archive will be migrated transparently to the new system between Friday, Oct. 7, and Friday, Oct. 14. Benefits include increased transfer speed when using parallel FTP (PFTP) and the ability to provide scalable capacity and I/O performance.

We have scheduled a training session for 2:30-3:30 p.m. Tuesday, Oct. 4,in Stewart Center 214A to go over in more detail the new features and access methods for the HPSS software that will now support the Fortress service. Registration is not necessary, but to assist in attendance estimations please email Bret Madsen (bmadsen@purdue.edu) if you plan to attend.

The conversion will begin on Oct. 3 with the existing system placed in a read-only mode to allow for the migration. Fortress will be completely unavailable for the weekend starting on Friday, Oct. 7, with full service expected to return by 5 p.m. on Oct. 14. During the migration, the cluster scratch space will be used to allow for additional storage needs. Please email rcac-help@purdue.edu if you need a temporary increase of your scratch quota during this time. We encourage any user who has data to be uploaded to the system to plan uploads around the migration window.

Frequently Asked Questions (FAQ)

How does HPSS differ as far as access to my data?

Plain-text FTP will NOT be supported, as it presents a security risk that Purdue's security policies no longer allow. In its place, we will have the more powerful PFTP, which is both secure and parallelizes data transfers so that the full network bandwidth can be utilized.
NFS mounting is possible, but will be limited to RCAC cluster front-end hosts and certain RCAC assets where it is deemed essential because there is potential for NFS calls to cause performance issues. NFS mounts outside of the rcac.purdue.edu domain, such as on departmental systems, will not be supported.
Access via CIFS and FTP will remain, although these protocols will be less able to take advantage of the optimization of the new system to parallelize data streams.
HSI and HTAR have been added to provide additional features, such as a Unix-style environment interface taking advantage of the power of HPSS without requiring any special user interaction, and a utility to aggregate a set of files from the local file system along with an index file stored in the same directory as the HTAR file.

How can I find out more?

You can find out more about HPSS on the collaboration's web site: http://www.hpss-collaboration.org/.
New online documentation has been created to help with getting to know the system.

ITaP is very excited about the new archive offering and we hope you will find the HPSS system to be useful.
Fortress information and future updates to the project's status can be found on the RCAC web site at: http://www.rcac.purdue.edu/userinfo/resources/fortress/

If you have any questions about this upgrade, please contact us at rcac-help@purdue.edu.

Hansen Cluster is in Production

rcac@purdue.edu — Thu, 15 Sep 2011 10:10:00 PST

As of 10:00 AM, September 15, 2011, the Hansen Community cluster has gone into production.

Hansen is comprised of 200 Dell computer nodes, with four 12-core AMD Opteron 6176 processors (48 cores per node), and is interconnected with RDMA-enabled 10Gb Ethernet.
In conjunction with Hansen's go-live this morning (Sept. 15, 2011), we are pleased to announce the availability of online self-service user and queue maintenance.
The cluster also features a high-performance 300TB Lustre parallel filesystem for scratch storage. In conjunction with Hansen's go-live this morning (Sept. 15, 2011), we are pleased to announce the availability of online self-service user and queue maintenance.
This application will enable you (or a designee of your choosing) to directly manage the list of users who are authorized to access your queue on the Hansen cluster.
Simply by typing their first name, last name, or career account username into the "add a new user" search box, you can add your students and collaborators as members of your queue!
New accounts will be created via a batch process overnight, and will, for the time being, turn around in approximately 1 business day.
The interface is available at:
https://web.rcac.purdue.edu/account/user/

If you are a owner-partner on another cluster, you will notice that you can see your queue lists for those clusters. However, self-service user and queue maintenance has not been enabled on the other clusters (Steele, Coates, Rossmann). You can expect this to occur during the next month.

If you encounter any issues with the user/queue application or with getting started on Hansen, please contact us at rcac-help@purdue.edu.

Coates PBS scheduler issues - 8/17/2011 (Updated 8/26/2011)

rcac@purdue.edu — Thu, 06 Oct 2011 09:39:00 PST

This week, ITaP engineers have been troubleshooting issues with the Coates cluster, with the most common symptom being PBS jobs that abort or restart after some period of run time.
Late yesterday afternoon, a change was made to the cluster's networking configuration that appears to have dramatically improved this issue.
During this coming weekend, monitoring is in place to watch for further PBS issues
Please report to rcac-help@purdue.edu :

If your jobs unexpectedly requeue and start back over from the beginning
If you see any MPI communication errors in job output or error files
Be sure to include any job IDs or approximate timestamps of any observed issues.

Thank you again for your continued understanding.

Previous Message
An update on the previously described PBS issues:

Over the past several days, ITaP engineers have successfully resolved two separate network issues that were responsible for the symptoms described below.
occurrences of the issues below with "qstat" and "qsub" have been greatly reduced, but do not appear completely eliminated.

Systems engineers are, however, still investigating open issues regarding the responsiveness of "qstat", and aborted or restarted jobs, including a number of jobs that were requeued shortly before 7:00pm yesterday evening (August 23).
We appreciate your continued understanding while this issue is being investigated.

Original message:
The Coates cluster is experiencing intermittent issues with its PBS job scheduler. These issues may manifest in any of the following ways:

qstat/qsub: cannot connect to server coates-adm.rcac.purdue.edu (errno=15007)
Slow response to "qstat" or "qsub" commands
Slow job scheduling, usually appearing as jobs remaining idle in the queue for longer than expected

ITaP systems engineers are troubleshooting this issue and hope to have a solution in place soon.

Please contact rcac-help@purdue.edu if you experience any other issues with the community clusters.

Fortress archive system being upgraded to HPSS

rcac@purdue.edu — Thu, 18 Aug 2011 09:41:00 PST

Information Technology at Purdue (ITaP) is upgrading the research computing archival storage system Fortress. Currently based upon EMC's DiskXtender (DXUL), Fortress is being upgraded to new, more powerful software: the High Performance Storage System (HPSS).
Data currently in the DiskXtender archive will be migrated transparently to the new system this fall. Benefits include increased transfer speed when using parallel FTP (PFTP) and the ability to provide scalable capacity and I/O performance.
WHAT IS HPSS?

HPSS is the result of over a decade of collaboration among five Department of Energy laboratories and IBM, with significant contributions by universities and other laboratories worldwide.

HPSS provides highly flexible and scalable hierarchical storage management that keeps recently used data on disk and less recently used data on tape. HPSS uses cluster, LAN, and/or SAN technology to aggregate the capacity and performance of many computers, disks, and tape drives into a single virtual file system of exceptional size and versatility. This approach enables HPSS to easily meet otherwise unachievable demands of total storage capacity, file sizes, data rates, and number of objects stored. HPSS provides a variety of user and filesystem interfaces ranging from the ubiquitous vfs, ftp, CIFS, and NFS to higher performance pftp, client API, local file mover, and third party SAN (SAN3P).

WHY MOVE TO HPSS?

The reason for the move is two-fold.

DiskXtender is no longer supported by EMC as of August 11, 2011.

Purdue's research data archive use increased five-fold in the past five years and it was determined that Purdue's usage pattern was reaching the limits of what our installation of DiskXtender could accommodate, especially in the areas of stability and the ability to access the archive at speeds now possible with modern networking and hardware.

WHAT DOES THIS MEAN FOR ME?

For the most part, it means quicker, more robust, and more secure access to the archive. HPSS is very modular and is able to bring the power of multiple servers to bear on any data flow bottlenecks. After the transition, data will continue to be accessible in the new archive system. DiskXtender tapes will be readable by the new system, so only the metadata associated with your files will be converted to HPSS format. The conversion process and testing is currently in progress and scheduled to be completed this summer to ensure that all your files will be available after the transition.

IS ANYTHING DIFFERENT ABOUT HOW I ACCESS MY DATA?

Yes. Some protocols will not be supported in the new system.

Plain-text FTP will NOT be supported, as this is a security risk that Purdue's security policies no longer allow. In its place, we will have the more powerful PFTP, which is both secure and parallelizes data transfers so that the full network bandwidth can be utilized.
NFS mounting is supported, but will be limited to RCAC cluster front-end hosts and certain RCAC assets where it is deemed essential because there is the potential for NFS calls to cause performance issues. NFS mounts outside of the rcac.purdue.edu domain, such as on departmental systems, will not be supported.
Access via CIFS, SSH copy, and FTP will remain, although these protocols will be less able to take advantage of the optimization of the new system to parallelize data streams.

HOW CAN I FIND OUT MORE?

You can find out more about HPSS on the collaboration's web site: http://www.hpss-collaboration.org/. RCAC will also hold training sessions on using HPSS as the conversion date grows closer, and new online documentation will be created to help with getting to know the system. Notice will be sent about opportunities to attend training sessions.

ITaP is very excited about the new archive offering, and we hope you will find the new system to be useful. As mentioned, we are expecting a fall rollout of the new system, but the exact date is uncertain as yet. Further communications will be forthcoming as we learn more.

If you have any questions about this, please contact us at rcac-help@purdue.edu.

MATH Datacenter upgrades, starting Friday, August 5

rcac@purdue.edu — Mon, 15 Aug 2011 17:32:00 PST

Beginning at 5:00 pm, Friday, August 5^th, the Coates and Rossmann supercomputer clusters will be unavailable due to work to complete a power and cooling upgrade to the Math Sciences building datacenter. We estimate that these clusters will be unavailable for use until approximately Wednesday, August 17. Also affected are the Radon cluster and the Moffett SiCortex system.

While the Steele cluster is undergoing separate maintenance on August 8-9, it is largely not affected by this work.

Only those PBS jobs with walltimes short enough that they will complete prior to 5pm Friday, August 5, will be scheduled for execution -- jobs with longer walltimes will be held in the queue until the systems have been returned to production after the maintenance on August 17^th.

The upgrade project will install two new electrical transformers and a secondary cooling loop to handle the additional power and cooling demands of new supercomputers, as well as provide redundant power to the datacenter to help avoid unscheduled outages. An unscheduled loss of power to the facility can mean millions of hours of lost computing time.

While this work will unfortunately impact on the use of Coates and Rossmann in the near term, it does include a great deal of facilities work that will allow ITaP to better serve existing and future community cluster partners. A brief summary of the work being performed includes:

Work that will enable ITaP to install up to three additional clusters into the Math Sciences building, and provide redundancy to the data center

o Installation of two new redundant transformers

o Installation of a larger capacity chilled water feed to the data center.

o Installation of a secondary cooling loop that will allow the datacenter to continue operation in the event of a disruption to the campus chilled water service

The work will involve

Two full-building power shutdowns, over the weekends of both August 6-7 and August 13-14
Removal and replacement of old electrical gear to improve the reliability of the existing electrical feeds into MATH.
Disconnecting all air conditioners and cooling distribution units (CDUs) from old plumbing and reconnect to new cooling loop.
Reconnecting all data center electrical service to the new transformers.

The project is funded by a $2 million National Science Foundation grant to upgrade and improve Purdue’s main research datacenter, located in the basement of the Mathematical Sciences Building.

Further information about this outage may be found at: http://www.rcac.purdue.edu/news/detail.cfm?newsId=471

Further information about the Steele outage may be found at: http://www.rcac.purdue.edu/news/detail.cfm?newsId=469

Storage maintenance: August 8, 2011

rcac@purdue.edu — Thu, 18 Aug 2011 09:38:00 PST

RCAC home directories, application software space, and BlueArc scratch directories will be unavailable from 8am to 5pm Monday, August 8, while firmware upgrades are made on RCAC's BlueArc storage systems.
While the Coates, Steele, and Rossmann clusters will already be unavailable for use due to other scheduled maintenance, the following systems will be unavailable for use during this maintenance while home directories are offline.

CMS Tier-2 nodes in MANN 112
- (CMS storage will not be affected)
Calumet campus Miner cluster
User login to the Wispy experimental cloud system
User login to Science Gateway portal systems
Condor frontends (condor.rcac.purdue.edu)

The Fortress archival data storage system will remain accessible during this outage.

Steele Cluster Maintenance: August 8-9, 2011

rcac@purdue.edu — Tue, 09 Aug 2011 19:03:00 PST

The Steele cluster will be unavailable Monday and Tuesday, August 8-9, while maintenance is performed on the cooling equipment serving Steele’s POD datacenter, and maintenance is performed on the network switch serving as Steele's interconnect.

Steele will be unavailable beginning at 8 a.m. Monday, August 8, and it is expected to be unavailable until 7 p.m. Tuesday, August 9.

Only those PBS jobs with walltimes short enough that they will complete prior to 6 a.m. Monday, August 8, will be scheduled for execution -- jobs with longer walltimes will be held in the queue until the systems have been returned to production after the maintenance.

Please monitor this news item for any updates, and direct any questions to rcac-help@purdue.edu.

Calumet-West Lafayette Network Disruption

rcac@purdue.edu — Tue, 05 Jul 2011 17:27:00 PST

The network link between Miner and the West Lafayette campus went down July 4th at approximately 5:30pm due to some physical failure in the iLight fiber between Calumet and West Lafayette. Crews are working on this problem now, and hope to have it restored before noon July 5th.
Update: This problem was resolved late in the afternoon on July 5th.

All RCAC systems unavailable some portion of Tue-Fri, 3/29-4/1

rcac@purdue.edu — Fri, 01 Apr 2011 11:37:00 PST

All RCAC systems will be unavailable on Tuesday, March 29th from 3:00am – 6:00pm. The Rossmann, Coates, Radon, and Moffett clusters will remain down through 6:00pm Thursday, March 31st. More details on what is being done and why is available at: http://www.itap.purdue.edu/newsroom/detail.cfm?newsId=2306

Update, 9:00am, March 29: Power has been restored to the Math building and storage upgrades are now proceeding as planned.

Update, 2:30pm, March 29: Storage upgrades continue as planned.

Update, 5:00pm, March 29: Miner has been returned to production service. Steele, Radon, and CMS will be returned to production service as soon as the vendor can correct a problem with their scratch filesystems. Rossmann, Coates, Radon, and Moffett will remain down through Thursday as planned to allow for upgrades to the chilled water service.

Update, 9:30pm, March 29: Steele, CMS, and Miner have been returned to production service. Rossmann, Coates, Radon, and Moffett will remain down through Thursday as planned to allow for upgrades to the chilled water service.

Update, 6:00pm, March 30: Work on the chilled water service to the Math building is proceeding as planned. Rossmann, Coates, Radon, and Moffett are expected remain down through 6:00pm Thursday.

Update, 12:00pm, March 31: Work on the chilled water service continues and has been extended for at least an extra hour, possibly more. Rossmann, Coates, Radon, and Moffett are expected remain down until at least 7:00pm Thursday.

Update, 3:00pm, March 31: Work on the chilled water service continues. Rossmann, Coates, Radon, and Moffett are expected remain down until at least 7:00pm Thursday.

Update, 5:00pm, March 31: Work on the chilled water service has been completed. Systems staff are in the process of bringing up Rossmann, Coates, Radon, and Moffett now. Current expectation for a return to production service is 8:00pm.

Update, 6:30pm, March 31: Radon and Moffett have been returned to production service. Systems staff are working on an issue with the scratch filesystem shared by Rossmann and Coates. Current expectation for a return to production service is 10:00pm.

Update, 8:30pm, March 31: Rossmann and Coates are going through final testing now. Current expectation for a return to production service is 9:30pm.

Update, 9:30pm, March 31: Rossmann and Coates have been returned to production service. However, approximately half of the Coates "A" nodes are still offline due to networking issues. The networking on these will be examined Friday. Because of this, queues on Coates-A may progress more slowly than usual until this is corrected. To try to compensate for this in the interim, all standby jobs have been temporarily held so owner queues can be assured maximum throughput.

Update, 11:30am, April 1: All Coates nodes have been returned to production service. The standby queue on Coates has been re-enabled. All systems are now operating normally.

No further updates are planned on this maintenance.

Moffett.rcac SiCortex system unavailable 3/21-22

rcac@purdue.edu — Tue, 22 Mar 2011 17:19:00 PST

The SiCortex, moffett.rcac, crashed Monday, March 21, 2011, and ITaP's HPC Systems Engineers' initial attempts to restart it failed. They were eventually able to identify and disable a compute node that was experiencing hardware problems, and were able to return the system to service at 4pm Tuesday, March 22.

DXUL/Fortress archival storage system in degraded mode 2/23

rcac@purdue.edu — Wed, 23 Feb 2011 11:30:00 PST

The DXUL/Fortress archival storage system became unable to access tapes in its robotic tape libary at approximately 9:25am Wednesday, February 23. Until ITaP Storage Engineers could correct this problem, it was not possible to stage files from tape into the DXUL/Fortress disk cache for access.
This problem was resolved at 11:10am Wednesday, February 23.
Please refer questions to "rcac-help@purdue.edu".

Coates scratch directory changes on 2/22 and 3/31

rcac@purdue.edu — Fri, 01 Apr 2011 09:54:00 PST

The default scratch directory location for the Coates Linux cluster has changed from the BlueArc storage servers to the Lustre storage system installed last summer.
Starting March 31st, your BlueArc scratch directory will be read-only from Coates nodes. You must now use your Lustre scratch directory in all Coates jobs.

As part of this change, your $RCAC_SCRATCH environment variable on Coates has been changed so its contains the location of your Lustre scratch directory, which is also reported by the "findscratch" command. For compatibility, your $RCAC_SCRATCH_OLD environment variable contains the location of your BlueArc scratch directory.

You can use these environment variables to copy the contents of your BlueArc scratch directory into your Lustre directory. One way to do that would be to log onto coates.rcac.purdue.edu and run the command

rsync -auH $RCAC_SCRATCH_OLD/ $RCAC_SCRATCH/

If you would like to check to see which files and directories would be copied without actually copying them, you could run the command

rsync -auHnv $RCAC_SCRATCH_OLD/ $RCAC_SCRATCH/

Please refer questions to "rcac-help@purdue.edu".

Lustre scratch storage system unavailable 2/3-4

rcac@purdue.edu — Thu, 10 Feb 2011 08:26:00 PST

The Lustre storage system that provides scratch storage on the Rossmann and Coates Linux clusters (via /scratch/lustreA) failed at approximately 1:30pm Thursday, February 3. ITaP Storage Engineers are in MATH working on the problem, but we are currently unable to say when the storage system will be returned to service. As a result, PBS job scheduling on Rossmann and Coates has been paused until the storage system is back in operation.

9pm Thursday, 2/3: Diagnostics run by ITaP and vendor Storage Engineers indicated that part of the disk system in one of the servers that make up LustreA suffered a hardware failure. Replacement parts are being express-shipped to Purdue and are expected to arrive Friday morning, 2/4.

11:30am Friday, 2/4: We have been notified that the replacement parts needed to repair the Lustre storage system are expected to arrive at Purdue early Friday afternoon, 2/4.

8pm Friday, 2/4: ITaP and vendor storage engineers are running diagnostics and checking the integrity of the Lustre storage system but are not yet ready to release it for production.

9pm Friday, 2/4: This outage has been resolved. The Rossmann and Coates Linux clusters' Lustre file system was returned to service and PBS job scheduling resumed at 9pm Friday, 2/4.

Steele "c" nodes temporarily unavailable 1/14

rcac@purdue.edu — Fri, 14 Jan 2011 14:08:00 PST

Type "c" nodes in the Steele Community Cluster lost network connectivity from 11:45am-1:45pm ET Friday, January 14, and PBS jobs executing on them at the time the problem occurred were requeued.

Coates and Rossmann cluster job scheduling temporarily suspended 10/30

rcac@purdue.edu — Sat, 30 Oct 2010 22:42:00 PST

Job scheduling on the Coates and Rossmann Linux cluster was disabled from 7:15-10:20pm Saturday, October 30, due to a partial cooling loss in the MATH datacenter.

RCAC front-end/job-submission system reboots 10/29

rcac@purdue.edu — Sat, 30 Oct 2010 22:41:00 PST

RCAC front-end/job-submission systems will be rebooted Friday morning, October 29. Reboots will be begin at 10am ET and each system is expected to be unavailable for 30 minutes or less.
Please refer questions to "rcac-help@purdue.edu".

Steele Linux cluster temporarily unavailable, 10/11-12/2010

rcac@purdue.edu — Mon, 18 Oct 2010 16:56:00 PST

The Steele Linux cluster lost cooling and was shut down shortly before 3am ET Monday, October 11, 2010. ITaP staff and cooling contractors are working to restore the cluster to service, but are currently unable to estimate when it will be returned to production.
1pm 10/11 update: The Steele cluster continues to be out of service while its cooling equipment is evaluated and the reason for its shutdown is determined.
5pm 10/11 update: The Steele cluster will remain out of service overnight due to ongoing work with its cooling equipment.
10am 10/12 update: The Steele cluster remains out of service as maintenance and testing of its cooling system continues.
2:15pm 10/12 update: The Steele cluster remains out of service so that a bypass valve on a chilled water line can be replaced.
10pm 10/12 update: The Stele cluster was returned to production at 9:45pm ET Tuesday, 10/12.

Steele Linux cluster temporarily unavailable, 9/8/2010

rcac@purdue.edu — Fri, 17 Sep 2010 08:24:00 PST

The Steele Linux cluster went offline at approximately 11:30am ET Wednesday, September 8, due to a cooling-related problem. ITaP staff members are working to restore the system to service but are currently unable to estimate when it will be back in production.
Update: As of 1:45pm,the Steele servers were being powered up and prepared for their return to production. Normal job scheduling is expected to resume by 2:30pm.
Final update: The Steele cluster was returned to service at 2:50pm ET. PBS jobs executing when the cluster went offline were requeued if possible, although some jobs may have been lost.

DXUL/Fortress archival storage system temporarily unavailable, 5/25

rcac@purdue.edu — Tue, 25 May 2010 16:01:00 PST

The DXUL/Fortress archival storage system became unavailable at approximately 11:30am ET on Tuesday, May 25, due to hardware problems in its dish cache. ITaP storage engineers replaced a disk drive that had failed and returned the system to service by 3pm.

Cooling problems on coates-b, -c, and -e nodes Tuesday, 4/6

rcac@purdue.edu — Tue, 06 Apr 2010 20:08:00 PST

Coates-b, -c, and -e nodes have been powered down due to a problem with a CDU (cooling distribution unit) that cools those systems. PBS jobs running on those nodes at the time have been requeued for execution after cooling has been restored and the nodes have been returned to production. We are working to correct the cooling problem but are currently unable to say when the nodes will be returned to service.
Update as of 3pm on 4/6/2010: Coate-b, -c, and -e nodes are being powered back up in preparation for their being returned to production, and PBS job scheduling is expected to be re-enabled on them by 5pm.
Final update: Coates-b, -c, and -e nodes were powered up by 4:30pm Tuesday, 4/6, and normal job scheduling on all nodes in the cluster resumed by 5pm.

Emergency VMWare System Maintenance Friday, 3/26

rcac@purdue.edu — Fri, 26 Mar 2010 14:15:00 PST

ITaP staff will be conducting emergency VMWare system maintenenance from noon-2pm today, March 26. During this time, access to RCAC home and scratch directories may not be possible via SMB/CIFS provided by samba.rcac.purdue.edu or data.rcac.purdue.edu, which is one of the VM systems that will be out of service due to this emergency maintenance.
Update, Friday 3/26 2:00pm: We believe the VM systems to now be stable, but are still doing further testing. We do not plan on any further reboots at this time, but may have to do so if further problems are discovered.

Emergency VMWare system maintenance Tuesday, 3/23

rcac@purdue.edu — Wed, 24 Mar 2010 01:51:00 PST

ITaP staff will be conducting emergency VMWare system maintenenance from 6-8pm today, March 23. During this time, access to RCAC home and scratch directories will not be possible via SMB/CIFS provided by samba.rcac.purdue.edu/data.rcac.purdue.edu, which is one of the VM systems that will out of service due to this emergency maintenance.
Update: The VM maintenance was completed and all RCAC services returned to normal operation by 9pm Tuesday, March 23.

Scratch filesystem work notice Mon, 2/8

rcac@purdue.edu — Fri, 19 Mar 2010 12:03:00 PST

Colleagues,
As you are aware, we have been experiencing performance problems with our scratch file system since the fall semester. Our read performance has dipped from 100 MB/sec to below 10 MB/sec. Similarly, our write performance has also made a corresponding dip. To address these problems, we have rebalanced the load within the storage system by moving users, eliminated old files to decrease fragmentation, and installed additional disks. Each of these actions has improved the overall system performance through December.
On January 18th, the scratch storage performance degraded significantly. Initial indications led us to believe that a great increase in job starts might have been part of the issue – with a spike of more than 14,000 jobs on that day. In order to diagnose the problem, we suspended the execution of standby and condor jobs on the clusters beginning Friday, February 5. Unfortunately this change did not significantly improve the overall performance.
To further diagnose our problems, we will be taking the following steps during the remainder of this week:

Working with selected users to reduce particular types of computational jobs to better characterize the problems.
Rebooting each of the storage head nodes.
Installing and activating additional equipment scheduled to arrive mid-week.
Temporarily suspending PBS and Condor job scheduling, which was done from 1:30-6pm Tuesday, 2/9.

We apologize for the inconvenience and we will update you throughout the week. We are taking this approach in the hope to avoid complete cluster downtimes. Please refer any questions about these temporary measures to rcac-help@purdue.edu. Thank you for your patience in this matter.

Temporary job scheduling change Friday, 2/5

rcac@purdue.edu — Fri, 19 Mar 2010 12:04:00 PST

Beginning at 5pm on Friday, February 5th, RCAC temporarily suspended execution of newly-submitted or currently-queued jobs using shared scratch file systems for selected user groups across all compute resources in order to diagnose a file system performance issue. Jobs using shared scratch file systems submitted after this time may remain queued, but any jobs running as of 5pm February 5th will not be affected.
Please refer questions about this temporary change to rcac-help@purdue.edu.

Coates cluster network problems Tue-Wed, 1/5-6

rcac@purdue.edu — Wed, 06 Jan 2010 15:16:00 PST

Network problems arose following Coates cluster maintenance Tuesday, January 5. ITaP staff are working to resolve these problems, but we are currently unable to say when Coates will be returned to production.

Final Update: The problems which arose following Tuesday's maintenance on the Coates cluster have been resolved, and the cluster was returned to production at 2:40 P.M. Wednesday, January 6.

RCAC front-end systems to be rebooted Tuesday, 11/24.

rcac@purdue.edu — Wed, 25 Nov 2009 07:32:00 PST

RCAC front-end/job-submission systems will be rebooted between 1-3pm on Tuesday, November 24, 2009, to install system updates. Each front-end system will be unavailable for approximately 30 minutes during this time period while it is rebooted.

Pete cluster front-end systems to be rebooted Wednesday, 11/11

rcac@purdue.edu — Fri, 13 Nov 2009 07:46:00 PST

The Pete Linux cluster front-end/job-submission systems will be rebooted at 10am Wednesday, November 11, 2009 to install kernel updates. Queued and executing non-interactive PBS jobs should be unaffected, but the front-end/login systems will be unavailable for approximately 30 minutes while they are rebooted.

Purdue partnership for putting idle computers to work on research wins international award

rcac@purdue.edu — Thu, 05 Nov 2009 17:03:00 PST

Purdue is being recognized as a worldwide campus technology innovator for its DiaGrid system, harnessing what would be otherwise wasted computing power for major research projects.

Campus Technology Magazine has selected DiaGrid for one of its 2009 Campus Technology Innovators Awards. The annual awards, highlighted in the magazine’s current issue, recognize higher education institutions for groundbreaking initiatives in educational technology that are models for other schools. Campus Technology is a leading monthly publication produced by 1105 Media Education Group to deliver news and real-world examples covering the use of technology to advance the way colleges and universities operate and educate today.
Purdue’s partners in DiaGrid include Purdue's Calumet and North Central campuses, Indiana University-Purdue University Fort Wayne, Indiana University, Indiana State University and the universities of Notre Dame, Louisville and Wisconsin.
Purdue was one of 11 award winners selected out of 349 nominations from higher education institutions across the globe. The winners are highlighted in the magazine’s August edition and on CampusTechnology.com.

"We are delighted to present Purdue with a 2009 Campus Technology Innovators award," said Geoffrey Fletcher, editorial director of Campus Technology. "DiaGrid provides millions of hours of computation that would otherwise be wasted, without the need for additional technology or facilities purchases. The sheer size and ingenuity of the initiative, as well as the diversity of computing resources represented in the grid, really set the project apart."

At heart, DiaGrid is a working example from the hot field of network-centered "cloud" computing services.
"As cloud computing becomes the new standard, Purdue has positioned itself as an ongoing leader in this technology, especially for scientific research and educational purposes," said Gerry McCartney, Purdue's vice president for information technology and chief information officer.

Operated by ITaP, Purdue's central information technology organization, DiaGrid started as an on-campus system. But the system now includes nearly 27,000 processors at Purdue and the partner campuses. Purdue has an ongoing effort to add partners, with a goal of incorporating 100,000 processors.

"We continue to explore new applications and are seeking additional institutions to join the project," said John Campbell, associate vice president in charge of research computing for ITaP. "DiaGrid allows institutions to maximize their technology investments while supporting the diverse needs of faculty."

Researchers at Purdue and elsewhere use DiaGrid for purposes such as imaging the structure of viruses at near-atomic resolutions — the better to develop new ways of battling viral illnesses ranging from swine flu and the common cold to West Nile virus and AIDS.

The distributed computing pool also has been used to, among other things: understand the Solar System’s formation; project the reliability of Indiana’s electrical supply; and model the spread of water pollutants to develop best practices for ensuring water quality and availability.

DiaGrid works by pooling computers over the Purdue campus network and off campus via the Internet and fast research networks. Whenever computers in the pool are idle — at night, when their owners are at lunch and otherwise — the system sends work to them. When computer owners need their computers, jobs in progress automatically get shifted to other machines. The variety of hardware in DiaGrid includes computers in student computer labs, offices, server rooms and supercomputing clusters. It also is the first multi-campus collaboration of its kind.

Photo caption: Purdue’s DiaGrid team (L-R) Andy Howard, Phil Cheeseman, John Campbell, David Braun, Preston Smith and Carol Song. The team is posing with images from the scientific research enabled by DiaGrid, projected in a multi-walled virtual environment at ITaP’s visualization facility.

Writer: Greg Kline, science and technology writer, ITaP, (765) 494-8167, gkline@purdue.edu

Sources: Gerry McCartney, (765) 496-2270, mccart@purdue.edu
John Campbell, (765) 494-1289, john-campbell@purdue.edu

DXUL/Fortress archival storage system unavailable Thursday, 9/17

rcac@purdue.edu — Fri, 18 Sep 2009 09:38:00 PST

The DXUL/Fortress system's ADIC Scalar 10K tape libary will be unavailable from 8am-noon on Thursday, September 17, for routine maintenance and firmware updates for its LTO tape drives. During this time, it will not be possible to stage files from tape into the DXUL/Fortress disk cache for access.
Please refer questions about this service interruption to rcac-help@purdue.edu.

RCAC system maintenance scheduled for October 9-13

rcac@purdue.edu — Mon, 19 Oct 2009 16:33:00 PST

RCAC systems will be unavailable from 8am-6pm Friday, October 9, for electrical work in the MATH data center and system maintenance. The Coates Linux cluster will not be returned to service until Tuesday afternoon, October 13, so RCAC staff can conduct whole-cluster performance benchmarking runs.
Only those PBS jobs will walltimes short enough that they will complete prior to 8am Friday, October 9, will be scheduled for execution -- jobs with longer walltimes will be held in the queue until the systems have been returned to production after the maintenance.
Please refer questions about this maintenance period to rcac-help@purdue.edu.

DXUL/Fortress archival storage system unavailable Monday, 9/14

rcac@purdue.edu — Mon, 14 Sep 2009 13:37:00 PST

The DXUL/Fortress archival storage system became unavailable Monday morning, September 14, due to a problem with its internal diskserver. RCAC and DXUL vendor storage engineers are working to restore the system to service.
Update: The DXUL/Fortress archival storage system was restored to service at 1:30pm ET on Monday, September 14.

Scratch storage outage Thursday, 9/10

rcac@purdue.edu — Mon, 14 Sep 2009 11:38:00 PST

Access to scratch storage on RCAC research systems became unavailable at approximately 3pm ET on Thursday, Septemeber 10. RCAC and Blue Arc storage engineers are working to correct the problem and return the scratch storage system to normal operation as soon as possible.
Update: Scratch storage on RCAC systems was restored to service at 3:50pm ET Thursday, September 10.

Radon cluster unavailable Tuesday-Wednesday, 8/25-26

rcac@purdue.edu — Wed, 26 Aug 2009 21:38:00 PST

The Radon Linux cluster will be unavailable Tuesday and Wednesday, August 25-26, for hardware upgrades and a conversion from Debian Linux to Red Hat Enterprise Linux 5 (RHEL5). Since this upgrade changes the version of Linux used on the cluster, all user programs should be recompiled after the upgrade and any PBS jobs that do not complete prior to the upgrade will be deleted and need to be resubmitted.
Update: The Radon cluster's upgrade to Red Hat Enterprise Linux 5 (RHEL5) has been completed and PBS job scheduling was resumed Wednesday evening, August 26. Although some applications compiled prior to the upgrade may continue to work, Radon users are encouraged to recompile their applications for best results.
Please refer questions about this system maintenance to rcac-help@purdue.edu.

Steele job scheduling temporarily paused Monday, 8/10

rcac@purdue.edu — Tue, 25 Aug 2009 13:36:00 PST

PBS job scheduling on the Steele cluster has been paused so that a Linux kernel update can be applied to the cluster's compute nodes. Steele users may continue to submit jobs but they will not be scheduled for execution until after job scheduling is resumed.
Update: As of 6pm ET Monday, August 10, approximately half of the Steele cluster's 903 compute nodes had received the Linux kernel update, and PBS job scheduling has resumed. The remaining nodes will be updated and made available for job scheduling as the PBS jobs running on them complete.
Please refer questions and/or comments to rcac-help@purdue.edu.

Submission to Steele standby queues temporarily disabled Monday, 7/28

rcac@purdue.edu — Tue, 04 Aug 2009 07:44:00 PST

Job submission to the Steele cluster's standby and standby-8 queues has been temporarily disabled due to the extremely large number of jobs that have been submitted to the cluster recently. Submission to these queues will be reenabled when the number of jobs in the system has dropped to a more manageable level.
Please refer questions or comments about this temporary scheduling change to rcac-help@purdue.edu

Steele Linux cluster unavailable Tuesday-Wednesday, 8/4-5

rcac@purdue.edu — Mon, 10 Aug 2009 12:15:00 PST

The Steele Linux cluster will be unavailable from 8am Tuesday, August 4, through 6pm Wednesday, August 5, for an operating system upgrade from Red Hat Enterprise Linux 4 to Red Hat Enterprise Linux 5. Only those PBS jobs with walltimes short enough that they will finish prior to the beginning of this maintenance period are being scheduled for execution -- jobs with longer walltimes will be held in the input queue until the operating system upgrade has been completed.
Update: The Steele cluster's upgrade to Red Hat Enterprise Linux 5 (RHEL5) has been completed and PBS job scheduling was resumed at 6pm Wednesday, August 5. Although many applications compiled prior to the upgrade will continue to work, Steele users are encouraged to recompile their applications for best results.
Please refer questions about the upgrade and/or problem reports to rcac-help@purdue.edu.

Steele Linux Cluster software upgrade postponed

rcac@purdue.edu — Tue, 14 Jul 2009 16:07:00 PST

The Steele Linux cluster system software upgrade that was scheduled for Tuesday-Thursday, July 14-16, has been postponed due to NFS automount problems detected by RCAC staff during pre-production testing. New dates for the software upgrade will be announced after RCAC staff have confirmed the NFS automount problems have been resolved.
The Steele cluster will be unavailable from 8am-11am Tuesday, July 14, while firmware upgrades are applied to the RCAC home and scratch BlueArc file servers. Only those PBS jobs with walltimes short enough that they will finish prior to the beginning of this maintenance period are being scheduled for execution -- jobs with longer walltimes will be held in the input queue until the BlueArc maintenance has been completed.

Steele Linux Cluster unavailable Tuesday-Thursday, 7/14-16

rcac@purdue.edu — Sun, 12 Jul 2009 20:52:00 PST

The Steele Linux cluster will be unavailable from 8am Tuesday, July 14, through 6pm Thursday, July 16, for an operating system upgrade from Red Hat Enterprise Linux 4 to Red Hat Enterprise Linux 5. Only those PBS jobs with walltimes short enough that they will finish prior to the beginning of this maintenance period are being scheduled for execution -- jobs with longer walltimes will be held in the input queue until the operating system upgrade has been completed.

RCAC Home and Scratch directories unavailable Tuesday, 7/14

rcac@purdue.edu — Thu, 16 Jul 2009 07:29:00 PST

RCAC home and scratch directories will be unavailable from 8am to 11am Tuesday, July 14, while firmware upgrades are made on RCAC's BlueArc storage systems. Only those PBS jobs with walltimes short enough that they will finish prior to the beginning of this maintenance period are being scheduled for execution -- jobs with longer walltimes will be held in the input queue until the BlueArc maintenance has been completed.

DXUL/fortress tape library out of service, Friday, June 5

rcac@purdue.edu — Sun, 07 Jun 2009 16:25:00 PST

The robotic tape library used by the DXUL/fortress archival storage system is experiencing hardware problems and is unable to mount tapes. DXUL remains in operation and files in the archival storage system's disk cache can be accessed normally, but it will not be possible to retrieve files from or store new files in the archive until the library has been repaired.
Please refer questions to "rcac-help@purdue.edu".
Update: The tape library was repaired and the DXUL/fortress archival storage system returned to normal operation at 4:30pm Friday, June 5.

Steele "C" nodes unavailable Friday, May 29

rcac@purdue.edu — Sun, 07 Jun 2009 16:27:00 PST

The 48 Steele "C" nodes, which have 32GB of memory and Gigabit Ethernet, will be unavailable from 8am to noon Friday, may 29, while a problematic network switch that connects them to the RCAC research network is replaced. Only those PBS jobs with walltimes short enough that they will finish prior to 8am Friday morning are being scheduled on these nodes.

SiCortext system unavailable Wednesday, 5/27

rcac@purdue.edu — Sun, 07 Jun 2009 16:27:00 PST

The SiCortex system, moffett.rcac.purdue.edu, will be unavailable from 8am-5pm Wednesday, May, while its system software is upgraded.

RCAC system and data center maintenance 3/17-19

rcac@purdue.edu — Mon, 23 Mar 2009 10:41:00 PST

RCAC systems including the DXUL/fortress archival storage system will be unavailable beginning at 8am Tuesday, 3/17, while system and MATH data center maintenance is performed. All systems are expected to be back in service by 6pm Thursday, 3/19, and updates on the status of individual systems will be shown here throughout the maintenance period.
Final update: All RCAC systems were back in production as of 1:45pm Friday, 3/20. Please refer any questions about this maintenance window to rcac-help@purdue.edu.

System	Expected Return To Service	Status
CMS/gluon	Wed, 3/18, noon	Available Tue, 3/17, 10pm
Condor	Wed, 3/18, noon	Available Wed, 3/18, 10:30am
Account Request System	Wed, 3/18, 6pm	Available Tue, 3/17, 10pm
Beaver	Wed, 3/18, 6pm	Available Wed, 3/18, 6pm
Gray	Wed, 3/18, 6pm	Available Wed, 3/18, 8pm
DXUL/fortress	Wed, 3/18, 6pm	Available Thu, 3/19, 2:30am
Moffett	Wed, 3/18, 6pm	Available Thu, 3/19, 10:30pm
Gold	Wed, 3/18, 6pm	Available Thu, 3/19, 9am
Pete	Wed, 3/18, 6pm	Available Thu, 3/19, 9am
Prospero	Wed, 3/18, 6pm	Available Thu, 3/19, 9am
Radon	Wed, 3/18, 6pm	Available Thu, 3/19, 9am
Rossmann	Wed, 3/18, 6pm	Available Thu, 3/19, 9am
Venice	Wed, 3/18, 6pm	Available Thu, 3/19, 9am
Curator/PiiMS	Thu, 3/19, noon	Available Wed, 3/18, 10:30am
Steele	Thu, 3/19, noon	Available Fri, 3/20, 10:30am
Julius	Thu, 3/19, noon	Available Fri, 3/20, 1:45pm

Unscheduled power outage in MATH Tuesday, 2/10

rcac@purdue.edu — Thu, 12 Feb 2009 20:31:00 PST

RCAC experienced an unscheduled power failure in its MATH data center at approximately 8:20pm Tuesday, February 10.

The Pete, Steele, and Radon Linux clusters, and the Julius/Caesar SGI Altix 4700 were affected and went out of service.

The Rossmann, Prospero, and Venice Linux clusters, the Condor servers, the CMS systems in MANN Hall, the Moffett SiCortex system, and the RCAC storage servers including the DXUL/fortress archival storage system remained in operation.

As of 9:15pm, electricians and RCAC systems staff were on-site in MATH but were unable to provide an estimate of when power will be restored so the systems could be returned to service.

The next update on this outage is scheduled for 12 midnight Tuesday, February 10.

Update as of midnight Tuesday, February 10: Electricians restored power in the MATH data center at approximately 10:35pm, at which point RCAC system engineers started rebooting the affected systems. Batch job scheduling was resumed on a cluster-by-cluster basis starting at 11:15pm and all systems were back in service by 11:45pm.

We regret any inconvenience caused by this outage, and will be conducting a review to determine its cause and explore ways to mitigate the impact of future power-related problems.

PBS job scheduling temporarily suspended Friday, 1/30

rcac@purdue.edu — Fri, 30 Jan 2009 23:41:00 PST

PBS job scheduling on RCAC systems was suspended from 11-11:30pm Friday, 1/30, while problems with access to home directories was diagnosed and corrected by RCAC system engineers.

SiCortex system unavailable Wednesday-Thursday, 1/21-22

rcac@purdue.edu — Tue, 27 Jan 2009 07:44:00 PST

The SiCortex system, moffett.rcac.purdue.edu, will be unavailable from 8am Wednesday, January 21, through 5pm Thursday, January 22, while hardware upgrades are performed.
Update as of 5:45 PM Thursday, 1/22: Diagnostics following this week's SiCortex maintenance indicated possible hardware problems, which RCAC and SiCortex systems engineers are working to resolve. As a result, the SiCortex system remains out of service. The next update on its status will be made at 10PM Thursday evening, 1/22.
Update as of 10:00 PM Thursday, 1/22: The previously mentioned diagnostics indicate the presense of multiple memory modules that need to be replaced. Replacement modules are being acquired but the system will be out of service until they arrive and have been installed and tested. The next update on the system's status will be made at 10AM Friday morning, 1/23.
Update as of 10:00 AM Friday, 1/23: The SiCortex remains out of service while SiCortex engineers continue to run hardware diagnostics. The next update on the systems status is scheduled for 3PM Friday, 1/23.
Update as of 3pm Friday, 1/23: We are continuing to run diagnostics on the SiCortex system. The next update on its status is scheduled for 10PM Friday, 1/23.
Update as of 6PM Friday, 1/23: The SiCortex system, moffett.rcac.purdue.edu, has been returned to service after the hardware upgrades that were made on January 21-23. A few hardware-related issues remain, however, and will be addressed during the system's next maintenance window. Please report any problems you might encounter via e-mail to "rcac-help@purdue.edu".

SGI Altix system maintenance Tuesday, January 13

rcac@purdue.edu — Tue, 13 Jan 2009 14:08:00 PST

The SGI Altix 4700 system, caesar.rcac.purdue.edu, will be unavailable from 10-11am Tuesday, January 13, while a memory module is replaced. Only those PBS jobs with walltimes short enough that they will finish prior to this maintenance are being scheduled for execution -- jobs with longer walltimes will be held in the queue until the system has been returned to service following this maintenance.

RCAC system maintenance scheduled for January 6-7

rcac@purdue.edu — Thu, 08 Jan 2009 13:49:00 PST

RCAC systems will be unavailable from 8am-5pm on Tuesday, January 6, for system maintenance. The Pete Linux cluster will not be returned to service until Wednesday afternoon, January 7, so RCAC systems staff and technical staff from the PBSPro vendor can work to diagnose and correct problems with its PBS server installation.
Only those PBS jobs will walltimes short enough that they will complete prior to 8am Tuesday, January 6, will be scheduled for execution -- jobs with longer walltimes will be held in the queue until the systems have been returned to production after the maintenance.

SiCortex system unavailable Thursday-Friday, 12/11-12

rcac@purdue.edu — Mon, 15 Dec 2008 08:20:00 PST

The SiCortex system, moffett.rcac.purdue.edu, will be unavailable from noon Thursday, December 11, through late afternoon Friday, December 12, while its software is updated to release level 3.0. Release notes for this updated software version , which corrects several known problems and includes some new features, may be found on the SiCortex web site.

DXUL/fortress system maintenance scheduled for Thursday, Nov 13

rcac@purdue.edu — Mon, 17 Nov 2008 11:46:00 PST

The Fortress/DXUL tape robot system will be undergoing hardware maintenance from 10:00am-4:00pm Thursday, November 13. During this time, it will not be possible to retrieve files from the tape archive nor will it be possible for the DXUL system to migrate files from its disk cache into the tape library. RCAC storage engineers will monitor the disk cache and disable all DXUL access if the cache fills during this maintenance window.

SGI Altix system unavailable Wednesday-Thursday, 11/5-6

rcac@purdue.edu — Mon, 17 Nov 2008 11:46:00 PST

The SGI Altix 4700 system, caesar.rcac.purdue.edu, will be unavailable from 8am Wednesday morning, November 5, until late afternoon Thursday, November 6, so SGI and Purdue system engineers can apply firmware upgrades to the system's /scratch/caesar file system and conduct tests to evaluate its stability. The results of those tests will determine whether or not the file system will be made available Thursday afternoon when caesar.rcac.purdue.edu is returned to service.
Only those PBS jobs with walltimes less than the time remaining until the beginning of this maintenance period Wednesday morning, November 5, will be scheduled for execution. Jobs with longer walltimes will be held in the queue for execution after the system is returned to service Thursday afternoon.

Caesar PBS Scheduling and Scratch Down, Oct. 21-22

rcac@purdue.edu — Wed, 22 Oct 2008 19:26:00 PST

PBS scheduling on Caesar was been stopped in the evening of October 21, and the scratch filesystem /scratch/caesar/ is unavailable. Engineers are working on the problem with the vendor, but do not yet have an estimate of when PBS will return to service.
PBS job scheduling on the Caesar SGI Altix 4700 was restarted at approaimately 4:00P.M. Wednesday,October 23. However, its /scratch/caesar file system has been removed from service due to hardware problems. In addition, the system's /tmp file system has been reduced in size to 10GB. Please use your RCAC scratch directory as named in your $RCAC_SCRATCH environment variable instead.
Questions about this outage and system change should be referred to "rcac-help@purdue.edu".

Steele PBS Scheduling Down, Sept. 11-12

rcac@purdue.edu — Fri, 12 Sep 2008 18:27:00 PST

PBS scheduling of any new jobs on Steele stopped working around 11:30pm on Sept. 11 and is still down now. Engineers are working on the problem with the vendor, but do not yet have an estimate of when PBS will return to service.
Update on 9/12, 4:30pm: Our engineers are still in contact with the PBS vendor, but the vendor has not been able to resolve the problem yet. We will post another update at noon tomorrow.
Update on 9/12, 6:25pm: Steele PBS is back in operation now. All jobs should be resuming normally and new submissions should work as well. Sorry for any inconvenience this may have caused. If you see any problems, please let us know by emailing rcac-help@purdue.edu.

Steele-A Partial Power Outage, Sept. 12

rcac@purdue.edu — Fri, 12 Sep 2008 16:35:00 PST

Numerous Steele-A nodes were unavailable due to a problem with a power distribution unit (PDU) in the Math data center from sometime early this morning until 11:00am. Electricians were able to restore power at that time, but PBS scheduling is still not working. That issue is being tracked in another system news item.

Fortress/DXUL Emergency Follow-up Maintenance Wednesday, September 10

rcac@purdue.edu — Thu, 11 Sep 2008 14:36:00 PST

The Fortress/DXUL tape robot system will be undergoing emergency hardware maintenance from 8:00am-5:00pm on Wednesday, September 10. This maintenance is a follow-up to the work done August 25-26. During this time, it will not be possible to retrieve files from the tape archive nor will it be possible for the DXUL system to migrate files from its disk cache into the tape library. RCAC storage engineers will monitor the disk cache and disable all DXUL access if the cache fills during this maintenance window. Fortress/DXUL users are asked to please keep their use of the system to a minimum during this time.

Pete cluster temporarily unavailable Tuesday, September 2

rcac@purdue.edu — Wed, 03 Sep 2008 22:34:00 PST

The pete.rcac back-end server for the Pete cluster hung earlier today, Tuesday, September 2. As a result, pete.rcac had to be rebooted and PBS jobs executing at the time had to be requeued.

Part of the process of restarting the PBS server involves its reestablishing communication with the PBS software running on each compute node. Under normal circumstances, reestablishing a cluster's nodes should take 15-20 minutes.

However, this operation is taking much longer than that on the Pete cluster -- our current estimate is that it will complete at approximately 5pm, at which point PBS job scheduling will be reenabled. We have escalated this issue to our software vendor, and are working toward a better turnaround in this sort of situation.

Update: PBS job scheduling on the Pete cluster was reenabled at 5pm Tuesday, September 2.

Please refer questions about this outage to rcac-help@purdue.edu.

SGI Altix system maintenance Tuesday, September 2

rcac@purdue.edu — Wed, 03 Sep 2008 22:13:00 PST

The four RCAC SGI Altix systems (julius, caesar, octavius, and brutus.rcac.purdue.edu) will be unavailable from 8am-5pm Tuesday, 9/2, while operating system upgrades are installed.
Update as of 5pm Tuesday, September 2: RCAC systems engineers have encountered problems updating the Altix systems and are waiting for a response from SGI technical support. The Altix systems will remain unavailable until SGI can address the problems we have encountered with the system updates.
Update as of 3pm Wednesday, September 3: RCAC system engineers have returned the SGI Altix systems to service and PBS job scheduling on the Altix 4700, caesar.rcac.purdue.edu, has been re-enabled.
Please refer questions about this outage to rcac-help@purdue.edu.

Emergency maintenance for RCAC Linux systems Wed-Fri, August 27-29

rcac@purdue.edu — Mon, 01 Sep 2008 11:37:00 PST

RCAC will be performing emergency maintenance on all its Linux systems Wednesday-Friday, August 27-29. This work includes the application of patches, InfiniBand upgrades, and downtime for system reboots as described in the schedule below. Job scheduling reservations are in place to prevent longer jobs from starting prior to each cluster's reboot time. When possible, currently-executing jobs will be requeued and restarted after cluster nodes have been returned to service.
Please refer questions and comments about this maintenance to rcac-help@purdue.edu.

Resource	Downtime	Notes	Status
Gray	Tue, 08/26, noon-5pm		Completed 5pm Wed, 8/27
CMS	Wed, 08/27, 3-8am		Completed 7:30am Wed, 8/27
Radon	Wed, 08/27, noon-5pm		Completed 5:30pm Wed, 8/27
Steele	Wed, 08/27, noon-5pm	1	Completed 10:20pm Wed, 8/27
Venice	Wed, 08/27, noon-5pm		Completed 2:30pm Thu, 8/28
Julius/SGI Altix	Thu, 08/28, 8am-6pm		Completed 5:00pm Thu, 8/28
Pete	Fri, 08/29, 8am-5pm	2	Completed 12:30pm Sat, 8/30
Prospero	Fri, 08/29, 8am-5pm	1	Completed 7:30pm Fri, 8/29
Rossmann	Fri, 08/29, 8am-5pm	1	Completed 7:30pm Fri, 8/29
1) It is recommended that applications using InfiniBand message-passing libraries be recompiled following the maintenance
2) Due to complications with the integration of SFS/Lustre with a new Linux kernel installed as part of the system maintenance, the contents of /scratch/lustre1 have been copied onto an RCAC BlueArc file server, where they will be available for use on the Pete cluster while RCAC works with HP to address the SFS/Lustre upgrade issues.

Resolved: Emergency DXUL/fortress maintenance Tuesday, August 26

rcac@purdue.edu — Tue, 26 Aug 2008 12:57:00 PST

Emergency hardware maintenance has been scheduled for the tape robot used by the DXUL/fortress archival storage system from 10:30am-12:30pm on Tuesday, August 26. This maintenance is in follow-up to the work done on Monday, August 25. During this time, it will not be possible to retrieve files from the tape archive nor will it be possible for the DXUL system to migrate files from its disk cache into the tape library. RCAC storage engineers will monitor the disk cache and disable all DXUL access if the cache fills during this maintenance window. DXUL users are encouraged to keep their use of the system to a minimum during this period.
Update as of 1pm Tuesday, August 26: Maintenance on the tape robot used by the DXUL/fortress system has been completed and the system has been returned to service.
Please refer questions about this maintenance activity to rcac-help@purdue.edu.

Resolved: Emergency DXUL/fortress maintenance Monday, August 25

rcac@purdue.edu — Mon, 25 Aug 2008 16:56:00 PST

Emergency hardware maintenance has been scheduled for the tape robot used by the DXUL/fortress archival storage system from 6:30am-noon on Monday, August 25. During this time, it will not be possible to retrieve files from the tape archive nor will it be possible for the DXUL system to migrate files from its disk cache into the tape library. RCAC storage engineers will monitor the disk cache and disable all DXUL access if the cache fills during this maintenance window. DXUL users are encouraged to keep their use of the system to a minimum during this period.
Update as of 9am Monday, August 25: The DXUL cache has filled, so DXUL access has been disabled until maintenance on the tape robot has been completed and migration of files from the disk cache into the tape library has resumed. The DXUL/fortress system is expected to be returned to service by 1pm.
Update as of 4:15pm Monday, August 25: Maintenance on the tape robot used by the DXUL/fortress system has been completed and the system has been returned to service.
Please refer questions about this maintenance activity to rcac-help@purdue.edu.

DXUL/fortress maintenance scheduled for Thursday, August 21

rcac@purdue.edu — Thu, 21 Aug 2008 20:30:00 PST

The DXUL/fortress archival storage system will be unavailable from 6:30-9:00am Thursday, August 21, while hardware maintenance to upgrade its Fibre Channel switches is performed.

Resolved; Unscheduled outage of CMS equipment in MANN, Monday August 18

rcac@purdue.edu — Mon, 18 Aug 2008 10:09:00 PST

A chilled water loss has occured at the MANN datacenter housing CMS Tier-2 equipment and all dedicated CMS systems housed there are unavailable until cooling has been restored.
Update -- 3pm Monday, August 18: The chilled water supply to MANN Hall has been restored and the CMS systems housed there were returned to production at 3pm, August 18.
Please refer questions about this outage to "rcac-help@purdue.edu".

Resolved: Unscheduled SGI Altix outage Thursday, August 14

rcac@purdue.edu — Thu, 14 Aug 2008 10:09:00 PST

The SGI Altix 4700, caesar.rcac, was unavailable for several hours on Thursday, August 14, due to hardware problems. RCAC systems engineers diagnosed and corrected the problems, and the system was returned to production at approximately 10:45am.

Please refer questions about this service interruption to "rcac-help@purdue.edu".

SGI Altix system unavailable Thursday, August 28

rcac@purdue.edu — Mon, 01 Sep 2008 11:38:00 PST

The Julius/Caesar SGI Altix 4700 system will be unavailable from 8am-5pm on Thursday, August 28, while its job scheduler is upgraded from PBSPro version 8.0 to version 9.1. A scheduling reservation has been installed and only those PBS jobs with walltimes less than the time remaining until 8am Thursday, August 28, will be scheduled for execution. All other jobs will be held for execution after the software upgrade has been completed.
Please refer questions about this service interruption to rcac-help@purdue.edu.

Radon Linux cluster unavailable Thursday, August 14

rcac@purdue.edu — Mon, 04 Aug 2008 22:38:00 PST

The Radon Linux cluster will be unavailable from 8am-5pm on Thursday, August 14, while its job scheduler is upgraded from PBSPro version 8.0 to version 9.1. A scheduling reservation has been installed and only those PBS jobs with walltimes less than the time remaining until 8am Thursday, August 14, will be scheduled for execution. All other jobs will be held for execution after the software upgrade has been completed.
Please refer questions about this service interruption to "rcac-help@purdue.edu".

Gold IBM AIX cluster to be rebooted Tuesday, August 5

rcac@purdue.edu — Mon, 04 Aug 2008 22:30:00 PST

Systems in the Gold IBM cluster will will be unavailable between noon-1pm on Tuesday, August 5, while they are rebooted to complete the installation of AIX system-software updates.

Resolved: Unscheduled DXUL archival storage system outage Wednesday, July 30

rcac@purdue.edu — Wed, 30 Jul 2008 11:55:00 PST

The DXUL/fortress.rcac.purdue.edu archival storage system was unavailable from approximately 3am-11:45am Wednesday, July 30, due to problems with its internal databases. RCAC storage engineers corrected those problems and the system appears to be operating normally.
Please refer questions or comments about this outage to rcac-help@purdue.edu.

SiCortex system to be rebooted Wednesday, July 16

rcac@purdue.edu — Wed, 16 Jul 2008 10:08:00 PST

The SiCortex system, moffett.rcac, will be rebooted between 2-3pm Wednesday, July 16, to allow several system changes to take effect. Jobs executing at that time will be terminated and the owners of currently-executing jobs have been notified of the reboot.

Resolved: Power outage affects CMS-related systems in MANN 112 Monday, July 14

rcac@purdue.edu — Mon, 14 Jul 2008 12:30:00 PST

CMS-related systems housed in MANN 112 are out of service due to an unplanned power outage that occurred shortly after 11am on Monday, July 14, Updates on the status of the systems will be posted here as additional information about the outage becomes available.
5:30pm July 14 Update: Power was restored to MANN hall and the CMS-related systems housed in MANN 112 were returned to service at approximately 2:30pm Monday, July 14.

Resolved: Omni.cc unavailable due to disk drive failure Thursday, July 10

rcac@purdue.edu — Thu, 10 Jul 2008 21:27:00 PST

Omni.cc.purdue.edu suffered a hardware failure on its system boot disk drive Thursday, July 10. RCAC system engineers have located a replacement, and are working to replace the failed drive and restore the system to operation.
7PM Friday, July 11 update: Omni.cc was returned to service at approximately 6:30PM Friday, July 11.
As of 9pm Thursday, July 10, our estimate is that the system will be back in service by early evening Friday, July 11. An update will be posted here if problems arise that will delay its return to service.
Important note for omni.cc users: Due to its age, which significantly complicates system maintenance and repair, and its limited use in recent years, omni.cc.purdue.edu will be retired and removed from service on Friday, October 10. Please contact rcac-help@purdue.edu if you have questions or need assistance in migrating the work you do on omni.cc to another system.

System maintenance scheduled for gray.rcac Thursday, July 10

rcac@purdue.edu — Tue, 08 Jul 2008 13:19:00 PST

The gray.rcac.purdue.edu system that serves as a front-end to the IBM e1350 BladeCenter in Bloomington will be unavailable from noon-5pm Thursday, July 10, while system maintenance is performed.
Please refer questions about this service interruption to "rcac-help@purdue.edu".

Radon.rcac to be rebooted Tuesday afternoon, July 8

rcac@purdue.edu — Tue, 01 Jul 2008 20:20:00 PST

The radon.rcac.purdue.edu Linux cluster front-end system will unavailable for a few minutes starting at 2PM Tuesday, July 8, so it can be rebooted to correct a problem related to running X Window applications such as Matlab. Non-interactive PBS jobs and Condor jobs should be unaffected by the reboot.
Please refer questions about this service interruption to "rcac-help@purdue.edu".

Resolved: RCAC system maintenance scheduled for July 1

rcac@purdue.edu — Mon, 30 Jun 2008 14:50:00 PST

RCAC systems including the Linux compute clusters, the SGI Altix, the IBM SP and Regatta systems, and the Sun F6800 systems will be out of service from 8am-midnight Tuesday, July 1, for hardware and software maintenance. Only those PBS jobs with walltimes short enough that they would complete prior to this maintenance period will be scheduled for execution -- jobs with longer walltimes will remain in the queue until after the maintenance has been completed.
10:00am July 3: The Pete and Steele clusters' PBS job schedulers appear to be operating normally. We will continue to closely monitor their behavior and are working with the PBS vendor to determine and correct the cause of the instabilities observed on July 1-2.
3:30pm July 2 update: The PBS job scheduler on the Pete cluster appears to be stable but the PBS scheduler on the Steele cluster remains unstable. While Steele jobs are being scheduled properly much of the time, frequent PBS crashes may temporarily interfere with job submission or the receipt of output from recently-completed jobs. We continue to closely monitor both the Pete and Steele clusters and have contacted the PBS vendor for further support. Updates will be posted as additional information becomes available.
10:30am July 2 update: There are problems with the Steele cluster's PBS job scheduler which are causing the scheduler to abort. RCAC system engineers are working to resolve the problems and are in contact with the PBS vendor's tech support staff.
12am July 2 update: The maintenance scheduled for Tuesday, July 1, has been completed. RCAC systems have been returned to service, but a problem has been observed with the Pete cluster's PBS job scheduler in that nodes frequently falsely report themselves as offline to the scheduler. RCAC systems engineers are working on the problem and will be contacting the PBS vendor's tech support staff Wednesday morning, July 2.

Resolved: DXUL/fortress system unavailable as of Wednesday, June 18

rcac@purdue.edu — Wed, 18 Jun 2008 10:32:00 PST

The DXUL/fortress.rcac archival storage system is unavailable effective 10AM Wednesday, June 18, due to a mechanical failure in its robotic tape library. Replacement parts and service personnel are en route and we estimate it will remain out of service until 5PM Thursday, June 19.
3:15 PM Thursday June 19: The DXUl/fortress.rcac archival storage system was returned to service at approximately 3 PM Thursday, June 19, following repairs to its robotic tape library. Please refer questions about this outage to rcac-help@purdue.edu.

Resolved: DXUL/fortress system unavailable Monday, June 16

rcac@purdue.edu — Mon, 16 Jun 2008 12:27:00 PST

The DXUl/fortress.rcac archival storage system is unavailable due to a mechanical failure in its robotic tape library. Replacement parts and service personnel are en route but we currently estimate it will remain out of service until 5PM Tuesday, June 17.
3PM Monday, June 16: The DXUl/fortress.rcac archival storage system was returned to service at approximately 2:15PM Monday, 6/16, following repairs to its robotic tape library.
Please refer questions about this outage to rcac-help@purdue.edu.

Resolved: Unscheduled service interruption Friday, June 13

rcac@purdue.edu — Fri, 13 Jun 2008 17:40:00 PST

RCAC systems are currently out of service due to file server network interface problems that developed at 3:20PM Friday, June 13. RCAC system engineers are working to correct the problem but are currently unable to estimate when normal service will be restored. The next update on this outage will be at 9PM Friday, June 13.
9PM 6/13 Update: The network interface problems that were preventing access to home directories on RCAC systems have been resolved, and all systems were restored to normal operation at approximately 8:45PM Friday, June 13.

Resolved: Unscheduled RCAC scratch fileserver outage Wednesday, June 11

rcac@purdue.edu — Wed, 11 Jun 2008 14:17:00 PST

The RCAC scratch file systems are currently unavailable due to problems with one of the BlueArc file servers. As a result, PBS and Condor job scheduling on all RCAC systems has been halted pending resolution of the problem. A BlueArc technician is on-site working on the problem, which occurred following maintenance on one of the redundant BlueArc servers.

2:30PM Friday, 6/13: The maintenance on the RCAC scratch file servers was completed and normal PBS and Condor job scheduling was enabled at 2PM Friday, 6/13.

4PM 6/12 Update: The RCAC scratch file systems were returned to service at 3:30PM Thursday afternoon, 6/12. PBS and Condor job scheduling has been re-enabled, but some additional file server maintenance remains to be done and has been scheduled for 1-2PM Friday, 6/13. As a precaution, a PBS reservation has been installed such that only those jobs with walltimes short enough to finish prior to 1PM Friday will be scheduled for execution. Jobs with longer walltimes will be held in the queue until after the the maintenance has been completed. The next update on this work will be posted at 2:30PM Friday, 6/13.

12noon 6/12 Update: One of the scratch file systems, scratch95, became unavailable shortly before 12noon, so PBS and Condor job scheduling have been halted until the installation of the replacement BlueArc hardware has been completed and the problems can be resolved. The next update on this outage will be posted at 4pm, June 12.

9AM 6/12 Update: The scratch file systems have been operating normally since 1:30AM Thursday morning, June 12. A replacement for hardware suspected of causing Wednesday's outage is expected to arrive and be installed later Thursday morning. The next update on this outage will be posted at 12noon, June 12.

1:45AM 6/12 Update: The scratch file systems have been returned to service and both PBS and Condor job scheduling have resumed. A replacement for a hardware component believed to have caused the problem is being express-shipped to Purdue for installation Thursday morning, 6/12.

9PM 6/11 Update: RCAC system engineers and a BlueArc technician continue to work on the problem but are unable to
estimate when the file systems will be returned to service. The next update on their status will be posted at 9AM, June 12.

5PM 6/11 Update: System engineers continue to work on the scratch file server problem but are unable to estimate when it will be resolved. The next update will be posted at 9PM.

3PM 6/11 Update: RCAC and BlueArc system engineers continue to work on the problem, but we are unable to estimate when the file systems will be returned to service. The next update on their status will be posted at 5PM.

Please refer questions about this outage to "rcac-help@purdue.edu".

Pete Linux cluster reboots scheduled for Thursday, June 5

rcac@purdue.edu — Wed, 04 Jun 2008 12:02:00 PST

In order to install an updated version of the Pete Linux cluster's Lustre/SFS client software, its compute nodes and front-end/login hosts will be rebooted between 2:30-3:30 PM on Thursday, June 6. To reduce the impact these reboots have on executing PBS jobs, a scheduling reservation has been installed for this time period and PBS jobs requesting more walltime than remains prior to 2:30 PM Thursday will be held in the queue until after the systems have been rebooted and returned to service.
Please refer questions or comments about this system maintenance to rcac-help@purdue.edu.

DXUL/fortress system maintenance scheduled for Thursday, May 29

rcac@purdue.edu — Tue, 27 May 2008 09:03:00 PST

Access to the DXUL/fortress.rcac.purdue.edu archival storage system will be unavailable from 8am-5pm Thursday, May 29, while system maintenance is performed to update software, install a new tape drive, and update firmware on existing tape drives.
Please refer questions or comments about this scheduled outage to "rcac-help@purdue.edu".

Resolved: Unscheduled DXUL/fortress outage Monday, May 26

rcac@purdue.edu — Mon, 26 May 2008 14:57:00 PST

The DXUL/fortress.rcac.purdue.edu was unavailable for approximately three hours on Monday, May 26, due to a recurrence of a problem first observed on Saturday, May 24. RCAC and EMC/DXUL storage engineers were able to return the system to service at approximately 4:30 pm and are working to diagnose and correct the problem.

Please refer questions or comments about this outage to "rcac-help@purdue.edu".

Resolved: Unscheduled DXUL/fortress outage Sunday, May 25

rcac@purdue.edu — Sun, 25 May 2008 11:12:00 PST

The DXUL/fortress.rcac.purdue.edu was unavailable for several hours on Sunday, May 25, due to a recurrence of a problem first observed on Saturday, May 24. RCAC and EMC/DXUL storage engineers were able to return the system to service at approximately 4:00 pm.
Please refer questions or comments about this outage to rcac-help@purdue.edu.

Resolved: Unscheduled DXUL/fortress outage Saturday, May 24

rcac@purdue.edu — Sat, 24 May 2008 19:26:00 PST

The DXUL/fortress.rcac.purdue.edu archival storage system was unavailable for several hours on Saturday, May 24, due to system problems. RCAC system and storage engineers corrected the problems and returned the system to operation at approximately 8pm.
Please refer questions or comments about this outage to rcac-help@purdue.edu.

Final update: Unscheduled Steele cluster outage Thursday, May 15-16

rcac@purdue.edu — Thu, 15 May 2008 18:21:00 PST

The Steele cluster has been returned to production and PBS job scheduling was re-enabled at approximately 3:45 PM Friday, May 16. Several PBS jobs were lost during the cluster's network problems and will need to be resubmitted by their owners.

We will continue to closely monitor the cluster's network and anticipate needing to schedule and announce an outage for network maintenance and tuning in the relatively near future.

Thank you for your patience during this outage -- please refer any questions or comments about it to rcac-help@purdue.edu.

SGI Altix 4700 hardware maintenance scheduled for Wednesday, May 14

rcac@purdue.edu — Thu, 08 May 2008 07:24:00 PST

The SGI Altix 4700 will be unavailable from 8AM-noon Wednesday, May 14, while hardware maintenance is performed. Access to the julius.rcac.purdue.edu front-end system should not be affected. Only those PBS jobs with walltimes short enough to complete prior to this maintenance period will be scheduled for execution -- jobs with longer walltimes will remain in the queue until after the maintenance has been completed.

IBM SP hardware maintenance scheduled for Wednesday, May 14

rcac@purdue.edu — Thu, 08 May 2008 07:23:00 PST

The IBM SP will be unavailable from 8AM-noon on Wednesday, May 14, while hardware maintenance is performed. Access to the cloud.cc.purdue.edu front-end system should not be affected. Only those PBS jobs with walltimes short enough to complete prior to this maintenance period will be scheduled for execution -- jobs with longer walltimes will remain in the queue until after the maintenance has been completed.

Steele cluster available for use on Tuesday, 5/6

rcac@purdue.edu — Tue, 06 May 2008 22:21:00 PST

After a very successful Installation Day on Monday, May 5, 799 compute nodes were made available for use in the Steele cluster on Tuesday, May 6. In addition to the "standby" PBS queue and its 4-hour maximum walltime limit, queues with 720-hour walltime limits are available for the research groups that helped fund the cluster's acquisition.

Network maintenance scheduled for Steele cluster on Tuesday, 5/6

rcac@purdue.edu — Sat, 03 May 2008 21:22:00 PST

Pre-production access to Steele compute nodes will be unavailable from 10AM-2PM on Tuesday, 5/6, while network maintenance is performed. The steele.rcac front-end system should not be affected. PBS jobs may be submited during this time, but they will be held in the queue until the cluster's network has been restored to service.

Brief interruption to Steele cluster access on Friday, 5/2

rcac@purdue.edu — Thu, 01 May 2008 16:30:00 PST

Pre-production access to Steele compute nodes will be unavailable from 1-2PM on Friday, 5/2, while network maintenance is performed. The steele.rcac front-end system should not be affected. PBS jobs may be submited during this time, but they will be held in the queue until the cluster's network has been restored to service.

RCAC systems returned to service Wednesday, April 30

rcac@purdue.edu — Wed, 30 Apr 2008 21:04:00 PST

The failing disk drives in RCAC's scratch storage server were replaced and the storage system was returned to service in early afternoon on Wednesday, April 30. PBS and Condor job scheduling was re-enabled at approximately 2:30 PM.

RCAC systems unavailable Tuesday and Wednesday, April 29-30

rcac@purdue.edu — Tue, 29 Apr 2008 11:42:00 PST

PBS and Condor job scheduling has been disabled on all RCAC HPC systems due to disk drive failures in the BlueArc scratch storage server. Systems affected include all RCAC Linux compute clusters, the SGI Altix, the IBM SP and Regatta systems, and the Sun F6800 systems. Replacement drives are being shipped and job scheduling is expected to resume by 5 PM on Wednesday, April 30.

RCAC systems unavailable Monday, April 28, 2008

rcac@purdue.edu — Thu, 17 Apr 2008 22:47:00 PST

RCAC HPC systems including the Linux compute clusters, the SGI Altix,
the IBM SP and Regatta systems, and the Sun F6800 systems will be out
of service from 8am-5pm Monday, April 28, while critical firmware
upgrades are applied to the the home and scratch file servers. Only
those PBS jobs with walltimes short enough to complete prior to this
maintenance period will be scheduled for execution -- jobs with longer
walltimes will remain in the queue until after the maintenance has been
completed.

Security announcement for cloud.cc, IBM SP,and Regatta users

rcac@purdue.edu — Thu, 03 Apr 2008 23:21:00 PST

Cloud.cc.purdue.edu and the IBM SP and Regatta systems have been placed behind an internet firewall to provide protection from a recently-announced security vulnerability in the AIX operating system they run. As a result, connections from non-RCAC systems will no longer be supported and cloud.cc/SP users will need to access those systems by first logging onto another RCAC system (e.g. radon.rcac.purdue.edu) then connecting to cloud.cc.purdue.edu via "ssh".

Please refer any questions about this change to "rcac-help@purdue.edu".

RCAC scratch file server outage

rcac@purdue.edu — Thu, 03 Apr 2008 12:00:00 PST

The RCAC scratch file servers experienced a failure Thursday morning, April 3, 2008. Our systems engineering staff is working to resolve the problem but are currently unable to estimate when the scratch storage areas will be returned to service. Updates will be posted here as additional information becomes available.

RCAC system maintenance scheduled for March 11-13, 2008

rcac@purdue.edu — Wed, 20 Feb 2008 17:15:00 PST

RCAC systems will be unavailable from 7am Tuesday, March 11, thru approximately 6pm Thursday, March 13, for data center and system maintenance.
In preparation for this maintenance period, PBS scheduling reservations have been installed such that only those jobs with walltimes short enough to ensure job completion prior to 7am Tuesday, March 11, will be scheduled for execution – jobs with longer walltimes will remain in the queue until the systems have been returned to production following the maintenance.
Several system changes will be made as part of this maintenance. They are summarized in the table below and described in the paragraphs that follow. Please refer any questions or comments about these changes to rcac-help@purdue.edu.

3/11 March maintenance period begins

PBS and Condor job scheduling suspended
Legacy scratch storage areas made read-only
TeraGrid $TG_CLUSTER_SCRATCH environment variable changed to match $RCAC_SCRATCH
Access to home directories via /home/rcac paths terminated

3/13 March Maintenance ends

PBS and Condor job scheduling resumed

3/31 Legacy scratch storage areas removed

Transition to new BlueArc scratch storage system

In January, 2008, RCAC deployed a new, high-performance scratch storage system utilizing clustered BlueArc Titan 2500 systems and 120 TB of SATA storage. This system provides significantly higher capacity and better performance than RCAC’s older, legacy scratch storage systems and is accessible from all RCAC systems. As a result, the older scratch areas will be made read-only on March 11, 2008, and removed from service on March 31.
Unlike the older scratch areas, each RCAC user has a pre-created working directory on the new scratch server and a default quota there of 250 GB. The location of this directory is stored in the $RCAC_SCRATCH environment variable, which may be referenced in shell scripts and PBS job scripts. Its location may also be found by running the command "myscratch". Also unlike the older scratch areas, files stored in the new scratch system will be automatically removed 60 days after their creation.
The older scratch areas being retired have been accessible as /scratch/local and, alternately, with names like /usr/rmt_share/scratchNN where NN is one of 69, 72, 73, 83-88. To prevent data loss when these older scratch areas are removed from service, they will be made read-only on March 11 and any important files they contain should be copied into the new scratch system (i.e. into $RCAC_SCRATCH) or some other location prior to their removal on March 31. Any new scratch files should be stored in the directory named in the $RCAC_SCRATCH environment variable.
TeraGrid users of Purdue systems should note that on March 11, the $TG_CLUSTER_SCRATCH environment variable will be changed to match the contents of $RCAC_SCRATCH and will contain the location of their working directory in the new scratch system rather than "/scratch/teragrid". Also, the /scratch/teragrid file system, aka /usr/rmt_share/scratch69, will become read-only on March 11 and will be removed on March 31.

Home directory paths

Several older RCAC systems use NFS automounter software to provide a short-hand path of the form /home/rcac/loginname to home directories. This software is somewhat fragile, so it not being installed on new RCAC systems as they are deployed. To provide a more consistent environment across all RCAC systems, its use will be terminated on all RCAC systems during the March maintenance period.
After March 11, references of the form /home/rcac/loginname will fail and RCAC users should refer to their home directories in shell scripts and PBS job scripts via the $HOME environment variable. In any situation in which an absolute path is required, the path displayed by running the command "echo $HOME" should be used.

Normal job scheduling resumed on the Lear and Hamlet clusters

rcac@purdue.edu — Tue, 29 Jan 2008 16:12:00 PST

The cooling-system maintenance scheduled in the MATH data center on January 29 has been completed and normal job scheduling has resumed.

Lear Cluster Idled Monday Night, January 28, 2008

rcac@purdue.edu — Mon, 28 Jan 2008 11:35:00 PST

To reduce heat generation prior to cooling system maintenance in the MATH datacenter, compute nodes in the Lear cluster will be idled from midnight Monday night, January 28, thru 10am Tuesday morning when the maintenance is expected to be completed. PBS jobs having walltimes long enough that they would not complete by midnight Monday will be held in the queue until after the cooling-system work has been completed. Please refer questions about this service reduction to rcac-help@purdue.edu.

PBS job scheduling reenabled on Radon Linux cluster

rcac@purdue.edu — Fri, 18 Jan 2008 18:59:00 PST

The RCAC systems engineering staff has analyzed the cause of recent system crashes on the Radon cluster and is testing what is hoped to be a fix for the problem on a subset of the cluster's compute nodes. PBS job scheduling has been restarted and the systems staff will monitor the status of the cluster over the weekend of January 19-21 and decide whether or not to apply this fix to the remainder of its systems on Tuesday, January 22.
Please refer questions or comments to rcac-help@purdue.edu.

PBS job scheduling temporarily disabled on Radon Linux cluster

rcac@purdue.edu — Fri, 18 Jan 2008 15:08:00 PST

PBS job scheduling on the Radon Linux cluster has been temporarily disabled while the RCAC systems engineering staff works to diagnose and correct a problem that has caused crashes on many of the systems in the cluster recently.
Please refer questions or comments about this service interruption to rcac-help@purdue.edu

RCAC systems unavailable January 8-10, 2007

rcac@purdue.edu — Tue, 08 Jan 2008 11:16:00 PST

RCAC systems will be unavailable from 7am Tuesday morning, January 8, thru Thursday afternoon, January 10, for system maintenance, home and scratch directory file server upgrades, and data center maintenance. PBS jobs submitted prior to this outage will be held in queue for execution after the systems have been returned to service on January 10.
Please refer questions about this service interruption to rcac-help@purdue.edu.