RCAC - System News

Purdue partnership for putting idle computers to work on research wins international award

Thu, 05 Nov 2009

Purdue is being recognized as a worldwide campus technology innovator for its DiaGrid system, harnessing what would be otherwise wasted computing power for major research projects.

Campus Technology Magazine has selected DiaGrid for one of its 2009 Campus Technology Innovators Awards. The annual awards, highlighted in the magazine’s current issue, recognize higher education institutions for groundbreaking initiatives in educational technology that are models for other schools. Campus Technology is a leading monthly publication produced by 1105 Media Education Group to deliver news and real-world examples covering the use of technology to advance the way colleges and universities operate and educate today.

Purdue’s partners in DiaGrid include Purdue's Calumet and North Central campuses, Indiana University-Purdue University Fort Wayne, Indiana University, Indiana State University and the universities of Notre Dame, Louisville and Wisconsin.

Purdue was one of 11 award winners selected out of 349 nominations from higher education institutions across the globe. The winners are highlighted in the magazine’s August edition and on CampusTechnology.com.

“We are delighted to present Purdue with a 2009 Campus Technology Innovators award,” said Geoffrey Fletcher, editorial director of Campus Technology. “DiaGrid provides millions of hours of computation that would otherwise be wasted, without the need for additional technology or facilities purchases. The sheer size and ingenuity of the initiative, as well as the diversity of computing resources represented in the grid, really set the project apart.”

At heart, DiaGrid is a working example from the hot field of network-centered “cloud” computing services.

“As cloud computing becomes the new standard, Purdue has positioned itself as an ongoing leader in this technology, especially for scientific research and educational purposes,” said Gerry McCartney, Purdue's vice president for information technology and chief information officer.

Operated by ITaP, Purdue's central information technology organization, DiaGrid started as an on-campus system. But the system now includes nearly 27,000 processors at Purdue and the partner campuses. Purdue has an ongoing effort to add partners, with a goal of incorporating 100,000 processors.

“We continue to explore new applications and are seeking additional institutions to join the project,” said John Campbell, associate vice president in charge of research computing for ITaP. “DiaGrid allows institutions to maximize their technology investments while supporting the diverse needs of faculty.”

Researchers at Purdue and elsewhere use DiaGrid for purposes such as imaging the structure of viruses at near-atomic resolutions — the better to develop new ways of battling viral illnesses ranging from swine flu and the common cold to West Nile virus and AIDS.

The distributed computing pool also has been used to, among other things: understand the Solar System’s formation; project the reliability of Indiana’s electrical supply; and model the spread of water pollutants to develop best practices for ensuring water quality and availability.

DiaGrid works by pooling computers over the Purdue campus network and off campus via the Internet and fast research networks. Whenever computers in the pool are idle — at night, when their owners are at lunch and otherwise — the system sends work to them. When computer owners need their computers, jobs in progress automatically get shifted to other machines. The variety of hardware in DiaGrid includes computers in student computer labs, offices, server rooms and supercomputing clusters. It also is the first multi-campus collaboration of its kind.

Photo caption: Purdue’s DiaGrid team (L-R) Andy Howard, Phil Cheeseman, John Campbell, David Braun, Preston Smith and Carol Song. The team is posing with images from the scientific research enabled by DiaGrid, projected in a multi-walled virtual environment at ITaP’s visualization facility.

Writer: Greg Kline, science and technology writer, ITaP, (765) 494-8167, gkline@purdue.edu

Sources: Gerry McCartney, (765) 496-2270, mccart@purdue.edu
John Campbell, (765) 494-1289, john-campbell@purdue.edu

DXUL/Fortress archival storage system unavailable Thursday, 9/17

Fri, 18 Sep 2009

The DXUL/Fortress system's ADIC Scalar 10K tape libary will be unavailable from 8am-noon on Thursday, September 17, for routine maintenance and firmware updates for its LTO tape drives. During this time, it will not be possible to stage files from tape into the DXUL/Fortress disk cache for access.

Please refer questions about this service interruption to rcac-help@purdue.edu.

RCAC system maintenance scheduled for October 9-13

Mon, 19 Oct 2009

RCAC systems will be unavailable from 8am-6pm Friday, October 9, for electrical work in the MATH data center and system maintenance. The Coates Linux cluster will not be returned to service until Tuesday afternoon, October 13, so RCAC staff can conduct whole-cluster performance benchmarking runs.

Only those PBS jobs will walltimes short enough that they will complete prior to 8am Friday, October 9, will be scheduled for execution -- jobs with longer walltimes will be held in the queue until the systems have been returned to production after the maintenance.

Please refer questions about this maintenance period to rcac-help@purdue.edu.

DXUL/Fortress archival storage system unavailable Monday, 9/14

Mon, 14 Sep 2009

The DXUL/Fortress archival storage system became unavailable Monday morning, September 14, due to a problem with its internal diskserver. RCAC and DXUL vendor storage engineers are working to restore the system to service.

Update: The DXUL/Fortress archival storage system was restored to service at 1:30pm ET on Monday, September 14.

Scratch storage outage Thursday, 9/10

Mon, 14 Sep 2009

Access to scratch storage on RCAC research systems became unavailable at approximately 3pm ET on Thursday, Septemeber 10. RCAC and Blue Arc storage engineers are working to correct the problem and return the scratch storage system to normal operation as soon as possible.

Update: Scratch storage on RCAC systems was restored to service at 3:50pm ET Thursday, September 10.

Radon cluster unavailable Tuesday-Wednesday, 8/25-26

Wed, 26 Aug 2009

The Radon Linux cluster will be unavailable Tuesday and Wednesday, August 25-26, for hardware upgrades and a conversion from Debian Linux to Red Hat Enterprise Linux 5 (RHEL5). Since this upgrade changes the version of Linux used on the cluster, all user programs should be recompiled after the upgrade and any PBS jobs that do not complete prior to the upgrade will be deleted and need to be resubmitted.

Update: The Radon cluster's upgrade to Red Hat Enterprise Linux 5 (RHEL5) has been completed and PBS job scheduling was resumed Wednesday evening, August 26. Although some applications compiled prior to the upgrade may continue to work, Radon users are encouraged to recompile their applications for best results.

Please refer questions about this system maintenance to rcac-help@purdue.edu.

Steele job scheduling temporarily paused Monday, 8/10

Tue, 25 Aug 2009

PBS job scheduling on the Steele cluster has been paused so that a Linux kernel update can be applied to the cluster's compute nodes. Steele users may continue to submit jobs but they will not be scheduled for execution until after job scheduling is resumed.

Update: As of 6pm ET Monday, August 10, approximately half of the Steele cluster's 903 compute nodes had received the Linux kernel update, and PBS job scheduling has resumed. The remaining nodes will be updated and made available for job scheduling as the PBS jobs running on them complete.

Please refer questions and/or comments to rcac-help@purdue.edu.

Submission to Steele standby queues temporarily disabled Monday, 7/28

Tue, 04 Aug 2009

Job submission to the Steele cluster's standby and standby-8 queues has been temporarily disabled due to the extremely large number of jobs that have been submitted to the cluster recently. Submission to these queues will be reenabled when the number of jobs in the system has dropped to a more manageable level.

Please refer questions or comments about this temporary scheduling change to rcac-help@purdue.edu

Steele Linux cluster unavailable Tuesday-Wednesday, 8/4-5

Mon, 10 Aug 2009

The Steele Linux cluster will be unavailable from 8am Tuesday, August 4, through 6pm Wednesday, August 5, for an operating system upgrade from Red Hat Enterprise Linux 4 to Red Hat Enterprise Linux 5. Only those PBS jobs with walltimes short enough that they will finish prior to the beginning of this maintenance period are being scheduled for execution -- jobs with longer walltimes will be held in the input queue until the operating system upgrade has been completed.

Update: The Steele cluster's upgrade to Red Hat Enterprise Linux 5 (RHEL5) has been completed and PBS job scheduling was resumed at 6pm Wednesday, August 5. Although many applications compiled prior to the upgrade will continue to work, Steele users are encouraged to recompile their applications for best results.

Please refer questions about the upgrade and/or problem reports to rcac-help@purdue.edu.

Steele Linux Cluster software upgrade postponed

Tue, 14 Jul 2009

The Steele Linux cluster system software upgrade that was scheduled for Tuesday-Thursday, July 14-16, has been postponed due to NFS automount problems detected by RCAC staff during pre-production testing. New dates for the software upgrade will be announced after RCAC staff have confirmed the NFS automount problems have been resolved.

The Steele cluster will be unavailable from 8am-11am Tuesday, July 14, while firmware upgrades are applied to the RCAC home and scratch BlueArc file servers. Only those PBS jobs with walltimes short enough that they will finish prior to the beginning of this maintenance period are being scheduled for execution -- jobs with longer walltimes will be held in the input queue until the BlueArc maintenance has been completed.

Steele Linux Cluster unavailable Tuesday-Thursday, 7/14-16

Sun, 12 Jul 2009

The Steele Linux cluster will be unavailable from 8am Tuesday, July 14, through 6pm Thursday, July 16, for an operating system upgrade from Red Hat Enterprise Linux 4 to Red Hat Enterprise Linux 5. Only those PBS jobs with walltimes short enough that they will finish prior to the beginning of this maintenance period are being scheduled for execution -- jobs with longer walltimes will be held in the input queue until the operating system upgrade has been completed.

RCAC Home and Scratch directories unavailable Tuesday, 7/14

Thu, 16 Jul 2009

RCAC home and scratch directories will be unavailable from 8am to 11am Tuesday, July 14, while firmware upgrades are made on RCAC's BlueArc storage systems. Only those PBS jobs with walltimes short enough that they will finish prior to the beginning of this maintenance period are being scheduled for execution -- jobs with longer walltimes will be held in the input queue until the BlueArc maintenance has been completed.

DXUL/fortress tape library out of service, Friday, June 5

Sun, 07 Jun 2009

The robotic tape library used by the DXUL/fortress archival storage system is experiencing hardware problems and is unable to mount tapes. DXUL remains in operation and files in the archival storage system's disk cache can be accessed normally, but it will not be possible to retrieve files from or store new files in the archive until the library has been repaired.

Please refer questions to "rcac-help@purdue.edu".

Update: The tape library was repaired and the DXUL/fortress archival storage system returned to normal operation at 4:30pm Friday, June 5.

Steele andquot;Candquot; nodes unavailable Friday, May 29

Sun, 07 Jun 2009

The 48 Steele "C" nodes, which have 32GB of memory and Gigabit Ethernet, will be unavailable from 8am to noon Friday, may 29, while a problematic network switch that connects them to the RCAC research network is replaced. Only those PBS jobs with walltimes short enough that they will finish prior to 8am Friday morning are being scheduled on these nodes.

SiCortext system unavailable Wednesday, 5/27

Sun, 07 Jun 2009

The SiCortex system, moffett.rcac.purdue.edu, will be unavailable from 8am-5pm Wednesday, May, while its system software is upgraded.

RCAC system and data center maintenance 3/17-19

Mon, 23 Mar 2009

RCAC systems including the DXUL/fortress archival storage system will be unavailable beginning at 8am Tuesday, 3/17, while system and MATH data center maintenance is performed. All systems are expected to be back in service by 6pm Thursday, 3/19, and updates on the status of individual systems will be shown here throughout the maintenance period.

Final update: All RCAC systems were back in production as of 1:45pm Friday, 3/20. Please refer any questions about this maintenance window to rcac-help@purdue.edu.

System	Expected Return To Service	Status
CMS/gluon	Wed, 3/18, noon	Available Tue, 3/17, 10pm
Condor	Wed, 3/18, noon	Available Wed, 3/18, 10:30am
Account Request System	Wed, 3/18, 6pm	Available Tue, 3/17, 10pm
Beaver	Wed, 3/18, 6pm	Available Wed, 3/18, 6pm
Gray	Wed, 3/18, 6pm	Available Wed, 3/18, 8pm
DXUL/fortress	Wed, 3/18, 6pm	Available Thu, 3/19, 2:30am
Moffett	Wed, 3/18, 6pm	Available Thu, 3/19, 10:30pm
Gold	Wed, 3/18, 6pm	Available Thu, 3/19, 9am
Pete	Wed, 3/18, 6pm	Available Thu, 3/19, 9am
Prospero	Wed, 3/18, 6pm	Available Thu, 3/19, 9am
Radon	Wed, 3/18, 6pm	Available Thu, 3/19, 9am
Rossmann	Wed, 3/18, 6pm	Available Thu, 3/19, 9am
Venice	Wed, 3/18, 6pm	Available Thu, 3/19, 9am
Curator/PiiMS	Thu, 3/19, noon	Available Wed, 3/18, 10:30am
Steele	Thu, 3/19, noon	Available Fri, 3/20, 10:30am
Julius	Thu, 3/19, noon	Available Fri, 3/20, 1:45pm

Unscheduled power outage in MATH Tuesday, 2/10

Thu, 12 Feb 2009

RCAC experienced an unscheduled power failure in its MATH data center at approximately 8:20pm Tuesday, February 10.

The Pete, Steele, and Radon Linux clusters, and the Julius/Caesar SGI Altix 4700 were affected and went out of service.

The Rossmann, Prospero, and Venice Linux clusters, the Condor servers, the CMS systems in MANN Hall, the Moffett SiCortex system, and the RCAC storage servers including the DXUL/fortress archival storage system remained in operation.

As of 9:15pm, electricians and RCAC systems staff were on-site in MATH but were unable to provide an estimate of when power will be restored so the systems could be returned to service.

The next update on this outage is scheduled for 12 midnight Tuesday, February 10.

Update as of midnight Tuesday, February 10: Electricians restored power in the MATH data center at approximately 10:35pm, at which point RCAC system engineers started rebooting the affected systems. Batch job scheduling was resumed on a cluster-by-cluster basis starting at 11:15pm and all systems were back in service by 11:45pm.

We regret any inconvenience caused by this outage, and will be conducting a review to determine its cause and explore ways to mitigate the impact of future power-related problems.

PBS job scheduling temporarily suspended Friday, 1/30

Fri, 30 Jan 2009

PBS job scheduling on RCAC systems was suspended from 11-11:30pm Friday, 1/30, while problems with access to home directories was diagnosed and corrected by RCAC system engineers.

SiCortex system unavailable Wednesday-Thursday, 1/21-22

Tue, 27 Jan 2009

The SiCortex system, moffett.rcac.purdue.edu, will be unavailable from 8am Wednesday, January 21, through 5pm Thursday, January 22, while hardware upgrades are performed.

Update as of 5:45 PM Thursday, 1/22: Diagnostics following this week's SiCortex maintenance indicated possible hardware problems, which RCAC and SiCortex systems engineers are working to resolve. As a result, the SiCortex system remains out of service. The next update on its status will be made at 10PM Thursday evening, 1/22.

Update as of 10:00 PM Thursday, 1/22: The previously mentioned diagnostics indicate the presense of multiple memory modules that need to be replaced. Replacement modules are being acquired but the system will be out of service until they arrive and have been installed and tested. The next update on the system's status will be made at 10AM Friday morning, 1/23.

Update as of 10:00 AM Friday, 1/23: The SiCortex remains out of service while SiCortex engineers continue to run hardware diagnostics. The next update on the systems status is scheduled for 3PM Friday, 1/23.

Update as of 3pm Friday, 1/23: We are continuing to run diagnostics on the SiCortex system. The next update on its status is scheduled for 10PM Friday, 1/23.

Update as of 6PM Friday, 1/23: The SiCortex system, moffett.rcac.purdue.edu, has been returned to service after the hardware upgrades that were made on January 21-23. A few hardware-related issues remain, however, and will be addressed during the system's next maintenance window. Please report any problems you might encounter via e-mail to "rcac-help@purdue.edu".

SGI Altix system maintenance Tuesday, January 13

Tue, 13 Jan 2009

The SGI Altix 4700 system, caesar.rcac.purdue.edu, will be unavailable from 10-11am Tuesday, January 13, while a memory module is replaced. Only those PBS jobs with walltimes short enough that they will finish prior to this maintenance are being scheduled for execution -- jobs with longer walltimes will be held in the queue until the system has been returned to service following this maintenance.

RCAC system maintenance scheduled for January 6-7

Thu, 08 Jan 2009

RCAC systems will be unavailable from 8am-5pm on Tuesday, January 6, for system maintenance. The Pete Linux cluster will not be returned to service until Wednesday afternoon, January 7, so RCAC systems staff and technical staff from the PBSPro vendor can work to diagnose and correct problems with its PBS server installation.

Only those PBS jobs will walltimes short enough that they will complete prior to 8am Tuesday, January 6, will be scheduled for execution -- jobs with longer walltimes will be held in the queue until the systems have been returned to production after the maintenance.

SiCortex system unavailable Thursday-Friday, 12/11-12

Mon, 15 Dec 2008

The SiCortex system, moffett.rcac.purdue.edu, will be unavailable from noon Thursday, December 11, through late afternoon Friday, December 12, while its software is updated to release level 3.0. Release notes for this updated software version , which corrects several known problems and includes some new features, may be found on the SiCortex web site.

DXUL/fortress system maintenance scheduled for Thursday, Nov 13

Mon, 17 Nov 2008

The Fortress/DXUL tape robot system will be undergoing hardware maintenance from 10:00am-4:00pm Thursday, November 13. During this time, it will not be possible to retrieve files from the tape archive nor will it be possible for the DXUL system to migrate files from its disk cache into the tape library. RCAC storage engineers will monitor the disk cache and disable all DXUL access if the cache fills during this maintenance window.

SGI Altix system unavailable Wednesday-Thursday, 11/5-6

Mon, 17 Nov 2008

The SGI Altix 4700 system, caesar.rcac.purdue.edu, will be unavailable from 8am Wednesday morning, November 5, until late afternoon Thursday, November 6, so SGI and Purdue system engineers can apply firmware upgrades to the system's /scratch/caesar file system and conduct tests to evaluate its stability. The results of those tests will determine whether or not the file system will be made available Thursday afternoon when caesar.rcac.purdue.edu is returned to service.

Only those PBS jobs with walltimes less than the time remaining until the beginning of this maintenance period Wednesday morning, November 5, will be scheduled for execution. Jobs with longer walltimes will be held in the queue for execution after the system is returned to service Thursday afternoon.

Caesar PBS Scheduling andamp; Scratch Down, Oct. 21-22

Wed, 22 Oct 2008

PBS scheduling on Caesar was been stopped in the evening of October 21, and the scratch filesystem /scratch/caesar/ is unavailable. Engineers are working on the problem with the vendor, but do not yet have an estimate of when PBS will return to service.

PBS job scheduling on the Caesar SGI Altix 4700 was restarted at approaimately 4:00P.M. Wednesday,October 23. However, its /scratch/caesar file system has been removed from service due to hardware problems. In addition, the system's /tmp file system has been reduced in size to 10GB. Please use your RCAC scratch directory as named in your $RCAC_SCRATCH environment variable instead.

Questions about this outage and system change should be referred to "rcac-help@purdue.edu".

Steele PBS Scheduling Down, Sept. 11-12

Fri, 12 Sep 2008

PBS scheduling of any new jobs on Steele stopped working around 11:30pm on Sept. 11 and is still down now. Engineers are working on the problem with the vendor, but do not yet have an estimate of when PBS will return to service.

Update on 9/12, 4:30pm: Our engineers are still in contact with the PBS vendor, but the vendor has not been able to resolve the problem yet. We will post another update at noon tomorrow.

Update on 9/12, 6:25pm: Steele PBS is back in operation now. All jobs should be resuming normally and new submissions should work as well. Sorry for any inconvenience this may have caused. If you see any problems, please let us know by emailing rcac-help@purdue.edu.

Steele-A Partial Power Outage, Sept. 12

Fri, 12 Sep 2008

Numerous Steele-A nodes were unavailable due to a problem with a power distribution unit (PDU) in the Math data center from sometime early this morning until 11:00am. Electricians were able to restore power at that time, but PBS scheduling is still not working. That issue is being tracked in another system news item.

Fortress/DXUL Emergency Follow-up Maintenance Wednesday, September 10

Thu, 11 Sep 2008

The Fortress/DXUL tape robot system will be undergoing emergency hardware maintenance from 8:00am-5:00pm on Wednesday, September 10. This maintenance is a follow-up to the work done August 25-26. During this time, it will not be possible to retrieve files from the tape archive nor will it be possible for the DXUL system to migrate files from its disk cache into the tape library. RCAC storage engineers will monitor the disk cache and disable all DXUL access if the cache fills during this maintenance window. Fortress/DXUL users are asked to please keep their use of the system to a minimum during this time.

Pete cluster temporarily unavailable Tuesday, September 2

Wed, 03 Sep 2008

The pete.rcac back-end server for the Pete cluster hung earlier today, Tuesday, September 2. As a result, pete.rcac had to be rebooted and PBS jobs executing at the time had to be requeued.

Part of the process of restarting the PBS server involves its reestablishing communication with the PBS software running on each compute node. Under normal circumstances, reestablishing a cluster's nodes should take 15-20 minutes.

However, this operation is taking much longer than that on the Pete cluster -- our current estimate is that it will complete at approximately 5pm, at which point PBS job scheduling will be reenabled. We have escalated this issue to our software vendor, and are working toward a better turnaround in this sort of situation.

Update: PBS job scheduling on the Pete cluster was reenabled at 5pm Tuesday, September 2.

Please refer questions about this outage to rcac-help@purdue.edu.

SGI Altix system maintenance Tuesday, September 2

Wed, 03 Sep 2008

The four RCAC SGI Altix systems (julius, caesar, octavius, and brutus.rcac.purdue.edu) will be unavailable from 8am-5pm Tuesday, 9/2, while operating system upgrades are installed.

Update as of 5pm Tuesday, September 2: RCAC systems engineers have encountered problems updating the Altix systems and are waiting for a response from SGI technical support. The Altix systems will remain unavailable until SGI can address the problems we have encountered with the system updates.

Update as of 3pm Wednesday, September 3: RCAC system engineers have returned the SGI Altix systems to service and PBS job scheduling on the Altix 4700, caesar.rcac.purdue.edu, has been re-enabled.

Please refer questions about this outage to rcac-help@purdue.edu.

Emergency maintenance for RCAC Linux systems Wed-Fri, August 27-29

Mon, 01 Sep 2008

RCAC will be performing emergency maintenance on all its Linux systems Wednesday-Friday, August 27-29. This work includes the application of patches, InfiniBand upgrades, and downtime for system reboots as described in the schedule below. Job scheduling reservations are in place to prevent longer jobs from starting prior to each cluster's reboot time. When possible, currently-executing jobs will be requeued and restarted after cluster nodes have been returned to service.

Please refer questions and comments about this maintenance to rcac-help@purdue.edu.

Resource	Downtime	Notes	Status
Gray	Tue, 08/26, noon-5pm		Completed 5pm Wed, 8/27
CMS	Wed, 08/27, 3-8am		Completed 7:30am Wed, 8/27
Radon	Wed, 08/27, noon-5pm		Completed 5:30pm Wed, 8/27
Steele	Wed, 08/27, noon-5pm	1	Completed 10:20pm Wed, 8/27
Venice	Wed, 08/27, noon-5pm		Completed 2:30pm Thu, 8/28
Julius/SGI Altix	Thu, 08/28, 8am-6pm		Completed 5:00pm Thu, 8/28
Pete	Fri, 08/29, 8am-5pm	2	Completed 12:30pm Sat, 8/30
Prospero	Fri, 08/29, 8am-5pm	1	Completed 7:30pm Fri, 8/29
Rossmann	Fri, 08/29, 8am-5pm	1	Completed 7:30pm Fri, 8/29
1) It is recommended that applications using InfiniBand message-passing libraries be recompiled following the maintenance
2) Due to complications with the integration of SFS/Lustre with a new Linux kernel installed as part of the system maintenance, the contents of /scratch/lustre1 have been copied onto an RCAC BlueArc file server, where they will be available for use on the Pete cluster while RCAC works with HP to address the SFS/Lustre upgrade issues.

Resolved: Emergency DXUL/fortress maintenance Tuesday, August 26

Tue, 26 Aug 2008

Emergency hardware maintenance has been scheduled for the tape robot used by the DXUL/fortress archival storage system from 10:30am-12:30pm on Tuesday, August 26. This maintenance is in follow-up to the work done on Monday, August 25. During this time, it will not be possible to retrieve files from the tape archive nor will it be possible for the DXUL system to migrate files from its disk cache into the tape library. RCAC storage engineers will monitor the disk cache and disable all DXUL access if the cache fills during this maintenance window. DXUL users are encouraged to keep their use of the system to a minimum during this period.

Update as of 1pm Tuesday, August 26: Maintenance on the tape robot used by the DXUL/fortress system has been completed and the system has been returned to service.

Please refer questions about this maintenance activity to rcac-help@purdue.edu.

Resolved: Emergency DXUL/fortress maintenance Monday, August 25

Mon, 25 Aug 2008

Emergency hardware maintenance has been scheduled for the tape robot used by the DXUL/fortress archival storage system from 6:30am-noon on Monday, August 25. During this time, it will not be possible to retrieve files from the tape archive nor will it be possible for the DXUL system to migrate files from its disk cache into the tape library. RCAC storage engineers will monitor the disk cache and disable all DXUL access if the cache fills during this maintenance window. DXUL users are encouraged to keep their use of the system to a minimum during this period.

Update as of 9am Monday, August 25: The DXUL cache has filled, so DXUL access has been disabled until maintenance on the tape robot has been completed and migration of files from the disk cache into the tape library has resumed. The DXUL/fortress system is expected to be returned to service by 1pm.

Update as of 4:15pm Monday, August 25: Maintenance on the tape robot used by the DXUL/fortress system has been completed and the system has been returned to service.

Please refer questions about this maintenance activity to rcac-help@purdue.edu.

DXUL/fortress maintenance scheduled for Thursday, August 21

Thu, 21 Aug 2008

The DXUL/fortress archival storage system will be unavailable from 6:30-9:00am Thursday, August 21, while hardware maintenance to upgrade its Fibre Channel switches is performed.

Resolved; Unscheduled outage of CMS equipment in MANN, Monday August 18

Mon, 18 Aug 2008

A chilled water loss has occured at the MANN datacenter housing CMS Tier-2 equipment and all dedicated CMS systems housed there are unavailable until cooling has been restored.

Update -- 3pm Monday, August 18: The chilled water supply to MANN Hall has been restored and the CMS systems housed there were returned to production at 3pm, August 18.

Please refer questions about this outage to "rcac-help@purdue.edu".

Resolved: Unscheduled SGI Altix outage Thursday, August 14

Thu, 14 Aug 2008

The SGI Altix 4700, caesar.rcac, was unavailable for several hours on Thursday, August 14, due to hardware problems. RCAC systems engineers diagnosed and corrected the problems, and the system was returned to production at approximately 10:45am.

Please refer questions about this service interruption to "rcac-help@purdue.edu".

SGI Altix system unavailable Thursday, August 28

Mon, 01 Sep 2008

The Julius/Caesar SGI Altix 4700 system will be unavailable from 8am-5pm on Thursday, August 28, while its job scheduler is upgraded from PBSPro version 8.0 to version 9.1. A scheduling reservation has been installed and only those PBS jobs with walltimes less than the time remaining until 8am Thursday, August 28, will be scheduled for execution. All other jobs will be held for execution after the software upgrade has been completed.

Please refer questions about this service interruption to rcac-help@purdue.edu.

Radon Linux cluster unavailable Thursday, August 14

Mon, 04 Aug 2008

The Radon Linux cluster will be unavailable from 8am-5pm on Thursday, August 14, while its job scheduler is upgraded from PBSPro version 8.0 to version 9.1. A scheduling reservation has been installed and only those PBS jobs with walltimes less than the time remaining until 8am Thursday, August 14, will be scheduled for execution. All other jobs will be held for execution after the software upgrade has been completed.

Please refer questions about this service interruption to "rcac-help@purdue.edu".

Gold IBM AIX cluster to be rebooted Tuesday, August 5

Mon, 04 Aug 2008

Systems in the Gold IBM cluster will will be unavailable between noon-1pm on Tuesday, August 5, while they are rebooted to complete the installation of AIX system-software updates.

Resolved: Unscheduled DXUL archival storage system outage Wednesday, July 30

Wed, 30 Jul 2008

The DXUL/fortress.rcac.purdue.edu archival storage system was unavailable from approximately 3am-11:45am Wednesday, July 30, due to problems with its internal databases. RCAC storage engineers corrected those problems and the system appears to be operating normally.

Please refer questions or comments about this outage to rcac-help@purdue.edu.

SiCortex system to be rebooted Wednesday, July 16

Wed, 16 Jul 2008

The SiCortex system, moffett.rcac, will be rebooted between 2-3pm Wednesday, July 16, to allow several system changes to take effect. Jobs executing at that time will be terminated and the owners of currently-executing jobs have been notified of the reboot.

Resolved: Power outage affects CMS-related systems in MANN 112 Monday, July 14

Mon, 14 Jul 2008

CMS-related systems housed in MANN 112 are out of service due to an unplanned power outage that occurred shortly after 11am on Monday, July 14, Updates on the status of the systems will be posted here as additional information about the outage becomes available.

5:30pm July 14 Update: Power was restored to MANN hall and the CMS-related systems housed in MANN 112 were returned to service at approximately 2:30pm Monday, July 14.

Resolved: Omni.cc unavailable due to disk drive failure Thursday, July 10

Thu, 10 Jul 2008

Omni.cc.purdue.edu suffered a hardware failure on its system boot disk drive Thursday, July 10. RCAC system engineers have located a replacement, and are working to replace the failed drive and restore the system to operation.

7PM Friday, July 11 update: Omni.cc was returned to service at approximately 6:30PM Friday, July 11.

As of 9pm Thursday, July 10, our estimate is that the system will be back in service by early evening Friday, July 11. An update will be posted here if problems arise that will delay its return to service.

Important note for omni.cc users: Due to its age, which significantly complicates system maintenance and repair, and its limited use in recent years, omni.cc.purdue.edu will be retired and removed from service on Friday, October 10. Please contact rcac-help@purdue.edu if you have questions or need assistance in migrating the work you do on omni.cc to another system.

System maintenance scheduled for gray.rcac Thursday, July 10

Tue, 08 Jul 2008

The gray.rcac.purdue.edu system that serves as a front-end to the IBM e1350 BladeCenter in Bloomington will be unavailable from noon-5pm Thursday, July 10, while system maintenance is performed.

Please refer questions about this service interruption to "rcac-help@purdue.edu".

Radon.rcac to be rebooted Tuesday afternoon, July 8

Tue, 01 Jul 2008

The radon.rcac.purdue.edu Linux cluster front-end system will unavailable for a few minutes starting at 2PM Tuesday, July 8, so it can be rebooted to correct a problem related to running X Window applications such as Matlab. Non-interactive PBS jobs and Condor jobs should be unaffected by the reboot.

Please refer questions about this service interruption to "rcac-help@purdue.edu".

Resolved: RCAC system maintenance scheduled for July 1

Mon, 30 Jun 2008

RCAC systems including the Linux compute clusters, the SGI Altix, the IBM SP and Regatta systems, and the Sun F6800 systems will be out of service from 8am-midnight Tuesday, July 1, for hardware and software maintenance. Only those PBS jobs with walltimes short enough that they would complete prior to this maintenance period will be scheduled for execution -- jobs with longer walltimes will remain in the queue until after the maintenance has been completed.

10:00am July 3: The Pete and Steele clusters' PBS job schedulers appear to be operating normally. We will continue to closely monitor their behavior and are working with the PBS vendor to determine and correct the cause of the instabilities observed on July 1-2.

3:30pm July 2 update: The PBS job scheduler on the Pete cluster appears to be stable but the PBS scheduler on the Steele cluster remains unstable. While Steele jobs are being scheduled properly much of the time, frequent PBS crashes may temporarily interfere with job submission or the receipt of output from recently-completed jobs. We continue to closely monitor both the Pete and Steele clusters and have contacted the PBS vendor for further support. Updates will be posted as additional information becomes available.

10:30am July 2 update: There are problems with the Steele cluster's PBS job scheduler which are causing the scheduler to abort. RCAC system engineers are working to resolve the problems and are in contact with the PBS vendor's tech support staff.

12am July 2 update: The maintenance scheduled for Tuesday, July 1, has been completed. RCAC systems have been returned to service, but a problem has been observed with the Pete cluster's PBS job scheduler in that nodes frequently falsely report themselves as offline to the scheduler. RCAC systems engineers are working on the problem and will be contacting the PBS vendor's tech support staff Wednesday morning, July 2.

Resolved: DXUL/fortress system unavailable as of Wednesday, June 18

Wed, 18 Jun 2008

The DXUL/fortress.rcac archival storage system is unavailable effective 10AM Wednesday, June 18, due to a mechanical failure in its robotic tape library. Replacement parts and service personnel are en route and we estimate it will remain out of service until 5PM Thursday, June 19.

3:15 PM Thursday June 19: The DXUl/fortress.rcac archival storage system was returned to service at approximately 3 PM Thursday, June 19, following repairs to its robotic tape library.

Please refer questions about this outage to rcac-help@purdue.edu.

Resolved: DXUL/fortress system unavailable Monday, June 16

Mon, 16 Jun 2008

The DXUl/fortress.rcac archival storage system is unavailable due to a mechanical failure in its robotic tape library. Replacement parts and service personnel are en route but we currently estimate it will remain out of service until 5PM Tuesday, June 17.

3PM Monday, June 16: The DXUl/fortress.rcac archival storage system was returned to service at approximately 2:15PM Monday, 6/16, following repairs to its robotic tape library.

Please refer questions about this outage to rcac-help@purdue.edu.

Resolved: Unscheduled service interruption Friday, June 13

Fri, 13 Jun 2008

RCAC systems are currently out of service due to file server network interface problems that developed at 3:20PM Friday, June 13. RCAC system engineers are working to correct the problem but are currently unable to estimate when normal service will be restored. The next update on this outage will be at 9PM Friday, June 13.

9PM 6/13 Update: The network interface problems that were preventing access to home directories on RCAC systems have been resolved, and all systems were restored to normal operation at approximately 8:45PM Friday, June 13.

Resolved: Unscheduled RCAC scratch fileserver outage Wednesday, June 11

Wed, 11 Jun 2008

The RCAC scratch file systems are currently unavailable due to problems with one of the BlueArc file servers. As a result, PBS and Condor job scheduling on all RCAC systems has been halted pending resolution of the problem. A BlueArc technician is on-site working on the problem, which occurred following maintenance on one of the redundant BlueArc servers.

2:30PM Friday, 6/13: The maintenance on the RCAC scratch file servers was completed and normal PBS and Condor job scheduling was enabled at 2PM Friday, 6/13.

4PM 6/12 Update: The RCAC scratch file systems were returned to service at 3:30PM Thursday afternoon, 6/12. PBS and Condor job scheduling has been re-enabled, but some additional file server maintenance remains to be done and has been scheduled for 1-2PM Friday, 6/13. As a precaution, a PBS reservation has been installed such that only those jobs with walltimes short enough to finish prior to 1PM Friday will be scheduled for execution. Jobs with longer walltimes will be held in the queue until after the the maintenance has been completed. The next update on this work will be posted at 2:30PM Friday, 6/13.

12noon 6/12 Update: One of the scratch file systems, scratch95, became unavailable shortly before 12noon, so PBS and Condor job scheduling have been halted until the installation of the replacement BlueArc hardware has been completed and the problems can be resolved. The next update on this outage will be posted at 4pm, June 12.

9AM 6/12 Update: The scratch file systems have been operating normally since 1:30AM Thursday morning, June 12. A replacement for hardware suspected of causing Wednesday's outage is expected to arrive and be installed later Thursday morning. The next update on this outage will be posted at 12noon, June 12.

1:45AM 6/12 Update: The scratch file systems have been returned to service and both PBS and Condor job scheduling have resumed. A replacement for a hardware component believed to have caused the problem is being express-shipped to Purdue for installation Thursday morning, 6/12.

9PM 6/11 Update: RCAC system engineers and a BlueArc technician continue to work on the problem but are unable to
estimate when the file systems will be returned to service. The next update on their status will be posted at 9AM, June 12.

5PM 6/11 Update: System engineers continue to work on the scratch file server problem but are unable to estimate when it will be resolved. The next update will be posted at 9PM.

3PM 6/11 Update: RCAC and BlueArc system engineers continue to work on the problem, but we are unable to estimate when the file systems will be returned to service. The next update on their status will be posted at 5PM.

Please refer questions about this outage to "rcac-help@purdue.edu".

Pete Linux cluster reboots scheduled for Thursday, June 5

Wed, 04 Jun 2008

In order to install an updated version of the Pete Linux cluster's Lustre/SFS client software, its compute nodes and front-end/login hosts will be rebooted between 2:30-3:30 PM on Thursday, June 6. To reduce the impact these reboots have on executing PBS jobs, a scheduling reservation has been installed for this time period and PBS jobs requesting more walltime than remains prior to 2:30 PM Thursday will be held in the queue until after the systems have been rebooted and returned to service.

Please refer questions or comments about this system maintenance to rcac-help@purdue.edu.

DXUL/fortress system maintenance scheduled for Thursday, May 29

Tue, 27 May 2008

Access to the DXUL/fortress.rcac.purdue.edu archival storage system will be unavailable from 8am-5pm Thursday, May 29, while system maintenance is performed to update software, install a new tape drive, and update firmware on existing tape drives.

Please refer questions or comments about this scheduled outage to "rcac-help@purdue.edu".

Resolved: Unscheduled DXUL/fortress outage Monday, May 26

Mon, 26 May 2008

The DXUL/fortress.rcac.purdue.edu was unavailable for approximately three hours on Monday, May 26, due to a recurrence of a problem first observed on Saturday, May 24. RCAC and EMC/DXUL storage engineers were able to return the system to service at approximately 4:30 pm and are working to diagnose and correct the problem.

Please refer questions or comments about this outage to "rcac-help@purdue.edu".

Resolved: Unscheduled DXUL/fortress outage Sunday, May 25

Sun, 25 May 2008

The DXUL/fortress.rcac.purdue.edu was unavailable for several hours on Sunday, May 25, due to a recurrence of a problem first observed on Saturday, May 24. RCAC and EMC/DXUL storage engineers were able to return the system to service at approximately 4:00 pm.

Please refer questions or comments about this outage to rcac-help@purdue.edu.

Resolved: Unscheduled DXUL/fortress outage Saturday, May 24

Sat, 24 May 2008

The DXUL/fortress.rcac.purdue.edu archival storage system was unavailable for several hours on Saturday, May 24, due to system problems. RCAC system and storage engineers corrected the problems and returned the system to operation at approximately 8pm.

Please refer questions or comments about this outage to rcac-help@purdue.edu.

Final update: Unscheduled Steele cluster outage Thursday, May 15-16

Thu, 15 May 2008

The Steele cluster has been returned to production and PBS job scheduling was re-enabled at approximately 3:45 PM Friday, May 16. Several PBS jobs were lost during the cluster's network problems and will need to be resubmitted by their owners.

We will continue to closely monitor the cluster's network and anticipate needing to schedule and announce an outage for network maintenance and tuning in the relatively near future.

Thank you for your patience during this outage -- please refer any questions or comments about it to rcac-help@purdue.edu.

SGI Altix 4700 hardware maintenance scheduled for Wednesday, May 14

Thu, 08 May 2008

The SGI Altix 4700 will be unavailable from 8AM-noon Wednesday, May 14, while hardware maintenance is performed. Access to the julius.rcac.purdue.edu front-end system should not be affected. Only those PBS jobs with walltimes short enough to complete prior to this maintenance period will be scheduled for execution -- jobs with longer walltimes will remain in the queue until after the maintenance has been completed.

IBM SP hardware maintenance scheduled for Wednesday, May 14

Thu, 08 May 2008

The IBM SP will be unavailable from 8AM-noon on Wednesday, May 14, while hardware maintenance is performed. Access to the cloud.cc.purdue.edu front-end system should not be affected. Only those PBS jobs with walltimes short enough to complete prior to this maintenance period will be scheduled for execution -- jobs with longer walltimes will remain in the queue until after the maintenance has been completed.

Steele cluster available for use on Tuesday, 5/6

Tue, 06 May 2008

After a very successful Installation Day on Monday, May 5, 799 compute nodes were made available for use in the Steele cluster on Tuesday, May 6. In addition to the "standby" PBS queue and its 4-hour maximum walltime limit, queues with 720-hour walltime limits are available for the research groups that helped fund the cluster's acquisition.

Network maintenance scheduled for Steele cluster on Tuesday, 5/6

Sat, 03 May 2008

Pre-production access to Steele compute nodes will be unavailable from 10AM-2PM on Tuesday, 5/6, while network maintenance is performed. The steele.rcac front-end system should not be affected. PBS jobs may be submited during this time, but they will be held in the queue until the cluster's network has been restored to service.

Brief interruption to Steele cluster access on Friday, 5/2

Thu, 01 May 2008

Pre-production access to Steele compute nodes will be unavailable from 1-2PM on Friday, 5/2, while network maintenance is performed. The steele.rcac front-end system should not be affected. PBS jobs may be submited during this time, but they will be held in the queue until the cluster's network has been restored to service.

RCAC systems returned to service Wednesday, April 30

Wed, 30 Apr 2008

The failing disk drives in RCAC's scratch storage server were replaced and the storage system was returned to service in early afternoon on Wednesday, April 30. PBS and Condor job scheduling was re-enabled at approximately 2:30 PM.

RCAC systems unavailable Tuesday and Wednesday, April 29-30

Tue, 29 Apr 2008

PBS and Condor job scheduling has been disabled on all RCAC HPC systems due to disk drive failures in the BlueArc scratch storage server. Systems affected include all RCAC Linux compute clusters, the SGI Altix, the IBM SP and Regatta systems, and the Sun F6800 systems. Replacement drives are being shipped and job scheduling is expected to resume by 5 PM on Wednesday, April 30.

RCAC systems unavailable Monday, April 28, 2008

Thu, 17 Apr 2008

RCAC HPC systems including the Linux compute clusters, the SGI Altix,
the IBM SP and Regatta systems, and the Sun F6800 systems will be out
of service from 8am-5pm Monday, April 28, while critical firmware
upgrades are applied to the the home and scratch file servers. Only
those PBS jobs with walltimes short enough to complete prior to this
maintenance period will be scheduled for execution -- jobs with longer
walltimes will remain in the queue until after the maintenance has been
completed.

Security announcement for cloud.cc, IBM SP,and Regatta users

Thu, 03 Apr 2008

Cloud.cc.purdue.edu and the IBM SP and Regatta systems have been placed behind an internet firewall to provide protection from a recently-announced security vulnerability in the AIX operating system they run. As a result, connections from non-RCAC systems will no longer be supported and cloud.cc/SP users will need to access those systems by first logging onto another RCAC system (e.g. radon.rcac.purdue.edu) then connecting to cloud.cc.purdue.edu via "ssh".

Please refer any questions about this change to "rcac-help@purdue.edu".

RCAC scratch file server outage

Thu, 03 Apr 2008

The RCAC scratch file servers experienced a failure Thursday morning, April 3, 2008. Our systems engineering staff is working to resolve the problem but are currently unable to estimate when the scratch storage areas will be returned to service. Updates will be posted here as additional information becomes available.

RCAC system maintenance scheduled for March 11-13, 2008

Wed, 20 Feb 2008

RCAC systems will be unavailable from 7am Tuesday, March 11, thru approximately 6pm Thursday, March 13, for data center and system maintenance.

In preparation for this maintenance period, PBS scheduling reservations have been installed such that only those jobs with walltimes short enough to ensure job completion prior to 7am Tuesday, March 11, will be scheduled for execution – jobs with longer walltimes will remain in the queue until the systems have been returned to production following the maintenance.

Several system changes will be made as part of this maintenance. They are summarized in the table below and described in the paragraphs that follow. Please refer any questions or comments about these changes to rcac-help@purdue.edu.

3/11 March maintenance period begins

PBS and Condor job scheduling suspended
Legacy scratch storage areas made read-only
TeraGrid $TG_CLUSTER_SCRATCH environment variable changed to match $RCAC_SCRATCH
Access to home directories via /home/rcac paths terminated

3/13 March Maintenance ends

PBS and Condor job scheduling resumed

3/31 Legacy scratch storage areas removed

Transition to new BlueArc scratch storage system

In January, 2008, RCAC deployed a new, high-performance scratch storage system utilizing clustered BlueArc Titan 2500 systems and 120 TB of SATA storage. This system provides significantly higher capacity and better performance than RCAC’s older, legacy scratch storage systems and is accessible from all RCAC systems. As a result, the older scratch areas will be made read-only on March 11, 2008, and removed from service on March 31.

Unlike the older scratch areas, each RCAC user has a pre-created working directory on the new scratch server and a default quota there of 250 GB. The location of this directory is stored in the $RCAC_SCRATCH environment variable, which may be referenced in shell scripts and PBS job scripts. Its location may also be found by running the command “myscratch”. Also unlike the older scratch areas, files stored in the new scratch system will be automatically removed 60 days after their creation.

The older scratch areas being retired have been accessible as /scratch/local and, alternately, with names like /usr/rmt_share/scratchNN where NN is one of 69, 72, 73, 83-88. To prevent data loss when these older scratch areas are removed from service, they will be made read-only on March 11 and any important files they contain should be copied into the new scratch system (i.e. into $RCAC_SCRATCH) or some other location prior to their removal on March 31. Any new scratch files should be stored in the directory named in the $RCAC_SCRATCH environment variable.

TeraGrid users of Purdue systems should note that on March 11, the $TG_CLUSTER_SCRATCH environment variable will be changed to match the contents of $RCAC_SCRATCH and will contain the location of their working directory in the new scratch system rather than “/scratch/teragrid”. Also, the /scratch/teragrid file system, aka /usr/rmt_share/scratch69, will become read-only on March 11 and will be removed on March 31.

Home directory paths

Several older RCAC systems use NFS automounter software to provide a short-hand path of the form /home/rcac/loginname to home directories. This software is somewhat fragile, so it not being installed on new RCAC systems as they are deployed. To provide a more consistent environment across all RCAC systems, its use will be terminated on all RCAC systems during the March maintenance period.

After March 11, references of the form /home/rcac/loginname will fail and RCAC users should refer to their home directories in shell scripts and PBS job scripts via the $HOME environment variable. In any situation in which an absolute path is required, the path displayed by running the command “echo $HOME” should be used.

Normal job scheduling resumed on the Lear and Hamlet clusters

Tue, 29 Jan 2008

The cooling-system maintenance scheduled in the MATH data center on January 29 has been completed and normal job scheduling has resumed.

Lear Cluster Idled Monday Night, January 28, 2008

Mon, 28 Jan 2008

To reduce heat generation prior to cooling system maintenance in the MATH datacenter, compute nodes in the Lear cluster will be idled from midnight Monday night, January 28, thru 10am Tuesday morning when the maintenance is expected to be completed. PBS jobs having walltimes long enough that they would not complete by midnight Monday will be held in the queue until after the cooling-system work has been completed. Please refer questions about this service reduction to rcac-help@purdue.edu.

PBS job scheduling reenabled on Radon Linux cluster

Fri, 18 Jan 2008

The RCAC systems engineering staff has analyzed the cause of recent system crashes on the Radon cluster and is testing what is hoped to be a fix for the problem on a subset of the cluster's compute nodes. PBS job scheduling has been restarted and the systems staff will monitor the status of the cluster over the weekend of January 19-21 and decide whether or not to apply this fix to the remainder of its systems on Tuesday, January 22.

Please refer questions or comments to rcac-help@purdue.edu.

PBS job scheduling temporarily disabled on Radon Linux cluster

Fri, 18 Jan 2008

PBS job scheduling on the Radon Linux cluster has been temporarily disabled while the RCAC systems engineering staff works to diagnose and correct a problem that has caused crashes on many of the systems in the cluster recently.

Please refer questions or comments about this service interruption to rcac-help@purdue.edu

RCAC systems unavailable January 8-10, 2007

Tue, 08 Jan 2008

RCAC systems will be unavailable from 7am Tuesday morning, January 8, thru Thursday afternoon, January 10, for system maintenance, home and scratch directory file server upgrades, and data center maintenance. PBS jobs submitted prior to this outage will be held in queue for execution after the systems have been returned to service on January 10.

Please refer questions about this service interruption to rcac-help@purdue.edu.