BoilerGrid - Getting Started
- Overview of BoilerGrid
- Accounts on BoilerGrid
- File Storage and Transfer for BoilerGrid
- Compiling Source Code on BoilerGrid
- Running Jobs on BoilerGrid
Overview of BoilerGrid
BoilerGrid is a large, high-throughput, distributed computing system provided by RCAC and using the Condor system developed by the Condor Project at the University of Wisconsin. BoilerGrid provides a means for users to run programs on large numbers of otherwise idle computers in various locations, including both high-performance resources momentarily under-utilized and desktop lab machines not currently in use. Whenever a local user or scheduled job needs a given machine, the Condor job is stopped and sent to another Condor node as soon as possible. Because this model limits the ability to accomplish parallel processing and communications, RCAC decided to limit access to smaller, serial jobs. Condor jobs can be submitted from most of the RCAC systems (Gray, Pete, Prospero, Radon, Rossmann, Steele, Venice). You may also install Condor on your own desktop machine, and submit from that.
Detailed Hardware Specification
BoilerGrid scavenges cycles from nearly all RCAC systems, including community clusters, specialized systems, and the recycled cluster. BoilerGrid also uses idle time of machines in student labs on the Purdue West Lafayette campus, the Purdue Calumet campus and the University of Notre Dame. Whenever the normal scheduling system on these machines sends a job to a node, Condor preempts or (if possible) checkpoints its work, then immediately surrenders the node to the scheduled job.
BoilerGrid currently consists of over 20,000 processors. Of these, about 10,500 are Linux/x86_64, approximately 600 are Linux/Intel (ia32), and approximately 11,000 are WinNT51/Intel. There are also small numbers of Itanium Linux, Solaris and Mac OSX nodes. Memory on compute nodes ranges from 512 MB to 32 GB, and most processors run at 3 GHz or faster. With a total of over 60 TFLOPS available, BoilerGrid can provide large numbers of cycles in a short amount of time. All shared areas and software packages available on the RCAC systems are available on Condor. Condor is designed for high-throughput computing and is excellent for parameter sweeps, Monte Carlo simulations, or nearly any serial application.
| Owner | Arch/OS | Processors |
|---|---|---|
| ITaP - RCAC | x86_64/Linux | ~10500 |
| ITaP - RCAC | Intel/Linux | ~660 |
| ITaP - Envision Center | Intel/Linux | 48 |
| ITaP - Teaching & Learning | Intel/WinNTXX | ~9300 |
| Purdue Calumet | Intel/WinNT51 | ~250 |
| Notre Dame CSE | Intel/Linux, Sun4u/Solaris28, PPC/OSX, x86_64/Linux | ~230 |
| Purdue Biology, Libraries, & other ITaP | Intel/Linux, Intel/WinNT51 | 187 |
BoilerGrid currently runs the latest stable release of Condor: 7.0.1. BoilerGrid status may be monitored using CondorView.
Obtaining an Account
Purdue faculty, staff, and students with the approval of their advisor may request access to BoilerGrid using the online Research Computing Account Request Form. However, if you have an account on Radon or any of the RCAC Community Clusters (Steele, Venice, Rossmann, Prospero, Pete), you have automatically been given access to BoilerGrid already.
Login / SSH
To issue jobs on BoilerGrid, users may log in to the front-end host condor.rcac.purdue.edu via SSH, or submit using "condor_submit" directly from Radon or any of the Community Clusters (Steele, Venice, Rossmann, Prospero, Pete).
SSH Client Software
All access to the RCAC systems must be through secure (encrypted) connections. Standard telnet and FTP are not supported. SSH, SCP, and SFTP may be used instead.
Secure Shell or SSH is a way of establishing a secure channel between a local and a remote computer. It uses public-key cryptography to authenticate the remote computer and (optionally) to allow the remote computer to authenticate the user. It is usually used to log in to a remote machine and execute commands similar to telnet, but it also supports tunneling and forwarding of X11 or arbitrary TCP connections. The associated SFTP and SCP protocols may be used to transfer files. There are many SSH clients available, depending on the operating system you use.
Linux / Solaris / AIX / HP-UX / Unix:
- "ssh", "sftp", and "scp" are pre-installed. Log in using ssh myusername@servername.
Microsoft Windows:
- PuTTY is an extremely small download and includes free SSH, SCP, and SFTP clients.
- Secure CRT (SSH client) and Secure FX (SFTP client) are commercial clients which are available free to Purdue students, faculty, and staff with a Purdue career account.
Mac OS X:
- "ssh", "sftp", and "scp" are pre-installed. You may start a local terminal window from "Applications->Utilities". Log in using ssh myusername@servername.
- MacSSH and MacSFTP
- NiftyTelnet 1.1 SSH
SSH Keys
SSH can be used in conjunction with many different means of authentication. One popular authentication method is called Public Key Authentication (PKA). PKA is a method of establishing your identity to a remote computer using related sets of encryption data called keys. PKA is a more secure alternative to traditional password-based authentication with which you are probably familiar.
To employ PKA via SSH, you manually generate a keypair (also called SSH keys) in the location from where you wish to initiate a connection to a remote machine. This keypair consists of two text files, one which is called a private key and one which is called a public key. You keep the private key file confidential on your local machine or local home directory (hence the name "private" key). You then login to a remote machine (if possible) and append the corresponding public key text to the end of a specific file, or have a system administrator do so on your behalf. In future login attempts, the public and private keys are compared to verify your identity, which then grants you access to the remote machine.
As a user, you can create, maintain, and employ as many keypairs as you wish. If you connect to a computational resource from your work laptop, your work desktop, and your home desktop, you can create and employ keypairs on each. You can also create multiple keypairs on a single local machine to serve different purposes, such as establishing access to different remote machines, or establishing different types of access to a single remote machine. In short, PKA via SSH offers a secure but flexible means of identifying yourself to all kinds computational resources.
Passphrases and SSH Keys
When a you create a keypair, you are prompted to provide a passphrase for the private key. This passphrase is different than a password in a number of ways. First, a passphrase is, as the name implies, a phrase. It can include most types of characters, including spaces, and has no limits on length. Second, this passphrase is not transmitted to the remote machine for verification. It is used only to allow the use of your local private key and is specific to a specific local private key.
Perhaps you are wondering why you would need a private key passphrase at all when using PKA. If the private key is kept secure, why the need for a passphrase just to use it? Indeed, if the location of your private keys were always completely secure, a passphrase might not be needed. In reality, a number of situations could arise in which someone may improperly gain access to your private key files. In these situations, a passphrase offers another level of security for you, the user who created the keypair.
Think of the private key/passphrase combination as being analogous to your ATM card/PIN combination. The ATM card itself is the object that grants access to your important accounts, and as such, should be kept secure at all times—just as a private key should. But if you ever lose your wallet or your ATM card is stolen, you are glad that your PIN exists to offer you another level of protection. The same is true for a private key passphrase.
When you create a keypair, you should always provide a corresponding private key passphrase. For security purposes, avoid using phrases that would be guessed by automated programs (e.g. phrases that consist solely of words in English-language dictionaries). This passphrase can never be recovered if forgotten, so make note of it. There are only limited situations when the use of a non-passphrase-protected private key is warranted—conducting automated file backups is one such situation. If you need to use a non-passphrase-protected private key to conduct automated backups to Fortress, see the No-Passphrase SSH Keys section.
Passwords
If you have received a default password as part of the process of obtaining your account, you should change it immediately when you log on for the first time. This can be done from any terminal/SSH session with the command "passwd". You will have the same password on all RCAC systems. If you change your password on any one RCAC system, it will change on all RCAC systems.
If you already have a Purdue career account, then you will initially be given the same userid and password as your career account. There is no need to change your career account password because you have received an account on RCAC systems.
There is not currently any requirement regarding how often you must change your password within RCAC, but for security reasons changing a password every six months, preferably every three months, is good practice.
All passwords should:
- Be something you have never used as a password before, on this or any other system.
- Be easy for you to remember and difficult for others to guess.
- Be at least eight characters long.
- Be a combination of upper and lowercase letters, numbers, and symbols.
- TIP: Abbreviate a sentence or song lyric: "The dog Samson ate 4 new slippers!" = "TdSa4ns!"
Never share your password with another user or make your password known to anyone else. Systems staff will NEVER ask for your password, by email or otherwise.
Storage Options
File storage options on RCAC systems include home directories, scratch file systems, /tmp, and long-term or permanent storage. Each of these have different performance and intended uses, and some vary from system to system as well. Home directories and long-term storage are backed up nightly, but scratch and /tmp are not and may be occasionally purged without warning. Below is more detail about each of these storage options.
Home Directories
Your home directory is the default directory you are placed in when you log in.
You should use this space for storing files you want to keep long term such as source code, scripts, input data sets, etc. It should also be used for files you want to keep and which you use often. The home directory will physically reside on the BlueArc NFS Server. You can find the path to your home directory by logging in, and typing pwd:
$ pwd /home/ba01/u103/myusername
The second component of the reply indicates the name of the host where your home directory physically resides. In this example, the home directory is on the RCAC home directory file server named "ba01" under area "u103". This will vary from person to person. Remember, you can always check where your home directory is located by doing a pwd command in your home directory.
Regardless of its physical location, your home directory and its contents are available on almost all the RCAC front-end hosts and their nodes via the Network File System (NFS). The only exception is Black.
Note that your home directory has a quota capping the size and/or number of files you may store within. For more information, refer to the Storage Quotas / Limits Section.
Scratch Directories
Scratch directories are provided by RCAC and are intended for short-term file storage only.
Backups are not performed on scratch directories. In the event of a disk crash or file purge, files in scratch directories can not be recovered. Please be sure to copy any important files to more permanent storage.
All files stored in RCAC scratch directories older than 90 days will be automatically removed (purged). Owners of these files will be notified one week before removal via email. For more information, please refer to our Scratch File Purging Policy.
RCAC scratch directories are provided by a central BlueArc server and are accessible from most RCAC systems. There are two primary scratch file systems: scratch95 and scratch96. A scratch directory already exists for all BoilerGrid users. Your RCAC scratch directory is located under scratch95 or scratch96 within a subdirectory by the first letter of your username.
To find the path to your RCAC scratch directory, run myscratch:
$ myscratch /scratch/scratch96/m/myusername
The variable $RCAC_SCRATCH is also set to your RCAC scratch directory path. Use this variable in any scripts. Your actual scratch directory path may change without warning, but this variable will remain current.
$ echo $RCAC_SCRATCH /scratch/scratch96/m/myusername
To find the path to someone else's RCAC scratch directory, use the command findscratch:
$ findscratch someuser /scratch/scratch95/s/someuser
Note that your RCAC scratch directory has a quota capping the size and/or number of files you may store within. For more information, refer to the Storage Quotas / Limits Section.
/tmp Directory
The /tmp directory is intended for temporary files that are used during the execution of a process or job or while you examine files created by your jobs. Used properly, /tmp may provide faster local storage to an active process than any other storage option. However, do not use it for longer-term storage or critical results.
Files stored in /tmp are not backed up and are removed whenever space is low or whenever the system is rebooted. In the event of a loss, files in /tmp can not be recovered, so use it only for files that can be recreated relatively easily.
Long-Term Storage
Long-term Storage or Permanent Storage is available to RCAC users on the DXUL/UniTree archival storage system, commonly referred to as "Fortress". DXUL (DiskXtender for Unix and Linux) and UniTree are a software package that manages a hierarchical storage system. Program files, data files and any other files which are not used often, but which must be saved, can be put in permanent storage. Fortress currently has a 1.2 PB capacity. However, since two copies are retained for every file, the usable capacity is only 600 TB.
Recently used files smaller than 0.5 MB have their primary copy stored on low-cost disks, but the second copy is on tape or optical disks. This provides a rapid restore time to the disk cache. However, the large latency to access a larger file (usually involving a copy from a tape cartridge) makes it unsuitable for use as active storage.
In addition to poor performance, these two uses can cause severe problems with the system itself:
- DO NOT store any actively used files on Fortress.
- DO NOT store large collections of small files on Fortress.
Do not use Fortress as a second home directory. Instead, use tar or some similar archive tool to combine all the smaller files you wish to store into a single large file first.
For active data storage you should use either local storage or a scratch file system. You may then copy any results you wish to archive to Fortress when computation is complete.
Fortress is directly accessible (via FTP, SSH, SCP, SFTP, and NFS) from all RCAC systems, as well as most systems in ECN and CS and from several other major servers on campus. To access Fortress in any way other than NFS, you must login to fortress.rcac.purdue.edu. RCAC has more information about Fortress, including how to obtain a Fortress account and how to access your files on Fortress.
Environment Variables
There are many environment variables related to storage locations and paths which are automatically set for you upon log in, or may be changed if necessary.
Use environment variables instead of actual paths whenever possible to avoid problems if the specific paths to any of these change. Some of the environment variables you should have are:
- $USER: your username
- $HOME: path to your home directory
- $PWD: path to your current directory
- $RCAC_SCRATCH: path to scratch filesystem
- $PATH: all directories searched for commands/applications
- $HOSTNAME: name of the machine you are on
- $SHELL: your current shell (bash, tcsh, csh, ksh)
- $SSH_CLIENT: your local client's IP address
- $TERM: type of terminal or terminal emulator being used
- $OMP_NUM_THREADS: OpenMP number of threads
All environment variables begin with the dollar sign ($) and are all uppercase. These may be used on the command line or in any scripts in place of and in combination with hard-coded values:
$ ls $HOME ... $ ls $RCAC_SCRATCH/myproject ... $ ls $RCAC_SCRATCH/myproject/$HOSTNAME_data ...
You may find the value of any environment variable by using the echo command:
$ echo $RCAC_SCRATCH /scratch/scratch95/m/myusername $ echo $SHELL /usr/local/bin/tcsh
You may list the values of all environment variable using the env command:
$ env USER=myusername HOME=/home/ba01/u101/myusername RCAC_SCRATCH=/scratch/scratch95/m/myusername SHELL=/usr/local/bin/tcsh ...
You may create or overwrite an environment variable using either export or setenv, depending on your shell:
(for bash and sh) $ export VARIABLE=value (for tcsh and csh) % setenv VARIABLE value
Storage Quotas / Limits
Your disk usage is limited on RCAC systems. However, each filesystem (scratch, home directory, etc.) may have a different limit. If you exceed the soft limit or quota, you will see warnings whenever writing to the disk that you are over quota, but the write will still succeed. If you exceed the hard limit or limit, your write will fail until you either remove other files or your quota is increased. Generally, RCAC systems do not impose a soft limit—only a hard limit.
Checking Quota Usage
You may find out what your current quota is by using the quota command:
$ quota
Disk quotas for user myusername (uid 12345):
Filesystem blocks quota limit grace files quota limit grace
ba01:/u103 2346272 0 5000000 17508 0 65535
The columns are as follows:
- Filesystem: This indicates the line is for the user's files on /u103/, which doing echo $HOME confirms is the user's home directory filesystem.
- Blocks: This shows how many 1 KB blocks the user's files take up. In this case, 2346272 KB / 1024 = 2291 MB, or 2291 MB / 1024 = 2.24 GB.
- Quota: This shows that soft limits are not being imposed (0).
- Limit: This shows how many 1 KB blocks the user's hard limit is. In this case, (5000000 KB / 1024) / 1024 = 4.77 GB.
- Grace: This would show the grace period (in days) for any soft limit (none in this case).
- Files: This shows how many file pointers (inodes) the user is currently using. This is based more on the number of files and directories and not the size.
- Quota: This shows that soft limits are not being imposed for file pointers (0).
- Limit: This shows the user's file pointer hard limit. It is possible, though unlikely, to hit this and not the size limit if you create a large number of very small files.
- Grace: This would show the grace period (in days) for any file pointer soft limit (none in this case).
You may also see the disk usage of any given directory by using du:
$ du -hs 1.1G . $ du -hs $HOME 138M /home/ba01/u103/myusername
This can be very helpful in figuring out where your largest files or directories are, so that you may clean out unneeded large files and avoid hitting your quota.
Requesting Quota Increase
If you find you need additional disk space on an RCAC account, please first consider archiving and compressing old files and moving them to long-term storage. If this option does not resolve the issue, you may send an email to rcac-help@purdue.edu and request additional space.
Archive and Compression
There are several options for archiving and compressing groups of files or directories on RCAC systems. All of the following tools are provided:
-
zip
(more information)
Simple compression and file packaging utility.
Examples:(compress file somefile.c) $ zip somefile.zip somefile.c (extract contents of somefile.zip) $ unzip somefile.zip (compress all files in a directory into one archive file) $ zip -r somefile.zip somedirectory/ (compress all ".c" files in current directory into one archive file) $ zip -r somefile.zip . -i \*.c
-
tar
(more information)
Saves many files together into a single archive file, and restores individual files from the archive. Includes automatic archive compression/decompression options and special features that allow tar to be used for incremental and full backups.
Examples:(archive file somefile.c) $ tar cvf somefile.tar somefile.c (archive and compress file somefile.c) $ tar czvf somefile.tar.gz somefile.c (list contents of archive somefile.tar) $ tar tvf somefile.tar (extract contents of somefile.tar) $ tar xvf somefile.tar (extract contents of gzipped archive somefile.tar.gz) $ tar xzvf somefile.tar.gz (archive and compress all files in a directory into one archive file) $ tar czvf somefile.tar.gz somedirectory/ (archive and compress all ".c" files in current directory into one archive file) $ tar czvf somefile.tar.gz *.c
-
gzip
(more information)
Compression utility designed as a replacement for compress, with much better compression and no patented algorithms. The standard compression system for all GNU software.
Examples:(compress file somefile - also removes uncompressed file) $ gzip somefile (uncompress file somefile.gz - also removes compressed file) $ gunzip somefile.gz
-
bzip2
(more information)
Strong, lossless data compressor based on the Burrows-Wheeler transform. Also available as a library.
Examples:(compress file somefile - also removes uncompressed file) $ bzip2 somefile (uncompress file somefile.bz2 - also removes compressed file) $ bunzip2 somefile.bz2
-
compress
(more information)
Adaptive Lempel-Ziv compressor. Not often used today.
Windows users can work with these same formats using some of the following software:
-
7-Zip
Free Windows software package that can handle all the above formats. -
WinZip
Commercial Windows software package that can handle all the above formats. -
WinRAR
Commercial Windows software package that can handle all the above formats.
File Transfer
There are a variety of ways to transfer data to and from RCAC systems. Which you should use depends on several factors, including the ease of use for you personally, connection speed and bandwidth, the size and number of files to be transferred. For more details on file transfer methods and applications, refer to the BoilerGrid Complete User Guide.
Provided Compilers
The compilers available on the Community Clusters (Steele, Venice, Rossmann, Prospero, Pete) and the Recycled Cluster (Radon) can all be used to compile code for Condor. They are: Intel, GNU, and PGI compilers for Fortran 77, Fortran 90, Fortran 95 (only PGI), C, and C++. The compilers can produce general-purpose and architecture-specific optimizations to improve performance. These include loop-level optimizations, inter-procedural analysis and cache optimizations. The compilers support automatic and user-directed parallelization of Fortran, C, and C++ applications for multiprocessing execution.
Aside from these compilers, the command condor_compile can be used to compile code that is to be relinked with the Condor libraries for submission into Condor's Standard Universe. The Condor libraries provide the program with additional support, such as the capability to checkpoint, which is required in Condor's Standard Universe mode of operation. condor_ compile requires access to the source or object code of the program to be submitted; if source or object code for the program is not available (i.e. only an executable binary, or if it is a shell script), then the program must be submitted into Condor's Vanilla Universe.
The command for condor_compile is:
condor_compile [compiler] -O -o myprogram.condor file1.suffix file2.suffix ... where [compiler] is one of cc (the system C compiler) CC (the system C++ compiler) f77 (the system FORTRAN compiler) gcc (the GNU C compiler) g++ (the GNU C++ compiler) g77 (the GNU FORTRAN compiler) f90 (the system FORTRAN 90 compiler)
Notice from the preceding example that only the GNU compilers are compatible with condor_compile and thus the Standard Universe. To load the GNU compilers, type module load gcc.
Compiler commands for the ordinary compilers
This section looks at just compiling with the standard C/C++ and Fortran compilers, as opposed to compiling with condor_compile. When you are not compiling with condor_compile you will have to submit to the Vanilla Universe, not the Standard Universe. One reason for choosing not to use condor_compile and the Standard Universe is that we might wish to take advantage of the features of a compiler which is not compatible with condor_compile. To load the Intel compilers, type module load intel; to load the GNU compilers, type module load gcc; and to load the PGI compilers type module load pgi.
Example: To load the Intel C/C++/Fortran compilers
-bash-3.00$ module load intel -bash-3.00$
There are several versions installed for most of the compilers. Typing module avail will show all the modules possible to load.
This table below shows the different compiler names.
Note about reading the tables: for shortness, I have collected the commands for the Intel, GNU, and PGI compilers in the same table. They are shown like this (Ex: Serial Fortran 77): ifort/g77/pgf77. This means that you should use ifort to compile with Intel, g77 to compile with GNU, and pgf77 to compile with PGI. A '-' means that the compiler is not available for that type of compiler.
| Serial Intel/GNU/PGI | |
|---|---|
| Fortran 77 | ifort/g77/pgf77 |
| Fortran 90 | ifort/gfortran/pgf90 |
| Fortran 95 | ifort/gfortran/pgf95 |
| C | icc/gcc/pgcc |
| C++ | icc/g++/pgCC |
Here is a list of man pages for the various compilers. They can also be seen by issuing the command man <compiler> (only when said compiler has been loaded with the module load command). They will contain various useful information, such as compiler/linker options.
- ifort (Intel Fortran compiler)
- icc (Intel C/C++ compiler)
- gfortran (GNU Fortran compiler)
- gcc (GNU C compiler)
- g++ (GNU C++ compiler)
- pgf77 (PGI Fortran 77 compiler)
- pgf90 (PGI Fortran 90 compiler)
- pgf95 (PGI Fortran 95 compiler)
- pgcc (PGI C compiler)
- pgCC (PGI C++ compiler)
This table shows some examples on how to compile a program. Note that not all compilers are available on all systems.
| Program Type | Intel | GNU | PGI |
|---|---|---|---|
| Fortran 77 serial program | ifort program.f -o program | g77 program.f -o program | pgf77 program.f -o program |
| Fortran 90 serial program | ifort program.f90 -o program | gfortran program.f90 -o program | pgf90 program.f90 -o program |
| Fortran 95 serial program | icc myprogram.c -o myprogram | gfortran program.f95 -o program< | pgf95 program.f95 -o program< |
| C serial program | icc program.c -o program | gcc program.c -o program | pgcc program.c -o program |
| C++ serial program (1) | icc program.cpp -o program | g++ program.cpp -o program | pgCC program.cpp -o program |
(1) the suffix of a C++ file may be .C, .c, .cc, .cpp, .cxx, or .c++.
Here is a list of "getting started" guides on the various compilers etc.:
- Intel Fortran Language Manual [PDF]
- Intel Fortran Libraries Reference [PDF]
- Intel C++ Compiler documentation [ZIP]
- Getting started with Intel C++
- Getting started with Intel Fortran
- condor_compile
Running Jobs on BoilerGrid
Only serial jobs can currently be submitted to BoilerGrid.
Running Jobs via Condor
Condor is one of several distributed computing resources RCAC provides. Like other similar resources, Condor provides a framework for running programs on otherwise idle computers. While this has serious limitations for parallel jobs and codes with serious I/O or memory requirements, Condor can provide a large quantity of cycles for researchers who need to run hundreds of smaller jobs.
Condor is a specialized batch system for managing compute-intensive jobs. Condor provides a queuing mechanism, scheduling policy, priority scheme, and resource classifications. Users submit their jobs to Condor, which then puts these jobs in a queue, runs them, and reports back with the results.
In some ways, Condor is different from other batch systems. They usually only operate on dedicated machines/compute servers. Instead, Condor can both schedule jobs on dedicated machines and effectively utilize non-dedicated machines to run jobs. It only runs jobs on machines which are currently not being used (no keyboard activity, no load average, no active telnet users, etc). In this way, Condor effectively harnesses otherwise idle machines throughout a pool of machines.
Currently we use Condor to collect idle cycles on all i386-compatible Linux-based computational resources which RCAC maintains. Condor also runs on the Linux Clusters, both community (Steele) and recycled (Radon). All of these resources are scheduled with PBS; however, when a node is not running a PBS job, it is free to execute Condor jobs. When PBS elects to run a new job on such a node, any active Condor job is immediately checkpointed (if possible) and removed from the node. Condor jobs can be submitted from most of the RCAC systems (Gray, Pete, Prospero, Radon, Rossmann, Steele, Venice). You may also install Condor on your own desktop machine, and submit from that.
See here for a more detailed description of the resources in the Condor pools.
The status of the Condor pools can always be monitored with CondorView.
Look here for Condor tutorials and slides from the 'Condor Boot Camp' that was held at Purdue University. They are very good and useful.
Most of the information in this manual is taken from either the Condor Version 6.8.0 Manual or the man pages for the various commands.
BoilerGrid Condor Tips
- Do not queue up thousands and thousands of jobs in a queue. Submit fewer jobs at a time or use DAGman to divide your jobs into reasonably-sized chunks. (500 runs or so)
- When a submit node is heavily used, do not run condor_q constantly. The condor_schedd is single-threaded, and schedules work in the same thread that you are using to list the queue, so do not take resources away from the scheduler.
- Long jobs should run in the Standard Universe, not in the Vanilla Universe, since they will otherwise never finish.
- Vanilla Universe can use Intel compilers (may run 30%-40% faster). Thus it may even be faster for somewhat longer jobs, because the speed gain may be bigger than the advantage from the checkpoint availability.
- All Standard Universe executables must be statically linked since we cannot be certain that the dynamic libraries on all machines in the flock will be the same version. That way we will be sending along the same run-time to all machines. There is also the problem that you can land on a system that is not even the same version as your build system. Static linkage for Vanilla Universe jobs helps to protect against problems here but we have seen a case or two where the execution machines were far enough out of synch that an application would not run. Thus, use static linkage if at all possible. The condor_compile command specifies static linkage as part of its arguments to the linker. they are shown by condor_compile in the 'LINKING FOR' message.
- Generally, if the execution of your job runs less than 1/2 hour, then there is almost no eviction. If it is shorter than 1 hour, there will still only be a few evictions.
- Purdue has both a scavenging/preempting and a scheduling system. Remember that the Condor pool is very heterogeneous, both regarding processor versions and OS versions/types (both Linux of different varieties and some Windows.)
- It is a good idea to use static links regardless of universe, since you will never know which version of SUN libraries etc. you will find on a given machine.
- Why no middleware (like Mycluster at TACC)? Middleware can be easier for the user, since it does Condor (and other stuff) 'behind the scene'. Middlewares are schedulers also and will not start the job until it is guaranteed to run to completion (no eviction). However, it has a lot of job restarts and thus much overhead on many jobs. Therefore, for a large number of jobs the 'pure' Condor (Condor without middleware) is better.
BoilerGrid Condor Submission Script
Example 1
Here is first the simplest possible job submission file. It will put one copy of the program hello (which has first been created by condor_compile) in queue for execution by Condor. There is no definition of platform, so Condor will just use its default, which is to run the job on a machine which has the same architecture and operating system as the machine from which it was submitted.
No input, output, and error commands are given in the job submission file, so the files stdin, stdout, and stderr will all refer to /dev/null (a.k.a. the null device. It is a special file that discards all data written to it, but reports that the write operation succeeded. It provides no data to any process that reads from it - returning EOF). The program may produce output by explicitly opening a file and writing to it. A log file, hello.log, will also be produced. This log file will contain events the job had during its lifetime inside of Condor, such as any possible errors. When the job finishes, its exit conditions will also be noted in the log file. It is recommended that you always have a log file so you know what happened to your jobs.
If your program only returns output to the screen (like the hello.c program below does), then you should include Output = hello.out or something like it somewhere before Queue. Otherwise you will not see the output.
The default universe is the Standard Universe. This is what will be assumed if you do not explicitly chose a universe. This is a problem if you have not compiled with condor_compile and thus included the Condor libraries. If you have a program that is compiled with the normal compilers and thus does not have the Condor libraries linked in, then you should run in the Vanilla Universe.
####################
#
# Example 1
# Simple condor job description file
#
####################
Executable = hello
Log = hello.log
Queue
Example 2
In this example (from the Condor manual), we queue two copies of the program Mathematica. The first copy will run in directory run_1, and the second will run in directory run_2. For both queued copies, stdin will be test.data, stdout will be loop.out, and stderr will be loop.error. There will be two sets of files written, as the files are each written to their own directories. This is a convenient way to organize data if you have a large group of Condor jobs to run. The example file shows program submission of Mathematica as a Vanilla Universe job. This may be necessary if the source and/or object code to program Mathematica is not available.
Condor recommends using a single log file.
####################
#
# Example 2: demonstrate use of multiple
# directories for data organization.
#
####################
Executable = mathematica
Universe = Vanilla
Input = test.data
Output = loop.out
Error = loop.error
Log = loop.log
Initialdir = run_1
Queue
Initialdir = run_2
Queue
Example 3
In this example (also from the Condor manual, the job submission file queues 150 runs of program foo which has been compiled and linked for Silicon Graphics workstations running IRIX 6.5. This job requires Condor to run the program on machines which have greater than 32 megabytes of physical memory, and expresses a preference to run the program on machines with more than 64 megabytes, if such machines are available. It also advises Condor that it will use up to 28 megabytes of memory when running. Each of the 150 runs of the program is given its own process number, starting with process number 0. So, files stdin, stdout, and stderr will refer to in.0, out.0, and err.0 for the first run of the program, in.1, out.1, and err.1 for the second run of the program, and so forth. A log file containing entries about when and where Condor runs, checkpoints, and migrates processes for the 150 queued programs will be written into file foo.log.
#################### # # Example 3: Show off some fancy features including # use of pre-defined macros and logging. # #################### Executable = foo Requirements = Memory >= 32 && OpSys == "IRIX65" && Arch =="SGI" Rank = Memory >= 64 Image_Size = 28 Meg Error = err.$(Process) Input = in.$(Process) Output = out.$(Process) Log = foo.log Queue 150
Condor Job Submission
To submit a job to Condor for execution, you must use the condor_submit command. This command takes as an argument the job submission file. As described above, this file contains the commands and keywords used to direct the queuing of jobs - the name of the executable to run, which universe to run in, any requirements and rank info, how many times to run the program, any command line arguments, etc. Based on this information, condor_submit will create a job ClassAd to use for matching with a machine ClassAd. When this has been done, Condor can queue the job for running on that machine.
One of the many advantages of using a job description file involves running the same program many times, each time with a different input data set (say, 500 times with 500 different input data sets). It is then easy to tell Condor to do this. Just arrange your data files accordingly so that each run reads its own input, and each run writes its own output. Each individual run may have its own initial working directory, stdin, stdout, stderr, command line arguments, and shell environment. A program that directly opens its own files will read the file names to use either from stdin or from the command line. A program that opens a static filename every time will need to use a separate subdirectory for the output of each run.
Write a job submission file and submit it:
condor_submit file
Example (my job submission file is called run_hello):
condor_submit run_hello
See condor_submit in the manual pages, for a more complete description of how to use it.
Condor Job Status
To just see the status of a job, type condor_q. Since there will often be many jobs scheduled at the same time, using condor_q <username> will limit the output to those jobs scheduled by <username>.
Example:
radon-fe00:~/condor_running$ condor_q <username> -- Submitter: radon-fe00.rcac.purdue.edu : <128.211.157.42:42163> : radon-fe00.rcac.purdue.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 1100900.0 <username> 2/20 15:13 0+00:00:00 I 0 0.0 Hello 1 jobs; 1 idle, 0 running, 0 held radon-fe00:~/condor_running$
Affecting the job's execution
Placing a job on hold
There are many reasons to put a job on hold. One could be if you do not have enough space to hold all the results at the same time, but need to move those somewhere else. You could queue all jobs, but put them on hold immediately. Then release a few job at a time (with a -constraint to condor_release, can be scripted), and move the results as they are produced, then release some more. To place a job in the queue on hold, use the command condor_hold <jobid>.
Releasing a held job
A job that is in the hold state remains there until later released for execution by the command condor_release. If a running job is placed on hold, it is killed and put back in the queue (assuming Vanilla jobs without checkpointing). A job still in the queue will stay there until released. Jobs can be held manually or may get held by the Condor Scheduler for various reasons (unable to write to your directory, etc.)condor_release jobid or condor_release user
Example
radon-fe00:~/condor_running$ condor_q <username> -- Submitter: radon-fe00.rcac.purdue.edu : <128.211.157.42:42163> : radon-fe00.rcac.purdue.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 1101790.0 <username> 2/24 14:53 0+00:00:00 I 0 0.0 Hello 1 jobs; 1 idle, 0 running, 0 held radon-fe00:~/condor_running$ condor_hold 1101790.0 Job 1101790.0 held radon-fe00:~/condor_running$ condor_q <username> -- Submitter: radon-fe00.rcac.purdue.edu : <128.211.157.42:42163> : radon-fe00.rcac.purdue.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 1101790.0 <username> 2/24 14:53 0+00:00:00 H 0 0.0 Hello 1 jobs; 0 idle, 0 running, 1 held radon-fe00:~/condor_running$ condor_release 1101790.0 Job 1101790.0 released radon-fe00:~/condor_running$ condor_q <username> -- Submitter: radon-fe00.rcac.purdue.edu : <128.211.157.42:42163> : radon-fe00.rcac.purdue.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 1101790.0 <username> 2/24 14:53 0+00:00:00 I 0 0.0 Hello 1 jobs; 1 idle, 0 running, 0 held radon-fe00:~/condor_running$
See the manual page for more information.
Checking on the progress of jobs
To check on the status of your jobs, use the command condor_q. This command will display the status of all the queued jobs, not just your own.
That is, however, not the only way of tracking the progress of your jobs. Another way of doing this is through the user log. In your job submission file, you can specify a log command (by adding Log = <name>.log somewhere before the Queue command). When you have done this, the progress of the job may be followed by viewing the log file. Various events such as execution commencement, checkpoint, eviction and termination are logged in the file. Also logged is the time at which the event occurred.
As soon as your job begins executing, Condor will start up a condor_shadow process on the submit machine. The shadow process is the mechanism by which the remotely executing jobs can access the environment from which it was submitted, such as input and output files. It is normal for a machine which has submitted hundreds of jobs to have hundreds of shadows running on the machine. Since the text segments of all these processes is the same, the load on the submit machine is usually not significant. If, however, you notice degraded performance, you can limit the number of jobs that can run simultaneously through the MAX_JOBS_RUNNING configuration parameter. Please talk to your system administrator for the necessary configuration change.
To find all the machines which are running your job, use the command condor_status. Example: say you wish to find all the machines which runs jobs submitted by user123@purdue.edu. You would then type condor_status -constraint 'RemoteUser == "user123@rcac.purdue.edu"'.
user123@radon-fe00:~$ condor_status -constraint 'RemoteUser == "user123@rcac.purdue.edu"' Name OpSys Arch State Activity LoadAv Mem ActvtyTime ba-005.rcac.p LINUX INTEL Claimed Busy 1.000 502 0+00:24:44 ba-006.rcac.p LINUX INTEL Claimed Busy 0.990 502 0+00:20:22 ba-007.rcac.p LINUX INTEL Claimed Busy 1.000 502 0+00:23:16 ba-008.rcac.p LINUX INTEL Claimed Busy 1.000 502 0+00:30:20 ...
Status of Machines
Condor allocates resources by matching the submitted jobs with the machines. It does this by matching ClassAds. Condor's ClassAds are analogous to the classified advertising section of the newspaper. Sellers/buyers advertise specifics about what they have to sell/want to buy. Both buyers and sellers have some constraints which must be satisfied, like buyers only being able to pay a certain sum of money or sellers asking for no less than a certain price. Sellers and buyers both want to rank requests to their own advantage, for example, the seller would give a higher rank to a higher price offer. In Condor, users submitting jobs can be thought of as buyers of compute resources and machine owners are sellers.
All the machines in a Condor pool advertise their attributes. These could be available RAM memory, CPU type, CPU speed, virtual memory size, current load average, or other static and dynamic properties. This machine ClassAd also advertises under what conditions it is willing to run a Condor job and what type of job it would prefer.
The owners who allow their machines to be part of the Condor pool may set individual terms and preferences - maybe specifying that their machines may be used to run jobs only at night or that they have a preference/higher rank for running jobs submitted by their own department.
A very useful program for finding out which machines and architectures are out there is the program condor_all. It should be noted that even though it is located in the "official" Condor directory - /opt/condor/bin, it is a locally (Purdue) developed tool. It is very handy for finding out how many machines of a certain architecture are available - useful for the job submission file. The program is in the default path on tg_login, but is installed on most RCAC resources where it can be run as /opt/condor/bin/condor_all.
Just as the machines have requirements and preferences, the same is true for the users submitting a job. The users specify a ClassAd with their requirements and preferences when they submit a job. This ClassAd includes the type of machine you wish to use - you would perhaps like to use the machine with the fastest floating-point performance available and you thus want Condor to rank the available machines based upon their floating-point performance.
Another example could be that your job requires a machine with a minimum of, say, 4 GB of RAM and you thus only want Condor to consider machines which fulfill this requirement.
Sometimes, the user may be ready to use any machine available and this too can be communicated to Condor through the job ClassAd.
Condor's job then is to read all the machine ClassAds and all the user job ClassAds and match them up. Condor makes certain that all requirements in both ClassAds are satisfied, if possible.
To get a feel for what a machine ClassAd does, try typing the commands condor_status. This will give you a summary of the information in the resource ClassAds in your Condor pool. Click here to see an example of running this command. The list was generated by running condor_status on Radon, August 25, 2006.
There are many attributes. Some of them are used by Condor for scheduling. Other attributes are for information purposes. An important point is that any of the attributes in a machine ad can be utilized at job submission time as part of a request or preference on what machine to use. Additional attributes can be easily added. For example, your site administrator can add a physical location attribute to your machine ClassAds.
Condor Job Cancellation
Removing a job from the queue
The command condor_rm can be used at any time to remove a job from the queue. If the job has already started running, then the job will be killed without a checkpoint, and its queue entry is removed. Use condor_q to get the ID of the job. Here is an example:
Queue of jobs before:
user123@radon-fe00:~$ Submitter: radon-fe00.rcac.purdue.edu : <128.210.9.35:35407> : radon-fe00.rcac.purdue.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD ... 260076.7 nice-user.user1 8/18 00:05 0+00:21:47 I 0 29.3 startfah.sh -oneun 260076.9 nice-user.user1 8/18 00:05 0+01:40:44 I 0 136.7 startfah.sh -oneun 260185.0 user123 8/30 13:01 0+00:00:00 R 0 19.5 hello ...
Queues of jobs after:
user123@radon-fe00:~$ condor_rm 260185.0 Job 260185.0 marked for removal user123@radon-fe00:~$ condor_q Submitter: radon-fe00.rcac.purdue.edu : <128.210.9.35:35407> : radon-fe00.rcac.purdue.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD ... 260076.7 nice-user.user1 8/18 00:05 0+00:21:47 I 0 29.3 startfah.sh -oneun 260076.9 nice-user.user1 8/18 00:05 0+01:40:44 I 0 136.7 startfah.sh -oneun ...
Condor DAG (Workflows)
DAG means Directed Acyclic Graph and is a way of submitting many jobs at the same time. DAGMan is the Directed Acyclic Graph Manager. In short, DAGMAn, lets you submit complex sequences of jobs as long as they can be expressed as a directed acylic graph. For example, you may wish to run a large parameter sweep but before the sweep run you need to prepare your data. After the sweep runs, you need to collate the results. This might look like this, assuming you want to sweep over five parameters:
DAGMan has many abilities, such as throttling jobs, recovery from failures, and more. More information about DAGMan can be found in the Condor manual.
Submitting a simple DAG
As an example, consider the DAGMan submit file dagfile.dag:
Job 1 testrun1.submit Job 2 testrun2.submit Job 3 testrun3.submit PARENT 1 CHILD 2 PARENT 2 CHILD 3
We then have Condor submit files for each part, this could be 'testrun1.submit' (in the below example the DAG is submitted across the grid - you can just change universe and remove globusscheduler for a local submission):
Universe = globus Executable = testrun1 Transfer_Executable = true Globusscheduler = tg-login1.ncsa.teragrid.org/jobmanager Output = testrun1.out Error = testrun1.error Log = testrun1.log Queue
The files testrun2.submit and testrun3.submit would be similar, with just 1 changed to 2 or 3. Create the file testrun1, testrun2, testrun3. Your directory should contain the following files:
cu12:~/dagtest238% ls dagfile.dag testrun1.submit testrun2.submit testrun3.submit testrun1 testrun2 testrun3 cu12:~/dagtest239%
To submit this DAG, give the command:
condor_submit_dag dagfile.dag
This gives the output:
cu12:~/dagtest239% condor_submit_dag dagfile.dag Checking all your submit files for log file names. This might take a while... Done. ----------------------------------------------------------------------- File for submitting this DAG to Condor : dagfile.dag.condor.sub Log of DAGMan debugging messages : dagfile.dag.dagman.out Log of Condor library debug messages : dagfile.dag.lib.out Log of the life of condor_dagman itself : dagfile.dag.dagman.log Condor Log file for all jobs of this DAG : /u/ncsa/user123/dagtest/testrun1.log Submitting job(s). Logging submit event(s). 1 job(s) submitted to cluster 58. ----------------------------------------------------------------------- cu12:~/dagtest240%
Just as for the ordinary condor_submit, the status of the job can be checked with condor_q and a job can be removed with condor_rm.
There are some examples of using DAG here and here.
The manual page for Condor DAG can be found here.
Click here to see another example of submitting a simple DAG.
Click here to see an example of a somewhat more complex DAG.
Click here to see an example of handling a DAG that fails.
Condor Examples
These instructions are very short and merely meant to give you the ability to run a small example immediately. Read the rest of the sections, and maybe the Condor manual for more details on how to use other features of Condor.
Simple Condor Example
Compiling
condor_compile <compiler> <program>.<extension> -o <program name>
Example:
condor_compile gcc hello.c -o hello
Submitting
It is very simple to submit the job to Condor, when the job submission file has been written. At the command prompt, just type condor_submit <job-name>, where job-name is the name of the job submission file.
Example: Here I am using a very simple job submission file, namely:
Executable = hello Log = hello.log Output = hello.out Queue
Where hello is a C program which have already been compiled with the command condor_compile gcc hello.c -o hello. I have named this job submission file 'run_hello'. In the following, I am running on Radon:
user123@radon-fe00:~$ condor_submit run_hello Submitting job(s). Logging submit event(s). 1 job(s) submitted to cluster 260182. user123@radon-fe00:~$
It may take a (sometimes long) while before the job is submitted and finishes running, depending on how many others are using the machines, your rank, the requirements you have given for the job, etc. The progress can be checked with the command condor_status. When the job has completed, I have the two files hello.log and hello.out in my directory - just as I asked for in the job submission file. You should always use a log file.
The contents of the files are:
hello.log:
000 (260182.000.000) 08/29 16:21:31 Job submitted from host: <128.210.9.35:35407> ... 001 (260182.000.000) 08/29 16:22:42 Job executing on host: <128.211.131.51:32780> ... 005 (260182.000.000) 08/29 16:22:42 Job terminated. (1) Normal termination (return value 13) Usr 0 00:00:00, Sys 0 00:00:00 - Run Remote Usage Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage Usr 0 00:00:00, Sys 0 00:00:00 - Total Remote Usage Usr 0 00:00:00, Sys 0 00:00:00 - Total Local Usage 830 - Run Bytes Sent By Job 13490672 - Run Bytes Received By Job 830 - Total Bytes Sent By Job 13490672 - Total Bytes Received By Job ...
and
hello.out:
Hello World!
which was the output the program would otherwise have written to the screen. You will also receive an email, unless otherwise specified.
Click here for another simple example.
Click here for an example of submitting a Standard Universe job.
Click here for an example of submitting a Java Universe job.