Other Materials
Implementing an Industrial-Strength Academic Cyberinfrastructure at Purdue University
Implementing a Central Quill Database in a Large Condor Installation (Condor Week 2008)
BoilerGrid for cyro-EM image processing (Condor Week 2008)
Examples, file transfer
The program executable is called my_program, and it uses three command-line arguments as it executes: two input file names and an output file name. The program executable and the submit description file for this job are located in a directory test, under a user home directory.The directory tree for all these examples:
/userhomedir/test (directory)
my_program.condor (the submit description file)
my_program (the executable)
files (directory)
logs2 (directory)
in1 (file)
in2 (file)
logs (directory)
Example 1: This example explicitly transfers input files. The input files to be transferred are specified relative to the directory where the job is submitted. The single output file, out1, created when the job is executed will be transferred back into the directory /userhomedir/test, not the files directory. # file name: my_program.condor # Condor submit description file for my_program Executable = my_program Universe = vanilla Error = logs/err.$(cluster) Output = logs/out.$(cluster) Log = logs/log.$(cluster) should_transfer_files = YES when_to_transfer_output = ON_EXIT transfer_input_files = files/in1, files/in2 Arguments = in1 in2 out1 QueueExample 2: This second example is identical to Example 1, except that absolute paths to the input files are specified, instead of relative paths to the input files.
# file name: my_program.condor # Condor submit description file for my_program Executable = my_program Universe = vanilla Error = logs/err.$(cluster) Output = logs/out.$(cluster) Log = logs/log.$(cluster) should_transfer_files = YES when_to_transfer_output = ON_EXIT transfer_input_files = /userhomedir/test/files/in1, /userhomedir/test/files/in2 Arguments = in1 in2 out1 QueueExample 3: This third example illustrates the use of the submit command initialdir, and its effect on the paths used for the various files. The expected location of the executable is not affected by the initialdir command. All other files (specified by input, output, transfer_input_files, as well as files modified or created by the job and automatically transferred back) are located relative to the specified initialdir. Therefore, the output file, out1, will be placed in the files directory. Note that the logs2 directory exists to make this example work correctly.
# file name: my_program.condor # Condor submit description file for my_program Executable = my_program Universe = vanilla Error = logs2/err.$(cluster) Output = logs2/out.$(cluster) Log = logs2/log.$(cluster) initialdir = files should_transfer_files = YES when_to_transfer_output = ON_EXIT transfer_input_files = in1, in2 Arguments = in1 in2 out1 QueueExample 4: Illustrates an Error. This example illustrates a job that will fail. The files specified using the transfer_input_files command work correctly (see Example 1). However, relative paths to files in the arguments command cause the executing program to fail. The file system on the submission side may utilize relative paths to files, however those files are placed into a single, flat, temporary directory on the execute machine.
Note that this specification and submission will cause the job to fail and reexecute.
# file name: my_program.condor # Condor submit description file for my_program Executable = my_program Universe = vanilla Error = logs/err.$(cluster) Output = logs/out.$(cluster) Log = logs/log.$(cluster) should_transfer_files = YES when_to_transfer_output = ON_EXIT transfer_input_files = files/in1, files/in2 Arguments = files/in1 files/in2 files/out1 QueueThis example fails with the following error:
err: files/out1: No such file or directory.Example 5: Illustrates an Error. As with Example 4, this example illustrates a job that will fail. The executing program's use of absolute paths cannot work.
# file name: my_program.condor # Condor submit description file for my_program Executable = my_program Universe = vanilla Error = logs/err.$(cluster) Output = logs/out.$(cluster) Log = logs/log.$(cluster) should_transfer_files = YES when_to_transfer_output = ON_EXIT transfer_input_files = /username/test/files/in1, /username/test/files/in2 Arguments = /username/test/files/in1 /username/test/files/in2 /username/test/files/out1 QueueThe job fails with the following error:
err: /username/test/files/out1: No such file or directory.Example 6: Illustrates an Error. This example illustrates a failure case where the executing program creates an output file in a directory other than within the single, flat, temporary directory that the program executes within. The file creation may or may not cause an error, depending on the existence and permissions of the directories on the remote file system.
Further incorrect usage is seen during the attempt to transfer the output file back using the transfer_output_files command. The behavior of Condor for this case is undefined.
# file name: my_program.condor # Condor submit description file for my_program Executable = my_program Universe = vanilla Error = logs/err.$(cluster) Output = logs/out.$(cluster) Log = logs/log.$(cluster) should_transfer_files = YES when_to_transfer_output = ON_EXIT transfer_input_files = files/in1, files/in2 transfer_output_files = /tmp/out1 Arguments = in1 in2 /tmp/out1 Queue
