TeraGrid.org Parent Organization Links

Purdue FPGA Resources

Purdue as part of NSF’s TeraGrid initiative, has undertaken providing limited FPGA resources to TeraGrid users generously donated by the NorthWest Indiana Computational Grid (NWICG). These resources consist of an SGI 450 (Brutus) with two RC100 FPGA blades, totaling 4 available FPGAs. Also available is a Sun Fire X2200 M2 (Portia) which serves both as a serialize, place & route (SPR) node for preparing FPGA code for use on Brutus and as an entry point for GSI-SSH and job submission to Brutus by TeraGrid users.

Programs using the FPGAs will generally consist of one IA64 binary compiled from C or partially written in C, which makes calls to the SGI RASC library to invoke FPGA algorithms as needed. These FPGA algorithms are the critical sections of the overall computation which the FPGAs are configured to run using an optimized architecture specific to each algorithm. These critical sections are written in either VHDL or Mitrion-C and then put through a "Serialize, Place, & Route" (SPR) process similar to a compile for ordinary code. These two pieces of code (the IA64 binary and the FPGA algorithm) will then be used together. First the FPGA algorithm is SPRed and then the output bitfile is registered for future use. Then the IA64 binary is submitted via Condor to run on Brutus. The FPGAs on Brutus will run the algorithm specified by the RASC calls in the IA64 binary. Some algorithms are already provided by systems administrators for anyone to use.

All users of the FPGA resource are expected to login to fpga.purdue.teragrid.org (Portia) via GSI-SSH using their grid credentials. From there, you will be able to submit FPGA-using IA64 jobs to Brutus via Condor and also build (SPR) and register code for new FPGA algorithms.

Building New FPGA Algorithms

To serialize, place & route new Mitrion-C code on Portia, you will first need to load the mitrion module. You must do this every time you log in: 

bash-3.00$ module load mitrion

Serialize, place, & route using Mitrion is most easily done using the “mcc” script. You may simply modify the following Makefile to suit your code:

Note that you will need both Mitrion-C code for your FPGA algorithm as well as C code that invokes your algorithm via the RASC library from SGI. Here is a brief example of both of these for the above Makefile example:

These example files and the above Makefile are also locally available on Portia under /opt/mitrion/examples/.

Compiling C Source Code that uses an FPGA Algorithm

To compile an IA64 binary on Portia which calls an FPGA algorithm (or multiple FPGA algorithms), you will need to either load the mitrion module as for SPR above or load both the xgcc-ia64 and rasc modules: 

bash-3.00$ module load mitrion
   (or)
bash-3.00$ module load xgcc-ia64
bash-3.00$ module load rasc

Then use a Makefile similar to the Makefile offered above for SPR, but simply remove the portions relating the the SPR process and mcc:

You will need to use a Makefile such as this which contains the relevant RASC library environment variables. An example of a RASC calls in a C application is here: This example and the above Makefile are also locally available on Portia under /opt/mitrion/examples/.

Registering and Managing FPGA Algorithms

If you are building a custom FPGA algorithm and application, you should now have a total of four necessary files:

  1. Your application (IA64 Linux Executable linked to RASC libraries)
  2. Your algorithm bitfile (.bin)
  3. Your algorithm bitstream config file (.cfg)
  4. Your algorithm service config file (.svc)
If you are only using an already registered system-wide FPGA algorithm, you need only have the first item above (the IA64 binary).

You may now register your new algorithm so that the FPGAs will be aware of it and RASC may call upon it. To do this, use the udevmgr utility: 

bash-3.00$ udevmgr -a -n my_algorithm_name -b my_algorithm.bin

The name you register your algorithm under ("my_algorithm_name" above) must match the name invoked in the application's RASC call. The same command will also update or replace the algorithm if necessary.

You may review all the algorithms you have registered or have been made available to all users by systems administrators by using udevmgr: 

bash-3.00$ udevmgr -l

Note that the latter section of algorithms listed here are not under your control, but are provided for convenience by systems administrators for any user to invoke if they wish. If you register an algorithm by the same name, your personally-registered algorithm will be invoked instead of the system-wide algorithm. If you have an algorithm you feel would be useful to other users, please contact systems administrators to have it registered as a system-wide algorithm.

You may also remove algorithms you have registered at any time: 

bash-3.00$ udevmgr -d -n my_algorithm_name

Submitting Jobs that use an FPGA

To actually submit a job to the FPGAs on Brutus, you will need to compose a simple Condor submission script, then submit this script using “condor_submit”. Here is an example Condor submission script for the above example code:

To submit this from Portia: 

bash-3.00$ condor_submit multi4_large.condor

To check on the status of jobs in the queue: 

bash-3.00$ condor_q

To remove a job from the queue: 

bash-3.00$ condor_rm my_job_number

To better evaluate why a job in the queue is not running: 

bash-3.00$ condor_q -better-analyze my_job_number