File storage options on RCAC systems include long-term storage (home directories, Fortress) and short-term storage (scratch directories, /tmp directory). Each option has different performance and intended uses, and some options vary from system to system as well. Daily snapshots of home directories are provided for a limited time for accidental deletion recovery. Scratch directories and temporary storage are not backed up and old files are regularly purged from scratch and /tmp directories. More details about each storage option appear below.
Home directories are provided for long-term file storage. Each user has one home directory. You should use your home directory for storing important program files, scripts, input data sets, critical results, and frequently used files. You should store infrequently used files on Fortress. Your home directory becomes your current working directory, by default, when you log in.
Daily snapshots of your home directory are provided for a limited period of time in the event of accidental deletion. For additional security, you should store another copy of your files on more permanent storage, such as the Fortress HPSS Archive.
Your home directory physically resides on a dedicated storage system only accessible for Brown. To find the path to your home directory, first log in then immediately enter the following:
$ pwd /home/myusername
Or from any subdirectory:
$ echo $HOME /home/myusername
Please note that your Brown home directory and its contents are exclusive to Brown cluster, including front-end hosts and compute nodes. This home directory is not available on other RCAC machines but Brown. There is no automatic copying or synchronization between home directories, but at your discretion you can manually copy all or parts of your main home to Brown using one of the suggested methods.
Your home directory has a quota limiting the total size of files you may store within. For more information, refer to the Storage Quotas / Limits Section.
Link to section 'Lost File Recovery' of 'Home Directory' Lost File Recovery
Nightly snapshots for 7 days, weekly snapshots for 4 weeks, and monthly snapshots for 3 months are kept. This means you will find snapshots from the last 7 nights, the last 4 Sundays, and the last 3 first of the months. Files are available going back between two and three months, depending on how long ago the last first of the month was. Snapshots beyond this are not kept. For additional security, you should store another copy of your files on more permanent storage, such as the Fortress HPSS Archive.
Link to section 'Performance' of 'Home Directory' Performance
Your home directory is medium-performance, non-purged space suitable for tasks like sharing data, editing files, developing and building software, and many other uses.
Your home directory is not designed or intended for use as high-performance working space for running data-intensive jobs with heavy I/O demands.
Link to section 'Long-Term Storage' of 'Long-Term Storage' Long-Term Storage
Long-term Storage or Permanent Storage is available to users on the High Performance Storage System (HPSS), an archival storage system, called Fortress. Program files, data files and any other files which are not used often, but which must be saved, can be put in permanent storage. Fortress currently has over 10PB of capacity.
For more information about Fortress, how it works, and user guides, and how to obtain an account:
Scratch directories are provided for short-term file storage only. The quota of your scratch directory is much greater than the quota of your home directory. You should use your scratch directory for storing temporary input files which your job reads or for writing temporary output files which you may examine after execution of your job. You should use your home directory and Fortress for longer-term storage or for holding critical results. The hsi and htar commands provide easy-to-use interfaces into the archive and can be used to copy files into the archive interactively or even automatically at the end of your regular job submission scripts.
Files in scratch directories are not recoverable. Files in scratch directories are not backed up. If you accidentally delete a file, a disk crashes, or old files are purged, they cannot be restored.
Files are purged from scratch directories not accessed or had content modified in 60 days. Owners of these files receive a notice one week before removal via email. Be sure to regularly check your Purdue email account or set up mail forwarding to an email account you do regularly check. For more information, please refer to our Scratch File Purging Policy.
All users may access scratch directories on Brown. To find the path to your scratch directory:
$ findscratch /scratch/brown/myusername
The value of variable $RCAC_SCRATCH is your scratch directory path. Use this variable in any scripts. Your actual scratch directory path may change without warning, but this variable will remain current.
$ echo $RCAC_SCRATCH /scratch/brown/myusername
Scratch directories are specific per cluster. I.e. only the /scratch/brown directory is available on Brown front-end and compute nodes. No other scratch directories are available on Brown.
Your scratch directory has a quota capping the total size and number of files you may store in it. For more information, refer to the section Storage Quotas / Limits .
Link to section 'Performance' of 'Scratch Space' Performance
Your scratch directory is located on a high-performance, large-capacity parallel filesystem engineered to provide work-area storage optimized for a wide variety of job types. It is designed to perform well with data-intensive computations, while scaling well to large numbers of simultaneous connections.
/tmp directories are provided for short-term file storage only. Each front-end and compute node has a /tmp directory. Your program may write temporary data to the
/tmp directory of the compute node on which it is running. That data is available for as long as your program is active. Once your program terminates, that temporary data is no longer available. When used properly,
/tmp may provide faster local storage to an active process than any other storage option. You should use your home directory and Fortress for longer-term storage or for holding critical results.
Backups are not performed for the
/tmp directory and removes files from
/tmp whenever space is low or whenever the system needs a reboot. In the event of a disk crash or file purge, files in
/tmp are not recoverable. You should copy any important files to more permanent storage.