File Transfer

Gilbreth supports several methods for file transfer. Use the links below to learn more about these methods.

SCP

SCP (Secure CoPy) is a simple way of transferring files between two machines that use the SSH protocol. SCP is available as a protocol choice in some graphical file transfer programs and also as a command line program on most Linux, Unix, and Mac OS X systems. SCP can copy single files, but will also recursively copy directory contents if given a directory name.

After Aug 17, 2020, the community clusters will not support password-based authentication for login. Methods that can be used include two-factor authentication (Purdue Login) or SSH keys. If you do not have SSH keys installed, you would need to type your Purdue Login response into the SFTP's "Password" prompt.

Link to section 'Command-line usage:' of 'SCP' Command-line usage:

You can transfer files both to and from Gilbreth while initiating an SCP session on either some other computer or on Gilbreth (in other words, directionality of connection and directionality of data flow are independent from each other). The scp command appears somewhat similar to the familiar cp command, with an extra user@host:file syntax to denote files and directories on a remote host. Either Gilbreth or another computer can be a remote.

Example: Initiating SCP session on some other computer (i.e. you are on some other computer, connecting to Gilbreth):

      (transfer TO Gilbreth)
      (Individual files) 
$ scp  sourcefile  myusername@gilbreth.rcac.purdue.edu:somedir/destinationfile
$ scp  sourcefile  myusername@gilbreth.rcac.purdue.edu:somedir/
      (Recursive directory copy)
$ scp -pr sourcedirectory/  myusername@gilbreth.rcac.purdue.edu:somedir/

      (transfer FROM Gilbreth)
      (Individual files)
$ scp  myusername@gilbreth.rcac.purdue.edu:somedir/sourcefile  destinationfile
$ scp  myusername@gilbreth.rcac.purdue.edu:somedir/sourcefile  somedir/
      (Recursive directory copy)
$ scp -pr myusername@gilbreth.rcac.purdue.edu:sourcedirectory  somedir/

The -p flag is optional. When used, it will cause the transfer to preserve file attributes and permissions. The -r flag is required for recursive transfers of entire directories.

Example: Initiating SCP session on Gilbreth (i.e. you are on Gilbreth, connecting to some other computer):

      (transfer TO Gilbreth)
      (Individual files) 
$ scp  myusername@$another.computer.example.com:sourcefile  somedir/destinationfile
$ scp  myusername@$another.computer.example.com:sourcefile  somedir/
      (Recursive directory copy)
$ scp -pr myusername@$another.computer.example.com:sourcedirectory/  somedir/

      (transfer FROM Gilbreth)
      (Individual files)
$ scp  somedir/sourcefile  myusername@$another.computer.example.com:destinationfile
$ scp  somedir/sourcefile  myusername@$another.computer.example.com:somedir/
      (Recursive directory copy)
$ scp -pr sourcedirectory  myusername@$another.computer.example.com:somedir/

The -p flag is optional. When used, it will cause the transfer to preserve file attributes and permissions. The -r flag is required for recursive transfers of entire directories.

Link to section 'Software (SCP clients)' of 'SCP' Software (SCP clients)

Linux and other Unix-like systems:

The scp command-line program should already be installed.

Microsoft Windows:

MobaXterm
Free, full-featured, graphical Windows SSH, SCP, and SFTP client.
Command-line scp program can be installed as part of Windows Subsystem for Linux (WSL), or Git-Bash.

Mac OS X:

The scp command-line program should already be installed. You may start a local terminal window from "Applications->Utilities".
Cyberduck is a full-featured and free graphical SFTP and SCP client.

Globus

Globus, previously known as Globus Online, is a powerful and easy to use file transfer service for transferring files virtually anywhere. It works within RCAC's various research storage systems; it connects between RCAC and remote research sites running Globus; and it connects research systems to personal systems. You may use Globus to connect to your home, scratch, and Fortress storage directories. Since Globus is web-based, it works on any operating system that is connected to the internet. The Globus Personal client is available on Windows, Linux, and Mac OS X. It is primarily used as a graphical means of transfer but it can also be used over the command line.

Link to section 'Globus Web:' of 'Globus' Globus Web:

Navigate to http://transfer.rcac.purdue.edu
Click "Proceed" to log in with your Purdue Career Account.
On your first login it will ask to make a connection to a Globus account. Accept the conditions.
Now you are at the main screen. Click "File Transfer" which will bring you to a two-panel interface (if you only see one panel, you can use selector in the top-right corner to switch the view).
You will need to select one collection and file path on one side as the source, and the second collection on the other as the destination. This can be one of several Purdue endpoints, or another University, or even your personal computer (see Personal Client section below).

The RCAC collections are as follows. A search for "Purdue" will give you several suggested results you can choose from, or you can give a more specific search.

Home Directory storage: "Purdue Gilbreth Cluster - Home Directories", however, you can start typing "Purdue" and "Gilbreth" and it will suggest appropriate matches.
Gilbreth scratch storage: "Purdue Gilbreth Cluster - Scratch", however, you can start typing "Purdue" and "Gilbreth and it will suggest appropriate matches. From here you will need to navigate into the first letter of your username, and then into your username.

Research Data Depot: "Purdue Research Computing - Data Depot", a search for "Depot" should provide appropriate matches to choose from.
Fortress: "Purdue Fortress HPSS Archive", a search for "Fortress" should provide appropriate matches to choose from.

From here, select a file or folder in either side of the two-pane window, and then use the arrows in the top-middle of the interface to instruct Globus to move files from one side to the other. You can transfer files in either direction. You will receive an email once the transfer is completed.

Link to section 'Globus Personal Client setup:' of 'Globus' Globus Personal Client setup:

Globus Connect Personal is a small software tool you can install to make your own computer a Globus endpoint on its own. It is useful if you need to transfer files via Globus to and from your computer directly.

On the "Collections" page from earlier, click "Get Globus Connect Personal" or download a version for your operating system it from here: Globus Connect Personal
Name this particular personal system and follow the setup prompts to create your Globus Connect Personal endpoint.
Your personal system is now available as a collection within the Globus transfer interface.

Link to section 'Globus Command Line:' of 'Globus' Globus Command Line:

Globus supports command line interface, allowing advanced automation of your transfers.

To use the recommended standalone Globus CLI application (the globus command):

First time use: issue the globus login command and follow instructions for initial login.
Commands for interfacing with the CLI can be found via Using the Command Line Interface, as well as the Globus CLI Examples pages.

Link to section 'Sharing Data with Outside Collaborators' of 'Globus' Sharing Data with Outside Collaborators

Globus allows convenient sharing of data with outside collaborators. Data can be shared with collaborators' personal computers or directly with many other computing resources at other institutions. See the Globus documentation on how to share data:

https://docs.globus.org/how-to/share-files/

For links to more information, please see Globus Support page and RCAC Globus presentation.

Windows Network Drive / SMB

SMB (Server Message Block), also known as CIFS, is an easy to use file transfer protocol that is useful for transferring files between RCAC systems and a desktop or laptop. You may use SMB to connect to your home, scratch, and Fortress storage directories. The SMB protocol is available on Windows, Linux, and Mac OS X. It is primarily used as a graphical means of transfer but it can also be used over the command line.

Note: to access Gilbreth through SMB file sharing, you must be on a Purdue campus network or connected through VPN.

Link to section 'Windows:' of 'Windows Network Drive / SMB' Windows:

Windows 7: Click Windows menu > Computer, then click Map Network Drive in the top bar
Windows 8 & 10: Tap the Windows key, type computer, select This PC, click Computer > Map Network Drive in the top bar
In the folder location enter the following information and click Finish:
- To access your Gilbreth home directory, enter \\home.gilbreth.rcac.purdue.edu\gilbreth-home.
- To access your scratch space on Gilbreth, enter \\scratch.gilbreth.rcac.purdue.edu\gilbreth-scratch. Once mapped, you will be able to navigate to your scratch directory.
- Note: Use your career account login name and password when prompted. (You will not need to add ",push" nor use your Purdue Duo client.)
Your home or scratch directory should now be mounted as a drive in the Computer window.

Link to section 'Mac OS X:' of 'Windows Network Drive / SMB' Mac OS X:

In the Finder, click Go > Connect to Server
In the Server Address enter the following information and click Connect:
- To access your Gilbreth home directory, enter smb://home.gilbreth.rcac.purdue.edu/gilbreth-home.
- To access your scratch space on Gilbreth, enter smb://scratch.gilbreth.rcac.purdue.edu/gilbreth-scratch. Once mapped, you will be able to navigate to your scratch directory.
- Note: Use your career account login name and password when prompted. (You will not need to add ",push" nor use your Purdue Duo client.)
Note: Use your career account login name and password when prompted. (You will not need to add ",push" nor use your Purdue Duo client.)

Link to section 'Linux:' of 'Windows Network Drive / SMB' Linux:

There are several graphical methods to connect in Linux depending on your desktop environment. Once you find out how to connect to a network server on your desktop environment, choose the Samba/SMB protocol and adapt the information from the Mac OS X section to connect.
If you would like access via samba on the command line you may install smbclient which will give you FTP-like access and can be used as shown below. For all the possible ways to connect look at the Mac OS X instructions.
```
smbclient //home.gilbreth.rcac.purdue.edu/gilbreth-home -U myusername
smbclient //scratch.gilbreth.rcac.purdue.edu/gilbreth-scratch -U myusername
```
Note: Use your career account login name and password when prompted. (You will not need to add ",push" nor use your Purdue Duo client.)

FTP / SFTP

FTP is not supported on any research systems because it does not allow for secure transmission of data. Use SFTP instead, as described below.

SFTP (Secure File Transfer Protocol) is a reliable way of transferring files between two machines. SFTP is available as a protocol choice in some graphical file transfer programs and also as a command-line program on most Linux, Unix, and Mac OS X systems. SFTP has more features than SCP and allows for other operations on remote files, remote directory listing, and resuming interrupted transfers. Command-line SFTP cannot recursively copy directory contents; to do so, try using SCP or graphical SFTP client.

Link to section 'Command-line usage' of 'FTP / SFTP' Command-line usage

You can transfer files both to and from Gilbreth while initiating an SFTP session on either some other computer or on Gilbreth (in other words, directionality of connection and directionality of data flow are independent from each other). Once the connection is established, you use put or get subcommands between "local" and "remote" computers. Either Gilbreth or another computer can be a remote.

Example: Initiating SFTP session on some other computer (i.e. you are on another computer, connecting to Gilbreth):

$ sftp myusername@gilbreth.rcac.purdue.edu

      (transfer TO Gilbreth)
sftp> put sourcefile somedir/destinationfile
sftp> put -P sourcefile somedir/

      (transfer FROM Gilbreth)
sftp> get sourcefile somedir/destinationfile
sftp> get -P sourcefile somedir/

sftp> exit

The -P flag is optional. When used, it will cause the transfer to preserve file attributes and permissions.

Example: Initiating SFTP session on Gilbreth (i.e. you are on Gilbreth, connecting to some other computer):

$ sftp myusername@$another.computer.example.com

      (transfer TO Gilbreth)
sftp> get sourcefile somedir/destinationfile
sftp> get -P sourcefile somedir/

      (transfer FROM Gilbreth)
sftp> put sourcefile somedir/destinationfile
sftp> put -P sourcefile somedir/

sftp> exit

The -P flag is optional. When used, it will cause the transfer to preserve file attributes and permissions.

Link to section 'Software (SFTP clients)' of 'FTP / SFTP' Software (SFTP clients)

Linux and other Unix-like systems:

The sftp command-line program should already be installed.

Microsoft Windows:

MobaXterm
Free, full-featured, graphical Windows SSH, SCP, and SFTP client.
Command-line sftp program can be installed as part of Windows Subsystem for Linux (WSL), or Git-Bash.

Mac OS X:

The sftp command-line program should already be installed. You may start a local terminal window from "Applications->Utilities".
Cyberduck is a full-featured and free graphical SFTP and SCP client.

Copying files from Purdue IT research computing home directory to Gilbreth

The Gilbreth home directory and its contents are specific to the Gilbreth cluster, and are not available on other RCAC machines. For people having access to other Community Clusters and Gilbreth, there is no automatic copying or synchronization between main and Gilbreth home directories. At your discretion, you can manually copy all or parts of your main research computing home to Gilbreth using one of the methods described below.

Please note that copying may fail if the size of your research computing home directory is larger than the Gilbreth one's quota. Please check usage and limits before proceeding!

Link to section 'Complete copy' of 'Copying files from Purdue IT research computing home directory to Gilbreth' Complete copy

For your convenience, a custom tool copy-rcac-home is provided to simplify at-will duplication of your main research computing home directory into Gilbreth. The tool performs a complete 1-to-1 copy using rsync -auH (with exception of a narrow subset of system-specific service files).

To use the tool, simply type copy-rcac-home in a terminal window on a Gilbreth front-end or compute node:

$ copy-rcac-home

   This script will copy entire contents of your main RCAC
   home directory into your Gilbreth cluster's $HOME.

   Note: copying may fail if the size of your RCAC home directory
   is larger than your quota on the Gilbreth one (25GB).
   BEFORE PROCEEDING, please run 'myquota' command on another
   cluster to see your usage there and judge whether it would fit!

Would you like to proceed? [Y/n]:

At this stage answering yes will proceed with copying, or you can respond with a no (or Ctrl-C) to cancel. See copy-rcac-home --help for more details on the tool.

Link to section 'Partial copy' of 'Copying files from Purdue IT research computing home directory to Gilbreth' Partial copy

Desired parts (or whole) of your research computing home directories can be copied to Gilbreth via any of the home directories' supported transfer methods, such as SCP, SFTP, rsync, or Globus.

Example: recursive copying of a subdirectory from RCAC home directory into Gilbreth home using scp.

   (if you are on Gilbreth, use other cluster name for the remote part)
$ scp -pr myothercluster.rcac.purdue.edu:somedirectory/  ~/

   (if you are on another cluster, use Gilbreth for the remote part)
$ scp -pr somedirectory/ myusername@gilbreth.rcac.purdue.edu:~/

Example: copying using Globus.

Search collections for "Purdue Research Computing - Home Directories" and "Purdue Gilbreth Cluster - Home" endpoints, respectively, then transfer desired files and/or directories as usual.

Migrating Your Current Purdue IT Research Computing Home Directory to the New Gilbreth Home Directory

In an upcoming maintenance, the Gilbreth home directory and its contents will become specific to the Gilbreth and will no longer be available on other RCAC machines. New Gilbreth home directories will be given to all Gilbreth users, and these home directories will be empty. The new home directories on Gilbreth are already available and are located at /home-new/$USER. There will be no automatic copying or synchronization between your current Gilbreth home (also referred to as your main RCAC home directory) and your new Gilbreth home directories. At your discretion, you can manually copy all or parts of your current Gilbreth home directory to your new Gilbreth home directory using one of the methods described below.

Please note that copying may fail if the size of your main research computing home directory is larger than the new Gilbreth one's quota of 25 GB. Please check usage and limits before proceeding!

Link to section 'Complete copy' of 'Migrating Your Current Purdue IT Research Computing Home Directory to the New Gilbreth Home Directory' Complete copy

To use the tool, simply type copy-rcac-home in a terminal window on a Gilbreth front-end or compute node:

$ copy-rcac-home

   This script will copy entire contents of your main RCAC
   home directory into your Gilbreth cluster's $HOME.

   Note: copying may fail if the size of your RCAC home directory
   is larger than your quota on the Gilbreth one (25GB).
   BEFORE PROCEEDING, please run 'myquota' command on another
   cluster to see your usage there and judge whether it would fit!

Would you like to proceed? [Y/n]:

At this stage answering yes will proceed with copying, or you can respond with a no (or Ctrl-C) to cancel. See copy-rcac-home --help for more details on the tool.

Link to section 'Partial copy' of 'Migrating Your Current Purdue IT Research Computing Home Directory to the New Gilbreth Home Directory' Partial copy

Desired parts (or whole) of your research computing home directories can be copied to Gilbreth via any of the home directories' supported transfer methods, such as SCP, SFTP, rsync, or Globus.

Example: recursive copying of a subdirectory from your current home directory on Gilbreth into the new Gilbreth home using scp and cp.

   (if you are on Gilbreth)
$ cp -pr somedirectory/  /home-new/$USER/

   (if you are on another cluster)
$ scp -pr somedirectory/ $USER@gilbreth.rcac.purdue.edu:/home-new/$USER

Example: copying using Globus.
Search collections for "Purdue Research Computing - Home Directories" and "Purdue Gilbreth Cluster - Home Directories" endpoints, respectively, then transfer desired files and/or directories as usual. For hidden files such as a .bashrc file, you will need to make sure to toggle the "Show Hidden Items" button shown below.