Data
Frequently asked questions about data and data management.
What is the best way to access my data?
HSI and HTAR: HSI provides a FTP-style interface taking advantage of the power of HPSS without requiring any special user knowledge. HTAR is a utility to aggregate a set of files into a single tar archive directly into Fortress, without requiring space to first create an archive.
Can I set up a shared space for my research group to share data?
Research groups are assigned a group data storage space within Fortress with each Data Depot group space. Faculty should request a Data Depot trial to create a shared Fortress space for their research group.
RCAC resources are not intended to store data protected by Federal privacy and security laws (e.g., HIPAA, ITAR, classified, etc.). It is the responsibility of the faculty partner to ensure that no protected data is stored on the systems.
Please keep in mind that such spaces are, by design, accessible by others and should not be used to store private information such as grades, login credentials, or personal data. Contact us to create a group space for your group.
How can I verify the contents of my archives while using HTAR?
You can ask HTAR to verify the contents of an archive during/after creation using the '-Hverify' switch. Please see the Fortress User Guide for details.
HSI/HTAR: put: Error -5 on transfer
First, check your firewall settings, and ensure that there are no firewall rules interfering with connecting to Fortress. For firewall configuration, please see "Do I need to do anything to my firewall to access Fortress?" If firewalls are not responsible:
Open the file named /etc/hosts on your workstation, especially if you run a Debian or Ubuntu Linux distribution. Look for a line like:
127.0.1.1 hostname.dept.purdue.edu hostname
Replace the IP address 127.0.1.1 with the real IP address for your system. If you don't know your IP address, you can find it with the command:
host `hostname --fqdn`
Can I share data with outside collaborators?
Yes! Globus allows convenient sharing of data with outside collaborators. Data can be shared with collaborators' personal computers or directly with many other computing resources at other institutions. See the Globus documentation on how to share data:
HSI/HTAR: Unable to authenticate user with remote gateway (error 2 or 9)
There could be a variety of such errors, with wordings along the lines of
Could not initialize keytab on remote server.
result = -2, errno = 2rver connection
*** hpssex_OpenConnection: Unable to authenticate user with remote gateway at 128.211.138.40.1217result = -2, errno = 9
Unable to setup communication to HPSS...
ERROR (main) unable to open remote gateway server connection
HTAR: HTAR FAILED
and
*** hpssex_OpenConnection: Unable to authenticate user with remote gateway at 128.211.138.40.1217result = -11000, errno = 9
Unable to setup communication to HPSS...
*** HSI: error opening logging
Error - authentication/initialization failed
The root cause for these errors is an expired or non-existent keytab file (a special authentication token stored in your home directory). These keytabs are valid for 90 days and on most RCAC resources they are usually automatically checked and regenerated when you execute hsi or htar commands. However, if the keytab is invalid, or fails to generate, Fortress may be unable to authenticate you and you would see the above errors. This is especially common on those RCAC clusters that have their own dedicated home directories (such as Bell), or on standalone installations (such as if you downloaded and installed HSI and HTAR on your non-RCAC computer).
This is a temporary problem and a permanent system-wide solution is being developed. In the interim, the recommended workaround is to generate a new valid keytab file in your main research computing home directory, and then copy it to your home directory on Fortress. The fortresskey command is used to generate the keytab and can be executed on another cluster or a dedicated data management host data.rcac.purdue.edu:
$ ssh myusername@data.rcac.purdue.edu fortresskey
$ scp -pr myusername@data.rcac.purdue.edu:~/.private $HOME
With a valid keytab in place, you should then be able to use hsi and htar commands to access Fortress from Fortress. Note that only one keytab can be valid at any given time (i.e. if you regenerated it, you may have to copy the new keytab to all systems that you intend to use hsi or htar from if they do not share the main research computing home directory).