5 Data transfer with globus-url-copy
To transfer files from your local workstation to DEISA, you have have Globus installation available and to use one of the DEISA GridFTP door nodes listed in the Table 4.
On the GridFTP door node server
- Globus toolkit has been installed,
- connections to the DEISA network and thus to the GridFTP servers at every DEISA site
- the machine can be accessed from the public internet,
- the machine has mounted all the DEISA GPFS file systems.
Three GridFTP servers can be accessed currently from the public internet:
Use your DEISA_HOME or DEISA_DATA directories for uploading and downloading data, as these are transparently available at nearly all the DEISA sites. On most of the DEISA HPC system data can be managed using the Global Parallel File System GPFS. GridFTP can be used in case that data has to be moved into a non-GPFS storage system.
This requires access to DEISA systems. The user can access these machines in two ways:
- From a machine that has GSISSH-Term installed, as described in the Interactive access User Guide section 4.5 Using GSISSH-Term.
- From a machine with a Globus installation that provides the gsissh client.
In the second case, after generating a proxy credential with grid-proxy-init, the user will connect to one of the "door nodes" by using the gsissh command:
gsissh -p 2222 a01.hlrb2.lrz-muenchen.de
After successful login to the DEISA machine with either GSISSH-Term or gsissh, the user is mapped to a DEISA user account, whose name depends on the user's Home Site.
For example, a user whose Home site is LRZ will be mapped to an account whose name starts with "lrz":
In the next subsections we give some examples that show how to use globus-url-copy for different purposes.
We show how to copy files and entire directories.
Before using globus-url-copy you have to generate a proxy credential based on your credentials (that means your permanent public/private key pair) with grid-proxy-init. What you have to do with your credentials and the trusted CA certificates has also been described in DEISA Certificates FAQ.
Enter GRID pass phrase for this identity:
Creating proxy .................................. Done
Your proxy is valid until: Fri Mar 10 05:09:41 2008
Copy a file
Assume that the user has stored a large file "myfile" in the current working directory of his local workstation. He wants to use it as input file for a calculation on some DEISA production system. To upload it to DEISA using GridFTP, he has to use either CINECA's or LRZ's
globus-url-copy file://`pwd`/myfile gsiftp://gridftp.deisa.eu/<DEISA_HOME>/myfile
For <DEISA_HOME> he has to enter the absolute path to his DEISA_HOME directory. DEISA_HOME and DEISA_DATA follow a specific pattern (please refer also to section 4.1 of the DEISA Primer):
For example, the values for the DEISA user lrz015ab with home site LRZ are:
id uid=1054392(lrz015ab) gid=1000001(lrz00001) groups=1000001(lrz00001)
module load deisa
echo $DEISA_HOME /deisa/lrz/home/lrz00001/lrz015ab
echo $DEISA_DATA /deisa/lrz/data/lrz00001/lrz015a
Thus, the user can construct the values of DEISA_HOME and DEISA_DATA, once he knows the DEISA user ID and group. However, this requires the user to find out the DEISA user and group IDs and store these values on the client and reuse them each time she want to access DEISA_HOME and DEISA_DATA using the pattern shown above.
Copy a directory
We will copy the subdirectory "mydirectory" of the current directory to the user's remote DEISA_HOME directory:
globus-url-copy -cd -r file://`pwd`/mydirectory/ gsiftp://gridftp.deisa.eu/<DEISA_HOME>/mydirectory/
where the -cd option stands for "create directory" and its purpose is to create the directory "mydirectory" on the GridFTP server as a subdirectory of the remote DEISA_HOME directory. To include subdirectories is used recursive copy option -r. Note that we terminate the URLs with a / to indicate that we refer to a directory. As 2811 is the GridFTP default port we can omit it here.