Personal tools
You are here: Home → User Support → User Documentation → Data transfer with GridFTP → 5 Data transfer with globus-url-copy
 

5 Data transfer with globus-url-copy

private

In this subsection we describe how you can use globus-url-copy to

  • copy data between your local workstation and the DEISA infrastructure
  • copy data from one DEISA platform to another DEISA platform

After that we give some concrete examples that show how to use the globus-url-copy command.

private

5.1 Copying data between a local workstation and the DEISA infrastructure

To transfer files from your local workstation to DEISA, you have have Globus installation available and to use one of the DEISA GridFTP door nodes listed in the Table 4.

On the GridFTP door node server

  • Globus toolkit has been installed,
  • connections to the DEISA network and thus to the GridFTP servers at every DEISA site
  • the machine can be accessed from the public internet,
  • the machine has mounted all the DEISA GPFS file systems.

Three GridFTP servers can be accessed currently from the public internet:

Use your DEISA_HOME or DEISA_DATA directories for uploading and downloading data, as these are transparently available at nearly all the DEISA sites. On most of the DEISA HPC system data can be managed using the Global Parallel File System GPFS. GridFTP can be used in case that data has to be moved into a non-GPFS storage system.

private

5.2 Copying data inside the DEISA infrastructure

This requires access to DEISA systems. The user can access these machines in two ways:

  • From a machine that has GSISSH-Term installed, as described in the Interactive access User Guide section 4.5 Using GSISSH-Term.
  • From a machine with a Globus installation that provides the gsissh client.

In the second case, after generating a proxy credential with grid-proxy-init, the user will connect to one of the "door nodes" by using the gsissh command:

gsissh -p 2222 a01.hlrb2.lrz-muenchen.de

After successful login to the DEISA machine with either GSISSH-Term or gsissh, the user is mapped to a DEISA user account, whose name depends on the user's Home Site.

For example, a user whose Home site is LRZ will be mapped to an account whose name starts with "lrz":

whoami
lrz015ab

In the next subsections we give some examples that show how to use globus-url-copy for different purposes.

private

5.3 Copying files and directories with globus-url-copy

We show how to copy files and entire directories.

Before using globus-url-copy you have to generate a proxy credential based on your credentials (that means your permanent public/private key pair) with grid-proxy-init. What you have to do with your credentials and the trusted CA certificates has also been described in DEISA Certificates FAQ.

grid-proxy-init
Your identity:
    /C=DE/O=GridGermany/OU=Leibniz-Rechenzentrum/OU=HLS/CN=Gabriel Mateescu
Enter GRID pass phrase for this identity:
Creating proxy .................................. Done
Your proxy is valid until: Fri Mar 10 05:09:41 2008  

 

Copy a file

Assume that the user has stored a large file "myfile" in the current working directory of his local workstation. He wants to use it as input file for a calculation on some DEISA production system. To upload it to DEISA using GridFTP, he has to use either CINECA's or LRZ's

globus-url-copy file://`pwd`/myfile gsiftp://gridftp.deisa.eu/<DEISA_HOME>/myfile 

For <DEISA_HOME> he has to enter the absolute path to his DEISA_HOME directory. DEISA_HOME and DEISA_DATA follow a specific pattern (please refer also to section 4.1 of the DEISA Primer):

DEISA_HOME=/deisa/SITE_NAME/home/DEISA_GROUP_NAME/DEISA_USER_NAME
DEISA_DATA=/deisa/SITE_NAME/data/DEISA_GROUP_NAME/DEISA_USER_NAME

For example, the values for the DEISA user lrz015ab with home site LRZ are:

id
uid=1054392(lrz015ab)
gid=1000001(lrz00001)
groups=1000001(lrz00001)
module load deisa
echo $DEISA_HOME /deisa/lrz/home/lrz00001/lrz015ab
echo $DEISA_DATA /deisa/lrz/data/lrz00001/lrz015a

Thus, the user can construct the values of DEISA_HOME and DEISA_DATA, once he knows the DEISA user ID and group. However, this requires the user to find out the DEISA user and group IDs and store these values on the client and reuse them each time she want to access DEISA_HOME and DEISA_DATA using the pattern shown above.

Copy a directory

We will copy the subdirectory "mydirectory" of the current directory to the user's remote DEISA_HOME directory:

globus-url-copy -cd -r file://`pwd`/mydirectory/ gsiftp://gridftp.deisa.eu/<DEISA_HOME>/mydirectory/

where the -cd option stands for "create directory" and its purpose is to create the directory "mydirectory" on the GridFTP server as a subdirectory of the remote DEISA_HOME directory. To include subdirectories is used recursive copy option -r.  Note that we terminate the URLs with a / to indicate that we refer to a directory. As 2811 is the GridFTP default port we can omit it here.

Document Actions