Data Management Tutorial Grid Induction NA3 Training Team, Israel

So far we've used files located on the UI to compute our task.
These were transferred to the WN via the Input / Output Sandbox jdl attribute.
This is not suitable for data files larger than 10Mb.

After this tutorial you will be able to use files stored on the Grid for your computational task and store files created by your job on Grid SE.

In this tutorial we will run a complete example that

• Prepares input files on an SE for running jobs.
• Create a job which downloads these input files and uses them.
• Take the job output and upload it to the SE
• Download the job output to the UI

Finding Storage Elements

Run the following command to list the accessible SE:

lcg-infosites can query other things like CEs, CEs close to SEs etc.
For more info type lcg-infosites -help

Creating a directory

Before we upload the files, each of you will create your own directory on the file catalog.
The file catalog convention is that all logical file names start with /grid/MyVO/ as the base directory. However, some VOs require further hierarchy.
Gilda VO will be under /grid/gilda/tutorials as base name.
Put your user name as your base home directory.
The following command is used to create a directory on the file catalog:

(e.g. lfc-mkdir -p /grid/see/assaf for see VO or lfc-mkdir -p /grid/gilda/tutorials/assaf for gilda VO)
Be'er sheva users: Please see Be'er Sheva-specific base directory

The flags are similar to regular mkdir, so -p means create the parents if they do not exist.

Uploading files to the directory

The output of the command is the guid of the uploaded file:
guid:399099e7-6333-45e9-8029-67aa6097d16a

Please download the following files to your account:

• ariadne executable
• proteins1.fasta
• proteins2.fasta
• blosum62.txt

We will use the lcg-cr command to place the files needed on the grid before running.
You can use the shell script below, while replacing the SE with a chosen one (you can use more than one)

Browsing your directory

You can always get the guid from lfn or surl or vice-versa.
For example the lcg-lg (list GUID) command returns the guid associated with a specified lfn or surl.
Use the lfn you've given to the file you uploaded to the SE to retrieve its guid.

% lcg-lg --vo MyVO lfn:/grid/MyVO/MyUserName/MyFileName

(e.g. for see VO, lcg-lg --vo see lfn:/grid/see/assaf/assaf.txt).

The output of the command is:

guid:399099e7-6333-45e9-8029-67aa6097d16a

Tip: If you don't want to type the entire path of the lfn every time, you can set the following environment variable to your base directory:

Preparing the jdl

Type="Job";
JobType="Normal";
VirtualOrganisation="MyVO";
Executable="/bin/sh";
Arguments="alignment.sh alignment.exe proteins1.fasta proteins2.fasta blosum62.txt my_alignment";
StdOutput="alignment.out";
StdError="alignment.err";
InputSandbox={"alignment.sh"};
OutputSandbox={"alignment.err", "alignment.out"};
RetryCount=3;
ShallowRetryCount=7;
DataRequirements = {
        [
        DataCatalogType = "DLI";
        InputData = {"lfn:/grid/MyVO/MyUserName/proteins1.fasta",
                "lfn:/grid/MyVO/MyUserName/proteins2.fasta",
                "lfn:/grid/MyVO/MyUserName/alignment.exe",
                "lfn:/grid/MyVO/MyUserName/blosum62.txt"
                };
        ]
};
DataAccessProtocol = {"rfio","gsiftp","gsidcap","https"};
OutputSE="<SE_name>";

As you can see, it's a normal job, running the shell script alignment.sh with the relevant attributes. The shell script is described below.

• The output file is also uploaded to the SE, so the OutputSsandbox is returning a success/failure information.
• The new attributes are a helper for the Resource Broker to select the CE close to your selected SE.
  DataRequirements specifies the input files in InputData. SEE supports only the DataCatalogType DLI currently, so use it as written.
• DataAccessProtocol is a mandatory attribute if DataRequirements are specified. You can add many types, but the above will suffice.
• Last, OutputSE specifies the location of the SE selected. The RB will choose the closest CE according to this attribute.
  For some reason, if the input files and the output SE differ, the RB has problem finding a match, so choose the same for each.

1. Downloads the required files from the Grid (by using the lfn) to the WN (where the computation itself takes place).
2. Runs the alignment.exe using the protein sequences in the files proteins1.fasta and proteins2.fasta and using blosum62.txt.
3. When the program terminates, the output file is uploaded to the SE and registered in the file catalog (using the lcg-cr command).

Pay attention that the environment variables LCG_GFAL_INFOSYS and LFC_HOST are VO dependant.
Use these commands to query them:

%lcg-infosites --vo MyVO tag %lcg-infosites --vo MyVO lfc

The first value in the % lcg-infosites --vo MyVO tag command is the requested value.
You can verify that they match the already set environment variables:

%echo $LCG_GFAL_INFOSYS %echo $LFC_HOST

The values in the script below take the values on the CE
You can specify directly the values as follows:
For gilda VO, these are
export LCG_GFAL_INFOSYS="glite-rb.ct.infn.it:2170";
export LFC_HOST="lfc-gilda.ct.infn.it";
For see VO, these are
export LCG_GFAL_INFOSYS="bdii.isabella.grnet.gr:2170";
export LFC_HOST="lfc.isabella.grnet.gr";

Running the jdl

Do you remember how to clear your files and directory you've created?
For files use:

For the empty directory

Congratulations. This is essentially what is needed to know for using the grid.

If you are feeling in the mood, let's learn about Advanced topics