Personal tools
You are here: Home User Support User Documentation DEISA Batch Systems 2 Load Leveler

2 Load Leveler

This Section describes common LoadLeveler batch commands, LoadLeveler directives and, finally, an example MPI batch script.

2.1 LoadLeveler Commands

llsubmit <job_script>

Submit a job script, called ‘job_script’, for execution

llq

Check the status of your job(s).

 llcancel <job_id>

Cancel a job.

llstatus

Returns status of machine.

2.2 Remote job management

If your Home Site is either CINECA, IDRIS, FZJ or RZG and your Execution Platforms include CINECA CNE-SP6, IDRIS Vargas and/or RZG VIP, then you can manage your remote jobs directly from your Home Site using the –X flag. For instance, if you wish to submit your job to a remote site, use

llsubmit –X <site>

where <site> is one of the following: deisa_cne, deisa_idr, deisa_fzj or deisa_rzg. 
The –X flag can be used with all the LoadLeveler commands listed above, i.e. if you wish to check the status of your remote job, use

llq –X <site>

etc.

This section lists some typical and important LoadLeveler directives accepted or unadvised. A general overview of LoadLeveler is provided by the corresponding IBM documentation.

2.4.1 LoadLeveler General Directives

#@ wall_clock_limit = <HH:MM:SS> [,HH:MM:SS]

The first time sets the hard limit of the execution time in hours, minutes and seconds. The second optional time sets a soft limit whereby the process receives a warning at this time.

#@ requirements = (Feature == “DEISA”)

This directive is required for all DEISA jobs, and must be employed.

#@ notify_user = <your-personal-email>

Emails the user once the job is complete. Note: Use your private email and not an  account local to the execution platform.

#@ notification = error

Notifies the job only when the job fails.

#@ shell = /bin/bash

Specifies which shell to employ.

#@ queue

Must appear at the end of the list of directives.

Please be aware that using some other keywords can on the contrary produce unexpected behavior. Therefore, the following keywords must be avoided in DEISA jobs:

#@ requirements = (X)  with a value for X different from Feature == "DEISA"

#@ class

2.4.2 LoadLeveler “POWER” Directives

#@ total_tasks = <number of cores>

Sets number of cores for an MPI job.

#@ resources = ConsumableCpus(N)

For multithread parallel jobs.

#@ job_type = parallel

Otherwise, could be set to serial if needed.

#@ data_limit = MEMORY

where MEMORY is the amount of data memory required. This directive is optional: the default value is set to the recommended value.

#@ stack_limit = MEMORY

same as data_limit above, but for stack memory. This directive is optional: the default value is set to the recommended value.

Please be aware that using some other keywords can on the contrary produce unexpected behavior. Therefore, the following keywords must be avoided in DEISA jobs:

#@ blockingwith a value different from unlimited

#@ max_processors

#@ min_processors

#@ network.X=Y

#@ node

#@ node_usage

#@ task_geometry

#@ tasks_per_node

Example of a POWER MPI script for DEISA users

#@ job_name = MyJob
#@ output = MyModel/output.$(jobid).log_ll
#@ error = MyModel/output.$(jobid).log_ll
#@ notify_user = <replace with your email>
#@ notification = error
#@ shell = /bin/bash
#@ requirements = (Feature == “DEISA”)
#@ job_type = parallel
#@ total_tasks = 128
#@ wall_clock_limit = 06:00:00,05:50:00
#@ data_limit = 512mb
#@ stack_limit = 400mb
#@ queue

module load deisa cpmd
cd $DEISA_SCRATCH
cp $DEISA_DATA/MyModel/CPMD/* .
$CPMD input.in > cpmd.out
cp * $DEISA_DATA/MyModel/CPMD

Note that, in this example, both the standard output and standard error are sent into the same file MyModel/output.$(jobid).log_ll which will reside in the $DEISA_HOME directory.

2.4.3 LoadLeveler “BlueGene” Directives

#@ job_type = BLUEGENE

is required to setup a step running on the BG/P.

#@ bg_size = xxxxx

is required to design the size of a BG job

#@ bg_connection = {MESH|TORUS|PREFER_TORUS}

Multisteps jobs are accepted (with steps running on the Front-end Node for pre- and post-processing purposes).

You must also be aware that using some other keywords can on the contrary produce unexpected behavior. Therefore, the following keywords must be avoided in DEISA jobs:

#@ bg_shape

#@ bg_partition

 

Example of BlueGene  MPI script for DEISA users

#!/bin/bash 
# @ job_name = myjob
# @ error = $(job_name).$(jobid).out
# @ output = $(job_name).$(jobid).out
# @ wall_clock_limit = 00:30:00
# @ notify_user = <replace with your email>
# @ notification = error
# @ job_type = bluegene
# @ bg_size = 128
# @ queue
#
# Run the program in Virtual Node Mode on the BlueGene/P:
#
# Executable statements follow

Executable statements for BlueGene DEISA users

module load deisa
cp my_code $DEISA_SCRATCH
# Warning: if you need to transfer important volumes
# of data, please use a multi-step job

cp input.data $DEISA_SCRATCH
cd $DEISA_SCRATCH

mpirun -mode VN -np 256 -mapfile TXYZ -exe ./my_code

# $LOADL_STEP_INITDIR is the submission directory
cp output.data $LOADL_STEP_INITDIR

Please be aware that $DEISA_DATA and $DEISA_HOME are not usable under BlueGene job step type, and using some can produce unexpected behavior.

Document Actions