Running Jobs – High Performance Computing

1. Simple CPU job
2. GPU job
3. MPI job
4. Array job
5. Interactive job
6. Graphical Application job
7. Python job
8. MATLAB job
9. COMSOL job
10. Slurm commands

IDUN uses the Slurm Workload Manager to manage the provided resources and to schedule jobs on these resources.

If you need more, start your job with 7 days and send a request to help desk, with the job id, about more hours for this specific job.

We have several videos on our IDUN HPC Youtube channel with some practical examples.

IDUN has 3 partitions (queues):

CPUQ - for jobs that needs only CPU.
GPUQ - for jobs that will use GPUs.
short - for test and development with time limit 20 minutes.

1. Simple CPU job

CPUQ is the default partition for pure CPU jobs. If you need GPU's, please use the GPUQ partition.

#!/bin/sh
#SBATCH --partition=CPUQ
#SBATCH --account=<account>
#SBATCH --time=0-00:15:00     # 0 days and 15 minutes limit
#SBATCH --nodes=1             # 1 compute nodes
#SBATCH --cpus-per-task=2     # 2 CPU cores
#SBATCH --mem=5G              # 5 gigabytes memory
#SBATCH --output=hello.txt    # Log file
echo "Hello IDUN"

Note, that you need to replace <account> with your allocation account, respectively.

Save the script, e.g. as job.slurm, and submit the job using command sbatch:

$ sbatch job.slurm

2. GPU job

To get access to gpu resources, you need to add a request for GPU resources. You may choose between:

p100 - NVIDIA GPU with 16GB
v100 - NVIDIA GPU with 16GB or 32GB
a100 - NVIDIA GPU with 40GB or 80GB. Also available PCIE or NVLink cards (sxm4).
h100 - NVIDIA GPU with 80GB

NVIDIA specifications:

	p100	v100	a100	h100
Single-Precision FP32 TFLOPS	9.3	14	19.5	67
Double-Precision FP64 TFLOPS	4.7	7	9.7	34

Our benchmark results:

	p100	v100	a100	h100
Tensorflow ResNet-50	239 images/sec	379 images/sec	891 images/sec
HPCG CUDA	95 GFLOPS	140 GFLOPS	241 GFLOPS	534 GFLOPS
Hashcat MD5	27129 MH/s	50663 MH/s	60503 MH/s	120.2 GH/s
Blender 4.0.1 Classroom	00:58.27 sec	00:30.41 sec	00:20.28 sec	00:12.99 sec

GPUs are available in GPUQ partition. Use this line in the job script:

#SBATCH --partition=GPUQ

IDUN uses these GPU names:

p100
v100
a100

IDUN uses these constraint names (features):

p100
v100
a100
gpu16g
gpu32g
gpu40g
gpu80g
sxm4

Examples

If you need one gpu:

#SBATCH --gres=gpu:1

if you need two p100 gpus:

#SBATCH --gres=gpu:p100:2

If you are dependent on specific type of GPU (p100,v100,a100), for example 2 Volta GPUs and only with 32GB memory:

#SBATCH --gres=gpu:v100:2
#SBATCH --constraint=gpu32g

It is also possible combine constraints with logical AND "&", logical OR "|":

Three GPUs but only v100 or a100:

#SBATCH --gres=gpu:3
#SBATCH --constraint="v100|a100"

Four GPUs but only a100 with sxm4

#SBATCH --gres=gpu:4
#SBATCH --constraint="a100&sxm4"

Five a100 GPUs but only with 40 or 80 GB memory

#SBATCH --gres=gpu:a100:5
#SBATCH --constraint="gpu40g|gpu80g"

Example:

#!/bin/sh
#SBATCH --partition=GPUQ
#SBATCH --account=<account>
#SBATCH --time=00:30:00
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=2
#SBATCH --gres=gpu:2  
#SBATCH --job-name="LBM_CUDA"
#SBATCH --output=lbm_cuda.out
module purge
module load fosscuda/2018b
mpirun hostname
srun ./my cudacode

3. MPI job

job.slurm (example with compiled c-code or fortran code).

#!/bin/sh
#SBATCH --partition=CPUQ
#SBATCH --account=<account>
#SBATCH --time=00:15:00
#SBATCH --nodes=2              # 2 compute nodes
#SBATCH --ntasks-per-node=1    # 1 mpi process each node
#SBATCH --mem=12000            # 12GB - in megabytes
#SBATCH --job-name="hello_test"
#SBATCH --output=test-srun.out
#SBATCH --mail-user=<email>
#SBATCH --mail-type=ALL

WORKDIR=${SLURM_SUBMIT_DIR}
cd ${WORKDIR}
echo "we are running from this directory: $SLURM_SUBMIT_DIR"
echo " the name of the job is: $SLURM_JOB_NAME"
echo "Th job ID is $SLURM_JOB_ID"
echo "The job was run on these nodes: $SLURM_JOB_NODELIST"
echo "Number of nodes: $SLURM_JOB_NUM_NODES"
echo "We are using $SLURM_CPUS_ON_NODE cores"
echo "We are using $SLURM_CPUS_ON_NODE cores per node"
echo "Total of $SLURM_NTASKS cores"

module purge
module load intel/2020b
module list
srun myprogram

We enabled switch topology on IDUN. You can list topology with this command:

$ scontrol show topo

In case you need to start job only on the nodes connected to one switch add this line to the job script:

#SBATCH --switches=1

Run several tasks and several nodes

If you your job is MPI and you want to run on several tasks and nodes, apply corresponding changes.

If you need to run your job on two nodes and ten tasks (cores) on each node:

#SBATCH --nodes=2
#SBATCH --ntasks-per-node=10

Use same module in the script as for the compilation of the code.

4. Array jobs:

To run many sub jobs from same job, use --array as

#SBATCH --array=1-10

Starts 10 sub jobs and you get a array ID from each sub job with environment variable $SLURM_ARRAY_TASK_ID

Example:

#!/bin/sh
#SBATCH --partition=CPUQ
#SBATCH --account=<account>
#SBATCH --time=00:15:00
#SBATCH --nodes=1
#SBATCH -c 28
#SBATCH --mem=12000
#SBATCH --array=1-10
#SBATCH --job-name="hello_test"
#SBATCH --output=test-srun.out
#SBATCH --mail-user=<email>
#SBATCH --mail-type=ALL
 
WORKDIR=${SLURM_SUBMIT_DIR}
cd ${WORKDIR}

module purge
module load intel/2020b
module list
./myprogram $SLURM_ARRAY_TASK_ID

This example will start 10 sub jobs and the running program will get an input argument with individual Task ID, which is from 1 to 10.

Other array jobs settings:

Specification   Resulting SLURM_ARRAY_TASK_IDs
1,4,42          # 1, 4, 42
1-5             # 1, 2, 3, 4, 5
0-10:2          # 0, 2, 4, 6, 8, 10 (step 2)
32,56,100-200   # 32, 56, 100, 101, 102, ..., 200
1-200%10        # 1, 2, ..., 200, but maximum 10 running at the same time

5. Interactive Job:

A user can also request node(s) for an interactive job:

Interactive Job
[<username>@idun-login1 ~]$ srun --account=<account> --nodes=1 --partition=CPUQ --time=00:30:00 --pty bash 
srun: job 416172 queued and waiting for resources
srun: job 416172 has been allocated resources

[compute-1-0-27 ~]$

The above example requests a single node from the CPUQpartition for 30 minutes. After the allocation, the user is prompted with a bash shell on the requested node.

An alternative way to accomplish the same interactive job is:

[<username>@idun-login1 ~]$ salloc --account=<account> --nodes=1 --partition=CPUQ --time=00:30:00
salloc: Pending job allocation 416189
salloc: job 416189 queued and waiting for resources
salloc: job 416189 has been allocated resources
salloc: Granted job allocation 416189
salloc: Waiting for resource configuration
salloc: Nodes compute-1-0-27 are ready for job

[<username>@idun-login1 ~]$ ssh compute-1-0-27

[reissman@compute-1-0-27 ~]$

The first command reserves again a single node from the WORKQ partition for 30 minutes, but in contrast to srun, does not automatically log into the allocated node. The login is then achieved by using the ssh command.

6. Running Graphical User Interface (GUI) Applications:

In order to use GUI applications, it is necessary to use the -X flag when logging into the cluster:

XForwarding
$ ssh -X <username>@idun-login1.hpc.ntnu.no
<username>@idun-login1.hpc.ntnu.no's password:

[<username>@idun-login1 ~]$

This will enable the forwarding of a GUI applications' windows to your desktop machine. After login, an interactive job can be started for the execution of the GUI application:

[<username>@idun-login1 ~] salloc --account=<account> --nodes=1 --partition=CPUQ --time=00:30:00
salloc: Pending job allocation 416194
salloc: job 416194 queued and waiting for resources
salloc: job 416194 has been allocated resources
salloc: Granted job allocation 416194
salloc: Waiting for resource configuration
salloc: Nodes compute-1-0-27 are ready for job

[<username>@idun-login1 ~]@ ssh -X compute-1-0-27

[<username>@compute-1-0-27 ~] xclock

The first command reserves a single node from the WORKQ partition for 30 minutes. After the allocation has been granted, it is possible to log into this node using the -X flag and start the GUI application (xclock in the above example).

7. Python job script example (Slurm):

Exampel Python/3.8.6-GCCcore-10.2.0. Use "module spider python" to find versions.

#!/bin/bash
#SBATCH --job-name="my-job"   # Sensible name for the job
#SBATCH --account=<account>   # Account for consumed resources
#SBATCH --nodes=1             # Allocate 1 nodes for the job
#SBATCH -c28                  # Number of cores (can vary)
#SBATCH --time=00-00:10:00    # Upper time limit for the job (DD-HH:MM:SS)
#SBATCH --partition=CPUQ

module load Python/3.8.6-GCCcore-10.2.0

python mypython.py

For parallel python see: Parallel Python – High Performance Computing Group (ntnu.no)

8. Matlab job script example (Slurm):

#!/bin/bash
#SBATCH --job-name="my-job"   # Sensible name for the job
#SBATCH --account=<account>   # Account for consumed resources
#SBATCH --nodes=1             # Allocate 1 nodes for the job
#SBATCH -c28                  # Number of cores (can vary)
#SBATCH --time=00-00:10:00    # Upper time limit (DD-HH:MM:SS)
#SBATCH --partition=CPUQ

module load MATLAB/2021b

matlab -nodisplay -nodesktop -nosplash -nojvm -r "test"

(NOTE! If using Parallel Computing Toolbox, remove -nojvm as: matlab -nodisplay -nodesktop -nosplash -r "test")

To check Matlab versions, type: module spider matlab

For parallel Matlab (using MPI) see: Distributed Matlab(using MPI) – High Performance Computing Group (ntnu.no)

9. COMSOL job script example (Slurm):

#!/bin/bash
########################################################
#
#  Running COMSOL cluster job
#
#  The default setup is to run COMSOL in hybrid mode
#  with 2 MPI processes per node ('mpiprocs'), one per 
#  socket, and 8 threads ('ompthreads') per MPI process
#
########################################################
#
#SBATCH --partition=CPUQ    # partition the batch job will be put in
#SABTCH --account=<account>  # Account for consumed resources
#SBATCH --time=00:30:00      # the walltime length of the job (30 minutes)
#SBATCH --nodes=2            # number of nodes requested
#SBATCH --ntasks-per-node=2  # number of processes per node
#SBATCH --job-name="comsol_example"  # name of the job
#SBATCH --output=comsol.out  # name of output file
#SBATCH -J COMSOL_example   # Name for the job 
 
module load COMSOL/5.3a


# ${SLURM_SUBMIT_DIR} - directory from where the job were submitted 
# Create (if necessary) the working directory
w=${SLURM_SUBMIT_DIR}/comsol/$case
if [ ! -d $w ]; then mkdir -p $w; fi
 
# Copy inputfile and move to working directory
cp $case.mph $w
cd $w
 
comsol batch -mpibootstrap ssh -inputfile $case.mph -outputfile out.mph -tmpdir $w

Controlling Slurm jobs

# Get all jobs
squeue
 
# get all jobs for user < only pending | only running > in <partition>
squeue -u username <-t PENDING|-t RUNNING> <-p partition>
 
# Show detailed info on <jobid>
scontrol show jobid -dd <jobid>
 
# cancel specific <jobid>
scancel < jobid >
 
# cancel all <pending> jobs for <username>
scancel <-t PENDING> -u <username>

# show available resources
sinfo -o "%10P %5D %34N  %5c  %7m  %47f  %23G"

Table of contents