How to use Slurm
Contents
What is Slurm?
Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system used by various entities across UCSB. The College of Engineering (CoE) is using Slurm on our GPU cluster to give additional computing options for those in need of heavy calculations.
Please note that these are generic instructions and exact procedure will vary.
I Thought UCSB was using Torque?
SLURM is similar to Torque, which was used by various UCSB entities in the past, but Slurm has the advantage of being free and scalable when applied to multiple systems at the same time. Which is especially relevant if we want the same cluster to be available from ALL CoE lab and research systems, which run a variety of operating systems such as Fedora, Centos and Windows. We will outline some differences below.
The major differences to be aware of:
Queues are known as Partitions, which means instead of the argument when submitting a job "-q short" to send something to the short (or some other) queue. on slurm it is now "-p short" (p for partition).
You'll need to change the various 'PBS' variables in your script. The common ones are listed below.
SBATCH partitions you can submit to: sbatch my.job (standard compute node) , sbatch -p short my.job (short queue, 1 hour - for testing), sbatch -p gpu my.job (GPU nodes), sbatch -p largemem my.job (large memory 1.5TB nodes)
What | Torque | Slurm |
|---|---|---|
Nodes/Cores | #PBS -l nodes=1:ppn=10 | #SBATCH --nodes=1 --ntasks-per-node=10 |
Walltime | #PBS -l walltime=1:00:00 | #SBATCH --time=1:00:00 |
mail to user | #PBS -M username@ucsb.edu | #SBATCH --mail-user=user@ucsb.edu |
Mail begin/end | #PBS -m be | #SBATCH --mail-type=start,end |
Working Directory | $PBS_O_WORKDIR | $SLURM_SUBMIT_DIR |
Overview on how to Submit Jobs
Please note that these are generic instructions and exact procedure will vary.
Slurm will take all of your environment variables that your login shell has, so if you need a compiler, or Matlab, etc., do the 'module load' for it before you submit your job.
Basic to run a job is 'sbatch' (from Torque it was 'qsub'), e.g. you have a file named 'test.slurm' that looks like this (for first a serial, then a parallel job)
#!/bin/bash -l
#Serial (1 core on one node) job...
#SBATCH --nodes=1 --ntasks-per-node=1
cd $SLURM_SUBMIT_DIR
time ./a.out >& logfileand a simple parallel (MPI) example from CSC and CNSI
https://csc.cnsi.ucsb.edu/docs/slurm-job-scheduler
#!/bin/bash -l
# ask for 16 cores on two nodes
#SBATCH --nodes=2 --ntasks-per-node=16
cd $SLURM_SUBMIT_DIR
/bin/hostname
mpirun -np $SLURM_NTASKS ./a.outNote: not all clusters have the same number of cores, CSC and CNSI have super computing clusters and can thus use more cores than most systems on campus. If you are unsure on number of available cores, use only a single core.
Notice that the main changes from PBS are a slightly different format on choosing the number of nodes/cores and also the directory name to CD to. For MPI jobs, it's actually somewhat simplified in you don't need to give it a nodes file.
You run this job with 'sbatch test.slurm' (you used to use 'qsub')
You can check on the status with squeue (formerly 'qstat') e.g.
squeue -u $USER (to see only your jobs, 'squeue' will show every job on the system)
You can look at details with 'scontrol show job JIOBID', sort of like the old 'qstat -f' command.
To kill a job you use 'scancel -i JOBID' (formerly 'qdel JOBID')
If you want an interactive node to test some things to make sure your job will run, you can do this with
srun -N 1 -p short --ntasks-per-node=4 --pty bash (which asks for 8 cores on the short queue node, which will run for up to an hour)
Torque | Slurm |
|---|---|
#PBS -J myjob | #SBATCH -J myjob |
CoE Specific Instructions
From Within the GPU Cluster
You can use the srun command to run a command within the cluster such as, srun python3 pytorch.py or srum ./pytorch.py if the Python script is executable and has the python interpreter set.
The more important command would be sbatch. You can create an sbatch file and setup your parameters in there and submit the job and retrieve the output at a later point in time. Below is the example sbatch file used to test the system.
To create the file:
cat sbatch.batchexample contents of file:
Then run
Once you've submitted your job it will tell you the job id. You can follow the job's progress with squeue and the stdout will be logged into slurm-jobid.out.