Job dependencies

Video: Job Dependencies (9 mins) - note this is not for ada.uea.ac.uk, but demonstrates the principle.

Job dependencies can be used to build complex pipelines or chain together long jobs requiring multiple steps.

You can run a set of jobs sequentially, so that the second job runs only after the first one has completed. For example, if you have two jobs, Job1.sub and Job2.sub, you can utilize job dependencies from the command line

sbatch Job1.sub    (gets allocated jobID 123213)

sbatch --dependency=afterany:123213 Job2.sub  (will start after the fist one ends)
 

The flag --dependency=afterany:123213 tells the queueing system to start the second job only after completion of the first job.

afterany indicates that Job2 will run whatever the exit status of Job1, i.e. regardless of whether Job1 completed successfully or not.

Exit status

The exit status of a job is the exit status of the last command that was run in the batch script. An exit status of '0' means that the queueing system considers the job completed successfully - this does not necessarily mean that all commands in the batch script completed successfully.

You can decide what the dependency to rhe previous job should be.

--dependency=afterany:Job1 Job2 will start after Job1 completes with any exit status
--dependency=after:Job1 Job2 will start any time after Job1 starts
--dependency=afterok:Job1 Job2 will run only if Job1 completed with an exit status of 0
--dependency=afternotok:Job1 Job2 will run only if Job1 completed with a non-zero exit status

 

Multiple jobs

Making several jobs depend on the completion of a single job is simple from the command line, eg

sbatch Job1.sub    (gets allocated jobID 123213)

sbatch --dependency=afterany:123213 Job2.sub  (allocated jobID 123215)

sbatch --dependency=afterany:123213 Job3.sub  (allocated jobID 123219)

sbatch --dependency=afterok:123219 Job4.sub   (will run once Job3.sub completes)

Make a job depend on the completion of several other jobs, eg

sbatch Job1.sub    (gets allocated jobID 123213)

sbatch Job2.sub    (gets allocated jobID 123216)

sbatch --dependency=afterany:123213,123216 Job3.sub

 --parsable option to sbatch

Using --parsable to determine the job id can simplify working with job dependencies. 

jobid=$(sbatch --parsable first_job.sub)  
sbatch --dependency=afterok:$jobid second_job.sub

Chaining jobs

To join jobs together in a submission script follows the same principles by adding extra lines to your script for the first job.

#!/bin/bash
<snip commands for first job>
run_some_command
sbatch --dependency=afterany:$SLURM_JOB_ID  second_job.sub