Please enable JavaScript for full functionality

IT Services Group D-PHYS

How to handle long running jobs

If you want to run long running jobs on our Linux workstations you have to abide by a few rules.

Best practices

Start long-running, cpu intensive or memory intensive jobs in a systemd scope unit in the pre-defined user background.slice (control group):

systemd-run --user --scope --slice background -u myscope1 -p OOMPolicy=continue screen -S myscreen1

This automatically applies less priority to those processes started in such a slice and keeps your foreground processes responsive (desktop, shell, etc.). It also shields all other processes from out-of-memory (OOM) kills when your long running job triggers an OOM.

Multiple jobs are best started in separate scopes (and screens) to fully isolate them (so that if one of the jobs exceeds a cgroup memory/swap limit, only the cgroup (scope) containing that job will be killed):

systemd-run --user --scope --slice background -u myscope2 -p OOMPolicy=continue screen -S myscreen2
systemd-run --user --scope --slice background -u myscope3 -p OOMPolicy=continue screen -S myscreen3 path/to/compute_job.py

Use custom cgroup settings to limit resources of a slice (example: 1GiB memory, 100MiB swap):

systemd-run --user --scope --slice background -u myscope1 -p OOMPolicy=continue -p MemoryMax=1G -p MemorySwapMax=100M screen -S myscreen4

See background or compute jobs and custom user scopes and settings for more information.

General rules
  • Do not bother anyone with your jobs. Keep the machine usable. Use slices and cgroup settings or in additionally: nice or renice with at least nice level 15, better 18.
  • Do not run RAM or CPU time intensive jobs at all on our login server login.phys.ethz.ch as there are usually dozens of users logged in and try to do their daily work on it. Clogging up those machines causes many users not being able to work anymore, so no long running or heavily resource consuming computing jobs are allowed on these two machines. Please use one of the computing nodes instead.
  • You can use ps, top, htop, oomctl, systemd-cgtop, psi, psitop or webbased performance metrics to monitor CPU and memory usage of the running jobs. Use nvtop or webbased gpu performance metrics to monitor GPU and GPU memory.
  • Keep an eye on the RAM usage of your programs and avoid using too much memory.
  • Do not run computational jobs on a network filesystem (group share, nor home share)! Use the local disks (e.g. in /scratch/) instead for such cases.
  • You should not run several jobs at once unless you know what you are doing. In many cases, it's less efficient than running jobs in sequence.
Personal Workstations

If you run long running jobs on your personal workstation you do not need to look after keeping the machine usable for other users. Though

  • you may not stress your machine so much that it runs out of memory or disk space or system services stop working. If it starts swapping, it will become much slower and inefficient. Use free, swapon, or vmstat to check swap usage.