How to handle long running jobs

If you want to run long running jobs on our Linux workstations you have to abide by a few rules.

General rules
  • Do not bother anyone with your jobs. Keep the machine usable. Use nice or renice with at least nice level 15, better 18.
  • Do not run RAM or CPU time intensive jobs at all on our login server as there are usually dozens of users logged in and try to do their daily work on it. Clogging up those machines causes many users not being able to work anymore, so no long running or heavily resource consuming computing jobs are allowed on these two machines. Please use one of the computing nodes instead.
  • You can use ps, top or htop to monitor CPU and memory usage of the running jobs. Use nvtop to monitor GPU and GPU memory.
  • Keep an eye on the RAM usage of your programs and avoid using too much memory.
  • Do not run computational jobs on a network filesystem (group share, nor home share)! Use the local disks (e.g. in /scratch/) instead for such cases.
  • You should not run several jobs at once unless you know what you are doing. In many cases, it's less efficient than running jobs in sequence.
Personal Workstations

If you run long running jobs on your personal workstation you do not need to look after keeping the machine usable for other users. Though

  • you may not stress your machine so much that it runs out of memory or disk space or system services stop working. If it starts swapping, it will become much slower and inefficient. Use free, swapon, or vmstat to check swap usage.