Discovery quickstart

For a free Slurm account on Dartmouth's Discovery cluster. Research Computing's own guides are on the Resources page.

Getting on

Off campus you need the Dartmouth VPN first. Then authenticate with Kerberos and connect.

kinit <netid>@KIEWIT.DARTMOUTH.EDU
ssh <netid>@discovery.dartmouth.edu

Where to put your files

Logging in lands you in your rc-home directory, which is not automounted on the compute nodes. A job that tries to read or write there fails with a permission error that has nothing to do with permissions.

Do your work in scratch instead.

cd /dartfs-hpc/scratch/<netid>

Keep code, environments and outputs under that path. Scratch is not backed up, so anything you want to keep should also live in a git repository or in your rc-home.

What the free account allows

The free account has one cap that binds, a limit of 32 CPUs summed across all your running jobs at once. There is no cap on the number of GPUs you may use. If your jobs stop starting, check your current CPU total before assuming the cluster is full.

squeue --me -h -t R -o "%C" | paste -sd+ | bc

With preprocessing to keep CPU requirements down I'm usually able to get away with 1 CPU per GPU, enabling me to use 32 GPUs concurrently.

Walltime decides which partition you can use

This is the rule that governs everything else. The partitions with plenty of free GPUs have short time limits, and the ones with long time limits are small. So the --time you request decides where your job can be scheduled.

Every partition in that table is open to any free account. A partition also accepts anything shorter than its limit, so a5500_w will take an 8 hour job too.

If you genuinely need more than 7 days, v100_preemptable allows up to 30 and is open to any free account. It is easy to miss because the name suggests it only holds v100 cards; it also carries l40s nodes at 48 GB.

Even so, prefer short jobs. If your work can save its state and restart, ask for 8 hours and resume across several submissions. That gets you into the large pools and will usually finish sooner in wall-clock terms than a single long request that queues for days, and because every partition can preempt you, a job that checkpoints is the only kind that reliably finishes.

Every partition on Discovery can stop your job partway to give the node back, not only the ones whose names end in _preempt or _preemptable. Checked on November 3, 2026, all 28 partitions report PreemptMode=REQUEUE, including a5500, a5500_w, a5000, a5000_w, gpuq and 2080ti. The suffix tells you who owns the hardware, not whether you can be interrupted.

So treat every job as interruptible. Set --requeue on all of them, and checkpoint your progress regularly, or a stop costs you everything since the last save. This is the single most expensive mistake on this cluster. A long job with no checkpoint that gets preempted at hour 39 has to start again from zero.

scontrol show partition <name> | tr ' ' '\n' | grep PreemptMode

Some partitions are reserved for specific labs and a free account cannot use them. Listing one only produces a job that pends forever. Confirm what any partition allows before relying on it.

scontrol show partition <name> | tr ' ' '\n' | grep -E 'AllowAccounts|MaxTime'

Picking a GPU by memory

Pick by how much GPU memory your work needs rather than by GPU name.

A model that needs 40 GB will not fit on a 24 GB card, and the job will fail after it has already waited in the queue.

A job script that works

Save this as run.sbatch and edit the top few lines.

#!/bin/bash
#SBATCH --job-name=myjob
#SBATCH --partition=gpu_preempt
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=4
#SBATCH --time=08:00:00
#SBATCH --requeue
#SBATCH --output=logs/%x_%j.log

set -euo pipefail
mkdir -p logs
cd /dartfs-hpc/scratch/$USER/myproject

source /optnfs/common/miniconda3/etc/profile.d/conda.sh
conda activate /dartfs-hpc/scratch/$USER/envs/myenv

python train.py

The two lines that load conda matter more than they look. Without them the script only works when the shell you submitted from already had conda available, so it runs by hand and then fails from a scheduled job or a fresh login.

Submitting and watching

For a quick test, ask for a session on a compute node and run commands by hand rather than submitting a script.

srun --partition=gpu_preempt --gres=gpu:1 --cpus-per-task=2 --time=01:00:00 --pty bash

When a job stays pending

Free GPUs on the cluster and a pending job usually means something other than the cluster being full. Work through these in order.

1. Your account cannot use the partition you asked for. Check AllowAccounts.
2. Your --time is longer than that partition allows. Check MaxTime.
3. You pinned a GPU type that is scarce or reserved.
4. You are at your own 32-CPU limit across running jobs.
5. GPUs are free but their nodes have no free CPUs left.

The reason field often names the cause directly.

squeue --me -o "%.10i %.9P %.8T %.10M %R"

When a job dies in a few seconds

An exit code of 127 with a log line saying conda was not found is the conda problem described above. Add the two conda lines to your script.

A job that ends almost immediately with no output at all has usually failed on a path. Confirm the directory you cd into exists on the compute node, and remember that your rc-home does not.