Slurm/slurm-setup.md

SLURM Setup SOP (Single Node CPU + GPU)

Overview

This document is a single-node SLURM cluster setup guide where:

• CPU + GPU scheduling is enabled
• Munge authentication is used
• Basic job submission is tested

---

Architecture (High Level)

Component	Detail
Controller + Compute Node	Same machine
Scheduler	SLURM
Auth	Munge
GPU Support	GRES (Generic Resource Scheduling)

---

Prerequisites

OS Setup

apt update && apt upgrade -y

GPU Driver Install

ubuntu-drivers autoinstall
reboot
nvidia-smi

Notes:

• ubuntu-drivers autoinstall — automatically installs the best NVIDIA driver
• nvidia-smi — verifies that the GPU is being detected

---

Munge Setup (Authentication Layer)

Install + Configure

sudo apt install munge -y

sudo mkdir -p /etc/munge
sudo dd if=/dev/urandom of=/etc/munge/munge.key bs=1 count=1024

sudo chown munge:munge /etc/munge/munge.key
sudo chmod 400 /etc/munge/munge.key

sudo systemctl enable --now munge

Test

munge -n | unmunge

Notes:

• Munge = SLURM's authentication backbone
• A shared key establishes trust between nodes
• munge -n | unmunge — generates and verifies a token

---

SLURM Installation

sudo apt install slurm-wlm slurmd slurmctld slurm-client -y

Components

Package	Role
slurmctld	Controller daemon (scheduler brain)
slurmd	Node agent
slurm-client	CLI tools (sbatch, squeue)
slurm-wlm	Core workload manager

---

SLURM Configuration

Main Config File — /etc/slurm/slurm.conf

ClusterName=single
SlurmctldHost=brilliant

SlurmUser=slurm
SlurmdUser=root
AuthType=auth/munge

StateSaveLocation=/tmp/slurmctld
SlurmdSpoolDir=/tmp/slurmd

SelectType=select/cons_tres
SelectTypeParameters=CR_Core
SchedulerType=sched/backfill

GresTypes=gpu

NodeName=brilliant CPUs=4 RealMemory=32000 Gres=gpu:1 State=UNKNOWN
PartitionName=debug Nodes=brilliant Default=YES State=UP

Config Reference

Config	Meaning
ClusterName	Cluster name
SlurmctldHost	Controller node hostname
SlurmUser	SLURM service user
AuthType=auth/munge	Munge-based authentication
StateSaveLocation	Scheduler state store location
SlurmdSpoolDir	Node job spool directory
SelectType=cons_tres	Resource tracking (CPU, GPU, mem)
GresTypes=gpu	Enables GPU resource
NodeName	Node definition
CPUs	Available CPU cores
RealMemory	RAM in MB
Gres=gpu:1	GPU count
PartitionName	Job queue

---

Directory Setup

sudo mkdir -p /tmp/slurmctld /tmp/slurmd
sudo chown -R slurm:slurm /tmp/slurmctld

---

Service Fix (Common Issue)

sudo nano /etc/default/slurmctld

Uncomment the following line:

SLURM_OPTIONS=""

Note: Without this fix, the SLURM daemon may fail to start due to missing default options.

---

Restart Services

sudo systemctl restart munge
sudo systemctl restart slurmctld
sudo systemctl restart slurmd

---

Basic Test (CPU Job)

sbatch --wrap="hostname; sleep 5"
squeue

Notes:

• sbatch — submits a job
• --wrap — runs an inline command
• squeue — displays the job queue

---

GPU Configuration

GRES Config File — /etc/slurm/gres.conf

NodeName=brilliant Name=gpu File=/dev/nvidia0

Note: This maps the GPU device to SLURM. /dev/nvidia0 refers to the first GPU.

GPU Validation

scontrol show node

Expected output:

Gres=gpu:1

GPU Job Test

sbatch --gres=gpu:1 --wrap="nvidia-smi"

Notes:

• --gres=gpu:1 — allocates 1 GPU to the job
• nvidia-smi runs inside the job to confirm GPU access