# SLURM Setup SOP (Single Node CPU + GPU)

## Overview

This document is a **single-node SLURM cluster setup guide** where:
- CPU + GPU scheduling is enabled
- Munge authentication is used
- Basic job submission is tested

---

## Architecture (High Level)

| Component | Detail |
|---|---|
| Controller + Compute Node | Same machine |
| Scheduler | SLURM |
| Auth | Munge |
| GPU Support | GRES (Generic Resource Scheduling) |

---

## Prerequisites

### OS Setup

```bash
apt update && apt upgrade -y
```

### GPU Driver Install

```bash
ubuntu-drivers autoinstall
reboot
nvidia-smi
```

> **Notes:**
> - `ubuntu-drivers autoinstall` — automatically installs the best NVIDIA driver
> - `nvidia-smi` — verifies that the GPU is being detected

---

## Munge Setup (Authentication Layer)

### Install + Configure

```bash
sudo apt install munge -y

sudo mkdir -p /etc/munge
sudo dd if=/dev/urandom of=/etc/munge/munge.key bs=1 count=1024

sudo chown munge:munge /etc/munge/munge.key
sudo chmod 400 /etc/munge/munge.key

sudo systemctl enable --now munge
```

### Test

```bash
munge -n | unmunge
```

> **Notes:**
> - Munge = SLURM's authentication backbone
> - A shared key establishes trust between nodes
> - `munge -n | unmunge` — generates and verifies a token

---

## SLURM Installation

```bash
sudo apt install slurm-wlm slurmd slurmctld slurm-client -y
```

### Components

| Package | Role |
|---|---|
| `slurmctld` | Controller daemon (scheduler brain) |
| `slurmd` | Node agent |
| `slurm-client` | CLI tools (`sbatch`, `squeue`) |
| `slurm-wlm` | Core workload manager |

---

## SLURM Configuration

### Main Config File — `/etc/slurm/slurm.conf`

```ini
ClusterName=single
SlurmctldHost=brilliant

SlurmUser=slurm
SlurmdUser=root
AuthType=auth/munge

StateSaveLocation=/tmp/slurmctld
SlurmdSpoolDir=/tmp/slurmd

SelectType=select/cons_tres
SelectTypeParameters=CR_Core
SchedulerType=sched/backfill

GresTypes=gpu

NodeName=brilliant CPUs=4 RealMemory=32000 Gres=gpu:1 State=UNKNOWN
PartitionName=debug Nodes=brilliant Default=YES State=UP
```

### Config Reference

| Config | Meaning |
|---|---|
| `ClusterName` | Cluster name |
| `SlurmctldHost` | Controller node hostname |
| `SlurmUser` | SLURM service user |
| `AuthType=auth/munge` | Munge-based authentication |
| `StateSaveLocation` | Scheduler state store location |
| `SlurmdSpoolDir` | Node job spool directory |
| `SelectType=cons_tres` | Resource tracking (CPU, GPU, mem) |
| `GresTypes=gpu` | Enables GPU resource |
| `NodeName` | Node definition |
| `CPUs` | Available CPU cores |
| `RealMemory` | RAM in MB |
| `Gres=gpu:1` | GPU count |
| `PartitionName` | Job queue |

---

## Directory Setup

```bash
sudo mkdir -p /tmp/slurmctld /tmp/slurmd
sudo chown -R slurm:slurm /tmp/slurmctld
```

---

## Service Fix (Common Issue)

```bash
sudo nano /etc/default/slurmctld
```

Uncomment the following line:

```
SLURM_OPTIONS=""
```

> **Note:** Without this fix, the SLURM daemon may fail to start due to missing default options.

---

## Restart Services

```bash
sudo systemctl restart munge
sudo systemctl restart slurmctld
sudo systemctl restart slurmd
```

---

## Basic Test (CPU Job)

```bash
sbatch --wrap="hostname; sleep 5"
squeue
```

> **Notes:**
> - `sbatch` — submits a job
> - `--wrap` — runs an inline command
> - `squeue` — displays the job queue

---

## GPU Configuration

### GRES Config File — `/etc/slurm/gres.conf`

```ini
NodeName=brilliant Name=gpu File=/dev/nvidia0
```

> **Note:** This maps the GPU device to SLURM. `/dev/nvidia0` refers to the first GPU.

### GPU Validation

```bash
scontrol show node
```

**Expected output:**

```
Gres=gpu:1
```

### GPU Job Test

```bash
sbatch --gres=gpu:1 --wrap="nvidia-smi"
```

> **Notes:**
> - `--gres=gpu:1` — allocates 1 GPU to the job
> - `nvidia-smi` runs inside the job to confirm GPU access