215 lines
3.8 KiB
Markdown
215 lines
3.8 KiB
Markdown
# SLURM Setup SOP (Single Node CPU + GPU)
|
|
|
|
## Overview
|
|
|
|
This document is a **single-node SLURM cluster setup guide** where:
|
|
- CPU + GPU scheduling is enabled
|
|
- Munge authentication is used
|
|
- Basic job submission is tested
|
|
|
|
---
|
|
|
|
## Architecture (High Level)
|
|
|
|
| Component | Detail |
|
|
|---|---|
|
|
| Controller + Compute Node | Same machine |
|
|
| Scheduler | SLURM |
|
|
| Auth | Munge |
|
|
| GPU Support | GRES (Generic Resource Scheduling) |
|
|
|
|
---
|
|
|
|
## Prerequisites
|
|
|
|
### OS Setup
|
|
|
|
```bash
|
|
apt update && apt upgrade -y
|
|
```
|
|
|
|
### GPU Driver Install
|
|
|
|
```bash
|
|
ubuntu-drivers autoinstall
|
|
reboot
|
|
nvidia-smi
|
|
```
|
|
|
|
> **Notes:**
|
|
> - `ubuntu-drivers autoinstall` — automatically installs the best NVIDIA driver
|
|
> - `nvidia-smi` — verifies that the GPU is being detected
|
|
|
|
---
|
|
|
|
## Munge Setup (Authentication Layer)
|
|
|
|
### Install + Configure
|
|
|
|
```bash
|
|
sudo apt install munge -y
|
|
|
|
sudo mkdir -p /etc/munge
|
|
sudo dd if=/dev/urandom of=/etc/munge/munge.key bs=1 count=1024
|
|
|
|
sudo chown munge:munge /etc/munge/munge.key
|
|
sudo chmod 400 /etc/munge/munge.key
|
|
|
|
sudo systemctl enable --now munge
|
|
```
|
|
|
|
### Test
|
|
|
|
```bash
|
|
munge -n | unmunge
|
|
```
|
|
|
|
> **Notes:**
|
|
> - Munge = SLURM's authentication backbone
|
|
> - A shared key establishes trust between nodes
|
|
> - `munge -n | unmunge` — generates and verifies a token
|
|
|
|
---
|
|
|
|
## SLURM Installation
|
|
|
|
```bash
|
|
sudo apt install slurm-wlm slurmd slurmctld slurm-client -y
|
|
```
|
|
|
|
### Components
|
|
|
|
| Package | Role |
|
|
|---|---|
|
|
| `slurmctld` | Controller daemon (scheduler brain) |
|
|
| `slurmd` | Node agent |
|
|
| `slurm-client` | CLI tools (`sbatch`, `squeue`) |
|
|
| `slurm-wlm` | Core workload manager |
|
|
|
|
---
|
|
|
|
## SLURM Configuration
|
|
|
|
### Main Config File — `/etc/slurm/slurm.conf`
|
|
|
|
```ini
|
|
ClusterName=single
|
|
SlurmctldHost=brilliant
|
|
|
|
SlurmUser=slurm
|
|
SlurmdUser=root
|
|
AuthType=auth/munge
|
|
|
|
StateSaveLocation=/tmp/slurmctld
|
|
SlurmdSpoolDir=/tmp/slurmd
|
|
|
|
SelectType=select/cons_tres
|
|
SelectTypeParameters=CR_Core
|
|
SchedulerType=sched/backfill
|
|
|
|
GresTypes=gpu
|
|
|
|
NodeName=brilliant CPUs=4 RealMemory=32000 Gres=gpu:1 State=UNKNOWN
|
|
PartitionName=debug Nodes=brilliant Default=YES State=UP
|
|
```
|
|
|
|
### Config Reference
|
|
|
|
| Config | Meaning |
|
|
|---|---|
|
|
| `ClusterName` | Cluster name |
|
|
| `SlurmctldHost` | Controller node hostname |
|
|
| `SlurmUser` | SLURM service user |
|
|
| `AuthType=auth/munge` | Munge-based authentication |
|
|
| `StateSaveLocation` | Scheduler state store location |
|
|
| `SlurmdSpoolDir` | Node job spool directory |
|
|
| `SelectType=cons_tres` | Resource tracking (CPU, GPU, mem) |
|
|
| `GresTypes=gpu` | Enables GPU resource |
|
|
| `NodeName` | Node definition |
|
|
| `CPUs` | Available CPU cores |
|
|
| `RealMemory` | RAM in MB |
|
|
| `Gres=gpu:1` | GPU count |
|
|
| `PartitionName` | Job queue |
|
|
|
|
---
|
|
|
|
## Directory Setup
|
|
|
|
```bash
|
|
sudo mkdir -p /tmp/slurmctld /tmp/slurmd
|
|
sudo chown -R slurm:slurm /tmp/slurmctld
|
|
```
|
|
|
|
---
|
|
|
|
## Service Fix (Common Issue)
|
|
|
|
```bash
|
|
sudo nano /etc/default/slurmctld
|
|
```
|
|
|
|
Uncomment the following line:
|
|
|
|
```
|
|
SLURM_OPTIONS=""
|
|
```
|
|
|
|
> **Note:** Without this fix, the SLURM daemon may fail to start due to missing default options.
|
|
|
|
---
|
|
|
|
## Restart Services
|
|
|
|
```bash
|
|
sudo systemctl restart munge
|
|
sudo systemctl restart slurmctld
|
|
sudo systemctl restart slurmd
|
|
```
|
|
|
|
---
|
|
|
|
## Basic Test (CPU Job)
|
|
|
|
```bash
|
|
sbatch --wrap="hostname; sleep 5"
|
|
squeue
|
|
```
|
|
|
|
> **Notes:**
|
|
> - `sbatch` — submits a job
|
|
> - `--wrap` — runs an inline command
|
|
> - `squeue` — displays the job queue
|
|
|
|
---
|
|
|
|
## GPU Configuration
|
|
|
|
### GRES Config File — `/etc/slurm/gres.conf`
|
|
|
|
```ini
|
|
NodeName=brilliant Name=gpu File=/dev/nvidia0
|
|
```
|
|
|
|
> **Note:** This maps the GPU device to SLURM. `/dev/nvidia0` refers to the first GPU.
|
|
|
|
### GPU Validation
|
|
|
|
```bash
|
|
scontrol show node
|
|
```
|
|
|
|
**Expected output:**
|
|
|
|
```
|
|
Gres=gpu:1
|
|
```
|
|
|
|
### GPU Job Test
|
|
|
|
```bash
|
|
sbatch --gres=gpu:1 --wrap="nvidia-smi"
|
|
```
|
|
|
|
> **Notes:**
|
|
> - `--gres=gpu:1` — allocates 1 GPU to the job
|
|
> - `nvidia-smi` runs inside the job to confirm GPU access |