Add slurm-setup.md

This is the first slurm setup with GPU server in standalone machine

slurm-setup.md

@@ -0,0 +1,215 @@
+# SLURM Setup SOP (Single Node CPU + GPU)
+
+## Overview
+
+This document is a **single-node SLURM cluster setup guide** where:
+- CPU + GPU scheduling is enabled
+- Munge authentication is used
+- Basic job submission is tested
+
+---
+
+## Architecture (High Level)
+
+| Component | Detail |
+|---|---|
+| Controller + Compute Node | Same machine |
+| Scheduler | SLURM |
+| Auth | Munge |
+| GPU Support | GRES (Generic Resource Scheduling) |
+
+---
+
+## Prerequisites
+
+### OS Setup
+
+```bash
+apt update && apt upgrade -y
+```
+
+### GPU Driver Install
+
+```bash
+ubuntu-drivers autoinstall
+reboot
+nvidia-smi
+```
+
+> **Notes:**
+> - `ubuntu-drivers autoinstall` — automatically installs the best NVIDIA driver
+> - `nvidia-smi` — verifies that the GPU is being detected
+
+---
+
+## Munge Setup (Authentication Layer)
+
+### Install + Configure
+
+```bash
+sudo apt install munge -y
+
+sudo mkdir -p /etc/munge
+sudo dd if=/dev/urandom of=/etc/munge/munge.key bs=1 count=1024
+
+sudo chown munge:munge /etc/munge/munge.key
+sudo chmod 400 /etc/munge/munge.key
+
+sudo systemctl enable --now munge
+```
+
+### Test
+
+```bash
+munge -n | unmunge
+```
+
+> **Notes:**
+> - Munge = SLURM's authentication backbone
+> - A shared key establishes trust between nodes
+> - `munge -n | unmunge` — generates and verifies a token
+
+---
+
+## SLURM Installation
+
+```bash
+sudo apt install slurm-wlm slurmd slurmctld slurm-client -y
+```
+
+### Components
+
+| Package | Role |
+|---|---|
+| `slurmctld` | Controller daemon (scheduler brain) |
+| `slurmd` | Node agent |
+| `slurm-client` | CLI tools (`sbatch`, `squeue`) |
+| `slurm-wlm` | Core workload manager |
+
+---
+
+## SLURM Configuration
+
+### Main Config File — `/etc/slurm/slurm.conf`
+
+```ini
+ClusterName=single
+SlurmctldHost=brilliant
+
+SlurmUser=slurm
+SlurmdUser=root
+AuthType=auth/munge
+
+StateSaveLocation=/tmp/slurmctld
+SlurmdSpoolDir=/tmp/slurmd
+
+SelectType=select/cons_tres
+SelectTypeParameters=CR_Core
+SchedulerType=sched/backfill
+
+GresTypes=gpu
+
+NodeName=brilliant CPUs=4 RealMemory=32000 Gres=gpu:1 State=UNKNOWN
+PartitionName=debug Nodes=brilliant Default=YES State=UP
+```
+
+### Config Reference
+
+| Config | Meaning |
+|---|---|
+| `ClusterName` | Cluster name |
+| `SlurmctldHost` | Controller node hostname |
+| `SlurmUser` | SLURM service user |
+| `AuthType=auth/munge` | Munge-based authentication |
+| `StateSaveLocation` | Scheduler state store location |
+| `SlurmdSpoolDir` | Node job spool directory |
+| `SelectType=cons_tres` | Resource tracking (CPU, GPU, mem) |
+| `GresTypes=gpu` | Enables GPU resource |
+| `NodeName` | Node definition |
+| `CPUs` | Available CPU cores |
+| `RealMemory` | RAM in MB |
+| `Gres=gpu:1` | GPU count |
+| `PartitionName` | Job queue |
+
+---
+
+## Directory Setup
+
+```bash
+sudo mkdir -p /tmp/slurmctld /tmp/slurmd
+sudo chown -R slurm:slurm /tmp/slurmctld
+```
+
+---
+
+## Service Fix (Common Issue)
+
+```bash
+sudo nano /etc/default/slurmctld
+```
+
+Uncomment the following line:
+
+```
+SLURM_OPTIONS=""
+```
+
+> **Note:** Without this fix, the SLURM daemon may fail to start due to missing default options.
+
+---
+
+## Restart Services
+
+```bash
+sudo systemctl restart munge
+sudo systemctl restart slurmctld
+sudo systemctl restart slurmd
+```
+
+---
+
+## Basic Test (CPU Job)
+
+```bash
+sbatch --wrap="hostname; sleep 5"
+squeue
+```
+
+> **Notes:**
+> - `sbatch` — submits a job
+> - `--wrap` — runs an inline command
+> - `squeue` — displays the job queue
+
+---
+
+## GPU Configuration
+
+### GRES Config File — `/etc/slurm/gres.conf`
+
+```ini
+NodeName=brilliant Name=gpu File=/dev/nvidia0
+```
+
+> **Note:** This maps the GPU device to SLURM. `/dev/nvidia0` refers to the first GPU.
+
+### GPU Validation
+
+```bash
+scontrol show node
+```
+
+**Expected output:**
+
+```
+Gres=gpu:1
+```
+
+### GPU Job Test
+
+```bash
+sbatch --gres=gpu:1 --wrap="nvidia-smi"
+```
+
+> **Notes:**
+> - `--gres=gpu:1` — allocates 1 GPU to the job
+> - `nvidia-smi` runs inside the job to confirm GPU access