# IPTSP Troubleshooting Guide --- ## Problem Overview ### CPU Steal * Occurs when the **guest VM wants CPU time but the hypervisor does not schedule it** * Indicates contention **outside the VM** * **Root Cause Domain:** Host-level CPU scheduling --- ### CPU I/O Wait * Occurs when the **guest is blocked waiting for disk I/O** * Typically caused by: * Storage latency * Backend storage bottlenecks * **Root Cause Domain:** Storage subsystem --- ## Likely Root Causes * CPU scheduling inefficiency * Storage / I/O latency issues --- # Deep Analysis --- ## 1. CPU Steal Analysis ### Key Concept: Emulator Threads Even with CPU pinning: * Each **vCPU = 1 thread** * Controlled via: ```ini hw:cpu_policy=dedicated ``` These threads handle guest workload. --- ### Hidden Workload: Emulator Threads QEMU introduces additional threads responsible for: * Device emulation (disk, NIC) * I/O handling * VM exits (hypercalls, interrupts) * Internal QEMU operations These **exist even when VM is idle** --- ### Critical Problem * Emulator threads are **NOT automatically pinned** * They are scheduled by **default Linux scheduler** * Can run on **any CPU** --- ### Identify Threads ```bash ps -eLo pid,tid,psr,comm | grep qemu ``` Example: ``` 1234 1235 2 qemu-system-x86 ← vCPU thread 1234 1236 3 qemu-system-x86 ← vCPU thread 1234 1237 2 qemu-system-x86 ← emulator thread (!!) ``` --- ### Proper CPU Segregation ```ini [compute] cpu_dedicated_set = 2-9 cpu_shared_set = 0-1 ``` * `cpu_dedicated_set` → VM vCPUs * `cpu_shared_set` → Emulator threads, host processes, kernel --- ### Important Limitation > CPU pinning **only isolates vCPU execution**, NOT the full VM execution path Still shared: * Emulator threads * Interrupts * Kernel scheduling --- ## 2. Interrupt & Host Noise Issues Even with no other VMs, host can interfere: ### Sources of Noise * `ksoftirqd` * `irqbalance` * `ovs-vswitchd` * `ceph-osd` --- ### Mitigation #### Kernel Isolation ```bash isolcpus= nohz_full= rcu_nocbs= ``` --- #### Interrupt Check ```bash cat /proc/interrupts ``` If interrupts hit pinned CPUs → **BAD** --- #### Fix Strategy * Manually tune IRQ affinity * Or disable `irqbalance` --- ## 3. Hyper-Threading (SMT) Consideration * Avoid pinning **single threads** * Always pin **full core pairs (siblings)** OR * Disable HT for critical workloads --- ## 4. NUMA Alignment Ensure VM stays within a single NUMA node: ```ini hw:numa_nodes=1 hw:cpu_policy=dedicated hw:mem_page_size=large ``` --- # I/O Wait Analysis --- ## VM-Level Checks ```bash iostat -x 1 mpstat -P ALL 1 top / htop ``` --- ## Host-Level Checks ```bash virsh domblkstat virsh dumpxml | grep numa ``` --- ## Expected Healthy Output (mpstat) * `steal ≈ 0` * `iowait ≈ 0` * High idle % Example: ``` %steal → 0.00 %iowait → 0.00 %idle → ~99% ``` --- # Advanced Debugging (eBPF / perf) --- ## CPU Scheduling Analysis ```bash runqlat runqslower perf sched record perf sched latency ``` --- ## Storage Latency Analysis ```bash biolatency biosnoop ``` --- # Troubleshooting Workflow --- ## Step 1 — Validate CPU Pinning * Check vCPU pinning * Verify emulator thread placement * Ensure NUMA alignment * Validate HT sibling usage --- ## Step 2 — Analyze Host CPU Noise * `mpstat` per core * Interrupt distribution * Background system processes --- ## Step 3 — Validate Storage Performance * `iostat` (VM + host) * Backend metrics (e.g., Ceph) --- ## Step 4 — Use eBPF Tools * `runqlat` → CPU bottleneck * `biolatency` → Storage bottleneck --- ## Step 5 — Correlate Metrics Match spikes across: * CPU steal * I/O wait * Host load Time correlation is key to identifying root cause --- # Key Takeaways * CPU pinning ≠ full isolation * Emulator threads are often the hidden bottleneck * Interrupts can silently break isolation * NUMA + HT alignment is critical * Always correlate CPU + storage metrics ---