Add iptsp-cpuio.md

This is the initail analysis for troubleshoots

iptsp-cpuio.md

@@ -0,0 +1,298 @@
+# IPTSP Troubleshooting Guide
+
+---
+
+##  Problem Overview
+
+### CPU Steal
+
+* Occurs when the **guest VM wants CPU time but the hypervisor does not schedule it**
+* Indicates contention **outside the VM**
+* **Root Cause Domain:** Host-level CPU scheduling
+
+---
+
+### CPU I/O Wait
+
+* Occurs when the **guest is blocked waiting for disk I/O**
+* Typically caused by:
+
+  * Storage latency
+  * Backend storage bottlenecks
+* **Root Cause Domain:** Storage subsystem
+
+---
+
+##  Likely Root Causes
+
+* CPU scheduling inefficiency
+* Storage / I/O latency issues
+
+---
+
+#  Deep Analysis
+
+---
+
+## 1. CPU Steal Analysis
+
+###  Key Concept: Emulator Threads
+
+Even with CPU pinning:
+
+* Each **vCPU = 1 thread**
+* Controlled via:
+
+  ```ini
+  hw:cpu_policy=dedicated
+  ```
+
+These threads handle guest workload.
+
+---
+
+###  Hidden Workload: Emulator Threads
+
+QEMU introduces additional threads responsible for:
+
+* Device emulation (disk, NIC)
+* I/O handling
+* VM exits (hypercalls, interrupts)
+* Internal QEMU operations
+
+ These **exist even when VM is idle**
+
+---
+
+###  Critical Problem
+
+* Emulator threads are **NOT automatically pinned**
+* They are scheduled by **default Linux scheduler**
+* Can run on **any CPU**
+
+---
+
+###  Identify Threads
+
+```bash
+ps -eLo pid,tid,psr,comm | grep qemu
+```
+
+Example:
+
+```
+1234 1235 2 qemu-system-x86  ← vCPU thread
+1234 1236 3 qemu-system-x86  ← vCPU thread
+1234 1237 2 qemu-system-x86  ← emulator thread (!!)
+```
+
+---
+
+###  Proper CPU Segregation
+
+```ini
+[compute]
+cpu_dedicated_set = 2-9
+cpu_shared_set = 0-1
+```
+
+* `cpu_dedicated_set` → VM vCPUs
+* `cpu_shared_set` → Emulator threads, host processes, kernel
+
+---
+
+###  Important Limitation
+
+> CPU pinning **only isolates vCPU execution**, NOT the full VM execution path
+
+Still shared:
+
+* Emulator threads
+* Interrupts
+* Kernel scheduling
+
+---
+
+## 2. Interrupt & Host Noise Issues
+
+Even with no other VMs, host can interfere:
+
+### Sources of Noise
+
+* `ksoftirqd`
+* `irqbalance`
+* `ovs-vswitchd`
+* `ceph-osd`
+
+---
+
+###  Mitigation
+
+#### Kernel Isolation
+
+```bash
+isolcpus=
+nohz_full=
+rcu_nocbs=
+```
+
+---
+
+#### Interrupt Check
+
+```bash
+cat /proc/interrupts
+```
+
+ If interrupts hit pinned CPUs → **BAD**
+
+---
+
+#### Fix Strategy
+
+* Manually tune IRQ affinity
+* Or disable `irqbalance`
+
+---
+
+## 3. Hyper-Threading (SMT) Consideration
+
+* Avoid pinning **single threads**
+* Always pin **full core pairs (siblings)**
+
+OR
+
+* Disable HT for critical workloads
+
+---
+
+## 4. NUMA Alignment
+
+Ensure VM stays within a single NUMA node:
+
+```ini
+hw:numa_nodes=1
+hw:cpu_policy=dedicated
+hw:mem_page_size=large
+```
+
+---
+
+#  I/O Wait Analysis
+
+---
+
+##  VM-Level Checks
+
+```bash
+iostat -x 1
+mpstat -P ALL 1
+top / htop
+```
+
+---
+
+##  Host-Level Checks
+
+```bash
+virsh domblkstat <vm>
+virsh dumpxml <vm> | grep numa
+```
+
+---
+
+##  Expected Healthy Output (mpstat)
+
+* `steal ≈ 0`
+* `iowait ≈ 0`
+* High idle %
+
+Example:
+
+```
+%steal   → 0.00
+%iowait  → 0.00
+%idle    → ~99%
+```
+
+---
+
+#  Advanced Debugging (eBPF / perf)
+
+---
+
+## CPU Scheduling Analysis
+
+```bash
+runqlat
+runqslower
+perf sched record
+perf sched latency
+```
+
+---
+
+## Storage Latency Analysis
+
+```bash
+biolatency
+biosnoop
+```
+
+---
+
+#  Troubleshooting Workflow
+
+---
+
+## Step 1 — Validate CPU Pinning
+
+* Check vCPU pinning
+* Verify emulator thread placement
+* Ensure NUMA alignment
+* Validate HT sibling usage
+
+---
+
+## Step 2 — Analyze Host CPU Noise
+
+* `mpstat` per core
+* Interrupt distribution
+* Background system processes
+
+---
+
+## Step 3 — Validate Storage Performance
+
+* `iostat` (VM + host)
+* Backend metrics (e.g., Ceph)
+
+---
+
+## Step 4 — Use eBPF Tools
+
+* `runqlat` → CPU bottleneck
+* `biolatency` → Storage bottleneck
+
+---
+
+## Step 5 — Correlate Metrics
+
+Match spikes across:
+
+* CPU steal
+* I/O wait
+* Host load
+
+ Time correlation is key to identifying root cause
+
+---
+
+#  Key Takeaways
+
+* CPU pinning ≠ full isolation
+* Emulator threads are often the hidden bottleneck
+* Interrupts can silently break isolation
+* NUMA + HT alignment is critical
+* Always correlate CPU + storage metrics
+---