IPTSP-Troubleshoots/iptsp-cpuio.md

IPTSP Troubleshooting Guide

---

Problem Overview

CPU Steal

• Occurs when the guest VM wants CPU time but the hypervisor does not schedule it
• Indicates contention outside the VM
• Root Cause Domain: Host-level CPU scheduling

---

CPU I/O Wait

• Occurs when the guest is blocked waiting for disk I/O

• Typically caused by:

  • Storage latency
  • Backend storage bottlenecks

• Root Cause Domain: Storage subsystem

---

Likely Root Causes

• CPU scheduling inefficiency
• Storage / I/O latency issues

---

Deep Analysis

---

1. CPU Steal Analysis

Key Concept: Emulator Threads

Even with CPU pinning:

• Each vCPU = 1 thread

• Controlled via:

  hw:cpu_policy=dedicated
  

These threads handle guest workload.

---

Hidden Workload: Emulator Threads

QEMU introduces additional threads responsible for:

• Device emulation (disk, NIC)
• I/O handling
• VM exits (hypercalls, interrupts)
• Internal QEMU operations

These exist even when VM is idle

---

Critical Problem

• Emulator threads are NOT automatically pinned
• They are scheduled by default Linux scheduler
• Can run on any CPU

---

Identify Threads

ps -eLo pid,tid,psr,comm | grep qemu

Example:

1234 1235 2 qemu-system-x86  ← vCPU thread
1234 1236 3 qemu-system-x86  ← vCPU thread
1234 1237 2 qemu-system-x86  ← emulator thread (!!)

---

Proper CPU Segregation

[compute]
cpu_dedicated_set = 2-9
cpu_shared_set = 0-1

• cpu_dedicated_set → VM vCPUs
• cpu_shared_set → Emulator threads, host processes, kernel

---

Important Limitation

CPU pinning only isolates vCPU execution, NOT the full VM execution path

Still shared:

• Emulator threads
• Interrupts
• Kernel scheduling

---

2. Interrupt & Host Noise Issues

Even with no other VMs, host can interfere:

Sources of Noise

• ksoftirqd
• irqbalance
• ovs-vswitchd
• ceph-osd

---

Mitigation

Kernel Isolation

isolcpus=
nohz_full=
rcu_nocbs=

---

Interrupt Check

cat /proc/interrupts

If interrupts hit pinned CPUs → BAD

---

Fix Strategy

• Manually tune IRQ affinity
• Or disable irqbalance

---

3. Hyper-Threading (SMT) Consideration

• Avoid pinning single threads
• Always pin full core pairs (siblings)

OR

• Disable HT for critical workloads

---

4. NUMA Alignment

Ensure VM stays within a single NUMA node:

hw:numa_nodes=1
hw:cpu_policy=dedicated
hw:mem_page_size=large

---

I/O Wait Analysis

---

VM-Level Checks

iostat -x 1
mpstat -P ALL 1
top / htop

---

Host-Level Checks

virsh domblkstat <vm>
virsh dumpxml <vm> | grep numa

---

Expected Healthy Output (mpstat)

• steal ≈ 0
• iowait ≈ 0
• High idle %

Example:

%steal   → 0.00
%iowait  → 0.00
%idle    → ~99%

---

Advanced Debugging (eBPF / perf)

---

CPU Scheduling Analysis

runqlat
runqslower
perf sched record
perf sched latency

---

Storage Latency Analysis

biolatency
biosnoop

---

Troubleshooting Workflow

---

Step 1 — Validate CPU Pinning

• Check vCPU pinning
• Verify emulator thread placement
• Ensure NUMA alignment
• Validate HT sibling usage

---

Step 2 — Analyze Host CPU Noise

• mpstat per core
• Interrupt distribution
• Background system processes

---

Step 3 — Validate Storage Performance

• iostat (VM + host)
• Backend metrics (e.g., Ceph)

---

Step 4 — Use eBPF Tools

• runqlat → CPU bottleneck
• biolatency → Storage bottleneck

---

Step 5 — Correlate Metrics

Match spikes across:

• CPU steal
• I/O wait
• Host load

Time correlation is key to identifying root cause

---

Key Takeaways

• CPU pinning ≠ full isolation
• Emulator threads are often the hidden bottleneck
• Interrupts can silently break isolation
• NUMA + HT alignment is critical
• Always correlate CPU + storage metrics

---