298 lines
3.9 KiB
Markdown
298 lines
3.9 KiB
Markdown
# IPTSP Troubleshooting Guide
|
|
|
|
---
|
|
|
|
## Problem Overview
|
|
|
|
### CPU Steal
|
|
|
|
* Occurs when the **guest VM wants CPU time but the hypervisor does not schedule it**
|
|
* Indicates contention **outside the VM**
|
|
* **Root Cause Domain:** Host-level CPU scheduling
|
|
|
|
---
|
|
|
|
### CPU I/O Wait
|
|
|
|
* Occurs when the **guest is blocked waiting for disk I/O**
|
|
* Typically caused by:
|
|
|
|
* Storage latency
|
|
* Backend storage bottlenecks
|
|
* **Root Cause Domain:** Storage subsystem
|
|
|
|
---
|
|
|
|
## Likely Root Causes
|
|
|
|
* CPU scheduling inefficiency
|
|
* Storage / I/O latency issues
|
|
|
|
---
|
|
|
|
# Deep Analysis
|
|
|
|
---
|
|
|
|
## 1. CPU Steal Analysis
|
|
|
|
### Key Concept: Emulator Threads
|
|
|
|
Even with CPU pinning:
|
|
|
|
* Each **vCPU = 1 thread**
|
|
* Controlled via:
|
|
|
|
```ini
|
|
hw:cpu_policy=dedicated
|
|
```
|
|
|
|
These threads handle guest workload.
|
|
|
|
---
|
|
|
|
### Hidden Workload: Emulator Threads
|
|
|
|
QEMU introduces additional threads responsible for:
|
|
|
|
* Device emulation (disk, NIC)
|
|
* I/O handling
|
|
* VM exits (hypercalls, interrupts)
|
|
* Internal QEMU operations
|
|
|
|
These **exist even when VM is idle**
|
|
|
|
---
|
|
|
|
### Critical Problem
|
|
|
|
* Emulator threads are **NOT automatically pinned**
|
|
* They are scheduled by **default Linux scheduler**
|
|
* Can run on **any CPU**
|
|
|
|
---
|
|
|
|
### Identify Threads
|
|
|
|
```bash
|
|
ps -eLo pid,tid,psr,comm | grep qemu
|
|
```
|
|
|
|
Example:
|
|
|
|
```
|
|
1234 1235 2 qemu-system-x86 ← vCPU thread
|
|
1234 1236 3 qemu-system-x86 ← vCPU thread
|
|
1234 1237 2 qemu-system-x86 ← emulator thread (!!)
|
|
```
|
|
|
|
---
|
|
|
|
### Proper CPU Segregation
|
|
|
|
```ini
|
|
[compute]
|
|
cpu_dedicated_set = 2-9
|
|
cpu_shared_set = 0-1
|
|
```
|
|
|
|
* `cpu_dedicated_set` → VM vCPUs
|
|
* `cpu_shared_set` → Emulator threads, host processes, kernel
|
|
|
|
---
|
|
|
|
### Important Limitation
|
|
|
|
> CPU pinning **only isolates vCPU execution**, NOT the full VM execution path
|
|
|
|
Still shared:
|
|
|
|
* Emulator threads
|
|
* Interrupts
|
|
* Kernel scheduling
|
|
|
|
---
|
|
|
|
## 2. Interrupt & Host Noise Issues
|
|
|
|
Even with no other VMs, host can interfere:
|
|
|
|
### Sources of Noise
|
|
|
|
* `ksoftirqd`
|
|
* `irqbalance`
|
|
* `ovs-vswitchd`
|
|
* `ceph-osd`
|
|
|
|
---
|
|
|
|
### Mitigation
|
|
|
|
#### Kernel Isolation
|
|
|
|
```bash
|
|
isolcpus=
|
|
nohz_full=
|
|
rcu_nocbs=
|
|
```
|
|
|
|
---
|
|
|
|
#### Interrupt Check
|
|
|
|
```bash
|
|
cat /proc/interrupts
|
|
```
|
|
|
|
If interrupts hit pinned CPUs → **BAD**
|
|
|
|
---
|
|
|
|
#### Fix Strategy
|
|
|
|
* Manually tune IRQ affinity
|
|
* Or disable `irqbalance`
|
|
|
|
---
|
|
|
|
## 3. Hyper-Threading (SMT) Consideration
|
|
|
|
* Avoid pinning **single threads**
|
|
* Always pin **full core pairs (siblings)**
|
|
|
|
OR
|
|
|
|
* Disable HT for critical workloads
|
|
|
|
---
|
|
|
|
## 4. NUMA Alignment
|
|
|
|
Ensure VM stays within a single NUMA node:
|
|
|
|
```ini
|
|
hw:numa_nodes=1
|
|
hw:cpu_policy=dedicated
|
|
hw:mem_page_size=large
|
|
```
|
|
|
|
---
|
|
|
|
# I/O Wait Analysis
|
|
|
|
---
|
|
|
|
## VM-Level Checks
|
|
|
|
```bash
|
|
iostat -x 1
|
|
mpstat -P ALL 1
|
|
top / htop
|
|
```
|
|
|
|
---
|
|
|
|
## Host-Level Checks
|
|
|
|
```bash
|
|
virsh domblkstat <vm>
|
|
virsh dumpxml <vm> | grep numa
|
|
```
|
|
|
|
---
|
|
|
|
## Expected Healthy Output (mpstat)
|
|
|
|
* `steal ≈ 0`
|
|
* `iowait ≈ 0`
|
|
* High idle %
|
|
|
|
Example:
|
|
|
|
```
|
|
%steal → 0.00
|
|
%iowait → 0.00
|
|
%idle → ~99%
|
|
```
|
|
|
|
---
|
|
|
|
# Advanced Debugging (eBPF / perf)
|
|
|
|
---
|
|
|
|
## CPU Scheduling Analysis
|
|
|
|
```bash
|
|
runqlat
|
|
runqslower
|
|
perf sched record
|
|
perf sched latency
|
|
```
|
|
|
|
---
|
|
|
|
## Storage Latency Analysis
|
|
|
|
```bash
|
|
biolatency
|
|
biosnoop
|
|
```
|
|
|
|
---
|
|
|
|
# Troubleshooting Workflow
|
|
|
|
---
|
|
|
|
## Step 1 — Validate CPU Pinning
|
|
|
|
* Check vCPU pinning
|
|
* Verify emulator thread placement
|
|
* Ensure NUMA alignment
|
|
* Validate HT sibling usage
|
|
|
|
---
|
|
|
|
## Step 2 — Analyze Host CPU Noise
|
|
|
|
* `mpstat` per core
|
|
* Interrupt distribution
|
|
* Background system processes
|
|
|
|
---
|
|
|
|
## Step 3 — Validate Storage Performance
|
|
|
|
* `iostat` (VM + host)
|
|
* Backend metrics (e.g., Ceph)
|
|
|
|
---
|
|
|
|
## Step 4 — Use eBPF Tools
|
|
|
|
* `runqlat` → CPU bottleneck
|
|
* `biolatency` → Storage bottleneck
|
|
|
|
---
|
|
|
|
## Step 5 — Correlate Metrics
|
|
|
|
Match spikes across:
|
|
|
|
* CPU steal
|
|
* I/O wait
|
|
* Host load
|
|
|
|
Time correlation is key to identifying root cause
|
|
|
|
---
|
|
|
|
# Key Takeaways
|
|
|
|
* CPU pinning ≠ full isolation
|
|
* Emulator threads are often the hidden bottleneck
|
|
* Interrupts can silently break isolation
|
|
* NUMA + HT alignment is critical
|
|
* Always correlate CPU + storage metrics
|
|
--- |