Upload files to "/"
This commit is contained in:
826
openstack-monasca-monitoring-hld-lld-sop.md
Normal file
826
openstack-monasca-monitoring-hld-lld-sop.md
Normal file
@@ -0,0 +1,826 @@
|
|||||||
|
# OpenStack Monasca Monitoring Integration --- Architecture & Investigation SOP
|
||||||
|
|
||||||
|
**Environment:** OpenStack Rocky + Monasca + Grafana + Prometheus +
|
||||||
|
OpenSearch\
|
||||||
|
**Purpose:** Future-reference documentation for understanding the
|
||||||
|
existing Monasca deployment and integrating its VM metrics into the
|
||||||
|
modern monitoring platform.
|
||||||
|
|
||||||
|
------------------------------------------------------------------------
|
||||||
|
|
||||||
|
## 1. Executive Summary
|
||||||
|
|
||||||
|
The OpenStack environment already uses **Monasca** for VM/infrastructure
|
||||||
|
metric collection. Ceilometer is intentionally not part of this design.
|
||||||
|
|
||||||
|
The investigation established that:
|
||||||
|
|
||||||
|
- Monasca VM metrics are being collected from OpenStack compute nodes.
|
||||||
|
- Monasca metrics are **tenant/project scoped**.
|
||||||
|
- The Monasca `resource_id` dimension maps to the **Nova server
|
||||||
|
UUID**.
|
||||||
|
- Compute nodes run the Monasca Agent components:
|
||||||
|
- `monasca-collector`
|
||||||
|
- `monasca-forwarder`
|
||||||
|
- `monasca-statsd`
|
||||||
|
- VM monitoring is performed through the Monasca Agent `libvirt`
|
||||||
|
check.
|
||||||
|
- The central Monasca platform is deployed in an **older Kubernetes
|
||||||
|
cluster**, namespace `monasca`.
|
||||||
|
- The central pipeline includes **Monasca API, Kafka, Monasca
|
||||||
|
Persister, and InfluxDB**.
|
||||||
|
- The Monasca metrics database is named **`mon`**.
|
||||||
|
- An old Grafana deployment exists in the Monasca namespace.
|
||||||
|
- That Grafana uses the legacy `monasca-grafana-datasource` plugin.
|
||||||
|
- The installed plugin is version **1.2.1**, updated in 2017, and
|
||||||
|
declares compatibility with **Grafana 3.x.x**.
|
||||||
|
- Therefore, copying that plugin into a modern Grafana installation is
|
||||||
|
not considered a suitable production integration path.
|
||||||
|
- The preferred direction is to evaluate **modern Grafana → InfluxDB →
|
||||||
|
Monasca metrics**, while preserving the existing Prometheus and
|
||||||
|
OpenSearch data sources for their respective workloads.
|
||||||
|
|
||||||
|
------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# 2. Bird's-Eye HLD
|
||||||
|
|
||||||
|
## 2.1 Existing Observability Landscape
|
||||||
|
|
||||||
|
``` text
|
||||||
|
OPENSTACK CLOUD
|
||||||
|
│
|
||||||
|
┌────────────────┴────────────────┐
|
||||||
|
│ │
|
||||||
|
Compute Nodes OpenStack Control Plane
|
||||||
|
│
|
||||||
|
│ Monasca Agent
|
||||||
|
│ collector / forwarder / statsd
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Monasca ingestion
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Kafka
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Monasca Persister
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
InfluxDB
|
||||||
|
database: mon
|
||||||
|
│
|
||||||
|
├──────────────► Legacy Monasca Grafana
|
||||||
|
│ + legacy Monasca datasource
|
||||||
|
│
|
||||||
|
└──────────────► Proposed Modern Grafana integration
|
||||||
|
|
||||||
|
|
||||||
|
MODERN OBSERVABILITY PLATFORM
|
||||||
|
|
||||||
|
┌──────────────────────────────────────────────┐
|
||||||
|
│ Grafana │
|
||||||
|
└──────────┬────────────────┬──────────────────┘
|
||||||
|
│ │
|
||||||
|
Prometheus OpenSearch
|
||||||
|
│ │
|
||||||
|
infra/K8s metrics logs
|
||||||
|
│
|
||||||
|
└──── existing monitoring stack
|
||||||
|
|
||||||
|
Proposed additional path:
|
||||||
|
|
||||||
|
Monasca InfluxDB ─────────────► Grafana InfluxDB datasource
|
||||||
|
```
|
||||||
|
|
||||||
|
## 2.2 Target Responsibility Model
|
||||||
|
|
||||||
|
System Primary responsibility
|
||||||
|
------------ -------------------------------------------------
|
||||||
|
Monasca OpenStack VM/service metric collection
|
||||||
|
InfluxDB Persistent Monasca time-series storage
|
||||||
|
Prometheus Existing Kubernetes/infrastructure metrics
|
||||||
|
OpenSearch Centralized logs
|
||||||
|
Grafana Unified visualization
|
||||||
|
Nova Authoritative VM inventory and VM UUID metadata
|
||||||
|
|
||||||
|
The objective is **not** to replace Monasca with Prometheus. Monasca
|
||||||
|
remains the OpenStack metric collection system.
|
||||||
|
|
||||||
|
------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# 3. Low-Level Design (LLD)
|
||||||
|
|
||||||
|
## 3.1 Compute-Node Metric Collection
|
||||||
|
|
||||||
|
Observed services on compute nodes:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
monasca-collector.service
|
||||||
|
monasca-forwarder.service
|
||||||
|
monasca-statsd.service
|
||||||
|
```
|
||||||
|
|
||||||
|
Example collector process:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
/root/monasca-agent/bin/python
|
||||||
|
/root/monasca-agent/bin/monasca-collector foreground
|
||||||
|
```
|
||||||
|
|
||||||
|
The service runs as:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
User=mon-agent
|
||||||
|
Group=mon-agent
|
||||||
|
```
|
||||||
|
|
||||||
|
Agent configuration is located under:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
/etc/monasca/agent/
|
||||||
|
├── agent.yaml
|
||||||
|
└── conf.d/
|
||||||
|
├── cpu.yaml
|
||||||
|
├── crash.yaml
|
||||||
|
├── disk.yaml
|
||||||
|
├── libvirt.yaml
|
||||||
|
├── load.yaml
|
||||||
|
├── memory.yaml
|
||||||
|
├── network.yaml
|
||||||
|
└── process.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
The discovered `libvirt.yaml` confirms VM metric collection is enabled,
|
||||||
|
including CPU, disk, network, and ping checks.
|
||||||
|
|
||||||
|
Important settings observed:
|
||||||
|
|
||||||
|
``` yaml
|
||||||
|
nova_refresh: 14400
|
||||||
|
vm_cpu_check_enable: true
|
||||||
|
vm_disks_check_enable: true
|
||||||
|
vm_network_check_enable: true
|
||||||
|
vm_ping_check_enable: true
|
||||||
|
vm_probation: 300
|
||||||
|
```
|
||||||
|
|
||||||
|
`nova_refresh: 14400` represents a four-hour refresh interval.
|
||||||
|
|
||||||
|
> Security: the inspected configuration contained a plaintext OpenStack
|
||||||
|
> service credential. Treat exposed credentials as compromised and
|
||||||
|
> rotate them through the normal credential-management process.
|
||||||
|
|
||||||
|
------------------------------------------------------------------------
|
||||||
|
|
||||||
|
## 3.2 Monasca Metric Identity
|
||||||
|
|
||||||
|
A typical VM CPU metric has dimensions similar to:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
name: cpu.utilization_perc
|
||||||
|
|
||||||
|
dimensions:
|
||||||
|
hostname: <VM name>
|
||||||
|
component: vm
|
||||||
|
service: compute
|
||||||
|
zone: nova
|
||||||
|
resource_id: <Nova server UUID>
|
||||||
|
```
|
||||||
|
|
||||||
|
The investigation confirmed:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
Monasca resource_id == Nova server UUID
|
||||||
|
```
|
||||||
|
|
||||||
|
This is critical for dashboard design because Nova inventory and Monasca
|
||||||
|
metrics can be joined conceptually using the server UUID.
|
||||||
|
|
||||||
|
------------------------------------------------------------------------
|
||||||
|
|
||||||
|
## 3.3 Tenant / Project Scoping
|
||||||
|
|
||||||
|
A major investigation result was that:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
monasca metric-list
|
||||||
|
```
|
||||||
|
|
||||||
|
returns metrics for the **current Keystone tenant/project**.
|
||||||
|
|
||||||
|
The Monasca CLI exposes a tenant override concept:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
--tenant-id <PROJECT_ID>
|
||||||
|
```
|
||||||
|
|
||||||
|
Therefore, seeing only a few `resource_id` values without specifying the
|
||||||
|
intended tenant does **not** mean that Monasca only monitors those VMs.
|
||||||
|
|
||||||
|
Logical query model:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
Keystone credentials
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Current project scope
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Monasca API
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Metrics belonging to that tenant
|
||||||
|
```
|
||||||
|
|
||||||
|
For operator-level investigation across projects, project context must
|
||||||
|
be considered explicitly.
|
||||||
|
|
||||||
|
------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# 4. Nova ↔ Monasca Correlation
|
||||||
|
|
||||||
|
For a selected project, Nova server inventory was compared with Monasca
|
||||||
|
metrics.
|
||||||
|
|
||||||
|
Nova provides:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
Server UUID
|
||||||
|
VM name
|
||||||
|
Status
|
||||||
|
Project ID
|
||||||
|
Compute host
|
||||||
|
Hypervisor hostname
|
||||||
|
Instance name
|
||||||
|
```
|
||||||
|
|
||||||
|
Example mapping:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
Nova server UUID
|
||||||
|
│
|
||||||
|
├── Monasca dimension: resource_id
|
||||||
|
│
|
||||||
|
└── Nova instance_name → libvirt domain
|
||||||
|
```
|
||||||
|
|
||||||
|
Example conceptual relationship:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
OpenStack Server
|
||||||
|
UUID: 758f...
|
||||||
|
Name: Jump&Proxy-VM-1p
|
||||||
|
Host: compute23
|
||||||
|
Instance: instance-00017c71
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
compute23 libvirt
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
instance-00017c71
|
||||||
|
```
|
||||||
|
|
||||||
|
The selected VM was verified as a running libvirt domain on its
|
||||||
|
Nova-assigned compute node.
|
||||||
|
|
||||||
|
This demonstrated the useful troubleshooting chain:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
Monasca metric
|
||||||
|
↓ resource_id
|
||||||
|
Nova server UUID
|
||||||
|
↓ host / instance_name
|
||||||
|
Compute node
|
||||||
|
↓
|
||||||
|
libvirt domain
|
||||||
|
↓
|
||||||
|
Monasca libvirt collector
|
||||||
|
```
|
||||||
|
|
||||||
|
------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# 5. Central Monasca Platform
|
||||||
|
|
||||||
|
The central Monasca components were eventually located in Kubernetes.
|
||||||
|
|
||||||
|
Namespace:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
monasca
|
||||||
|
```
|
||||||
|
|
||||||
|
Observed components include:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
monasca-api
|
||||||
|
monasca-client
|
||||||
|
monasca-grafana
|
||||||
|
monasca-influxdb-new
|
||||||
|
monasca-memcached
|
||||||
|
monasca-mysql
|
||||||
|
monasca-persister
|
||||||
|
standalone-kafka
|
||||||
|
standalone-kafka-zookeeper
|
||||||
|
```
|
||||||
|
|
||||||
|
Helm releases observed:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
monasca
|
||||||
|
standalone-kafka
|
||||||
|
standalone-monasca-influxdb
|
||||||
|
standalone-monasca-mysql
|
||||||
|
kafka-manager
|
||||||
|
```
|
||||||
|
|
||||||
|
This establishes the central data flow:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
Compute Monasca Agent
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Monasca API
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Kafka
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Monasca Persister
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
InfluxDB
|
||||||
|
```
|
||||||
|
|
||||||
|
The Monasca API DNS name resolves through the Kubernetes-facing
|
||||||
|
endpoint:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
mon-api.brilliant.com.bd
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
k8s.brilliant.com.bd
|
||||||
|
```
|
||||||
|
|
||||||
|
------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# 6. InfluxDB Backend
|
||||||
|
|
||||||
|
The deployment configuration confirmed InfluxDB as the Monasca metrics
|
||||||
|
backend.
|
||||||
|
|
||||||
|
Database:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
mon
|
||||||
|
```
|
||||||
|
|
||||||
|
Conceptual configuration:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
Monasca API ─────► InfluxDB database "mon"
|
||||||
|
▲
|
||||||
|
│
|
||||||
|
Monasca Persister
|
||||||
|
```
|
||||||
|
|
||||||
|
This is important because modern Grafana has a maintained InfluxDB
|
||||||
|
datasource, creating a potential integration route without relying on
|
||||||
|
the obsolete Monasca Grafana plugin.
|
||||||
|
|
||||||
|
------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# 7. Existing Legacy Grafana
|
||||||
|
|
||||||
|
Existing deployment:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
monasca-grafana
|
||||||
|
```
|
||||||
|
|
||||||
|
Image:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
monasca/grafana
|
||||||
|
```
|
||||||
|
|
||||||
|
The deployment uses:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
Keystone authentication
|
||||||
|
Monasca MySQL as Grafana metadata DB
|
||||||
|
ConfigMap-mounted Grafana configuration
|
||||||
|
```
|
||||||
|
|
||||||
|
Inside the container, the following plugins were discovered:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
/var/lib/grafana/plugins/
|
||||||
|
├── monasca-grafana-app
|
||||||
|
├── monasca-grafana-datasource
|
||||||
|
└── natel-discrete-panel
|
||||||
|
```
|
||||||
|
|
||||||
|
## 7.1 Legacy Monasca Datasource Metadata
|
||||||
|
|
||||||
|
Discovered plugin:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
Name: Monasca
|
||||||
|
ID: monasca-datasource
|
||||||
|
Type: datasource
|
||||||
|
Version: 1.2.1
|
||||||
|
Updated: 2017-10-23
|
||||||
|
Grafana dependency: 3.x.x
|
||||||
|
```
|
||||||
|
|
||||||
|
The package identifies itself as:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
Grafana datasource for use with OpenStack Monasca API
|
||||||
|
```
|
||||||
|
|
||||||
|
Therefore, the historical architecture was:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
Legacy Grafana
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Monasca datasource plugin
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Monasca API
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
InfluxDB
|
||||||
|
```
|
||||||
|
|
||||||
|
This explains why searching the modern Grafana plugin catalog did not
|
||||||
|
reveal a usable Monasca datasource.
|
||||||
|
|
||||||
|
------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# 8. Why the Legacy Plugin Should Not Be the Default Modern Design
|
||||||
|
|
||||||
|
The existing plugin is tied to a Grafana 3.x-era plugin model.
|
||||||
|
|
||||||
|
The source tree contains legacy-style components such as:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
datasource.js
|
||||||
|
module.js
|
||||||
|
query_ctrl.js
|
||||||
|
partials/
|
||||||
|
```
|
||||||
|
|
||||||
|
Its metadata explicitly declares:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
grafanaVersion: 3.x.x
|
||||||
|
```
|
||||||
|
|
||||||
|
Therefore:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
Copy old plugin into new Grafana
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Legacy frontend/plugin APIs
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
High compatibility and maintenance risk
|
||||||
|
```
|
||||||
|
|
||||||
|
A production monitoring deployment should avoid making this abandoned
|
||||||
|
legacy component a new critical dependency unless it is deliberately
|
||||||
|
modernized, tested, and maintained internally.
|
||||||
|
|
||||||
|
------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# 9. Proposed Modern Integration
|
||||||
|
|
||||||
|
The current preferred design to evaluate is:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
Modern Grafana
|
||||||
|
/ | \
|
||||||
|
/ | \
|
||||||
|
▼ ▼ ▼
|
||||||
|
Prometheus OpenSearch InfluxDB
|
||||||
|
│ │ │
|
||||||
|
K8s/infra Logs Monasca
|
||||||
|
metrics
|
||||||
|
```
|
||||||
|
|
||||||
|
Specifically:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
Monasca Agent
|
||||||
|
↓
|
||||||
|
Monasca API / Kafka
|
||||||
|
↓
|
||||||
|
Monasca Persister
|
||||||
|
↓
|
||||||
|
InfluxDB ("mon")
|
||||||
|
↓
|
||||||
|
Grafana InfluxDB datasource
|
||||||
|
```
|
||||||
|
|
||||||
|
This keeps each existing monitoring technology in its current role
|
||||||
|
instead of forcing Monasca metrics through Prometheus.
|
||||||
|
|
||||||
|
------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# 10. Important Design Concern: Multi-Tenancy
|
||||||
|
|
||||||
|
Directly querying InfluxDB is **not architecturally identical** to
|
||||||
|
querying the Monasca API.
|
||||||
|
|
||||||
|
Monasca API provides tenant-aware behavior through Keystone/project
|
||||||
|
context.
|
||||||
|
|
||||||
|
Direct InfluxDB access potentially bypasses that API authorization
|
||||||
|
boundary.
|
||||||
|
|
||||||
|
Therefore distinguish two Grafana use cases:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
Operator Grafana
|
||||||
|
│
|
||||||
|
└── Cloud administrators can see all projects
|
||||||
|
→ direct backend access may be acceptable after security review
|
||||||
|
|
||||||
|
Tenant-facing Grafana
|
||||||
|
│
|
||||||
|
└── Customer must only see own project
|
||||||
|
→ direct DB access requires much stronger isolation design
|
||||||
|
```
|
||||||
|
|
||||||
|
Before production implementation, verify exactly how tenant/project
|
||||||
|
identity is represented in the stored InfluxDB series and how Grafana
|
||||||
|
access control will prevent cross-project visibility.
|
||||||
|
|
||||||
|
------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# 11. Current Deployment Status
|
||||||
|
|
||||||
|
## Confirmed
|
||||||
|
|
||||||
|
- [x] Monasca is already collecting OpenStack VM metrics.
|
||||||
|
- [x] Ceilometer is not required for the selected architecture.
|
||||||
|
- [x] Monasca metrics are project/tenant scoped.
|
||||||
|
- [x] `resource_id` maps to Nova VM UUID.
|
||||||
|
- [x] Compute nodes run Monasca Agent.
|
||||||
|
- [x] VM metrics are collected through the libvirt check.
|
||||||
|
- [x] Central Monasca is deployed on Kubernetes.
|
||||||
|
- [x] Kafka is part of the central pipeline.
|
||||||
|
- [x] Monasca Persister is running.
|
||||||
|
- [x] InfluxDB is the metric backend.
|
||||||
|
- [x] Monasca metric database is `mon`.
|
||||||
|
- [x] Legacy Monasca Grafana exists.
|
||||||
|
- [x] Legacy Grafana contains `monasca-grafana-datasource`.
|
||||||
|
- [x] Plugin version is 1.2.1 and targets Grafana 3.x.
|
||||||
|
- [x] Modern Grafana should not depend on that plugin by default.
|
||||||
|
|
||||||
|
## Still To Be Determined
|
||||||
|
|
||||||
|
- [ ] Exact InfluxDB Kubernetes Service name.
|
||||||
|
- [ ] InfluxDB service port and exposure type.
|
||||||
|
- [ ] Whether authentication is enabled and which credentials should
|
||||||
|
be used.
|
||||||
|
- [ ] Whether the modern Grafana cluster can route to the old Monasca
|
||||||
|
Kubernetes cluster.
|
||||||
|
- [ ] Exact InfluxDB schema/measurements used by Monasca.
|
||||||
|
- [ ] How tenant/project identity appears in InfluxDB tags.
|
||||||
|
- [ ] Appropriate Grafana datasource configuration.
|
||||||
|
- [ ] Dashboard variable design for project, VM, resource UUID, host,
|
||||||
|
and metric.
|
||||||
|
- [ ] Security model for operator-only versus tenant-facing
|
||||||
|
dashboards.
|
||||||
|
- [ ] Alerting design.
|
||||||
|
- [ ] Production validation and performance testing.
|
||||||
|
|
||||||
|
------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# 12. Next Deployment Phase
|
||||||
|
|
||||||
|
The next investigation should focus only on the InfluxDB access path:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
Modern Grafana
|
||||||
|
│
|
||||||
|
│ network?
|
||||||
|
▼
|
||||||
|
Old Monasca Kubernetes cluster
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
InfluxDB Service
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
database: mon
|
||||||
|
```
|
||||||
|
|
||||||
|
Information to collect:
|
||||||
|
|
||||||
|
1. InfluxDB Kubernetes Service name.
|
||||||
|
2. Service type: ClusterIP / NodePort / LoadBalancer.
|
||||||
|
3. Service port.
|
||||||
|
4. Endpoints/pods behind the Service.
|
||||||
|
5. InfluxDB version.
|
||||||
|
6. Authentication configuration.
|
||||||
|
7. Whether the new Grafana cluster has network reachability to it.
|
||||||
|
8. Sample Monasca measurement/tag structure.
|
||||||
|
|
||||||
|
Do not expose InfluxDB publicly merely to make Grafana connectivity
|
||||||
|
easier. Prefer a controlled private network path.
|
||||||
|
|
||||||
|
------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# 13. Reference Investigation Commands
|
||||||
|
|
||||||
|
These are reference commands captured from the investigation. Validate
|
||||||
|
them against the environment before operational use.
|
||||||
|
|
||||||
|
### Monasca tenant metrics
|
||||||
|
|
||||||
|
``` bash
|
||||||
|
monasca --monasca-api-url http://mon-api.brilliant.com.bd/v2.0 \
|
||||||
|
--monasca-api-version 2_0 \
|
||||||
|
metric-list \
|
||||||
|
--name cpu.utilization_perc \
|
||||||
|
--tenant-id <PROJECT_ID>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Nova project inventory
|
||||||
|
|
||||||
|
``` bash
|
||||||
|
openstack server list --project <PROJECT_ID> --fit
|
||||||
|
```
|
||||||
|
|
||||||
|
### Nova VM placement
|
||||||
|
|
||||||
|
``` bash
|
||||||
|
openstack server show <SERVER_UUID> \
|
||||||
|
-f table \
|
||||||
|
-c OS-EXT-SRV-ATTR:host \
|
||||||
|
-c OS-EXT-SRV-ATTR:hypervisor_hostname \
|
||||||
|
-c OS-EXT-SRV-ATTR:instance_name \
|
||||||
|
-c status \
|
||||||
|
-c project_id
|
||||||
|
```
|
||||||
|
|
||||||
|
### Verify local libvirt domain
|
||||||
|
|
||||||
|
``` bash
|
||||||
|
virsh list --all
|
||||||
|
```
|
||||||
|
|
||||||
|
### Compute-node Monasca services
|
||||||
|
|
||||||
|
``` bash
|
||||||
|
systemctl status monasca-collector
|
||||||
|
systemctl status monasca-forwarder
|
||||||
|
systemctl status monasca-statsd
|
||||||
|
```
|
||||||
|
|
||||||
|
### Monasca Agent VM configuration
|
||||||
|
|
||||||
|
``` text
|
||||||
|
/etc/monasca/agent/conf.d/libvirt.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
### Central Monasca Kubernetes resources
|
||||||
|
|
||||||
|
``` bash
|
||||||
|
kubectl get pods -n monasca
|
||||||
|
kubectl get svc -n monasca
|
||||||
|
kubectl get deploy -n monasca
|
||||||
|
```
|
||||||
|
|
||||||
|
### Existing Grafana deployment
|
||||||
|
|
||||||
|
``` bash
|
||||||
|
kubectl describe deploy monasca-grafana -n monasca
|
||||||
|
```
|
||||||
|
|
||||||
|
### Legacy plugin location
|
||||||
|
|
||||||
|
``` text
|
||||||
|
/var/lib/grafana/plugins/monasca-grafana-datasource
|
||||||
|
```
|
||||||
|
|
||||||
|
------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# 14. Troubleshooting Notes Kept Separate From Deployment
|
||||||
|
|
||||||
|
During discovery, two current VMs appeared absent from one
|
||||||
|
`cpu.utilization_perc` comparison. One was traced to `compute23`, and
|
||||||
|
its corresponding libvirt domain was confirmed running.
|
||||||
|
|
||||||
|
That investigation was intentionally deferred because the primary
|
||||||
|
objective is **deploying the monitoring integration**, not achieving
|
||||||
|
immediate 100% metric-coverage troubleshooting.
|
||||||
|
|
||||||
|
If metric coverage becomes a production acceptance criterion later, use
|
||||||
|
this chain:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
Nova ACTIVE VM inventory
|
||||||
|
↓
|
||||||
|
Monasca recent resource_id inventory
|
||||||
|
↓
|
||||||
|
Identify missing UUID
|
||||||
|
↓
|
||||||
|
Nova host + instance_name
|
||||||
|
↓
|
||||||
|
libvirt domain
|
||||||
|
↓
|
||||||
|
Monasca libvirt collector
|
||||||
|
↓
|
||||||
|
forwarder
|
||||||
|
↓
|
||||||
|
Monasca API / Kafka
|
||||||
|
↓
|
||||||
|
persister
|
||||||
|
↓
|
||||||
|
InfluxDB
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not mix this troubleshooting track with the main Grafana integration
|
||||||
|
unless missing metrics materially block deployment.
|
||||||
|
|
||||||
|
------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# 15. Operational Risks Observed
|
||||||
|
|
||||||
|
### Credentials
|
||||||
|
|
||||||
|
Plaintext service credentials were visible during configuration
|
||||||
|
inspection. Secrets should be rotated where exposure occurred and
|
||||||
|
ultimately moved to an appropriate secret-management mechanism.
|
||||||
|
|
||||||
|
### Legacy Software
|
||||||
|
|
||||||
|
The Monasca Kubernetes installation and its bundled Grafana/plugins are
|
||||||
|
old. Avoid making obsolete frontend plugins dependencies of the new
|
||||||
|
monitoring platform.
|
||||||
|
|
||||||
|
### Persister Restarts
|
||||||
|
|
||||||
|
Several `monasca-persister` pods showed restart counts in the thousands.
|
||||||
|
This does not block architecture discovery, but it deserves a separate
|
||||||
|
reliability investigation before treating the Monasca backend as
|
||||||
|
production-grade for a new visualization dependency.
|
||||||
|
|
||||||
|
### Cross-Cluster Connectivity
|
||||||
|
|
||||||
|
The modern Grafana and old Monasca stack may be in different Kubernetes
|
||||||
|
clusters. A Kubernetes `ClusterIP` is normally cluster-internal;
|
||||||
|
connectivity must therefore be deliberately designed rather than
|
||||||
|
assuming Kubernetes service DNS will work cross-cluster.
|
||||||
|
|
||||||
|
------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# 16. Final Architecture Baseline
|
||||||
|
|
||||||
|
Use this as the working architecture until the InfluxDB
|
||||||
|
connectivity/schema validation phase proves otherwise:
|
||||||
|
|
||||||
|
``` text
|
||||||
|
┌─────────────────────┐
|
||||||
|
│ OpenStack │
|
||||||
|
└──────────┬──────────┘
|
||||||
|
│
|
||||||
|
Nova / libvirt
|
||||||
|
│
|
||||||
|
┌──────────▼──────────┐
|
||||||
|
│ Monasca Agents │
|
||||||
|
│ on compute nodes │
|
||||||
|
└──────────┬──────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Monasca API
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Kafka
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Monasca Persister
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────────────┐
|
||||||
|
│ InfluxDB │
|
||||||
|
│ database: mon │
|
||||||
|
└──────────┬──────────┘
|
||||||
|
│
|
||||||
|
proposed datasource
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌──────────────┐ ┌─────────────────────┐ ┌──────────────┐
|
||||||
|
│ Prometheus │─────────►│ Modern Grafana │◄─────────│ OpenSearch │
|
||||||
|
│ K8s/infra │ │ unified dashboards │ │ logs │
|
||||||
|
└──────────────┘ └─────────────────────┘ └──────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
**Next milestone:** prove secure network connectivity from modern
|
||||||
|
Grafana to the Monasca InfluxDB service and inspect the `mon` database
|
||||||
|
schema before creating the Grafana datasource.
|
||||||
Reference in New Issue
Block a user