From adc3c1038ca595dd17f4de4cebbcd9ba03d9dba4 Mon Sep 17 00:00:00 2001 From: mosiur Date: Thu, 20 Aug 2026 10:38:36 +0000 Subject: [PATCH] Upload files to "/" --- openstack-monasca-monitoring-hld-lld-sop.md | 826 ++++++++++++++++++++ 1 file changed, 826 insertions(+) create mode 100644 openstack-monasca-monitoring-hld-lld-sop.md diff --git a/openstack-monasca-monitoring-hld-lld-sop.md b/openstack-monasca-monitoring-hld-lld-sop.md new file mode 100644 index 0000000..8448cc9 --- /dev/null +++ b/openstack-monasca-monitoring-hld-lld-sop.md @@ -0,0 +1,826 @@ +# OpenStack Monasca Monitoring Integration --- Architecture & Investigation SOP + +**Environment:** OpenStack Rocky + Monasca + Grafana + Prometheus + +OpenSearch\ +**Purpose:** Future-reference documentation for understanding the +existing Monasca deployment and integrating its VM metrics into the +modern monitoring platform. + +------------------------------------------------------------------------ + +## 1. Executive Summary + +The OpenStack environment already uses **Monasca** for VM/infrastructure +metric collection. Ceilometer is intentionally not part of this design. + +The investigation established that: + +- Monasca VM metrics are being collected from OpenStack compute nodes. +- Monasca metrics are **tenant/project scoped**. +- The Monasca `resource_id` dimension maps to the **Nova server + UUID**. +- Compute nodes run the Monasca Agent components: + - `monasca-collector` + - `monasca-forwarder` + - `monasca-statsd` +- VM monitoring is performed through the Monasca Agent `libvirt` + check. +- The central Monasca platform is deployed in an **older Kubernetes + cluster**, namespace `monasca`. +- The central pipeline includes **Monasca API, Kafka, Monasca + Persister, and InfluxDB**. +- The Monasca metrics database is named **`mon`**. +- An old Grafana deployment exists in the Monasca namespace. +- That Grafana uses the legacy `monasca-grafana-datasource` plugin. +- The installed plugin is version **1.2.1**, updated in 2017, and + declares compatibility with **Grafana 3.x.x**. +- Therefore, copying that plugin into a modern Grafana installation is + not considered a suitable production integration path. +- The preferred direction is to evaluate **modern Grafana → InfluxDB → + Monasca metrics**, while preserving the existing Prometheus and + OpenSearch data sources for their respective workloads. + +------------------------------------------------------------------------ + +# 2. Bird's-Eye HLD + +## 2.1 Existing Observability Landscape + +``` text + OPENSTACK CLOUD + │ + ┌────────────────┴────────────────┐ + │ │ + Compute Nodes OpenStack Control Plane + │ + │ Monasca Agent + │ collector / forwarder / statsd + │ + ▼ + Monasca ingestion + │ + ▼ + Kafka + │ + ▼ + Monasca Persister + │ + ▼ + InfluxDB + database: mon + │ + ├──────────────► Legacy Monasca Grafana + │ + legacy Monasca datasource + │ + └──────────────► Proposed Modern Grafana integration + + + MODERN OBSERVABILITY PLATFORM + + ┌──────────────────────────────────────────────┐ + │ Grafana │ + └──────────┬────────────────┬──────────────────┘ + │ │ + Prometheus OpenSearch + │ │ + infra/K8s metrics logs + │ + └──── existing monitoring stack + +Proposed additional path: + + Monasca InfluxDB ─────────────► Grafana InfluxDB datasource +``` + +## 2.2 Target Responsibility Model + + System Primary responsibility + ------------ ------------------------------------------------- + Monasca OpenStack VM/service metric collection + InfluxDB Persistent Monasca time-series storage + Prometheus Existing Kubernetes/infrastructure metrics + OpenSearch Centralized logs + Grafana Unified visualization + Nova Authoritative VM inventory and VM UUID metadata + +The objective is **not** to replace Monasca with Prometheus. Monasca +remains the OpenStack metric collection system. + +------------------------------------------------------------------------ + +# 3. Low-Level Design (LLD) + +## 3.1 Compute-Node Metric Collection + +Observed services on compute nodes: + +``` text +monasca-collector.service +monasca-forwarder.service +monasca-statsd.service +``` + +Example collector process: + +``` text +/root/monasca-agent/bin/python +/root/monasca-agent/bin/monasca-collector foreground +``` + +The service runs as: + +``` text +User=mon-agent +Group=mon-agent +``` + +Agent configuration is located under: + +``` text +/etc/monasca/agent/ +├── agent.yaml +└── conf.d/ + ├── cpu.yaml + ├── crash.yaml + ├── disk.yaml + ├── libvirt.yaml + ├── load.yaml + ├── memory.yaml + ├── network.yaml + └── process.yaml +``` + +The discovered `libvirt.yaml` confirms VM metric collection is enabled, +including CPU, disk, network, and ping checks. + +Important settings observed: + +``` yaml +nova_refresh: 14400 +vm_cpu_check_enable: true +vm_disks_check_enable: true +vm_network_check_enable: true +vm_ping_check_enable: true +vm_probation: 300 +``` + +`nova_refresh: 14400` represents a four-hour refresh interval. + +> Security: the inspected configuration contained a plaintext OpenStack +> service credential. Treat exposed credentials as compromised and +> rotate them through the normal credential-management process. + +------------------------------------------------------------------------ + +## 3.2 Monasca Metric Identity + +A typical VM CPU metric has dimensions similar to: + +``` text +name: cpu.utilization_perc + +dimensions: + hostname: + component: vm + service: compute + zone: nova + resource_id: +``` + +The investigation confirmed: + +``` text +Monasca resource_id == Nova server UUID +``` + +This is critical for dashboard design because Nova inventory and Monasca +metrics can be joined conceptually using the server UUID. + +------------------------------------------------------------------------ + +## 3.3 Tenant / Project Scoping + +A major investigation result was that: + +``` text +monasca metric-list +``` + +returns metrics for the **current Keystone tenant/project**. + +The Monasca CLI exposes a tenant override concept: + +``` text +--tenant-id +``` + +Therefore, seeing only a few `resource_id` values without specifying the +intended tenant does **not** mean that Monasca only monitors those VMs. + +Logical query model: + +``` text +Keystone credentials + │ + ▼ +Current project scope + │ + ▼ +Monasca API + │ + ▼ +Metrics belonging to that tenant +``` + +For operator-level investigation across projects, project context must +be considered explicitly. + +------------------------------------------------------------------------ + +# 4. Nova ↔ Monasca Correlation + +For a selected project, Nova server inventory was compared with Monasca +metrics. + +Nova provides: + +``` text +Server UUID +VM name +Status +Project ID +Compute host +Hypervisor hostname +Instance name +``` + +Example mapping: + +``` text +Nova server UUID + │ + ├── Monasca dimension: resource_id + │ + └── Nova instance_name → libvirt domain +``` + +Example conceptual relationship: + +``` text +OpenStack Server +UUID: 758f... +Name: Jump&Proxy-VM-1p +Host: compute23 +Instance: instance-00017c71 + │ + ▼ +compute23 libvirt + │ + ▼ +instance-00017c71 +``` + +The selected VM was verified as a running libvirt domain on its +Nova-assigned compute node. + +This demonstrated the useful troubleshooting chain: + +``` text +Monasca metric + ↓ resource_id +Nova server UUID + ↓ host / instance_name +Compute node + ↓ +libvirt domain + ↓ +Monasca libvirt collector +``` + +------------------------------------------------------------------------ + +# 5. Central Monasca Platform + +The central Monasca components were eventually located in Kubernetes. + +Namespace: + +``` text +monasca +``` + +Observed components include: + +``` text +monasca-api +monasca-client +monasca-grafana +monasca-influxdb-new +monasca-memcached +monasca-mysql +monasca-persister +standalone-kafka +standalone-kafka-zookeeper +``` + +Helm releases observed: + +``` text +monasca +standalone-kafka +standalone-monasca-influxdb +standalone-monasca-mysql +kafka-manager +``` + +This establishes the central data flow: + +``` text +Compute Monasca Agent + │ + ▼ + Monasca API + │ + ▼ + Kafka + │ + ▼ + Monasca Persister + │ + ▼ + InfluxDB +``` + +The Monasca API DNS name resolves through the Kubernetes-facing +endpoint: + +``` text +mon-api.brilliant.com.bd + │ + ▼ +k8s.brilliant.com.bd +``` + +------------------------------------------------------------------------ + +# 6. InfluxDB Backend + +The deployment configuration confirmed InfluxDB as the Monasca metrics +backend. + +Database: + +``` text +mon +``` + +Conceptual configuration: + +``` text +Monasca API ─────► InfluxDB database "mon" + ▲ + │ + Monasca Persister +``` + +This is important because modern Grafana has a maintained InfluxDB +datasource, creating a potential integration route without relying on +the obsolete Monasca Grafana plugin. + +------------------------------------------------------------------------ + +# 7. Existing Legacy Grafana + +Existing deployment: + +``` text +monasca-grafana +``` + +Image: + +``` text +monasca/grafana +``` + +The deployment uses: + +``` text +Keystone authentication +Monasca MySQL as Grafana metadata DB +ConfigMap-mounted Grafana configuration +``` + +Inside the container, the following plugins were discovered: + +``` text +/var/lib/grafana/plugins/ +├── monasca-grafana-app +├── monasca-grafana-datasource +└── natel-discrete-panel +``` + +## 7.1 Legacy Monasca Datasource Metadata + +Discovered plugin: + +``` text +Name: Monasca +ID: monasca-datasource +Type: datasource +Version: 1.2.1 +Updated: 2017-10-23 +Grafana dependency: 3.x.x +``` + +The package identifies itself as: + +``` text +Grafana datasource for use with OpenStack Monasca API +``` + +Therefore, the historical architecture was: + +``` text +Legacy Grafana + │ + ▼ +Monasca datasource plugin + │ + ▼ +Monasca API + │ + ▼ +InfluxDB +``` + +This explains why searching the modern Grafana plugin catalog did not +reveal a usable Monasca datasource. + +------------------------------------------------------------------------ + +# 8. Why the Legacy Plugin Should Not Be the Default Modern Design + +The existing plugin is tied to a Grafana 3.x-era plugin model. + +The source tree contains legacy-style components such as: + +``` text +datasource.js +module.js +query_ctrl.js +partials/ +``` + +Its metadata explicitly declares: + +``` text +grafanaVersion: 3.x.x +``` + +Therefore: + +``` text +Copy old plugin into new Grafana + │ + ▼ +Legacy frontend/plugin APIs + │ + ▼ +High compatibility and maintenance risk +``` + +A production monitoring deployment should avoid making this abandoned +legacy component a new critical dependency unless it is deliberately +modernized, tested, and maintained internally. + +------------------------------------------------------------------------ + +# 9. Proposed Modern Integration + +The current preferred design to evaluate is: + +``` text + Modern Grafana + / | \ + / | \ + ▼ ▼ ▼ + Prometheus OpenSearch InfluxDB + │ │ │ + K8s/infra Logs Monasca + metrics +``` + +Specifically: + +``` text +Monasca Agent + ↓ +Monasca API / Kafka + ↓ +Monasca Persister + ↓ +InfluxDB ("mon") + ↓ +Grafana InfluxDB datasource +``` + +This keeps each existing monitoring technology in its current role +instead of forcing Monasca metrics through Prometheus. + +------------------------------------------------------------------------ + +# 10. Important Design Concern: Multi-Tenancy + +Directly querying InfluxDB is **not architecturally identical** to +querying the Monasca API. + +Monasca API provides tenant-aware behavior through Keystone/project +context. + +Direct InfluxDB access potentially bypasses that API authorization +boundary. + +Therefore distinguish two Grafana use cases: + +``` text +Operator Grafana + │ + └── Cloud administrators can see all projects + → direct backend access may be acceptable after security review + +Tenant-facing Grafana + │ + └── Customer must only see own project + → direct DB access requires much stronger isolation design +``` + +Before production implementation, verify exactly how tenant/project +identity is represented in the stored InfluxDB series and how Grafana +access control will prevent cross-project visibility. + +------------------------------------------------------------------------ + +# 11. Current Deployment Status + +## Confirmed + +- [x] Monasca is already collecting OpenStack VM metrics. +- [x] Ceilometer is not required for the selected architecture. +- [x] Monasca metrics are project/tenant scoped. +- [x] `resource_id` maps to Nova VM UUID. +- [x] Compute nodes run Monasca Agent. +- [x] VM metrics are collected through the libvirt check. +- [x] Central Monasca is deployed on Kubernetes. +- [x] Kafka is part of the central pipeline. +- [x] Monasca Persister is running. +- [x] InfluxDB is the metric backend. +- [x] Monasca metric database is `mon`. +- [x] Legacy Monasca Grafana exists. +- [x] Legacy Grafana contains `monasca-grafana-datasource`. +- [x] Plugin version is 1.2.1 and targets Grafana 3.x. +- [x] Modern Grafana should not depend on that plugin by default. + +## Still To Be Determined + +- [ ] Exact InfluxDB Kubernetes Service name. +- [ ] InfluxDB service port and exposure type. +- [ ] Whether authentication is enabled and which credentials should + be used. +- [ ] Whether the modern Grafana cluster can route to the old Monasca + Kubernetes cluster. +- [ ] Exact InfluxDB schema/measurements used by Monasca. +- [ ] How tenant/project identity appears in InfluxDB tags. +- [ ] Appropriate Grafana datasource configuration. +- [ ] Dashboard variable design for project, VM, resource UUID, host, + and metric. +- [ ] Security model for operator-only versus tenant-facing + dashboards. +- [ ] Alerting design. +- [ ] Production validation and performance testing. + +------------------------------------------------------------------------ + +# 12. Next Deployment Phase + +The next investigation should focus only on the InfluxDB access path: + +``` text +Modern Grafana + │ + │ network? + ▼ +Old Monasca Kubernetes cluster + │ + ▼ +InfluxDB Service + │ + ▼ +database: mon +``` + +Information to collect: + +1. InfluxDB Kubernetes Service name. +2. Service type: ClusterIP / NodePort / LoadBalancer. +3. Service port. +4. Endpoints/pods behind the Service. +5. InfluxDB version. +6. Authentication configuration. +7. Whether the new Grafana cluster has network reachability to it. +8. Sample Monasca measurement/tag structure. + +Do not expose InfluxDB publicly merely to make Grafana connectivity +easier. Prefer a controlled private network path. + +------------------------------------------------------------------------ + +# 13. Reference Investigation Commands + +These are reference commands captured from the investigation. Validate +them against the environment before operational use. + +### Monasca tenant metrics + +``` bash +monasca --monasca-api-url http://mon-api.brilliant.com.bd/v2.0 \ + --monasca-api-version 2_0 \ + metric-list \ + --name cpu.utilization_perc \ + --tenant-id +``` + +### Nova project inventory + +``` bash +openstack server list --project --fit +``` + +### Nova VM placement + +``` bash +openstack server show \ + -f table \ + -c OS-EXT-SRV-ATTR:host \ + -c OS-EXT-SRV-ATTR:hypervisor_hostname \ + -c OS-EXT-SRV-ATTR:instance_name \ + -c status \ + -c project_id +``` + +### Verify local libvirt domain + +``` bash +virsh list --all +``` + +### Compute-node Monasca services + +``` bash +systemctl status monasca-collector +systemctl status monasca-forwarder +systemctl status monasca-statsd +``` + +### Monasca Agent VM configuration + +``` text +/etc/monasca/agent/conf.d/libvirt.yaml +``` + +### Central Monasca Kubernetes resources + +``` bash +kubectl get pods -n monasca +kubectl get svc -n monasca +kubectl get deploy -n monasca +``` + +### Existing Grafana deployment + +``` bash +kubectl describe deploy monasca-grafana -n monasca +``` + +### Legacy plugin location + +``` text +/var/lib/grafana/plugins/monasca-grafana-datasource +``` + +------------------------------------------------------------------------ + +# 14. Troubleshooting Notes Kept Separate From Deployment + +During discovery, two current VMs appeared absent from one +`cpu.utilization_perc` comparison. One was traced to `compute23`, and +its corresponding libvirt domain was confirmed running. + +That investigation was intentionally deferred because the primary +objective is **deploying the monitoring integration**, not achieving +immediate 100% metric-coverage troubleshooting. + +If metric coverage becomes a production acceptance criterion later, use +this chain: + +``` text +Nova ACTIVE VM inventory + ↓ +Monasca recent resource_id inventory + ↓ +Identify missing UUID + ↓ +Nova host + instance_name + ↓ +libvirt domain + ↓ +Monasca libvirt collector + ↓ +forwarder + ↓ +Monasca API / Kafka + ↓ +persister + ↓ +InfluxDB +``` + +Do not mix this troubleshooting track with the main Grafana integration +unless missing metrics materially block deployment. + +------------------------------------------------------------------------ + +# 15. Operational Risks Observed + +### Credentials + +Plaintext service credentials were visible during configuration +inspection. Secrets should be rotated where exposure occurred and +ultimately moved to an appropriate secret-management mechanism. + +### Legacy Software + +The Monasca Kubernetes installation and its bundled Grafana/plugins are +old. Avoid making obsolete frontend plugins dependencies of the new +monitoring platform. + +### Persister Restarts + +Several `monasca-persister` pods showed restart counts in the thousands. +This does not block architecture discovery, but it deserves a separate +reliability investigation before treating the Monasca backend as +production-grade for a new visualization dependency. + +### Cross-Cluster Connectivity + +The modern Grafana and old Monasca stack may be in different Kubernetes +clusters. A Kubernetes `ClusterIP` is normally cluster-internal; +connectivity must therefore be deliberately designed rather than +assuming Kubernetes service DNS will work cross-cluster. + +------------------------------------------------------------------------ + +# 16. Final Architecture Baseline + +Use this as the working architecture until the InfluxDB +connectivity/schema validation phase proves otherwise: + +``` text + ┌─────────────────────┐ + │ OpenStack │ + └──────────┬──────────┘ + │ + Nova / libvirt + │ + ┌──────────▼──────────┐ + │ Monasca Agents │ + │ on compute nodes │ + └──────────┬──────────┘ + │ + ▼ + Monasca API + │ + ▼ + Kafka + │ + ▼ + Monasca Persister + │ + ▼ + ┌─────────────────────┐ + │ InfluxDB │ + │ database: mon │ + └──────────┬──────────┘ + │ + proposed datasource + │ + ▼ +┌──────────────┐ ┌─────────────────────┐ ┌──────────────┐ +│ Prometheus │─────────►│ Modern Grafana │◄─────────│ OpenSearch │ +│ K8s/infra │ │ unified dashboards │ │ logs │ +└──────────────┘ └─────────────────────┘ └──────────────┘ +``` + +**Next milestone:** prove secure network connectivity from modern +Grafana to the Monasca InfluxDB service and inspect the `mon` database +schema before creating the Grafana datasource.