# OpenStack Monasca Monitoring Integration --- Architecture & Investigation SOP **Environment:** OpenStack Rocky + Monasca + Grafana + Prometheus + OpenSearch\ **Purpose:** Future-reference documentation for understanding the existing Monasca deployment and integrating its VM metrics into the modern monitoring platform. ------------------------------------------------------------------------ ## 1. Executive Summary The OpenStack environment already uses **Monasca** for VM/infrastructure metric collection. Ceilometer is intentionally not part of this design. The investigation established that: - Monasca VM metrics are being collected from OpenStack compute nodes. - Monasca metrics are **tenant/project scoped**. - The Monasca `resource_id` dimension maps to the **Nova server UUID**. - Compute nodes run the Monasca Agent components: - `monasca-collector` - `monasca-forwarder` - `monasca-statsd` - VM monitoring is performed through the Monasca Agent `libvirt` check. - The central Monasca platform is deployed in an **older Kubernetes cluster**, namespace `monasca`. - The central pipeline includes **Monasca API, Kafka, Monasca Persister, and InfluxDB**. - The Monasca metrics database is named **`mon`**. - An old Grafana deployment exists in the Monasca namespace. - That Grafana uses the legacy `monasca-grafana-datasource` plugin. - The installed plugin is version **1.2.1**, updated in 2017, and declares compatibility with **Grafana 3.x.x**. - Therefore, copying that plugin into a modern Grafana installation is not considered a suitable production integration path. - The preferred direction is to evaluate **modern Grafana → InfluxDB → Monasca metrics**, while preserving the existing Prometheus and OpenSearch data sources for their respective workloads. ------------------------------------------------------------------------ # 2. Bird's-Eye HLD ## 2.1 Existing Observability Landscape ``` text OPENSTACK CLOUD │ ┌────────────────┴────────────────┐ │ │ Compute Nodes OpenStack Control Plane │ │ Monasca Agent │ collector / forwarder / statsd │ ▼ Monasca ingestion │ ▼ Kafka │ ▼ Monasca Persister │ ▼ InfluxDB database: mon │ ├──────────────► Legacy Monasca Grafana │ + legacy Monasca datasource │ └──────────────► Proposed Modern Grafana integration MODERN OBSERVABILITY PLATFORM ┌──────────────────────────────────────────────┐ │ Grafana │ └──────────┬────────────────┬──────────────────┘ │ │ Prometheus OpenSearch │ │ infra/K8s metrics logs │ └──── existing monitoring stack Proposed additional path: Monasca InfluxDB ─────────────► Grafana InfluxDB datasource ``` ## 2.2 Target Responsibility Model System Primary responsibility ------------ ------------------------------------------------- Monasca OpenStack VM/service metric collection InfluxDB Persistent Monasca time-series storage Prometheus Existing Kubernetes/infrastructure metrics OpenSearch Centralized logs Grafana Unified visualization Nova Authoritative VM inventory and VM UUID metadata The objective is **not** to replace Monasca with Prometheus. Monasca remains the OpenStack metric collection system. ------------------------------------------------------------------------ # 3. Low-Level Design (LLD) ## 3.1 Compute-Node Metric Collection Observed services on compute nodes: ``` text monasca-collector.service monasca-forwarder.service monasca-statsd.service ``` Example collector process: ``` text /root/monasca-agent/bin/python /root/monasca-agent/bin/monasca-collector foreground ``` The service runs as: ``` text User=mon-agent Group=mon-agent ``` Agent configuration is located under: ``` text /etc/monasca/agent/ ├── agent.yaml └── conf.d/ ├── cpu.yaml ├── crash.yaml ├── disk.yaml ├── libvirt.yaml ├── load.yaml ├── memory.yaml ├── network.yaml └── process.yaml ``` The discovered `libvirt.yaml` confirms VM metric collection is enabled, including CPU, disk, network, and ping checks. Important settings observed: ``` yaml nova_refresh: 14400 vm_cpu_check_enable: true vm_disks_check_enable: true vm_network_check_enable: true vm_ping_check_enable: true vm_probation: 300 ``` `nova_refresh: 14400` represents a four-hour refresh interval. > Security: the inspected configuration contained a plaintext OpenStack > service credential. Treat exposed credentials as compromised and > rotate them through the normal credential-management process. ------------------------------------------------------------------------ ## 3.2 Monasca Metric Identity A typical VM CPU metric has dimensions similar to: ``` text name: cpu.utilization_perc dimensions: hostname: component: vm service: compute zone: nova resource_id: ``` The investigation confirmed: ``` text Monasca resource_id == Nova server UUID ``` This is critical for dashboard design because Nova inventory and Monasca metrics can be joined conceptually using the server UUID. ------------------------------------------------------------------------ ## 3.3 Tenant / Project Scoping A major investigation result was that: ``` text monasca metric-list ``` returns metrics for the **current Keystone tenant/project**. The Monasca CLI exposes a tenant override concept: ``` text --tenant-id ``` Therefore, seeing only a few `resource_id` values without specifying the intended tenant does **not** mean that Monasca only monitors those VMs. Logical query model: ``` text Keystone credentials │ ▼ Current project scope │ ▼ Monasca API │ ▼ Metrics belonging to that tenant ``` For operator-level investigation across projects, project context must be considered explicitly. ------------------------------------------------------------------------ # 4. Nova ↔ Monasca Correlation For a selected project, Nova server inventory was compared with Monasca metrics. Nova provides: ``` text Server UUID VM name Status Project ID Compute host Hypervisor hostname Instance name ``` Example mapping: ``` text Nova server UUID │ ├── Monasca dimension: resource_id │ └── Nova instance_name → libvirt domain ``` Example conceptual relationship: ``` text OpenStack Server UUID: 758f... Name: Jump&Proxy-VM-1p Host: compute23 Instance: instance-00017c71 │ ▼ compute23 libvirt │ ▼ instance-00017c71 ``` The selected VM was verified as a running libvirt domain on its Nova-assigned compute node. This demonstrated the useful troubleshooting chain: ``` text Monasca metric ↓ resource_id Nova server UUID ↓ host / instance_name Compute node ↓ libvirt domain ↓ Monasca libvirt collector ``` ------------------------------------------------------------------------ # 5. Central Monasca Platform The central Monasca components were eventually located in Kubernetes. Namespace: ``` text monasca ``` Observed components include: ``` text monasca-api monasca-client monasca-grafana monasca-influxdb-new monasca-memcached monasca-mysql monasca-persister standalone-kafka standalone-kafka-zookeeper ``` Helm releases observed: ``` text monasca standalone-kafka standalone-monasca-influxdb standalone-monasca-mysql kafka-manager ``` This establishes the central data flow: ``` text Compute Monasca Agent │ ▼ Monasca API │ ▼ Kafka │ ▼ Monasca Persister │ ▼ InfluxDB ``` The Monasca API DNS name resolves through the Kubernetes-facing endpoint: ``` text mon-api.brilliant.com.bd │ ▼ k8s.brilliant.com.bd ``` ------------------------------------------------------------------------ # 6. InfluxDB Backend The deployment configuration confirmed InfluxDB as the Monasca metrics backend. Database: ``` text mon ``` Conceptual configuration: ``` text Monasca API ─────► InfluxDB database "mon" ▲ │ Monasca Persister ``` This is important because modern Grafana has a maintained InfluxDB datasource, creating a potential integration route without relying on the obsolete Monasca Grafana plugin. ------------------------------------------------------------------------ # 7. Existing Legacy Grafana Existing deployment: ``` text monasca-grafana ``` Image: ``` text monasca/grafana ``` The deployment uses: ``` text Keystone authentication Monasca MySQL as Grafana metadata DB ConfigMap-mounted Grafana configuration ``` Inside the container, the following plugins were discovered: ``` text /var/lib/grafana/plugins/ ├── monasca-grafana-app ├── monasca-grafana-datasource └── natel-discrete-panel ``` ## 7.1 Legacy Monasca Datasource Metadata Discovered plugin: ``` text Name: Monasca ID: monasca-datasource Type: datasource Version: 1.2.1 Updated: 2017-10-23 Grafana dependency: 3.x.x ``` The package identifies itself as: ``` text Grafana datasource for use with OpenStack Monasca API ``` Therefore, the historical architecture was: ``` text Legacy Grafana │ ▼ Monasca datasource plugin │ ▼ Monasca API │ ▼ InfluxDB ``` This explains why searching the modern Grafana plugin catalog did not reveal a usable Monasca datasource. ------------------------------------------------------------------------ # 8. Why the Legacy Plugin Should Not Be the Default Modern Design The existing plugin is tied to a Grafana 3.x-era plugin model. The source tree contains legacy-style components such as: ``` text datasource.js module.js query_ctrl.js partials/ ``` Its metadata explicitly declares: ``` text grafanaVersion: 3.x.x ``` Therefore: ``` text Copy old plugin into new Grafana │ ▼ Legacy frontend/plugin APIs │ ▼ High compatibility and maintenance risk ``` A production monitoring deployment should avoid making this abandoned legacy component a new critical dependency unless it is deliberately modernized, tested, and maintained internally. ------------------------------------------------------------------------ # 9. Proposed Modern Integration The current preferred design to evaluate is: ``` text Modern Grafana / | \ / | \ ▼ ▼ ▼ Prometheus OpenSearch InfluxDB │ │ │ K8s/infra Logs Monasca metrics ``` Specifically: ``` text Monasca Agent ↓ Monasca API / Kafka ↓ Monasca Persister ↓ InfluxDB ("mon") ↓ Grafana InfluxDB datasource ``` This keeps each existing monitoring technology in its current role instead of forcing Monasca metrics through Prometheus. ------------------------------------------------------------------------ # 10. Important Design Concern: Multi-Tenancy Directly querying InfluxDB is **not architecturally identical** to querying the Monasca API. Monasca API provides tenant-aware behavior through Keystone/project context. Direct InfluxDB access potentially bypasses that API authorization boundary. Therefore distinguish two Grafana use cases: ``` text Operator Grafana │ └── Cloud administrators can see all projects → direct backend access may be acceptable after security review Tenant-facing Grafana │ └── Customer must only see own project → direct DB access requires much stronger isolation design ``` Before production implementation, verify exactly how tenant/project identity is represented in the stored InfluxDB series and how Grafana access control will prevent cross-project visibility. ------------------------------------------------------------------------ # 11. Current Deployment Status ## Confirmed - [x] Monasca is already collecting OpenStack VM metrics. - [x] Ceilometer is not required for the selected architecture. - [x] Monasca metrics are project/tenant scoped. - [x] `resource_id` maps to Nova VM UUID. - [x] Compute nodes run Monasca Agent. - [x] VM metrics are collected through the libvirt check. - [x] Central Monasca is deployed on Kubernetes. - [x] Kafka is part of the central pipeline. - [x] Monasca Persister is running. - [x] InfluxDB is the metric backend. - [x] Monasca metric database is `mon`. - [x] Legacy Monasca Grafana exists. - [x] Legacy Grafana contains `monasca-grafana-datasource`. - [x] Plugin version is 1.2.1 and targets Grafana 3.x. - [x] Modern Grafana should not depend on that plugin by default. ## Still To Be Determined - [ ] Exact InfluxDB Kubernetes Service name. - [ ] InfluxDB service port and exposure type. - [ ] Whether authentication is enabled and which credentials should be used. - [ ] Whether the modern Grafana cluster can route to the old Monasca Kubernetes cluster. - [ ] Exact InfluxDB schema/measurements used by Monasca. - [ ] How tenant/project identity appears in InfluxDB tags. - [ ] Appropriate Grafana datasource configuration. - [ ] Dashboard variable design for project, VM, resource UUID, host, and metric. - [ ] Security model for operator-only versus tenant-facing dashboards. - [ ] Alerting design. - [ ] Production validation and performance testing. ------------------------------------------------------------------------ # 12. Next Deployment Phase The next investigation should focus only on the InfluxDB access path: ``` text Modern Grafana │ │ network? ▼ Old Monasca Kubernetes cluster │ ▼ InfluxDB Service │ ▼ database: mon ``` Information to collect: 1. InfluxDB Kubernetes Service name. 2. Service type: ClusterIP / NodePort / LoadBalancer. 3. Service port. 4. Endpoints/pods behind the Service. 5. InfluxDB version. 6. Authentication configuration. 7. Whether the new Grafana cluster has network reachability to it. 8. Sample Monasca measurement/tag structure. Do not expose InfluxDB publicly merely to make Grafana connectivity easier. Prefer a controlled private network path. ------------------------------------------------------------------------ # 13. Reference Investigation Commands These are reference commands captured from the investigation. Validate them against the environment before operational use. ### Monasca tenant metrics ``` bash monasca --monasca-api-url http://mon-api.brilliant.com.bd/v2.0 \ --monasca-api-version 2_0 \ metric-list \ --name cpu.utilization_perc \ --tenant-id ``` ### Nova project inventory ``` bash openstack server list --project --fit ``` ### Nova VM placement ``` bash openstack server show \ -f table \ -c OS-EXT-SRV-ATTR:host \ -c OS-EXT-SRV-ATTR:hypervisor_hostname \ -c OS-EXT-SRV-ATTR:instance_name \ -c status \ -c project_id ``` ### Verify local libvirt domain ``` bash virsh list --all ``` ### Compute-node Monasca services ``` bash systemctl status monasca-collector systemctl status monasca-forwarder systemctl status monasca-statsd ``` ### Monasca Agent VM configuration ``` text /etc/monasca/agent/conf.d/libvirt.yaml ``` ### Central Monasca Kubernetes resources ``` bash kubectl get pods -n monasca kubectl get svc -n monasca kubectl get deploy -n monasca ``` ### Existing Grafana deployment ``` bash kubectl describe deploy monasca-grafana -n monasca ``` ### Legacy plugin location ``` text /var/lib/grafana/plugins/monasca-grafana-datasource ``` ------------------------------------------------------------------------ # 14. Troubleshooting Notes Kept Separate From Deployment During discovery, two current VMs appeared absent from one `cpu.utilization_perc` comparison. One was traced to `compute23`, and its corresponding libvirt domain was confirmed running. That investigation was intentionally deferred because the primary objective is **deploying the monitoring integration**, not achieving immediate 100% metric-coverage troubleshooting. If metric coverage becomes a production acceptance criterion later, use this chain: ``` text Nova ACTIVE VM inventory ↓ Monasca recent resource_id inventory ↓ Identify missing UUID ↓ Nova host + instance_name ↓ libvirt domain ↓ Monasca libvirt collector ↓ forwarder ↓ Monasca API / Kafka ↓ persister ↓ InfluxDB ``` Do not mix this troubleshooting track with the main Grafana integration unless missing metrics materially block deployment. ------------------------------------------------------------------------ # 15. Operational Risks Observed ### Credentials Plaintext service credentials were visible during configuration inspection. Secrets should be rotated where exposure occurred and ultimately moved to an appropriate secret-management mechanism. ### Legacy Software The Monasca Kubernetes installation and its bundled Grafana/plugins are old. Avoid making obsolete frontend plugins dependencies of the new monitoring platform. ### Persister Restarts Several `monasca-persister` pods showed restart counts in the thousands. This does not block architecture discovery, but it deserves a separate reliability investigation before treating the Monasca backend as production-grade for a new visualization dependency. ### Cross-Cluster Connectivity The modern Grafana and old Monasca stack may be in different Kubernetes clusters. A Kubernetes `ClusterIP` is normally cluster-internal; connectivity must therefore be deliberately designed rather than assuming Kubernetes service DNS will work cross-cluster. ------------------------------------------------------------------------ # 16. Final Architecture Baseline Use this as the working architecture until the InfluxDB connectivity/schema validation phase proves otherwise: ``` text ┌─────────────────────┐ │ OpenStack │ └──────────┬──────────┘ │ Nova / libvirt │ ┌──────────▼──────────┐ │ Monasca Agents │ │ on compute nodes │ └──────────┬──────────┘ │ ▼ Monasca API │ ▼ Kafka │ ▼ Monasca Persister │ ▼ ┌─────────────────────┐ │ InfluxDB │ │ database: mon │ └──────────┬──────────┘ │ proposed datasource │ ▼ ┌──────────────┐ ┌─────────────────────┐ ┌──────────────┐ │ Prometheus │─────────►│ Modern Grafana │◄─────────│ OpenSearch │ │ K8s/infra │ │ unified dashboards │ │ logs │ └──────────────┘ └─────────────────────┘ └──────────────┘ ``` **Next milestone:** prove secure network connectivity from modern Grafana to the Monasca InfluxDB service and inspect the `mon` database schema before creating the Grafana datasource.