21 KiB
OpenStack Monasca Monitoring Integration --- Architecture & Investigation SOP
Environment: OpenStack Rocky + Monasca + Grafana + Prometheus +
OpenSearch
Purpose: Future-reference documentation for understanding the
existing Monasca deployment and integrating its VM metrics into the
modern monitoring platform.
1. Executive Summary
The OpenStack environment already uses Monasca for VM/infrastructure metric collection. Ceilometer is intentionally not part of this design.
The investigation established that:
- Monasca VM metrics are being collected from OpenStack compute nodes.
- Monasca metrics are tenant/project scoped.
- The Monasca
resource_iddimension maps to the Nova server UUID. - Compute nodes run the Monasca Agent components:
monasca-collectormonasca-forwardermonasca-statsd
- VM monitoring is performed through the Monasca Agent
libvirtcheck. - The central Monasca platform is deployed in an older Kubernetes
cluster, namespace
monasca. - The central pipeline includes Monasca API, Kafka, Monasca Persister, and InfluxDB.
- The Monasca metrics database is named
mon. - An old Grafana deployment exists in the Monasca namespace.
- That Grafana uses the legacy
monasca-grafana-datasourceplugin. - The installed plugin is version 1.2.1, updated in 2017, and declares compatibility with Grafana 3.x.x.
- Therefore, copying that plugin into a modern Grafana installation is not considered a suitable production integration path.
- The preferred direction is to evaluate modern Grafana → InfluxDB → Monasca metrics, while preserving the existing Prometheus and OpenSearch data sources for their respective workloads.
2. Bird's-Eye HLD
2.1 Existing Observability Landscape
OPENSTACK CLOUD
│
┌────────────────┴────────────────┐
│ │
Compute Nodes OpenStack Control Plane
│
│ Monasca Agent
│ collector / forwarder / statsd
│
▼
Monasca ingestion
│
▼
Kafka
│
▼
Monasca Persister
│
▼
InfluxDB
database: mon
│
├──────────────► Legacy Monasca Grafana
│ + legacy Monasca datasource
│
└──────────────► Proposed Modern Grafana integration
MODERN OBSERVABILITY PLATFORM
┌──────────────────────────────────────────────┐
│ Grafana │
└──────────┬────────────────┬──────────────────┘
│ │
Prometheus OpenSearch
│ │
infra/K8s metrics logs
│
└──── existing monitoring stack
Proposed additional path:
Monasca InfluxDB ─────────────► Grafana InfluxDB datasource
2.2 Target Responsibility Model
System Primary responsibility
Monasca OpenStack VM/service metric collection InfluxDB Persistent Monasca time-series storage Prometheus Existing Kubernetes/infrastructure metrics OpenSearch Centralized logs Grafana Unified visualization Nova Authoritative VM inventory and VM UUID metadata
The objective is not to replace Monasca with Prometheus. Monasca remains the OpenStack metric collection system.
3. Low-Level Design (LLD)
3.1 Compute-Node Metric Collection
Observed services on compute nodes:
monasca-collector.service
monasca-forwarder.service
monasca-statsd.service
Example collector process:
/root/monasca-agent/bin/python
/root/monasca-agent/bin/monasca-collector foreground
The service runs as:
User=mon-agent
Group=mon-agent
Agent configuration is located under:
/etc/monasca/agent/
├── agent.yaml
└── conf.d/
├── cpu.yaml
├── crash.yaml
├── disk.yaml
├── libvirt.yaml
├── load.yaml
├── memory.yaml
├── network.yaml
└── process.yaml
The discovered libvirt.yaml confirms VM metric collection is enabled,
including CPU, disk, network, and ping checks.
Important settings observed:
nova_refresh: 14400
vm_cpu_check_enable: true
vm_disks_check_enable: true
vm_network_check_enable: true
vm_ping_check_enable: true
vm_probation: 300
nova_refresh: 14400 represents a four-hour refresh interval.
Security: the inspected configuration contained a plaintext OpenStack service credential. Treat exposed credentials as compromised and rotate them through the normal credential-management process.
3.2 Monasca Metric Identity
A typical VM CPU metric has dimensions similar to:
name: cpu.utilization_perc
dimensions:
hostname: <VM name>
component: vm
service: compute
zone: nova
resource_id: <Nova server UUID>
The investigation confirmed:
Monasca resource_id == Nova server UUID
This is critical for dashboard design because Nova inventory and Monasca metrics can be joined conceptually using the server UUID.
3.3 Tenant / Project Scoping
A major investigation result was that:
monasca metric-list
returns metrics for the current Keystone tenant/project.
The Monasca CLI exposes a tenant override concept:
--tenant-id <PROJECT_ID>
Therefore, seeing only a few resource_id values without specifying the
intended tenant does not mean that Monasca only monitors those VMs.
Logical query model:
Keystone credentials
│
▼
Current project scope
│
▼
Monasca API
│
▼
Metrics belonging to that tenant
For operator-level investigation across projects, project context must be considered explicitly.
4. Nova ↔ Monasca Correlation
For a selected project, Nova server inventory was compared with Monasca metrics.
Nova provides:
Server UUID
VM name
Status
Project ID
Compute host
Hypervisor hostname
Instance name
Example mapping:
Nova server UUID
│
├── Monasca dimension: resource_id
│
└── Nova instance_name → libvirt domain
Example conceptual relationship:
OpenStack Server
UUID: 758f...
Name: Jump&Proxy-VM-1p
Host: compute23
Instance: instance-00017c71
│
▼
compute23 libvirt
│
▼
instance-00017c71
The selected VM was verified as a running libvirt domain on its Nova-assigned compute node.
This demonstrated the useful troubleshooting chain:
Monasca metric
↓ resource_id
Nova server UUID
↓ host / instance_name
Compute node
↓
libvirt domain
↓
Monasca libvirt collector
5. Central Monasca Platform
The central Monasca components were eventually located in Kubernetes.
Namespace:
monasca
Observed components include:
monasca-api
monasca-client
monasca-grafana
monasca-influxdb-new
monasca-memcached
monasca-mysql
monasca-persister
standalone-kafka
standalone-kafka-zookeeper
Helm releases observed:
monasca
standalone-kafka
standalone-monasca-influxdb
standalone-monasca-mysql
kafka-manager
This establishes the central data flow:
Compute Monasca Agent
│
▼
Monasca API
│
▼
Kafka
│
▼
Monasca Persister
│
▼
InfluxDB
The Monasca API DNS name resolves through the Kubernetes-facing endpoint:
mon-api.brilliant.com.bd
│
▼
k8s.brilliant.com.bd
6. InfluxDB Backend
The deployment configuration confirmed InfluxDB as the Monasca metrics backend.
Database:
mon
Conceptual configuration:
Monasca API ─────► InfluxDB database "mon"
▲
│
Monasca Persister
This is important because modern Grafana has a maintained InfluxDB datasource, creating a potential integration route without relying on the obsolete Monasca Grafana plugin.
7. Existing Legacy Grafana
Existing deployment:
monasca-grafana
Image:
monasca/grafana
The deployment uses:
Keystone authentication
Monasca MySQL as Grafana metadata DB
ConfigMap-mounted Grafana configuration
Inside the container, the following plugins were discovered:
/var/lib/grafana/plugins/
├── monasca-grafana-app
├── monasca-grafana-datasource
└── natel-discrete-panel
7.1 Legacy Monasca Datasource Metadata
Discovered plugin:
Name: Monasca
ID: monasca-datasource
Type: datasource
Version: 1.2.1
Updated: 2017-10-23
Grafana dependency: 3.x.x
The package identifies itself as:
Grafana datasource for use with OpenStack Monasca API
Therefore, the historical architecture was:
Legacy Grafana
│
▼
Monasca datasource plugin
│
▼
Monasca API
│
▼
InfluxDB
This explains why searching the modern Grafana plugin catalog did not reveal a usable Monasca datasource.
8. Why the Legacy Plugin Should Not Be the Default Modern Design
The existing plugin is tied to a Grafana 3.x-era plugin model.
The source tree contains legacy-style components such as:
datasource.js
module.js
query_ctrl.js
partials/
Its metadata explicitly declares:
grafanaVersion: 3.x.x
Therefore:
Copy old plugin into new Grafana
│
▼
Legacy frontend/plugin APIs
│
▼
High compatibility and maintenance risk
A production monitoring deployment should avoid making this abandoned legacy component a new critical dependency unless it is deliberately modernized, tested, and maintained internally.
9. Proposed Modern Integration
The current preferred design to evaluate is:
Modern Grafana
/ | \
/ | \
▼ ▼ ▼
Prometheus OpenSearch InfluxDB
│ │ │
K8s/infra Logs Monasca
metrics
Specifically:
Monasca Agent
↓
Monasca API / Kafka
↓
Monasca Persister
↓
InfluxDB ("mon")
↓
Grafana InfluxDB datasource
This keeps each existing monitoring technology in its current role instead of forcing Monasca metrics through Prometheus.
10. Important Design Concern: Multi-Tenancy
Directly querying InfluxDB is not architecturally identical to querying the Monasca API.
Monasca API provides tenant-aware behavior through Keystone/project context.
Direct InfluxDB access potentially bypasses that API authorization boundary.
Therefore distinguish two Grafana use cases:
Operator Grafana
│
└── Cloud administrators can see all projects
→ direct backend access may be acceptable after security review
Tenant-facing Grafana
│
└── Customer must only see own project
→ direct DB access requires much stronger isolation design
Before production implementation, verify exactly how tenant/project identity is represented in the stored InfluxDB series and how Grafana access control will prevent cross-project visibility.
11. Current Deployment Status
Confirmed
- Monasca is already collecting OpenStack VM metrics.
- Ceilometer is not required for the selected architecture.
- Monasca metrics are project/tenant scoped.
resource_idmaps to Nova VM UUID.- Compute nodes run Monasca Agent.
- VM metrics are collected through the libvirt check.
- Central Monasca is deployed on Kubernetes.
- Kafka is part of the central pipeline.
- Monasca Persister is running.
- InfluxDB is the metric backend.
- Monasca metric database is
mon. - Legacy Monasca Grafana exists.
- Legacy Grafana contains
monasca-grafana-datasource. - Plugin version is 1.2.1 and targets Grafana 3.x.
- Modern Grafana should not depend on that plugin by default.
Still To Be Determined
- Exact InfluxDB Kubernetes Service name.
- InfluxDB service port and exposure type.
- Whether authentication is enabled and which credentials should be used.
- Whether the modern Grafana cluster can route to the old Monasca Kubernetes cluster.
- Exact InfluxDB schema/measurements used by Monasca.
- How tenant/project identity appears in InfluxDB tags.
- Appropriate Grafana datasource configuration.
- Dashboard variable design for project, VM, resource UUID, host, and metric.
- Security model for operator-only versus tenant-facing dashboards.
- Alerting design.
- Production validation and performance testing.
12. Next Deployment Phase
The next investigation should focus only on the InfluxDB access path:
Modern Grafana
│
│ network?
▼
Old Monasca Kubernetes cluster
│
▼
InfluxDB Service
│
▼
database: mon
Information to collect:
- InfluxDB Kubernetes Service name.
- Service type: ClusterIP / NodePort / LoadBalancer.
- Service port.
- Endpoints/pods behind the Service.
- InfluxDB version.
- Authentication configuration.
- Whether the new Grafana cluster has network reachability to it.
- Sample Monasca measurement/tag structure.
Do not expose InfluxDB publicly merely to make Grafana connectivity easier. Prefer a controlled private network path.
13. Reference Investigation Commands
These are reference commands captured from the investigation. Validate them against the environment before operational use.
Monasca tenant metrics
monasca --monasca-api-url http://mon-api.brilliant.com.bd/v2.0 \
--monasca-api-version 2_0 \
metric-list \
--name cpu.utilization_perc \
--tenant-id <PROJECT_ID>
Nova project inventory
openstack server list --project <PROJECT_ID> --fit
Nova VM placement
openstack server show <SERVER_UUID> \
-f table \
-c OS-EXT-SRV-ATTR:host \
-c OS-EXT-SRV-ATTR:hypervisor_hostname \
-c OS-EXT-SRV-ATTR:instance_name \
-c status \
-c project_id
Verify local libvirt domain
virsh list --all
Compute-node Monasca services
systemctl status monasca-collector
systemctl status monasca-forwarder
systemctl status monasca-statsd
Monasca Agent VM configuration
/etc/monasca/agent/conf.d/libvirt.yaml
Central Monasca Kubernetes resources
kubectl get pods -n monasca
kubectl get svc -n monasca
kubectl get deploy -n monasca
Existing Grafana deployment
kubectl describe deploy monasca-grafana -n monasca
Legacy plugin location
/var/lib/grafana/plugins/monasca-grafana-datasource
14. Troubleshooting Notes Kept Separate From Deployment
During discovery, two current VMs appeared absent from one
cpu.utilization_perc comparison. One was traced to compute23, and
its corresponding libvirt domain was confirmed running.
That investigation was intentionally deferred because the primary objective is deploying the monitoring integration, not achieving immediate 100% metric-coverage troubleshooting.
If metric coverage becomes a production acceptance criterion later, use this chain:
Nova ACTIVE VM inventory
↓
Monasca recent resource_id inventory
↓
Identify missing UUID
↓
Nova host + instance_name
↓
libvirt domain
↓
Monasca libvirt collector
↓
forwarder
↓
Monasca API / Kafka
↓
persister
↓
InfluxDB
Do not mix this troubleshooting track with the main Grafana integration unless missing metrics materially block deployment.
15. Operational Risks Observed
Credentials
Plaintext service credentials were visible during configuration inspection. Secrets should be rotated where exposure occurred and ultimately moved to an appropriate secret-management mechanism.
Legacy Software
The Monasca Kubernetes installation and its bundled Grafana/plugins are old. Avoid making obsolete frontend plugins dependencies of the new monitoring platform.
Persister Restarts
Several monasca-persister pods showed restart counts in the thousands.
This does not block architecture discovery, but it deserves a separate
reliability investigation before treating the Monasca backend as
production-grade for a new visualization dependency.
Cross-Cluster Connectivity
The modern Grafana and old Monasca stack may be in different Kubernetes
clusters. A Kubernetes ClusterIP is normally cluster-internal;
connectivity must therefore be deliberately designed rather than
assuming Kubernetes service DNS will work cross-cluster.
16. Final Architecture Baseline
Use this as the working architecture until the InfluxDB connectivity/schema validation phase proves otherwise:
┌─────────────────────┐
│ OpenStack │
└──────────┬──────────┘
│
Nova / libvirt
│
┌──────────▼──────────┐
│ Monasca Agents │
│ on compute nodes │
└──────────┬──────────┘
│
▼
Monasca API
│
▼
Kafka
│
▼
Monasca Persister
│
▼
┌─────────────────────┐
│ InfluxDB │
│ database: mon │
└──────────┬──────────┘
│
proposed datasource
│
▼
┌──────────────┐ ┌─────────────────────┐ ┌──────────────┐
│ Prometheus │─────────►│ Modern Grafana │◄─────────│ OpenSearch │
│ K8s/infra │ │ unified dashboards │ │ logs │
└──────────────┘ └─────────────────────┘ └──────────────┘
Next milestone: prove secure network connectivity from modern
Grafana to the Monasca InfluxDB service and inspect the mon database
schema before creating the Grafana datasource.