4925 lines
116 KiB
Markdown
4925 lines
116 KiB
Markdown
# OpenSearch + cPanel/Linux Observability Platform — Complete SOP
|
|
|
|
> **Canonical combined runbook** for deploying and operating OpenSearch on Kubernetes, OpenSearch Dashboards, Fluent Bit, Data Prepper, cPanel/Linux log ingestion, Linux Audit ingestion, dashboards, anomaly detection, forecasting, alerting, and troubleshooting.
|
|
>
|
|
> **Prepared by:** Md. Mosiur Rahnman
|
|
> **Document type:** Standard Operating Procedure (SOP) / Future Reference Runbook
|
|
> **Primary environment documented:** Kubernetes/K3s, OpenSearch 3.7.0, OpenSearch Dashboards 3.7.0, Data Prepper 2.8.0, Longhorn, NGINX Ingress, cPanel/Linux
|
|
> **Primary namespace:** `observability`
|
|
|
|
---
|
|
|
|
## Document Control
|
|
|
|
| Item | Value |
|
|
|---|---|
|
|
| Prepared by | Md. Mosiur Rahnman |
|
|
| Scope | OpenSearch platform + cPanel/Linux log observability |
|
|
| OpenSearch version used in the implementation | 3.7.0 |
|
|
| OpenSearch Dashboards version used | 3.7.0 |
|
|
| Data Prepper version used | 2.8.0 |
|
|
| Kubernetes namespace | `observability` |
|
|
| OpenSearch Helm release | `brilliant-opensearch` |
|
|
| OpenSearch cluster name | `brilliant-opensearch-cluster` |
|
|
| OpenSearch StatefulSet | `brilliant-opensearch-cluster-master` |
|
|
| OpenSearch Service | `brilliant-opensearch-cluster-master` |
|
|
| Data Prepper Service | `data-prepper` |
|
|
| StorageClass | `longhorn` |
|
|
| OpenSearch pod count | 3 |
|
|
| OpenSearch PVC size | 120Gi per pod |
|
|
| Dashboards external path | `/opensearch` |
|
|
| Data Prepper ingestion path | `/logs` |
|
|
| NGINX HTTP NodePort observed | `31599` |
|
|
| NGINX HTTPS NodePort observed | `31186` |
|
|
| cPanel access log | `/usr/local/cpanel/logs/access_log` |
|
|
|
|
> [!IMPORTANT]
|
|
> Values such as node IPs, NodePorts, release names, Service names, chart paths, certificate DNs, and storage topology are **environment-specific**. Verify them before reusing this SOP in another cluster.
|
|
|
|
---
|
|
|
|
## Table of Contents
|
|
|
|
1. [Purpose and Scope](#1-purpose-and-scope)
|
|
2. [Final Architecture](#2-final-architecture)
|
|
3. [Core Design Principles](#3-core-design-principles)
|
|
4. [Environment Variables and Naming](#4-environment-variables-and-naming)
|
|
5. [Pre-Deployment Prerequisites](#5-pre-deployment-prerequisites)
|
|
6. [OpenSearch PKI and TLS](#6-opensearch-pki-and-tls)
|
|
7. [Kubernetes Secrets](#7-kubernetes-secrets)
|
|
8. [OpenSearch Helm Configuration](#8-opensearch-helm-configuration)
|
|
9. [Render, Validate, and Deploy OpenSearch](#9-render-validate-and-deploy-opensearch)
|
|
10. [OpenSearch Security Initialization and Admin Password](#10-opensearch-security-initialization-and-admin-password)
|
|
11. [OpenSearch Cluster Verification](#11-opensearch-cluster-verification)
|
|
12. [OpenSearch Dashboards Deployment](#12-opensearch-dashboards-deployment)
|
|
13. [Data Prepper — Full Deployment, Routing, Parsing, and Verification](#13-data-prepper--full-deployment-routing-parsing-and-verification)
|
|
14. [Fluent Bit — Full cPanel/Linux Collection Procedure](#14-fluent-bit--full-cpanellinux-collection-procedure)
|
|
15. [cPanel Access Log Parsing](#15-cpanel-access-log-parsing)
|
|
16. [Linux Syslog and Other Log Sources](#16-linux-syslog-and-other-log-sources)
|
|
17. [Linux Audit Log Parsing](#17-linux-audit-log-parsing)
|
|
18. [OpenSearch Index Templates and Mapping](#18-opensearch-index-templates-and-mapping)
|
|
19. [End-to-End Verification](#19-end-to-end-verification)
|
|
20. [OpenSearch Dashboards Data Source and Workspace](#20-opensearch-dashboards-data-source-and-workspace)
|
|
21. [cPanel Access Dashboard](#21-cpanel-access-dashboard)
|
|
22. [Linux Audit Security Dashboard](#22-linux-audit-security-dashboard)
|
|
23. [Anomaly Detection, Forecasting, and Alerting](#23-anomaly-detection-forecasting-and-alerting)
|
|
24. [SMTP Notification Configuration](#24-smtp-notification-configuration)
|
|
25. [Reliability, Backup, and Recovery Tests](#25-reliability-backup-and-recovery-tests)
|
|
26. [Security Hardening](#26-security-hardening)
|
|
27. [Operational Monitoring](#27-operational-monitoring)
|
|
28. [Troubleshooting Matrix](#28-troubleshooting-matrix)
|
|
29. [Troubleshooting Decision Trees](#29-troubleshooting-decision-trees)
|
|
30. [Daily / Weekly Operational Checklists](#30-daily--weekly-operational-checklists)
|
|
31. [Windows Fluent Bit Variant](#31-windows-fluent-bit-variant)
|
|
32. [Cheat Sheet](#32-cheat-sheet)
|
|
33. [Lessons Learned and Known Limitations](#33-lessons-learned-and-known-limitations)
|
|
34. [Final Definition of Done](#34-final-definition-of-done)
|
|
|
|
---
|
|
|
|
# 1. Purpose and Scope
|
|
|
|
This SOP consolidates the complete implementation into one operational reference. It covers the path from Kubernetes infrastructure through searchable and visualized cPanel/Linux logs:
|
|
|
|
```text
|
|
cPanel / Linux Servers
|
|
|
|
|
| log files
|
|
v
|
|
Fluent Bit
|
|
- Tail input
|
|
- Offset DB
|
|
- Filesystem buffering
|
|
- source_file
|
|
- log_source classification
|
|
|
|
|
| HTTP/HTTPS POST /logs
|
|
v
|
|
NGINX Ingress
|
|
|
|
|
v
|
|
Data Prepper :2021
|
|
- One HTTP source
|
|
- Router
|
|
- Grok/date/key-value processors
|
|
|
|
|
| HTTPS :9200
|
|
v
|
|
OpenSearch 3-node cluster
|
|
- Longhorn PVCs
|
|
- REST TLS
|
|
- Transport TLS
|
|
- Security plugin
|
|
|
|
|
v
|
|
OpenSearch Dashboards
|
|
- Discover / Logs
|
|
- Workspaces
|
|
- Dashboards
|
|
- Anomaly Detection
|
|
- Forecasting
|
|
- Alerting
|
|
- Email notifications
|
|
```
|
|
|
|
This document preserves the tested implementation while separating:
|
|
|
|
- **Production baseline** — settings that should normally remain enabled.
|
|
- **Environment-specific values** — values that must be rechecked before reuse.
|
|
- **Diagnostic workarounds** — temporary troubleshooting settings that should not become permanent defaults.
|
|
- **Known limitations** — parser or reliability behaviors that need to be understood rather than hidden.
|
|
|
|
---
|
|
|
|
# 2. Final Architecture
|
|
|
|
## 2.1 Kubernetes/OpenSearch topology
|
|
|
|
```text
|
|
+--------------------------------------+
|
|
| Kubernetes / K3s |
|
|
| Namespace: observability |
|
|
| |
|
|
Browser | OpenSearch Dashboards |
|
|
| | :5601 |
|
|
| HTTP/HTTPS | | HTTPS |
|
|
v | v |
|
|
NGINX Ingress ---------->| OpenSearch Service :9200 |
|
|
^ | | |
|
|
| | +------+------+------+ |
|
|
| | | | | |
|
|
| | pod-0 pod-1 pod-2 |
|
|
| | | | | |
|
|
| | PVC-0 PVC-1 PVC-2 |
|
|
| | \________ Transport TLS :9300 ____/|
|
|
| | |
|
|
| | Data Prepper :2021 |
|
|
| | | HTTPS :9200 |
|
|
| | +----------------------------->|
|
|
| +--------------------------------------+
|
|
|
|
|
| HTTP/HTTPS /logs
|
|
|
|
|
Fluent Bit on cPanel/Linux servers
|
|
```
|
|
|
|
## 2.2 Log-routing architecture
|
|
|
|
```text
|
|
Fluent Bit
|
|
|
|
|
| JSON + source_file + log_source
|
|
v
|
|
Data Prepper router
|
|
|
|
|
+--> cpanel.access_log --> access pipeline --> cpanel-access-log-YYYY.MM.dd
|
|
|
|
|
+--> system.syslog ----> syslog pipeline --> cpanel-system-syslog-YYYY.MM.dd
|
|
|
|
|
+--> audit.audit.log --> audit pipeline ---> cpanel-audit-log-YYYY.MM.dd
|
|
|
|
|
+--> unknown sources --> catch-all --------> cpanel-all-log-YYYY.MM.dd
|
|
```
|
|
|
|
---
|
|
|
|
# 3. Core Design Principles
|
|
|
|
1. **Validate one layer at a time.** Do not debug OpenSearch if Fluent Bit is not reading the source file.
|
|
2. **Render Helm before applying it.** A valid `values.yaml` can still render an invalid or unintended Kubernetes manifest.
|
|
3. **Keep transport, parsing, routing, and storage failures separate.** `parsed: {}` is a parser problem; `502` is normally an upstream availability problem.
|
|
4. **Use one Data Prepper HTTP listener per port.** Multiple pipelines must not bind `:2021` independently.
|
|
5. **Classify logs early.** Fluent Bit knows the original file path, so create `source_file` and normalized `log_source` before transport.
|
|
6. **Use dedicated downstream pipelines for different formats.** cPanel access logs, syslog, and Auditd have fundamentally different schemas.
|
|
7. **Preserve the raw log.** Parsing should enrich the event, not destroy the original message.
|
|
8. **Use least privilege.** Do not reuse the OpenSearch admin credential for Dashboards or ingestion.
|
|
9. **Treat TLS trust and TLS hostname validation as different checks.** A trusted CA does not fix a SAN mismatch.
|
|
10. **Treat Kubernetes `Running` and application health as different states.** A pod can run while OpenSearch cluster formation, authentication, or authorization is broken.
|
|
11. **OpenSearch shard replicas are not backups.** Use snapshots to independent storage.
|
|
12. **Base64 is encoding, not encryption.** Kubernetes Secret output must still be treated as sensitive.
|
|
13. **Do not commit real passwords, private keys, SMTP credentials, or encryption master keys to Git.**
|
|
|
|
---
|
|
|
|
# 4. Environment Variables and Naming
|
|
|
|
Use variables in shell sessions to reduce copy/paste mistakes.
|
|
|
|
```bash
|
|
export NS="observability"
|
|
export OS_RELEASE="brilliant-opensearch"
|
|
export OS_CLUSTER="brilliant-opensearch-cluster"
|
|
export OS_STS="brilliant-opensearch-cluster-master"
|
|
export OS_SVC="brilliant-opensearch-cluster-master"
|
|
export OS_HEADLESS_SVC="brilliant-opensearch-cluster-master-headless"
|
|
export DP_SVC="data-prepper"
|
|
export DP_PORT="2021"
|
|
export DASH_BASEPATH="/opensearch"
|
|
export INGRESS_HTTP_NODEPORT="31599"
|
|
export INGRESS_HTTPS_NODEPORT="31186"
|
|
```
|
|
|
|
Check the real values instead of assuming them:
|
|
|
|
```bash
|
|
kubectl get ns
|
|
helm list -A
|
|
kubectl get sts,deploy,svc,ingress -A
|
|
kubectl get nodes -o wide
|
|
```
|
|
|
|
For the local Helm chart used in the implementation:
|
|
|
|
```bash
|
|
export OS_CHART="${HOME}/monitoring/helm-charts/charts/opensearch"
|
|
```
|
|
|
|
Verify it exists:
|
|
|
|
```bash
|
|
test -f "$OS_CHART/Chart.yaml" && echo "OpenSearch chart found"
|
|
```
|
|
|
|
---
|
|
|
|
# 5. Pre-Deployment Prerequisites
|
|
|
|
## 5.1 Client tools
|
|
|
|
Verify required tools:
|
|
|
|
```bash
|
|
kubectl version --client
|
|
helm version
|
|
openssl version
|
|
curl --version
|
|
```
|
|
|
|
Optional but useful:
|
|
|
|
```bash
|
|
jq --version
|
|
yq --version
|
|
```
|
|
|
|
## 5.2 Kubernetes nodes
|
|
|
|
```bash
|
|
kubectl get nodes -o wide
|
|
kubectl get nodes --show-labels
|
|
kubectl describe node k8s-storage-01 | sed -n '/Taints:/,/Unschedulable:/p'
|
|
```
|
|
|
|
Target placement from the implementation:
|
|
|
|
```text
|
|
k8s-storage-01 -> eligible, dedicated=storage:NoSchedule
|
|
k8s-worker-01 -> eligible
|
|
k8s-worker-02 -> eligible
|
|
k8s-cp-01 -> excluded from OpenSearch
|
|
```
|
|
|
|
If `k8s-storage-01` is tainted:
|
|
|
|
```text
|
|
dedicated=storage:NoSchedule
|
|
```
|
|
|
|
OpenSearch requires a matching toleration if that node should be eligible.
|
|
|
|
## 5.3 Longhorn
|
|
|
|
```bash
|
|
kubectl get storageclass
|
|
kubectl get pods -n longhorn-system
|
|
```
|
|
|
|
Expected StorageClass:
|
|
|
|
```text
|
|
longhorn
|
|
```
|
|
|
|
Storage planning for the documented cluster:
|
|
|
|
```text
|
|
3 OpenSearch pods x 120Gi = 360Gi logical PVC capacity
|
|
```
|
|
|
|
Longhorn replication consumes additional physical storage. OpenSearch shard replication consumes additional logical data copies as well; they are separate redundancy layers.
|
|
|
|
## 5.4 `vm.max_map_count`
|
|
|
|
Check every eligible OpenSearch node:
|
|
|
|
```bash
|
|
for node in k8s-storage-01 k8s-worker-01 k8s-worker-02; do
|
|
echo "=== $node ==="
|
|
ssh "$node" 'sysctl vm.max_map_count'
|
|
done
|
|
```
|
|
|
|
Target used by the Helm chart:
|
|
|
|
```text
|
|
vm.max_map_count = 262144
|
|
```
|
|
|
|
Prefer configuring the kernel permanently on the host:
|
|
|
|
```bash
|
|
cat >/etc/sysctl.d/99-opensearch.conf <<'EOF_SYSCTL'
|
|
vm.max_map_count=262144
|
|
EOF_SYSCTL
|
|
|
|
sysctl --system
|
|
```
|
|
|
|
If the hosts are already configured, disable privileged sysctl init mechanisms in Helm. Do not enable multiple sysctl methods unnecessarily.
|
|
|
|
## 5.5 Namespace
|
|
|
|
```bash
|
|
kubectl create namespace "$NS" --dry-run=client -o yaml | kubectl apply -f -
|
|
```
|
|
|
|
---
|
|
|
|
# 6. OpenSearch PKI and TLS
|
|
|
|
## 6.1 Certificate roles
|
|
|
|
Use separate certificate purposes:
|
|
|
|
```text
|
|
Root CA
|
|
├── signs node certificates
|
|
└── signs admin certificate
|
|
|
|
Node certificates
|
|
├── REST TLS server identity
|
|
├── transport TLS server identity
|
|
└── transport TLS client identity
|
|
|
|
Admin certificate
|
|
└── privileged OpenSearch Security administration
|
|
```
|
|
|
|
Never mount the Root CA private key into OpenSearch.
|
|
|
|
```text
|
|
root-ca-key.pem -> offline/encrypted CA material only
|
|
root-ca.pem -> safe to distribute as trust anchor
|
|
```
|
|
|
|
## 6.2 Expected PKI files
|
|
|
|
```text
|
|
root-ca.pem
|
|
root-ca-key.pem
|
|
|
|
admin.pem
|
|
admin-key.pem
|
|
|
|
node-0.pem
|
|
node-0-key.pem
|
|
node-1.pem
|
|
node-1-key.pem
|
|
node-2.pem
|
|
node-2-key.pem
|
|
```
|
|
|
|
## 6.3 SAN design
|
|
|
|
Use a short certificate CN and put Kubernetes DNS identities in SANs.
|
|
|
|
Common Service DNS names:
|
|
|
|
```text
|
|
brilliant-opensearch-cluster-master
|
|
brilliant-opensearch-cluster-master.observability
|
|
brilliant-opensearch-cluster-master.observability.svc
|
|
brilliant-opensearch-cluster-master.observability.svc.cluster.local
|
|
|
|
brilliant-opensearch-cluster-master-headless
|
|
brilliant-opensearch-cluster-master-headless.observability
|
|
brilliant-opensearch-cluster-master-headless.observability.svc
|
|
brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local
|
|
```
|
|
|
|
Pod-specific names should also be included where transport communication uses them:
|
|
|
|
```text
|
|
brilliant-opensearch-cluster-master-0.brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local
|
|
brilliant-opensearch-cluster-master-1.brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local
|
|
brilliant-opensearch-cluster-master-2.brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local
|
|
```
|
|
|
|
## 6.4 Optional OpenSSL generation template
|
|
|
|
If certificates already exist, skip generation and perform verification only.
|
|
|
|
### Root CA
|
|
|
|
```bash
|
|
openssl genrsa -out root-ca-key.pem 4096
|
|
|
|
openssl req -x509 -new -sha256 \
|
|
-key root-ca-key.pem \
|
|
-days 3650 \
|
|
-out root-ca.pem \
|
|
-subj '/C=BD/ST=Dhaka/L=Dhaka/O=Brilliant/OU=OpenSearch CA/CN=Brilliant OpenSearch Root CA'
|
|
```
|
|
|
|
### Admin certificate
|
|
|
|
```bash
|
|
openssl genrsa -out admin-key-rsa.pem 2048
|
|
openssl pkcs8 -topk8 -nocrypt \
|
|
-in admin-key-rsa.pem \
|
|
-out admin-key.pem
|
|
|
|
openssl req -new \
|
|
-key admin-key.pem \
|
|
-out admin.csr \
|
|
-subj '/C=BD/ST=Dhaka/L=Dhaka/O=Brilliant/OU=OpenSearch Administrators/CN=opensearch-admin'
|
|
|
|
cat >admin.ext <<'EOF_ADMIN_EXT'
|
|
basicConstraints=CA:FALSE
|
|
keyUsage=digitalSignature,keyEncipherment
|
|
extendedKeyUsage=clientAuth
|
|
EOF_ADMIN_EXT
|
|
|
|
openssl x509 -req \
|
|
-in admin.csr \
|
|
-CA root-ca.pem \
|
|
-CAkey root-ca-key.pem \
|
|
-CAcreateserial \
|
|
-out admin.pem \
|
|
-days 825 \
|
|
-sha256 \
|
|
-extfile admin.ext
|
|
```
|
|
|
|
### Node certificate example for pod 0
|
|
|
|
Create a SAN file containing the actual Service and pod DNS names:
|
|
|
|
```bash
|
|
cat >node-0.ext <<'EOF_NODE0_EXT'
|
|
basicConstraints=CA:FALSE
|
|
keyUsage=digitalSignature,keyEncipherment
|
|
extendedKeyUsage=serverAuth,clientAuth
|
|
subjectAltName=@alt_names
|
|
|
|
[alt_names]
|
|
DNS.1=brilliant-opensearch-cluster-master
|
|
DNS.2=brilliant-opensearch-cluster-master.observability
|
|
DNS.3=brilliant-opensearch-cluster-master.observability.svc
|
|
DNS.4=brilliant-opensearch-cluster-master.observability.svc.cluster.local
|
|
DNS.5=brilliant-opensearch-cluster-master-headless
|
|
DNS.6=brilliant-opensearch-cluster-master-headless.observability
|
|
DNS.7=brilliant-opensearch-cluster-master-headless.observability.svc
|
|
DNS.8=brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local
|
|
DNS.9=brilliant-opensearch-cluster-master-0.brilliant-opensearch-cluster-master-headless
|
|
DNS.10=brilliant-opensearch-cluster-master-0.brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local
|
|
EOF_NODE0_EXT
|
|
|
|
openssl genrsa -out node-0-key-rsa.pem 2048
|
|
openssl pkcs8 -topk8 -nocrypt \
|
|
-in node-0-key-rsa.pem \
|
|
-out node-0-key.pem
|
|
|
|
openssl req -new \
|
|
-key node-0-key.pem \
|
|
-out node-0.csr \
|
|
-subj '/C=BD/ST=Dhaka/L=Dhaka/O=Brilliant/OU=OpenSearch Nodes/CN=opensearch-master-0'
|
|
|
|
openssl x509 -req \
|
|
-in node-0.csr \
|
|
-CA root-ca.pem \
|
|
-CAkey root-ca-key.pem \
|
|
-CAcreateserial \
|
|
-out node-0.pem \
|
|
-days 825 \
|
|
-sha256 \
|
|
-extfile node-0.ext
|
|
```
|
|
|
|
Repeat with the correct CN and pod-specific SAN for node 1 and node 2.
|
|
|
|
## 6.5 Certificate verification
|
|
|
|
Verify chain:
|
|
|
|
```bash
|
|
openssl verify -CAfile root-ca.pem admin.pem
|
|
openssl verify -CAfile root-ca.pem node-0.pem
|
|
openssl verify -CAfile root-ca.pem node-1.pem
|
|
openssl verify -CAfile root-ca.pem node-2.pem
|
|
```
|
|
|
|
Verify subjects:
|
|
|
|
```bash
|
|
openssl x509 -in admin.pem -noout -subject -nameopt RFC2253
|
|
openssl x509 -in node-0.pem -noout -subject -nameopt RFC2253
|
|
openssl x509 -in node-1.pem -noout -subject -nameopt RFC2253
|
|
openssl x509 -in node-2.pem -noout -subject -nameopt RFC2253
|
|
```
|
|
|
|
Verify SANs:
|
|
|
|
```bash
|
|
openssl x509 -in node-0.pem -noout -text | sed -n '/Subject Alternative Name/,+2p'
|
|
```
|
|
|
|
Verify EKU:
|
|
|
|
```bash
|
|
openssl x509 -in node-0.pem -noout -text | sed -n '/Extended Key Usage/,+2p'
|
|
```
|
|
|
|
Verify certificate/key match:
|
|
|
|
```bash
|
|
openssl x509 -in node-0.pem -pubkey -noout | sha256sum
|
|
openssl pkey -in node-0-key.pem -pubout | sha256sum
|
|
```
|
|
|
|
The hashes must match.
|
|
|
|
Verify validity:
|
|
|
|
```bash
|
|
openssl x509 -in node-0.pem -noout -dates
|
|
```
|
|
|
|
---
|
|
|
|
# 7. Kubernetes Secrets
|
|
|
|
## 7.1 Initial admin password
|
|
|
|
Do not place the real password in Git.
|
|
|
|
```bash
|
|
read -rsp 'Initial OpenSearch admin password: ' OPENSEARCH_INITIAL_ADMIN_PASSWORD
|
|
echo
|
|
|
|
kubectl create secret generic opensearch-secret \
|
|
-n "$NS" \
|
|
--from-literal=OPENSEARCH_INITIAL_ADMIN_PASSWORD="$OPENSEARCH_INITIAL_ADMIN_PASSWORD" \
|
|
--dry-run=client -o yaml | kubectl apply -f -
|
|
|
|
unset OPENSEARCH_INITIAL_ADMIN_PASSWORD
|
|
```
|
|
|
|
## 7.2 Node certificate Secrets
|
|
|
|
```bash
|
|
kubectl create secret generic opensearch-node-0-cert \
|
|
-n "$NS" \
|
|
--from-file=node-0.pem \
|
|
--from-file=node-0-key.pem \
|
|
--from-file=root-ca.pem \
|
|
--dry-run=client -o yaml | kubectl apply -f -
|
|
|
|
kubectl create secret generic opensearch-node-1-cert \
|
|
-n "$NS" \
|
|
--from-file=node-1.pem \
|
|
--from-file=node-1-key.pem \
|
|
--from-file=root-ca.pem \
|
|
--dry-run=client -o yaml | kubectl apply -f -
|
|
|
|
kubectl create secret generic opensearch-node-2-cert \
|
|
-n "$NS" \
|
|
--from-file=node-2.pem \
|
|
--from-file=node-2-key.pem \
|
|
--from-file=root-ca.pem \
|
|
--dry-run=client -o yaml | kubectl apply -f -
|
|
```
|
|
|
|
## 7.3 Admin certificate Secret
|
|
|
|
```bash
|
|
kubectl create secret generic opensearch-admin-cert \
|
|
-n "$NS" \
|
|
--from-file=admin.pem \
|
|
--from-file=admin-key.pem \
|
|
--from-file=root-ca.pem \
|
|
--dry-run=client -o yaml | kubectl apply -f -
|
|
```
|
|
|
|
## 7.4 Verify Secret metadata without dumping data
|
|
|
|
```bash
|
|
kubectl get secret -n "$NS" \
|
|
opensearch-secret \
|
|
opensearch-node-0-cert \
|
|
opensearch-node-1-cert \
|
|
opensearch-node-2-cert \
|
|
opensearch-admin-cert
|
|
```
|
|
|
|
Avoid:
|
|
|
|
```bash
|
|
kubectl get secret <name> -o yaml
|
|
```
|
|
|
|
when the output would expose sensitive Base64 data in terminals, tickets, screenshots, or chat logs.
|
|
|
|
---
|
|
|
|
# 8. OpenSearch Helm Configuration
|
|
|
|
## 8.1 Resource design
|
|
|
|
Per OpenSearch pod:
|
|
|
|
```text
|
|
CPU request: 2
|
|
CPU limit: 4
|
|
Memory request: 6Gi
|
|
Memory limit: 8Gi
|
|
JVM heap: 3Gi
|
|
PVC: 120Gi
|
|
```
|
|
|
|
## 8.2 Helm values baseline
|
|
|
|
The exact chart schema must always be checked with the pinned chart:
|
|
|
|
```bash
|
|
helm show values "$OS_CHART" > /tmp/opensearch-default-values.yaml
|
|
```
|
|
|
|
Critical values for this environment:
|
|
|
|
```yaml
|
|
clusterName: brilliant-opensearch-cluster
|
|
nodeGroup: master
|
|
singleNode: false
|
|
masterService: brilliant-opensearch-cluster-master
|
|
|
|
roles:
|
|
- master
|
|
- ingest
|
|
- data
|
|
- remote_cluster_client
|
|
|
|
replicas: 3
|
|
|
|
image:
|
|
repository: opensearchproject/opensearch
|
|
tag: "3.7.0"
|
|
pullPolicy: IfNotPresent
|
|
|
|
opensearchJavaOpts: "-Xms3g -Xmx3g"
|
|
|
|
resources:
|
|
requests:
|
|
cpu: "2"
|
|
memory: 6Gi
|
|
limits:
|
|
cpu: "4"
|
|
memory: 8Gi
|
|
|
|
persistence:
|
|
enabled: true
|
|
storageClass: longhorn
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
size: 120Gi
|
|
|
|
antiAffinity: hard
|
|
antiAffinityTopologyKey: kubernetes.io/hostname
|
|
podManagementPolicy: Parallel
|
|
|
|
protocol: https
|
|
httpPort: 9200
|
|
transportPort: 9300
|
|
metricsPort: 9600
|
|
|
|
podSecurityContext:
|
|
fsGroup: 1000
|
|
runAsUser: 1000
|
|
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 1000
|
|
capabilities:
|
|
drop:
|
|
- ALL
|
|
|
|
envFrom:
|
|
- secretRef:
|
|
name: opensearch-secret
|
|
|
|
secretMounts: []
|
|
|
|
sysctl:
|
|
enabled: false
|
|
|
|
sysctlInit:
|
|
enabled: false
|
|
```
|
|
|
|
> [!NOTE]
|
|
> If the worker nodes are **not** preconfigured with `vm.max_map_count=262144`, enable exactly one supported sysctl method. Do not enable both pod-level sysctl and a privileged sysctl init container unless you have a specific reason.
|
|
|
|
## 8.3 Node scheduling
|
|
|
|
Storage taint toleration:
|
|
|
|
```yaml
|
|
tolerations:
|
|
- key: dedicated
|
|
operator: Equal
|
|
value: storage
|
|
effect: NoSchedule
|
|
```
|
|
|
|
Use labels/affinity to include only the intended storage/worker nodes. Do not rely on anti-affinity to exclude the control plane; anti-affinity controls pod co-location, not arbitrary node blacklisting.
|
|
|
|
Verify rendered affinity before install.
|
|
|
|
## 8.4 Per-pod certificate selection
|
|
|
|
The main OpenSearch container should see only its selected runtime certificate set.
|
|
|
|
```yaml
|
|
extraVolumes:
|
|
- name: node-0-source
|
|
secret:
|
|
secretName: opensearch-node-0-cert
|
|
defaultMode: 0440
|
|
|
|
- name: node-1-source
|
|
secret:
|
|
secretName: opensearch-node-1-cert
|
|
defaultMode: 0440
|
|
|
|
- name: node-2-source
|
|
secret:
|
|
secretName: opensearch-node-2-cert
|
|
defaultMode: 0440
|
|
|
|
- name: runtime-certs
|
|
emptyDir: {}
|
|
|
|
extraVolumeMounts:
|
|
- name: runtime-certs
|
|
mountPath: /usr/share/opensearch/config/certs
|
|
readOnly: true
|
|
```
|
|
|
|
Init container:
|
|
|
|
```yaml
|
|
extraInitContainers:
|
|
- name: select-node-certificate
|
|
image: busybox:1.36
|
|
command:
|
|
- sh
|
|
- -ec
|
|
- |
|
|
ordinal="${HOSTNAME##*-}"
|
|
|
|
case "$ordinal" in
|
|
0|1|2) ;;
|
|
*)
|
|
echo "Unexpected StatefulSet ordinal: $ordinal" >&2
|
|
exit 1
|
|
;;
|
|
esac
|
|
|
|
src="/source/node-${ordinal}"
|
|
cert="${src}/node-${ordinal}.pem"
|
|
key="${src}/node-${ordinal}-key.pem"
|
|
ca="${src}/root-ca.pem"
|
|
|
|
test -r "$cert"
|
|
test -r "$key"
|
|
test -r "$ca"
|
|
|
|
cp "$cert" /runtime/node.pem
|
|
cp "$key" /runtime/node-key.pem
|
|
cp "$ca" /runtime/root-ca.pem
|
|
|
|
chown 1000:1000 /runtime/node.pem /runtime/node-key.pem /runtime/root-ca.pem
|
|
chmod 0644 /runtime/node.pem /runtime/root-ca.pem
|
|
chmod 0600 /runtime/node-key.pem
|
|
|
|
ls -l /runtime
|
|
volumeMounts:
|
|
- name: node-0-source
|
|
mountPath: /source/node-0
|
|
readOnly: true
|
|
- name: node-1-source
|
|
mountPath: /source/node-1
|
|
readOnly: true
|
|
- name: node-2-source
|
|
mountPath: /source/node-2
|
|
readOnly: true
|
|
- name: runtime-certs
|
|
mountPath: /runtime
|
|
```
|
|
|
|
> [!IMPORTANT]
|
|
> The `chown 1000:1000` matches the documented chart runtime user. Verify the actual image UID/GID before copying this into a different image or chart.
|
|
|
|
```bash
|
|
kubectl exec -n "$NS" "${OS_STS}-0" -c opensearch -- id
|
|
```
|
|
|
|
The init container and main container mount the same `emptyDir` at different paths:
|
|
|
|
```text
|
|
init container: /runtime/node.pem
|
|
main container: /usr/share/opensearch/config/certs/node.pem
|
|
```
|
|
|
|
These are the same underlying file.
|
|
|
|
## 8.5 OpenSearch TLS configuration
|
|
|
|
Use fixed runtime paths:
|
|
|
|
```yaml
|
|
config:
|
|
opensearch.yml: |
|
|
cluster.name: brilliant-opensearch-cluster
|
|
network.host: 0.0.0.0
|
|
|
|
plugins.security.ssl.transport.pemcert_filepath: certs/node.pem
|
|
plugins.security.ssl.transport.pemkey_filepath: certs/node-key.pem
|
|
plugins.security.ssl.transport.pemtrustedcas_filepath: certs/root-ca.pem
|
|
plugins.security.ssl.transport.enforce_hostname_verification: true
|
|
|
|
plugins.security.ssl.http.enabled: true
|
|
plugins.security.ssl.http.pemcert_filepath: certs/node.pem
|
|
plugins.security.ssl.http.pemkey_filepath: certs/node-key.pem
|
|
plugins.security.ssl.http.pemtrustedcas_filepath: certs/root-ca.pem
|
|
|
|
plugins.security.nodes_dn:
|
|
- 'CN=opensearch-master-0,OU=OpenSearch Nodes,O=Brilliant,L=Dhaka,ST=Dhaka,C=BD'
|
|
- 'CN=opensearch-master-1,OU=OpenSearch Nodes,O=Brilliant,L=Dhaka,ST=Dhaka,C=BD'
|
|
- 'CN=opensearch-master-2,OU=OpenSearch Nodes,O=Brilliant,L=Dhaka,ST=Dhaka,C=BD'
|
|
|
|
plugins.security.authcz.admin_dn:
|
|
- 'CN=opensearch-admin,OU=OpenSearch Administrators,O=Brilliant,L=Dhaka,ST=Dhaka,C=BD'
|
|
```
|
|
|
|
Use the **actual RFC2253 certificate DNs** from `openssl x509`, not guessed strings.
|
|
|
|
## 8.6 Temporary transport hostname-verification workaround
|
|
|
|
If logs show:
|
|
|
|
```text
|
|
No subject alternative DNS name matching brilliant-opensearch-cluster-master-headless found
|
|
```
|
|
|
|
then the certificate SAN is wrong for the DNS name used by transport communication.
|
|
|
|
Temporary diagnostic setting:
|
|
|
|
```yaml
|
|
plugins.security.ssl.transport.enforce_hostname_verification: false
|
|
```
|
|
|
|
This is not the desired production state. Regenerate node certificates with correct SANs, deploy them, verify transport TLS, then restore:
|
|
|
|
```yaml
|
|
plugins.security.ssl.transport.enforce_hostname_verification: true
|
|
```
|
|
|
|
---
|
|
|
|
# 9. Render, Validate, and Deploy OpenSearch
|
|
|
|
Use this order every time:
|
|
|
|
```text
|
|
values.yaml
|
|
-> helm lint
|
|
-> helm template
|
|
-> inspect rendered YAML
|
|
-> Kubernetes server-side dry-run
|
|
-> helm upgrade/install
|
|
-> runtime validation
|
|
```
|
|
|
|
## 9.1 Helm lint
|
|
|
|
```bash
|
|
helm lint "$OS_CHART" -f values.yaml
|
|
```
|
|
|
|
## 9.2 Render
|
|
|
|
```bash
|
|
helm template "$OS_RELEASE" "$OS_CHART" \
|
|
-n "$NS" \
|
|
-f values.yaml \
|
|
> rendered-opensearch.yaml
|
|
```
|
|
|
|
## 9.3 Inspect rendered manifest
|
|
|
|
```bash
|
|
grep -nE 'kind: StatefulSet|replicas:|storageClassName:|120Gi|runtime-certs|node-[012]-source|select-node-certificate|nodes_dn|admin_dn|9200|9300' rendered-opensearch.yaml
|
|
```
|
|
|
|
Verify:
|
|
|
|
```text
|
|
[ ] replicas = 3
|
|
[ ] image/version pinned
|
|
[ ] requests/limits present
|
|
[ ] 120Gi Longhorn PVC template
|
|
[ ] hard anti-affinity
|
|
[ ] expected toleration / affinity
|
|
[ ] runtime-certs emptyDir
|
|
[ ] init container selects certificate by ordinal
|
|
[ ] main container mounts runtime-certs only
|
|
[ ] node DNs and admin DN correct
|
|
[ ] REST port 9200
|
|
[ ] transport port 9300
|
|
[ ] bootstrap Secret reference present
|
|
```
|
|
|
|
## 9.4 Kubernetes API dry-run
|
|
|
|
```bash
|
|
kubectl apply --dry-run=server -f rendered-opensearch.yaml
|
|
```
|
|
|
|
## 9.5 Deploy
|
|
|
|
```bash
|
|
helm upgrade --install "$OS_RELEASE" "$OS_CHART" \
|
|
-n "$NS" \
|
|
-f values.yaml
|
|
```
|
|
|
|
## 9.6 Observe rollout
|
|
|
|
```bash
|
|
kubectl get pods -n "$NS" -w
|
|
```
|
|
|
|
In another terminal:
|
|
|
|
```bash
|
|
kubectl rollout status statefulset/"$OS_STS" -n "$NS"
|
|
```
|
|
|
|
---
|
|
|
|
# 10. OpenSearch Security Initialization and Admin Password
|
|
|
|
## 10.1 Important distinction
|
|
|
|
These are not the same thing:
|
|
|
|
```text
|
|
Kubernetes Secret / OPENSEARCH_INITIAL_ADMIN_PASSWORD
|
|
!=
|
|
active OpenSearch Security index
|
|
```
|
|
|
|
For new demo-security initialization, `OPENSEARCH_INITIAL_ADMIN_PASSWORD` is used during setup. Once the Security configuration is initialized, changing an environment variable or mounted YAML file does not automatically rewrite the active security index.
|
|
|
|
## 10.2 Inspect effective internal user configuration
|
|
|
|
```bash
|
|
kubectl exec -n "$NS" "${OS_STS}-0" -c opensearch -- \
|
|
grep -A8 '^admin:' \
|
|
/usr/share/opensearch/config/opensearch-security/internal_users.yml
|
|
```
|
|
|
|
If the mounted file still contains a demo hash, it does not prove the intended password is active.
|
|
|
|
## 10.3 Generate a bcrypt hash
|
|
|
|
```bash
|
|
kubectl exec -it -n "$NS" "${OS_STS}-0" -c opensearch -- \
|
|
/usr/share/opensearch/plugins/opensearch-security/tools/hash.sh \
|
|
-p '<STRONG_NEW_PASSWORD>'
|
|
```
|
|
|
|
Avoid placing real passwords in shell history. Prefer an interactive/secure workflow where possible.
|
|
|
|
## 10.4 Create custom `internal_users.yml`
|
|
|
|
```yaml
|
|
---
|
|
_meta:
|
|
type: internalusers
|
|
config_version: 2
|
|
|
|
admin:
|
|
hash: "$2y$12$<GENERATED_BCRYPT_HASH>"
|
|
reserved: true
|
|
backend_roles:
|
|
- admin
|
|
description: OpenSearch administrator
|
|
```
|
|
|
|
## 10.5 Critical Secret key name
|
|
|
|
The chart expects the exact key:
|
|
|
|
```text
|
|
internal_users.yml
|
|
```
|
|
|
|
Not:
|
|
|
|
```text
|
|
internal_users.yaml
|
|
```
|
|
|
|
Create/update correctly even if your local filename is `.yaml`:
|
|
|
|
```bash
|
|
kubectl create secret generic opensearch-internal-users \
|
|
-n "$NS" \
|
|
--from-file=internal_users.yml=./internal_users.yaml \
|
|
--dry-run=client -o yaml | kubectl apply -f -
|
|
```
|
|
|
|
Verify only the key names:
|
|
|
|
```bash
|
|
kubectl get secret opensearch-internal-users \
|
|
-n "$NS" \
|
|
-o json | jq -r '.data | keys[]'
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
internal_users.yml
|
|
```
|
|
|
|
## 10.6 Helm security configuration
|
|
|
|
```yaml
|
|
securityConfig:
|
|
enabled: true
|
|
internalUsersSecret: opensearch-internal-users
|
|
```
|
|
|
|
Do not duplicate `securityConfig:` blocks.
|
|
|
|
## 10.7 Admin certificate mount
|
|
|
|
The admin certificate should be separate from normal node runtime TLS material.
|
|
|
|
```yaml
|
|
extraVolumes:
|
|
# existing volumes ...
|
|
- name: admin-certs
|
|
secret:
|
|
secretName: opensearch-admin-cert
|
|
defaultMode: 0440
|
|
|
|
extraVolumeMounts:
|
|
# existing mounts ...
|
|
- name: admin-certs
|
|
mountPath: /usr/share/opensearch/config/admin-certs
|
|
readOnly: true
|
|
```
|
|
|
|
With the chart's `fsGroup: 1000`, mode `0440` allows the non-root OpenSearch process to read these files through the pod group. Verify the effective UID/GID before changing this mode. A stricter production design is to mount the admin key only into a temporary administrative Job/Pod rather than every OpenSearch runtime pod.
|
|
|
|
## 10.8 Apply only internal users with `securityadmin.sh`
|
|
|
|
After the cluster is healthy and the admin certificate is trusted:
|
|
|
|
```bash
|
|
kubectl exec -it -n "$NS" "${OS_STS}-0" -c opensearch -- \
|
|
/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
|
|
-f /usr/share/opensearch/config/opensearch-security/internal_users.yml \
|
|
-t internalusers \
|
|
-icl \
|
|
-nhnv \
|
|
-cacert /usr/share/opensearch/config/admin-certs/root-ca.pem \
|
|
-cert /usr/share/opensearch/config/admin-certs/admin.pem \
|
|
-key /usr/share/opensearch/config/admin-certs/admin-key.pem
|
|
```
|
|
|
|
Using `-f` + `-t internalusers` is safer than unintentionally overwriting the complete Security configuration.
|
|
|
|
---
|
|
|
|
# 11. OpenSearch Cluster Verification
|
|
|
|
## 11.1 Kubernetes state
|
|
|
|
```bash
|
|
kubectl get sts -n "$NS"
|
|
kubectl get pods -n "$NS" -o wide
|
|
kubectl get pvc -n "$NS"
|
|
kubectl get svc -n "$NS"
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
3 OpenSearch pods
|
|
3 Bound PVCs
|
|
1 PVC per pod
|
|
pods spread across intended nodes
|
|
ClusterIP Service present
|
|
headless Service present
|
|
```
|
|
|
|
## 11.2 Init container logs
|
|
|
|
```bash
|
|
kubectl logs -n "$NS" "${OS_STS}-0" -c select-node-certificate
|
|
kubectl logs -n "$NS" "${OS_STS}-1" -c select-node-certificate
|
|
kubectl logs -n "$NS" "${OS_STS}-2" -c select-node-certificate
|
|
```
|
|
|
|
## 11.3 Runtime certificate files
|
|
|
|
```bash
|
|
kubectl exec -n "$NS" "${OS_STS}-0" -c opensearch -- \
|
|
ls -l /usr/share/opensearch/config/certs
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
node.pem
|
|
node-key.pem
|
|
root-ca.pem
|
|
```
|
|
|
|
## 11.4 OpenSearch logs
|
|
|
|
```bash
|
|
kubectl logs -n "$NS" "${OS_STS}-0" -c opensearch --tail=200
|
|
```
|
|
|
|
If a container restarted:
|
|
|
|
```bash
|
|
kubectl logs -n "$NS" "${OS_STS}-0" -c opensearch --previous --tail=200
|
|
```
|
|
|
|
Correct flag:
|
|
|
|
```text
|
|
--previous
|
|
```
|
|
|
|
not:
|
|
|
|
```text
|
|
--privious
|
|
```
|
|
|
|
## 11.5 Local diagnostic port-forward
|
|
|
|
```bash
|
|
kubectl port-forward -n "$NS" svc/"$OS_SVC" 9200:9200
|
|
```
|
|
|
|
Because the certificate may not contain `localhost`, `curl -k` is acceptable only as a **local diagnostic**:
|
|
|
|
```bash
|
|
curl -k -u 'admin:<PASSWORD>' \
|
|
'https://localhost:9200/_cluster/health?pretty'
|
|
|
|
curl -k -u 'admin:<PASSWORD>' \
|
|
'https://localhost:9200/_cat/nodes?v'
|
|
```
|
|
|
|
Do not interpret `-k` as production TLS validation.
|
|
|
|
## 11.6 Cluster health meaning
|
|
|
|
```text
|
|
GREEN -> all primary and replica shards allocated
|
|
YELLOW -> all primaries allocated; at least one replica unassigned
|
|
RED -> at least one primary shard unavailable
|
|
```
|
|
|
|
## 11.7 Replica count
|
|
|
|
OpenSearch defaults to one replica unless changed. A 3-node cluster **can** use two replicas to keep three shard copies, but this is a capacity/durability choice, not an automatic requirement.
|
|
|
|
Before setting `number_of_replicas: 2`, account for storage, indexing cost, and failure-domain design.
|
|
|
|
---
|
|
|
|
# 12. OpenSearch Dashboards Deployment
|
|
|
|
## 12.1 Architecture
|
|
|
|
```text
|
|
Browser
|
|
|
|
|
| HTTP :31599 /opensearch
|
|
v
|
|
NGINX Ingress
|
|
|
|
|
v
|
|
OpenSearch Dashboards Service :5601
|
|
|
|
|
v
|
|
Dashboards pod
|
|
|
|
|
| HTTPS :9200 + CA + hostname validation
|
|
v
|
|
OpenSearch Service
|
|
```
|
|
|
|
## 12.2 Verify OpenSearch Service certificate SAN
|
|
|
|
The hostname used by Dashboards must exist in the OpenSearch certificate SAN:
|
|
|
|
```text
|
|
brilliant-opensearch-cluster-master
|
|
```
|
|
|
|
A failure against `https://localhost:9200` such as:
|
|
|
|
```text
|
|
no alternative certificate subject name matches target hostname 'localhost'
|
|
```
|
|
|
|
means CA trust may be correct but hostname verification failed.
|
|
|
|
## 12.3 Root CA Secret for Dashboards
|
|
|
|
```bash
|
|
kubectl create secret generic opensearch-dashboard-ca \
|
|
-n "$NS" \
|
|
--from-file=root-ca.pem \
|
|
--dry-run=client -o yaml | kubectl apply -f -
|
|
```
|
|
|
|
## 12.4 `kibanaserver` password Secret
|
|
|
|
```bash
|
|
read -rsp 'kibanaserver password: ' KIBANASERVER_PASSWORD
|
|
echo
|
|
|
|
kubectl create secret generic opensearch-kibanaserver-password \
|
|
-n "$NS" \
|
|
--from-literal=KIBANASERVER_PASSWORD="$KIBANASERVER_PASSWORD" \
|
|
--dry-run=client -o yaml | kubectl apply -f -
|
|
|
|
unset KIBANASERVER_PASSWORD
|
|
```
|
|
|
|
The Secret object name and the Secret key are different concepts:
|
|
|
|
```text
|
|
secretKeyRef.name -> opensearch-kibanaserver-password
|
|
secretKeyRef.key -> KIBANASERVER_PASSWORD
|
|
```
|
|
|
|
## 12.5 Dashboards Helm values
|
|
|
|
```yaml
|
|
opensearchHosts: "https://brilliant-opensearch-cluster-master:9200"
|
|
|
|
extraEnvs:
|
|
- name: KIBANASERVER_PASSWORD
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: opensearch-kibanaserver-password
|
|
key: KIBANASERVER_PASSWORD
|
|
|
|
extraVolumes:
|
|
- name: root-cert
|
|
secret:
|
|
secretName: opensearch-dashboard-ca
|
|
|
|
extraVolumeMounts:
|
|
- name: root-cert
|
|
readOnly: true
|
|
mountPath: /usr/share/opensearch-dashboards/certs
|
|
|
|
config:
|
|
opensearch_dashboards.yml: |
|
|
server:
|
|
name: dashboards
|
|
host: "0.0.0.0"
|
|
basePath: /opensearch
|
|
rewriteBasePath: true
|
|
|
|
opensearch:
|
|
username: kibanaserver
|
|
password: ${KIBANASERVER_PASSWORD}
|
|
ssl:
|
|
verificationMode: full
|
|
certificateAuthorities:
|
|
- /usr/share/opensearch-dashboards/certs/root-ca.pem
|
|
|
|
opensearch_security.multitenancy.enabled: false
|
|
|
|
data_source:
|
|
enabled: true
|
|
|
|
workspace:
|
|
enabled: true
|
|
```
|
|
|
|
Optional feature keys such as `explore` and `datasetManagement` must be checked against the exact OpenSearch Dashboards version before enabling them.
|
|
|
|
## 12.6 Workspaces and multi-tenancy
|
|
|
|
When using Workspaces together with the OpenSearch Security plugin:
|
|
|
|
```yaml
|
|
opensearch_security.multitenancy.enabled: false
|
|
```
|
|
|
|
is required to avoid feature conflict.
|
|
|
|
## 12.7 `server.publicBaseUrl`
|
|
|
|
In the tested OpenSearch Dashboards 3.7.0 deployment, adding `server.publicBaseUrl` caused startup validation failure:
|
|
|
|
```text
|
|
"publicBaseUrl" is not allowed
|
|
```
|
|
|
|
Therefore it is intentionally absent from this SOP.
|
|
|
|
## 12.8 Render Dashboards before deploy
|
|
|
|
```bash
|
|
helm template brilliant-opensearch-dashboard <DASHBOARDS_CHART_PATH> \
|
|
-n "$NS" \
|
|
-f dashboards-values.yaml \
|
|
> rendered-dashboards.yaml
|
|
```
|
|
|
|
Inspect:
|
|
|
|
```bash
|
|
grep -nE 'KIBANASERVER_PASSWORD|root-cert|basePath|rewriteBasePath|verificationMode|5601' rendered-dashboards.yaml
|
|
```
|
|
|
|
## 12.9 Ingress conflict check
|
|
|
|
Before creating a hostless path, inspect existing Ingress objects:
|
|
|
|
```bash
|
|
kubectl get ingress -A
|
|
```
|
|
|
|
Grafana already used hostless `/` in the implementation. A second hostless `/` route under the same NGINX controller conflicted even though it was in another namespace.
|
|
|
|
Use `/opensearch` instead.
|
|
|
|
## 12.10 Dashboards Ingress
|
|
|
|
```yaml
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: Ingress
|
|
metadata:
|
|
name: opensearch-dashboards-ingress
|
|
namespace: observability
|
|
spec:
|
|
ingressClassName: nginx
|
|
rules:
|
|
- http:
|
|
paths:
|
|
- path: /opensearch
|
|
pathType: Prefix
|
|
backend:
|
|
service:
|
|
name: brilliant-opensearch-dashboard-opensearch-dashboards
|
|
port:
|
|
number: 5601
|
|
```
|
|
|
|
Apply:
|
|
|
|
```bash
|
|
kubectl apply -f opensearch-dashboards-ingress.yaml
|
|
```
|
|
|
|
## 12.11 Runtime verification
|
|
|
|
```bash
|
|
kubectl get deploy,pods,svc,ingress -n "$NS"
|
|
```
|
|
|
|
Current logs:
|
|
|
|
```bash
|
|
kubectl logs -n "$NS" deploy/brilliant-opensearch-dashboard-opensearch-dashboards --tail=200
|
|
```
|
|
|
|
Previous container logs after a restart:
|
|
|
|
```bash
|
|
kubectl logs -n "$NS" deploy/brilliant-opensearch-dashboard-opensearch-dashboards --previous --tail=200
|
|
```
|
|
|
|
Expected external URL pattern:
|
|
|
|
```text
|
|
http://<KUBERNETES-NODE-IP>:31599/opensearch
|
|
```
|
|
|
|
## 12.12 `401 -> 404` diagnostic meaning
|
|
|
|
During troubleshooting:
|
|
|
|
```text
|
|
unauthenticated /opensearch -> 401 Authentication required
|
|
valid authentication -> 404 Not Found
|
|
```
|
|
|
|
This indicated traffic was reaching Dashboards and authentication worked, while Dashboards did not yet recognize `/opensearch` as its base path. `server.basePath` + `rewriteBasePath` resolved that routing layer.
|
|
|
|
---
|
|
|
|
# 13. Data Prepper — Full Deployment, Routing, Parsing, and Verification
|
|
|
|
This chapter is the canonical Data Prepper procedure for this environment. It consolidates the Data Prepper material from the cPanel ingestion, multi-source routing, and Linux Audit SOPs.
|
|
|
|
## 13.1 Role in the architecture
|
|
|
|
```text
|
|
Fluent Bit
|
|
|
|
|
| HTTP POST /logs
|
|
v
|
|
NGINX Ingress / NodePort
|
|
|
|
|
v
|
|
Data Prepper HTTP source :2021
|
|
|
|
|
v
|
|
Router pipeline
|
|
|
|
|
+--> cPanel access pipeline
|
|
+--> system syslog pipeline
|
|
+--> Linux audit pipeline
|
|
+--> catch-all / unknown pipeline
|
|
|
|
|
v
|
|
OpenSearch HTTPS :9200
|
|
```
|
|
|
|
Data Prepper is responsible for receiving events, routing them by `log_source`, parsing/normalizing selected formats, and writing documents to the correct OpenSearch index.
|
|
|
|
## 13.2 Environment values used in this implementation
|
|
|
|
| Setting | Value used |
|
|
|---|---|
|
|
| Data Prepper version | `2.8.0` |
|
|
| Namespace | `observability` |
|
|
| Helm release | `data-prepper` |
|
|
| HTTP source port | `2021` |
|
|
| Server API / metrics port | `4900` |
|
|
| HTTP path | `/logs` |
|
|
| OpenSearch endpoint | `https://brilliant-opensearch-cluster-master:9200` |
|
|
| Root CA path inside Data Prepper | `/usr/share/data-prepper/cert/root-ca.pem` |
|
|
| cPanel access index | `cpanel-access-log-%{yyyy.MM.dd}` |
|
|
| Syslog index | `cpanel-system-syslog-%{yyyy.MM.dd}` |
|
|
| Audit index | `cpanel-audit-log-%{yyyy.MM.dd}` |
|
|
| Catch-all index | `cpanel-all-log-%{yyyy.MM.dd}` |
|
|
|
|
Verify the actual release and namespace before modifying anything:
|
|
|
|
```bash
|
|
helm list -n "$NS"
|
|
kubectl get deploy,svc,pod -n "$NS" | grep -i data-prepper
|
|
```
|
|
|
|
Expected Helm release:
|
|
|
|
```text
|
|
data-prepper
|
|
```
|
|
|
|
> [!WARNING]
|
|
> One troubleshooting incident installed/upgraded a second `data-prepper` release in the `default` namespace. Always specify `-n observability` / `--namespace observability`.
|
|
|
|
## 13.3 Back up the current Data Prepper configuration before changes
|
|
|
|
```bash
|
|
mkdir -p ~/backup/data-prepper
|
|
|
|
helm get values data-prepper -n "$NS" -a \
|
|
> ~/backup/data-prepper/values-$(date +%F-%H%M%S).yaml
|
|
|
|
helm get manifest data-prepper -n "$NS" \
|
|
> ~/backup/data-prepper/manifest-$(date +%F-%H%M%S).yaml
|
|
```
|
|
|
|
Also export the currently running pipeline/config objects if they are stored in ConfigMaps or Secrets:
|
|
|
|
```bash
|
|
kubectl get configmap -n "$NS" -o name | grep -i prepper
|
|
kubectl get secret -n "$NS" -o name | grep -i prepper
|
|
```
|
|
|
|
Do not dump Secret data into Git, screenshots, or terminal transcripts.
|
|
|
|
## 13.4 One HTTP source per listener port
|
|
|
|
Only one pipeline in a Data Prepper process should own HTTP port `2021`.
|
|
|
|
Incorrect design:
|
|
|
|
```text
|
|
access pipeline -> HTTP :2021
|
|
syslog pipeline -> HTTP :2021
|
|
audit pipeline -> HTTP :2021
|
|
```
|
|
|
|
Typical failure:
|
|
|
|
```text
|
|
bind(..) failed: Address already in use
|
|
```
|
|
|
|
Correct design:
|
|
|
|
```text
|
|
ONE HTTP source :2021
|
|
|
|
|
v
|
|
router pipeline
|
|
|
|
|
+--> access pipeline
|
|
+--> syslog pipeline
|
|
+--> audit pipeline
|
|
+--> catch-all pipeline
|
|
```
|
|
|
|
Downstream pipelines use a Data Prepper `pipeline` source, not another HTTP source.
|
|
|
|
## 13.5 Complete router pipeline
|
|
|
|
The normalized `log_source` field created by Fluent Bit is the routing key.
|
|
|
|
```yaml
|
|
version: "2"
|
|
|
|
cpanel-log-router:
|
|
source:
|
|
http:
|
|
path: /logs
|
|
ssl: false
|
|
port: 2021
|
|
health_check_service: true
|
|
unauthenticated_health_check: true
|
|
|
|
route:
|
|
- accessLog: '/log_source == "cpanel.access_log"'
|
|
- systemSyslog: '/log_source == "system.syslog"'
|
|
- auditLog: '/log_source == "audit.audit.log"'
|
|
- unhandled: '/log_source != "cpanel.access_log" and /log_source != "system.syslog" and /log_source != "audit.audit.log"'
|
|
|
|
sink:
|
|
- pipeline:
|
|
name: cpanel-access-pipeline
|
|
routes:
|
|
- accessLog
|
|
|
|
- pipeline:
|
|
name: cpanel-system-syslog-pipeline
|
|
routes:
|
|
- systemSyslog
|
|
|
|
- pipeline:
|
|
name: cpanel-audit-pipeline
|
|
routes:
|
|
- auditLog
|
|
|
|
- pipeline:
|
|
name: cpanel-catch-all-pipeline
|
|
routes:
|
|
- unhandled
|
|
```
|
|
|
|
### Why the `unhandled` route is explicit
|
|
|
|
A pipeline sink without a `routes:` restriction can receive all events. If a catch-all sink is left unqualified, dedicated events may also be copied into the catch-all path. The explicit `unhandled` condition prevents that duplication.
|
|
|
|
## 13.6 Complete cPanel access pipeline
|
|
|
|
The tested raw access format is:
|
|
|
|
```text
|
|
20.100.201.54 proxy - [08/09/2026:07:37:27 -0000] "GET /wp-content/themes/twentytwenty/404.php HTTP/1.1" 301 0 "-" "-" "-" "X-Forwarded-For: 20.100.201.54" 80
|
|
```
|
|
|
|
Pipeline:
|
|
|
|
```yaml
|
|
cpanel-access-pipeline:
|
|
workers: 2
|
|
delay: 100
|
|
|
|
source:
|
|
pipeline:
|
|
name: cpanel-log-router
|
|
|
|
processor:
|
|
- drop_events:
|
|
drop_when: '/log_source != "cpanel.access_log"'
|
|
|
|
- grok:
|
|
break_on_match: false
|
|
match:
|
|
log:
|
|
- '^%{IP:client_ip} %{NOTSPACE:identity} %{NOTSPACE:user} \[%{GREEDYDATA:timestamp}\] "%{WORD:method} %{URIPATHPARAM:url} HTTP/%{NUMBER:http_version}" %{INT:http_status:int} %{NONNEGINT:bytes:long} "%{NOTSPACE:referrer}" "%{GREEDYDATA:user_agent}" "%{NOTSPACE:auth_method}" "%{GREEDYDATA:x_forwarded_for}" %{POSINT:service_port:int}$'
|
|
target_key: parsed
|
|
|
|
- date:
|
|
match:
|
|
- key: /parsed/timestamp
|
|
patterns:
|
|
- "MM/dd/yyyy:HH:mm:ss Z"
|
|
destination: "@timestamp"
|
|
|
|
sink:
|
|
- opensearch:
|
|
hosts:
|
|
- "https://brilliant-opensearch-cluster-master:9200"
|
|
insecure: false
|
|
username: logger
|
|
password: "<SECRET-MANAGED-VALUE>"
|
|
cert: /usr/share/data-prepper/cert/root-ca.pem
|
|
index_type: custom
|
|
index: cpanel-access-log-%{yyyy.MM.dd}
|
|
```
|
|
|
|
Expected parsed fields:
|
|
|
|
```text
|
|
parsed.client_ip
|
|
parsed.identity
|
|
parsed.user
|
|
parsed.timestamp
|
|
parsed.method
|
|
parsed.url
|
|
parsed.http_version
|
|
parsed.http_status
|
|
parsed.bytes
|
|
parsed.referrer
|
|
parsed.user_agent
|
|
parsed.auth_method
|
|
parsed.x_forwarded_for
|
|
parsed.service_port
|
|
@timestamp
|
|
```
|
|
|
|
### Important Grok compatibility issue from the implementation
|
|
|
|
The tested Data Prepper 2.8.0 setup failed when this was used:
|
|
|
|
```text
|
|
%{PORT:service_port}
|
|
```
|
|
|
|
with:
|
|
|
|
```text
|
|
No definition for key 'PORT' found, aborting
|
|
```
|
|
|
|
The working replacement was:
|
|
|
|
```text
|
|
%{POSINT:service_port}
|
|
```
|
|
|
|
## 13.7 Syslog pipeline
|
|
|
|
Syslog records observed in the environment included both:
|
|
|
|
```text
|
|
2026-08-11T10:42:39.679625+00:00 bwebhosting kernel: Firewall: ...
|
|
```
|
|
|
|
and:
|
|
|
|
```text
|
|
2026-08-11T10:42:39.434068+00:00 bwebhosting fluent-bit[1241036]: ...
|
|
```
|
|
|
|
The process token therefore needs to tolerate both a plain process name and a process name with an optional PID.
|
|
|
|
A safe baseline is to preserve raw data while the exact parser is being tuned:
|
|
|
|
```yaml
|
|
cpanel-system-syslog-pipeline:
|
|
source:
|
|
pipeline:
|
|
name: cpanel-log-router
|
|
|
|
processor:
|
|
- drop_events:
|
|
drop_when: '/log_source != "system.syslog"'
|
|
|
|
sink:
|
|
- opensearch:
|
|
hosts:
|
|
- "https://brilliant-opensearch-cluster-master:9200"
|
|
insecure: false
|
|
username: logger
|
|
password: "<SECRET-MANAGED-VALUE>"
|
|
cert: /usr/share/data-prepper/cert/root-ca.pem
|
|
index_type: custom
|
|
index: cpanel-system-syslog-%{yyyy.MM.dd}
|
|
```
|
|
|
|
Add Grok only after representative samples are verified. If a document reaches OpenSearch with `parsed: {}`, transport/routing may be healthy while the Grok expression is not matching.
|
|
|
|
## 13.8 Linux Audit pipeline
|
|
|
|
Audit records are multi-type and are handled in the dedicated Linux Audit chapter of this SOP. The router must send only:
|
|
|
|
```text
|
|
log_source = audit.audit.log
|
|
```
|
|
|
|
to:
|
|
|
|
```text
|
|
cpanel-audit-pipeline
|
|
```
|
|
|
|
The audit pipeline should additionally protect itself with:
|
|
|
|
```yaml
|
|
- drop_events:
|
|
drop_when: '/log_source != "audit.audit.log"'
|
|
```
|
|
|
|
The full Audit Grok, date, `EXECVE`, `SYSCALL`, `LOGIN`, `USER_ACCT`, `USER_START`, `CRED_ACQ`, and `CWD` parsing procedure is documented in Section 17.
|
|
|
|
## 13.9 Catch-all pipeline
|
|
|
|
Use the catch-all to retain unknown/new sources without forcing every source into a dedicated parser on day one.
|
|
|
|
```yaml
|
|
cpanel-catch-all-pipeline:
|
|
source:
|
|
pipeline:
|
|
name: cpanel-log-router
|
|
|
|
sink:
|
|
- opensearch:
|
|
hosts:
|
|
- "https://brilliant-opensearch-cluster-master:9200"
|
|
insecure: false
|
|
username: logger
|
|
password: "<SECRET-MANAGED-VALUE>"
|
|
cert: /usr/share/data-prepper/cert/root-ca.pem
|
|
index_type: custom
|
|
index: cpanel-all-log-%{yyyy.MM.dd}
|
|
```
|
|
|
|
Recommended model:
|
|
|
|
```text
|
|
Known/high-value source
|
|
-> dedicated route
|
|
-> dedicated parser
|
|
-> dedicated index
|
|
|
|
Unknown/new source
|
|
-> catch-all
|
|
-> raw visibility
|
|
-> inspect
|
|
-> promote later if required
|
|
```
|
|
|
|
## 13.10 OpenSearch Root CA mount
|
|
|
|
Data Prepper must trust the CA that signed the OpenSearch HTTP certificate.
|
|
|
|
Conceptual Kubernetes volume:
|
|
|
|
```yaml
|
|
volumes:
|
|
- name: opensearch-root-ca
|
|
secret:
|
|
secretName: opensearch-dashboard-ca
|
|
```
|
|
|
|
Conceptual container mount:
|
|
|
|
```yaml
|
|
volumeMounts:
|
|
- name: opensearch-root-ca
|
|
mountPath: /usr/share/data-prepper/cert
|
|
readOnly: true
|
|
```
|
|
|
|
Verify inside the pod:
|
|
|
|
```bash
|
|
kubectl exec -n "$NS" deploy/data-prepper -- \
|
|
ls -l /usr/share/data-prepper/cert/root-ca.pem
|
|
```
|
|
|
|
Do not set `insecure: true` merely to bypass a certificate problem. Correct the CA/SAN/TLS configuration instead.
|
|
|
|
## 13.11 Data Prepper credential handling
|
|
|
|
The source SOP used an OpenSearch user named:
|
|
|
|
```text
|
|
logger
|
|
```
|
|
|
|
The real password must not be committed to Git.
|
|
|
|
Required security model:
|
|
|
|
```text
|
|
Data Prepper identity
|
|
-> authenticate to OpenSearch
|
|
-> only required write/create-index permissions
|
|
-> no admin/security-management permissions
|
|
```
|
|
|
|
If the Helm chart or deployment mechanism stores `pipelines.yaml` in a Secret, keep the sensitive pipeline configuration in that Secret or use the secret-management mechanism supported by the deployment. Do not assume arbitrary environment-variable interpolation in every Data Prepper configuration field without testing the exact deployed version.
|
|
|
|
## 13.12 Kubernetes Service
|
|
|
|
Expected Service path:
|
|
|
|
```text
|
|
Ingress
|
|
-> data-prepper Service :2021
|
|
-> Data Prepper Pod :2021
|
|
```
|
|
|
|
Service example:
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: data-prepper
|
|
namespace: observability
|
|
spec:
|
|
type: ClusterIP
|
|
selector:
|
|
app: data-prepper
|
|
ports:
|
|
- name: ingest
|
|
port: 2021
|
|
targetPort: 2021
|
|
- name: metrics
|
|
port: 4900
|
|
targetPort: 4900
|
|
```
|
|
|
|
The selector is chart/deployment-specific. Verify that it matches the actual pod labels:
|
|
|
|
```bash
|
|
kubectl get pod -n "$NS" --show-labels | grep data-prepper
|
|
kubectl get svc data-prepper -n "$NS" -o yaml
|
|
```
|
|
|
|
Verify the backend endpoint:
|
|
|
|
```bash
|
|
kubectl get endpointslice \
|
|
-n "$NS" \
|
|
-l kubernetes.io/service-name=data-prepper \
|
|
-o wide
|
|
```
|
|
|
|
Expected result: the current Data Prepper pod IP and port `2021` are present.
|
|
|
|
## 13.13 NGINX Ingress
|
|
|
|
The implementation used:
|
|
|
|
```text
|
|
http://118.67.213.76:31599/logs
|
|
```
|
|
|
|
Flow:
|
|
|
|
```text
|
|
118.67.213.76:31599
|
|
-> ingress-nginx controller :80
|
|
-> prepper-ingress /logs
|
|
-> data-prepper:2021
|
|
```
|
|
|
|
Ingress:
|
|
|
|
```yaml
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: Ingress
|
|
metadata:
|
|
name: prepper-ingress
|
|
namespace: observability
|
|
spec:
|
|
ingressClassName: nginx
|
|
rules:
|
|
- http:
|
|
paths:
|
|
- path: /logs
|
|
pathType: Prefix
|
|
backend:
|
|
service:
|
|
name: data-prepper
|
|
port:
|
|
number: 2021
|
|
```
|
|
|
|
Apply and verify:
|
|
|
|
```bash
|
|
kubectl apply -f prepper-ingress.yaml
|
|
kubectl get ingress prepper-ingress -n "$NS" -o wide
|
|
kubectl describe ingress prepper-ingress -n "$NS"
|
|
```
|
|
|
|
## 13.14 Data Prepper Helm upgrade procedure
|
|
|
|
The exact chart path depends on how Data Prepper was installed. Use the real chart reference/path instead of guessing.
|
|
|
|
```bash
|
|
export DP_RELEASE="data-prepper"
|
|
export DP_CHART="<DATA_PREPPER_CHART_OR_LOCAL_PATH>"
|
|
export DP_VALUES="<DATA_PREPPER_VALUES_FILE>"
|
|
```
|
|
|
|
Confirm the current release first:
|
|
|
|
```bash
|
|
helm list -n "$NS"
|
|
```
|
|
|
|
Render before upgrading:
|
|
|
|
```bash
|
|
helm template "$DP_RELEASE" "$DP_CHART" \
|
|
-n "$NS" \
|
|
-f "$DP_VALUES" \
|
|
> /tmp/data-prepper-rendered.yaml
|
|
```
|
|
|
|
Inspect the important items:
|
|
|
|
```bash
|
|
grep -nE 'kind: (Deployment|Service|ConfigMap|Secret)|2021|4900|pipelines|data-prepper' \
|
|
/tmp/data-prepper-rendered.yaml | head -200
|
|
```
|
|
|
|
Then upgrade the correct release in the correct namespace:
|
|
|
|
```bash
|
|
helm upgrade "$DP_RELEASE" "$DP_CHART" \
|
|
-n "$NS" \
|
|
-f "$DP_VALUES"
|
|
```
|
|
|
|
Watch rollout:
|
|
|
|
```bash
|
|
kubectl get pods -n "$NS" -w
|
|
```
|
|
|
|
## 13.15 Pod and listener verification
|
|
|
|
```bash
|
|
kubectl get pods -n "$NS" | grep data-prepper
|
|
kubectl logs -n "$NS" deploy/data-prepper --tail=200
|
|
```
|
|
|
|
Expected health:
|
|
|
|
```text
|
|
READY: 1/1
|
|
STATUS: Running
|
|
```
|
|
|
|
If `ss`/`netstat` do not exist in the image, `/proc/net/tcp` can verify the listener.
|
|
|
|
For port `2021`:
|
|
|
|
```text
|
|
2021 decimal = 07E5 hexadecimal
|
|
0A = LISTEN
|
|
01 = ESTABLISHED
|
|
```
|
|
|
|
Example:
|
|
|
|
```bash
|
|
kubectl exec -n "$NS" deploy/data-prepper -- \
|
|
sh -c 'grep -i ":07E5" /proc/net/tcp /proc/net/tcp6 || true'
|
|
```
|
|
|
|
## 13.16 Health endpoint and metrics
|
|
|
|
When the HTTP source health service is enabled:
|
|
|
|
```bash
|
|
kubectl run dp-curl --rm -i --restart=Never \
|
|
-n "$NS" \
|
|
--image=curlimages/curl:latest \
|
|
-- \
|
|
curl -sS http://data-prepper:2021/health
|
|
```
|
|
|
|
Data Prepper server/metrics API:
|
|
|
|
```bash
|
|
kubectl exec -n "$NS" deploy/data-prepper -- \
|
|
curl -s http://127.0.0.1:4900/metrics/prometheus | head -100
|
|
```
|
|
|
|
Useful metric families from the audit/cPanel work include:
|
|
|
|
```text
|
|
grokProcessingMatch
|
|
grokProcessingMismatch
|
|
grokProcessingErrors
|
|
dateProcessingMatchSuccess
|
|
dateProcessingMatchFailure
|
|
documentsSuccess
|
|
documentErrors
|
|
```
|
|
|
|
Pipeline/plugin names can be normalized into the full metric name.
|
|
|
|
## 13.17 Controlled event test inside Kubernetes
|
|
|
|
The HTTP source accepts JSON events. Use a controlled event before involving Fluent Bit:
|
|
|
|
```bash
|
|
kubectl run dp-curl --rm -i --restart=Never \
|
|
-n "$NS" \
|
|
--image=curlimages/curl:latest \
|
|
-- \
|
|
curl -sS -X POST http://data-prepper:2021/logs \
|
|
-H 'Content-Type: application/json' \
|
|
--data '[{"log":"test","source_file":"manual-test","log_source":"test.manual"}]'
|
|
```
|
|
|
|
Then verify that the catch-all path receives it.
|
|
|
|
## 13.18 Controlled event test through NGINX
|
|
|
|
From a host that can reach the NGINX NodePort:
|
|
|
|
```bash
|
|
curl -i -X POST \
|
|
'http://118.67.213.76:31599/logs' \
|
|
-H 'Content-Type: application/json' \
|
|
--data '[{"log":"ingress-test","source_file":"manual-test","log_source":"test.manual"}]'
|
|
```
|
|
|
|
A successful test proves:
|
|
|
|
```text
|
|
client
|
|
-> NodePort
|
|
-> NGINX
|
|
-> Ingress rule
|
|
-> Service
|
|
-> EndpointSlice
|
|
-> Data Prepper :2021
|
|
```
|
|
|
|
## 13.19 OpenSearch sink verification
|
|
|
|
Do not use index existence alone as proof of successful ingestion.
|
|
|
|
Example:
|
|
|
|
```text
|
|
green open cpanel-access-log-2026.08.09 ... docs.count=0
|
|
```
|
|
|
|
means the index exists but has no documents.
|
|
|
|
Check index list:
|
|
|
|
```bash
|
|
curl --cacert <ROOT_CA> -u '<USER>:<PASSWORD>' \
|
|
'https://<OPENSEARCH>:9200/_cat/indices/cpanel-*?v'
|
|
```
|
|
|
|
Check document count:
|
|
|
|
```bash
|
|
curl --cacert <ROOT_CA> -u '<USER>:<PASSWORD>' \
|
|
'https://<OPENSEARCH>:9200/cpanel-access-log-*/_count?pretty'
|
|
```
|
|
|
|
Check parsed fields:
|
|
|
|
```bash
|
|
curl --cacert <ROOT_CA> -u '<USER>:<PASSWORD>' \
|
|
-H 'Content-Type: application/json' \
|
|
'https://<OPENSEARCH>:9200/cpanel-access-log-*/_search?pretty' \
|
|
-d '{
|
|
"size": 5,
|
|
"sort": [{"@timestamp": "desc"}],
|
|
"_source": ["@timestamp","log_source","source_file","log","parsed"]
|
|
}'
|
|
```
|
|
|
|
Interpretation:
|
|
|
|
```text
|
|
log exists + parsed exists -> parser matched
|
|
log exists + parsed missing -> parser did not match / processor issue
|
|
index exists + docs.count = 0 -> no successful document insertion
|
|
```
|
|
|
|
## 13.20 `502 Bad Gateway` troubleshooting
|
|
|
|
Fluent Bit previously reported:
|
|
|
|
```text
|
|
HTTP status=502
|
|
502 Bad Gateway
|
|
nginx
|
|
```
|
|
|
|
This means Fluent Bit reached NGINX, but NGINX did not have a working upstream path to Data Prepper.
|
|
|
|
Check in this order:
|
|
|
|
```bash
|
|
kubectl get pod -n "$NS" | grep data-prepper
|
|
kubectl get svc data-prepper -n "$NS" -o wide
|
|
kubectl get endpointslice -n "$NS" \
|
|
-l kubernetes.io/service-name=data-prepper -o wide
|
|
kubectl logs -n "$NS" deploy/data-prepper --tail=200
|
|
kubectl describe ingress prepper-ingress -n "$NS"
|
|
```
|
|
|
|
Do not change Fluent Bit Grok/parsing when NGINX is returning a backend/upstream error.
|
|
|
|
## 13.21 OpenSearch sink failure troubleshooting
|
|
|
|
Observed sink errors included:
|
|
|
|
```text
|
|
Failed to initialize OpenSearch sink with a retryable exception
|
|
Request execution cancelled
|
|
I/O session is closed
|
|
```
|
|
|
|
Troubleshoot this path independently:
|
|
|
|
```text
|
|
Data Prepper
|
|
-> Kubernetes DNS
|
|
-> brilliant-opensearch-cluster-master
|
|
-> TCP 9200
|
|
-> TLS handshake
|
|
-> Root CA validation
|
|
-> hostname/SAN validation
|
|
-> logger authentication
|
|
-> logger authorization
|
|
-> index write
|
|
```
|
|
|
|
Do not modify Grok while the failure is DNS/TCP/TLS/authentication/authorization at the sink.
|
|
|
|
## 13.22 YAML and processor hierarchy checks
|
|
|
|
Correct Grok hierarchy:
|
|
|
|
```yaml
|
|
processor:
|
|
- grok:
|
|
break_on_match: false
|
|
match:
|
|
log:
|
|
- '<GROK_PATTERN>'
|
|
target_key: parsed
|
|
```
|
|
|
|
A prior indentation mistake caused Data Prepper to interpret options such as `false` at the wrong level and fail pipeline construction.
|
|
|
|
Validate indentation and quoting before every Helm upgrade.
|
|
|
|
## 13.23 Production hardening for Data Prepper
|
|
|
|
The original implementation used:
|
|
|
|
```yaml
|
|
ssl: false
|
|
```
|
|
|
|
on the Data Prepper HTTP source and exposed it through an NGINX NodePort. Before considering the ingestion endpoint production-hardened, use the controls appropriate to the environment:
|
|
|
|
```text
|
|
TLS on the ingestion path
|
|
HTTP authentication where supported/configured
|
|
private network routing
|
|
firewall/source-IP restrictions
|
|
NetworkPolicy
|
|
Ingress authentication / mTLS where appropriate
|
|
least-privilege OpenSearch sink identity
|
|
persistent DLQ if required
|
|
resource limits
|
|
metrics and alerting
|
|
```
|
|
|
|
The endpoint receives security-sensitive data including IP addresses, usernames, URLs, audit records, processes, and commands. Protect access and retention accordingly.
|
|
|
|
---
|
|
|
|
# 14. Fluent Bit — Full cPanel/Linux Collection Procedure
|
|
|
|
This chapter is the canonical Fluent Bit procedure for the cPanel/Linux sources in this project. It includes the access-only baseline, the expanded multi-source configuration, source classification, persistent offset state, filesystem buffering, verification, and outage-recovery testing.
|
|
|
|
## 14.1 Role in the architecture
|
|
|
|
```text
|
|
cPanel/Linux log files
|
|
|
|
|
v
|
|
Fluent Bit Tail inputs
|
|
|
|
|
+--> persistent Tail DB
|
|
+--> filesystem buffer
|
|
+--> source_file
|
|
+--> Lua log_source classification
|
|
|
|
|
v
|
|
HTTP POST /logs
|
|
|
|
|
v
|
|
NGINX -> Data Prepper
|
|
```
|
|
|
|
Fluent Bit must:
|
|
|
|
- Continuously tail the required log files.
|
|
- Preserve the original path as `source_file`.
|
|
- Maintain independent Tail DB state for each input.
|
|
- Use filesystem buffering for undelivered chunks.
|
|
- Normalize the source into a stable `log_source` value.
|
|
- Send JSON to Data Prepper.
|
|
- Retry transient failures.
|
|
- Resume correctly after service/server restarts.
|
|
|
|
## 14.2 Source inventory used in the project
|
|
|
|
| Category | Example path | Expected `log_source` |
|
|
|---|---|---|
|
|
| cPanel access | `/usr/local/cpanel/logs/access_log` | `cpanel.access_log` |
|
|
| Syslog | `/var/log/syslog` | `system.syslog` |
|
|
| Kernel | `/var/log/kern.log` | `system.kern.log` |
|
|
| Exim | `/var/log/exim_mainlog` | `system.exim_mainlog` |
|
|
| Auditd | `/var/log/audit/audit.log` | `audit.audit.log` |
|
|
| Apache | `/var/log/apache2/...` | `apache.<filename>` |
|
|
|
|
Before configuration, verify which files actually exist on the server:
|
|
|
|
```bash
|
|
for f in \
|
|
/usr/local/cpanel/logs/access_log \
|
|
/var/log/syslog \
|
|
/var/log/kern.log \
|
|
/var/log/exim_mainlog \
|
|
/var/log/audit/audit.log; do
|
|
[ -e "$f" ] && ls -lh "$f"
|
|
done
|
|
```
|
|
|
|
Inspect representative records before writing parsers:
|
|
|
|
```bash
|
|
tail -n 20 /usr/local/cpanel/logs/access_log
|
|
sudo tail -n 20 /var/log/audit/audit.log
|
|
```
|
|
|
|
## 14.3 Optional installation verification
|
|
|
|
The source SOPs started from an already installed Fluent Bit service. If deploying on a new cPanel host, use the package procedure for the actual OS and pin/test the version according to your change policy.
|
|
|
|
First identify the OS:
|
|
|
|
```bash
|
|
cat /etc/os-release
|
|
```
|
|
|
|
After installation, the service must exist:
|
|
|
|
```bash
|
|
systemctl status fluent-bit --no-pager
|
|
fluent-bit --version
|
|
```
|
|
|
|
Do not replace a working production package with a newer one during an ingestion troubleshooting session unless the upgrade itself is planned and tested.
|
|
|
|
## 14.4 Service account and permissions
|
|
|
|
Determine the service account:
|
|
|
|
```bash
|
|
systemctl cat fluent-bit | grep -E '^(User|Group)=' || true
|
|
ps -eo user,group,pid,cmd | grep '[f]luent-bit'
|
|
```
|
|
|
|
The Fluent Bit process needs read permission on every selected source file and write permission to the Tail DB/buffer directories.
|
|
|
|
For audit logs, verify access explicitly:
|
|
|
|
```bash
|
|
namei -l /var/log/audit/audit.log
|
|
```
|
|
|
|
Do not run Fluent Bit as root merely to bypass a permissions problem unless that is the deliberately approved security model.
|
|
|
|
## 14.5 Persistent state and buffer directories
|
|
|
|
Recommended separation:
|
|
|
|
```text
|
|
/var/lib/fluent-bit/
|
|
├── state/
|
|
│ ├── cpanel-access.db
|
|
│ ├── syslog.db
|
|
│ ├── kern.db
|
|
│ ├── exim.db
|
|
│ └── audit.db
|
|
└── storage/
|
|
└── filesystem chunks
|
|
```
|
|
|
|
Create the directories using the actual Fluent Bit service account:
|
|
|
|
```bash
|
|
install -d -m 0750 -o fluent-bit -g fluent-bit /var/lib/fluent-bit/state
|
|
install -d -m 0750 -o fluent-bit -g fluent-bit /var/lib/fluent-bit/storage
|
|
```
|
|
|
|
If the service uses another user/group, replace `fluent-bit:fluent-bit` accordingly.
|
|
|
|
Keep these directories outside any broad Tail wildcard so Fluent Bit does not ingest its own SQLite/WAL or buffer files.
|
|
|
|
## 14.6 Access-log-only baseline configuration
|
|
|
|
Use this first when proving the cPanel access path end to end:
|
|
|
|
```ini
|
|
[SERVICE]
|
|
Flush 1
|
|
Daemon Off
|
|
Log_Level info
|
|
|
|
Parsers_File parsers.conf
|
|
Plugins_File plugins.conf
|
|
|
|
storage.metrics on
|
|
storage.path /var/lib/fluent-bit/storage
|
|
storage.sync full
|
|
storage.checksum on
|
|
storage.backlog.mem_limit 500M
|
|
|
|
[INPUT]
|
|
Name tail
|
|
Path /usr/local/cpanel/logs/access_log
|
|
Path_Key source_file
|
|
Tag access_log
|
|
|
|
DB /var/lib/fluent-bit/state/cpanel-access.db
|
|
DB.Journal_Mode wal
|
|
|
|
Refresh_Interval 5
|
|
Skip_Long_Lines true
|
|
Mem_Buf_Limit 512MB
|
|
Storage.Type filesystem
|
|
|
|
[FILTER]
|
|
Name modify
|
|
Match access_log
|
|
Add log_source cpanel.access_log
|
|
|
|
[OUTPUT]
|
|
Name http
|
|
Host 118.67.213.76
|
|
Port 31599
|
|
Format json
|
|
HTTP_Method POST
|
|
URI /logs
|
|
Match access_log
|
|
Retry_Limit no_limits
|
|
Storage.Total_Limit_Size 4G
|
|
```
|
|
|
|
### Notes
|
|
|
|
`DB` stores file identity/offset state. It is not the durable event queue.
|
|
|
|
`Storage.Type filesystem` allows chunks to be backed by the filesystem storage layer.
|
|
|
|
`Storage.Total_Limit_Size 4G` caps the logical filesystem queue for this output. If the queue fills, old queued chunks can be discarded to make room for new data; monitor it.
|
|
|
|
`Retry_Limit no_limits` avoids abandoning a failed chunk merely because a finite retry count was reached. This still does not make local disk capacity infinite.
|
|
|
|
## 14.7 Historical ingestion behavior
|
|
|
|
For a file with no existing Tail DB state:
|
|
|
|
```ini
|
|
Read_from_Head true
|
|
```
|
|
|
|
means read from the beginning.
|
|
|
|
Without it, a new input normally begins at the end of the file and follows new records.
|
|
|
|
Use historical backfill intentionally. Deleting/replacing a Tail DB and then setting `Read_from_Head true` can re-ingest already indexed data.
|
|
|
|
For static backfill testing, use a copy of the source file and a dedicated DB/index whenever possible.
|
|
|
|
## 14.8 Multi-source Tail configuration
|
|
|
|
After the access-only path is proven, expand to the required sources. Each input should have a unique DB.
|
|
|
|
```ini
|
|
[SERVICE]
|
|
Flush 1
|
|
Daemon Off
|
|
Log_Level info
|
|
|
|
Parsers_File parsers.conf
|
|
Plugins_File plugins.conf
|
|
|
|
storage.metrics on
|
|
storage.path /var/lib/fluent-bit/storage
|
|
storage.sync full
|
|
storage.checksum on
|
|
storage.backlog.mem_limit 500M
|
|
|
|
[INPUT]
|
|
Name tail
|
|
Path /usr/local/cpanel/logs/access_log
|
|
Path_Key source_file
|
|
Tag cpanel.files
|
|
DB /var/lib/fluent-bit/state/cpanel-access.db
|
|
DB.Journal_Mode wal
|
|
Refresh_Interval 5
|
|
Skip_Long_Lines true
|
|
Mem_Buf_Limit 512MB
|
|
Storage.Type filesystem
|
|
|
|
[INPUT]
|
|
Name tail
|
|
Path /var/log/syslog
|
|
Path_Key source_file
|
|
Tag cpanel.files
|
|
DB /var/lib/fluent-bit/state/syslog.db
|
|
DB.Journal_Mode wal
|
|
Refresh_Interval 5
|
|
Skip_Long_Lines true
|
|
Mem_Buf_Limit 256MB
|
|
Storage.Type filesystem
|
|
|
|
[INPUT]
|
|
Name tail
|
|
Path /var/log/kern.log
|
|
Path_Key source_file
|
|
Tag cpanel.files
|
|
DB /var/lib/fluent-bit/state/kern.db
|
|
DB.Journal_Mode wal
|
|
Refresh_Interval 5
|
|
Skip_Long_Lines true
|
|
Mem_Buf_Limit 256MB
|
|
Storage.Type filesystem
|
|
|
|
[INPUT]
|
|
Name tail
|
|
Path /var/log/exim_mainlog
|
|
Path_Key source_file
|
|
Tag cpanel.files
|
|
DB /var/lib/fluent-bit/state/exim.db
|
|
DB.Journal_Mode wal
|
|
Refresh_Interval 5
|
|
Skip_Long_Lines true
|
|
Mem_Buf_Limit 256MB
|
|
Storage.Type filesystem
|
|
|
|
[INPUT]
|
|
Name tail
|
|
Path /var/log/audit/audit.log
|
|
Path_Key source_file
|
|
Tag cpanel.files
|
|
DB /var/lib/fluent-bit/state/audit.db
|
|
DB.Journal_Mode wal
|
|
Refresh_Interval 5
|
|
Skip_Long_Lines true
|
|
Mem_Buf_Limit 512MB
|
|
Storage.Type filesystem
|
|
|
|
[INPUT]
|
|
Name tail
|
|
Path /var/log/apache2/*.log
|
|
Path_Key source_file
|
|
Tag cpanel.files
|
|
DB /var/lib/fluent-bit/state/apache.db
|
|
DB.Journal_Mode wal
|
|
Refresh_Interval 5
|
|
Skip_Long_Lines true
|
|
Mem_Buf_Limit 512MB
|
|
Storage.Type filesystem
|
|
|
|
[FILTER]
|
|
Name lua
|
|
Match cpanel.files
|
|
Script /etc/fluent-bit/classify.lua
|
|
Call classify
|
|
|
|
[OUTPUT]
|
|
Name http
|
|
Host 118.67.213.76
|
|
Port 31599
|
|
Format json
|
|
HTTP_Method POST
|
|
URI /logs
|
|
Match cpanel.files
|
|
Retry_Limit no_limits
|
|
Storage.Total_Limit_Size 4G
|
|
```
|
|
|
|
Only keep input paths that actually exist and are required. If Apache/cPanel stores logs elsewhere on a specific host, use the real path.
|
|
|
|
## 14.9 `source_file` and `log_source`
|
|
|
|
A Tail event should preserve its file:
|
|
|
|
```json
|
|
{
|
|
"log": "...",
|
|
"source_file": "/var/log/syslog"
|
|
}
|
|
```
|
|
|
|
The Lua filter converts that into a stable routing field:
|
|
|
|
```json
|
|
{
|
|
"log": "...",
|
|
"source_file": "/var/log/syslog",
|
|
"log_source": "system.syslog"
|
|
}
|
|
```
|
|
|
|
Data Prepper routes on `log_source`, not on assumptions derived from the destination index name.
|
|
|
|
## 14.10 Lua source-classification script
|
|
|
|
Create:
|
|
|
|
```text
|
|
/etc/fluent-bit/classify.lua
|
|
```
|
|
|
|
Content:
|
|
|
|
```lua
|
|
function classify(tag, timestamp, record)
|
|
local path = record["source_file"] or ""
|
|
|
|
if path == "/usr/local/cpanel/logs/access_log" then
|
|
record["log_source"] = "cpanel.access_log"
|
|
elseif path == "/var/log/syslog" then
|
|
record["log_source"] = "system.syslog"
|
|
elseif path == "/var/log/kern.log" then
|
|
record["log_source"] = "system.kern.log"
|
|
elseif path == "/var/log/exim_mainlog" then
|
|
record["log_source"] = "system.exim_mainlog"
|
|
elseif path == "/var/log/audit/audit.log" then
|
|
record["log_source"] = "audit.audit.log"
|
|
elseif string.match(path, "^/var/log/apache2/") then
|
|
local filename = string.match(path, "([^/]+)$") or "unknown"
|
|
record["log_source"] = "apache." .. filename
|
|
else
|
|
record["log_source"] = "unknown"
|
|
end
|
|
|
|
return 1, timestamp, record
|
|
end
|
|
```
|
|
|
|
Set ownership/permissions appropriate to the service:
|
|
|
|
```bash
|
|
chown root:root /etc/fluent-bit/classify.lua
|
|
chmod 0644 /etc/fluent-bit/classify.lua
|
|
```
|
|
|
|
The earlier implementation attempted inline Lua in the classic configuration and Fluent Bit failed with:
|
|
|
|
```text
|
|
extra indentation level found
|
|
invalid indentation level
|
|
```
|
|
|
|
Keeping Lua in a separate file avoids that configuration-indentation problem and is easier to test/version.
|
|
|
|
## 14.11 Validate configuration before restart
|
|
|
|
If supported by the installed Fluent Bit version/build:
|
|
|
|
```bash
|
|
fluent-bit --dry-run -c /etc/fluent-bit/fluent-bit.conf
|
|
```
|
|
|
|
If the binary is not in `PATH`:
|
|
|
|
```bash
|
|
command -v fluent-bit
|
|
systemctl cat fluent-bit
|
|
```
|
|
|
|
Then restart:
|
|
|
|
```bash
|
|
systemctl restart fluent-bit
|
|
systemctl status fluent-bit --no-pager
|
|
journalctl -u fluent-bit --since '5 minutes ago' --no-pager
|
|
```
|
|
|
|
Healthy state should not show sustained:
|
|
|
|
```text
|
|
configuration parser errors
|
|
502 Bad Gateway
|
|
connection refused
|
|
failed to flush
|
|
chunk retry exhaustion
|
|
```
|
|
|
|
## 14.12 Prove the source is generating data
|
|
|
|
For cPanel access logs, generate or wait for a request and confirm the file changes:
|
|
|
|
```bash
|
|
tail -Fn0 /usr/local/cpanel/logs/access_log
|
|
```
|
|
|
|
For audit:
|
|
|
|
```bash
|
|
sudo tail -Fn0 /var/log/audit/audit.log
|
|
```
|
|
|
|
If no fresh event exists in the source file, do not troubleshoot Data Prepper/OpenSearch yet.
|
|
|
|
## 14.13 Verify Tail DB creation and persistence
|
|
|
|
```bash
|
|
ls -lh /var/lib/fluent-bit/state/
|
|
```
|
|
|
|
If SQLite is installed and Fluent Bit is stopped or the DB can be safely inspected:
|
|
|
|
```bash
|
|
sqlite3 /var/lib/fluent-bit/state/cpanel-access.db \
|
|
'SELECT id,name,offset,inode FROM in_tail_files;'
|
|
```
|
|
|
|
The Tail database tells Fluent Bit where it last read each tracked file.
|
|
|
|
## 14.14 Verify filesystem buffering
|
|
|
|
```bash
|
|
find /var/lib/fluent-bit/storage -maxdepth 2 -type f -ls | head -50
|
|
```
|
|
|
|
Remember:
|
|
|
|
```text
|
|
Tail DB
|
|
-> protects file read position
|
|
|
|
filesystem storage
|
|
-> holds chunks that have been read but are not yet successfully delivered
|
|
```
|
|
|
|
They are separate mechanisms.
|
|
|
|
## 14.15 Verify HTTP output to Data Prepper
|
|
|
|
Follow Fluent Bit logs:
|
|
|
|
```bash
|
|
journalctl -u fluent-bit -f
|
|
```
|
|
|
|
The target used in this implementation is:
|
|
|
|
```text
|
|
http://118.67.213.76:31599/logs
|
|
```
|
|
|
|
Test the receiver directly from the cPanel server:
|
|
|
|
```bash
|
|
curl -i -X POST \
|
|
'http://118.67.213.76:31599/logs' \
|
|
-H 'Content-Type: application/json' \
|
|
--data '[{"log":"manual-cpanel-host-test","source_file":"manual-test","log_source":"test.manual"}]'
|
|
```
|
|
|
|
If the manual curl fails, fix network/Ingress/Data Prepper before modifying Fluent Bit parsing or Tail settings.
|
|
|
|
## 14.16 Interpretation of common Fluent Bit output failures
|
|
|
|
### `502 Bad Gateway`
|
|
|
|
```text
|
|
Fluent Bit -> NGINX = reachable
|
|
NGINX -> Data Prepper upstream = unhealthy/unreachable
|
|
```
|
|
|
|
Check Data Prepper pod, Service, EndpointSlice, listener, and Ingress.
|
|
|
|
### `connection refused`
|
|
|
|
The target IP/port is reachable at the network level but nothing is accepting the connection, or a firewall is rejecting it.
|
|
|
|
### `failed to flush chunk`
|
|
|
|
Inspect the associated HTTP/network error. With filesystem buffering enabled, confirm the queue is growing and not at the storage limit.
|
|
|
|
### configuration parser error
|
|
|
|
Use the exact reported file/line. Do not assume the systemd unit failure text is the root cause; Fluent Bit's own startup output contains the useful parser error.
|
|
|
|
## 14.17 Durable buffering recovery test
|
|
|
|
Perform this only after normal ingestion is stable.
|
|
|
|
1. Record the current OpenSearch count.
|
|
2. Make Data Prepper unavailable intentionally.
|
|
3. Generate several test requests/log records.
|
|
4. Confirm Fluent Bit reports retries.
|
|
5. Confirm filesystem backlog activity in `/var/lib/fluent-bit/storage`.
|
|
6. Restart Fluent Bit while Data Prepper is still unavailable.
|
|
7. Confirm the Tail DB and storage directories survive the restart.
|
|
8. Restore Data Prepper.
|
|
9. Confirm queued events are forwarded.
|
|
10. Verify the expected documents appear in OpenSearch.
|
|
|
|
Useful commands:
|
|
|
|
```bash
|
|
systemctl restart fluent-bit
|
|
journalctl -u fluent-bit --since '10 minutes ago' --no-pager
|
|
find /var/lib/fluent-bit/storage -type f | head
|
|
```
|
|
|
|
This test validates both offset persistence and undelivered-event buffering.
|
|
|
|
## 14.18 Long-line behavior
|
|
|
|
The configuration uses:
|
|
|
|
```ini
|
|
Skip_Long_Lines true
|
|
```
|
|
|
|
This prevents a single oversized line from stopping monitoring of the entire file, but the oversized event can be skipped.
|
|
|
|
If large audit/application records matter, test representative maximum line sizes and tune `Buffer_Chunk_Size` / `Buffer_Max_Size` rather than assuming the defaults are sufficient.
|
|
|
|
## 14.19 Log rotation
|
|
|
|
Fluent Bit Tail tracks files using file metadata/offset state. Verify rotation behavior for each high-value log:
|
|
|
|
```bash
|
|
ls -li /usr/local/cpanel/logs/access_log*
|
|
ls -li /var/log/audit/audit.log*
|
|
```
|
|
|
|
Test that rotation does not cause either permanent loss or unexpected full-file replay. Do not delete Tail DB files as a routine rotation fix.
|
|
|
|
## 14.20 Multiple files and inotify limits
|
|
|
|
If the deployment expands to a large number of Apache/cPanel files, inspect inotify capacity:
|
|
|
|
```bash
|
|
sysctl fs.inotify.max_user_watches
|
|
sysctl fs.inotify.max_user_instances
|
|
```
|
|
|
|
Increase them only if monitoring scale requires it and persist the chosen values through the normal OS configuration-management process.
|
|
|
|
## 14.21 Security hardening for Fluent Bit
|
|
|
|
Use the minimum filesystem permissions required to read the configured logs and write its own state/buffer directories.
|
|
|
|
Protect:
|
|
|
|
```text
|
|
/etc/fluent-bit/fluent-bit.conf
|
|
/etc/fluent-bit/classify.lua
|
|
/var/lib/fluent-bit/state/
|
|
/var/lib/fluent-bit/storage/
|
|
```
|
|
|
|
The original ingestion route used plain HTTP to the NGINX NodePort. For production, protect the transport with private routing, firewall restrictions, authentication and/or TLS as appropriate to the environment.
|
|
|
|
Do not store OpenSearch credentials in Fluent Bit when Fluent Bit is intentionally sending only to Data Prepper.
|
|
|
|
## 14.22 Fluent Bit acceptance checklist
|
|
|
|
- [ ] Required source files exist and receive new records.
|
|
- [ ] Fluent Bit service is running.
|
|
- [ ] No classic-config indentation/parser error remains.
|
|
- [ ] Each Tail input has a unique persistent DB.
|
|
- [ ] Tail DB directory is writable.
|
|
- [ ] Filesystem buffer directory is writable.
|
|
- [ ] `Storage.Type filesystem` is enabled for protected inputs.
|
|
- [ ] `source_file` is present.
|
|
- [ ] `log_source` matches the source file.
|
|
- [ ] `/usr/local/cpanel/logs/access_log` becomes `cpanel.access_log`.
|
|
- [ ] `/var/log/syslog` becomes `system.syslog`.
|
|
- [ ] `/var/log/audit/audit.log` becomes `audit.audit.log`.
|
|
- [ ] HTTP destination is the correct NGINX/Data Prepper endpoint.
|
|
- [ ] URI is `/logs`.
|
|
- [ ] Output format is JSON.
|
|
- [ ] Fluent Bit has no sustained 502/refused/flush failures.
|
|
- [ ] Restart resumes from stored offsets.
|
|
- [ ] Data Prepper outage/recovery test has been completed.
|
|
- [ ] Backlog disk usage is monitored.
|
|
|
|
---
|
|
|
|
# 15. cPanel Access Log Parsing
|
|
|
|
## 15.1 Example raw log
|
|
|
|
```text
|
|
20.100.201.54 proxy - [08/09/2026:07:37:27 -0000] "GET /wp-content/themes/twentytwenty/404.php HTTP/1.1" 301 0 "-" "-" "-" "X-Forwarded-For: 20.100.201.54" 80
|
|
```
|
|
|
|
## 15.2 Recommended parser
|
|
|
|
Use one timestamp capture and let the date processor create `@timestamp`.
|
|
|
|
```yaml
|
|
cpanel-access-pipeline:
|
|
source:
|
|
pipeline:
|
|
name: cpanel-log-router
|
|
|
|
processor:
|
|
- grok:
|
|
break_on_match: true
|
|
match:
|
|
log:
|
|
- '^%{IP:client_ip} %{NOTSPACE:identity} %{NOTSPACE:user} \[%{GREEDYDATA:timestamp}\] "%{WORD:method} %{URIPATHPARAM:url} HTTP/%{NUMBER:http_version}" %{INT:http_status:int} %{NONNEGINT:bytes:long} "%{NOTSPACE:referrer}" "%{GREEDYDATA:user_agent}" "%{NOTSPACE:auth_method}" "%{GREEDYDATA:x_forwarded_for}" %{POSINT:service_port:int}$'
|
|
target_key: parsed
|
|
|
|
- date:
|
|
match:
|
|
- key: /parsed/timestamp
|
|
patterns:
|
|
- "MM/dd/yyyy:HH:mm:ss Z"
|
|
destination: "@timestamp"
|
|
|
|
sink:
|
|
- opensearch:
|
|
hosts:
|
|
- "https://brilliant-opensearch-cluster-master:9200"
|
|
insecure: false
|
|
username: logger
|
|
password: "<SECRET-MANAGED-VALUE>"
|
|
cert: /usr/share/data-prepper/cert/root-ca.pem
|
|
index_type: custom
|
|
index: cpanel-access-log-%{yyyy.MM.dd}
|
|
```
|
|
|
|
## 15.3 Parsed fields
|
|
|
|
```text
|
|
parsed.client_ip
|
|
parsed.identity
|
|
parsed.user
|
|
parsed.timestamp
|
|
parsed.method
|
|
parsed.url
|
|
parsed.http_version
|
|
parsed.http_status
|
|
parsed.bytes
|
|
parsed.referrer
|
|
parsed.user_agent
|
|
parsed.auth_method
|
|
parsed.x_forwarded_for
|
|
parsed.service_port
|
|
@timestamp
|
|
```
|
|
|
|
## 15.4 Important Grok issue: `%{PORT}`
|
|
|
|
In the tested Data Prepper 2.8.0 environment:
|
|
|
|
```text
|
|
%{PORT:service_port}
|
|
```
|
|
|
|
failed with:
|
|
|
|
```text
|
|
No definition for key 'PORT' found
|
|
```
|
|
|
|
Use:
|
|
|
|
```text
|
|
%{POSINT:service_port:int}
|
|
```
|
|
|
|
instead.
|
|
|
|
## 15.5 YAML hierarchy
|
|
|
|
Correct:
|
|
|
|
```yaml
|
|
processor:
|
|
- grok:
|
|
break_on_match: true
|
|
match:
|
|
log:
|
|
- '<PATTERN>'
|
|
target_key: parsed
|
|
```
|
|
|
|
`break_on_match`, `match`, and `target_key` belong under `grok:`.
|
|
|
|
## 15.6 Parser validation test
|
|
|
|
```bash
|
|
curl -sS -X POST \
|
|
'http://<INGRESS-NODE-IP>:31599/logs' \
|
|
-H 'Content-Type: application/json' \
|
|
--data '[{
|
|
"log":"20.100.201.54 proxy - [08/09/2026:07:37:27 -0000] \"GET /wp-content/themes/twentytwenty/404.php HTTP/1.1\" 301 0 \"-\" \"-\" \"-\" \"X-Forwarded-For: 20.100.201.54\" 80",
|
|
"source_file":"/usr/local/cpanel/logs/access_log",
|
|
"log_source":"cpanel.access_log"
|
|
}]'
|
|
```
|
|
|
|
Then inspect a newly indexed document.
|
|
|
|
---
|
|
|
|
# 16. Linux Syslog and Other Log Sources
|
|
|
|
## 16.1 Example syslog records
|
|
|
|
```text
|
|
2026-08-11T10:42:39.679625+00:00 bwebhosting kernel: Firewall: ...
|
|
```
|
|
|
|
```text
|
|
2026-08-11T10:42:39.434068+00:00 bwebhosting fluent-bit[1241036]: ...
|
|
```
|
|
|
|
A parser must tolerate both:
|
|
|
|
```text
|
|
process:
|
|
process[pid]:
|
|
```
|
|
|
|
Do not assume a PID is always present.
|
|
|
|
## 16.2 Recommended syslog approach
|
|
|
|
Start with a small set of real samples from the actual host and build multiple Grok patterns if formats vary.
|
|
|
|
If a document contains:
|
|
|
|
```json
|
|
{
|
|
"parsed": {}
|
|
}
|
|
```
|
|
|
|
then transport and indexing may be working while the Grok pattern failed to match.
|
|
|
|
## 16.3 Do not infer source from index name
|
|
|
|
If a document in `cpanel-system-syslog-*` contains:
|
|
|
|
```json
|
|
{
|
|
"log_source": "system.exim_mainlog"
|
|
}
|
|
```
|
|
|
|
then routing/filtering is wrong even though the index name says `syslog`.
|
|
|
|
Verify:
|
|
|
|
```text
|
|
_index
|
|
source_file
|
|
log_source
|
|
log
|
|
parsed
|
|
@timestamp
|
|
```
|
|
|
|
## 16.4 Unknown/catch-all strategy
|
|
|
|
Use:
|
|
|
|
```text
|
|
cpanel-all-log-YYYY.MM.dd
|
|
```
|
|
|
|
for sources not yet assigned to a dedicated parser.
|
|
|
|
This provides a safe discovery path:
|
|
|
|
```text
|
|
unknown log appears
|
|
-> stored raw
|
|
-> inspect format
|
|
-> create dedicated parser if operationally valuable
|
|
```
|
|
|
|
---
|
|
|
|
# 17. Linux Audit Log Parsing
|
|
|
|
## 17.1 Audit event model
|
|
|
|
A single logical Audit event can create multiple records sharing one event ID:
|
|
|
|
```text
|
|
SYSCALL
|
|
EXECVE
|
|
CWD
|
|
PATH
|
|
PROCTITLE
|
|
```
|
|
|
|
Therefore:
|
|
|
|
```text
|
|
Total Audit documents != total commands executed
|
|
```
|
|
|
|
Use:
|
|
|
|
```text
|
|
audit.event_id
|
|
```
|
|
|
|
for correlation.
|
|
|
|
Use primarily:
|
|
|
|
```text
|
|
audit.log_type = EXECVE
|
|
```
|
|
|
|
when counting command execution records.
|
|
|
|
## 17.2 Audit pipeline source
|
|
|
|
```yaml
|
|
cpanel-audit-pipeline:
|
|
source:
|
|
pipeline:
|
|
name: cpanel-log-router
|
|
```
|
|
|
|
A defensive filter may be retained:
|
|
|
|
```yaml
|
|
processor:
|
|
- drop_events:
|
|
drop_when: '/log_source != "audit.audit.log"'
|
|
```
|
|
|
|
## 17.3 Common Audit header
|
|
|
|
Typical:
|
|
|
|
```text
|
|
type=EXECVE msg=audit(1787117401.120:11280588): ...
|
|
```
|
|
|
|
Parser:
|
|
|
|
```yaml
|
|
- grok:
|
|
break_on_match: true
|
|
match:
|
|
log:
|
|
- '^type=%{WORD:log_type} msg=audit\(%{INT:timestamp_seconds}\.%{INT:timestamp_fraction}:%{INT:event_id}\): %{GREEDYDATA:msg}'
|
|
target_key: audit
|
|
```
|
|
|
|
Expected:
|
|
|
|
```json
|
|
{
|
|
"audit": {
|
|
"log_type": "EXECVE",
|
|
"timestamp_seconds": "1787117401",
|
|
"timestamp_fraction": "120",
|
|
"event_id": "11280588",
|
|
"msg": "..."
|
|
}
|
|
}
|
|
```
|
|
|
|
## 17.4 Audit timestamp
|
|
|
|
```yaml
|
|
- date:
|
|
match:
|
|
- key: /audit/timestamp_seconds
|
|
patterns:
|
|
- epoch_second
|
|
destination: "@timestamp"
|
|
```
|
|
|
|
## 17.5 `SYSCALL`
|
|
|
|
Parse only when:
|
|
|
|
```text
|
|
/audit/log_type == "SYSCALL"
|
|
```
|
|
|
|
Common target fields:
|
|
|
|
```text
|
|
syscall.syscall
|
|
syscall.pid
|
|
syscall.ppid
|
|
syscall.success
|
|
syscall.exit_code
|
|
syscall.exe
|
|
syscall.comm
|
|
syscall.AUID
|
|
syscall.UID
|
|
syscall.terminal
|
|
```
|
|
|
|
Audit keys may be quoted or `(null)`, so maintain multiple patterns if necessary.
|
|
|
|
## 17.6 `EXECVE`
|
|
|
|
### Stage 1 — preserve argument count and raw argument text
|
|
|
|
```yaml
|
|
- grok:
|
|
break_on_match: true
|
|
grok_when: '/audit/log_type == "EXECVE"'
|
|
target_key: execve
|
|
match:
|
|
/audit/msg:
|
|
- '^argc=%{INT:arg_count} %{GREEDYDATA:args_raw}$'
|
|
```
|
|
|
|
### Stage 2 — optional key/value split
|
|
|
|
```yaml
|
|
- key_value:
|
|
key_value_when: '/audit/log_type == "EXECVE"'
|
|
source: /execve/args_raw
|
|
destination: /execve/args
|
|
field_delimiter_regex: '\s+'
|
|
value_split_characters: "="
|
|
delete_value_regex: '^"|"$'
|
|
```
|
|
|
|
Do **not** configure both `field_delimiter_regex` and a nonempty `field_split_characters`; these options are mutually exclusive.
|
|
|
|
### Known limitation
|
|
|
|
Splitting on whitespace can break an argument that contains literal spaces inside quotes, for example:
|
|
|
|
```text
|
|
a1="hello world"
|
|
```
|
|
|
|
Therefore always preserve:
|
|
|
|
```text
|
|
execve.args_raw
|
|
```
|
|
|
|
and do not treat `execve.args.*` as a lossless reconstruction of every possible command line. Test representative Audit samples before relying on argument-level fields for security logic.
|
|
|
|
## 17.7 Mapping-growth limitation
|
|
|
|
Dynamic fields such as:
|
|
|
|
```text
|
|
execve.args.a0
|
|
execve.args.a1
|
|
execve.args.a2
|
|
...
|
|
```
|
|
|
|
can create field-count growth.
|
|
|
|
Prefer long-term normalized fields such as:
|
|
|
|
```text
|
|
execve.command
|
|
execve.arg_count
|
|
execve.args_raw
|
|
```
|
|
|
|
and limit dynamic argument mappings.
|
|
|
|
## 17.8 `CWD`
|
|
|
|
```yaml
|
|
- grok:
|
|
break_on_match: true
|
|
grok_when: '/audit/log_type == "CWD"'
|
|
target_key: cwd
|
|
match:
|
|
/audit/msg:
|
|
- '^cwd="%{GREEDYDATA:path}"'
|
|
```
|
|
|
|
## 17.9 `USER_ACCT`, `USER_START`, `CRED_ACQ`
|
|
|
|
For each type, keep the full parser chain consistent:
|
|
|
|
```text
|
|
grok_when
|
|
-> target_key
|
|
-> key_value_when
|
|
-> source
|
|
-> destination
|
|
```
|
|
|
|
Example nested key/value processing:
|
|
|
|
```yaml
|
|
- key_value:
|
|
key_value_when: '/audit/log_type == "USER_ACCT"'
|
|
source: /user_account/msg
|
|
destination: /user_account/parsed_msg
|
|
field_split_characters: " "
|
|
value_split_characters: "="
|
|
delete_value_regex: '^"|"$'
|
|
```
|
|
|
|
Known limitation: simple space splitting can break values such as:
|
|
|
|
```text
|
|
acct="John Doe"
|
|
```
|
|
|
|
Preserve the raw message.
|
|
|
|
## 17.10 `LOGIN`
|
|
|
|
Some Audit LOGIN records contain the control separator `\u001d` before human-readable identity fields. Patterns should tolerate arbitrary content between result and `UID=` instead of assuming normal whitespace only.
|
|
|
|
## 17.11 Audit parser verification
|
|
|
|
Count total EXECVE:
|
|
|
|
```bash
|
|
curl -k -u 'USER:PASSWORD' \
|
|
-H 'Content-Type: application/json' \
|
|
'https://OPENSEARCH:9200/cpanel-audit-log-YYYY.MM.DD/_count' \
|
|
-d '{
|
|
"query": {
|
|
"match": {
|
|
"audit.log_type": "EXECVE"
|
|
}
|
|
}
|
|
}'
|
|
```
|
|
|
|
Count parsed EXECVE:
|
|
|
|
```bash
|
|
curl -k -u 'USER:PASSWORD' \
|
|
-H 'Content-Type: application/json' \
|
|
'https://OPENSEARCH:9200/cpanel-audit-log-YYYY.MM.DD/_count' \
|
|
-d '{
|
|
"query": {
|
|
"bool": {
|
|
"filter": [
|
|
{"match": {"audit.log_type": "EXECVE"}},
|
|
{"exists": {"field": "execve.arg_count"}}
|
|
]
|
|
}
|
|
}
|
|
}'
|
|
```
|
|
|
|
Ideal validation target:
|
|
|
|
```text
|
|
Total EXECVE == parsed EXECVE
|
|
```
|
|
|
|
Investigate mismatches with raw documents rather than only aggregate Grok metrics.
|
|
|
|
---
|
|
|
|
# 18. OpenSearch Index Templates and Mapping
|
|
|
|
## 18.1 Index model
|
|
|
|
```text
|
|
cpanel-access-log-* -> parsed cPanel HTTP access logs
|
|
cpanel-system-syslog-* -> system syslog
|
|
cpanel-audit-log-* -> Linux Audit records
|
|
cpanel-all-log-* -> unknown/unhandled raw logs
|
|
```
|
|
|
|
## 18.2 cPanel access mappings
|
|
|
|
Critical types:
|
|
|
|
```text
|
|
@timestamp -> date
|
|
parsed.client_ip -> ip
|
|
parsed.http_status -> integer
|
|
parsed.bytes -> long
|
|
parsed.service_port -> integer
|
|
```
|
|
|
|
Template example:
|
|
|
|
```json
|
|
{
|
|
"index_patterns": ["cpanel-access-log-*"],
|
|
"template": {
|
|
"mappings": {
|
|
"properties": {
|
|
"@timestamp": {
|
|
"type": "date"
|
|
},
|
|
"parsed": {
|
|
"properties": {
|
|
"client_ip": {
|
|
"type": "ip"
|
|
},
|
|
"http_status": {
|
|
"type": "integer"
|
|
},
|
|
"bytes": {
|
|
"type": "long"
|
|
},
|
|
"service_port": {
|
|
"type": "integer"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
Apply with Dev Tools or API:
|
|
|
|
```http
|
|
PUT _index_template/cpanel-access-log-template
|
|
{
|
|
"index_patterns": ["cpanel-access-log-*"],
|
|
"template": {
|
|
"mappings": {
|
|
"properties": {
|
|
"@timestamp": {"type": "date"},
|
|
"parsed": {
|
|
"properties": {
|
|
"client_ip": {"type": "ip"},
|
|
"http_status": {"type": "integer"},
|
|
"bytes": {"type": "long"},
|
|
"service_port": {"type": "integer"}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
## 18.3 Verify mapping
|
|
|
|
```http
|
|
GET cpanel-access-log-*/_field_caps?fields=@timestamp,parsed.client_ip,parsed.http_status,parsed.bytes,parsed.service_port
|
|
```
|
|
|
|
## 18.4 Audit default search fields
|
|
|
|
A large Audit mapping can cause:
|
|
|
|
```text
|
|
too_many_nested_clauses
|
|
maxClauseCount is set to 1024
|
|
```
|
|
|
|
Restrict default query fields:
|
|
|
|
```http
|
|
PUT cpanel-audit-log-*/_settings
|
|
{
|
|
"index.query.default_field": [
|
|
"log",
|
|
"audit.log_type",
|
|
"audit.event_id",
|
|
"syscall.exe",
|
|
"syscall.comm",
|
|
"syscall.AUID",
|
|
"execve.args.a0"
|
|
]
|
|
}
|
|
```
|
|
|
|
Add the same setting to the matching index template so future daily indexes inherit it.
|
|
|
|
Do not immediately increase the Boolean clause limit. Control field growth and default-field expansion first.
|
|
|
|
## 18.5 Replica policy
|
|
|
|
Set shard/replica counts in an index template deliberately. Do not assume Helm pod replicas automatically configure index replicas.
|
|
|
|
---
|
|
|
|
# 19. End-to-End Verification
|
|
|
|
Always test left-to-right.
|
|
|
|
## 19.1 Source file
|
|
|
|
```bash
|
|
tail -n 5 /usr/local/cpanel/logs/access_log
|
|
stat /usr/local/cpanel/logs/access_log
|
|
```
|
|
|
|
## 19.2 Fluent Bit
|
|
|
|
```bash
|
|
systemctl status fluent-bit --no-pager
|
|
journalctl -u fluent-bit --since '10 minutes ago' --no-pager
|
|
```
|
|
|
|
Check Tail DB exists:
|
|
|
|
```bash
|
|
ls -lh /var/lib/fluent-bit/state/
|
|
```
|
|
|
|
Check buffer path:
|
|
|
|
```bash
|
|
du -sh /var/lib/fluent-bit/storage/
|
|
```
|
|
|
|
## 19.3 Ingress
|
|
|
|
From the source server:
|
|
|
|
```bash
|
|
curl -i -X POST \
|
|
'http://118.67.213.76:31599/logs' \
|
|
-H 'Content-Type: application/json' \
|
|
--data '[{"log":"manual ingress test","source_file":"manual","log_source":"test.manual"}]'
|
|
```
|
|
|
|
## 19.4 Data Prepper pod
|
|
|
|
```bash
|
|
kubectl get pods -n "$NS" -l app=data-prepper -o wide
|
|
kubectl logs -n "$NS" deploy/data-prepper --tail=200
|
|
```
|
|
|
|
## 19.5 Service and endpoints
|
|
|
|
```bash
|
|
kubectl get svc data-prepper -n "$NS" -o wide
|
|
kubectl get endpointslice -n "$NS" \
|
|
-l kubernetes.io/service-name=data-prepper -o wide
|
|
```
|
|
|
|
## 19.6 Port 2021 listener
|
|
|
|
If the image lacks `ss` and `netstat`, inspect `/proc/net/tcp*`.
|
|
|
|
```text
|
|
2021 decimal = 07E5 hexadecimal
|
|
TCP state 0A = LISTEN
|
|
TCP state 01 = ESTABLISHED
|
|
```
|
|
|
|
Example:
|
|
|
|
```bash
|
|
kubectl exec -n "$NS" deploy/data-prepper -- \
|
|
sh -c 'grep -i ":07E5" /proc/net/tcp /proc/net/tcp6 || true'
|
|
```
|
|
|
|
## 19.7 OpenSearch indexes
|
|
|
|
```http
|
|
GET _cat/indices/cpanel-*?v
|
|
```
|
|
|
|
Remember:
|
|
|
|
```text
|
|
index exists + docs.count=0
|
|
```
|
|
|
|
does not prove successful ingestion.
|
|
|
|
## 19.8 Recent documents
|
|
|
|
```http
|
|
GET cpanel-access-log-*/_search
|
|
{
|
|
"size": 5,
|
|
"sort": [
|
|
{"@timestamp": "desc"}
|
|
],
|
|
"_source": [
|
|
"@timestamp",
|
|
"log_source",
|
|
"source_file",
|
|
"log",
|
|
"parsed"
|
|
]
|
|
}
|
|
```
|
|
|
|
## 19.9 Last 24 hours count
|
|
|
|
```http
|
|
GET cpanel-access-log-*/_count
|
|
{
|
|
"query": {
|
|
"range": {
|
|
"@timestamp": {
|
|
"gte": "now-24h",
|
|
"lte": "now"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
This proves:
|
|
|
|
```text
|
|
index wildcard works
|
|
+ data exists
|
|
+ @timestamp is queryable
|
|
+ time-range query works
|
|
```
|
|
|
|
---
|
|
|
|
# 20. OpenSearch Dashboards Data Source and Workspace
|
|
|
|
## 20.1 Enable data sources
|
|
|
|
```yaml
|
|
data_source:
|
|
enabled: true
|
|
```
|
|
|
|
## 20.2 Workspaces
|
|
|
|
```yaml
|
|
workspace:
|
|
enabled: true
|
|
|
|
opensearch_security.multitenancy.enabled: false
|
|
```
|
|
|
|
## 20.3 Data source encryption master key
|
|
|
|
The source SOP configured a data-source credential encryption master key on OpenSearch nodes.
|
|
|
|
Requirements:
|
|
|
|
```text
|
|
same key on every OpenSearch node
|
|
valid supported key length
|
|
never commit the key to Git
|
|
store using secure secret management
|
|
```
|
|
|
|
Before reusing the exact configuration key in a new OpenSearch version, verify the current OpenSearch documentation for the data-source encryption setting.
|
|
|
|
## 20.4 Create workspace
|
|
|
|
Recommended workspace:
|
|
|
|
```text
|
|
Cpanel
|
|
```
|
|
|
|
Inside the workspace:
|
|
|
|
1. Create an OpenSearch data source connection.
|
|
2. Use the cluster Service endpoint.
|
|
3. Use the required username/password authentication method.
|
|
4. Test the connection.
|
|
5. Create index pattern:
|
|
|
|
```text
|
|
cpanel-access-log-*
|
|
```
|
|
|
|
6. Select time field:
|
|
|
|
```text
|
|
@timestamp
|
|
```
|
|
|
|
---
|
|
|
|
# 21. cPanel Access Dashboard
|
|
|
|
Recommended dashboard:
|
|
|
|
```text
|
|
cPanel Access Overview
|
|
```
|
|
|
|
Suggested layout:
|
|
|
|
```text
|
|
+-------------------------------------------------------------------+
|
|
| Total Requests | Unique Clients | 2xx | 4xx | 5xx |
|
|
+-------------------------------------------------------------------+
|
|
| Requests Over Time |
|
|
+--------------------------------+----------------------------------+
|
|
| HTTP Status Distribution | Top Requested URLs |
|
|
+--------------------------------+----------------------------------+
|
|
| Top Client IPs | HTTP Methods |
|
|
+--------------------------------+----------------------------------+
|
|
| Top User Agents |
|
|
+-------------------------------------------------------------------+
|
|
```
|
|
|
|
## 21.1 Total Requests
|
|
|
|
```text
|
|
Visualization: Metric
|
|
Aggregation: Count
|
|
Label: Total Requests
|
|
```
|
|
|
|
## 21.2 Unique Clients
|
|
|
|
```text
|
|
Visualization: Metric
|
|
Aggregation: Unique Count / Cardinality
|
|
Field: parsed.client_ip
|
|
Label: Unique Clients
|
|
```
|
|
|
|
## 21.3 2xx
|
|
|
|
DQL:
|
|
|
|
```text
|
|
parsed.http_status >= 200 and parsed.http_status < 300
|
|
```
|
|
|
|
## 21.4 4xx
|
|
|
|
```text
|
|
parsed.http_status >= 400 and parsed.http_status < 500
|
|
```
|
|
|
|
## 21.5 5xx
|
|
|
|
```text
|
|
parsed.http_status >= 500 and parsed.http_status < 600
|
|
```
|
|
|
|
Do not use chained comparison syntax such as:
|
|
|
|
```text
|
|
200 <= parsed.http_status < 300
|
|
```
|
|
|
|
## 21.6 Requests over time
|
|
|
|
```text
|
|
Visualization: Area/Line
|
|
Metric: Count
|
|
X axis: Date Histogram
|
|
Field: @timestamp
|
|
Interval: Auto
|
|
```
|
|
|
|
## 21.7 HTTP status distribution
|
|
|
|
```text
|
|
Visualization: Pie/Donut
|
|
Metric: Count
|
|
Terms: parsed.http_status
|
|
Size: 10
|
|
```
|
|
|
|
## 21.8 Top URLs
|
|
|
|
```text
|
|
Visualization: Horizontal Bar
|
|
Metric: Count
|
|
Terms: parsed.url.keyword
|
|
Fallback: parsed.url if aggregatable
|
|
Size: 10
|
|
```
|
|
|
|
## 21.9 Top client IPs
|
|
|
|
```text
|
|
Terms: parsed.client_ip
|
|
Size: 10
|
|
```
|
|
|
|
## 21.10 HTTP methods
|
|
|
|
```text
|
|
Terms: parsed.method.keyword
|
|
Fallback: parsed.method if aggregatable
|
|
```
|
|
|
|
## 21.11 Top user agents
|
|
|
|
```text
|
|
Terms: parsed.user_agent.keyword
|
|
Fallback: parsed.user_agent if aggregatable
|
|
```
|
|
|
|
Useful for scanners, bots, crawlers, and automated clients.
|
|
|
|
---
|
|
|
|
# 22. Linux Audit Security Dashboard
|
|
|
|
Recommended dashboard:
|
|
|
|
```text
|
|
Linux Server Audit & Security Overview
|
|
```
|
|
|
|
Layout:
|
|
|
|
```text
|
|
+-----------------------------------------------------------+
|
|
| Commands | Logins | Failed Syscalls | Users | Audit Docs |
|
|
+-----------------------------------------------------------+
|
|
| Audit Activity Over Time |
|
|
+---------------------------+-------------------------------+
|
|
| Top Commands | Top Executables |
|
|
+---------------------------+-------------------------------+
|
|
| Top Users | Top Failed Syscalls |
|
|
+---------------------------+-------------------------------+
|
|
| Authentication & Session Activity |
|
|
+-----------------------------------------------------------+
|
|
| Recent Audit Activity |
|
|
+-----------------------------------------------------------+
|
|
```
|
|
|
|
## 22.1 Commands executed
|
|
|
|
DQL:
|
|
|
|
```text
|
|
audit.log_type: "EXECVE"
|
|
```
|
|
|
|
Metric: Count.
|
|
|
|
## 22.2 Login events
|
|
|
|
```text
|
|
audit.log_type: "LOGIN"
|
|
```
|
|
|
|
## 22.3 Failed syscalls
|
|
|
|
```text
|
|
audit.log_type: "SYSCALL" AND syscall.success: "no"
|
|
```
|
|
|
|
## 22.4 Top failed syscall names
|
|
|
|
```text
|
|
Terms: syscall.syscall.keyword
|
|
```
|
|
|
|
## 22.5 Top commands
|
|
|
|
```text
|
|
DQL: audit.log_type: "EXECVE"
|
|
Terms: execve.args.a0.keyword
|
|
```
|
|
|
|
Use a normalized `execve.command` field instead if you implement the recommended schema improvement.
|
|
|
|
## 22.6 Top users
|
|
|
|
```text
|
|
DQL: audit.log_type: "SYSCALL"
|
|
Terms: syscall.AUID.keyword
|
|
```
|
|
|
|
## 22.7 Authentication/session activity
|
|
|
|
DQL:
|
|
|
|
```text
|
|
audit.log_type: ("LOGIN" OR "USER_ACCT" OR "USER_START" OR "CRED_ACQ")
|
|
```
|
|
|
|
Use Date Histogram on `@timestamp` and split series by `audit.log_type.keyword`.
|
|
|
|
## 22.8 Recent activity
|
|
|
|
Use separate tables initially because one logical Audit event spans multiple documents.
|
|
|
|
Recent EXECVE fields:
|
|
|
|
```text
|
|
@timestamp
|
|
audit.event_id
|
|
execve.args.a0
|
|
execve.args.a1
|
|
execve.args.a2
|
|
```
|
|
|
|
Recent failed syscall fields:
|
|
|
|
```text
|
|
@timestamp
|
|
audit.event_id
|
|
syscall.AUID
|
|
syscall.exe
|
|
syscall.syscall
|
|
syscall.success
|
|
syscall.exit_code
|
|
```
|
|
|
|
Future correlation should group by `audit.event_id`.
|
|
|
|
---
|
|
|
|
# 23. Anomaly Detection, Forecasting, and Alerting
|
|
|
|
## 23.1 Request-volume anomaly detector
|
|
|
|
Recommended detector:
|
|
|
|
```text
|
|
Total-Request-Detector
|
|
```
|
|
|
|
Index:
|
|
|
|
```text
|
|
cpanel-access-log-*
|
|
```
|
|
|
|
Timestamp:
|
|
|
|
```text
|
|
@timestamp
|
|
```
|
|
|
|
Feature:
|
|
|
|
```text
|
|
Request-Count -> Count
|
|
```
|
|
|
|
Starting operational settings from the implementation:
|
|
|
|
```text
|
|
Interval: 10 minutes
|
|
Frequency: 10 minutes
|
|
Window delay: 1 minute
|
|
History: 144 intervals (~24 hours)
|
|
Shingle size: 8
|
|
Sparse data: Fill with Zeros
|
|
```
|
|
|
|
Do not split by high-cardinality client IP in the first detector.
|
|
|
|
Validate:
|
|
|
|
```text
|
|
actual request count
|
|
expected range
|
|
anomaly grade
|
|
confidence
|
|
historical timeline
|
|
```
|
|
|
|
## 23.2 Forecasting
|
|
|
|
Recommended forecaster:
|
|
|
|
```text
|
|
Total-Request-Forecast
|
|
```
|
|
|
|
Starting values used:
|
|
|
|
```text
|
|
Forecast interval: 3 minutes
|
|
Window delay: 2 minutes
|
|
Horizon: 24 intervals (~72 minutes)
|
|
History: 537 intervals (~26h 51m)
|
|
```
|
|
|
|
These are starting parameters, not universal defaults.
|
|
|
|
## 23.3 Alert monitor
|
|
|
|
Flow:
|
|
|
|
```text
|
|
Anomaly Detector
|
|
-> Monitor
|
|
-> Trigger
|
|
-> Alert
|
|
-> Action
|
|
-> Notification Channel
|
|
```
|
|
|
|
Starting trigger threshold:
|
|
|
|
```text
|
|
Anomaly grade > 0.7
|
|
AND
|
|
Confidence > 0.7
|
|
```
|
|
|
|
Tune after reviewing false positives and real incidents.
|
|
|
|
---
|
|
|
|
# 24. SMTP Notification Configuration
|
|
|
|
Recorded sender configuration:
|
|
|
|
```text
|
|
Sender name: cloud-operation
|
|
SMTP host: zsmtp.brilliant.com.bd
|
|
Port: 465
|
|
Method: SSL
|
|
From address: cloud.operation@brilliant.com.bd
|
|
```
|
|
|
|
Do not put SMTP passwords into Helm values or Git.
|
|
|
|
## 24.1 Create temporary credential files
|
|
|
|
```bash
|
|
printf '%s' 'cloud.operation@brilliant.com.bd' > /tmp/smtp-username
|
|
|
|
read -rsp 'SMTP password: ' SMTP_PASSWORD
|
|
echo
|
|
printf '%s' "$SMTP_PASSWORD" > /tmp/smtp-password
|
|
unset SMTP_PASSWORD
|
|
```
|
|
|
|
## 24.2 Create Secret for OpenSearch keystore
|
|
|
|
```bash
|
|
kubectl create secret generic opensearch-smtp-keystore \
|
|
-n "$NS" \
|
|
--from-file=opensearch.notifications.core.email.cloud-operation.username=/tmp/smtp-username \
|
|
--from-file=opensearch.notifications.core.email.cloud-operation.password=/tmp/smtp-password \
|
|
--dry-run=client -o yaml | kubectl apply -f -
|
|
|
|
rm -f /tmp/smtp-username /tmp/smtp-password
|
|
```
|
|
|
|
The key path uses the **sender name** `cloud-operation`, not the sender email address.
|
|
|
|
## 24.3 Helm keystore value
|
|
|
|
```yaml
|
|
keystore:
|
|
- secretName: opensearch-smtp-keystore
|
|
```
|
|
|
|
## 24.4 Upgrade
|
|
|
|
```bash
|
|
helm upgrade "$OS_RELEASE" "$OS_CHART" \
|
|
-n "$NS" \
|
|
-f values.yaml
|
|
```
|
|
|
|
## 24.5 Verify keystore
|
|
|
|
```bash
|
|
kubectl exec -n "$NS" "${OS_STS}-0" -c opensearch -- \
|
|
/usr/share/opensearch/bin/opensearch-keystore list
|
|
```
|
|
|
|
Expected entries:
|
|
|
|
```text
|
|
opensearch.notifications.core.email.cloud-operation.username
|
|
opensearch.notifications.core.email.cloud-operation.password
|
|
```
|
|
|
|
Verify all OpenSearch nodes if the chart injects keystore material per pod.
|
|
|
|
## 24.6 `Invalid Addresses` troubleshooting
|
|
|
|
Check:
|
|
|
|
```text
|
|
From address is valid
|
|
recipient is valid
|
|
SMTP host/port are correct
|
|
SSL/TLS mode matches the server
|
|
credentials exist in keystore
|
|
credentials are present on all relevant nodes
|
|
SMTP relay permits sender
|
|
SMTP account may use configured From address
|
|
recipient is allowed
|
|
```
|
|
|
|
---
|
|
|
|
# 25. Reliability, Backup, and Recovery Tests
|
|
|
|
## 25.1 Fluent Bit buffering recovery
|
|
|
|
After normal ingestion is stable:
|
|
|
|
1. Make Data Prepper temporarily unavailable.
|
|
2. Generate a few cPanel requests.
|
|
3. Confirm Fluent Bit retries.
|
|
4. Confirm filesystem backlog grows.
|
|
5. Restart Fluent Bit while Data Prepper is unavailable.
|
|
6. Restore Data Prepper.
|
|
7. Confirm buffered events drain.
|
|
8. Confirm documents reach OpenSearch.
|
|
|
|
Monitor:
|
|
|
|
```bash
|
|
du -sh /var/lib/fluent-bit/storage/
|
|
journalctl -u fluent-bit -f
|
|
```
|
|
|
|
## 25.2 OpenSearch pod recreation
|
|
|
|
Delete one pod:
|
|
|
|
```bash
|
|
kubectl delete pod -n "$NS" "${OS_STS}-0"
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
same StatefulSet ordinal recreated
|
|
same PVC reused
|
|
node rejoins cluster
|
|
```
|
|
|
|
Verify:
|
|
|
|
```bash
|
|
kubectl get pod,pvc -n "$NS" -o wide
|
|
```
|
|
|
|
## 25.3 Worker restart/failure test
|
|
|
|
Expected:
|
|
|
|
```text
|
|
remaining OpenSearch nodes continue serving
|
|
replica shard can be promoted if primary is lost
|
|
failed node/pod rejoins according to scheduling/storage policy
|
|
```
|
|
|
|
## 25.4 Snapshots
|
|
|
|
Shard replicas are not backups.
|
|
|
|
Snapshot storage should be independent from the OpenSearch cluster and Longhorn volumes.
|
|
|
|
Required operational practice:
|
|
|
|
```text
|
|
configure repository
|
|
schedule snapshots
|
|
retain snapshots
|
|
monitor failures
|
|
perform restore tests
|
|
```
|
|
|
|
A backup is not considered valid until restore has been tested.
|
|
|
|
---
|
|
|
|
# 26. Security Hardening
|
|
|
|
## 26.1 Credentials
|
|
|
|
Do not commit:
|
|
|
|
```text
|
|
OpenSearch admin password
|
|
kibanaserver password
|
|
Data Prepper logger password
|
|
SMTP password
|
|
data-source encryption key
|
|
private keys
|
|
Kubernetes Secret manifests containing real Base64 data
|
|
```
|
|
|
|
## 26.2 Identity separation
|
|
|
|
```text
|
|
admin -> security/cluster administration only
|
|
kibanaserver -> Dashboards backend only
|
|
logger -> Data Prepper ingestion only
|
|
human users -> read/search/dashboard permissions as required
|
|
```
|
|
|
|
## 26.3 Data Prepper ingress
|
|
|
|
The original implementation used:
|
|
|
|
```yaml
|
|
ssl: false
|
|
```
|
|
|
|
on the Data Prepper HTTP source and exposed it through NGINX NodePort. For production, protect the path with one or more of:
|
|
|
|
```text
|
|
TLS
|
|
HTTP basic authentication
|
|
private routing/VPN
|
|
source IP restriction
|
|
firewall ACL
|
|
Kubernetes NetworkPolicy
|
|
authenticated ingress
|
|
```
|
|
|
|
The Data Prepper HTTP source supports TLS and HTTP basic authentication.
|
|
|
|
## 26.4 OpenSearch ports
|
|
|
|
```text
|
|
9300 -> never expose publicly
|
|
9200 -> normally private/internal
|
|
5601 -> expose through controlled reverse proxy/Ingress
|
|
2021 -> restrict to approved log senders
|
|
```
|
|
|
|
## 26.5 Certificate management
|
|
|
|
Monitor:
|
|
|
|
```text
|
|
Not Before
|
|
Not After
|
|
SAN correctness
|
|
key/cert pair
|
|
CA trust
|
|
rotation date
|
|
```
|
|
|
|
`root-ca-key.pem` must remain offline/encrypted.
|
|
|
|
## 26.6 NetworkPolicy
|
|
|
|
If NetworkPolicy is used, explicitly permit:
|
|
|
|
```text
|
|
Dashboards -> OpenSearch :9200
|
|
Data Prepper -> OpenSearch :9200
|
|
OpenSearch nodes <-> OpenSearch nodes :9300
|
|
Ingress -> Dashboards :5601
|
|
Ingress -> Data Prepper :2021
|
|
```
|
|
|
|
---
|
|
|
|
# 27. Operational Monitoring
|
|
|
|
## 27.1 Kubernetes
|
|
|
|
Monitor:
|
|
|
|
```text
|
|
pod restarts
|
|
OOMKilled
|
|
CPU throttling
|
|
memory RSS
|
|
Pending pods
|
|
PVC usage
|
|
node pressure
|
|
scheduling failures
|
|
```
|
|
|
|
Commands:
|
|
|
|
```bash
|
|
kubectl get pods -n "$NS" -o wide
|
|
kubectl top pods -n "$NS"
|
|
kubectl get events -n "$NS" --sort-by=.lastTimestamp | tail -50
|
|
```
|
|
|
|
## 27.2 OpenSearch
|
|
|
|
Monitor:
|
|
|
|
```text
|
|
cluster health
|
|
node count
|
|
JVM heap pressure
|
|
GC
|
|
shard allocation
|
|
pending tasks
|
|
indexing latency
|
|
search latency
|
|
circuit breakers
|
|
disk watermarks
|
|
```
|
|
|
|
Useful APIs:
|
|
|
|
```http
|
|
GET _cluster/health
|
|
GET _cat/nodes?v
|
|
GET _cat/shards?v
|
|
GET _cat/indices?v
|
|
GET _cluster/pending_tasks
|
|
GET _nodes/stats
|
|
```
|
|
|
|
## 27.3 Fluent Bit
|
|
|
|
Monitor:
|
|
|
|
```text
|
|
input records
|
|
output records
|
|
retries
|
|
failed chunks
|
|
storage queue usage
|
|
service restarts
|
|
```
|
|
|
|
## 27.4 Data Prepper
|
|
|
|
Monitor:
|
|
|
|
```text
|
|
requestsReceived
|
|
successRequests
|
|
badRequests
|
|
requestsRejected
|
|
grokProcessingMismatch
|
|
grokProcessingErrors
|
|
date failures
|
|
OpenSearch document errors
|
|
```
|
|
|
|
## 27.5 Longhorn
|
|
|
|
Monitor:
|
|
|
|
```text
|
|
volume health
|
|
replica count
|
|
replica placement
|
|
disk usage
|
|
snapshot growth
|
|
attachment failures
|
|
```
|
|
|
|
---
|
|
|
|
# 28. Troubleshooting Matrix
|
|
|
|
| Symptom | Layer | Likely cause | First checks |
|
|
|---|---|---|---|
|
|
| Fluent Bit `extra indentation level found` | Fluent Bit config | classic config indentation / inline Lua | reported line; move Lua to separate file |
|
|
| Fluent Bit repeated `502 Bad Gateway` | Ingress/upstream | NGINX reachable but Data Prepper backend unhealthy | Data Prepper pod, Service, EndpointSlice, :2021 |
|
|
| Fluent Bit `connection refused` | Network/listener | endpoint unavailable | IP/port/firewall/listener |
|
|
| Data Prepper `Address already in use` | Pipeline | multiple HTTP sources on same port | keep one router HTTP source |
|
|
| Data Prepper `No definition for key 'PORT'` | Grok | unsupported pattern in tested 2.8 setup | use `POSINT` |
|
|
| Data Prepper pipeline YAML Boolean error | YAML | options at wrong hierarchy | indent under `grok:` correctly |
|
|
| `parsed: {}` | Parser | Grok did not match | compare raw line with regex |
|
|
| Wrong source in dedicated index | Routing | router/drop condition wrong | inspect `log_source`, route, new docs |
|
|
| Index exists, `docs.count=0` | Sink/ingestion | no successful documents | Data Prepper sink logs + pipeline metrics |
|
|
| Data Prepper `Request execution cancelled` / `I/O session is closed` | OpenSearch sink | DNS/TCP/TLS/auth/sink instability | test each layer to :9200 |
|
|
| OpenSearch pod `Running` but cluster unavailable | OpenSearch transport | TLS/discovery/cluster manager | pod logs, :9300, headless DNS, SAN |
|
|
| `No subject alternative DNS name ... headless` | TLS | SAN mismatch | regenerate certs with actual DNS SANs |
|
|
| `Cluster manager not present` | Cluster formation | transport TLS/discovery issue | headless Service, :9300, cert SAN, nodes_dn |
|
|
| `Unauthorized` | Authentication | wrong active credential | active Security config / user hash |
|
|
| `Resource 'admin' is reserved` | Security API | reserved resource cannot be changed by that API | update YAML + securityadmin using admin cert |
|
|
| Secret `subPath` `not a directory` | Kubernetes mount | Secret key name mismatch | `internal_users.yml` exactly |
|
|
| Dashboards `publicBaseUrl is not allowed` | Dashboards config | unsupported key in tested version | remove it |
|
|
| Dashboards 401 | Authentication | auth required/missing | security config / credential flow |
|
|
| Dashboards 404 after auth | Base path/routing | application not aware of ingress path | `basePath`, `rewriteBasePath` |
|
|
| Dashboards permission `indices:monitor/settings/get` | Authorization | backend role missing required permission | identify plugin + target index before grant |
|
|
| NGINX duplicate host/path | Ingress | catch-all route already exists | use unique host/path |
|
|
| `too_many_nested_clauses` | OpenSearch mapping/query | too many default searchable fields | restrict `index.query.default_field`; control mappings |
|
|
| SMTP `Invalid Addresses` | Notification | sender/recipient/relay/auth issue | address + keystore + relay policy |
|
|
|
|
---
|
|
|
|
# 29. Troubleshooting Decision Trees
|
|
|
|
## 29.1 Log ingestion
|
|
|
|
```text
|
|
Is the source file receiving new lines?
|
|
|
|
|
+-- NO --> troubleshoot cPanel/application logging
|
|
|
|
|
+-- YES
|
|
|
|
|
v
|
|
Did Fluent Bit discover/read the file?
|
|
|
|
|
+-- NO --> path, permissions, Tail DB, Read_from_Head policy
|
|
|
|
|
+-- YES
|
|
|
|
|
v
|
|
Does event contain source_file + log_source?
|
|
|
|
|
+-- NO --> Path_Key / Lua classification
|
|
|
|
|
+-- YES
|
|
|
|
|
v
|
|
Does Fluent Bit POST /logs successfully?
|
|
|
|
|
+-- 502 --> Ingress reached, upstream unhealthy
|
|
+-- refused --> listener/network/firewall issue
|
|
|
|
|
+-- success
|
|
|
|
|
v
|
|
Is Data Prepper 1/1 Running and :2021 listening?
|
|
|
|
|
+-- NO --> startup/pipeline configuration
|
|
|
|
|
+-- YES
|
|
|
|
|
v
|
|
Does the router match the expected log_source?
|
|
|
|
|
+-- NO --> route expression
|
|
|
|
|
+-- YES
|
|
|
|
|
v
|
|
Does the parser populate expected fields?
|
|
|
|
|
+-- NO --> Grok/date/key-value mismatch
|
|
|
|
|
+-- YES
|
|
|
|
|
v
|
|
Can Data Prepper reach OpenSearch :9200?
|
|
|
|
|
+-- NO --> DNS/TCP/TLS/CA
|
|
|
|
|
+-- YES
|
|
|
|
|
v
|
|
Does OpenSearch authenticate/authorize logger?
|
|
|
|
|
+-- NO --> user/role/index permissions
|
|
|
|
|
+-- YES
|
|
|
|
|
v
|
|
Does the expected index contain new documents?
|
|
```
|
|
|
|
## 29.2 OpenSearch deployment
|
|
|
|
```text
|
|
Pod Pending?
|
|
|
|
|
+--> node affinity
|
|
+--> hard anti-affinity
|
|
+--> taints/tolerations
|
|
+--> CPU/memory requests
|
|
+--> PVC/Longhorn
|
|
|
|
Pod starts but crashes?
|
|
|
|
|
+--> vm.max_map_count
|
|
+--> JVM heap / OOM
|
|
+--> certificate file permissions
|
|
+--> TLS key format
|
|
+--> opensearch.yml syntax
|
|
+--> bootstrap security
|
|
|
|
Pods run but cluster does not form?
|
|
|
|
|
+--> headless Service DNS
|
|
+--> transport :9300
|
|
+--> SANs
|
|
+--> nodes_dn
|
|
+--> discovery settings
|
|
|
|
REST HTTPS fails?
|
|
|
|
|
+--> REST TLS enabled
|
|
+--> cert/key paths
|
|
+--> CA
|
|
+--> hostname/SAN
|
|
+--> certificate expiry
|
|
|
|
Authentication fails?
|
|
|
|
|
+--> active Security index
|
|
+--> internal_users.yml
|
|
+--> securityadmin applied?
|
|
```
|
|
|
|
## 29.3 Dashboards
|
|
|
|
```text
|
|
Can browser reach NodePort?
|
|
|
|
|
+-- NO --> node firewall / NodePort / ingress-nginx Service
|
|
|
|
|
+-- YES
|
|
v
|
|
Does NGINX match /opensearch?
|
|
|
|
|
+-- NO --> Ingress rule / ingressClassName
|
|
|
|
|
+-- YES
|
|
v
|
|
Does Service have ready Dashboards endpoint?
|
|
|
|
|
+-- NO --> selector / pod readiness
|
|
|
|
|
+-- YES
|
|
v
|
|
Did Dashboards start without config validation errors?
|
|
|
|
|
+-- NO --> current + previous logs
|
|
|
|
|
+-- YES
|
|
v
|
|
Can Dashboards connect to OpenSearch with CA + kibanaserver?
|
|
|
|
|
+-- NO --> DNS/TLS/SAN/auth
|
|
|
|
|
+-- YES
|
|
v
|
|
Do assets/redirects stay under /opensearch?
|
|
|
|
|
+-- NO --> basePath/rewriteBasePath
|
|
|
|
|
+-- YES --> path is healthy
|
|
```
|
|
|
|
---
|
|
|
|
# 30. Daily / Weekly Operational Checklists
|
|
|
|
## 30.1 Daily
|
|
|
|
- [ ] All OpenSearch pods are `Running` and Ready.
|
|
- [ ] OpenSearch cluster health is expected.
|
|
- [ ] `_cat/nodes` lists all expected nodes.
|
|
- [ ] No recurring transport TLS errors.
|
|
- [ ] Data Prepper pod is Ready.
|
|
- [ ] Data Prepper has no sustained OpenSearch sink errors.
|
|
- [ ] Fluent Bit services are running.
|
|
- [ ] No repeated Fluent Bit 5xx/connection failures.
|
|
- [ ] New cPanel access documents are arriving.
|
|
- [ ] New Audit documents are arriving.
|
|
- [ ] `@timestamp` is current and sane.
|
|
- [ ] Grok processing error/mismatch rates are reviewed.
|
|
- [ ] Disk/PVC usage is below operational thresholds.
|
|
- [ ] Longhorn volumes are healthy.
|
|
- [ ] Dashboards loads and queries data.
|
|
|
|
## 30.2 Weekly
|
|
|
|
- [ ] Review OpenSearch JVM heap/GC trends.
|
|
- [ ] Review index growth and retention.
|
|
- [ ] Review Fluent Bit backlog utilization.
|
|
- [ ] Review Data Prepper rejected/bad requests.
|
|
- [ ] Review parsing failures/unhandled log sources.
|
|
- [ ] Review security/authentication failures.
|
|
- [ ] Review certificate expiry dates.
|
|
- [ ] Verify snapshots are succeeding.
|
|
- [ ] Periodically test restore procedure.
|
|
- [ ] Review alert quality and false positives.
|
|
|
|
---
|
|
|
|
# 31. Windows Fluent Bit Variant
|
|
|
|
This is an optional extension for downloaded Windows log files.
|
|
|
|
## 31.1 Recommended directory separation
|
|
|
|
```text
|
|
C:\fluent-bit\
|
|
├── logs\
|
|
├── state\
|
|
└── buffer\
|
|
```
|
|
|
|
Do not place the Tail DB inside a wildcard source directory if it could be ingested accidentally.
|
|
|
|
## 31.2 Example concept
|
|
|
|
```ini
|
|
[SERVICE]
|
|
Flush 1
|
|
Daemon Off
|
|
Log_Level info
|
|
storage.metrics On
|
|
storage.path C:\fluent-bit\buffer
|
|
storage.sync full
|
|
storage.backlog.mem_limit 500M
|
|
|
|
[INPUT]
|
|
Name tail
|
|
Path C:\fluent-bit\logs\*.log
|
|
Path_Key source_file
|
|
Tag downloaded.logs
|
|
DB C:\fluent-bit\state\tail.db
|
|
DB.Journal_Mode wal
|
|
Read_from_Head true
|
|
Refresh_Interval 5
|
|
Skip_Long_Lines true
|
|
Mem_Buf_Limit 512MB
|
|
Storage.Type filesystem
|
|
|
|
[OUTPUT]
|
|
Name http
|
|
Host 118.67.213.76
|
|
Port 31599
|
|
Format json
|
|
HTTP_Method POST
|
|
URI /logs
|
|
Match *
|
|
Retry_Limit no_limits
|
|
Storage.Total_Limit_Size 4G
|
|
```
|
|
|
|
Before ingesting all historical files:
|
|
|
|
1. Test one representative file.
|
|
2. Confirm existing content is read.
|
|
3. Restart Fluent Bit and confirm offset resume behavior.
|
|
4. Verify `source_file`.
|
|
5. Verify one test document in OpenSearch.
|
|
6. Only then expand the wildcard.
|
|
|
|
---
|
|
|
|
# 32. Cheat Sheet
|
|
|
|
## 32.1 Complete process summary
|
|
|
|
```text
|
|
1. Verify Kubernetes nodes, Longhorn, vm.max_map_count
|
|
2. Verify/generate Root CA, node certs, admin cert
|
|
3. Create Kubernetes Secrets
|
|
4. Configure OpenSearch Helm values
|
|
5. helm lint
|
|
6. helm template
|
|
7. kubectl --dry-run=server
|
|
8. helm upgrade --install OpenSearch
|
|
9. Verify pods/PVC/TLS/cluster health/security
|
|
10. Deploy Dashboards with CA + kibanaserver Secret
|
|
11. Configure /opensearch Ingress/basePath
|
|
12. Deploy Data Prepper with ONE HTTP :2021 router
|
|
13. Configure dedicated access/syslog/audit/catch-all pipelines
|
|
14. Configure Fluent Bit Tail DB + filesystem buffer + log_source
|
|
15. Verify /logs end-to-end
|
|
16. Create index templates/mappings
|
|
17. Create Dashboards workspace/index patterns
|
|
18. Build cPanel and Linux Audit dashboards
|
|
19. Configure anomaly detector / forecast / alerts
|
|
20. Configure SMTP keystore
|
|
21. Test failures, snapshots, and recovery
|
|
```
|
|
|
|
## 32.2 Kubernetes/OpenSearch quick commands
|
|
|
|
```bash
|
|
export NS=observability
|
|
export OS_RELEASE=brilliant-opensearch
|
|
export OS_STS=brilliant-opensearch-cluster-master
|
|
export OS_SVC=brilliant-opensearch-cluster-master
|
|
|
|
kubectl get pods -n "$NS" -o wide
|
|
kubectl get sts -n "$NS"
|
|
kubectl get svc -n "$NS"
|
|
kubectl get pvc -n "$NS"
|
|
kubectl get ingress -A
|
|
|
|
kubectl logs -n "$NS" "${OS_STS}-0" -c opensearch --tail=200
|
|
kubectl logs -n "$NS" "${OS_STS}-0" -c opensearch --previous --tail=200
|
|
|
|
kubectl port-forward -n "$NS" svc/"$OS_SVC" 9200:9200
|
|
curl -k -u 'admin:<PASSWORD>' https://localhost:9200/_cluster/health?pretty
|
|
curl -k -u 'admin:<PASSWORD>' 'https://localhost:9200/_cat/nodes?v'
|
|
```
|
|
|
|
## 32.3 Helm quick commands
|
|
|
|
```bash
|
|
helm lint "$OS_CHART" -f values.yaml
|
|
|
|
helm template "$OS_RELEASE" "$OS_CHART" \
|
|
-n "$NS" -f values.yaml > rendered-opensearch.yaml
|
|
|
|
kubectl apply --dry-run=server -f rendered-opensearch.yaml
|
|
|
|
helm upgrade --install "$OS_RELEASE" "$OS_CHART" \
|
|
-n "$NS" -f values.yaml
|
|
|
|
kubectl rollout status statefulset/"$OS_STS" -n "$NS"
|
|
```
|
|
|
|
## 32.4 TLS quick commands
|
|
|
|
```bash
|
|
openssl verify -CAfile root-ca.pem node-0.pem
|
|
openssl x509 -in node-0.pem -noout -subject -nameopt RFC2253
|
|
openssl x509 -in node-0.pem -noout -dates
|
|
openssl x509 -in node-0.pem -noout -text | sed -n '/Subject Alternative Name/,+2p'
|
|
```
|
|
|
|
## 32.5 Data Prepper quick commands
|
|
|
|
```bash
|
|
kubectl get pods -n observability -l app=data-prepper -o wide
|
|
kubectl get svc data-prepper -n observability -o wide
|
|
kubectl get endpointslice -n observability \
|
|
-l kubernetes.io/service-name=data-prepper -o wide
|
|
kubectl logs -n observability deploy/data-prepper --tail=200
|
|
|
|
kubectl exec -n observability deploy/data-prepper -- \
|
|
sh -c 'grep -i ":07E5" /proc/net/tcp /proc/net/tcp6 || true'
|
|
```
|
|
|
|
## 32.6 Fluent Bit quick commands
|
|
|
|
```bash
|
|
systemctl status fluent-bit --no-pager
|
|
journalctl -u fluent-bit --since '10 minutes ago' --no-pager
|
|
|
|
du -sh /var/lib/fluent-bit/storage/
|
|
ls -lh /var/lib/fluent-bit/state/
|
|
```
|
|
|
|
## 32.7 Test `/logs`
|
|
|
|
```bash
|
|
curl -i -X POST \
|
|
'http://118.67.213.76:31599/logs' \
|
|
-H 'Content-Type: application/json' \
|
|
--data '[{"log":"manual test","source_file":"manual","log_source":"test.manual"}]'
|
|
```
|
|
|
|
## 32.8 OpenSearch Dev Tools quick queries
|
|
|
|
```http
|
|
GET _cluster/health
|
|
GET _cat/nodes?v
|
|
GET _cat/indices/cpanel-*?v
|
|
|
|
GET cpanel-access-log-*/_count
|
|
{
|
|
"query": {
|
|
"range": {
|
|
"@timestamp": {
|
|
"gte": "now-24h",
|
|
"lte": "now"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
GET cpanel-access-log-*/_search
|
|
{
|
|
"size": 5,
|
|
"sort": [{"@timestamp": "desc"}]
|
|
}
|
|
```
|
|
|
|
## 32.9 DQL quick filters
|
|
|
|
```text
|
|
2xx:
|
|
parsed.http_status >= 200 and parsed.http_status < 300
|
|
|
|
4xx:
|
|
parsed.http_status >= 400 and parsed.http_status < 500
|
|
|
|
5xx:
|
|
parsed.http_status >= 500 and parsed.http_status < 600
|
|
|
|
Audit commands:
|
|
audit.log_type: "EXECVE"
|
|
|
|
Failed syscalls:
|
|
audit.log_type: "SYSCALL" AND syscall.success: "no"
|
|
|
|
Authentication activity:
|
|
audit.log_type: ("LOGIN" OR "USER_ACCT" OR "USER_START" OR "CRED_ACQ")
|
|
```
|
|
|
|
## 32.10 Error-to-layer quick map
|
|
|
|
```text
|
|
502 from Fluent Bit -> NGINX reachable; check Data Prepper upstream
|
|
connection refused -> listener/network
|
|
parsed: {} -> Grok mismatch
|
|
index docs.count=0 -> ingestion/sink not proven
|
|
SAN mismatch -> certificate DNS identity
|
|
Unauthorized -> authentication
|
|
security_exception -> authorization
|
|
404 after Dashboards auth-> basePath/routing
|
|
CrashLoop after config -> application validation/logs
|
|
```
|
|
|
|
---
|
|
|
|
# 33. Lessons Learned and Known Limitations
|
|
|
|
1. **Helm rendering is mandatory.** Inspect final StatefulSet/Deployment, not only `values.yaml`.
|
|
2. **`Running` is not enough.** Validate readiness, application logs, OpenSearch cluster health, and backend connectivity separately.
|
|
3. **Transport TLS and REST TLS are different boundaries.** `curl -k` against port 9200 cannot repair transport TLS on 9300.
|
|
4. **SANs matter.** Correct CA trust is insufficient when the Service/headless DNS is missing from the certificate.
|
|
5. **`kibanaserver` is a user; `kibana_server` is a role.** Do not confuse them.
|
|
6. **A mounted Security YAML file is not automatically the active Security index.** Apply changes appropriately with Security tooling.
|
|
7. **Secret `subPath` names match exact keys.** `internal_users.yaml` and `internal_users.yml` are different keys.
|
|
8. **One Data Prepper HTTP source per listener.** Route internally to downstream pipelines.
|
|
9. **A sink without `routes:` receives all events.** Define an explicit unknown route if the catch-all must contain only unhandled sources.
|
|
10. **Fluent Bit Tail DB is not the event queue.** It remembers offsets; filesystem storage buffers undelivered chunks.
|
|
11. **Unlimited retries do not mean unlimited durability.** A finite filesystem queue can still overflow and drop old chunks.
|
|
12. **`parsed: {}` means transport can be healthy while parsing failed.** Debug the raw line against the parser.
|
|
13. **Audit records are multi-record logical events.** Correlate using `audit.event_id`.
|
|
14. **Whitespace-splitting EXECVE arguments is not fully quote-aware.** Preserve `args_raw`.
|
|
15. **Dynamic Audit arguments can create mapping explosion.** Normalize high-value fields and restrict default searchable fields.
|
|
16. **Index existence does not prove document ingestion.** Always inspect `docs.count` and sample `_source`.
|
|
17. **NGINX Ingress routing is controller-wide.** Duplicate hostless `/` rules can conflict across namespaces.
|
|
18. **Dashboards path hosting requires application-aware basePath configuration.**
|
|
19. **Version-specific settings matter.** A configuration key accepted by Kibana or another OpenSearch Dashboards version may be rejected by 3.7.0.
|
|
20. **Least privilege first.** Investigate the action and target resource before granting broad permissions.
|
|
21. **OpenSearch replicas and Longhorn replicas consume storage independently.**
|
|
22. **OpenSearch shard replicas are not backups.** Snapshot and restore testing remain mandatory.
|
|
|
|
---
|
|
|
|
# 34. Final Definition of Done
|
|
|
|
The platform is considered complete only when all of the following are true.
|
|
|
|
## OpenSearch
|
|
|
|
- [ ] Three OpenSearch pods are Ready.
|
|
- [ ] Pods are on separate eligible Kubernetes nodes.
|
|
- [ ] Three 120Gi Longhorn PVCs are Bound.
|
|
- [ ] Transport TLS works with correct SANs.
|
|
- [ ] REST HTTPS works.
|
|
- [ ] Node certificate DNs match `nodes_dn`.
|
|
- [ ] Admin certificate DN matches `admin_dn`.
|
|
- [ ] Cluster manager is elected.
|
|
- [ ] `_cat/nodes` lists all three nodes.
|
|
- [ ] Cluster health is expected.
|
|
- [ ] Default/demo credentials are removed or changed.
|
|
- [ ] Real secrets are absent from Git.
|
|
|
|
## Dashboards
|
|
|
|
- [ ] Dashboards is Ready.
|
|
- [ ] CA is mounted.
|
|
- [ ] `kibanaserver` password comes from a Secret.
|
|
- [ ] TLS verification is `full`.
|
|
- [ ] `/opensearch` basePath works.
|
|
- [ ] Ingress does not conflict with Grafana `/`.
|
|
- [ ] Workspaces work with Security multi-tenancy disabled.
|
|
- [ ] Data source connection succeeds.
|
|
|
|
## Fluent Bit
|
|
|
|
- [ ] Tail paths are correct.
|
|
- [ ] Tail DB is persistent and writable.
|
|
- [ ] Filesystem buffer is writable.
|
|
- [ ] `source_file` is present.
|
|
- [ ] `log_source` is correct.
|
|
- [ ] `/logs` delivery succeeds.
|
|
- [ ] Retry/backlog behavior is tested.
|
|
- [ ] Queue usage is monitored.
|
|
|
|
## Data Prepper
|
|
|
|
- [ ] Only one HTTP source owns port 2021.
|
|
- [ ] Service and EndpointSlice point to the current pod.
|
|
- [ ] Health endpoint is healthy.
|
|
- [ ] Router matches known log sources.
|
|
- [ ] Catch-all contains only unhandled sources.
|
|
- [ ] Access Grok populates `parsed.*`.
|
|
- [ ] Audit parsers populate expected type-specific fields.
|
|
- [ ] Date processors create correct `@timestamp`.
|
|
- [ ] OpenSearch sink has no persistent errors.
|
|
|
|
## OpenSearch data
|
|
|
|
- [ ] `cpanel-access-log-*` contains documents.
|
|
- [ ] `cpanel-system-syslog-*` contains expected syslog only.
|
|
- [ ] `cpanel-audit-log-*` contains Audit records.
|
|
- [ ] `cpanel-all-log-*` exposes unknown/unhandled sources.
|
|
- [ ] cPanel access field mappings are correct.
|
|
- [ ] Audit default query fields are controlled.
|
|
- [ ] Retention/ISM strategy is defined.
|
|
|
|
## Dashboards and alerting
|
|
|
|
- [ ] cPanel Access Overview exists.
|
|
- [ ] Linux Server Audit & Security Overview exists.
|
|
- [ ] Anomaly detector runs and produces sensible output.
|
|
- [ ] Forecasting runs if required.
|
|
- [ ] Monitor and trigger exist.
|
|
- [ ] SMTP credentials exist in OpenSearch keystore.
|
|
- [ ] Test notification succeeds.
|
|
- [ ] Real alert action succeeds.
|
|
|
|
## Operations
|
|
|
|
- [ ] OpenSearch pod-failure test completed.
|
|
- [ ] Fluent Bit/Data Prepper interruption recovery tested.
|
|
- [ ] Snapshot repository configured.
|
|
- [ ] Snapshot schedule configured.
|
|
- [ ] Restore test completed.
|
|
- [ ] Certificate-expiry monitoring configured.
|
|
- [ ] Daily/weekly checks assigned to operations.
|
|
|
|
---
|
|
|
|
## Verification References Used for This Consolidation
|
|
|
|
This combined SOP was cross-checked against the official documentation for:
|
|
|
|
- OpenSearch Helm installation and chart configuration.
|
|
- OpenSearch Security YAML configuration and `securityadmin.sh` behavior.
|
|
- OpenSearch index replica settings and cluster health semantics.
|
|
- OpenSearch Dashboards Workspaces and multi-tenancy compatibility.
|
|
- OpenSearch Dashboards multiple data sources.
|
|
- Data Prepper HTTP source, Grok processor, key-value processor, conditional routing, server API port, and OpenSearch sink.
|
|
- Fluent Bit Tail input, filesystem buffering, retry semantics, and output queue limits.
|
|
- Kubernetes StatefulSet/headless Service behavior and Kubernetes Secret/volume semantics.
|
|
|
|
When upgrading any major/minor component, rerun the same validation flow against the documentation for the exact version before applying old configuration unchanged.
|
|
|
|
---
|
|
|
|
**Prepared by: Md. Mosiur Rahnman**
|