diff --git a/OpenSearch_cPanel_Observability_Complete_SOP_v2.md b/OpenSearch_cPanel_Observability_Complete_SOP_v2.md new file mode 100644 index 0000000..9c642d0 --- /dev/null +++ b/OpenSearch_cPanel_Observability_Complete_SOP_v2.md @@ -0,0 +1,4924 @@ +# OpenSearch + cPanel/Linux Observability Platform — Complete SOP + +> **Canonical combined runbook** for deploying and operating OpenSearch on Kubernetes, OpenSearch Dashboards, Fluent Bit, Data Prepper, cPanel/Linux log ingestion, Linux Audit ingestion, dashboards, anomaly detection, forecasting, alerting, and troubleshooting. +> +> **Prepared by:** Md. Mosiur Rahnman +> **Document type:** Standard Operating Procedure (SOP) / Future Reference Runbook +> **Primary environment documented:** Kubernetes/K3s, OpenSearch 3.7.0, OpenSearch Dashboards 3.7.0, Data Prepper 2.8.0, Longhorn, NGINX Ingress, cPanel/Linux +> **Primary namespace:** `observability` + +--- + +## Document Control + +| Item | Value | +|---|---| +| Prepared by | Md. Mosiur Rahnman | +| Scope | OpenSearch platform + cPanel/Linux log observability | +| OpenSearch version used in the implementation | 3.7.0 | +| OpenSearch Dashboards version used | 3.7.0 | +| Data Prepper version used | 2.8.0 | +| Kubernetes namespace | `observability` | +| OpenSearch Helm release | `brilliant-opensearch` | +| OpenSearch cluster name | `brilliant-opensearch-cluster` | +| OpenSearch StatefulSet | `brilliant-opensearch-cluster-master` | +| OpenSearch Service | `brilliant-opensearch-cluster-master` | +| Data Prepper Service | `data-prepper` | +| StorageClass | `longhorn` | +| OpenSearch pod count | 3 | +| OpenSearch PVC size | 120Gi per pod | +| Dashboards external path | `/opensearch` | +| Data Prepper ingestion path | `/logs` | +| NGINX HTTP NodePort observed | `31599` | +| NGINX HTTPS NodePort observed | `31186` | +| cPanel access log | `/usr/local/cpanel/logs/access_log` | + +> [!IMPORTANT] +> Values such as node IPs, NodePorts, release names, Service names, chart paths, certificate DNs, and storage topology are **environment-specific**. Verify them before reusing this SOP in another cluster. + +--- + +## Table of Contents + +1. [Purpose and Scope](#1-purpose-and-scope) +2. [Final Architecture](#2-final-architecture) +3. [Core Design Principles](#3-core-design-principles) +4. [Environment Variables and Naming](#4-environment-variables-and-naming) +5. [Pre-Deployment Prerequisites](#5-pre-deployment-prerequisites) +6. [OpenSearch PKI and TLS](#6-opensearch-pki-and-tls) +7. [Kubernetes Secrets](#7-kubernetes-secrets) +8. [OpenSearch Helm Configuration](#8-opensearch-helm-configuration) +9. [Render, Validate, and Deploy OpenSearch](#9-render-validate-and-deploy-opensearch) +10. [OpenSearch Security Initialization and Admin Password](#10-opensearch-security-initialization-and-admin-password) +11. [OpenSearch Cluster Verification](#11-opensearch-cluster-verification) +12. [OpenSearch Dashboards Deployment](#12-opensearch-dashboards-deployment) +13. [Data Prepper — Full Deployment, Routing, Parsing, and Verification](#13-data-prepper--full-deployment-routing-parsing-and-verification) +14. [Fluent Bit — Full cPanel/Linux Collection Procedure](#14-fluent-bit--full-cpanellinux-collection-procedure) +15. [cPanel Access Log Parsing](#15-cpanel-access-log-parsing) +16. [Linux Syslog and Other Log Sources](#16-linux-syslog-and-other-log-sources) +17. [Linux Audit Log Parsing](#17-linux-audit-log-parsing) +18. [OpenSearch Index Templates and Mapping](#18-opensearch-index-templates-and-mapping) +19. [End-to-End Verification](#19-end-to-end-verification) +20. [OpenSearch Dashboards Data Source and Workspace](#20-opensearch-dashboards-data-source-and-workspace) +21. [cPanel Access Dashboard](#21-cpanel-access-dashboard) +22. [Linux Audit Security Dashboard](#22-linux-audit-security-dashboard) +23. [Anomaly Detection, Forecasting, and Alerting](#23-anomaly-detection-forecasting-and-alerting) +24. [SMTP Notification Configuration](#24-smtp-notification-configuration) +25. [Reliability, Backup, and Recovery Tests](#25-reliability-backup-and-recovery-tests) +26. [Security Hardening](#26-security-hardening) +27. [Operational Monitoring](#27-operational-monitoring) +28. [Troubleshooting Matrix](#28-troubleshooting-matrix) +29. [Troubleshooting Decision Trees](#29-troubleshooting-decision-trees) +30. [Daily / Weekly Operational Checklists](#30-daily--weekly-operational-checklists) +31. [Windows Fluent Bit Variant](#31-windows-fluent-bit-variant) +32. [Cheat Sheet](#32-cheat-sheet) +33. [Lessons Learned and Known Limitations](#33-lessons-learned-and-known-limitations) +34. [Final Definition of Done](#34-final-definition-of-done) + +--- + +# 1. Purpose and Scope + +This SOP consolidates the complete implementation into one operational reference. It covers the path from Kubernetes infrastructure through searchable and visualized cPanel/Linux logs: + +```text +cPanel / Linux Servers + | + | log files + v +Fluent Bit + - Tail input + - Offset DB + - Filesystem buffering + - source_file + - log_source classification + | + | HTTP/HTTPS POST /logs + v +NGINX Ingress + | + v +Data Prepper :2021 + - One HTTP source + - Router + - Grok/date/key-value processors + | + | HTTPS :9200 + v +OpenSearch 3-node cluster + - Longhorn PVCs + - REST TLS + - Transport TLS + - Security plugin + | + v +OpenSearch Dashboards + - Discover / Logs + - Workspaces + - Dashboards + - Anomaly Detection + - Forecasting + - Alerting + - Email notifications +``` + +This document preserves the tested implementation while separating: + +- **Production baseline** — settings that should normally remain enabled. +- **Environment-specific values** — values that must be rechecked before reuse. +- **Diagnostic workarounds** — temporary troubleshooting settings that should not become permanent defaults. +- **Known limitations** — parser or reliability behaviors that need to be understood rather than hidden. + +--- + +# 2. Final Architecture + +## 2.1 Kubernetes/OpenSearch topology + +```text + +--------------------------------------+ + | Kubernetes / K3s | + | Namespace: observability | + | | +Browser | OpenSearch Dashboards | + | | :5601 | + | HTTP/HTTPS | | HTTPS | + v | v | +NGINX Ingress ---------->| OpenSearch Service :9200 | + ^ | | | + | | +------+------+------+ | + | | | | | | + | | pod-0 pod-1 pod-2 | + | | | | | | + | | PVC-0 PVC-1 PVC-2 | + | | \________ Transport TLS :9300 ____/| + | | | + | | Data Prepper :2021 | + | | | HTTPS :9200 | + | | +----------------------------->| + | +--------------------------------------+ + | + | HTTP/HTTPS /logs + | +Fluent Bit on cPanel/Linux servers +``` + +## 2.2 Log-routing architecture + +```text +Fluent Bit + | + | JSON + source_file + log_source + v +Data Prepper router + | + +--> cpanel.access_log --> access pipeline --> cpanel-access-log-YYYY.MM.dd + | + +--> system.syslog ----> syslog pipeline --> cpanel-system-syslog-YYYY.MM.dd + | + +--> audit.audit.log --> audit pipeline ---> cpanel-audit-log-YYYY.MM.dd + | + +--> unknown sources --> catch-all --------> cpanel-all-log-YYYY.MM.dd +``` + +--- + +# 3. Core Design Principles + +1. **Validate one layer at a time.** Do not debug OpenSearch if Fluent Bit is not reading the source file. +2. **Render Helm before applying it.** A valid `values.yaml` can still render an invalid or unintended Kubernetes manifest. +3. **Keep transport, parsing, routing, and storage failures separate.** `parsed: {}` is a parser problem; `502` is normally an upstream availability problem. +4. **Use one Data Prepper HTTP listener per port.** Multiple pipelines must not bind `:2021` independently. +5. **Classify logs early.** Fluent Bit knows the original file path, so create `source_file` and normalized `log_source` before transport. +6. **Use dedicated downstream pipelines for different formats.** cPanel access logs, syslog, and Auditd have fundamentally different schemas. +7. **Preserve the raw log.** Parsing should enrich the event, not destroy the original message. +8. **Use least privilege.** Do not reuse the OpenSearch admin credential for Dashboards or ingestion. +9. **Treat TLS trust and TLS hostname validation as different checks.** A trusted CA does not fix a SAN mismatch. +10. **Treat Kubernetes `Running` and application health as different states.** A pod can run while OpenSearch cluster formation, authentication, or authorization is broken. +11. **OpenSearch shard replicas are not backups.** Use snapshots to independent storage. +12. **Base64 is encoding, not encryption.** Kubernetes Secret output must still be treated as sensitive. +13. **Do not commit real passwords, private keys, SMTP credentials, or encryption master keys to Git.** + +--- + +# 4. Environment Variables and Naming + +Use variables in shell sessions to reduce copy/paste mistakes. + +```bash +export NS="observability" +export OS_RELEASE="brilliant-opensearch" +export OS_CLUSTER="brilliant-opensearch-cluster" +export OS_STS="brilliant-opensearch-cluster-master" +export OS_SVC="brilliant-opensearch-cluster-master" +export OS_HEADLESS_SVC="brilliant-opensearch-cluster-master-headless" +export DP_SVC="data-prepper" +export DP_PORT="2021" +export DASH_BASEPATH="/opensearch" +export INGRESS_HTTP_NODEPORT="31599" +export INGRESS_HTTPS_NODEPORT="31186" +``` + +Check the real values instead of assuming them: + +```bash +kubectl get ns +helm list -A +kubectl get sts,deploy,svc,ingress -A +kubectl get nodes -o wide +``` + +For the local Helm chart used in the implementation: + +```bash +export OS_CHART="${HOME}/monitoring/helm-charts/charts/opensearch" +``` + +Verify it exists: + +```bash +test -f "$OS_CHART/Chart.yaml" && echo "OpenSearch chart found" +``` + +--- + +# 5. Pre-Deployment Prerequisites + +## 5.1 Client tools + +Verify required tools: + +```bash +kubectl version --client +helm version +openssl version +curl --version +``` + +Optional but useful: + +```bash +jq --version +yq --version +``` + +## 5.2 Kubernetes nodes + +```bash +kubectl get nodes -o wide +kubectl get nodes --show-labels +kubectl describe node k8s-storage-01 | sed -n '/Taints:/,/Unschedulable:/p' +``` + +Target placement from the implementation: + +```text +k8s-storage-01 -> eligible, dedicated=storage:NoSchedule +k8s-worker-01 -> eligible +k8s-worker-02 -> eligible +k8s-cp-01 -> excluded from OpenSearch +``` + +If `k8s-storage-01` is tainted: + +```text +dedicated=storage:NoSchedule +``` + +OpenSearch requires a matching toleration if that node should be eligible. + +## 5.3 Longhorn + +```bash +kubectl get storageclass +kubectl get pods -n longhorn-system +``` + +Expected StorageClass: + +```text +longhorn +``` + +Storage planning for the documented cluster: + +```text +3 OpenSearch pods x 120Gi = 360Gi logical PVC capacity +``` + +Longhorn replication consumes additional physical storage. OpenSearch shard replication consumes additional logical data copies as well; they are separate redundancy layers. + +## 5.4 `vm.max_map_count` + +Check every eligible OpenSearch node: + +```bash +for node in k8s-storage-01 k8s-worker-01 k8s-worker-02; do + echo "=== $node ===" + ssh "$node" 'sysctl vm.max_map_count' +done +``` + +Target used by the Helm chart: + +```text +vm.max_map_count = 262144 +``` + +Prefer configuring the kernel permanently on the host: + +```bash +cat >/etc/sysctl.d/99-opensearch.conf <<'EOF_SYSCTL' +vm.max_map_count=262144 +EOF_SYSCTL + +sysctl --system +``` + +If the hosts are already configured, disable privileged sysctl init mechanisms in Helm. Do not enable multiple sysctl methods unnecessarily. + +## 5.5 Namespace + +```bash +kubectl create namespace "$NS" --dry-run=client -o yaml | kubectl apply -f - +``` + +--- + +# 6. OpenSearch PKI and TLS + +## 6.1 Certificate roles + +Use separate certificate purposes: + +```text +Root CA + ├── signs node certificates + └── signs admin certificate + +Node certificates + ├── REST TLS server identity + ├── transport TLS server identity + └── transport TLS client identity + +Admin certificate + └── privileged OpenSearch Security administration +``` + +Never mount the Root CA private key into OpenSearch. + +```text +root-ca-key.pem -> offline/encrypted CA material only +root-ca.pem -> safe to distribute as trust anchor +``` + +## 6.2 Expected PKI files + +```text +root-ca.pem +root-ca-key.pem + +admin.pem +admin-key.pem + +node-0.pem +node-0-key.pem +node-1.pem +node-1-key.pem +node-2.pem +node-2-key.pem +``` + +## 6.3 SAN design + +Use a short certificate CN and put Kubernetes DNS identities in SANs. + +Common Service DNS names: + +```text +brilliant-opensearch-cluster-master +brilliant-opensearch-cluster-master.observability +brilliant-opensearch-cluster-master.observability.svc +brilliant-opensearch-cluster-master.observability.svc.cluster.local + +brilliant-opensearch-cluster-master-headless +brilliant-opensearch-cluster-master-headless.observability +brilliant-opensearch-cluster-master-headless.observability.svc +brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local +``` + +Pod-specific names should also be included where transport communication uses them: + +```text +brilliant-opensearch-cluster-master-0.brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local +brilliant-opensearch-cluster-master-1.brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local +brilliant-opensearch-cluster-master-2.brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local +``` + +## 6.4 Optional OpenSSL generation template + +If certificates already exist, skip generation and perform verification only. + +### Root CA + +```bash +openssl genrsa -out root-ca-key.pem 4096 + +openssl req -x509 -new -sha256 \ + -key root-ca-key.pem \ + -days 3650 \ + -out root-ca.pem \ + -subj '/C=BD/ST=Dhaka/L=Dhaka/O=Brilliant/OU=OpenSearch CA/CN=Brilliant OpenSearch Root CA' +``` + +### Admin certificate + +```bash +openssl genrsa -out admin-key-rsa.pem 2048 +openssl pkcs8 -topk8 -nocrypt \ + -in admin-key-rsa.pem \ + -out admin-key.pem + +openssl req -new \ + -key admin-key.pem \ + -out admin.csr \ + -subj '/C=BD/ST=Dhaka/L=Dhaka/O=Brilliant/OU=OpenSearch Administrators/CN=opensearch-admin' + +cat >admin.ext <<'EOF_ADMIN_EXT' +basicConstraints=CA:FALSE +keyUsage=digitalSignature,keyEncipherment +extendedKeyUsage=clientAuth +EOF_ADMIN_EXT + +openssl x509 -req \ + -in admin.csr \ + -CA root-ca.pem \ + -CAkey root-ca-key.pem \ + -CAcreateserial \ + -out admin.pem \ + -days 825 \ + -sha256 \ + -extfile admin.ext +``` + +### Node certificate example for pod 0 + +Create a SAN file containing the actual Service and pod DNS names: + +```bash +cat >node-0.ext <<'EOF_NODE0_EXT' +basicConstraints=CA:FALSE +keyUsage=digitalSignature,keyEncipherment +extendedKeyUsage=serverAuth,clientAuth +subjectAltName=@alt_names + +[alt_names] +DNS.1=brilliant-opensearch-cluster-master +DNS.2=brilliant-opensearch-cluster-master.observability +DNS.3=brilliant-opensearch-cluster-master.observability.svc +DNS.4=brilliant-opensearch-cluster-master.observability.svc.cluster.local +DNS.5=brilliant-opensearch-cluster-master-headless +DNS.6=brilliant-opensearch-cluster-master-headless.observability +DNS.7=brilliant-opensearch-cluster-master-headless.observability.svc +DNS.8=brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local +DNS.9=brilliant-opensearch-cluster-master-0.brilliant-opensearch-cluster-master-headless +DNS.10=brilliant-opensearch-cluster-master-0.brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local +EOF_NODE0_EXT + +openssl genrsa -out node-0-key-rsa.pem 2048 +openssl pkcs8 -topk8 -nocrypt \ + -in node-0-key-rsa.pem \ + -out node-0-key.pem + +openssl req -new \ + -key node-0-key.pem \ + -out node-0.csr \ + -subj '/C=BD/ST=Dhaka/L=Dhaka/O=Brilliant/OU=OpenSearch Nodes/CN=opensearch-master-0' + +openssl x509 -req \ + -in node-0.csr \ + -CA root-ca.pem \ + -CAkey root-ca-key.pem \ + -CAcreateserial \ + -out node-0.pem \ + -days 825 \ + -sha256 \ + -extfile node-0.ext +``` + +Repeat with the correct CN and pod-specific SAN for node 1 and node 2. + +## 6.5 Certificate verification + +Verify chain: + +```bash +openssl verify -CAfile root-ca.pem admin.pem +openssl verify -CAfile root-ca.pem node-0.pem +openssl verify -CAfile root-ca.pem node-1.pem +openssl verify -CAfile root-ca.pem node-2.pem +``` + +Verify subjects: + +```bash +openssl x509 -in admin.pem -noout -subject -nameopt RFC2253 +openssl x509 -in node-0.pem -noout -subject -nameopt RFC2253 +openssl x509 -in node-1.pem -noout -subject -nameopt RFC2253 +openssl x509 -in node-2.pem -noout -subject -nameopt RFC2253 +``` + +Verify SANs: + +```bash +openssl x509 -in node-0.pem -noout -text | sed -n '/Subject Alternative Name/,+2p' +``` + +Verify EKU: + +```bash +openssl x509 -in node-0.pem -noout -text | sed -n '/Extended Key Usage/,+2p' +``` + +Verify certificate/key match: + +```bash +openssl x509 -in node-0.pem -pubkey -noout | sha256sum +openssl pkey -in node-0-key.pem -pubout | sha256sum +``` + +The hashes must match. + +Verify validity: + +```bash +openssl x509 -in node-0.pem -noout -dates +``` + +--- + +# 7. Kubernetes Secrets + +## 7.1 Initial admin password + +Do not place the real password in Git. + +```bash +read -rsp 'Initial OpenSearch admin password: ' OPENSEARCH_INITIAL_ADMIN_PASSWORD +echo + +kubectl create secret generic opensearch-secret \ + -n "$NS" \ + --from-literal=OPENSEARCH_INITIAL_ADMIN_PASSWORD="$OPENSEARCH_INITIAL_ADMIN_PASSWORD" \ + --dry-run=client -o yaml | kubectl apply -f - + +unset OPENSEARCH_INITIAL_ADMIN_PASSWORD +``` + +## 7.2 Node certificate Secrets + +```bash +kubectl create secret generic opensearch-node-0-cert \ + -n "$NS" \ + --from-file=node-0.pem \ + --from-file=node-0-key.pem \ + --from-file=root-ca.pem \ + --dry-run=client -o yaml | kubectl apply -f - + +kubectl create secret generic opensearch-node-1-cert \ + -n "$NS" \ + --from-file=node-1.pem \ + --from-file=node-1-key.pem \ + --from-file=root-ca.pem \ + --dry-run=client -o yaml | kubectl apply -f - + +kubectl create secret generic opensearch-node-2-cert \ + -n "$NS" \ + --from-file=node-2.pem \ + --from-file=node-2-key.pem \ + --from-file=root-ca.pem \ + --dry-run=client -o yaml | kubectl apply -f - +``` + +## 7.3 Admin certificate Secret + +```bash +kubectl create secret generic opensearch-admin-cert \ + -n "$NS" \ + --from-file=admin.pem \ + --from-file=admin-key.pem \ + --from-file=root-ca.pem \ + --dry-run=client -o yaml | kubectl apply -f - +``` + +## 7.4 Verify Secret metadata without dumping data + +```bash +kubectl get secret -n "$NS" \ + opensearch-secret \ + opensearch-node-0-cert \ + opensearch-node-1-cert \ + opensearch-node-2-cert \ + opensearch-admin-cert +``` + +Avoid: + +```bash +kubectl get secret -o yaml +``` + +when the output would expose sensitive Base64 data in terminals, tickets, screenshots, or chat logs. + +--- + +# 8. OpenSearch Helm Configuration + +## 8.1 Resource design + +Per OpenSearch pod: + +```text +CPU request: 2 +CPU limit: 4 +Memory request: 6Gi +Memory limit: 8Gi +JVM heap: 3Gi +PVC: 120Gi +``` + +## 8.2 Helm values baseline + +The exact chart schema must always be checked with the pinned chart: + +```bash +helm show values "$OS_CHART" > /tmp/opensearch-default-values.yaml +``` + +Critical values for this environment: + +```yaml +clusterName: brilliant-opensearch-cluster +nodeGroup: master +singleNode: false +masterService: brilliant-opensearch-cluster-master + +roles: + - master + - ingest + - data + - remote_cluster_client + +replicas: 3 + +image: + repository: opensearchproject/opensearch + tag: "3.7.0" + pullPolicy: IfNotPresent + +opensearchJavaOpts: "-Xms3g -Xmx3g" + +resources: + requests: + cpu: "2" + memory: 6Gi + limits: + cpu: "4" + memory: 8Gi + +persistence: + enabled: true + storageClass: longhorn + accessModes: + - ReadWriteOnce + size: 120Gi + +antiAffinity: hard +antiAffinityTopologyKey: kubernetes.io/hostname +podManagementPolicy: Parallel + +protocol: https +httpPort: 9200 +transportPort: 9300 +metricsPort: 9600 + +podSecurityContext: + fsGroup: 1000 + runAsUser: 1000 + +securityContext: + runAsNonRoot: true + runAsUser: 1000 + capabilities: + drop: + - ALL + +envFrom: + - secretRef: + name: opensearch-secret + +secretMounts: [] + +sysctl: + enabled: false + +sysctlInit: + enabled: false +``` + +> [!NOTE] +> If the worker nodes are **not** preconfigured with `vm.max_map_count=262144`, enable exactly one supported sysctl method. Do not enable both pod-level sysctl and a privileged sysctl init container unless you have a specific reason. + +## 8.3 Node scheduling + +Storage taint toleration: + +```yaml +tolerations: + - key: dedicated + operator: Equal + value: storage + effect: NoSchedule +``` + +Use labels/affinity to include only the intended storage/worker nodes. Do not rely on anti-affinity to exclude the control plane; anti-affinity controls pod co-location, not arbitrary node blacklisting. + +Verify rendered affinity before install. + +## 8.4 Per-pod certificate selection + +The main OpenSearch container should see only its selected runtime certificate set. + +```yaml +extraVolumes: + - name: node-0-source + secret: + secretName: opensearch-node-0-cert + defaultMode: 0440 + + - name: node-1-source + secret: + secretName: opensearch-node-1-cert + defaultMode: 0440 + + - name: node-2-source + secret: + secretName: opensearch-node-2-cert + defaultMode: 0440 + + - name: runtime-certs + emptyDir: {} + +extraVolumeMounts: + - name: runtime-certs + mountPath: /usr/share/opensearch/config/certs + readOnly: true +``` + +Init container: + +```yaml +extraInitContainers: + - name: select-node-certificate + image: busybox:1.36 + command: + - sh + - -ec + - | + ordinal="${HOSTNAME##*-}" + + case "$ordinal" in + 0|1|2) ;; + *) + echo "Unexpected StatefulSet ordinal: $ordinal" >&2 + exit 1 + ;; + esac + + src="/source/node-${ordinal}" + cert="${src}/node-${ordinal}.pem" + key="${src}/node-${ordinal}-key.pem" + ca="${src}/root-ca.pem" + + test -r "$cert" + test -r "$key" + test -r "$ca" + + cp "$cert" /runtime/node.pem + cp "$key" /runtime/node-key.pem + cp "$ca" /runtime/root-ca.pem + + chown 1000:1000 /runtime/node.pem /runtime/node-key.pem /runtime/root-ca.pem + chmod 0644 /runtime/node.pem /runtime/root-ca.pem + chmod 0600 /runtime/node-key.pem + + ls -l /runtime + volumeMounts: + - name: node-0-source + mountPath: /source/node-0 + readOnly: true + - name: node-1-source + mountPath: /source/node-1 + readOnly: true + - name: node-2-source + mountPath: /source/node-2 + readOnly: true + - name: runtime-certs + mountPath: /runtime +``` + +> [!IMPORTANT] +> The `chown 1000:1000` matches the documented chart runtime user. Verify the actual image UID/GID before copying this into a different image or chart. + +```bash +kubectl exec -n "$NS" "${OS_STS}-0" -c opensearch -- id +``` + +The init container and main container mount the same `emptyDir` at different paths: + +```text +init container: /runtime/node.pem +main container: /usr/share/opensearch/config/certs/node.pem +``` + +These are the same underlying file. + +## 8.5 OpenSearch TLS configuration + +Use fixed runtime paths: + +```yaml +config: + opensearch.yml: | + cluster.name: brilliant-opensearch-cluster + network.host: 0.0.0.0 + + plugins.security.ssl.transport.pemcert_filepath: certs/node.pem + plugins.security.ssl.transport.pemkey_filepath: certs/node-key.pem + plugins.security.ssl.transport.pemtrustedcas_filepath: certs/root-ca.pem + plugins.security.ssl.transport.enforce_hostname_verification: true + + plugins.security.ssl.http.enabled: true + plugins.security.ssl.http.pemcert_filepath: certs/node.pem + plugins.security.ssl.http.pemkey_filepath: certs/node-key.pem + plugins.security.ssl.http.pemtrustedcas_filepath: certs/root-ca.pem + + plugins.security.nodes_dn: + - 'CN=opensearch-master-0,OU=OpenSearch Nodes,O=Brilliant,L=Dhaka,ST=Dhaka,C=BD' + - 'CN=opensearch-master-1,OU=OpenSearch Nodes,O=Brilliant,L=Dhaka,ST=Dhaka,C=BD' + - 'CN=opensearch-master-2,OU=OpenSearch Nodes,O=Brilliant,L=Dhaka,ST=Dhaka,C=BD' + + plugins.security.authcz.admin_dn: + - 'CN=opensearch-admin,OU=OpenSearch Administrators,O=Brilliant,L=Dhaka,ST=Dhaka,C=BD' +``` + +Use the **actual RFC2253 certificate DNs** from `openssl x509`, not guessed strings. + +## 8.6 Temporary transport hostname-verification workaround + +If logs show: + +```text +No subject alternative DNS name matching brilliant-opensearch-cluster-master-headless found +``` + +then the certificate SAN is wrong for the DNS name used by transport communication. + +Temporary diagnostic setting: + +```yaml +plugins.security.ssl.transport.enforce_hostname_verification: false +``` + +This is not the desired production state. Regenerate node certificates with correct SANs, deploy them, verify transport TLS, then restore: + +```yaml +plugins.security.ssl.transport.enforce_hostname_verification: true +``` + +--- + +# 9. Render, Validate, and Deploy OpenSearch + +Use this order every time: + +```text +values.yaml + -> helm lint + -> helm template + -> inspect rendered YAML + -> Kubernetes server-side dry-run + -> helm upgrade/install + -> runtime validation +``` + +## 9.1 Helm lint + +```bash +helm lint "$OS_CHART" -f values.yaml +``` + +## 9.2 Render + +```bash +helm template "$OS_RELEASE" "$OS_CHART" \ + -n "$NS" \ + -f values.yaml \ + > rendered-opensearch.yaml +``` + +## 9.3 Inspect rendered manifest + +```bash +grep -nE 'kind: StatefulSet|replicas:|storageClassName:|120Gi|runtime-certs|node-[012]-source|select-node-certificate|nodes_dn|admin_dn|9200|9300' rendered-opensearch.yaml +``` + +Verify: + +```text +[ ] replicas = 3 +[ ] image/version pinned +[ ] requests/limits present +[ ] 120Gi Longhorn PVC template +[ ] hard anti-affinity +[ ] expected toleration / affinity +[ ] runtime-certs emptyDir +[ ] init container selects certificate by ordinal +[ ] main container mounts runtime-certs only +[ ] node DNs and admin DN correct +[ ] REST port 9200 +[ ] transport port 9300 +[ ] bootstrap Secret reference present +``` + +## 9.4 Kubernetes API dry-run + +```bash +kubectl apply --dry-run=server -f rendered-opensearch.yaml +``` + +## 9.5 Deploy + +```bash +helm upgrade --install "$OS_RELEASE" "$OS_CHART" \ + -n "$NS" \ + -f values.yaml +``` + +## 9.6 Observe rollout + +```bash +kubectl get pods -n "$NS" -w +``` + +In another terminal: + +```bash +kubectl rollout status statefulset/"$OS_STS" -n "$NS" +``` + +--- + +# 10. OpenSearch Security Initialization and Admin Password + +## 10.1 Important distinction + +These are not the same thing: + +```text +Kubernetes Secret / OPENSEARCH_INITIAL_ADMIN_PASSWORD + != +active OpenSearch Security index +``` + +For new demo-security initialization, `OPENSEARCH_INITIAL_ADMIN_PASSWORD` is used during setup. Once the Security configuration is initialized, changing an environment variable or mounted YAML file does not automatically rewrite the active security index. + +## 10.2 Inspect effective internal user configuration + +```bash +kubectl exec -n "$NS" "${OS_STS}-0" -c opensearch -- \ + grep -A8 '^admin:' \ + /usr/share/opensearch/config/opensearch-security/internal_users.yml +``` + +If the mounted file still contains a demo hash, it does not prove the intended password is active. + +## 10.3 Generate a bcrypt hash + +```bash +kubectl exec -it -n "$NS" "${OS_STS}-0" -c opensearch -- \ + /usr/share/opensearch/plugins/opensearch-security/tools/hash.sh \ + -p '' +``` + +Avoid placing real passwords in shell history. Prefer an interactive/secure workflow where possible. + +## 10.4 Create custom `internal_users.yml` + +```yaml +--- +_meta: + type: internalusers + config_version: 2 + +admin: + hash: "$2y$12$" + reserved: true + backend_roles: + - admin + description: OpenSearch administrator +``` + +## 10.5 Critical Secret key name + +The chart expects the exact key: + +```text +internal_users.yml +``` + +Not: + +```text +internal_users.yaml +``` + +Create/update correctly even if your local filename is `.yaml`: + +```bash +kubectl create secret generic opensearch-internal-users \ + -n "$NS" \ + --from-file=internal_users.yml=./internal_users.yaml \ + --dry-run=client -o yaml | kubectl apply -f - +``` + +Verify only the key names: + +```bash +kubectl get secret opensearch-internal-users \ + -n "$NS" \ + -o json | jq -r '.data | keys[]' +``` + +Expected: + +```text +internal_users.yml +``` + +## 10.6 Helm security configuration + +```yaml +securityConfig: + enabled: true + internalUsersSecret: opensearch-internal-users +``` + +Do not duplicate `securityConfig:` blocks. + +## 10.7 Admin certificate mount + +The admin certificate should be separate from normal node runtime TLS material. + +```yaml +extraVolumes: + # existing volumes ... + - name: admin-certs + secret: + secretName: opensearch-admin-cert + defaultMode: 0440 + +extraVolumeMounts: + # existing mounts ... + - name: admin-certs + mountPath: /usr/share/opensearch/config/admin-certs + readOnly: true +``` + +With the chart's `fsGroup: 1000`, mode `0440` allows the non-root OpenSearch process to read these files through the pod group. Verify the effective UID/GID before changing this mode. A stricter production design is to mount the admin key only into a temporary administrative Job/Pod rather than every OpenSearch runtime pod. + +## 10.8 Apply only internal users with `securityadmin.sh` + +After the cluster is healthy and the admin certificate is trusted: + +```bash +kubectl exec -it -n "$NS" "${OS_STS}-0" -c opensearch -- \ + /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh \ + -f /usr/share/opensearch/config/opensearch-security/internal_users.yml \ + -t internalusers \ + -icl \ + -nhnv \ + -cacert /usr/share/opensearch/config/admin-certs/root-ca.pem \ + -cert /usr/share/opensearch/config/admin-certs/admin.pem \ + -key /usr/share/opensearch/config/admin-certs/admin-key.pem +``` + +Using `-f` + `-t internalusers` is safer than unintentionally overwriting the complete Security configuration. + +--- + +# 11. OpenSearch Cluster Verification + +## 11.1 Kubernetes state + +```bash +kubectl get sts -n "$NS" +kubectl get pods -n "$NS" -o wide +kubectl get pvc -n "$NS" +kubectl get svc -n "$NS" +``` + +Expected: + +```text +3 OpenSearch pods +3 Bound PVCs +1 PVC per pod +pods spread across intended nodes +ClusterIP Service present +headless Service present +``` + +## 11.2 Init container logs + +```bash +kubectl logs -n "$NS" "${OS_STS}-0" -c select-node-certificate +kubectl logs -n "$NS" "${OS_STS}-1" -c select-node-certificate +kubectl logs -n "$NS" "${OS_STS}-2" -c select-node-certificate +``` + +## 11.3 Runtime certificate files + +```bash +kubectl exec -n "$NS" "${OS_STS}-0" -c opensearch -- \ + ls -l /usr/share/opensearch/config/certs +``` + +Expected: + +```text +node.pem +node-key.pem +root-ca.pem +``` + +## 11.4 OpenSearch logs + +```bash +kubectl logs -n "$NS" "${OS_STS}-0" -c opensearch --tail=200 +``` + +If a container restarted: + +```bash +kubectl logs -n "$NS" "${OS_STS}-0" -c opensearch --previous --tail=200 +``` + +Correct flag: + +```text +--previous +``` + +not: + +```text +--privious +``` + +## 11.5 Local diagnostic port-forward + +```bash +kubectl port-forward -n "$NS" svc/"$OS_SVC" 9200:9200 +``` + +Because the certificate may not contain `localhost`, `curl -k` is acceptable only as a **local diagnostic**: + +```bash +curl -k -u 'admin:' \ + 'https://localhost:9200/_cluster/health?pretty' + +curl -k -u 'admin:' \ + 'https://localhost:9200/_cat/nodes?v' +``` + +Do not interpret `-k` as production TLS validation. + +## 11.6 Cluster health meaning + +```text +GREEN -> all primary and replica shards allocated +YELLOW -> all primaries allocated; at least one replica unassigned +RED -> at least one primary shard unavailable +``` + +## 11.7 Replica count + +OpenSearch defaults to one replica unless changed. A 3-node cluster **can** use two replicas to keep three shard copies, but this is a capacity/durability choice, not an automatic requirement. + +Before setting `number_of_replicas: 2`, account for storage, indexing cost, and failure-domain design. + +--- + +# 12. OpenSearch Dashboards Deployment + +## 12.1 Architecture + +```text +Browser + | + | HTTP :31599 /opensearch + v +NGINX Ingress + | + v +OpenSearch Dashboards Service :5601 + | + v +Dashboards pod + | + | HTTPS :9200 + CA + hostname validation + v +OpenSearch Service +``` + +## 12.2 Verify OpenSearch Service certificate SAN + +The hostname used by Dashboards must exist in the OpenSearch certificate SAN: + +```text +brilliant-opensearch-cluster-master +``` + +A failure against `https://localhost:9200` such as: + +```text +no alternative certificate subject name matches target hostname 'localhost' +``` + +means CA trust may be correct but hostname verification failed. + +## 12.3 Root CA Secret for Dashboards + +```bash +kubectl create secret generic opensearch-dashboard-ca \ + -n "$NS" \ + --from-file=root-ca.pem \ + --dry-run=client -o yaml | kubectl apply -f - +``` + +## 12.4 `kibanaserver` password Secret + +```bash +read -rsp 'kibanaserver password: ' KIBANASERVER_PASSWORD +echo + +kubectl create secret generic opensearch-kibanaserver-password \ + -n "$NS" \ + --from-literal=KIBANASERVER_PASSWORD="$KIBANASERVER_PASSWORD" \ + --dry-run=client -o yaml | kubectl apply -f - + +unset KIBANASERVER_PASSWORD +``` + +The Secret object name and the Secret key are different concepts: + +```text +secretKeyRef.name -> opensearch-kibanaserver-password +secretKeyRef.key -> KIBANASERVER_PASSWORD +``` + +## 12.5 Dashboards Helm values + +```yaml +opensearchHosts: "https://brilliant-opensearch-cluster-master:9200" + +extraEnvs: + - name: KIBANASERVER_PASSWORD + valueFrom: + secretKeyRef: + name: opensearch-kibanaserver-password + key: KIBANASERVER_PASSWORD + +extraVolumes: + - name: root-cert + secret: + secretName: opensearch-dashboard-ca + +extraVolumeMounts: + - name: root-cert + readOnly: true + mountPath: /usr/share/opensearch-dashboards/certs + +config: + opensearch_dashboards.yml: | + server: + name: dashboards + host: "0.0.0.0" + basePath: /opensearch + rewriteBasePath: true + + opensearch: + username: kibanaserver + password: ${KIBANASERVER_PASSWORD} + ssl: + verificationMode: full + certificateAuthorities: + - /usr/share/opensearch-dashboards/certs/root-ca.pem + + opensearch_security.multitenancy.enabled: false + + data_source: + enabled: true + + workspace: + enabled: true +``` + +Optional feature keys such as `explore` and `datasetManagement` must be checked against the exact OpenSearch Dashboards version before enabling them. + +## 12.6 Workspaces and multi-tenancy + +When using Workspaces together with the OpenSearch Security plugin: + +```yaml +opensearch_security.multitenancy.enabled: false +``` + +is required to avoid feature conflict. + +## 12.7 `server.publicBaseUrl` + +In the tested OpenSearch Dashboards 3.7.0 deployment, adding `server.publicBaseUrl` caused startup validation failure: + +```text +"publicBaseUrl" is not allowed +``` + +Therefore it is intentionally absent from this SOP. + +## 12.8 Render Dashboards before deploy + +```bash +helm template brilliant-opensearch-dashboard \ + -n "$NS" \ + -f dashboards-values.yaml \ + > rendered-dashboards.yaml +``` + +Inspect: + +```bash +grep -nE 'KIBANASERVER_PASSWORD|root-cert|basePath|rewriteBasePath|verificationMode|5601' rendered-dashboards.yaml +``` + +## 12.9 Ingress conflict check + +Before creating a hostless path, inspect existing Ingress objects: + +```bash +kubectl get ingress -A +``` + +Grafana already used hostless `/` in the implementation. A second hostless `/` route under the same NGINX controller conflicted even though it was in another namespace. + +Use `/opensearch` instead. + +## 12.10 Dashboards Ingress + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: opensearch-dashboards-ingress + namespace: observability +spec: + ingressClassName: nginx + rules: + - http: + paths: + - path: /opensearch + pathType: Prefix + backend: + service: + name: brilliant-opensearch-dashboard-opensearch-dashboards + port: + number: 5601 +``` + +Apply: + +```bash +kubectl apply -f opensearch-dashboards-ingress.yaml +``` + +## 12.11 Runtime verification + +```bash +kubectl get deploy,pods,svc,ingress -n "$NS" +``` + +Current logs: + +```bash +kubectl logs -n "$NS" deploy/brilliant-opensearch-dashboard-opensearch-dashboards --tail=200 +``` + +Previous container logs after a restart: + +```bash +kubectl logs -n "$NS" deploy/brilliant-opensearch-dashboard-opensearch-dashboards --previous --tail=200 +``` + +Expected external URL pattern: + +```text +http://:31599/opensearch +``` + +## 12.12 `401 -> 404` diagnostic meaning + +During troubleshooting: + +```text +unauthenticated /opensearch -> 401 Authentication required +valid authentication -> 404 Not Found +``` + +This indicated traffic was reaching Dashboards and authentication worked, while Dashboards did not yet recognize `/opensearch` as its base path. `server.basePath` + `rewriteBasePath` resolved that routing layer. + +--- + +# 13. Data Prepper — Full Deployment, Routing, Parsing, and Verification + +This chapter is the canonical Data Prepper procedure for this environment. It consolidates the Data Prepper material from the cPanel ingestion, multi-source routing, and Linux Audit SOPs. + +## 13.1 Role in the architecture + +```text +Fluent Bit + | + | HTTP POST /logs + v +NGINX Ingress / NodePort + | + v +Data Prepper HTTP source :2021 + | + v +Router pipeline + | + +--> cPanel access pipeline + +--> system syslog pipeline + +--> Linux audit pipeline + +--> catch-all / unknown pipeline + | + v +OpenSearch HTTPS :9200 +``` + +Data Prepper is responsible for receiving events, routing them by `log_source`, parsing/normalizing selected formats, and writing documents to the correct OpenSearch index. + +## 13.2 Environment values used in this implementation + +| Setting | Value used | +|---|---| +| Data Prepper version | `2.8.0` | +| Namespace | `observability` | +| Helm release | `data-prepper` | +| HTTP source port | `2021` | +| Server API / metrics port | `4900` | +| HTTP path | `/logs` | +| OpenSearch endpoint | `https://brilliant-opensearch-cluster-master:9200` | +| Root CA path inside Data Prepper | `/usr/share/data-prepper/cert/root-ca.pem` | +| cPanel access index | `cpanel-access-log-%{yyyy.MM.dd}` | +| Syslog index | `cpanel-system-syslog-%{yyyy.MM.dd}` | +| Audit index | `cpanel-audit-log-%{yyyy.MM.dd}` | +| Catch-all index | `cpanel-all-log-%{yyyy.MM.dd}` | + +Verify the actual release and namespace before modifying anything: + +```bash +helm list -n "$NS" +kubectl get deploy,svc,pod -n "$NS" | grep -i data-prepper +``` + +Expected Helm release: + +```text +data-prepper +``` + +> [!WARNING] +> One troubleshooting incident installed/upgraded a second `data-prepper` release in the `default` namespace. Always specify `-n observability` / `--namespace observability`. + +## 13.3 Back up the current Data Prepper configuration before changes + +```bash +mkdir -p ~/backup/data-prepper + +helm get values data-prepper -n "$NS" -a \ + > ~/backup/data-prepper/values-$(date +%F-%H%M%S).yaml + +helm get manifest data-prepper -n "$NS" \ + > ~/backup/data-prepper/manifest-$(date +%F-%H%M%S).yaml +``` + +Also export the currently running pipeline/config objects if they are stored in ConfigMaps or Secrets: + +```bash +kubectl get configmap -n "$NS" -o name | grep -i prepper +kubectl get secret -n "$NS" -o name | grep -i prepper +``` + +Do not dump Secret data into Git, screenshots, or terminal transcripts. + +## 13.4 One HTTP source per listener port + +Only one pipeline in a Data Prepper process should own HTTP port `2021`. + +Incorrect design: + +```text +access pipeline -> HTTP :2021 +syslog pipeline -> HTTP :2021 +audit pipeline -> HTTP :2021 +``` + +Typical failure: + +```text +bind(..) failed: Address already in use +``` + +Correct design: + +```text +ONE HTTP source :2021 + | + v +router pipeline + | + +--> access pipeline + +--> syslog pipeline + +--> audit pipeline + +--> catch-all pipeline +``` + +Downstream pipelines use a Data Prepper `pipeline` source, not another HTTP source. + +## 13.5 Complete router pipeline + +The normalized `log_source` field created by Fluent Bit is the routing key. + +```yaml +version: "2" + +cpanel-log-router: + source: + http: + path: /logs + ssl: false + port: 2021 + health_check_service: true + unauthenticated_health_check: true + + route: + - accessLog: '/log_source == "cpanel.access_log"' + - systemSyslog: '/log_source == "system.syslog"' + - auditLog: '/log_source == "audit.audit.log"' + - unhandled: '/log_source != "cpanel.access_log" and /log_source != "system.syslog" and /log_source != "audit.audit.log"' + + sink: + - pipeline: + name: cpanel-access-pipeline + routes: + - accessLog + + - pipeline: + name: cpanel-system-syslog-pipeline + routes: + - systemSyslog + + - pipeline: + name: cpanel-audit-pipeline + routes: + - auditLog + + - pipeline: + name: cpanel-catch-all-pipeline + routes: + - unhandled +``` + +### Why the `unhandled` route is explicit + +A pipeline sink without a `routes:` restriction can receive all events. If a catch-all sink is left unqualified, dedicated events may also be copied into the catch-all path. The explicit `unhandled` condition prevents that duplication. + +## 13.6 Complete cPanel access pipeline + +The tested raw access format is: + +```text +20.100.201.54 proxy - [08/09/2026:07:37:27 -0000] "GET /wp-content/themes/twentytwenty/404.php HTTP/1.1" 301 0 "-" "-" "-" "X-Forwarded-For: 20.100.201.54" 80 +``` + +Pipeline: + +```yaml +cpanel-access-pipeline: + workers: 2 + delay: 100 + + source: + pipeline: + name: cpanel-log-router + + processor: + - drop_events: + drop_when: '/log_source != "cpanel.access_log"' + + - grok: + break_on_match: false + match: + log: + - '^%{IP:client_ip} %{NOTSPACE:identity} %{NOTSPACE:user} \[%{GREEDYDATA:timestamp}\] "%{WORD:method} %{URIPATHPARAM:url} HTTP/%{NUMBER:http_version}" %{INT:http_status:int} %{NONNEGINT:bytes:long} "%{NOTSPACE:referrer}" "%{GREEDYDATA:user_agent}" "%{NOTSPACE:auth_method}" "%{GREEDYDATA:x_forwarded_for}" %{POSINT:service_port:int}$' + target_key: parsed + + - date: + match: + - key: /parsed/timestamp + patterns: + - "MM/dd/yyyy:HH:mm:ss Z" + destination: "@timestamp" + + sink: + - opensearch: + hosts: + - "https://brilliant-opensearch-cluster-master:9200" + insecure: false + username: logger + password: "" + cert: /usr/share/data-prepper/cert/root-ca.pem + index_type: custom + index: cpanel-access-log-%{yyyy.MM.dd} +``` + +Expected parsed fields: + +```text +parsed.client_ip +parsed.identity +parsed.user +parsed.timestamp +parsed.method +parsed.url +parsed.http_version +parsed.http_status +parsed.bytes +parsed.referrer +parsed.user_agent +parsed.auth_method +parsed.x_forwarded_for +parsed.service_port +@timestamp +``` + +### Important Grok compatibility issue from the implementation + +The tested Data Prepper 2.8.0 setup failed when this was used: + +```text +%{PORT:service_port} +``` + +with: + +```text +No definition for key 'PORT' found, aborting +``` + +The working replacement was: + +```text +%{POSINT:service_port} +``` + +## 13.7 Syslog pipeline + +Syslog records observed in the environment included both: + +```text +2026-08-11T10:42:39.679625+00:00 bwebhosting kernel: Firewall: ... +``` + +and: + +```text +2026-08-11T10:42:39.434068+00:00 bwebhosting fluent-bit[1241036]: ... +``` + +The process token therefore needs to tolerate both a plain process name and a process name with an optional PID. + +A safe baseline is to preserve raw data while the exact parser is being tuned: + +```yaml +cpanel-system-syslog-pipeline: + source: + pipeline: + name: cpanel-log-router + + processor: + - drop_events: + drop_when: '/log_source != "system.syslog"' + + sink: + - opensearch: + hosts: + - "https://brilliant-opensearch-cluster-master:9200" + insecure: false + username: logger + password: "" + cert: /usr/share/data-prepper/cert/root-ca.pem + index_type: custom + index: cpanel-system-syslog-%{yyyy.MM.dd} +``` + +Add Grok only after representative samples are verified. If a document reaches OpenSearch with `parsed: {}`, transport/routing may be healthy while the Grok expression is not matching. + +## 13.8 Linux Audit pipeline + +Audit records are multi-type and are handled in the dedicated Linux Audit chapter of this SOP. The router must send only: + +```text +log_source = audit.audit.log +``` + +to: + +```text +cpanel-audit-pipeline +``` + +The audit pipeline should additionally protect itself with: + +```yaml +- drop_events: + drop_when: '/log_source != "audit.audit.log"' +``` + +The full Audit Grok, date, `EXECVE`, `SYSCALL`, `LOGIN`, `USER_ACCT`, `USER_START`, `CRED_ACQ`, and `CWD` parsing procedure is documented in Section 17. + +## 13.9 Catch-all pipeline + +Use the catch-all to retain unknown/new sources without forcing every source into a dedicated parser on day one. + +```yaml +cpanel-catch-all-pipeline: + source: + pipeline: + name: cpanel-log-router + + sink: + - opensearch: + hosts: + - "https://brilliant-opensearch-cluster-master:9200" + insecure: false + username: logger + password: "" + cert: /usr/share/data-prepper/cert/root-ca.pem + index_type: custom + index: cpanel-all-log-%{yyyy.MM.dd} +``` + +Recommended model: + +```text +Known/high-value source + -> dedicated route + -> dedicated parser + -> dedicated index + +Unknown/new source + -> catch-all + -> raw visibility + -> inspect + -> promote later if required +``` + +## 13.10 OpenSearch Root CA mount + +Data Prepper must trust the CA that signed the OpenSearch HTTP certificate. + +Conceptual Kubernetes volume: + +```yaml +volumes: + - name: opensearch-root-ca + secret: + secretName: opensearch-dashboard-ca +``` + +Conceptual container mount: + +```yaml +volumeMounts: + - name: opensearch-root-ca + mountPath: /usr/share/data-prepper/cert + readOnly: true +``` + +Verify inside the pod: + +```bash +kubectl exec -n "$NS" deploy/data-prepper -- \ + ls -l /usr/share/data-prepper/cert/root-ca.pem +``` + +Do not set `insecure: true` merely to bypass a certificate problem. Correct the CA/SAN/TLS configuration instead. + +## 13.11 Data Prepper credential handling + +The source SOP used an OpenSearch user named: + +```text +logger +``` + +The real password must not be committed to Git. + +Required security model: + +```text +Data Prepper identity + -> authenticate to OpenSearch + -> only required write/create-index permissions + -> no admin/security-management permissions +``` + +If the Helm chart or deployment mechanism stores `pipelines.yaml` in a Secret, keep the sensitive pipeline configuration in that Secret or use the secret-management mechanism supported by the deployment. Do not assume arbitrary environment-variable interpolation in every Data Prepper configuration field without testing the exact deployed version. + +## 13.12 Kubernetes Service + +Expected Service path: + +```text +Ingress + -> data-prepper Service :2021 + -> Data Prepper Pod :2021 +``` + +Service example: + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: data-prepper + namespace: observability +spec: + type: ClusterIP + selector: + app: data-prepper + ports: + - name: ingest + port: 2021 + targetPort: 2021 + - name: metrics + port: 4900 + targetPort: 4900 +``` + +The selector is chart/deployment-specific. Verify that it matches the actual pod labels: + +```bash +kubectl get pod -n "$NS" --show-labels | grep data-prepper +kubectl get svc data-prepper -n "$NS" -o yaml +``` + +Verify the backend endpoint: + +```bash +kubectl get endpointslice \ + -n "$NS" \ + -l kubernetes.io/service-name=data-prepper \ + -o wide +``` + +Expected result: the current Data Prepper pod IP and port `2021` are present. + +## 13.13 NGINX Ingress + +The implementation used: + +```text +http://118.67.213.76:31599/logs +``` + +Flow: + +```text +118.67.213.76:31599 + -> ingress-nginx controller :80 + -> prepper-ingress /logs + -> data-prepper:2021 +``` + +Ingress: + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: prepper-ingress + namespace: observability +spec: + ingressClassName: nginx + rules: + - http: + paths: + - path: /logs + pathType: Prefix + backend: + service: + name: data-prepper + port: + number: 2021 +``` + +Apply and verify: + +```bash +kubectl apply -f prepper-ingress.yaml +kubectl get ingress prepper-ingress -n "$NS" -o wide +kubectl describe ingress prepper-ingress -n "$NS" +``` + +## 13.14 Data Prepper Helm upgrade procedure + +The exact chart path depends on how Data Prepper was installed. Use the real chart reference/path instead of guessing. + +```bash +export DP_RELEASE="data-prepper" +export DP_CHART="" +export DP_VALUES="" +``` + +Confirm the current release first: + +```bash +helm list -n "$NS" +``` + +Render before upgrading: + +```bash +helm template "$DP_RELEASE" "$DP_CHART" \ + -n "$NS" \ + -f "$DP_VALUES" \ + > /tmp/data-prepper-rendered.yaml +``` + +Inspect the important items: + +```bash +grep -nE 'kind: (Deployment|Service|ConfigMap|Secret)|2021|4900|pipelines|data-prepper' \ + /tmp/data-prepper-rendered.yaml | head -200 +``` + +Then upgrade the correct release in the correct namespace: + +```bash +helm upgrade "$DP_RELEASE" "$DP_CHART" \ + -n "$NS" \ + -f "$DP_VALUES" +``` + +Watch rollout: + +```bash +kubectl get pods -n "$NS" -w +``` + +## 13.15 Pod and listener verification + +```bash +kubectl get pods -n "$NS" | grep data-prepper +kubectl logs -n "$NS" deploy/data-prepper --tail=200 +``` + +Expected health: + +```text +READY: 1/1 +STATUS: Running +``` + +If `ss`/`netstat` do not exist in the image, `/proc/net/tcp` can verify the listener. + +For port `2021`: + +```text +2021 decimal = 07E5 hexadecimal +0A = LISTEN +01 = ESTABLISHED +``` + +Example: + +```bash +kubectl exec -n "$NS" deploy/data-prepper -- \ + sh -c 'grep -i ":07E5" /proc/net/tcp /proc/net/tcp6 || true' +``` + +## 13.16 Health endpoint and metrics + +When the HTTP source health service is enabled: + +```bash +kubectl run dp-curl --rm -i --restart=Never \ + -n "$NS" \ + --image=curlimages/curl:latest \ + -- \ + curl -sS http://data-prepper:2021/health +``` + +Data Prepper server/metrics API: + +```bash +kubectl exec -n "$NS" deploy/data-prepper -- \ + curl -s http://127.0.0.1:4900/metrics/prometheus | head -100 +``` + +Useful metric families from the audit/cPanel work include: + +```text +grokProcessingMatch +grokProcessingMismatch +grokProcessingErrors +dateProcessingMatchSuccess +dateProcessingMatchFailure +documentsSuccess +documentErrors +``` + +Pipeline/plugin names can be normalized into the full metric name. + +## 13.17 Controlled event test inside Kubernetes + +The HTTP source accepts JSON events. Use a controlled event before involving Fluent Bit: + +```bash +kubectl run dp-curl --rm -i --restart=Never \ + -n "$NS" \ + --image=curlimages/curl:latest \ + -- \ + curl -sS -X POST http://data-prepper:2021/logs \ + -H 'Content-Type: application/json' \ + --data '[{"log":"test","source_file":"manual-test","log_source":"test.manual"}]' +``` + +Then verify that the catch-all path receives it. + +## 13.18 Controlled event test through NGINX + +From a host that can reach the NGINX NodePort: + +```bash +curl -i -X POST \ + 'http://118.67.213.76:31599/logs' \ + -H 'Content-Type: application/json' \ + --data '[{"log":"ingress-test","source_file":"manual-test","log_source":"test.manual"}]' +``` + +A successful test proves: + +```text +client + -> NodePort + -> NGINX + -> Ingress rule + -> Service + -> EndpointSlice + -> Data Prepper :2021 +``` + +## 13.19 OpenSearch sink verification + +Do not use index existence alone as proof of successful ingestion. + +Example: + +```text +green open cpanel-access-log-2026.08.09 ... docs.count=0 +``` + +means the index exists but has no documents. + +Check index list: + +```bash +curl --cacert -u ':' \ + 'https://:9200/_cat/indices/cpanel-*?v' +``` + +Check document count: + +```bash +curl --cacert -u ':' \ + 'https://:9200/cpanel-access-log-*/_count?pretty' +``` + +Check parsed fields: + +```bash +curl --cacert -u ':' \ + -H 'Content-Type: application/json' \ + 'https://:9200/cpanel-access-log-*/_search?pretty' \ + -d '{ + "size": 5, + "sort": [{"@timestamp": "desc"}], + "_source": ["@timestamp","log_source","source_file","log","parsed"] + }' +``` + +Interpretation: + +```text +log exists + parsed exists -> parser matched +log exists + parsed missing -> parser did not match / processor issue +index exists + docs.count = 0 -> no successful document insertion +``` + +## 13.20 `502 Bad Gateway` troubleshooting + +Fluent Bit previously reported: + +```text +HTTP status=502 +502 Bad Gateway +nginx +``` + +This means Fluent Bit reached NGINX, but NGINX did not have a working upstream path to Data Prepper. + +Check in this order: + +```bash +kubectl get pod -n "$NS" | grep data-prepper +kubectl get svc data-prepper -n "$NS" -o wide +kubectl get endpointslice -n "$NS" \ + -l kubernetes.io/service-name=data-prepper -o wide +kubectl logs -n "$NS" deploy/data-prepper --tail=200 +kubectl describe ingress prepper-ingress -n "$NS" +``` + +Do not change Fluent Bit Grok/parsing when NGINX is returning a backend/upstream error. + +## 13.21 OpenSearch sink failure troubleshooting + +Observed sink errors included: + +```text +Failed to initialize OpenSearch sink with a retryable exception +Request execution cancelled +I/O session is closed +``` + +Troubleshoot this path independently: + +```text +Data Prepper + -> Kubernetes DNS + -> brilliant-opensearch-cluster-master + -> TCP 9200 + -> TLS handshake + -> Root CA validation + -> hostname/SAN validation + -> logger authentication + -> logger authorization + -> index write +``` + +Do not modify Grok while the failure is DNS/TCP/TLS/authentication/authorization at the sink. + +## 13.22 YAML and processor hierarchy checks + +Correct Grok hierarchy: + +```yaml +processor: + - grok: + break_on_match: false + match: + log: + - '' + target_key: parsed +``` + +A prior indentation mistake caused Data Prepper to interpret options such as `false` at the wrong level and fail pipeline construction. + +Validate indentation and quoting before every Helm upgrade. + +## 13.23 Production hardening for Data Prepper + +The original implementation used: + +```yaml +ssl: false +``` + +on the Data Prepper HTTP source and exposed it through an NGINX NodePort. Before considering the ingestion endpoint production-hardened, use the controls appropriate to the environment: + +```text +TLS on the ingestion path +HTTP authentication where supported/configured +private network routing +firewall/source-IP restrictions +NetworkPolicy +Ingress authentication / mTLS where appropriate +least-privilege OpenSearch sink identity +persistent DLQ if required +resource limits +metrics and alerting +``` + +The endpoint receives security-sensitive data including IP addresses, usernames, URLs, audit records, processes, and commands. Protect access and retention accordingly. + +--- + +# 14. Fluent Bit — Full cPanel/Linux Collection Procedure + +This chapter is the canonical Fluent Bit procedure for the cPanel/Linux sources in this project. It includes the access-only baseline, the expanded multi-source configuration, source classification, persistent offset state, filesystem buffering, verification, and outage-recovery testing. + +## 14.1 Role in the architecture + +```text +cPanel/Linux log files + | + v +Fluent Bit Tail inputs + | + +--> persistent Tail DB + +--> filesystem buffer + +--> source_file + +--> Lua log_source classification + | + v +HTTP POST /logs + | + v +NGINX -> Data Prepper +``` + +Fluent Bit must: + +- Continuously tail the required log files. +- Preserve the original path as `source_file`. +- Maintain independent Tail DB state for each input. +- Use filesystem buffering for undelivered chunks. +- Normalize the source into a stable `log_source` value. +- Send JSON to Data Prepper. +- Retry transient failures. +- Resume correctly after service/server restarts. + +## 14.2 Source inventory used in the project + +| Category | Example path | Expected `log_source` | +|---|---|---| +| cPanel access | `/usr/local/cpanel/logs/access_log` | `cpanel.access_log` | +| Syslog | `/var/log/syslog` | `system.syslog` | +| Kernel | `/var/log/kern.log` | `system.kern.log` | +| Exim | `/var/log/exim_mainlog` | `system.exim_mainlog` | +| Auditd | `/var/log/audit/audit.log` | `audit.audit.log` | +| Apache | `/var/log/apache2/...` | `apache.` | + +Before configuration, verify which files actually exist on the server: + +```bash +for f in \ + /usr/local/cpanel/logs/access_log \ + /var/log/syslog \ + /var/log/kern.log \ + /var/log/exim_mainlog \ + /var/log/audit/audit.log; do + [ -e "$f" ] && ls -lh "$f" +done +``` + +Inspect representative records before writing parsers: + +```bash +tail -n 20 /usr/local/cpanel/logs/access_log +sudo tail -n 20 /var/log/audit/audit.log +``` + +## 14.3 Optional installation verification + +The source SOPs started from an already installed Fluent Bit service. If deploying on a new cPanel host, use the package procedure for the actual OS and pin/test the version according to your change policy. + +First identify the OS: + +```bash +cat /etc/os-release +``` + +After installation, the service must exist: + +```bash +systemctl status fluent-bit --no-pager +fluent-bit --version +``` + +Do not replace a working production package with a newer one during an ingestion troubleshooting session unless the upgrade itself is planned and tested. + +## 14.4 Service account and permissions + +Determine the service account: + +```bash +systemctl cat fluent-bit | grep -E '^(User|Group)=' || true +ps -eo user,group,pid,cmd | grep '[f]luent-bit' +``` + +The Fluent Bit process needs read permission on every selected source file and write permission to the Tail DB/buffer directories. + +For audit logs, verify access explicitly: + +```bash +namei -l /var/log/audit/audit.log +``` + +Do not run Fluent Bit as root merely to bypass a permissions problem unless that is the deliberately approved security model. + +## 14.5 Persistent state and buffer directories + +Recommended separation: + +```text +/var/lib/fluent-bit/ + ├── state/ + │ ├── cpanel-access.db + │ ├── syslog.db + │ ├── kern.db + │ ├── exim.db + │ └── audit.db + └── storage/ + └── filesystem chunks +``` + +Create the directories using the actual Fluent Bit service account: + +```bash +install -d -m 0750 -o fluent-bit -g fluent-bit /var/lib/fluent-bit/state +install -d -m 0750 -o fluent-bit -g fluent-bit /var/lib/fluent-bit/storage +``` + +If the service uses another user/group, replace `fluent-bit:fluent-bit` accordingly. + +Keep these directories outside any broad Tail wildcard so Fluent Bit does not ingest its own SQLite/WAL or buffer files. + +## 14.6 Access-log-only baseline configuration + +Use this first when proving the cPanel access path end to end: + +```ini +[SERVICE] + Flush 1 + Daemon Off + Log_Level info + + Parsers_File parsers.conf + Plugins_File plugins.conf + + storage.metrics on + storage.path /var/lib/fluent-bit/storage + storage.sync full + storage.checksum on + storage.backlog.mem_limit 500M + +[INPUT] + Name tail + Path /usr/local/cpanel/logs/access_log + Path_Key source_file + Tag access_log + + DB /var/lib/fluent-bit/state/cpanel-access.db + DB.Journal_Mode wal + + Refresh_Interval 5 + Skip_Long_Lines true + Mem_Buf_Limit 512MB + Storage.Type filesystem + +[FILTER] + Name modify + Match access_log + Add log_source cpanel.access_log + +[OUTPUT] + Name http + Host 118.67.213.76 + Port 31599 + Format json + HTTP_Method POST + URI /logs + Match access_log + Retry_Limit no_limits + Storage.Total_Limit_Size 4G +``` + +### Notes + +`DB` stores file identity/offset state. It is not the durable event queue. + +`Storage.Type filesystem` allows chunks to be backed by the filesystem storage layer. + +`Storage.Total_Limit_Size 4G` caps the logical filesystem queue for this output. If the queue fills, old queued chunks can be discarded to make room for new data; monitor it. + +`Retry_Limit no_limits` avoids abandoning a failed chunk merely because a finite retry count was reached. This still does not make local disk capacity infinite. + +## 14.7 Historical ingestion behavior + +For a file with no existing Tail DB state: + +```ini +Read_from_Head true +``` + +means read from the beginning. + +Without it, a new input normally begins at the end of the file and follows new records. + +Use historical backfill intentionally. Deleting/replacing a Tail DB and then setting `Read_from_Head true` can re-ingest already indexed data. + +For static backfill testing, use a copy of the source file and a dedicated DB/index whenever possible. + +## 14.8 Multi-source Tail configuration + +After the access-only path is proven, expand to the required sources. Each input should have a unique DB. + +```ini +[SERVICE] + Flush 1 + Daemon Off + Log_Level info + + Parsers_File parsers.conf + Plugins_File plugins.conf + + storage.metrics on + storage.path /var/lib/fluent-bit/storage + storage.sync full + storage.checksum on + storage.backlog.mem_limit 500M + +[INPUT] + Name tail + Path /usr/local/cpanel/logs/access_log + Path_Key source_file + Tag cpanel.files + DB /var/lib/fluent-bit/state/cpanel-access.db + DB.Journal_Mode wal + Refresh_Interval 5 + Skip_Long_Lines true + Mem_Buf_Limit 512MB + Storage.Type filesystem + +[INPUT] + Name tail + Path /var/log/syslog + Path_Key source_file + Tag cpanel.files + DB /var/lib/fluent-bit/state/syslog.db + DB.Journal_Mode wal + Refresh_Interval 5 + Skip_Long_Lines true + Mem_Buf_Limit 256MB + Storage.Type filesystem + +[INPUT] + Name tail + Path /var/log/kern.log + Path_Key source_file + Tag cpanel.files + DB /var/lib/fluent-bit/state/kern.db + DB.Journal_Mode wal + Refresh_Interval 5 + Skip_Long_Lines true + Mem_Buf_Limit 256MB + Storage.Type filesystem + +[INPUT] + Name tail + Path /var/log/exim_mainlog + Path_Key source_file + Tag cpanel.files + DB /var/lib/fluent-bit/state/exim.db + DB.Journal_Mode wal + Refresh_Interval 5 + Skip_Long_Lines true + Mem_Buf_Limit 256MB + Storage.Type filesystem + +[INPUT] + Name tail + Path /var/log/audit/audit.log + Path_Key source_file + Tag cpanel.files + DB /var/lib/fluent-bit/state/audit.db + DB.Journal_Mode wal + Refresh_Interval 5 + Skip_Long_Lines true + Mem_Buf_Limit 512MB + Storage.Type filesystem + +[INPUT] + Name tail + Path /var/log/apache2/*.log + Path_Key source_file + Tag cpanel.files + DB /var/lib/fluent-bit/state/apache.db + DB.Journal_Mode wal + Refresh_Interval 5 + Skip_Long_Lines true + Mem_Buf_Limit 512MB + Storage.Type filesystem + +[FILTER] + Name lua + Match cpanel.files + Script /etc/fluent-bit/classify.lua + Call classify + +[OUTPUT] + Name http + Host 118.67.213.76 + Port 31599 + Format json + HTTP_Method POST + URI /logs + Match cpanel.files + Retry_Limit no_limits + Storage.Total_Limit_Size 4G +``` + +Only keep input paths that actually exist and are required. If Apache/cPanel stores logs elsewhere on a specific host, use the real path. + +## 14.9 `source_file` and `log_source` + +A Tail event should preserve its file: + +```json +{ + "log": "...", + "source_file": "/var/log/syslog" +} +``` + +The Lua filter converts that into a stable routing field: + +```json +{ + "log": "...", + "source_file": "/var/log/syslog", + "log_source": "system.syslog" +} +``` + +Data Prepper routes on `log_source`, not on assumptions derived from the destination index name. + +## 14.10 Lua source-classification script + +Create: + +```text +/etc/fluent-bit/classify.lua +``` + +Content: + +```lua +function classify(tag, timestamp, record) + local path = record["source_file"] or "" + + if path == "/usr/local/cpanel/logs/access_log" then + record["log_source"] = "cpanel.access_log" + elseif path == "/var/log/syslog" then + record["log_source"] = "system.syslog" + elseif path == "/var/log/kern.log" then + record["log_source"] = "system.kern.log" + elseif path == "/var/log/exim_mainlog" then + record["log_source"] = "system.exim_mainlog" + elseif path == "/var/log/audit/audit.log" then + record["log_source"] = "audit.audit.log" + elseif string.match(path, "^/var/log/apache2/") then + local filename = string.match(path, "([^/]+)$") or "unknown" + record["log_source"] = "apache." .. filename + else + record["log_source"] = "unknown" + end + + return 1, timestamp, record +end +``` + +Set ownership/permissions appropriate to the service: + +```bash +chown root:root /etc/fluent-bit/classify.lua +chmod 0644 /etc/fluent-bit/classify.lua +``` + +The earlier implementation attempted inline Lua in the classic configuration and Fluent Bit failed with: + +```text +extra indentation level found +invalid indentation level +``` + +Keeping Lua in a separate file avoids that configuration-indentation problem and is easier to test/version. + +## 14.11 Validate configuration before restart + +If supported by the installed Fluent Bit version/build: + +```bash +fluent-bit --dry-run -c /etc/fluent-bit/fluent-bit.conf +``` + +If the binary is not in `PATH`: + +```bash +command -v fluent-bit +systemctl cat fluent-bit +``` + +Then restart: + +```bash +systemctl restart fluent-bit +systemctl status fluent-bit --no-pager +journalctl -u fluent-bit --since '5 minutes ago' --no-pager +``` + +Healthy state should not show sustained: + +```text +configuration parser errors +502 Bad Gateway +connection refused +failed to flush +chunk retry exhaustion +``` + +## 14.12 Prove the source is generating data + +For cPanel access logs, generate or wait for a request and confirm the file changes: + +```bash +tail -Fn0 /usr/local/cpanel/logs/access_log +``` + +For audit: + +```bash +sudo tail -Fn0 /var/log/audit/audit.log +``` + +If no fresh event exists in the source file, do not troubleshoot Data Prepper/OpenSearch yet. + +## 14.13 Verify Tail DB creation and persistence + +```bash +ls -lh /var/lib/fluent-bit/state/ +``` + +If SQLite is installed and Fluent Bit is stopped or the DB can be safely inspected: + +```bash +sqlite3 /var/lib/fluent-bit/state/cpanel-access.db \ + 'SELECT id,name,offset,inode FROM in_tail_files;' +``` + +The Tail database tells Fluent Bit where it last read each tracked file. + +## 14.14 Verify filesystem buffering + +```bash +find /var/lib/fluent-bit/storage -maxdepth 2 -type f -ls | head -50 +``` + +Remember: + +```text +Tail DB + -> protects file read position + +filesystem storage + -> holds chunks that have been read but are not yet successfully delivered +``` + +They are separate mechanisms. + +## 14.15 Verify HTTP output to Data Prepper + +Follow Fluent Bit logs: + +```bash +journalctl -u fluent-bit -f +``` + +The target used in this implementation is: + +```text +http://118.67.213.76:31599/logs +``` + +Test the receiver directly from the cPanel server: + +```bash +curl -i -X POST \ + 'http://118.67.213.76:31599/logs' \ + -H 'Content-Type: application/json' \ + --data '[{"log":"manual-cpanel-host-test","source_file":"manual-test","log_source":"test.manual"}]' +``` + +If the manual curl fails, fix network/Ingress/Data Prepper before modifying Fluent Bit parsing or Tail settings. + +## 14.16 Interpretation of common Fluent Bit output failures + +### `502 Bad Gateway` + +```text +Fluent Bit -> NGINX = reachable +NGINX -> Data Prepper upstream = unhealthy/unreachable +``` + +Check Data Prepper pod, Service, EndpointSlice, listener, and Ingress. + +### `connection refused` + +The target IP/port is reachable at the network level but nothing is accepting the connection, or a firewall is rejecting it. + +### `failed to flush chunk` + +Inspect the associated HTTP/network error. With filesystem buffering enabled, confirm the queue is growing and not at the storage limit. + +### configuration parser error + +Use the exact reported file/line. Do not assume the systemd unit failure text is the root cause; Fluent Bit's own startup output contains the useful parser error. + +## 14.17 Durable buffering recovery test + +Perform this only after normal ingestion is stable. + +1. Record the current OpenSearch count. +2. Make Data Prepper unavailable intentionally. +3. Generate several test requests/log records. +4. Confirm Fluent Bit reports retries. +5. Confirm filesystem backlog activity in `/var/lib/fluent-bit/storage`. +6. Restart Fluent Bit while Data Prepper is still unavailable. +7. Confirm the Tail DB and storage directories survive the restart. +8. Restore Data Prepper. +9. Confirm queued events are forwarded. +10. Verify the expected documents appear in OpenSearch. + +Useful commands: + +```bash +systemctl restart fluent-bit +journalctl -u fluent-bit --since '10 minutes ago' --no-pager +find /var/lib/fluent-bit/storage -type f | head +``` + +This test validates both offset persistence and undelivered-event buffering. + +## 14.18 Long-line behavior + +The configuration uses: + +```ini +Skip_Long_Lines true +``` + +This prevents a single oversized line from stopping monitoring of the entire file, but the oversized event can be skipped. + +If large audit/application records matter, test representative maximum line sizes and tune `Buffer_Chunk_Size` / `Buffer_Max_Size` rather than assuming the defaults are sufficient. + +## 14.19 Log rotation + +Fluent Bit Tail tracks files using file metadata/offset state. Verify rotation behavior for each high-value log: + +```bash +ls -li /usr/local/cpanel/logs/access_log* +ls -li /var/log/audit/audit.log* +``` + +Test that rotation does not cause either permanent loss or unexpected full-file replay. Do not delete Tail DB files as a routine rotation fix. + +## 14.20 Multiple files and inotify limits + +If the deployment expands to a large number of Apache/cPanel files, inspect inotify capacity: + +```bash +sysctl fs.inotify.max_user_watches +sysctl fs.inotify.max_user_instances +``` + +Increase them only if monitoring scale requires it and persist the chosen values through the normal OS configuration-management process. + +## 14.21 Security hardening for Fluent Bit + +Use the minimum filesystem permissions required to read the configured logs and write its own state/buffer directories. + +Protect: + +```text +/etc/fluent-bit/fluent-bit.conf +/etc/fluent-bit/classify.lua +/var/lib/fluent-bit/state/ +/var/lib/fluent-bit/storage/ +``` + +The original ingestion route used plain HTTP to the NGINX NodePort. For production, protect the transport with private routing, firewall restrictions, authentication and/or TLS as appropriate to the environment. + +Do not store OpenSearch credentials in Fluent Bit when Fluent Bit is intentionally sending only to Data Prepper. + +## 14.22 Fluent Bit acceptance checklist + +- [ ] Required source files exist and receive new records. +- [ ] Fluent Bit service is running. +- [ ] No classic-config indentation/parser error remains. +- [ ] Each Tail input has a unique persistent DB. +- [ ] Tail DB directory is writable. +- [ ] Filesystem buffer directory is writable. +- [ ] `Storage.Type filesystem` is enabled for protected inputs. +- [ ] `source_file` is present. +- [ ] `log_source` matches the source file. +- [ ] `/usr/local/cpanel/logs/access_log` becomes `cpanel.access_log`. +- [ ] `/var/log/syslog` becomes `system.syslog`. +- [ ] `/var/log/audit/audit.log` becomes `audit.audit.log`. +- [ ] HTTP destination is the correct NGINX/Data Prepper endpoint. +- [ ] URI is `/logs`. +- [ ] Output format is JSON. +- [ ] Fluent Bit has no sustained 502/refused/flush failures. +- [ ] Restart resumes from stored offsets. +- [ ] Data Prepper outage/recovery test has been completed. +- [ ] Backlog disk usage is monitored. + +--- + +# 15. cPanel Access Log Parsing + +## 15.1 Example raw log + +```text +20.100.201.54 proxy - [08/09/2026:07:37:27 -0000] "GET /wp-content/themes/twentytwenty/404.php HTTP/1.1" 301 0 "-" "-" "-" "X-Forwarded-For: 20.100.201.54" 80 +``` + +## 15.2 Recommended parser + +Use one timestamp capture and let the date processor create `@timestamp`. + +```yaml +cpanel-access-pipeline: + source: + pipeline: + name: cpanel-log-router + + processor: + - grok: + break_on_match: true + match: + log: + - '^%{IP:client_ip} %{NOTSPACE:identity} %{NOTSPACE:user} \[%{GREEDYDATA:timestamp}\] "%{WORD:method} %{URIPATHPARAM:url} HTTP/%{NUMBER:http_version}" %{INT:http_status:int} %{NONNEGINT:bytes:long} "%{NOTSPACE:referrer}" "%{GREEDYDATA:user_agent}" "%{NOTSPACE:auth_method}" "%{GREEDYDATA:x_forwarded_for}" %{POSINT:service_port:int}$' + target_key: parsed + + - date: + match: + - key: /parsed/timestamp + patterns: + - "MM/dd/yyyy:HH:mm:ss Z" + destination: "@timestamp" + + sink: + - opensearch: + hosts: + - "https://brilliant-opensearch-cluster-master:9200" + insecure: false + username: logger + password: "" + cert: /usr/share/data-prepper/cert/root-ca.pem + index_type: custom + index: cpanel-access-log-%{yyyy.MM.dd} +``` + +## 15.3 Parsed fields + +```text +parsed.client_ip +parsed.identity +parsed.user +parsed.timestamp +parsed.method +parsed.url +parsed.http_version +parsed.http_status +parsed.bytes +parsed.referrer +parsed.user_agent +parsed.auth_method +parsed.x_forwarded_for +parsed.service_port +@timestamp +``` + +## 15.4 Important Grok issue: `%{PORT}` + +In the tested Data Prepper 2.8.0 environment: + +```text +%{PORT:service_port} +``` + +failed with: + +```text +No definition for key 'PORT' found +``` + +Use: + +```text +%{POSINT:service_port:int} +``` + +instead. + +## 15.5 YAML hierarchy + +Correct: + +```yaml +processor: + - grok: + break_on_match: true + match: + log: + - '' + target_key: parsed +``` + +`break_on_match`, `match`, and `target_key` belong under `grok:`. + +## 15.6 Parser validation test + +```bash +curl -sS -X POST \ + 'http://:31599/logs' \ + -H 'Content-Type: application/json' \ + --data '[{ + "log":"20.100.201.54 proxy - [08/09/2026:07:37:27 -0000] \"GET /wp-content/themes/twentytwenty/404.php HTTP/1.1\" 301 0 \"-\" \"-\" \"-\" \"X-Forwarded-For: 20.100.201.54\" 80", + "source_file":"/usr/local/cpanel/logs/access_log", + "log_source":"cpanel.access_log" + }]' +``` + +Then inspect a newly indexed document. + +--- + +# 16. Linux Syslog and Other Log Sources + +## 16.1 Example syslog records + +```text +2026-08-11T10:42:39.679625+00:00 bwebhosting kernel: Firewall: ... +``` + +```text +2026-08-11T10:42:39.434068+00:00 bwebhosting fluent-bit[1241036]: ... +``` + +A parser must tolerate both: + +```text +process: +process[pid]: +``` + +Do not assume a PID is always present. + +## 16.2 Recommended syslog approach + +Start with a small set of real samples from the actual host and build multiple Grok patterns if formats vary. + +If a document contains: + +```json +{ + "parsed": {} +} +``` + +then transport and indexing may be working while the Grok pattern failed to match. + +## 16.3 Do not infer source from index name + +If a document in `cpanel-system-syslog-*` contains: + +```json +{ + "log_source": "system.exim_mainlog" +} +``` + +then routing/filtering is wrong even though the index name says `syslog`. + +Verify: + +```text +_index +source_file +log_source +log +parsed +@timestamp +``` + +## 16.4 Unknown/catch-all strategy + +Use: + +```text +cpanel-all-log-YYYY.MM.dd +``` + +for sources not yet assigned to a dedicated parser. + +This provides a safe discovery path: + +```text +unknown log appears + -> stored raw + -> inspect format + -> create dedicated parser if operationally valuable +``` + +--- + +# 17. Linux Audit Log Parsing + +## 17.1 Audit event model + +A single logical Audit event can create multiple records sharing one event ID: + +```text +SYSCALL +EXECVE +CWD +PATH +PROCTITLE +``` + +Therefore: + +```text +Total Audit documents != total commands executed +``` + +Use: + +```text +audit.event_id +``` + +for correlation. + +Use primarily: + +```text +audit.log_type = EXECVE +``` + +when counting command execution records. + +## 17.2 Audit pipeline source + +```yaml +cpanel-audit-pipeline: + source: + pipeline: + name: cpanel-log-router +``` + +A defensive filter may be retained: + +```yaml +processor: + - drop_events: + drop_when: '/log_source != "audit.audit.log"' +``` + +## 17.3 Common Audit header + +Typical: + +```text +type=EXECVE msg=audit(1787117401.120:11280588): ... +``` + +Parser: + +```yaml +- grok: + break_on_match: true + match: + log: + - '^type=%{WORD:log_type} msg=audit\(%{INT:timestamp_seconds}\.%{INT:timestamp_fraction}:%{INT:event_id}\): %{GREEDYDATA:msg}' + target_key: audit +``` + +Expected: + +```json +{ + "audit": { + "log_type": "EXECVE", + "timestamp_seconds": "1787117401", + "timestamp_fraction": "120", + "event_id": "11280588", + "msg": "..." + } +} +``` + +## 17.4 Audit timestamp + +```yaml +- date: + match: + - key: /audit/timestamp_seconds + patterns: + - epoch_second + destination: "@timestamp" +``` + +## 17.5 `SYSCALL` + +Parse only when: + +```text +/audit/log_type == "SYSCALL" +``` + +Common target fields: + +```text +syscall.syscall +syscall.pid +syscall.ppid +syscall.success +syscall.exit_code +syscall.exe +syscall.comm +syscall.AUID +syscall.UID +syscall.terminal +``` + +Audit keys may be quoted or `(null)`, so maintain multiple patterns if necessary. + +## 17.6 `EXECVE` + +### Stage 1 — preserve argument count and raw argument text + +```yaml +- grok: + break_on_match: true + grok_when: '/audit/log_type == "EXECVE"' + target_key: execve + match: + /audit/msg: + - '^argc=%{INT:arg_count} %{GREEDYDATA:args_raw}$' +``` + +### Stage 2 — optional key/value split + +```yaml +- key_value: + key_value_when: '/audit/log_type == "EXECVE"' + source: /execve/args_raw + destination: /execve/args + field_delimiter_regex: '\s+' + value_split_characters: "=" + delete_value_regex: '^"|"$' +``` + +Do **not** configure both `field_delimiter_regex` and a nonempty `field_split_characters`; these options are mutually exclusive. + +### Known limitation + +Splitting on whitespace can break an argument that contains literal spaces inside quotes, for example: + +```text +a1="hello world" +``` + +Therefore always preserve: + +```text +execve.args_raw +``` + +and do not treat `execve.args.*` as a lossless reconstruction of every possible command line. Test representative Audit samples before relying on argument-level fields for security logic. + +## 17.7 Mapping-growth limitation + +Dynamic fields such as: + +```text +execve.args.a0 +execve.args.a1 +execve.args.a2 +... +``` + +can create field-count growth. + +Prefer long-term normalized fields such as: + +```text +execve.command +execve.arg_count +execve.args_raw +``` + +and limit dynamic argument mappings. + +## 17.8 `CWD` + +```yaml +- grok: + break_on_match: true + grok_when: '/audit/log_type == "CWD"' + target_key: cwd + match: + /audit/msg: + - '^cwd="%{GREEDYDATA:path}"' +``` + +## 17.9 `USER_ACCT`, `USER_START`, `CRED_ACQ` + +For each type, keep the full parser chain consistent: + +```text +grok_when + -> target_key + -> key_value_when + -> source + -> destination +``` + +Example nested key/value processing: + +```yaml +- key_value: + key_value_when: '/audit/log_type == "USER_ACCT"' + source: /user_account/msg + destination: /user_account/parsed_msg + field_split_characters: " " + value_split_characters: "=" + delete_value_regex: '^"|"$' +``` + +Known limitation: simple space splitting can break values such as: + +```text +acct="John Doe" +``` + +Preserve the raw message. + +## 17.10 `LOGIN` + +Some Audit LOGIN records contain the control separator `\u001d` before human-readable identity fields. Patterns should tolerate arbitrary content between result and `UID=` instead of assuming normal whitespace only. + +## 17.11 Audit parser verification + +Count total EXECVE: + +```bash +curl -k -u 'USER:PASSWORD' \ + -H 'Content-Type: application/json' \ + 'https://OPENSEARCH:9200/cpanel-audit-log-YYYY.MM.DD/_count' \ + -d '{ + "query": { + "match": { + "audit.log_type": "EXECVE" + } + } + }' +``` + +Count parsed EXECVE: + +```bash +curl -k -u 'USER:PASSWORD' \ + -H 'Content-Type: application/json' \ + 'https://OPENSEARCH:9200/cpanel-audit-log-YYYY.MM.DD/_count' \ + -d '{ + "query": { + "bool": { + "filter": [ + {"match": {"audit.log_type": "EXECVE"}}, + {"exists": {"field": "execve.arg_count"}} + ] + } + } + }' +``` + +Ideal validation target: + +```text +Total EXECVE == parsed EXECVE +``` + +Investigate mismatches with raw documents rather than only aggregate Grok metrics. + +--- + +# 18. OpenSearch Index Templates and Mapping + +## 18.1 Index model + +```text +cpanel-access-log-* -> parsed cPanel HTTP access logs +cpanel-system-syslog-* -> system syslog +cpanel-audit-log-* -> Linux Audit records +cpanel-all-log-* -> unknown/unhandled raw logs +``` + +## 18.2 cPanel access mappings + +Critical types: + +```text +@timestamp -> date +parsed.client_ip -> ip +parsed.http_status -> integer +parsed.bytes -> long +parsed.service_port -> integer +``` + +Template example: + +```json +{ + "index_patterns": ["cpanel-access-log-*"], + "template": { + "mappings": { + "properties": { + "@timestamp": { + "type": "date" + }, + "parsed": { + "properties": { + "client_ip": { + "type": "ip" + }, + "http_status": { + "type": "integer" + }, + "bytes": { + "type": "long" + }, + "service_port": { + "type": "integer" + } + } + } + } + } + } +} +``` + +Apply with Dev Tools or API: + +```http +PUT _index_template/cpanel-access-log-template +{ + "index_patterns": ["cpanel-access-log-*"], + "template": { + "mappings": { + "properties": { + "@timestamp": {"type": "date"}, + "parsed": { + "properties": { + "client_ip": {"type": "ip"}, + "http_status": {"type": "integer"}, + "bytes": {"type": "long"}, + "service_port": {"type": "integer"} + } + } + } + } + } +} +``` + +## 18.3 Verify mapping + +```http +GET cpanel-access-log-*/_field_caps?fields=@timestamp,parsed.client_ip,parsed.http_status,parsed.bytes,parsed.service_port +``` + +## 18.4 Audit default search fields + +A large Audit mapping can cause: + +```text +too_many_nested_clauses +maxClauseCount is set to 1024 +``` + +Restrict default query fields: + +```http +PUT cpanel-audit-log-*/_settings +{ + "index.query.default_field": [ + "log", + "audit.log_type", + "audit.event_id", + "syscall.exe", + "syscall.comm", + "syscall.AUID", + "execve.args.a0" + ] +} +``` + +Add the same setting to the matching index template so future daily indexes inherit it. + +Do not immediately increase the Boolean clause limit. Control field growth and default-field expansion first. + +## 18.5 Replica policy + +Set shard/replica counts in an index template deliberately. Do not assume Helm pod replicas automatically configure index replicas. + +--- + +# 19. End-to-End Verification + +Always test left-to-right. + +## 19.1 Source file + +```bash +tail -n 5 /usr/local/cpanel/logs/access_log +stat /usr/local/cpanel/logs/access_log +``` + +## 19.2 Fluent Bit + +```bash +systemctl status fluent-bit --no-pager +journalctl -u fluent-bit --since '10 minutes ago' --no-pager +``` + +Check Tail DB exists: + +```bash +ls -lh /var/lib/fluent-bit/state/ +``` + +Check buffer path: + +```bash +du -sh /var/lib/fluent-bit/storage/ +``` + +## 19.3 Ingress + +From the source server: + +```bash +curl -i -X POST \ + 'http://118.67.213.76:31599/logs' \ + -H 'Content-Type: application/json' \ + --data '[{"log":"manual ingress test","source_file":"manual","log_source":"test.manual"}]' +``` + +## 19.4 Data Prepper pod + +```bash +kubectl get pods -n "$NS" -l app=data-prepper -o wide +kubectl logs -n "$NS" deploy/data-prepper --tail=200 +``` + +## 19.5 Service and endpoints + +```bash +kubectl get svc data-prepper -n "$NS" -o wide +kubectl get endpointslice -n "$NS" \ + -l kubernetes.io/service-name=data-prepper -o wide +``` + +## 19.6 Port 2021 listener + +If the image lacks `ss` and `netstat`, inspect `/proc/net/tcp*`. + +```text +2021 decimal = 07E5 hexadecimal +TCP state 0A = LISTEN +TCP state 01 = ESTABLISHED +``` + +Example: + +```bash +kubectl exec -n "$NS" deploy/data-prepper -- \ + sh -c 'grep -i ":07E5" /proc/net/tcp /proc/net/tcp6 || true' +``` + +## 19.7 OpenSearch indexes + +```http +GET _cat/indices/cpanel-*?v +``` + +Remember: + +```text +index exists + docs.count=0 +``` + +does not prove successful ingestion. + +## 19.8 Recent documents + +```http +GET cpanel-access-log-*/_search +{ + "size": 5, + "sort": [ + {"@timestamp": "desc"} + ], + "_source": [ + "@timestamp", + "log_source", + "source_file", + "log", + "parsed" + ] +} +``` + +## 19.9 Last 24 hours count + +```http +GET cpanel-access-log-*/_count +{ + "query": { + "range": { + "@timestamp": { + "gte": "now-24h", + "lte": "now" + } + } + } +} +``` + +This proves: + +```text +index wildcard works ++ data exists ++ @timestamp is queryable ++ time-range query works +``` + +--- + +# 20. OpenSearch Dashboards Data Source and Workspace + +## 20.1 Enable data sources + +```yaml +data_source: + enabled: true +``` + +## 20.2 Workspaces + +```yaml +workspace: + enabled: true + +opensearch_security.multitenancy.enabled: false +``` + +## 20.3 Data source encryption master key + +The source SOP configured a data-source credential encryption master key on OpenSearch nodes. + +Requirements: + +```text +same key on every OpenSearch node +valid supported key length +never commit the key to Git +store using secure secret management +``` + +Before reusing the exact configuration key in a new OpenSearch version, verify the current OpenSearch documentation for the data-source encryption setting. + +## 20.4 Create workspace + +Recommended workspace: + +```text +Cpanel +``` + +Inside the workspace: + +1. Create an OpenSearch data source connection. +2. Use the cluster Service endpoint. +3. Use the required username/password authentication method. +4. Test the connection. +5. Create index pattern: + +```text +cpanel-access-log-* +``` + +6. Select time field: + +```text +@timestamp +``` + +--- + +# 21. cPanel Access Dashboard + +Recommended dashboard: + +```text +cPanel Access Overview +``` + +Suggested layout: + +```text ++-------------------------------------------------------------------+ +| Total Requests | Unique Clients | 2xx | 4xx | 5xx | ++-------------------------------------------------------------------+ +| Requests Over Time | ++--------------------------------+----------------------------------+ +| HTTP Status Distribution | Top Requested URLs | ++--------------------------------+----------------------------------+ +| Top Client IPs | HTTP Methods | ++--------------------------------+----------------------------------+ +| Top User Agents | ++-------------------------------------------------------------------+ +``` + +## 21.1 Total Requests + +```text +Visualization: Metric +Aggregation: Count +Label: Total Requests +``` + +## 21.2 Unique Clients + +```text +Visualization: Metric +Aggregation: Unique Count / Cardinality +Field: parsed.client_ip +Label: Unique Clients +``` + +## 21.3 2xx + +DQL: + +```text +parsed.http_status >= 200 and parsed.http_status < 300 +``` + +## 21.4 4xx + +```text +parsed.http_status >= 400 and parsed.http_status < 500 +``` + +## 21.5 5xx + +```text +parsed.http_status >= 500 and parsed.http_status < 600 +``` + +Do not use chained comparison syntax such as: + +```text +200 <= parsed.http_status < 300 +``` + +## 21.6 Requests over time + +```text +Visualization: Area/Line +Metric: Count +X axis: Date Histogram +Field: @timestamp +Interval: Auto +``` + +## 21.7 HTTP status distribution + +```text +Visualization: Pie/Donut +Metric: Count +Terms: parsed.http_status +Size: 10 +``` + +## 21.8 Top URLs + +```text +Visualization: Horizontal Bar +Metric: Count +Terms: parsed.url.keyword +Fallback: parsed.url if aggregatable +Size: 10 +``` + +## 21.9 Top client IPs + +```text +Terms: parsed.client_ip +Size: 10 +``` + +## 21.10 HTTP methods + +```text +Terms: parsed.method.keyword +Fallback: parsed.method if aggregatable +``` + +## 21.11 Top user agents + +```text +Terms: parsed.user_agent.keyword +Fallback: parsed.user_agent if aggregatable +``` + +Useful for scanners, bots, crawlers, and automated clients. + +--- + +# 22. Linux Audit Security Dashboard + +Recommended dashboard: + +```text +Linux Server Audit & Security Overview +``` + +Layout: + +```text ++-----------------------------------------------------------+ +| Commands | Logins | Failed Syscalls | Users | Audit Docs | ++-----------------------------------------------------------+ +| Audit Activity Over Time | ++---------------------------+-------------------------------+ +| Top Commands | Top Executables | ++---------------------------+-------------------------------+ +| Top Users | Top Failed Syscalls | ++---------------------------+-------------------------------+ +| Authentication & Session Activity | ++-----------------------------------------------------------+ +| Recent Audit Activity | ++-----------------------------------------------------------+ +``` + +## 22.1 Commands executed + +DQL: + +```text +audit.log_type: "EXECVE" +``` + +Metric: Count. + +## 22.2 Login events + +```text +audit.log_type: "LOGIN" +``` + +## 22.3 Failed syscalls + +```text +audit.log_type: "SYSCALL" AND syscall.success: "no" +``` + +## 22.4 Top failed syscall names + +```text +Terms: syscall.syscall.keyword +``` + +## 22.5 Top commands + +```text +DQL: audit.log_type: "EXECVE" +Terms: execve.args.a0.keyword +``` + +Use a normalized `execve.command` field instead if you implement the recommended schema improvement. + +## 22.6 Top users + +```text +DQL: audit.log_type: "SYSCALL" +Terms: syscall.AUID.keyword +``` + +## 22.7 Authentication/session activity + +DQL: + +```text +audit.log_type: ("LOGIN" OR "USER_ACCT" OR "USER_START" OR "CRED_ACQ") +``` + +Use Date Histogram on `@timestamp` and split series by `audit.log_type.keyword`. + +## 22.8 Recent activity + +Use separate tables initially because one logical Audit event spans multiple documents. + +Recent EXECVE fields: + +```text +@timestamp +audit.event_id +execve.args.a0 +execve.args.a1 +execve.args.a2 +``` + +Recent failed syscall fields: + +```text +@timestamp +audit.event_id +syscall.AUID +syscall.exe +syscall.syscall +syscall.success +syscall.exit_code +``` + +Future correlation should group by `audit.event_id`. + +--- + +# 23. Anomaly Detection, Forecasting, and Alerting + +## 23.1 Request-volume anomaly detector + +Recommended detector: + +```text +Total-Request-Detector +``` + +Index: + +```text +cpanel-access-log-* +``` + +Timestamp: + +```text +@timestamp +``` + +Feature: + +```text +Request-Count -> Count +``` + +Starting operational settings from the implementation: + +```text +Interval: 10 minutes +Frequency: 10 minutes +Window delay: 1 minute +History: 144 intervals (~24 hours) +Shingle size: 8 +Sparse data: Fill with Zeros +``` + +Do not split by high-cardinality client IP in the first detector. + +Validate: + +```text +actual request count +expected range +anomaly grade +confidence +historical timeline +``` + +## 23.2 Forecasting + +Recommended forecaster: + +```text +Total-Request-Forecast +``` + +Starting values used: + +```text +Forecast interval: 3 minutes +Window delay: 2 minutes +Horizon: 24 intervals (~72 minutes) +History: 537 intervals (~26h 51m) +``` + +These are starting parameters, not universal defaults. + +## 23.3 Alert monitor + +Flow: + +```text +Anomaly Detector + -> Monitor + -> Trigger + -> Alert + -> Action + -> Notification Channel +``` + +Starting trigger threshold: + +```text +Anomaly grade > 0.7 +AND +Confidence > 0.7 +``` + +Tune after reviewing false positives and real incidents. + +--- + +# 24. SMTP Notification Configuration + +Recorded sender configuration: + +```text +Sender name: cloud-operation +SMTP host: zsmtp.brilliant.com.bd +Port: 465 +Method: SSL +From address: cloud.operation@brilliant.com.bd +``` + +Do not put SMTP passwords into Helm values or Git. + +## 24.1 Create temporary credential files + +```bash +printf '%s' 'cloud.operation@brilliant.com.bd' > /tmp/smtp-username + +read -rsp 'SMTP password: ' SMTP_PASSWORD +echo +printf '%s' "$SMTP_PASSWORD" > /tmp/smtp-password +unset SMTP_PASSWORD +``` + +## 24.2 Create Secret for OpenSearch keystore + +```bash +kubectl create secret generic opensearch-smtp-keystore \ + -n "$NS" \ + --from-file=opensearch.notifications.core.email.cloud-operation.username=/tmp/smtp-username \ + --from-file=opensearch.notifications.core.email.cloud-operation.password=/tmp/smtp-password \ + --dry-run=client -o yaml | kubectl apply -f - + +rm -f /tmp/smtp-username /tmp/smtp-password +``` + +The key path uses the **sender name** `cloud-operation`, not the sender email address. + +## 24.3 Helm keystore value + +```yaml +keystore: + - secretName: opensearch-smtp-keystore +``` + +## 24.4 Upgrade + +```bash +helm upgrade "$OS_RELEASE" "$OS_CHART" \ + -n "$NS" \ + -f values.yaml +``` + +## 24.5 Verify keystore + +```bash +kubectl exec -n "$NS" "${OS_STS}-0" -c opensearch -- \ + /usr/share/opensearch/bin/opensearch-keystore list +``` + +Expected entries: + +```text +opensearch.notifications.core.email.cloud-operation.username +opensearch.notifications.core.email.cloud-operation.password +``` + +Verify all OpenSearch nodes if the chart injects keystore material per pod. + +## 24.6 `Invalid Addresses` troubleshooting + +Check: + +```text +From address is valid +recipient is valid +SMTP host/port are correct +SSL/TLS mode matches the server +credentials exist in keystore +credentials are present on all relevant nodes +SMTP relay permits sender +SMTP account may use configured From address +recipient is allowed +``` + +--- + +# 25. Reliability, Backup, and Recovery Tests + +## 25.1 Fluent Bit buffering recovery + +After normal ingestion is stable: + +1. Make Data Prepper temporarily unavailable. +2. Generate a few cPanel requests. +3. Confirm Fluent Bit retries. +4. Confirm filesystem backlog grows. +5. Restart Fluent Bit while Data Prepper is unavailable. +6. Restore Data Prepper. +7. Confirm buffered events drain. +8. Confirm documents reach OpenSearch. + +Monitor: + +```bash +du -sh /var/lib/fluent-bit/storage/ +journalctl -u fluent-bit -f +``` + +## 25.2 OpenSearch pod recreation + +Delete one pod: + +```bash +kubectl delete pod -n "$NS" "${OS_STS}-0" +``` + +Expected: + +```text +same StatefulSet ordinal recreated +same PVC reused +node rejoins cluster +``` + +Verify: + +```bash +kubectl get pod,pvc -n "$NS" -o wide +``` + +## 25.3 Worker restart/failure test + +Expected: + +```text +remaining OpenSearch nodes continue serving +replica shard can be promoted if primary is lost +failed node/pod rejoins according to scheduling/storage policy +``` + +## 25.4 Snapshots + +Shard replicas are not backups. + +Snapshot storage should be independent from the OpenSearch cluster and Longhorn volumes. + +Required operational practice: + +```text +configure repository +schedule snapshots +retain snapshots +monitor failures +perform restore tests +``` + +A backup is not considered valid until restore has been tested. + +--- + +# 26. Security Hardening + +## 26.1 Credentials + +Do not commit: + +```text +OpenSearch admin password +kibanaserver password +Data Prepper logger password +SMTP password +data-source encryption key +private keys +Kubernetes Secret manifests containing real Base64 data +``` + +## 26.2 Identity separation + +```text +admin -> security/cluster administration only +kibanaserver -> Dashboards backend only +logger -> Data Prepper ingestion only +human users -> read/search/dashboard permissions as required +``` + +## 26.3 Data Prepper ingress + +The original implementation used: + +```yaml +ssl: false +``` + +on the Data Prepper HTTP source and exposed it through NGINX NodePort. For production, protect the path with one or more of: + +```text +TLS +HTTP basic authentication +private routing/VPN +source IP restriction +firewall ACL +Kubernetes NetworkPolicy +authenticated ingress +``` + +The Data Prepper HTTP source supports TLS and HTTP basic authentication. + +## 26.4 OpenSearch ports + +```text +9300 -> never expose publicly +9200 -> normally private/internal +5601 -> expose through controlled reverse proxy/Ingress +2021 -> restrict to approved log senders +``` + +## 26.5 Certificate management + +Monitor: + +```text +Not Before +Not After +SAN correctness +key/cert pair +CA trust +rotation date +``` + +`root-ca-key.pem` must remain offline/encrypted. + +## 26.6 NetworkPolicy + +If NetworkPolicy is used, explicitly permit: + +```text +Dashboards -> OpenSearch :9200 +Data Prepper -> OpenSearch :9200 +OpenSearch nodes <-> OpenSearch nodes :9300 +Ingress -> Dashboards :5601 +Ingress -> Data Prepper :2021 +``` + +--- + +# 27. Operational Monitoring + +## 27.1 Kubernetes + +Monitor: + +```text +pod restarts +OOMKilled +CPU throttling +memory RSS +Pending pods +PVC usage +node pressure +scheduling failures +``` + +Commands: + +```bash +kubectl get pods -n "$NS" -o wide +kubectl top pods -n "$NS" +kubectl get events -n "$NS" --sort-by=.lastTimestamp | tail -50 +``` + +## 27.2 OpenSearch + +Monitor: + +```text +cluster health +node count +JVM heap pressure +GC +shard allocation +pending tasks +indexing latency +search latency +circuit breakers +disk watermarks +``` + +Useful APIs: + +```http +GET _cluster/health +GET _cat/nodes?v +GET _cat/shards?v +GET _cat/indices?v +GET _cluster/pending_tasks +GET _nodes/stats +``` + +## 27.3 Fluent Bit + +Monitor: + +```text +input records +output records +retries +failed chunks +storage queue usage +service restarts +``` + +## 27.4 Data Prepper + +Monitor: + +```text +requestsReceived +successRequests +badRequests +requestsRejected +grokProcessingMismatch +grokProcessingErrors +date failures +OpenSearch document errors +``` + +## 27.5 Longhorn + +Monitor: + +```text +volume health +replica count +replica placement +disk usage +snapshot growth +attachment failures +``` + +--- + +# 28. Troubleshooting Matrix + +| Symptom | Layer | Likely cause | First checks | +|---|---|---|---| +| Fluent Bit `extra indentation level found` | Fluent Bit config | classic config indentation / inline Lua | reported line; move Lua to separate file | +| Fluent Bit repeated `502 Bad Gateway` | Ingress/upstream | NGINX reachable but Data Prepper backend unhealthy | Data Prepper pod, Service, EndpointSlice, :2021 | +| Fluent Bit `connection refused` | Network/listener | endpoint unavailable | IP/port/firewall/listener | +| Data Prepper `Address already in use` | Pipeline | multiple HTTP sources on same port | keep one router HTTP source | +| Data Prepper `No definition for key 'PORT'` | Grok | unsupported pattern in tested 2.8 setup | use `POSINT` | +| Data Prepper pipeline YAML Boolean error | YAML | options at wrong hierarchy | indent under `grok:` correctly | +| `parsed: {}` | Parser | Grok did not match | compare raw line with regex | +| Wrong source in dedicated index | Routing | router/drop condition wrong | inspect `log_source`, route, new docs | +| Index exists, `docs.count=0` | Sink/ingestion | no successful documents | Data Prepper sink logs + pipeline metrics | +| Data Prepper `Request execution cancelled` / `I/O session is closed` | OpenSearch sink | DNS/TCP/TLS/auth/sink instability | test each layer to :9200 | +| OpenSearch pod `Running` but cluster unavailable | OpenSearch transport | TLS/discovery/cluster manager | pod logs, :9300, headless DNS, SAN | +| `No subject alternative DNS name ... headless` | TLS | SAN mismatch | regenerate certs with actual DNS SANs | +| `Cluster manager not present` | Cluster formation | transport TLS/discovery issue | headless Service, :9300, cert SAN, nodes_dn | +| `Unauthorized` | Authentication | wrong active credential | active Security config / user hash | +| `Resource 'admin' is reserved` | Security API | reserved resource cannot be changed by that API | update YAML + securityadmin using admin cert | +| Secret `subPath` `not a directory` | Kubernetes mount | Secret key name mismatch | `internal_users.yml` exactly | +| Dashboards `publicBaseUrl is not allowed` | Dashboards config | unsupported key in tested version | remove it | +| Dashboards 401 | Authentication | auth required/missing | security config / credential flow | +| Dashboards 404 after auth | Base path/routing | application not aware of ingress path | `basePath`, `rewriteBasePath` | +| Dashboards permission `indices:monitor/settings/get` | Authorization | backend role missing required permission | identify plugin + target index before grant | +| NGINX duplicate host/path | Ingress | catch-all route already exists | use unique host/path | +| `too_many_nested_clauses` | OpenSearch mapping/query | too many default searchable fields | restrict `index.query.default_field`; control mappings | +| SMTP `Invalid Addresses` | Notification | sender/recipient/relay/auth issue | address + keystore + relay policy | + +--- + +# 29. Troubleshooting Decision Trees + +## 29.1 Log ingestion + +```text +Is the source file receiving new lines? + | + +-- NO --> troubleshoot cPanel/application logging + | + +-- YES + | + v +Did Fluent Bit discover/read the file? + | + +-- NO --> path, permissions, Tail DB, Read_from_Head policy + | + +-- YES + | + v +Does event contain source_file + log_source? + | + +-- NO --> Path_Key / Lua classification + | + +-- YES + | + v +Does Fluent Bit POST /logs successfully? + | + +-- 502 --> Ingress reached, upstream unhealthy + +-- refused --> listener/network/firewall issue + | + +-- success + | + v +Is Data Prepper 1/1 Running and :2021 listening? + | + +-- NO --> startup/pipeline configuration + | + +-- YES + | + v +Does the router match the expected log_source? + | + +-- NO --> route expression + | + +-- YES + | + v +Does the parser populate expected fields? + | + +-- NO --> Grok/date/key-value mismatch + | + +-- YES + | + v +Can Data Prepper reach OpenSearch :9200? + | + +-- NO --> DNS/TCP/TLS/CA + | + +-- YES + | + v +Does OpenSearch authenticate/authorize logger? + | + +-- NO --> user/role/index permissions + | + +-- YES + | + v +Does the expected index contain new documents? +``` + +## 29.2 OpenSearch deployment + +```text +Pod Pending? + | + +--> node affinity + +--> hard anti-affinity + +--> taints/tolerations + +--> CPU/memory requests + +--> PVC/Longhorn + +Pod starts but crashes? + | + +--> vm.max_map_count + +--> JVM heap / OOM + +--> certificate file permissions + +--> TLS key format + +--> opensearch.yml syntax + +--> bootstrap security + +Pods run but cluster does not form? + | + +--> headless Service DNS + +--> transport :9300 + +--> SANs + +--> nodes_dn + +--> discovery settings + +REST HTTPS fails? + | + +--> REST TLS enabled + +--> cert/key paths + +--> CA + +--> hostname/SAN + +--> certificate expiry + +Authentication fails? + | + +--> active Security index + +--> internal_users.yml + +--> securityadmin applied? +``` + +## 29.3 Dashboards + +```text +Can browser reach NodePort? + | + +-- NO --> node firewall / NodePort / ingress-nginx Service + | + +-- YES + v +Does NGINX match /opensearch? + | + +-- NO --> Ingress rule / ingressClassName + | + +-- YES + v +Does Service have ready Dashboards endpoint? + | + +-- NO --> selector / pod readiness + | + +-- YES + v +Did Dashboards start without config validation errors? + | + +-- NO --> current + previous logs + | + +-- YES + v +Can Dashboards connect to OpenSearch with CA + kibanaserver? + | + +-- NO --> DNS/TLS/SAN/auth + | + +-- YES + v +Do assets/redirects stay under /opensearch? + | + +-- NO --> basePath/rewriteBasePath + | + +-- YES --> path is healthy +``` + +--- + +# 30. Daily / Weekly Operational Checklists + +## 30.1 Daily + +- [ ] All OpenSearch pods are `Running` and Ready. +- [ ] OpenSearch cluster health is expected. +- [ ] `_cat/nodes` lists all expected nodes. +- [ ] No recurring transport TLS errors. +- [ ] Data Prepper pod is Ready. +- [ ] Data Prepper has no sustained OpenSearch sink errors. +- [ ] Fluent Bit services are running. +- [ ] No repeated Fluent Bit 5xx/connection failures. +- [ ] New cPanel access documents are arriving. +- [ ] New Audit documents are arriving. +- [ ] `@timestamp` is current and sane. +- [ ] Grok processing error/mismatch rates are reviewed. +- [ ] Disk/PVC usage is below operational thresholds. +- [ ] Longhorn volumes are healthy. +- [ ] Dashboards loads and queries data. + +## 30.2 Weekly + +- [ ] Review OpenSearch JVM heap/GC trends. +- [ ] Review index growth and retention. +- [ ] Review Fluent Bit backlog utilization. +- [ ] Review Data Prepper rejected/bad requests. +- [ ] Review parsing failures/unhandled log sources. +- [ ] Review security/authentication failures. +- [ ] Review certificate expiry dates. +- [ ] Verify snapshots are succeeding. +- [ ] Periodically test restore procedure. +- [ ] Review alert quality and false positives. + +--- + +# 31. Windows Fluent Bit Variant + +This is an optional extension for downloaded Windows log files. + +## 31.1 Recommended directory separation + +```text +C:\fluent-bit\ + ├── logs\ + ├── state\ + └── buffer\ +``` + +Do not place the Tail DB inside a wildcard source directory if it could be ingested accidentally. + +## 31.2 Example concept + +```ini +[SERVICE] + Flush 1 + Daemon Off + Log_Level info + storage.metrics On + storage.path C:\fluent-bit\buffer + storage.sync full + storage.backlog.mem_limit 500M + +[INPUT] + Name tail + Path C:\fluent-bit\logs\*.log + Path_Key source_file + Tag downloaded.logs + DB C:\fluent-bit\state\tail.db + DB.Journal_Mode wal + Read_from_Head true + Refresh_Interval 5 + Skip_Long_Lines true + Mem_Buf_Limit 512MB + Storage.Type filesystem + +[OUTPUT] + Name http + Host 118.67.213.76 + Port 31599 + Format json + HTTP_Method POST + URI /logs + Match * + Retry_Limit no_limits + Storage.Total_Limit_Size 4G +``` + +Before ingesting all historical files: + +1. Test one representative file. +2. Confirm existing content is read. +3. Restart Fluent Bit and confirm offset resume behavior. +4. Verify `source_file`. +5. Verify one test document in OpenSearch. +6. Only then expand the wildcard. + +--- + +# 32. Cheat Sheet + +## 32.1 Complete process summary + +```text +1. Verify Kubernetes nodes, Longhorn, vm.max_map_count +2. Verify/generate Root CA, node certs, admin cert +3. Create Kubernetes Secrets +4. Configure OpenSearch Helm values +5. helm lint +6. helm template +7. kubectl --dry-run=server +8. helm upgrade --install OpenSearch +9. Verify pods/PVC/TLS/cluster health/security +10. Deploy Dashboards with CA + kibanaserver Secret +11. Configure /opensearch Ingress/basePath +12. Deploy Data Prepper with ONE HTTP :2021 router +13. Configure dedicated access/syslog/audit/catch-all pipelines +14. Configure Fluent Bit Tail DB + filesystem buffer + log_source +15. Verify /logs end-to-end +16. Create index templates/mappings +17. Create Dashboards workspace/index patterns +18. Build cPanel and Linux Audit dashboards +19. Configure anomaly detector / forecast / alerts +20. Configure SMTP keystore +21. Test failures, snapshots, and recovery +``` + +## 32.2 Kubernetes/OpenSearch quick commands + +```bash +export NS=observability +export OS_RELEASE=brilliant-opensearch +export OS_STS=brilliant-opensearch-cluster-master +export OS_SVC=brilliant-opensearch-cluster-master + +kubectl get pods -n "$NS" -o wide +kubectl get sts -n "$NS" +kubectl get svc -n "$NS" +kubectl get pvc -n "$NS" +kubectl get ingress -A + +kubectl logs -n "$NS" "${OS_STS}-0" -c opensearch --tail=200 +kubectl logs -n "$NS" "${OS_STS}-0" -c opensearch --previous --tail=200 + +kubectl port-forward -n "$NS" svc/"$OS_SVC" 9200:9200 +curl -k -u 'admin:' https://localhost:9200/_cluster/health?pretty +curl -k -u 'admin:' 'https://localhost:9200/_cat/nodes?v' +``` + +## 32.3 Helm quick commands + +```bash +helm lint "$OS_CHART" -f values.yaml + +helm template "$OS_RELEASE" "$OS_CHART" \ + -n "$NS" -f values.yaml > rendered-opensearch.yaml + +kubectl apply --dry-run=server -f rendered-opensearch.yaml + +helm upgrade --install "$OS_RELEASE" "$OS_CHART" \ + -n "$NS" -f values.yaml + +kubectl rollout status statefulset/"$OS_STS" -n "$NS" +``` + +## 32.4 TLS quick commands + +```bash +openssl verify -CAfile root-ca.pem node-0.pem +openssl x509 -in node-0.pem -noout -subject -nameopt RFC2253 +openssl x509 -in node-0.pem -noout -dates +openssl x509 -in node-0.pem -noout -text | sed -n '/Subject Alternative Name/,+2p' +``` + +## 32.5 Data Prepper quick commands + +```bash +kubectl get pods -n observability -l app=data-prepper -o wide +kubectl get svc data-prepper -n observability -o wide +kubectl get endpointslice -n observability \ + -l kubernetes.io/service-name=data-prepper -o wide +kubectl logs -n observability deploy/data-prepper --tail=200 + +kubectl exec -n observability deploy/data-prepper -- \ + sh -c 'grep -i ":07E5" /proc/net/tcp /proc/net/tcp6 || true' +``` + +## 32.6 Fluent Bit quick commands + +```bash +systemctl status fluent-bit --no-pager +journalctl -u fluent-bit --since '10 minutes ago' --no-pager + +du -sh /var/lib/fluent-bit/storage/ +ls -lh /var/lib/fluent-bit/state/ +``` + +## 32.7 Test `/logs` + +```bash +curl -i -X POST \ + 'http://118.67.213.76:31599/logs' \ + -H 'Content-Type: application/json' \ + --data '[{"log":"manual test","source_file":"manual","log_source":"test.manual"}]' +``` + +## 32.8 OpenSearch Dev Tools quick queries + +```http +GET _cluster/health +GET _cat/nodes?v +GET _cat/indices/cpanel-*?v + +GET cpanel-access-log-*/_count +{ + "query": { + "range": { + "@timestamp": { + "gte": "now-24h", + "lte": "now" + } + } + } +} + +GET cpanel-access-log-*/_search +{ + "size": 5, + "sort": [{"@timestamp": "desc"}] +} +``` + +## 32.9 DQL quick filters + +```text +2xx: +parsed.http_status >= 200 and parsed.http_status < 300 + +4xx: +parsed.http_status >= 400 and parsed.http_status < 500 + +5xx: +parsed.http_status >= 500 and parsed.http_status < 600 + +Audit commands: +audit.log_type: "EXECVE" + +Failed syscalls: +audit.log_type: "SYSCALL" AND syscall.success: "no" + +Authentication activity: +audit.log_type: ("LOGIN" OR "USER_ACCT" OR "USER_START" OR "CRED_ACQ") +``` + +## 32.10 Error-to-layer quick map + +```text +502 from Fluent Bit -> NGINX reachable; check Data Prepper upstream +connection refused -> listener/network +parsed: {} -> Grok mismatch +index docs.count=0 -> ingestion/sink not proven +SAN mismatch -> certificate DNS identity +Unauthorized -> authentication +security_exception -> authorization +404 after Dashboards auth-> basePath/routing +CrashLoop after config -> application validation/logs +``` + +--- + +# 33. Lessons Learned and Known Limitations + +1. **Helm rendering is mandatory.** Inspect final StatefulSet/Deployment, not only `values.yaml`. +2. **`Running` is not enough.** Validate readiness, application logs, OpenSearch cluster health, and backend connectivity separately. +3. **Transport TLS and REST TLS are different boundaries.** `curl -k` against port 9200 cannot repair transport TLS on 9300. +4. **SANs matter.** Correct CA trust is insufficient when the Service/headless DNS is missing from the certificate. +5. **`kibanaserver` is a user; `kibana_server` is a role.** Do not confuse them. +6. **A mounted Security YAML file is not automatically the active Security index.** Apply changes appropriately with Security tooling. +7. **Secret `subPath` names match exact keys.** `internal_users.yaml` and `internal_users.yml` are different keys. +8. **One Data Prepper HTTP source per listener.** Route internally to downstream pipelines. +9. **A sink without `routes:` receives all events.** Define an explicit unknown route if the catch-all must contain only unhandled sources. +10. **Fluent Bit Tail DB is not the event queue.** It remembers offsets; filesystem storage buffers undelivered chunks. +11. **Unlimited retries do not mean unlimited durability.** A finite filesystem queue can still overflow and drop old chunks. +12. **`parsed: {}` means transport can be healthy while parsing failed.** Debug the raw line against the parser. +13. **Audit records are multi-record logical events.** Correlate using `audit.event_id`. +14. **Whitespace-splitting EXECVE arguments is not fully quote-aware.** Preserve `args_raw`. +15. **Dynamic Audit arguments can create mapping explosion.** Normalize high-value fields and restrict default searchable fields. +16. **Index existence does not prove document ingestion.** Always inspect `docs.count` and sample `_source`. +17. **NGINX Ingress routing is controller-wide.** Duplicate hostless `/` rules can conflict across namespaces. +18. **Dashboards path hosting requires application-aware basePath configuration.** +19. **Version-specific settings matter.** A configuration key accepted by Kibana or another OpenSearch Dashboards version may be rejected by 3.7.0. +20. **Least privilege first.** Investigate the action and target resource before granting broad permissions. +21. **OpenSearch replicas and Longhorn replicas consume storage independently.** +22. **OpenSearch shard replicas are not backups.** Snapshot and restore testing remain mandatory. + +--- + +# 34. Final Definition of Done + +The platform is considered complete only when all of the following are true. + +## OpenSearch + +- [ ] Three OpenSearch pods are Ready. +- [ ] Pods are on separate eligible Kubernetes nodes. +- [ ] Three 120Gi Longhorn PVCs are Bound. +- [ ] Transport TLS works with correct SANs. +- [ ] REST HTTPS works. +- [ ] Node certificate DNs match `nodes_dn`. +- [ ] Admin certificate DN matches `admin_dn`. +- [ ] Cluster manager is elected. +- [ ] `_cat/nodes` lists all three nodes. +- [ ] Cluster health is expected. +- [ ] Default/demo credentials are removed or changed. +- [ ] Real secrets are absent from Git. + +## Dashboards + +- [ ] Dashboards is Ready. +- [ ] CA is mounted. +- [ ] `kibanaserver` password comes from a Secret. +- [ ] TLS verification is `full`. +- [ ] `/opensearch` basePath works. +- [ ] Ingress does not conflict with Grafana `/`. +- [ ] Workspaces work with Security multi-tenancy disabled. +- [ ] Data source connection succeeds. + +## Fluent Bit + +- [ ] Tail paths are correct. +- [ ] Tail DB is persistent and writable. +- [ ] Filesystem buffer is writable. +- [ ] `source_file` is present. +- [ ] `log_source` is correct. +- [ ] `/logs` delivery succeeds. +- [ ] Retry/backlog behavior is tested. +- [ ] Queue usage is monitored. + +## Data Prepper + +- [ ] Only one HTTP source owns port 2021. +- [ ] Service and EndpointSlice point to the current pod. +- [ ] Health endpoint is healthy. +- [ ] Router matches known log sources. +- [ ] Catch-all contains only unhandled sources. +- [ ] Access Grok populates `parsed.*`. +- [ ] Audit parsers populate expected type-specific fields. +- [ ] Date processors create correct `@timestamp`. +- [ ] OpenSearch sink has no persistent errors. + +## OpenSearch data + +- [ ] `cpanel-access-log-*` contains documents. +- [ ] `cpanel-system-syslog-*` contains expected syslog only. +- [ ] `cpanel-audit-log-*` contains Audit records. +- [ ] `cpanel-all-log-*` exposes unknown/unhandled sources. +- [ ] cPanel access field mappings are correct. +- [ ] Audit default query fields are controlled. +- [ ] Retention/ISM strategy is defined. + +## Dashboards and alerting + +- [ ] cPanel Access Overview exists. +- [ ] Linux Server Audit & Security Overview exists. +- [ ] Anomaly detector runs and produces sensible output. +- [ ] Forecasting runs if required. +- [ ] Monitor and trigger exist. +- [ ] SMTP credentials exist in OpenSearch keystore. +- [ ] Test notification succeeds. +- [ ] Real alert action succeeds. + +## Operations + +- [ ] OpenSearch pod-failure test completed. +- [ ] Fluent Bit/Data Prepper interruption recovery tested. +- [ ] Snapshot repository configured. +- [ ] Snapshot schedule configured. +- [ ] Restore test completed. +- [ ] Certificate-expiry monitoring configured. +- [ ] Daily/weekly checks assigned to operations. + +--- + +## Verification References Used for This Consolidation + +This combined SOP was cross-checked against the official documentation for: + +- OpenSearch Helm installation and chart configuration. +- OpenSearch Security YAML configuration and `securityadmin.sh` behavior. +- OpenSearch index replica settings and cluster health semantics. +- OpenSearch Dashboards Workspaces and multi-tenancy compatibility. +- OpenSearch Dashboards multiple data sources. +- Data Prepper HTTP source, Grok processor, key-value processor, conditional routing, server API port, and OpenSearch sink. +- Fluent Bit Tail input, filesystem buffering, retry semantics, and output queue limits. +- Kubernetes StatefulSet/headless Service behavior and Kubernetes Secret/volume semantics. + +When upgrading any major/minor component, rerun the same validation flow against the documentation for the exact version before applying old configuration unchanged. + +--- + +**Prepared by: Md. Mosiur Rahnman**