# OpenSearch + cPanel/Linux Observability Platform — Complete SOP > **Canonical combined runbook** for deploying and operating OpenSearch on Kubernetes, OpenSearch Dashboards, Fluent Bit, Data Prepper, cPanel/Linux log ingestion, Linux Audit ingestion, dashboards, anomaly detection, forecasting, alerting, and troubleshooting. > > **Prepared by:** Md. Mosiur Rahnman > **Document type:** Standard Operating Procedure (SOP) / Future Reference Runbook > **Primary environment documented:** Kubernetes/K3s, OpenSearch 3.7.0, OpenSearch Dashboards 3.7.0, Data Prepper 2.8.0, Longhorn, NGINX Ingress, cPanel/Linux > **Primary namespace:** `observability` --- ## Document Control | Item | Value | |---|---| | Prepared by | Md. Mosiur Rahnman | | Scope | OpenSearch platform + cPanel/Linux log observability | | OpenSearch version used in the implementation | 3.7.0 | | OpenSearch Dashboards version used | 3.7.0 | | Data Prepper version used | 2.8.0 | | Kubernetes namespace | `observability` | | OpenSearch Helm release | `brilliant-opensearch` | | OpenSearch cluster name | `brilliant-opensearch-cluster` | | OpenSearch StatefulSet | `brilliant-opensearch-cluster-master` | | OpenSearch Service | `brilliant-opensearch-cluster-master` | | Data Prepper Service | `data-prepper` | | StorageClass | `longhorn` | | OpenSearch pod count | 3 | | OpenSearch PVC size | 120Gi per pod | | Dashboards external path | `/opensearch` | | Data Prepper ingestion path | `/logs` | | NGINX HTTP NodePort observed | `31599` | | NGINX HTTPS NodePort observed | `31186` | | cPanel access log | `/usr/local/cpanel/logs/access_log` | > [!IMPORTANT] > Values such as node IPs, NodePorts, release names, Service names, chart paths, certificate DNs, and storage topology are **environment-specific**. Verify them before reusing this SOP in another cluster. --- ## Table of Contents 1. [Purpose and Scope](#1-purpose-and-scope) 2. [Final Architecture](#2-final-architecture) 3. [Core Design Principles](#3-core-design-principles) 4. [Environment Variables and Naming](#4-environment-variables-and-naming) 5. [Pre-Deployment Prerequisites](#5-pre-deployment-prerequisites) 6. [OpenSearch PKI and TLS](#6-opensearch-pki-and-tls) 7. [Kubernetes Secrets](#7-kubernetes-secrets) 8. [OpenSearch Helm Configuration](#8-opensearch-helm-configuration) 9. [Render, Validate, and Deploy OpenSearch](#9-render-validate-and-deploy-opensearch) 10. [OpenSearch Security Initialization and Admin Password](#10-opensearch-security-initialization-and-admin-password) 11. [OpenSearch Cluster Verification](#11-opensearch-cluster-verification) 12. [OpenSearch Dashboards Deployment](#12-opensearch-dashboards-deployment) 13. [Data Prepper — Full Deployment, Routing, Parsing, and Verification](#13-data-prepper--full-deployment-routing-parsing-and-verification) 14. [Fluent Bit — Full cPanel/Linux Collection Procedure](#14-fluent-bit--full-cpanellinux-collection-procedure) 15. [cPanel Access Log Parsing](#15-cpanel-access-log-parsing) 16. [Linux Syslog and Other Log Sources](#16-linux-syslog-and-other-log-sources) 17. [Linux Audit Log Parsing](#17-linux-audit-log-parsing) 18. [OpenSearch Index Templates and Mapping](#18-opensearch-index-templates-and-mapping) 19. [End-to-End Verification](#19-end-to-end-verification) 20. [OpenSearch Dashboards Data Source and Workspace](#20-opensearch-dashboards-data-source-and-workspace) 21. [cPanel Access Dashboard](#21-cpanel-access-dashboard) 22. [Linux Audit Security Dashboard](#22-linux-audit-security-dashboard) 23. [Anomaly Detection, Forecasting, and Alerting](#23-anomaly-detection-forecasting-and-alerting) 24. [SMTP Notification Configuration](#24-smtp-notification-configuration) 25. [Reliability, Backup, and Recovery Tests](#25-reliability-backup-and-recovery-tests) 26. [Security Hardening](#26-security-hardening) 27. [Operational Monitoring](#27-operational-monitoring) 28. [Troubleshooting Matrix](#28-troubleshooting-matrix) 29. [Troubleshooting Decision Trees](#29-troubleshooting-decision-trees) 30. [Daily / Weekly Operational Checklists](#30-daily--weekly-operational-checklists) 31. [Windows Fluent Bit Variant](#31-windows-fluent-bit-variant) 32. [Cheat Sheet](#32-cheat-sheet) 33. [Lessons Learned and Known Limitations](#33-lessons-learned-and-known-limitations) 34. [Final Definition of Done](#34-final-definition-of-done) --- # 1. Purpose and Scope This SOP consolidates the complete implementation into one operational reference. It covers the path from Kubernetes infrastructure through searchable and visualized cPanel/Linux logs: ```text cPanel / Linux Servers | | log files v Fluent Bit - Tail input - Offset DB - Filesystem buffering - source_file - log_source classification | | HTTP/HTTPS POST /logs v NGINX Ingress | v Data Prepper :2021 - One HTTP source - Router - Grok/date/key-value processors | | HTTPS :9200 v OpenSearch 3-node cluster - Longhorn PVCs - REST TLS - Transport TLS - Security plugin | v OpenSearch Dashboards - Discover / Logs - Workspaces - Dashboards - Anomaly Detection - Forecasting - Alerting - Email notifications ``` This document preserves the tested implementation while separating: - **Production baseline** — settings that should normally remain enabled. - **Environment-specific values** — values that must be rechecked before reuse. - **Diagnostic workarounds** — temporary troubleshooting settings that should not become permanent defaults. - **Known limitations** — parser or reliability behaviors that need to be understood rather than hidden. --- # 2. Final Architecture ## 2.1 Kubernetes/OpenSearch topology ```text +--------------------------------------+ | Kubernetes / K3s | | Namespace: observability | | | Browser | OpenSearch Dashboards | | | :5601 | | HTTP/HTTPS | | HTTPS | v | v | NGINX Ingress ---------->| OpenSearch Service :9200 | ^ | | | | | +------+------+------+ | | | | | | | | | pod-0 pod-1 pod-2 | | | | | | | | | PVC-0 PVC-1 PVC-2 | | | \________ Transport TLS :9300 ____/| | | | | | Data Prepper :2021 | | | | HTTPS :9200 | | | +----------------------------->| | +--------------------------------------+ | | HTTP/HTTPS /logs | Fluent Bit on cPanel/Linux servers ``` ## 2.2 Log-routing architecture ```text Fluent Bit | | JSON + source_file + log_source v Data Prepper router | +--> cpanel.access_log --> access pipeline --> cpanel-access-log-YYYY.MM.dd | +--> system.syslog ----> syslog pipeline --> cpanel-system-syslog-YYYY.MM.dd | +--> audit.audit.log --> audit pipeline ---> cpanel-audit-log-YYYY.MM.dd | +--> unknown sources --> catch-all --------> cpanel-all-log-YYYY.MM.dd ``` --- # 3. Core Design Principles 1. **Validate one layer at a time.** Do not debug OpenSearch if Fluent Bit is not reading the source file. 2. **Render Helm before applying it.** A valid `values.yaml` can still render an invalid or unintended Kubernetes manifest. 3. **Keep transport, parsing, routing, and storage failures separate.** `parsed: {}` is a parser problem; `502` is normally an upstream availability problem. 4. **Use one Data Prepper HTTP listener per port.** Multiple pipelines must not bind `:2021` independently. 5. **Classify logs early.** Fluent Bit knows the original file path, so create `source_file` and normalized `log_source` before transport. 6. **Use dedicated downstream pipelines for different formats.** cPanel access logs, syslog, and Auditd have fundamentally different schemas. 7. **Preserve the raw log.** Parsing should enrich the event, not destroy the original message. 8. **Use least privilege.** Do not reuse the OpenSearch admin credential for Dashboards or ingestion. 9. **Treat TLS trust and TLS hostname validation as different checks.** A trusted CA does not fix a SAN mismatch. 10. **Treat Kubernetes `Running` and application health as different states.** A pod can run while OpenSearch cluster formation, authentication, or authorization is broken. 11. **OpenSearch shard replicas are not backups.** Use snapshots to independent storage. 12. **Base64 is encoding, not encryption.** Kubernetes Secret output must still be treated as sensitive. 13. **Do not commit real passwords, private keys, SMTP credentials, or encryption master keys to Git.** --- # 4. Environment Variables and Naming Use variables in shell sessions to reduce copy/paste mistakes. ```bash export NS="observability" export OS_RELEASE="brilliant-opensearch" export OS_CLUSTER="brilliant-opensearch-cluster" export OS_STS="brilliant-opensearch-cluster-master" export OS_SVC="brilliant-opensearch-cluster-master" export OS_HEADLESS_SVC="brilliant-opensearch-cluster-master-headless" export DP_SVC="data-prepper" export DP_PORT="2021" export DASH_BASEPATH="/opensearch" export INGRESS_HTTP_NODEPORT="31599" export INGRESS_HTTPS_NODEPORT="31186" ``` Check the real values instead of assuming them: ```bash kubectl get ns helm list -A kubectl get sts,deploy,svc,ingress -A kubectl get nodes -o wide ``` For the local Helm chart used in the implementation: ```bash export OS_CHART="${HOME}/monitoring/helm-charts/charts/opensearch" ``` Verify it exists: ```bash test -f "$OS_CHART/Chart.yaml" && echo "OpenSearch chart found" ``` --- # 5. Pre-Deployment Prerequisites ## 5.1 Client tools Verify required tools: ```bash kubectl version --client helm version openssl version curl --version ``` Optional but useful: ```bash jq --version yq --version ``` ## 5.2 Kubernetes nodes ```bash kubectl get nodes -o wide kubectl get nodes --show-labels kubectl describe node k8s-storage-01 | sed -n '/Taints:/,/Unschedulable:/p' ``` Target placement from the implementation: ```text k8s-storage-01 -> eligible, dedicated=storage:NoSchedule k8s-worker-01 -> eligible k8s-worker-02 -> eligible k8s-cp-01 -> excluded from OpenSearch ``` If `k8s-storage-01` is tainted: ```text dedicated=storage:NoSchedule ``` OpenSearch requires a matching toleration if that node should be eligible. ## 5.3 Longhorn ```bash kubectl get storageclass kubectl get pods -n longhorn-system ``` Expected StorageClass: ```text longhorn ``` Storage planning for the documented cluster: ```text 3 OpenSearch pods x 120Gi = 360Gi logical PVC capacity ``` Longhorn replication consumes additional physical storage. OpenSearch shard replication consumes additional logical data copies as well; they are separate redundancy layers. ## 5.4 `vm.max_map_count` Check every eligible OpenSearch node: ```bash for node in k8s-storage-01 k8s-worker-01 k8s-worker-02; do echo "=== $node ===" ssh "$node" 'sysctl vm.max_map_count' done ``` Target used by the Helm chart: ```text vm.max_map_count = 262144 ``` Prefer configuring the kernel permanently on the host: ```bash cat >/etc/sysctl.d/99-opensearch.conf <<'EOF_SYSCTL' vm.max_map_count=262144 EOF_SYSCTL sysctl --system ``` If the hosts are already configured, disable privileged sysctl init mechanisms in Helm. Do not enable multiple sysctl methods unnecessarily. ## 5.5 Namespace ```bash kubectl create namespace "$NS" --dry-run=client -o yaml | kubectl apply -f - ``` --- # 6. OpenSearch PKI and TLS ## 6.1 Certificate roles Use separate certificate purposes: ```text Root CA ├── signs node certificates └── signs admin certificate Node certificates ├── REST TLS server identity ├── transport TLS server identity └── transport TLS client identity Admin certificate └── privileged OpenSearch Security administration ``` Never mount the Root CA private key into OpenSearch. ```text root-ca-key.pem -> offline/encrypted CA material only root-ca.pem -> safe to distribute as trust anchor ``` ## 6.2 Expected PKI files ```text root-ca.pem root-ca-key.pem admin.pem admin-key.pem node-0.pem node-0-key.pem node-1.pem node-1-key.pem node-2.pem node-2-key.pem ``` ## 6.3 SAN design Use a short certificate CN and put Kubernetes DNS identities in SANs. Common Service DNS names: ```text brilliant-opensearch-cluster-master brilliant-opensearch-cluster-master.observability brilliant-opensearch-cluster-master.observability.svc brilliant-opensearch-cluster-master.observability.svc.cluster.local brilliant-opensearch-cluster-master-headless brilliant-opensearch-cluster-master-headless.observability brilliant-opensearch-cluster-master-headless.observability.svc brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local ``` Pod-specific names should also be included where transport communication uses them: ```text brilliant-opensearch-cluster-master-0.brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local brilliant-opensearch-cluster-master-1.brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local brilliant-opensearch-cluster-master-2.brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local ``` ## 6.4 Optional OpenSSL generation template If certificates already exist, skip generation and perform verification only. ### Root CA ```bash openssl genrsa -out root-ca-key.pem 4096 openssl req -x509 -new -sha256 \ -key root-ca-key.pem \ -days 3650 \ -out root-ca.pem \ -subj '/C=BD/ST=Dhaka/L=Dhaka/O=Brilliant/OU=OpenSearch CA/CN=Brilliant OpenSearch Root CA' ``` ### Admin certificate ```bash openssl genrsa -out admin-key-rsa.pem 2048 openssl pkcs8 -topk8 -nocrypt \ -in admin-key-rsa.pem \ -out admin-key.pem openssl req -new \ -key admin-key.pem \ -out admin.csr \ -subj '/C=BD/ST=Dhaka/L=Dhaka/O=Brilliant/OU=OpenSearch Administrators/CN=opensearch-admin' cat >admin.ext <<'EOF_ADMIN_EXT' basicConstraints=CA:FALSE keyUsage=digitalSignature,keyEncipherment extendedKeyUsage=clientAuth EOF_ADMIN_EXT openssl x509 -req \ -in admin.csr \ -CA root-ca.pem \ -CAkey root-ca-key.pem \ -CAcreateserial \ -out admin.pem \ -days 825 \ -sha256 \ -extfile admin.ext ``` ### Node certificate example for pod 0 Create a SAN file containing the actual Service and pod DNS names: ```bash cat >node-0.ext <<'EOF_NODE0_EXT' basicConstraints=CA:FALSE keyUsage=digitalSignature,keyEncipherment extendedKeyUsage=serverAuth,clientAuth subjectAltName=@alt_names [alt_names] DNS.1=brilliant-opensearch-cluster-master DNS.2=brilliant-opensearch-cluster-master.observability DNS.3=brilliant-opensearch-cluster-master.observability.svc DNS.4=brilliant-opensearch-cluster-master.observability.svc.cluster.local DNS.5=brilliant-opensearch-cluster-master-headless DNS.6=brilliant-opensearch-cluster-master-headless.observability DNS.7=brilliant-opensearch-cluster-master-headless.observability.svc DNS.8=brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local DNS.9=brilliant-opensearch-cluster-master-0.brilliant-opensearch-cluster-master-headless DNS.10=brilliant-opensearch-cluster-master-0.brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local EOF_NODE0_EXT openssl genrsa -out node-0-key-rsa.pem 2048 openssl pkcs8 -topk8 -nocrypt \ -in node-0-key-rsa.pem \ -out node-0-key.pem openssl req -new \ -key node-0-key.pem \ -out node-0.csr \ -subj '/C=BD/ST=Dhaka/L=Dhaka/O=Brilliant/OU=OpenSearch Nodes/CN=opensearch-master-0' openssl x509 -req \ -in node-0.csr \ -CA root-ca.pem \ -CAkey root-ca-key.pem \ -CAcreateserial \ -out node-0.pem \ -days 825 \ -sha256 \ -extfile node-0.ext ``` Repeat with the correct CN and pod-specific SAN for node 1 and node 2. ## 6.5 Certificate verification Verify chain: ```bash openssl verify -CAfile root-ca.pem admin.pem openssl verify -CAfile root-ca.pem node-0.pem openssl verify -CAfile root-ca.pem node-1.pem openssl verify -CAfile root-ca.pem node-2.pem ``` Verify subjects: ```bash openssl x509 -in admin.pem -noout -subject -nameopt RFC2253 openssl x509 -in node-0.pem -noout -subject -nameopt RFC2253 openssl x509 -in node-1.pem -noout -subject -nameopt RFC2253 openssl x509 -in node-2.pem -noout -subject -nameopt RFC2253 ``` Verify SANs: ```bash openssl x509 -in node-0.pem -noout -text | sed -n '/Subject Alternative Name/,+2p' ``` Verify EKU: ```bash openssl x509 -in node-0.pem -noout -text | sed -n '/Extended Key Usage/,+2p' ``` Verify certificate/key match: ```bash openssl x509 -in node-0.pem -pubkey -noout | sha256sum openssl pkey -in node-0-key.pem -pubout | sha256sum ``` The hashes must match. Verify validity: ```bash openssl x509 -in node-0.pem -noout -dates ``` --- # 7. Kubernetes Secrets ## 7.1 Initial admin password Do not place the real password in Git. ```bash read -rsp 'Initial OpenSearch admin password: ' OPENSEARCH_INITIAL_ADMIN_PASSWORD echo kubectl create secret generic opensearch-secret \ -n "$NS" \ --from-literal=OPENSEARCH_INITIAL_ADMIN_PASSWORD="$OPENSEARCH_INITIAL_ADMIN_PASSWORD" \ --dry-run=client -o yaml | kubectl apply -f - unset OPENSEARCH_INITIAL_ADMIN_PASSWORD ``` ## 7.2 Node certificate Secrets ```bash kubectl create secret generic opensearch-node-0-cert \ -n "$NS" \ --from-file=node-0.pem \ --from-file=node-0-key.pem \ --from-file=root-ca.pem \ --dry-run=client -o yaml | kubectl apply -f - kubectl create secret generic opensearch-node-1-cert \ -n "$NS" \ --from-file=node-1.pem \ --from-file=node-1-key.pem \ --from-file=root-ca.pem \ --dry-run=client -o yaml | kubectl apply -f - kubectl create secret generic opensearch-node-2-cert \ -n "$NS" \ --from-file=node-2.pem \ --from-file=node-2-key.pem \ --from-file=root-ca.pem \ --dry-run=client -o yaml | kubectl apply -f - ``` ## 7.3 Admin certificate Secret ```bash kubectl create secret generic opensearch-admin-cert \ -n "$NS" \ --from-file=admin.pem \ --from-file=admin-key.pem \ --from-file=root-ca.pem \ --dry-run=client -o yaml | kubectl apply -f - ``` ## 7.4 Verify Secret metadata without dumping data ```bash kubectl get secret -n "$NS" \ opensearch-secret \ opensearch-node-0-cert \ opensearch-node-1-cert \ opensearch-node-2-cert \ opensearch-admin-cert ``` Avoid: ```bash kubectl get secret -o yaml ``` when the output would expose sensitive Base64 data in terminals, tickets, screenshots, or chat logs. --- # 8. OpenSearch Helm Configuration ## 8.1 Resource design Per OpenSearch pod: ```text CPU request: 2 CPU limit: 4 Memory request: 6Gi Memory limit: 8Gi JVM heap: 3Gi PVC: 120Gi ``` ## 8.2 Helm values baseline The exact chart schema must always be checked with the pinned chart: ```bash helm show values "$OS_CHART" > /tmp/opensearch-default-values.yaml ``` Critical values for this environment: ```yaml clusterName: brilliant-opensearch-cluster nodeGroup: master singleNode: false masterService: brilliant-opensearch-cluster-master roles: - master - ingest - data - remote_cluster_client replicas: 3 image: repository: opensearchproject/opensearch tag: "3.7.0" pullPolicy: IfNotPresent opensearchJavaOpts: "-Xms3g -Xmx3g" resources: requests: cpu: "2" memory: 6Gi limits: cpu: "4" memory: 8Gi persistence: enabled: true storageClass: longhorn accessModes: - ReadWriteOnce size: 120Gi antiAffinity: hard antiAffinityTopologyKey: kubernetes.io/hostname podManagementPolicy: Parallel protocol: https httpPort: 9200 transportPort: 9300 metricsPort: 9600 podSecurityContext: fsGroup: 1000 runAsUser: 1000 securityContext: runAsNonRoot: true runAsUser: 1000 capabilities: drop: - ALL envFrom: - secretRef: name: opensearch-secret secretMounts: [] sysctl: enabled: false sysctlInit: enabled: false ``` > [!NOTE] > If the worker nodes are **not** preconfigured with `vm.max_map_count=262144`, enable exactly one supported sysctl method. Do not enable both pod-level sysctl and a privileged sysctl init container unless you have a specific reason. ## 8.3 Node scheduling Storage taint toleration: ```yaml tolerations: - key: dedicated operator: Equal value: storage effect: NoSchedule ``` Use labels/affinity to include only the intended storage/worker nodes. Do not rely on anti-affinity to exclude the control plane; anti-affinity controls pod co-location, not arbitrary node blacklisting. Verify rendered affinity before install. ## 8.4 Per-pod certificate selection The main OpenSearch container should see only its selected runtime certificate set. ```yaml extraVolumes: - name: node-0-source secret: secretName: opensearch-node-0-cert defaultMode: 0440 - name: node-1-source secret: secretName: opensearch-node-1-cert defaultMode: 0440 - name: node-2-source secret: secretName: opensearch-node-2-cert defaultMode: 0440 - name: runtime-certs emptyDir: {} extraVolumeMounts: - name: runtime-certs mountPath: /usr/share/opensearch/config/certs readOnly: true ``` Init container: ```yaml extraInitContainers: - name: select-node-certificate image: busybox:1.36 command: - sh - -ec - | ordinal="${HOSTNAME##*-}" case "$ordinal" in 0|1|2) ;; *) echo "Unexpected StatefulSet ordinal: $ordinal" >&2 exit 1 ;; esac src="/source/node-${ordinal}" cert="${src}/node-${ordinal}.pem" key="${src}/node-${ordinal}-key.pem" ca="${src}/root-ca.pem" test -r "$cert" test -r "$key" test -r "$ca" cp "$cert" /runtime/node.pem cp "$key" /runtime/node-key.pem cp "$ca" /runtime/root-ca.pem chown 1000:1000 /runtime/node.pem /runtime/node-key.pem /runtime/root-ca.pem chmod 0644 /runtime/node.pem /runtime/root-ca.pem chmod 0600 /runtime/node-key.pem ls -l /runtime volumeMounts: - name: node-0-source mountPath: /source/node-0 readOnly: true - name: node-1-source mountPath: /source/node-1 readOnly: true - name: node-2-source mountPath: /source/node-2 readOnly: true - name: runtime-certs mountPath: /runtime ``` > [!IMPORTANT] > The `chown 1000:1000` matches the documented chart runtime user. Verify the actual image UID/GID before copying this into a different image or chart. ```bash kubectl exec -n "$NS" "${OS_STS}-0" -c opensearch -- id ``` The init container and main container mount the same `emptyDir` at different paths: ```text init container: /runtime/node.pem main container: /usr/share/opensearch/config/certs/node.pem ``` These are the same underlying file. ## 8.5 OpenSearch TLS configuration Use fixed runtime paths: ```yaml config: opensearch.yml: | cluster.name: brilliant-opensearch-cluster network.host: 0.0.0.0 plugins.security.ssl.transport.pemcert_filepath: certs/node.pem plugins.security.ssl.transport.pemkey_filepath: certs/node-key.pem plugins.security.ssl.transport.pemtrustedcas_filepath: certs/root-ca.pem plugins.security.ssl.transport.enforce_hostname_verification: true plugins.security.ssl.http.enabled: true plugins.security.ssl.http.pemcert_filepath: certs/node.pem plugins.security.ssl.http.pemkey_filepath: certs/node-key.pem plugins.security.ssl.http.pemtrustedcas_filepath: certs/root-ca.pem plugins.security.nodes_dn: - 'CN=opensearch-master-0,OU=OpenSearch Nodes,O=Brilliant,L=Dhaka,ST=Dhaka,C=BD' - 'CN=opensearch-master-1,OU=OpenSearch Nodes,O=Brilliant,L=Dhaka,ST=Dhaka,C=BD' - 'CN=opensearch-master-2,OU=OpenSearch Nodes,O=Brilliant,L=Dhaka,ST=Dhaka,C=BD' plugins.security.authcz.admin_dn: - 'CN=opensearch-admin,OU=OpenSearch Administrators,O=Brilliant,L=Dhaka,ST=Dhaka,C=BD' ``` Use the **actual RFC2253 certificate DNs** from `openssl x509`, not guessed strings. ## 8.6 Temporary transport hostname-verification workaround If logs show: ```text No subject alternative DNS name matching brilliant-opensearch-cluster-master-headless found ``` then the certificate SAN is wrong for the DNS name used by transport communication. Temporary diagnostic setting: ```yaml plugins.security.ssl.transport.enforce_hostname_verification: false ``` This is not the desired production state. Regenerate node certificates with correct SANs, deploy them, verify transport TLS, then restore: ```yaml plugins.security.ssl.transport.enforce_hostname_verification: true ``` --- # 9. Render, Validate, and Deploy OpenSearch Use this order every time: ```text values.yaml -> helm lint -> helm template -> inspect rendered YAML -> Kubernetes server-side dry-run -> helm upgrade/install -> runtime validation ``` ## 9.1 Helm lint ```bash helm lint "$OS_CHART" -f values.yaml ``` ## 9.2 Render ```bash helm template "$OS_RELEASE" "$OS_CHART" \ -n "$NS" \ -f values.yaml \ > rendered-opensearch.yaml ``` ## 9.3 Inspect rendered manifest ```bash grep -nE 'kind: StatefulSet|replicas:|storageClassName:|120Gi|runtime-certs|node-[012]-source|select-node-certificate|nodes_dn|admin_dn|9200|9300' rendered-opensearch.yaml ``` Verify: ```text [ ] replicas = 3 [ ] image/version pinned [ ] requests/limits present [ ] 120Gi Longhorn PVC template [ ] hard anti-affinity [ ] expected toleration / affinity [ ] runtime-certs emptyDir [ ] init container selects certificate by ordinal [ ] main container mounts runtime-certs only [ ] node DNs and admin DN correct [ ] REST port 9200 [ ] transport port 9300 [ ] bootstrap Secret reference present ``` ## 9.4 Kubernetes API dry-run ```bash kubectl apply --dry-run=server -f rendered-opensearch.yaml ``` ## 9.5 Deploy ```bash helm upgrade --install "$OS_RELEASE" "$OS_CHART" \ -n "$NS" \ -f values.yaml ``` ## 9.6 Observe rollout ```bash kubectl get pods -n "$NS" -w ``` In another terminal: ```bash kubectl rollout status statefulset/"$OS_STS" -n "$NS" ``` --- # 10. OpenSearch Security Initialization and Admin Password ## 10.1 Important distinction These are not the same thing: ```text Kubernetes Secret / OPENSEARCH_INITIAL_ADMIN_PASSWORD != active OpenSearch Security index ``` For new demo-security initialization, `OPENSEARCH_INITIAL_ADMIN_PASSWORD` is used during setup. Once the Security configuration is initialized, changing an environment variable or mounted YAML file does not automatically rewrite the active security index. ## 10.2 Inspect effective internal user configuration ```bash kubectl exec -n "$NS" "${OS_STS}-0" -c opensearch -- \ grep -A8 '^admin:' \ /usr/share/opensearch/config/opensearch-security/internal_users.yml ``` If the mounted file still contains a demo hash, it does not prove the intended password is active. ## 10.3 Generate a bcrypt hash ```bash kubectl exec -it -n "$NS" "${OS_STS}-0" -c opensearch -- \ /usr/share/opensearch/plugins/opensearch-security/tools/hash.sh \ -p '' ``` Avoid placing real passwords in shell history. Prefer an interactive/secure workflow where possible. ## 10.4 Create custom `internal_users.yml` ```yaml --- _meta: type: internalusers config_version: 2 admin: hash: "$2y$12$" reserved: true backend_roles: - admin description: OpenSearch administrator ``` ## 10.5 Critical Secret key name The chart expects the exact key: ```text internal_users.yml ``` Not: ```text internal_users.yaml ``` Create/update correctly even if your local filename is `.yaml`: ```bash kubectl create secret generic opensearch-internal-users \ -n "$NS" \ --from-file=internal_users.yml=./internal_users.yaml \ --dry-run=client -o yaml | kubectl apply -f - ``` Verify only the key names: ```bash kubectl get secret opensearch-internal-users \ -n "$NS" \ -o json | jq -r '.data | keys[]' ``` Expected: ```text internal_users.yml ``` ## 10.6 Helm security configuration ```yaml securityConfig: enabled: true internalUsersSecret: opensearch-internal-users ``` Do not duplicate `securityConfig:` blocks. ## 10.7 Admin certificate mount The admin certificate should be separate from normal node runtime TLS material. ```yaml extraVolumes: # existing volumes ... - name: admin-certs secret: secretName: opensearch-admin-cert defaultMode: 0440 extraVolumeMounts: # existing mounts ... - name: admin-certs mountPath: /usr/share/opensearch/config/admin-certs readOnly: true ``` With the chart's `fsGroup: 1000`, mode `0440` allows the non-root OpenSearch process to read these files through the pod group. Verify the effective UID/GID before changing this mode. A stricter production design is to mount the admin key only into a temporary administrative Job/Pod rather than every OpenSearch runtime pod. ## 10.8 Apply only internal users with `securityadmin.sh` After the cluster is healthy and the admin certificate is trusted: ```bash kubectl exec -it -n "$NS" "${OS_STS}-0" -c opensearch -- \ /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh \ -f /usr/share/opensearch/config/opensearch-security/internal_users.yml \ -t internalusers \ -icl \ -nhnv \ -cacert /usr/share/opensearch/config/admin-certs/root-ca.pem \ -cert /usr/share/opensearch/config/admin-certs/admin.pem \ -key /usr/share/opensearch/config/admin-certs/admin-key.pem ``` Using `-f` + `-t internalusers` is safer than unintentionally overwriting the complete Security configuration. --- # 11. OpenSearch Cluster Verification ## 11.1 Kubernetes state ```bash kubectl get sts -n "$NS" kubectl get pods -n "$NS" -o wide kubectl get pvc -n "$NS" kubectl get svc -n "$NS" ``` Expected: ```text 3 OpenSearch pods 3 Bound PVCs 1 PVC per pod pods spread across intended nodes ClusterIP Service present headless Service present ``` ## 11.2 Init container logs ```bash kubectl logs -n "$NS" "${OS_STS}-0" -c select-node-certificate kubectl logs -n "$NS" "${OS_STS}-1" -c select-node-certificate kubectl logs -n "$NS" "${OS_STS}-2" -c select-node-certificate ``` ## 11.3 Runtime certificate files ```bash kubectl exec -n "$NS" "${OS_STS}-0" -c opensearch -- \ ls -l /usr/share/opensearch/config/certs ``` Expected: ```text node.pem node-key.pem root-ca.pem ``` ## 11.4 OpenSearch logs ```bash kubectl logs -n "$NS" "${OS_STS}-0" -c opensearch --tail=200 ``` If a container restarted: ```bash kubectl logs -n "$NS" "${OS_STS}-0" -c opensearch --previous --tail=200 ``` Correct flag: ```text --previous ``` not: ```text --privious ``` ## 11.5 Local diagnostic port-forward ```bash kubectl port-forward -n "$NS" svc/"$OS_SVC" 9200:9200 ``` Because the certificate may not contain `localhost`, `curl -k` is acceptable only as a **local diagnostic**: ```bash curl -k -u 'admin:' \ 'https://localhost:9200/_cluster/health?pretty' curl -k -u 'admin:' \ 'https://localhost:9200/_cat/nodes?v' ``` Do not interpret `-k` as production TLS validation. ## 11.6 Cluster health meaning ```text GREEN -> all primary and replica shards allocated YELLOW -> all primaries allocated; at least one replica unassigned RED -> at least one primary shard unavailable ``` ## 11.7 Replica count OpenSearch defaults to one replica unless changed. A 3-node cluster **can** use two replicas to keep three shard copies, but this is a capacity/durability choice, not an automatic requirement. Before setting `number_of_replicas: 2`, account for storage, indexing cost, and failure-domain design. --- # 12. OpenSearch Dashboards Deployment ## 12.1 Architecture ```text Browser | | HTTP :31599 /opensearch v NGINX Ingress | v OpenSearch Dashboards Service :5601 | v Dashboards pod | | HTTPS :9200 + CA + hostname validation v OpenSearch Service ``` ## 12.2 Verify OpenSearch Service certificate SAN The hostname used by Dashboards must exist in the OpenSearch certificate SAN: ```text brilliant-opensearch-cluster-master ``` A failure against `https://localhost:9200` such as: ```text no alternative certificate subject name matches target hostname 'localhost' ``` means CA trust may be correct but hostname verification failed. ## 12.3 Root CA Secret for Dashboards ```bash kubectl create secret generic opensearch-dashboard-ca \ -n "$NS" \ --from-file=root-ca.pem \ --dry-run=client -o yaml | kubectl apply -f - ``` ## 12.4 `kibanaserver` password Secret ```bash read -rsp 'kibanaserver password: ' KIBANASERVER_PASSWORD echo kubectl create secret generic opensearch-kibanaserver-password \ -n "$NS" \ --from-literal=KIBANASERVER_PASSWORD="$KIBANASERVER_PASSWORD" \ --dry-run=client -o yaml | kubectl apply -f - unset KIBANASERVER_PASSWORD ``` The Secret object name and the Secret key are different concepts: ```text secretKeyRef.name -> opensearch-kibanaserver-password secretKeyRef.key -> KIBANASERVER_PASSWORD ``` ## 12.5 Dashboards Helm values ```yaml opensearchHosts: "https://brilliant-opensearch-cluster-master:9200" extraEnvs: - name: KIBANASERVER_PASSWORD valueFrom: secretKeyRef: name: opensearch-kibanaserver-password key: KIBANASERVER_PASSWORD extraVolumes: - name: root-cert secret: secretName: opensearch-dashboard-ca extraVolumeMounts: - name: root-cert readOnly: true mountPath: /usr/share/opensearch-dashboards/certs config: opensearch_dashboards.yml: | server: name: dashboards host: "0.0.0.0" basePath: /opensearch rewriteBasePath: true opensearch: username: kibanaserver password: ${KIBANASERVER_PASSWORD} ssl: verificationMode: full certificateAuthorities: - /usr/share/opensearch-dashboards/certs/root-ca.pem opensearch_security.multitenancy.enabled: false data_source: enabled: true workspace: enabled: true ``` Optional feature keys such as `explore` and `datasetManagement` must be checked against the exact OpenSearch Dashboards version before enabling them. ## 12.6 Workspaces and multi-tenancy When using Workspaces together with the OpenSearch Security plugin: ```yaml opensearch_security.multitenancy.enabled: false ``` is required to avoid feature conflict. ## 12.7 `server.publicBaseUrl` In the tested OpenSearch Dashboards 3.7.0 deployment, adding `server.publicBaseUrl` caused startup validation failure: ```text "publicBaseUrl" is not allowed ``` Therefore it is intentionally absent from this SOP. ## 12.8 Render Dashboards before deploy ```bash helm template brilliant-opensearch-dashboard \ -n "$NS" \ -f dashboards-values.yaml \ > rendered-dashboards.yaml ``` Inspect: ```bash grep -nE 'KIBANASERVER_PASSWORD|root-cert|basePath|rewriteBasePath|verificationMode|5601' rendered-dashboards.yaml ``` ## 12.9 Ingress conflict check Before creating a hostless path, inspect existing Ingress objects: ```bash kubectl get ingress -A ``` Grafana already used hostless `/` in the implementation. A second hostless `/` route under the same NGINX controller conflicted even though it was in another namespace. Use `/opensearch` instead. ## 12.10 Dashboards Ingress ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: opensearch-dashboards-ingress namespace: observability spec: ingressClassName: nginx rules: - http: paths: - path: /opensearch pathType: Prefix backend: service: name: brilliant-opensearch-dashboard-opensearch-dashboards port: number: 5601 ``` Apply: ```bash kubectl apply -f opensearch-dashboards-ingress.yaml ``` ## 12.11 Runtime verification ```bash kubectl get deploy,pods,svc,ingress -n "$NS" ``` Current logs: ```bash kubectl logs -n "$NS" deploy/brilliant-opensearch-dashboard-opensearch-dashboards --tail=200 ``` Previous container logs after a restart: ```bash kubectl logs -n "$NS" deploy/brilliant-opensearch-dashboard-opensearch-dashboards --previous --tail=200 ``` Expected external URL pattern: ```text http://:31599/opensearch ``` ## 12.12 `401 -> 404` diagnostic meaning During troubleshooting: ```text unauthenticated /opensearch -> 401 Authentication required valid authentication -> 404 Not Found ``` This indicated traffic was reaching Dashboards and authentication worked, while Dashboards did not yet recognize `/opensearch` as its base path. `server.basePath` + `rewriteBasePath` resolved that routing layer. --- # 13. Data Prepper — Full Deployment, Routing, Parsing, and Verification This chapter is the canonical Data Prepper procedure for this environment. It consolidates the Data Prepper material from the cPanel ingestion, multi-source routing, and Linux Audit SOPs. ## 13.1 Role in the architecture ```text Fluent Bit | | HTTP POST /logs v NGINX Ingress / NodePort | v Data Prepper HTTP source :2021 | v Router pipeline | +--> cPanel access pipeline +--> system syslog pipeline +--> Linux audit pipeline +--> catch-all / unknown pipeline | v OpenSearch HTTPS :9200 ``` Data Prepper is responsible for receiving events, routing them by `log_source`, parsing/normalizing selected formats, and writing documents to the correct OpenSearch index. ## 13.2 Environment values used in this implementation | Setting | Value used | |---|---| | Data Prepper version | `2.8.0` | | Namespace | `observability` | | Helm release | `data-prepper` | | HTTP source port | `2021` | | Server API / metrics port | `4900` | | HTTP path | `/logs` | | OpenSearch endpoint | `https://brilliant-opensearch-cluster-master:9200` | | Root CA path inside Data Prepper | `/usr/share/data-prepper/cert/root-ca.pem` | | cPanel access index | `cpanel-access-log-%{yyyy.MM.dd}` | | Syslog index | `cpanel-system-syslog-%{yyyy.MM.dd}` | | Audit index | `cpanel-audit-log-%{yyyy.MM.dd}` | | Catch-all index | `cpanel-all-log-%{yyyy.MM.dd}` | Verify the actual release and namespace before modifying anything: ```bash helm list -n "$NS" kubectl get deploy,svc,pod -n "$NS" | grep -i data-prepper ``` Expected Helm release: ```text data-prepper ``` > [!WARNING] > One troubleshooting incident installed/upgraded a second `data-prepper` release in the `default` namespace. Always specify `-n observability` / `--namespace observability`. ## 13.3 Back up the current Data Prepper configuration before changes ```bash mkdir -p ~/backup/data-prepper helm get values data-prepper -n "$NS" -a \ > ~/backup/data-prepper/values-$(date +%F-%H%M%S).yaml helm get manifest data-prepper -n "$NS" \ > ~/backup/data-prepper/manifest-$(date +%F-%H%M%S).yaml ``` Also export the currently running pipeline/config objects if they are stored in ConfigMaps or Secrets: ```bash kubectl get configmap -n "$NS" -o name | grep -i prepper kubectl get secret -n "$NS" -o name | grep -i prepper ``` Do not dump Secret data into Git, screenshots, or terminal transcripts. ## 13.4 One HTTP source per listener port Only one pipeline in a Data Prepper process should own HTTP port `2021`. Incorrect design: ```text access pipeline -> HTTP :2021 syslog pipeline -> HTTP :2021 audit pipeline -> HTTP :2021 ``` Typical failure: ```text bind(..) failed: Address already in use ``` Correct design: ```text ONE HTTP source :2021 | v router pipeline | +--> access pipeline +--> syslog pipeline +--> audit pipeline +--> catch-all pipeline ``` Downstream pipelines use a Data Prepper `pipeline` source, not another HTTP source. ## 13.5 Complete router pipeline The normalized `log_source` field created by Fluent Bit is the routing key. ```yaml version: "2" cpanel-log-router: source: http: path: /logs ssl: false port: 2021 health_check_service: true unauthenticated_health_check: true route: - accessLog: '/log_source == "cpanel.access_log"' - systemSyslog: '/log_source == "system.syslog"' - auditLog: '/log_source == "audit.audit.log"' - unhandled: '/log_source != "cpanel.access_log" and /log_source != "system.syslog" and /log_source != "audit.audit.log"' sink: - pipeline: name: cpanel-access-pipeline routes: - accessLog - pipeline: name: cpanel-system-syslog-pipeline routes: - systemSyslog - pipeline: name: cpanel-audit-pipeline routes: - auditLog - pipeline: name: cpanel-catch-all-pipeline routes: - unhandled ``` ### Why the `unhandled` route is explicit A pipeline sink without a `routes:` restriction can receive all events. If a catch-all sink is left unqualified, dedicated events may also be copied into the catch-all path. The explicit `unhandled` condition prevents that duplication. ## 13.6 Complete cPanel access pipeline The tested raw access format is: ```text 20.100.201.54 proxy - [08/09/2026:07:37:27 -0000] "GET /wp-content/themes/twentytwenty/404.php HTTP/1.1" 301 0 "-" "-" "-" "X-Forwarded-For: 20.100.201.54" 80 ``` Pipeline: ```yaml cpanel-access-pipeline: workers: 2 delay: 100 source: pipeline: name: cpanel-log-router processor: - drop_events: drop_when: '/log_source != "cpanel.access_log"' - grok: break_on_match: false match: log: - '^%{IP:client_ip} %{NOTSPACE:identity} %{NOTSPACE:user} \[%{GREEDYDATA:timestamp}\] "%{WORD:method} %{URIPATHPARAM:url} HTTP/%{NUMBER:http_version}" %{INT:http_status:int} %{NONNEGINT:bytes:long} "%{NOTSPACE:referrer}" "%{GREEDYDATA:user_agent}" "%{NOTSPACE:auth_method}" "%{GREEDYDATA:x_forwarded_for}" %{POSINT:service_port:int}$' target_key: parsed - date: match: - key: /parsed/timestamp patterns: - "MM/dd/yyyy:HH:mm:ss Z" destination: "@timestamp" sink: - opensearch: hosts: - "https://brilliant-opensearch-cluster-master:9200" insecure: false username: logger password: "" cert: /usr/share/data-prepper/cert/root-ca.pem index_type: custom index: cpanel-access-log-%{yyyy.MM.dd} ``` Expected parsed fields: ```text parsed.client_ip parsed.identity parsed.user parsed.timestamp parsed.method parsed.url parsed.http_version parsed.http_status parsed.bytes parsed.referrer parsed.user_agent parsed.auth_method parsed.x_forwarded_for parsed.service_port @timestamp ``` ### Important Grok compatibility issue from the implementation The tested Data Prepper 2.8.0 setup failed when this was used: ```text %{PORT:service_port} ``` with: ```text No definition for key 'PORT' found, aborting ``` The working replacement was: ```text %{POSINT:service_port} ``` ## 13.7 Syslog pipeline Syslog records observed in the environment included both: ```text 2026-08-11T10:42:39.679625+00:00 bwebhosting kernel: Firewall: ... ``` and: ```text 2026-08-11T10:42:39.434068+00:00 bwebhosting fluent-bit[1241036]: ... ``` The process token therefore needs to tolerate both a plain process name and a process name with an optional PID. A safe baseline is to preserve raw data while the exact parser is being tuned: ```yaml cpanel-system-syslog-pipeline: source: pipeline: name: cpanel-log-router processor: - drop_events: drop_when: '/log_source != "system.syslog"' sink: - opensearch: hosts: - "https://brilliant-opensearch-cluster-master:9200" insecure: false username: logger password: "" cert: /usr/share/data-prepper/cert/root-ca.pem index_type: custom index: cpanel-system-syslog-%{yyyy.MM.dd} ``` Add Grok only after representative samples are verified. If a document reaches OpenSearch with `parsed: {}`, transport/routing may be healthy while the Grok expression is not matching. ## 13.8 Linux Audit pipeline Audit records are multi-type and are handled in the dedicated Linux Audit chapter of this SOP. The router must send only: ```text log_source = audit.audit.log ``` to: ```text cpanel-audit-pipeline ``` The audit pipeline should additionally protect itself with: ```yaml - drop_events: drop_when: '/log_source != "audit.audit.log"' ``` The full Audit Grok, date, `EXECVE`, `SYSCALL`, `LOGIN`, `USER_ACCT`, `USER_START`, `CRED_ACQ`, and `CWD` parsing procedure is documented in Section 17. ## 13.9 Catch-all pipeline Use the catch-all to retain unknown/new sources without forcing every source into a dedicated parser on day one. ```yaml cpanel-catch-all-pipeline: source: pipeline: name: cpanel-log-router sink: - opensearch: hosts: - "https://brilliant-opensearch-cluster-master:9200" insecure: false username: logger password: "" cert: /usr/share/data-prepper/cert/root-ca.pem index_type: custom index: cpanel-all-log-%{yyyy.MM.dd} ``` Recommended model: ```text Known/high-value source -> dedicated route -> dedicated parser -> dedicated index Unknown/new source -> catch-all -> raw visibility -> inspect -> promote later if required ``` ## 13.10 OpenSearch Root CA mount Data Prepper must trust the CA that signed the OpenSearch HTTP certificate. Conceptual Kubernetes volume: ```yaml volumes: - name: opensearch-root-ca secret: secretName: opensearch-dashboard-ca ``` Conceptual container mount: ```yaml volumeMounts: - name: opensearch-root-ca mountPath: /usr/share/data-prepper/cert readOnly: true ``` Verify inside the pod: ```bash kubectl exec -n "$NS" deploy/data-prepper -- \ ls -l /usr/share/data-prepper/cert/root-ca.pem ``` Do not set `insecure: true` merely to bypass a certificate problem. Correct the CA/SAN/TLS configuration instead. ## 13.11 Data Prepper credential handling The source SOP used an OpenSearch user named: ```text logger ``` The real password must not be committed to Git. Required security model: ```text Data Prepper identity -> authenticate to OpenSearch -> only required write/create-index permissions -> no admin/security-management permissions ``` If the Helm chart or deployment mechanism stores `pipelines.yaml` in a Secret, keep the sensitive pipeline configuration in that Secret or use the secret-management mechanism supported by the deployment. Do not assume arbitrary environment-variable interpolation in every Data Prepper configuration field without testing the exact deployed version. ## 13.12 Kubernetes Service Expected Service path: ```text Ingress -> data-prepper Service :2021 -> Data Prepper Pod :2021 ``` Service example: ```yaml apiVersion: v1 kind: Service metadata: name: data-prepper namespace: observability spec: type: ClusterIP selector: app: data-prepper ports: - name: ingest port: 2021 targetPort: 2021 - name: metrics port: 4900 targetPort: 4900 ``` The selector is chart/deployment-specific. Verify that it matches the actual pod labels: ```bash kubectl get pod -n "$NS" --show-labels | grep data-prepper kubectl get svc data-prepper -n "$NS" -o yaml ``` Verify the backend endpoint: ```bash kubectl get endpointslice \ -n "$NS" \ -l kubernetes.io/service-name=data-prepper \ -o wide ``` Expected result: the current Data Prepper pod IP and port `2021` are present. ## 13.13 NGINX Ingress The implementation used: ```text http://118.67.213.76:31599/logs ``` Flow: ```text 118.67.213.76:31599 -> ingress-nginx controller :80 -> prepper-ingress /logs -> data-prepper:2021 ``` Ingress: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: prepper-ingress namespace: observability spec: ingressClassName: nginx rules: - http: paths: - path: /logs pathType: Prefix backend: service: name: data-prepper port: number: 2021 ``` Apply and verify: ```bash kubectl apply -f prepper-ingress.yaml kubectl get ingress prepper-ingress -n "$NS" -o wide kubectl describe ingress prepper-ingress -n "$NS" ``` ## 13.14 Data Prepper Helm upgrade procedure The exact chart path depends on how Data Prepper was installed. Use the real chart reference/path instead of guessing. ```bash export DP_RELEASE="data-prepper" export DP_CHART="" export DP_VALUES="" ``` Confirm the current release first: ```bash helm list -n "$NS" ``` Render before upgrading: ```bash helm template "$DP_RELEASE" "$DP_CHART" \ -n "$NS" \ -f "$DP_VALUES" \ > /tmp/data-prepper-rendered.yaml ``` Inspect the important items: ```bash grep -nE 'kind: (Deployment|Service|ConfigMap|Secret)|2021|4900|pipelines|data-prepper' \ /tmp/data-prepper-rendered.yaml | head -200 ``` Then upgrade the correct release in the correct namespace: ```bash helm upgrade "$DP_RELEASE" "$DP_CHART" \ -n "$NS" \ -f "$DP_VALUES" ``` Watch rollout: ```bash kubectl get pods -n "$NS" -w ``` ## 13.15 Pod and listener verification ```bash kubectl get pods -n "$NS" | grep data-prepper kubectl logs -n "$NS" deploy/data-prepper --tail=200 ``` Expected health: ```text READY: 1/1 STATUS: Running ``` If `ss`/`netstat` do not exist in the image, `/proc/net/tcp` can verify the listener. For port `2021`: ```text 2021 decimal = 07E5 hexadecimal 0A = LISTEN 01 = ESTABLISHED ``` Example: ```bash kubectl exec -n "$NS" deploy/data-prepper -- \ sh -c 'grep -i ":07E5" /proc/net/tcp /proc/net/tcp6 || true' ``` ## 13.16 Health endpoint and metrics When the HTTP source health service is enabled: ```bash kubectl run dp-curl --rm -i --restart=Never \ -n "$NS" \ --image=curlimages/curl:latest \ -- \ curl -sS http://data-prepper:2021/health ``` Data Prepper server/metrics API: ```bash kubectl exec -n "$NS" deploy/data-prepper -- \ curl -s http://127.0.0.1:4900/metrics/prometheus | head -100 ``` Useful metric families from the audit/cPanel work include: ```text grokProcessingMatch grokProcessingMismatch grokProcessingErrors dateProcessingMatchSuccess dateProcessingMatchFailure documentsSuccess documentErrors ``` Pipeline/plugin names can be normalized into the full metric name. ## 13.17 Controlled event test inside Kubernetes The HTTP source accepts JSON events. Use a controlled event before involving Fluent Bit: ```bash kubectl run dp-curl --rm -i --restart=Never \ -n "$NS" \ --image=curlimages/curl:latest \ -- \ curl -sS -X POST http://data-prepper:2021/logs \ -H 'Content-Type: application/json' \ --data '[{"log":"test","source_file":"manual-test","log_source":"test.manual"}]' ``` Then verify that the catch-all path receives it. ## 13.18 Controlled event test through NGINX From a host that can reach the NGINX NodePort: ```bash curl -i -X POST \ 'http://118.67.213.76:31599/logs' \ -H 'Content-Type: application/json' \ --data '[{"log":"ingress-test","source_file":"manual-test","log_source":"test.manual"}]' ``` A successful test proves: ```text client -> NodePort -> NGINX -> Ingress rule -> Service -> EndpointSlice -> Data Prepper :2021 ``` ## 13.19 OpenSearch sink verification Do not use index existence alone as proof of successful ingestion. Example: ```text green open cpanel-access-log-2026.08.09 ... docs.count=0 ``` means the index exists but has no documents. Check index list: ```bash curl --cacert -u ':' \ 'https://:9200/_cat/indices/cpanel-*?v' ``` Check document count: ```bash curl --cacert -u ':' \ 'https://:9200/cpanel-access-log-*/_count?pretty' ``` Check parsed fields: ```bash curl --cacert -u ':' \ -H 'Content-Type: application/json' \ 'https://:9200/cpanel-access-log-*/_search?pretty' \ -d '{ "size": 5, "sort": [{"@timestamp": "desc"}], "_source": ["@timestamp","log_source","source_file","log","parsed"] }' ``` Interpretation: ```text log exists + parsed exists -> parser matched log exists + parsed missing -> parser did not match / processor issue index exists + docs.count = 0 -> no successful document insertion ``` ## 13.20 `502 Bad Gateway` troubleshooting Fluent Bit previously reported: ```text HTTP status=502 502 Bad Gateway nginx ``` This means Fluent Bit reached NGINX, but NGINX did not have a working upstream path to Data Prepper. Check in this order: ```bash kubectl get pod -n "$NS" | grep data-prepper kubectl get svc data-prepper -n "$NS" -o wide kubectl get endpointslice -n "$NS" \ -l kubernetes.io/service-name=data-prepper -o wide kubectl logs -n "$NS" deploy/data-prepper --tail=200 kubectl describe ingress prepper-ingress -n "$NS" ``` Do not change Fluent Bit Grok/parsing when NGINX is returning a backend/upstream error. ## 13.21 OpenSearch sink failure troubleshooting Observed sink errors included: ```text Failed to initialize OpenSearch sink with a retryable exception Request execution cancelled I/O session is closed ``` Troubleshoot this path independently: ```text Data Prepper -> Kubernetes DNS -> brilliant-opensearch-cluster-master -> TCP 9200 -> TLS handshake -> Root CA validation -> hostname/SAN validation -> logger authentication -> logger authorization -> index write ``` Do not modify Grok while the failure is DNS/TCP/TLS/authentication/authorization at the sink. ## 13.22 YAML and processor hierarchy checks Correct Grok hierarchy: ```yaml processor: - grok: break_on_match: false match: log: - '' target_key: parsed ``` A prior indentation mistake caused Data Prepper to interpret options such as `false` at the wrong level and fail pipeline construction. Validate indentation and quoting before every Helm upgrade. ## 13.23 Production hardening for Data Prepper The original implementation used: ```yaml ssl: false ``` on the Data Prepper HTTP source and exposed it through an NGINX NodePort. Before considering the ingestion endpoint production-hardened, use the controls appropriate to the environment: ```text TLS on the ingestion path HTTP authentication where supported/configured private network routing firewall/source-IP restrictions NetworkPolicy Ingress authentication / mTLS where appropriate least-privilege OpenSearch sink identity persistent DLQ if required resource limits metrics and alerting ``` The endpoint receives security-sensitive data including IP addresses, usernames, URLs, audit records, processes, and commands. Protect access and retention accordingly. --- # 14. Fluent Bit — Full cPanel/Linux Collection Procedure This chapter is the canonical Fluent Bit procedure for the cPanel/Linux sources in this project. It includes the access-only baseline, the expanded multi-source configuration, source classification, persistent offset state, filesystem buffering, verification, and outage-recovery testing. ## 14.1 Role in the architecture ```text cPanel/Linux log files | v Fluent Bit Tail inputs | +--> persistent Tail DB +--> filesystem buffer +--> source_file +--> Lua log_source classification | v HTTP POST /logs | v NGINX -> Data Prepper ``` Fluent Bit must: - Continuously tail the required log files. - Preserve the original path as `source_file`. - Maintain independent Tail DB state for each input. - Use filesystem buffering for undelivered chunks. - Normalize the source into a stable `log_source` value. - Send JSON to Data Prepper. - Retry transient failures. - Resume correctly after service/server restarts. ## 14.2 Source inventory used in the project | Category | Example path | Expected `log_source` | |---|---|---| | cPanel access | `/usr/local/cpanel/logs/access_log` | `cpanel.access_log` | | Syslog | `/var/log/syslog` | `system.syslog` | | Kernel | `/var/log/kern.log` | `system.kern.log` | | Exim | `/var/log/exim_mainlog` | `system.exim_mainlog` | | Auditd | `/var/log/audit/audit.log` | `audit.audit.log` | | Apache | `/var/log/apache2/...` | `apache.` | Before configuration, verify which files actually exist on the server: ```bash for f in \ /usr/local/cpanel/logs/access_log \ /var/log/syslog \ /var/log/kern.log \ /var/log/exim_mainlog \ /var/log/audit/audit.log; do [ -e "$f" ] && ls -lh "$f" done ``` Inspect representative records before writing parsers: ```bash tail -n 20 /usr/local/cpanel/logs/access_log sudo tail -n 20 /var/log/audit/audit.log ``` ## 14.3 Optional installation verification The source SOPs started from an already installed Fluent Bit service. If deploying on a new cPanel host, use the package procedure for the actual OS and pin/test the version according to your change policy. First identify the OS: ```bash cat /etc/os-release ``` After installation, the service must exist: ```bash systemctl status fluent-bit --no-pager fluent-bit --version ``` Do not replace a working production package with a newer one during an ingestion troubleshooting session unless the upgrade itself is planned and tested. ## 14.4 Service account and permissions Determine the service account: ```bash systemctl cat fluent-bit | grep -E '^(User|Group)=' || true ps -eo user,group,pid,cmd | grep '[f]luent-bit' ``` The Fluent Bit process needs read permission on every selected source file and write permission to the Tail DB/buffer directories. For audit logs, verify access explicitly: ```bash namei -l /var/log/audit/audit.log ``` Do not run Fluent Bit as root merely to bypass a permissions problem unless that is the deliberately approved security model. ## 14.5 Persistent state and buffer directories Recommended separation: ```text /var/lib/fluent-bit/ ├── state/ │ ├── cpanel-access.db │ ├── syslog.db │ ├── kern.db │ ├── exim.db │ └── audit.db └── storage/ └── filesystem chunks ``` Create the directories using the actual Fluent Bit service account: ```bash install -d -m 0750 -o fluent-bit -g fluent-bit /var/lib/fluent-bit/state install -d -m 0750 -o fluent-bit -g fluent-bit /var/lib/fluent-bit/storage ``` If the service uses another user/group, replace `fluent-bit:fluent-bit` accordingly. Keep these directories outside any broad Tail wildcard so Fluent Bit does not ingest its own SQLite/WAL or buffer files. ## 14.6 Access-log-only baseline configuration Use this first when proving the cPanel access path end to end: ```ini [SERVICE] Flush 1 Daemon Off Log_Level info Parsers_File parsers.conf Plugins_File plugins.conf storage.metrics on storage.path /var/lib/fluent-bit/storage storage.sync full storage.checksum on storage.backlog.mem_limit 500M [INPUT] Name tail Path /usr/local/cpanel/logs/access_log Path_Key source_file Tag access_log DB /var/lib/fluent-bit/state/cpanel-access.db DB.Journal_Mode wal Refresh_Interval 5 Skip_Long_Lines true Mem_Buf_Limit 512MB Storage.Type filesystem [FILTER] Name modify Match access_log Add log_source cpanel.access_log [OUTPUT] Name http Host 118.67.213.76 Port 31599 Format json HTTP_Method POST URI /logs Match access_log Retry_Limit no_limits Storage.Total_Limit_Size 4G ``` ### Notes `DB` stores file identity/offset state. It is not the durable event queue. `Storage.Type filesystem` allows chunks to be backed by the filesystem storage layer. `Storage.Total_Limit_Size 4G` caps the logical filesystem queue for this output. If the queue fills, old queued chunks can be discarded to make room for new data; monitor it. `Retry_Limit no_limits` avoids abandoning a failed chunk merely because a finite retry count was reached. This still does not make local disk capacity infinite. ## 14.7 Historical ingestion behavior For a file with no existing Tail DB state: ```ini Read_from_Head true ``` means read from the beginning. Without it, a new input normally begins at the end of the file and follows new records. Use historical backfill intentionally. Deleting/replacing a Tail DB and then setting `Read_from_Head true` can re-ingest already indexed data. For static backfill testing, use a copy of the source file and a dedicated DB/index whenever possible. ## 14.8 Multi-source Tail configuration After the access-only path is proven, expand to the required sources. Each input should have a unique DB. ```ini [SERVICE] Flush 1 Daemon Off Log_Level info Parsers_File parsers.conf Plugins_File plugins.conf storage.metrics on storage.path /var/lib/fluent-bit/storage storage.sync full storage.checksum on storage.backlog.mem_limit 500M [INPUT] Name tail Path /usr/local/cpanel/logs/access_log Path_Key source_file Tag cpanel.files DB /var/lib/fluent-bit/state/cpanel-access.db DB.Journal_Mode wal Refresh_Interval 5 Skip_Long_Lines true Mem_Buf_Limit 512MB Storage.Type filesystem [INPUT] Name tail Path /var/log/syslog Path_Key source_file Tag cpanel.files DB /var/lib/fluent-bit/state/syslog.db DB.Journal_Mode wal Refresh_Interval 5 Skip_Long_Lines true Mem_Buf_Limit 256MB Storage.Type filesystem [INPUT] Name tail Path /var/log/kern.log Path_Key source_file Tag cpanel.files DB /var/lib/fluent-bit/state/kern.db DB.Journal_Mode wal Refresh_Interval 5 Skip_Long_Lines true Mem_Buf_Limit 256MB Storage.Type filesystem [INPUT] Name tail Path /var/log/exim_mainlog Path_Key source_file Tag cpanel.files DB /var/lib/fluent-bit/state/exim.db DB.Journal_Mode wal Refresh_Interval 5 Skip_Long_Lines true Mem_Buf_Limit 256MB Storage.Type filesystem [INPUT] Name tail Path /var/log/audit/audit.log Path_Key source_file Tag cpanel.files DB /var/lib/fluent-bit/state/audit.db DB.Journal_Mode wal Refresh_Interval 5 Skip_Long_Lines true Mem_Buf_Limit 512MB Storage.Type filesystem [INPUT] Name tail Path /var/log/apache2/*.log Path_Key source_file Tag cpanel.files DB /var/lib/fluent-bit/state/apache.db DB.Journal_Mode wal Refresh_Interval 5 Skip_Long_Lines true Mem_Buf_Limit 512MB Storage.Type filesystem [FILTER] Name lua Match cpanel.files Script /etc/fluent-bit/classify.lua Call classify [OUTPUT] Name http Host 118.67.213.76 Port 31599 Format json HTTP_Method POST URI /logs Match cpanel.files Retry_Limit no_limits Storage.Total_Limit_Size 4G ``` Only keep input paths that actually exist and are required. If Apache/cPanel stores logs elsewhere on a specific host, use the real path. ## 14.9 `source_file` and `log_source` A Tail event should preserve its file: ```json { "log": "...", "source_file": "/var/log/syslog" } ``` The Lua filter converts that into a stable routing field: ```json { "log": "...", "source_file": "/var/log/syslog", "log_source": "system.syslog" } ``` Data Prepper routes on `log_source`, not on assumptions derived from the destination index name. ## 14.10 Lua source-classification script Create: ```text /etc/fluent-bit/classify.lua ``` Content: ```lua function classify(tag, timestamp, record) local path = record["source_file"] or "" if path == "/usr/local/cpanel/logs/access_log" then record["log_source"] = "cpanel.access_log" elseif path == "/var/log/syslog" then record["log_source"] = "system.syslog" elseif path == "/var/log/kern.log" then record["log_source"] = "system.kern.log" elseif path == "/var/log/exim_mainlog" then record["log_source"] = "system.exim_mainlog" elseif path == "/var/log/audit/audit.log" then record["log_source"] = "audit.audit.log" elseif string.match(path, "^/var/log/apache2/") then local filename = string.match(path, "([^/]+)$") or "unknown" record["log_source"] = "apache." .. filename else record["log_source"] = "unknown" end return 1, timestamp, record end ``` Set ownership/permissions appropriate to the service: ```bash chown root:root /etc/fluent-bit/classify.lua chmod 0644 /etc/fluent-bit/classify.lua ``` The earlier implementation attempted inline Lua in the classic configuration and Fluent Bit failed with: ```text extra indentation level found invalid indentation level ``` Keeping Lua in a separate file avoids that configuration-indentation problem and is easier to test/version. ## 14.11 Validate configuration before restart If supported by the installed Fluent Bit version/build: ```bash fluent-bit --dry-run -c /etc/fluent-bit/fluent-bit.conf ``` If the binary is not in `PATH`: ```bash command -v fluent-bit systemctl cat fluent-bit ``` Then restart: ```bash systemctl restart fluent-bit systemctl status fluent-bit --no-pager journalctl -u fluent-bit --since '5 minutes ago' --no-pager ``` Healthy state should not show sustained: ```text configuration parser errors 502 Bad Gateway connection refused failed to flush chunk retry exhaustion ``` ## 14.12 Prove the source is generating data For cPanel access logs, generate or wait for a request and confirm the file changes: ```bash tail -Fn0 /usr/local/cpanel/logs/access_log ``` For audit: ```bash sudo tail -Fn0 /var/log/audit/audit.log ``` If no fresh event exists in the source file, do not troubleshoot Data Prepper/OpenSearch yet. ## 14.13 Verify Tail DB creation and persistence ```bash ls -lh /var/lib/fluent-bit/state/ ``` If SQLite is installed and Fluent Bit is stopped or the DB can be safely inspected: ```bash sqlite3 /var/lib/fluent-bit/state/cpanel-access.db \ 'SELECT id,name,offset,inode FROM in_tail_files;' ``` The Tail database tells Fluent Bit where it last read each tracked file. ## 14.14 Verify filesystem buffering ```bash find /var/lib/fluent-bit/storage -maxdepth 2 -type f -ls | head -50 ``` Remember: ```text Tail DB -> protects file read position filesystem storage -> holds chunks that have been read but are not yet successfully delivered ``` They are separate mechanisms. ## 14.15 Verify HTTP output to Data Prepper Follow Fluent Bit logs: ```bash journalctl -u fluent-bit -f ``` The target used in this implementation is: ```text http://118.67.213.76:31599/logs ``` Test the receiver directly from the cPanel server: ```bash curl -i -X POST \ 'http://118.67.213.76:31599/logs' \ -H 'Content-Type: application/json' \ --data '[{"log":"manual-cpanel-host-test","source_file":"manual-test","log_source":"test.manual"}]' ``` If the manual curl fails, fix network/Ingress/Data Prepper before modifying Fluent Bit parsing or Tail settings. ## 14.16 Interpretation of common Fluent Bit output failures ### `502 Bad Gateway` ```text Fluent Bit -> NGINX = reachable NGINX -> Data Prepper upstream = unhealthy/unreachable ``` Check Data Prepper pod, Service, EndpointSlice, listener, and Ingress. ### `connection refused` The target IP/port is reachable at the network level but nothing is accepting the connection, or a firewall is rejecting it. ### `failed to flush chunk` Inspect the associated HTTP/network error. With filesystem buffering enabled, confirm the queue is growing and not at the storage limit. ### configuration parser error Use the exact reported file/line. Do not assume the systemd unit failure text is the root cause; Fluent Bit's own startup output contains the useful parser error. ## 14.17 Durable buffering recovery test Perform this only after normal ingestion is stable. 1. Record the current OpenSearch count. 2. Make Data Prepper unavailable intentionally. 3. Generate several test requests/log records. 4. Confirm Fluent Bit reports retries. 5. Confirm filesystem backlog activity in `/var/lib/fluent-bit/storage`. 6. Restart Fluent Bit while Data Prepper is still unavailable. 7. Confirm the Tail DB and storage directories survive the restart. 8. Restore Data Prepper. 9. Confirm queued events are forwarded. 10. Verify the expected documents appear in OpenSearch. Useful commands: ```bash systemctl restart fluent-bit journalctl -u fluent-bit --since '10 minutes ago' --no-pager find /var/lib/fluent-bit/storage -type f | head ``` This test validates both offset persistence and undelivered-event buffering. ## 14.18 Long-line behavior The configuration uses: ```ini Skip_Long_Lines true ``` This prevents a single oversized line from stopping monitoring of the entire file, but the oversized event can be skipped. If large audit/application records matter, test representative maximum line sizes and tune `Buffer_Chunk_Size` / `Buffer_Max_Size` rather than assuming the defaults are sufficient. ## 14.19 Log rotation Fluent Bit Tail tracks files using file metadata/offset state. Verify rotation behavior for each high-value log: ```bash ls -li /usr/local/cpanel/logs/access_log* ls -li /var/log/audit/audit.log* ``` Test that rotation does not cause either permanent loss or unexpected full-file replay. Do not delete Tail DB files as a routine rotation fix. ## 14.20 Multiple files and inotify limits If the deployment expands to a large number of Apache/cPanel files, inspect inotify capacity: ```bash sysctl fs.inotify.max_user_watches sysctl fs.inotify.max_user_instances ``` Increase them only if monitoring scale requires it and persist the chosen values through the normal OS configuration-management process. ## 14.21 Security hardening for Fluent Bit Use the minimum filesystem permissions required to read the configured logs and write its own state/buffer directories. Protect: ```text /etc/fluent-bit/fluent-bit.conf /etc/fluent-bit/classify.lua /var/lib/fluent-bit/state/ /var/lib/fluent-bit/storage/ ``` The original ingestion route used plain HTTP to the NGINX NodePort. For production, protect the transport with private routing, firewall restrictions, authentication and/or TLS as appropriate to the environment. Do not store OpenSearch credentials in Fluent Bit when Fluent Bit is intentionally sending only to Data Prepper. ## 14.22 Fluent Bit acceptance checklist - [ ] Required source files exist and receive new records. - [ ] Fluent Bit service is running. - [ ] No classic-config indentation/parser error remains. - [ ] Each Tail input has a unique persistent DB. - [ ] Tail DB directory is writable. - [ ] Filesystem buffer directory is writable. - [ ] `Storage.Type filesystem` is enabled for protected inputs. - [ ] `source_file` is present. - [ ] `log_source` matches the source file. - [ ] `/usr/local/cpanel/logs/access_log` becomes `cpanel.access_log`. - [ ] `/var/log/syslog` becomes `system.syslog`. - [ ] `/var/log/audit/audit.log` becomes `audit.audit.log`. - [ ] HTTP destination is the correct NGINX/Data Prepper endpoint. - [ ] URI is `/logs`. - [ ] Output format is JSON. - [ ] Fluent Bit has no sustained 502/refused/flush failures. - [ ] Restart resumes from stored offsets. - [ ] Data Prepper outage/recovery test has been completed. - [ ] Backlog disk usage is monitored. --- # 15. cPanel Access Log Parsing ## 15.1 Example raw log ```text 20.100.201.54 proxy - [08/09/2026:07:37:27 -0000] "GET /wp-content/themes/twentytwenty/404.php HTTP/1.1" 301 0 "-" "-" "-" "X-Forwarded-For: 20.100.201.54" 80 ``` ## 15.2 Recommended parser Use one timestamp capture and let the date processor create `@timestamp`. ```yaml cpanel-access-pipeline: source: pipeline: name: cpanel-log-router processor: - grok: break_on_match: true match: log: - '^%{IP:client_ip} %{NOTSPACE:identity} %{NOTSPACE:user} \[%{GREEDYDATA:timestamp}\] "%{WORD:method} %{URIPATHPARAM:url} HTTP/%{NUMBER:http_version}" %{INT:http_status:int} %{NONNEGINT:bytes:long} "%{NOTSPACE:referrer}" "%{GREEDYDATA:user_agent}" "%{NOTSPACE:auth_method}" "%{GREEDYDATA:x_forwarded_for}" %{POSINT:service_port:int}$' target_key: parsed - date: match: - key: /parsed/timestamp patterns: - "MM/dd/yyyy:HH:mm:ss Z" destination: "@timestamp" sink: - opensearch: hosts: - "https://brilliant-opensearch-cluster-master:9200" insecure: false username: logger password: "" cert: /usr/share/data-prepper/cert/root-ca.pem index_type: custom index: cpanel-access-log-%{yyyy.MM.dd} ``` ## 15.3 Parsed fields ```text parsed.client_ip parsed.identity parsed.user parsed.timestamp parsed.method parsed.url parsed.http_version parsed.http_status parsed.bytes parsed.referrer parsed.user_agent parsed.auth_method parsed.x_forwarded_for parsed.service_port @timestamp ``` ## 15.4 Important Grok issue: `%{PORT}` In the tested Data Prepper 2.8.0 environment: ```text %{PORT:service_port} ``` failed with: ```text No definition for key 'PORT' found ``` Use: ```text %{POSINT:service_port:int} ``` instead. ## 15.5 YAML hierarchy Correct: ```yaml processor: - grok: break_on_match: true match: log: - '' target_key: parsed ``` `break_on_match`, `match`, and `target_key` belong under `grok:`. ## 15.6 Parser validation test ```bash curl -sS -X POST \ 'http://:31599/logs' \ -H 'Content-Type: application/json' \ --data '[{ "log":"20.100.201.54 proxy - [08/09/2026:07:37:27 -0000] \"GET /wp-content/themes/twentytwenty/404.php HTTP/1.1\" 301 0 \"-\" \"-\" \"-\" \"X-Forwarded-For: 20.100.201.54\" 80", "source_file":"/usr/local/cpanel/logs/access_log", "log_source":"cpanel.access_log" }]' ``` Then inspect a newly indexed document. --- # 16. Linux Syslog and Other Log Sources ## 16.1 Example syslog records ```text 2026-08-11T10:42:39.679625+00:00 bwebhosting kernel: Firewall: ... ``` ```text 2026-08-11T10:42:39.434068+00:00 bwebhosting fluent-bit[1241036]: ... ``` A parser must tolerate both: ```text process: process[pid]: ``` Do not assume a PID is always present. ## 16.2 Recommended syslog approach Start with a small set of real samples from the actual host and build multiple Grok patterns if formats vary. If a document contains: ```json { "parsed": {} } ``` then transport and indexing may be working while the Grok pattern failed to match. ## 16.3 Do not infer source from index name If a document in `cpanel-system-syslog-*` contains: ```json { "log_source": "system.exim_mainlog" } ``` then routing/filtering is wrong even though the index name says `syslog`. Verify: ```text _index source_file log_source log parsed @timestamp ``` ## 16.4 Unknown/catch-all strategy Use: ```text cpanel-all-log-YYYY.MM.dd ``` for sources not yet assigned to a dedicated parser. This provides a safe discovery path: ```text unknown log appears -> stored raw -> inspect format -> create dedicated parser if operationally valuable ``` --- # 17. Linux Audit Log Parsing ## 17.1 Audit event model A single logical Audit event can create multiple records sharing one event ID: ```text SYSCALL EXECVE CWD PATH PROCTITLE ``` Therefore: ```text Total Audit documents != total commands executed ``` Use: ```text audit.event_id ``` for correlation. Use primarily: ```text audit.log_type = EXECVE ``` when counting command execution records. ## 17.2 Audit pipeline source ```yaml cpanel-audit-pipeline: source: pipeline: name: cpanel-log-router ``` A defensive filter may be retained: ```yaml processor: - drop_events: drop_when: '/log_source != "audit.audit.log"' ``` ## 17.3 Common Audit header Typical: ```text type=EXECVE msg=audit(1787117401.120:11280588): ... ``` Parser: ```yaml - grok: break_on_match: true match: log: - '^type=%{WORD:log_type} msg=audit\(%{INT:timestamp_seconds}\.%{INT:timestamp_fraction}:%{INT:event_id}\): %{GREEDYDATA:msg}' target_key: audit ``` Expected: ```json { "audit": { "log_type": "EXECVE", "timestamp_seconds": "1787117401", "timestamp_fraction": "120", "event_id": "11280588", "msg": "..." } } ``` ## 17.4 Audit timestamp ```yaml - date: match: - key: /audit/timestamp_seconds patterns: - epoch_second destination: "@timestamp" ``` ## 17.5 `SYSCALL` Parse only when: ```text /audit/log_type == "SYSCALL" ``` Common target fields: ```text syscall.syscall syscall.pid syscall.ppid syscall.success syscall.exit_code syscall.exe syscall.comm syscall.AUID syscall.UID syscall.terminal ``` Audit keys may be quoted or `(null)`, so maintain multiple patterns if necessary. ## 17.6 `EXECVE` ### Stage 1 — preserve argument count and raw argument text ```yaml - grok: break_on_match: true grok_when: '/audit/log_type == "EXECVE"' target_key: execve match: /audit/msg: - '^argc=%{INT:arg_count} %{GREEDYDATA:args_raw}$' ``` ### Stage 2 — optional key/value split ```yaml - key_value: key_value_when: '/audit/log_type == "EXECVE"' source: /execve/args_raw destination: /execve/args field_delimiter_regex: '\s+' value_split_characters: "=" delete_value_regex: '^"|"$' ``` Do **not** configure both `field_delimiter_regex` and a nonempty `field_split_characters`; these options are mutually exclusive. ### Known limitation Splitting on whitespace can break an argument that contains literal spaces inside quotes, for example: ```text a1="hello world" ``` Therefore always preserve: ```text execve.args_raw ``` and do not treat `execve.args.*` as a lossless reconstruction of every possible command line. Test representative Audit samples before relying on argument-level fields for security logic. ## 17.7 Mapping-growth limitation Dynamic fields such as: ```text execve.args.a0 execve.args.a1 execve.args.a2 ... ``` can create field-count growth. Prefer long-term normalized fields such as: ```text execve.command execve.arg_count execve.args_raw ``` and limit dynamic argument mappings. ## 17.8 `CWD` ```yaml - grok: break_on_match: true grok_when: '/audit/log_type == "CWD"' target_key: cwd match: /audit/msg: - '^cwd="%{GREEDYDATA:path}"' ``` ## 17.9 `USER_ACCT`, `USER_START`, `CRED_ACQ` For each type, keep the full parser chain consistent: ```text grok_when -> target_key -> key_value_when -> source -> destination ``` Example nested key/value processing: ```yaml - key_value: key_value_when: '/audit/log_type == "USER_ACCT"' source: /user_account/msg destination: /user_account/parsed_msg field_split_characters: " " value_split_characters: "=" delete_value_regex: '^"|"$' ``` Known limitation: simple space splitting can break values such as: ```text acct="John Doe" ``` Preserve the raw message. ## 17.10 `LOGIN` Some Audit LOGIN records contain the control separator `\u001d` before human-readable identity fields. Patterns should tolerate arbitrary content between result and `UID=` instead of assuming normal whitespace only. ## 17.11 Audit parser verification Count total EXECVE: ```bash curl -k -u 'USER:PASSWORD' \ -H 'Content-Type: application/json' \ 'https://OPENSEARCH:9200/cpanel-audit-log-YYYY.MM.DD/_count' \ -d '{ "query": { "match": { "audit.log_type": "EXECVE" } } }' ``` Count parsed EXECVE: ```bash curl -k -u 'USER:PASSWORD' \ -H 'Content-Type: application/json' \ 'https://OPENSEARCH:9200/cpanel-audit-log-YYYY.MM.DD/_count' \ -d '{ "query": { "bool": { "filter": [ {"match": {"audit.log_type": "EXECVE"}}, {"exists": {"field": "execve.arg_count"}} ] } } }' ``` Ideal validation target: ```text Total EXECVE == parsed EXECVE ``` Investigate mismatches with raw documents rather than only aggregate Grok metrics. --- # 18. OpenSearch Index Templates and Mapping ## 18.1 Index model ```text cpanel-access-log-* -> parsed cPanel HTTP access logs cpanel-system-syslog-* -> system syslog cpanel-audit-log-* -> Linux Audit records cpanel-all-log-* -> unknown/unhandled raw logs ``` ## 18.2 cPanel access mappings Critical types: ```text @timestamp -> date parsed.client_ip -> ip parsed.http_status -> integer parsed.bytes -> long parsed.service_port -> integer ``` Template example: ```json { "index_patterns": ["cpanel-access-log-*"], "template": { "mappings": { "properties": { "@timestamp": { "type": "date" }, "parsed": { "properties": { "client_ip": { "type": "ip" }, "http_status": { "type": "integer" }, "bytes": { "type": "long" }, "service_port": { "type": "integer" } } } } } } } ``` Apply with Dev Tools or API: ```http PUT _index_template/cpanel-access-log-template { "index_patterns": ["cpanel-access-log-*"], "template": { "mappings": { "properties": { "@timestamp": {"type": "date"}, "parsed": { "properties": { "client_ip": {"type": "ip"}, "http_status": {"type": "integer"}, "bytes": {"type": "long"}, "service_port": {"type": "integer"} } } } } } } ``` ## 18.3 Verify mapping ```http GET cpanel-access-log-*/_field_caps?fields=@timestamp,parsed.client_ip,parsed.http_status,parsed.bytes,parsed.service_port ``` ## 18.4 Audit default search fields A large Audit mapping can cause: ```text too_many_nested_clauses maxClauseCount is set to 1024 ``` Restrict default query fields: ```http PUT cpanel-audit-log-*/_settings { "index.query.default_field": [ "log", "audit.log_type", "audit.event_id", "syscall.exe", "syscall.comm", "syscall.AUID", "execve.args.a0" ] } ``` Add the same setting to the matching index template so future daily indexes inherit it. Do not immediately increase the Boolean clause limit. Control field growth and default-field expansion first. ## 18.5 Replica policy Set shard/replica counts in an index template deliberately. Do not assume Helm pod replicas automatically configure index replicas. --- # 19. End-to-End Verification Always test left-to-right. ## 19.1 Source file ```bash tail -n 5 /usr/local/cpanel/logs/access_log stat /usr/local/cpanel/logs/access_log ``` ## 19.2 Fluent Bit ```bash systemctl status fluent-bit --no-pager journalctl -u fluent-bit --since '10 minutes ago' --no-pager ``` Check Tail DB exists: ```bash ls -lh /var/lib/fluent-bit/state/ ``` Check buffer path: ```bash du -sh /var/lib/fluent-bit/storage/ ``` ## 19.3 Ingress From the source server: ```bash curl -i -X POST \ 'http://118.67.213.76:31599/logs' \ -H 'Content-Type: application/json' \ --data '[{"log":"manual ingress test","source_file":"manual","log_source":"test.manual"}]' ``` ## 19.4 Data Prepper pod ```bash kubectl get pods -n "$NS" -l app=data-prepper -o wide kubectl logs -n "$NS" deploy/data-prepper --tail=200 ``` ## 19.5 Service and endpoints ```bash kubectl get svc data-prepper -n "$NS" -o wide kubectl get endpointslice -n "$NS" \ -l kubernetes.io/service-name=data-prepper -o wide ``` ## 19.6 Port 2021 listener If the image lacks `ss` and `netstat`, inspect `/proc/net/tcp*`. ```text 2021 decimal = 07E5 hexadecimal TCP state 0A = LISTEN TCP state 01 = ESTABLISHED ``` Example: ```bash kubectl exec -n "$NS" deploy/data-prepper -- \ sh -c 'grep -i ":07E5" /proc/net/tcp /proc/net/tcp6 || true' ``` ## 19.7 OpenSearch indexes ```http GET _cat/indices/cpanel-*?v ``` Remember: ```text index exists + docs.count=0 ``` does not prove successful ingestion. ## 19.8 Recent documents ```http GET cpanel-access-log-*/_search { "size": 5, "sort": [ {"@timestamp": "desc"} ], "_source": [ "@timestamp", "log_source", "source_file", "log", "parsed" ] } ``` ## 19.9 Last 24 hours count ```http GET cpanel-access-log-*/_count { "query": { "range": { "@timestamp": { "gte": "now-24h", "lte": "now" } } } } ``` This proves: ```text index wildcard works + data exists + @timestamp is queryable + time-range query works ``` --- # 20. OpenSearch Dashboards Data Source and Workspace ## 20.1 Enable data sources ```yaml data_source: enabled: true ``` ## 20.2 Workspaces ```yaml workspace: enabled: true opensearch_security.multitenancy.enabled: false ``` ## 20.3 Data source encryption master key The source SOP configured a data-source credential encryption master key on OpenSearch nodes. Requirements: ```text same key on every OpenSearch node valid supported key length never commit the key to Git store using secure secret management ``` Before reusing the exact configuration key in a new OpenSearch version, verify the current OpenSearch documentation for the data-source encryption setting. ## 20.4 Create workspace Recommended workspace: ```text Cpanel ``` Inside the workspace: 1. Create an OpenSearch data source connection. 2. Use the cluster Service endpoint. 3. Use the required username/password authentication method. 4. Test the connection. 5. Create index pattern: ```text cpanel-access-log-* ``` 6. Select time field: ```text @timestamp ``` --- # 21. cPanel Access Dashboard Recommended dashboard: ```text cPanel Access Overview ``` Suggested layout: ```text +-------------------------------------------------------------------+ | Total Requests | Unique Clients | 2xx | 4xx | 5xx | +-------------------------------------------------------------------+ | Requests Over Time | +--------------------------------+----------------------------------+ | HTTP Status Distribution | Top Requested URLs | +--------------------------------+----------------------------------+ | Top Client IPs | HTTP Methods | +--------------------------------+----------------------------------+ | Top User Agents | +-------------------------------------------------------------------+ ``` ## 21.1 Total Requests ```text Visualization: Metric Aggregation: Count Label: Total Requests ``` ## 21.2 Unique Clients ```text Visualization: Metric Aggregation: Unique Count / Cardinality Field: parsed.client_ip Label: Unique Clients ``` ## 21.3 2xx DQL: ```text parsed.http_status >= 200 and parsed.http_status < 300 ``` ## 21.4 4xx ```text parsed.http_status >= 400 and parsed.http_status < 500 ``` ## 21.5 5xx ```text parsed.http_status >= 500 and parsed.http_status < 600 ``` Do not use chained comparison syntax such as: ```text 200 <= parsed.http_status < 300 ``` ## 21.6 Requests over time ```text Visualization: Area/Line Metric: Count X axis: Date Histogram Field: @timestamp Interval: Auto ``` ## 21.7 HTTP status distribution ```text Visualization: Pie/Donut Metric: Count Terms: parsed.http_status Size: 10 ``` ## 21.8 Top URLs ```text Visualization: Horizontal Bar Metric: Count Terms: parsed.url.keyword Fallback: parsed.url if aggregatable Size: 10 ``` ## 21.9 Top client IPs ```text Terms: parsed.client_ip Size: 10 ``` ## 21.10 HTTP methods ```text Terms: parsed.method.keyword Fallback: parsed.method if aggregatable ``` ## 21.11 Top user agents ```text Terms: parsed.user_agent.keyword Fallback: parsed.user_agent if aggregatable ``` Useful for scanners, bots, crawlers, and automated clients. --- # 22. Linux Audit Security Dashboard Recommended dashboard: ```text Linux Server Audit & Security Overview ``` Layout: ```text +-----------------------------------------------------------+ | Commands | Logins | Failed Syscalls | Users | Audit Docs | +-----------------------------------------------------------+ | Audit Activity Over Time | +---------------------------+-------------------------------+ | Top Commands | Top Executables | +---------------------------+-------------------------------+ | Top Users | Top Failed Syscalls | +---------------------------+-------------------------------+ | Authentication & Session Activity | +-----------------------------------------------------------+ | Recent Audit Activity | +-----------------------------------------------------------+ ``` ## 22.1 Commands executed DQL: ```text audit.log_type: "EXECVE" ``` Metric: Count. ## 22.2 Login events ```text audit.log_type: "LOGIN" ``` ## 22.3 Failed syscalls ```text audit.log_type: "SYSCALL" AND syscall.success: "no" ``` ## 22.4 Top failed syscall names ```text Terms: syscall.syscall.keyword ``` ## 22.5 Top commands ```text DQL: audit.log_type: "EXECVE" Terms: execve.args.a0.keyword ``` Use a normalized `execve.command` field instead if you implement the recommended schema improvement. ## 22.6 Top users ```text DQL: audit.log_type: "SYSCALL" Terms: syscall.AUID.keyword ``` ## 22.7 Authentication/session activity DQL: ```text audit.log_type: ("LOGIN" OR "USER_ACCT" OR "USER_START" OR "CRED_ACQ") ``` Use Date Histogram on `@timestamp` and split series by `audit.log_type.keyword`. ## 22.8 Recent activity Use separate tables initially because one logical Audit event spans multiple documents. Recent EXECVE fields: ```text @timestamp audit.event_id execve.args.a0 execve.args.a1 execve.args.a2 ``` Recent failed syscall fields: ```text @timestamp audit.event_id syscall.AUID syscall.exe syscall.syscall syscall.success syscall.exit_code ``` Future correlation should group by `audit.event_id`. --- # 23. Anomaly Detection, Forecasting, and Alerting ## 23.1 Request-volume anomaly detector Recommended detector: ```text Total-Request-Detector ``` Index: ```text cpanel-access-log-* ``` Timestamp: ```text @timestamp ``` Feature: ```text Request-Count -> Count ``` Starting operational settings from the implementation: ```text Interval: 10 minutes Frequency: 10 minutes Window delay: 1 minute History: 144 intervals (~24 hours) Shingle size: 8 Sparse data: Fill with Zeros ``` Do not split by high-cardinality client IP in the first detector. Validate: ```text actual request count expected range anomaly grade confidence historical timeline ``` ## 23.2 Forecasting Recommended forecaster: ```text Total-Request-Forecast ``` Starting values used: ```text Forecast interval: 3 minutes Window delay: 2 minutes Horizon: 24 intervals (~72 minutes) History: 537 intervals (~26h 51m) ``` These are starting parameters, not universal defaults. ## 23.3 Alert monitor Flow: ```text Anomaly Detector -> Monitor -> Trigger -> Alert -> Action -> Notification Channel ``` Starting trigger threshold: ```text Anomaly grade > 0.7 AND Confidence > 0.7 ``` Tune after reviewing false positives and real incidents. --- # 24. SMTP Notification Configuration Recorded sender configuration: ```text Sender name: cloud-operation SMTP host: zsmtp.brilliant.com.bd Port: 465 Method: SSL From address: cloud.operation@brilliant.com.bd ``` Do not put SMTP passwords into Helm values or Git. ## 24.1 Create temporary credential files ```bash printf '%s' 'cloud.operation@brilliant.com.bd' > /tmp/smtp-username read -rsp 'SMTP password: ' SMTP_PASSWORD echo printf '%s' "$SMTP_PASSWORD" > /tmp/smtp-password unset SMTP_PASSWORD ``` ## 24.2 Create Secret for OpenSearch keystore ```bash kubectl create secret generic opensearch-smtp-keystore \ -n "$NS" \ --from-file=opensearch.notifications.core.email.cloud-operation.username=/tmp/smtp-username \ --from-file=opensearch.notifications.core.email.cloud-operation.password=/tmp/smtp-password \ --dry-run=client -o yaml | kubectl apply -f - rm -f /tmp/smtp-username /tmp/smtp-password ``` The key path uses the **sender name** `cloud-operation`, not the sender email address. ## 24.3 Helm keystore value ```yaml keystore: - secretName: opensearch-smtp-keystore ``` ## 24.4 Upgrade ```bash helm upgrade "$OS_RELEASE" "$OS_CHART" \ -n "$NS" \ -f values.yaml ``` ## 24.5 Verify keystore ```bash kubectl exec -n "$NS" "${OS_STS}-0" -c opensearch -- \ /usr/share/opensearch/bin/opensearch-keystore list ``` Expected entries: ```text opensearch.notifications.core.email.cloud-operation.username opensearch.notifications.core.email.cloud-operation.password ``` Verify all OpenSearch nodes if the chart injects keystore material per pod. ## 24.6 `Invalid Addresses` troubleshooting Check: ```text From address is valid recipient is valid SMTP host/port are correct SSL/TLS mode matches the server credentials exist in keystore credentials are present on all relevant nodes SMTP relay permits sender SMTP account may use configured From address recipient is allowed ``` --- # 25. Reliability, Backup, and Recovery Tests ## 25.1 Fluent Bit buffering recovery After normal ingestion is stable: 1. Make Data Prepper temporarily unavailable. 2. Generate a few cPanel requests. 3. Confirm Fluent Bit retries. 4. Confirm filesystem backlog grows. 5. Restart Fluent Bit while Data Prepper is unavailable. 6. Restore Data Prepper. 7. Confirm buffered events drain. 8. Confirm documents reach OpenSearch. Monitor: ```bash du -sh /var/lib/fluent-bit/storage/ journalctl -u fluent-bit -f ``` ## 25.2 OpenSearch pod recreation Delete one pod: ```bash kubectl delete pod -n "$NS" "${OS_STS}-0" ``` Expected: ```text same StatefulSet ordinal recreated same PVC reused node rejoins cluster ``` Verify: ```bash kubectl get pod,pvc -n "$NS" -o wide ``` ## 25.3 Worker restart/failure test Expected: ```text remaining OpenSearch nodes continue serving replica shard can be promoted if primary is lost failed node/pod rejoins according to scheduling/storage policy ``` ## 25.4 Snapshots Shard replicas are not backups. Snapshot storage should be independent from the OpenSearch cluster and Longhorn volumes. Required operational practice: ```text configure repository schedule snapshots retain snapshots monitor failures perform restore tests ``` A backup is not considered valid until restore has been tested. --- # 26. Security Hardening ## 26.1 Credentials Do not commit: ```text OpenSearch admin password kibanaserver password Data Prepper logger password SMTP password data-source encryption key private keys Kubernetes Secret manifests containing real Base64 data ``` ## 26.2 Identity separation ```text admin -> security/cluster administration only kibanaserver -> Dashboards backend only logger -> Data Prepper ingestion only human users -> read/search/dashboard permissions as required ``` ## 26.3 Data Prepper ingress The original implementation used: ```yaml ssl: false ``` on the Data Prepper HTTP source and exposed it through NGINX NodePort. For production, protect the path with one or more of: ```text TLS HTTP basic authentication private routing/VPN source IP restriction firewall ACL Kubernetes NetworkPolicy authenticated ingress ``` The Data Prepper HTTP source supports TLS and HTTP basic authentication. ## 26.4 OpenSearch ports ```text 9300 -> never expose publicly 9200 -> normally private/internal 5601 -> expose through controlled reverse proxy/Ingress 2021 -> restrict to approved log senders ``` ## 26.5 Certificate management Monitor: ```text Not Before Not After SAN correctness key/cert pair CA trust rotation date ``` `root-ca-key.pem` must remain offline/encrypted. ## 26.6 NetworkPolicy If NetworkPolicy is used, explicitly permit: ```text Dashboards -> OpenSearch :9200 Data Prepper -> OpenSearch :9200 OpenSearch nodes <-> OpenSearch nodes :9300 Ingress -> Dashboards :5601 Ingress -> Data Prepper :2021 ``` --- # 27. Operational Monitoring ## 27.1 Kubernetes Monitor: ```text pod restarts OOMKilled CPU throttling memory RSS Pending pods PVC usage node pressure scheduling failures ``` Commands: ```bash kubectl get pods -n "$NS" -o wide kubectl top pods -n "$NS" kubectl get events -n "$NS" --sort-by=.lastTimestamp | tail -50 ``` ## 27.2 OpenSearch Monitor: ```text cluster health node count JVM heap pressure GC shard allocation pending tasks indexing latency search latency circuit breakers disk watermarks ``` Useful APIs: ```http GET _cluster/health GET _cat/nodes?v GET _cat/shards?v GET _cat/indices?v GET _cluster/pending_tasks GET _nodes/stats ``` ## 27.3 Fluent Bit Monitor: ```text input records output records retries failed chunks storage queue usage service restarts ``` ## 27.4 Data Prepper Monitor: ```text requestsReceived successRequests badRequests requestsRejected grokProcessingMismatch grokProcessingErrors date failures OpenSearch document errors ``` ## 27.5 Longhorn Monitor: ```text volume health replica count replica placement disk usage snapshot growth attachment failures ``` --- # 28. Troubleshooting Matrix | Symptom | Layer | Likely cause | First checks | |---|---|---|---| | Fluent Bit `extra indentation level found` | Fluent Bit config | classic config indentation / inline Lua | reported line; move Lua to separate file | | Fluent Bit repeated `502 Bad Gateway` | Ingress/upstream | NGINX reachable but Data Prepper backend unhealthy | Data Prepper pod, Service, EndpointSlice, :2021 | | Fluent Bit `connection refused` | Network/listener | endpoint unavailable | IP/port/firewall/listener | | Data Prepper `Address already in use` | Pipeline | multiple HTTP sources on same port | keep one router HTTP source | | Data Prepper `No definition for key 'PORT'` | Grok | unsupported pattern in tested 2.8 setup | use `POSINT` | | Data Prepper pipeline YAML Boolean error | YAML | options at wrong hierarchy | indent under `grok:` correctly | | `parsed: {}` | Parser | Grok did not match | compare raw line with regex | | Wrong source in dedicated index | Routing | router/drop condition wrong | inspect `log_source`, route, new docs | | Index exists, `docs.count=0` | Sink/ingestion | no successful documents | Data Prepper sink logs + pipeline metrics | | Data Prepper `Request execution cancelled` / `I/O session is closed` | OpenSearch sink | DNS/TCP/TLS/auth/sink instability | test each layer to :9200 | | OpenSearch pod `Running` but cluster unavailable | OpenSearch transport | TLS/discovery/cluster manager | pod logs, :9300, headless DNS, SAN | | `No subject alternative DNS name ... headless` | TLS | SAN mismatch | regenerate certs with actual DNS SANs | | `Cluster manager not present` | Cluster formation | transport TLS/discovery issue | headless Service, :9300, cert SAN, nodes_dn | | `Unauthorized` | Authentication | wrong active credential | active Security config / user hash | | `Resource 'admin' is reserved` | Security API | reserved resource cannot be changed by that API | update YAML + securityadmin using admin cert | | Secret `subPath` `not a directory` | Kubernetes mount | Secret key name mismatch | `internal_users.yml` exactly | | Dashboards `publicBaseUrl is not allowed` | Dashboards config | unsupported key in tested version | remove it | | Dashboards 401 | Authentication | auth required/missing | security config / credential flow | | Dashboards 404 after auth | Base path/routing | application not aware of ingress path | `basePath`, `rewriteBasePath` | | Dashboards permission `indices:monitor/settings/get` | Authorization | backend role missing required permission | identify plugin + target index before grant | | NGINX duplicate host/path | Ingress | catch-all route already exists | use unique host/path | | `too_many_nested_clauses` | OpenSearch mapping/query | too many default searchable fields | restrict `index.query.default_field`; control mappings | | SMTP `Invalid Addresses` | Notification | sender/recipient/relay/auth issue | address + keystore + relay policy | --- # 29. Troubleshooting Decision Trees ## 29.1 Log ingestion ```text Is the source file receiving new lines? | +-- NO --> troubleshoot cPanel/application logging | +-- YES | v Did Fluent Bit discover/read the file? | +-- NO --> path, permissions, Tail DB, Read_from_Head policy | +-- YES | v Does event contain source_file + log_source? | +-- NO --> Path_Key / Lua classification | +-- YES | v Does Fluent Bit POST /logs successfully? | +-- 502 --> Ingress reached, upstream unhealthy +-- refused --> listener/network/firewall issue | +-- success | v Is Data Prepper 1/1 Running and :2021 listening? | +-- NO --> startup/pipeline configuration | +-- YES | v Does the router match the expected log_source? | +-- NO --> route expression | +-- YES | v Does the parser populate expected fields? | +-- NO --> Grok/date/key-value mismatch | +-- YES | v Can Data Prepper reach OpenSearch :9200? | +-- NO --> DNS/TCP/TLS/CA | +-- YES | v Does OpenSearch authenticate/authorize logger? | +-- NO --> user/role/index permissions | +-- YES | v Does the expected index contain new documents? ``` ## 29.2 OpenSearch deployment ```text Pod Pending? | +--> node affinity +--> hard anti-affinity +--> taints/tolerations +--> CPU/memory requests +--> PVC/Longhorn Pod starts but crashes? | +--> vm.max_map_count +--> JVM heap / OOM +--> certificate file permissions +--> TLS key format +--> opensearch.yml syntax +--> bootstrap security Pods run but cluster does not form? | +--> headless Service DNS +--> transport :9300 +--> SANs +--> nodes_dn +--> discovery settings REST HTTPS fails? | +--> REST TLS enabled +--> cert/key paths +--> CA +--> hostname/SAN +--> certificate expiry Authentication fails? | +--> active Security index +--> internal_users.yml +--> securityadmin applied? ``` ## 29.3 Dashboards ```text Can browser reach NodePort? | +-- NO --> node firewall / NodePort / ingress-nginx Service | +-- YES v Does NGINX match /opensearch? | +-- NO --> Ingress rule / ingressClassName | +-- YES v Does Service have ready Dashboards endpoint? | +-- NO --> selector / pod readiness | +-- YES v Did Dashboards start without config validation errors? | +-- NO --> current + previous logs | +-- YES v Can Dashboards connect to OpenSearch with CA + kibanaserver? | +-- NO --> DNS/TLS/SAN/auth | +-- YES v Do assets/redirects stay under /opensearch? | +-- NO --> basePath/rewriteBasePath | +-- YES --> path is healthy ``` --- # 30. Daily / Weekly Operational Checklists ## 30.1 Daily - [ ] All OpenSearch pods are `Running` and Ready. - [ ] OpenSearch cluster health is expected. - [ ] `_cat/nodes` lists all expected nodes. - [ ] No recurring transport TLS errors. - [ ] Data Prepper pod is Ready. - [ ] Data Prepper has no sustained OpenSearch sink errors. - [ ] Fluent Bit services are running. - [ ] No repeated Fluent Bit 5xx/connection failures. - [ ] New cPanel access documents are arriving. - [ ] New Audit documents are arriving. - [ ] `@timestamp` is current and sane. - [ ] Grok processing error/mismatch rates are reviewed. - [ ] Disk/PVC usage is below operational thresholds. - [ ] Longhorn volumes are healthy. - [ ] Dashboards loads and queries data. ## 30.2 Weekly - [ ] Review OpenSearch JVM heap/GC trends. - [ ] Review index growth and retention. - [ ] Review Fluent Bit backlog utilization. - [ ] Review Data Prepper rejected/bad requests. - [ ] Review parsing failures/unhandled log sources. - [ ] Review security/authentication failures. - [ ] Review certificate expiry dates. - [ ] Verify snapshots are succeeding. - [ ] Periodically test restore procedure. - [ ] Review alert quality and false positives. --- # 31. Windows Fluent Bit Variant This is an optional extension for downloaded Windows log files. ## 31.1 Recommended directory separation ```text C:\fluent-bit\ ├── logs\ ├── state\ └── buffer\ ``` Do not place the Tail DB inside a wildcard source directory if it could be ingested accidentally. ## 31.2 Example concept ```ini [SERVICE] Flush 1 Daemon Off Log_Level info storage.metrics On storage.path C:\fluent-bit\buffer storage.sync full storage.backlog.mem_limit 500M [INPUT] Name tail Path C:\fluent-bit\logs\*.log Path_Key source_file Tag downloaded.logs DB C:\fluent-bit\state\tail.db DB.Journal_Mode wal Read_from_Head true Refresh_Interval 5 Skip_Long_Lines true Mem_Buf_Limit 512MB Storage.Type filesystem [OUTPUT] Name http Host 118.67.213.76 Port 31599 Format json HTTP_Method POST URI /logs Match * Retry_Limit no_limits Storage.Total_Limit_Size 4G ``` Before ingesting all historical files: 1. Test one representative file. 2. Confirm existing content is read. 3. Restart Fluent Bit and confirm offset resume behavior. 4. Verify `source_file`. 5. Verify one test document in OpenSearch. 6. Only then expand the wildcard. --- # 32. Cheat Sheet ## 32.1 Complete process summary ```text 1. Verify Kubernetes nodes, Longhorn, vm.max_map_count 2. Verify/generate Root CA, node certs, admin cert 3. Create Kubernetes Secrets 4. Configure OpenSearch Helm values 5. helm lint 6. helm template 7. kubectl --dry-run=server 8. helm upgrade --install OpenSearch 9. Verify pods/PVC/TLS/cluster health/security 10. Deploy Dashboards with CA + kibanaserver Secret 11. Configure /opensearch Ingress/basePath 12. Deploy Data Prepper with ONE HTTP :2021 router 13. Configure dedicated access/syslog/audit/catch-all pipelines 14. Configure Fluent Bit Tail DB + filesystem buffer + log_source 15. Verify /logs end-to-end 16. Create index templates/mappings 17. Create Dashboards workspace/index patterns 18. Build cPanel and Linux Audit dashboards 19. Configure anomaly detector / forecast / alerts 20. Configure SMTP keystore 21. Test failures, snapshots, and recovery ``` ## 32.2 Kubernetes/OpenSearch quick commands ```bash export NS=observability export OS_RELEASE=brilliant-opensearch export OS_STS=brilliant-opensearch-cluster-master export OS_SVC=brilliant-opensearch-cluster-master kubectl get pods -n "$NS" -o wide kubectl get sts -n "$NS" kubectl get svc -n "$NS" kubectl get pvc -n "$NS" kubectl get ingress -A kubectl logs -n "$NS" "${OS_STS}-0" -c opensearch --tail=200 kubectl logs -n "$NS" "${OS_STS}-0" -c opensearch --previous --tail=200 kubectl port-forward -n "$NS" svc/"$OS_SVC" 9200:9200 curl -k -u 'admin:' https://localhost:9200/_cluster/health?pretty curl -k -u 'admin:' 'https://localhost:9200/_cat/nodes?v' ``` ## 32.3 Helm quick commands ```bash helm lint "$OS_CHART" -f values.yaml helm template "$OS_RELEASE" "$OS_CHART" \ -n "$NS" -f values.yaml > rendered-opensearch.yaml kubectl apply --dry-run=server -f rendered-opensearch.yaml helm upgrade --install "$OS_RELEASE" "$OS_CHART" \ -n "$NS" -f values.yaml kubectl rollout status statefulset/"$OS_STS" -n "$NS" ``` ## 32.4 TLS quick commands ```bash openssl verify -CAfile root-ca.pem node-0.pem openssl x509 -in node-0.pem -noout -subject -nameopt RFC2253 openssl x509 -in node-0.pem -noout -dates openssl x509 -in node-0.pem -noout -text | sed -n '/Subject Alternative Name/,+2p' ``` ## 32.5 Data Prepper quick commands ```bash kubectl get pods -n observability -l app=data-prepper -o wide kubectl get svc data-prepper -n observability -o wide kubectl get endpointslice -n observability \ -l kubernetes.io/service-name=data-prepper -o wide kubectl logs -n observability deploy/data-prepper --tail=200 kubectl exec -n observability deploy/data-prepper -- \ sh -c 'grep -i ":07E5" /proc/net/tcp /proc/net/tcp6 || true' ``` ## 32.6 Fluent Bit quick commands ```bash systemctl status fluent-bit --no-pager journalctl -u fluent-bit --since '10 minutes ago' --no-pager du -sh /var/lib/fluent-bit/storage/ ls -lh /var/lib/fluent-bit/state/ ``` ## 32.7 Test `/logs` ```bash curl -i -X POST \ 'http://118.67.213.76:31599/logs' \ -H 'Content-Type: application/json' \ --data '[{"log":"manual test","source_file":"manual","log_source":"test.manual"}]' ``` ## 32.8 OpenSearch Dev Tools quick queries ```http GET _cluster/health GET _cat/nodes?v GET _cat/indices/cpanel-*?v GET cpanel-access-log-*/_count { "query": { "range": { "@timestamp": { "gte": "now-24h", "lte": "now" } } } } GET cpanel-access-log-*/_search { "size": 5, "sort": [{"@timestamp": "desc"}] } ``` ## 32.9 DQL quick filters ```text 2xx: parsed.http_status >= 200 and parsed.http_status < 300 4xx: parsed.http_status >= 400 and parsed.http_status < 500 5xx: parsed.http_status >= 500 and parsed.http_status < 600 Audit commands: audit.log_type: "EXECVE" Failed syscalls: audit.log_type: "SYSCALL" AND syscall.success: "no" Authentication activity: audit.log_type: ("LOGIN" OR "USER_ACCT" OR "USER_START" OR "CRED_ACQ") ``` ## 32.10 Error-to-layer quick map ```text 502 from Fluent Bit -> NGINX reachable; check Data Prepper upstream connection refused -> listener/network parsed: {} -> Grok mismatch index docs.count=0 -> ingestion/sink not proven SAN mismatch -> certificate DNS identity Unauthorized -> authentication security_exception -> authorization 404 after Dashboards auth-> basePath/routing CrashLoop after config -> application validation/logs ``` --- # 33. Lessons Learned and Known Limitations 1. **Helm rendering is mandatory.** Inspect final StatefulSet/Deployment, not only `values.yaml`. 2. **`Running` is not enough.** Validate readiness, application logs, OpenSearch cluster health, and backend connectivity separately. 3. **Transport TLS and REST TLS are different boundaries.** `curl -k` against port 9200 cannot repair transport TLS on 9300. 4. **SANs matter.** Correct CA trust is insufficient when the Service/headless DNS is missing from the certificate. 5. **`kibanaserver` is a user; `kibana_server` is a role.** Do not confuse them. 6. **A mounted Security YAML file is not automatically the active Security index.** Apply changes appropriately with Security tooling. 7. **Secret `subPath` names match exact keys.** `internal_users.yaml` and `internal_users.yml` are different keys. 8. **One Data Prepper HTTP source per listener.** Route internally to downstream pipelines. 9. **A sink without `routes:` receives all events.** Define an explicit unknown route if the catch-all must contain only unhandled sources. 10. **Fluent Bit Tail DB is not the event queue.** It remembers offsets; filesystem storage buffers undelivered chunks. 11. **Unlimited retries do not mean unlimited durability.** A finite filesystem queue can still overflow and drop old chunks. 12. **`parsed: {}` means transport can be healthy while parsing failed.** Debug the raw line against the parser. 13. **Audit records are multi-record logical events.** Correlate using `audit.event_id`. 14. **Whitespace-splitting EXECVE arguments is not fully quote-aware.** Preserve `args_raw`. 15. **Dynamic Audit arguments can create mapping explosion.** Normalize high-value fields and restrict default searchable fields. 16. **Index existence does not prove document ingestion.** Always inspect `docs.count` and sample `_source`. 17. **NGINX Ingress routing is controller-wide.** Duplicate hostless `/` rules can conflict across namespaces. 18. **Dashboards path hosting requires application-aware basePath configuration.** 19. **Version-specific settings matter.** A configuration key accepted by Kibana or another OpenSearch Dashboards version may be rejected by 3.7.0. 20. **Least privilege first.** Investigate the action and target resource before granting broad permissions. 21. **OpenSearch replicas and Longhorn replicas consume storage independently.** 22. **OpenSearch shard replicas are not backups.** Snapshot and restore testing remain mandatory. --- # 34. Final Definition of Done The platform is considered complete only when all of the following are true. ## OpenSearch - [ ] Three OpenSearch pods are Ready. - [ ] Pods are on separate eligible Kubernetes nodes. - [ ] Three 120Gi Longhorn PVCs are Bound. - [ ] Transport TLS works with correct SANs. - [ ] REST HTTPS works. - [ ] Node certificate DNs match `nodes_dn`. - [ ] Admin certificate DN matches `admin_dn`. - [ ] Cluster manager is elected. - [ ] `_cat/nodes` lists all three nodes. - [ ] Cluster health is expected. - [ ] Default/demo credentials are removed or changed. - [ ] Real secrets are absent from Git. ## Dashboards - [ ] Dashboards is Ready. - [ ] CA is mounted. - [ ] `kibanaserver` password comes from a Secret. - [ ] TLS verification is `full`. - [ ] `/opensearch` basePath works. - [ ] Ingress does not conflict with Grafana `/`. - [ ] Workspaces work with Security multi-tenancy disabled. - [ ] Data source connection succeeds. ## Fluent Bit - [ ] Tail paths are correct. - [ ] Tail DB is persistent and writable. - [ ] Filesystem buffer is writable. - [ ] `source_file` is present. - [ ] `log_source` is correct. - [ ] `/logs` delivery succeeds. - [ ] Retry/backlog behavior is tested. - [ ] Queue usage is monitored. ## Data Prepper - [ ] Only one HTTP source owns port 2021. - [ ] Service and EndpointSlice point to the current pod. - [ ] Health endpoint is healthy. - [ ] Router matches known log sources. - [ ] Catch-all contains only unhandled sources. - [ ] Access Grok populates `parsed.*`. - [ ] Audit parsers populate expected type-specific fields. - [ ] Date processors create correct `@timestamp`. - [ ] OpenSearch sink has no persistent errors. ## OpenSearch data - [ ] `cpanel-access-log-*` contains documents. - [ ] `cpanel-system-syslog-*` contains expected syslog only. - [ ] `cpanel-audit-log-*` contains Audit records. - [ ] `cpanel-all-log-*` exposes unknown/unhandled sources. - [ ] cPanel access field mappings are correct. - [ ] Audit default query fields are controlled. - [ ] Retention/ISM strategy is defined. ## Dashboards and alerting - [ ] cPanel Access Overview exists. - [ ] Linux Server Audit & Security Overview exists. - [ ] Anomaly detector runs and produces sensible output. - [ ] Forecasting runs if required. - [ ] Monitor and trigger exist. - [ ] SMTP credentials exist in OpenSearch keystore. - [ ] Test notification succeeds. - [ ] Real alert action succeeds. ## Operations - [ ] OpenSearch pod-failure test completed. - [ ] Fluent Bit/Data Prepper interruption recovery tested. - [ ] Snapshot repository configured. - [ ] Snapshot schedule configured. - [ ] Restore test completed. - [ ] Certificate-expiry monitoring configured. - [ ] Daily/weekly checks assigned to operations. --- ## Verification References Used for This Consolidation This combined SOP was cross-checked against the official documentation for: - OpenSearch Helm installation and chart configuration. - OpenSearch Security YAML configuration and `securityadmin.sh` behavior. - OpenSearch index replica settings and cluster health semantics. - OpenSearch Dashboards Workspaces and multi-tenancy compatibility. - OpenSearch Dashboards multiple data sources. - Data Prepper HTTP source, Grok processor, key-value processor, conditional routing, server API port, and OpenSearch sink. - Fluent Bit Tail input, filesystem buffering, retry semantics, and output queue limits. - Kubernetes StatefulSet/headless Service behavior and Kubernetes Secret/volume semantics. When upgrading any major/minor component, rerun the same validation flow against the documentation for the exact version before applying old configuration unchanged. --- **Prepared by: Md. Mosiur Rahnman**