116 KiB
OpenSearch + cPanel/Linux Observability Platform — Complete SOP
Canonical combined runbook for deploying and operating OpenSearch on Kubernetes, OpenSearch Dashboards, Fluent Bit, Data Prepper, cPanel/Linux log ingestion, Linux Audit ingestion, dashboards, anomaly detection, forecasting, alerting, and troubleshooting.
Prepared by: Md. Mosiur Rahnman
Document type: Standard Operating Procedure (SOP) / Future Reference Runbook
Primary environment documented: Kubernetes/K3s, OpenSearch 3.7.0, OpenSearch Dashboards 3.7.0, Data Prepper 2.8.0, Longhorn, NGINX Ingress, cPanel/Linux
Primary namespace:observability
Document Control
| Item | Value |
|---|---|
| Prepared by | Md. Mosiur Rahnman |
| Scope | OpenSearch platform + cPanel/Linux log observability |
| OpenSearch version used in the implementation | 3.7.0 |
| OpenSearch Dashboards version used | 3.7.0 |
| Data Prepper version used | 2.8.0 |
| Kubernetes namespace | observability |
| OpenSearch Helm release | brilliant-opensearch |
| OpenSearch cluster name | brilliant-opensearch-cluster |
| OpenSearch StatefulSet | brilliant-opensearch-cluster-master |
| OpenSearch Service | brilliant-opensearch-cluster-master |
| Data Prepper Service | data-prepper |
| StorageClass | longhorn |
| OpenSearch pod count | 3 |
| OpenSearch PVC size | 120Gi per pod |
| Dashboards external path | /opensearch |
| Data Prepper ingestion path | /logs |
| NGINX HTTP NodePort observed | 31599 |
| NGINX HTTPS NodePort observed | 31186 |
| cPanel access log | /usr/local/cpanel/logs/access_log |
Important
Values such as node IPs, NodePorts, release names, Service names, chart paths, certificate DNs, and storage topology are environment-specific. Verify them before reusing this SOP in another cluster.
Table of Contents
- Purpose and Scope
- Final Architecture
- Core Design Principles
- Environment Variables and Naming
- Pre-Deployment Prerequisites
- OpenSearch PKI and TLS
- Kubernetes Secrets
- OpenSearch Helm Configuration
- Render, Validate, and Deploy OpenSearch
- OpenSearch Security Initialization and Admin Password
- OpenSearch Cluster Verification
- OpenSearch Dashboards Deployment
- Data Prepper — Full Deployment, Routing, Parsing, and Verification
- Fluent Bit — Full cPanel/Linux Collection Procedure
- cPanel Access Log Parsing
- Linux Syslog and Other Log Sources
- Linux Audit Log Parsing
- OpenSearch Index Templates and Mapping
- End-to-End Verification
- OpenSearch Dashboards Data Source and Workspace
- cPanel Access Dashboard
- Linux Audit Security Dashboard
- Anomaly Detection, Forecasting, and Alerting
- SMTP Notification Configuration
- Reliability, Backup, and Recovery Tests
- Security Hardening
- Operational Monitoring
- Troubleshooting Matrix
- Troubleshooting Decision Trees
- Daily / Weekly Operational Checklists
- Windows Fluent Bit Variant
- Cheat Sheet
- Lessons Learned and Known Limitations
- Final Definition of Done
1. Purpose and Scope
This SOP consolidates the complete implementation into one operational reference. It covers the path from Kubernetes infrastructure through searchable and visualized cPanel/Linux logs:
cPanel / Linux Servers
|
| log files
v
Fluent Bit
- Tail input
- Offset DB
- Filesystem buffering
- source_file
- log_source classification
|
| HTTP/HTTPS POST /logs
v
NGINX Ingress
|
v
Data Prepper :2021
- One HTTP source
- Router
- Grok/date/key-value processors
|
| HTTPS :9200
v
OpenSearch 3-node cluster
- Longhorn PVCs
- REST TLS
- Transport TLS
- Security plugin
|
v
OpenSearch Dashboards
- Discover / Logs
- Workspaces
- Dashboards
- Anomaly Detection
- Forecasting
- Alerting
- Email notifications
This document preserves the tested implementation while separating:
- Production baseline — settings that should normally remain enabled.
- Environment-specific values — values that must be rechecked before reuse.
- Diagnostic workarounds — temporary troubleshooting settings that should not become permanent defaults.
- Known limitations — parser or reliability behaviors that need to be understood rather than hidden.
2. Final Architecture
2.1 Kubernetes/OpenSearch topology
+--------------------------------------+
| Kubernetes / K3s |
| Namespace: observability |
| |
Browser | OpenSearch Dashboards |
| | :5601 |
| HTTP/HTTPS | | HTTPS |
v | v |
NGINX Ingress ---------->| OpenSearch Service :9200 |
^ | | |
| | +------+------+------+ |
| | | | | |
| | pod-0 pod-1 pod-2 |
| | | | | |
| | PVC-0 PVC-1 PVC-2 |
| | \________ Transport TLS :9300 ____/|
| | |
| | Data Prepper :2021 |
| | | HTTPS :9200 |
| | +----------------------------->|
| +--------------------------------------+
|
| HTTP/HTTPS /logs
|
Fluent Bit on cPanel/Linux servers
2.2 Log-routing architecture
Fluent Bit
|
| JSON + source_file + log_source
v
Data Prepper router
|
+--> cpanel.access_log --> access pipeline --> cpanel-access-log-YYYY.MM.dd
|
+--> system.syslog ----> syslog pipeline --> cpanel-system-syslog-YYYY.MM.dd
|
+--> audit.audit.log --> audit pipeline ---> cpanel-audit-log-YYYY.MM.dd
|
+--> unknown sources --> catch-all --------> cpanel-all-log-YYYY.MM.dd
3. Core Design Principles
- Validate one layer at a time. Do not debug OpenSearch if Fluent Bit is not reading the source file.
- Render Helm before applying it. A valid
values.yamlcan still render an invalid or unintended Kubernetes manifest. - Keep transport, parsing, routing, and storage failures separate.
parsed: {}is a parser problem;502is normally an upstream availability problem. - Use one Data Prepper HTTP listener per port. Multiple pipelines must not bind
:2021independently. - Classify logs early. Fluent Bit knows the original file path, so create
source_fileand normalizedlog_sourcebefore transport. - Use dedicated downstream pipelines for different formats. cPanel access logs, syslog, and Auditd have fundamentally different schemas.
- Preserve the raw log. Parsing should enrich the event, not destroy the original message.
- Use least privilege. Do not reuse the OpenSearch admin credential for Dashboards or ingestion.
- Treat TLS trust and TLS hostname validation as different checks. A trusted CA does not fix a SAN mismatch.
- Treat Kubernetes
Runningand application health as different states. A pod can run while OpenSearch cluster formation, authentication, or authorization is broken. - OpenSearch shard replicas are not backups. Use snapshots to independent storage.
- Base64 is encoding, not encryption. Kubernetes Secret output must still be treated as sensitive.
- Do not commit real passwords, private keys, SMTP credentials, or encryption master keys to Git.
4. Environment Variables and Naming
Use variables in shell sessions to reduce copy/paste mistakes.
export NS="observability"
export OS_RELEASE="brilliant-opensearch"
export OS_CLUSTER="brilliant-opensearch-cluster"
export OS_STS="brilliant-opensearch-cluster-master"
export OS_SVC="brilliant-opensearch-cluster-master"
export OS_HEADLESS_SVC="brilliant-opensearch-cluster-master-headless"
export DP_SVC="data-prepper"
export DP_PORT="2021"
export DASH_BASEPATH="/opensearch"
export INGRESS_HTTP_NODEPORT="31599"
export INGRESS_HTTPS_NODEPORT="31186"
Check the real values instead of assuming them:
kubectl get ns
helm list -A
kubectl get sts,deploy,svc,ingress -A
kubectl get nodes -o wide
For the local Helm chart used in the implementation:
export OS_CHART="${HOME}/monitoring/helm-charts/charts/opensearch"
Verify it exists:
test -f "$OS_CHART/Chart.yaml" && echo "OpenSearch chart found"
5. Pre-Deployment Prerequisites
5.1 Client tools
Verify required tools:
kubectl version --client
helm version
openssl version
curl --version
Optional but useful:
jq --version
yq --version
5.2 Kubernetes nodes
kubectl get nodes -o wide
kubectl get nodes --show-labels
kubectl describe node k8s-storage-01 | sed -n '/Taints:/,/Unschedulable:/p'
Target placement from the implementation:
k8s-storage-01 -> eligible, dedicated=storage:NoSchedule
k8s-worker-01 -> eligible
k8s-worker-02 -> eligible
k8s-cp-01 -> excluded from OpenSearch
If k8s-storage-01 is tainted:
dedicated=storage:NoSchedule
OpenSearch requires a matching toleration if that node should be eligible.
5.3 Longhorn
kubectl get storageclass
kubectl get pods -n longhorn-system
Expected StorageClass:
longhorn
Storage planning for the documented cluster:
3 OpenSearch pods x 120Gi = 360Gi logical PVC capacity
Longhorn replication consumes additional physical storage. OpenSearch shard replication consumes additional logical data copies as well; they are separate redundancy layers.
5.4 vm.max_map_count
Check every eligible OpenSearch node:
for node in k8s-storage-01 k8s-worker-01 k8s-worker-02; do
echo "=== $node ==="
ssh "$node" 'sysctl vm.max_map_count'
done
Target used by the Helm chart:
vm.max_map_count = 262144
Prefer configuring the kernel permanently on the host:
cat >/etc/sysctl.d/99-opensearch.conf <<'EOF_SYSCTL'
vm.max_map_count=262144
EOF_SYSCTL
sysctl --system
If the hosts are already configured, disable privileged sysctl init mechanisms in Helm. Do not enable multiple sysctl methods unnecessarily.
5.5 Namespace
kubectl create namespace "$NS" --dry-run=client -o yaml | kubectl apply -f -
6. OpenSearch PKI and TLS
6.1 Certificate roles
Use separate certificate purposes:
Root CA
├── signs node certificates
└── signs admin certificate
Node certificates
├── REST TLS server identity
├── transport TLS server identity
└── transport TLS client identity
Admin certificate
└── privileged OpenSearch Security administration
Never mount the Root CA private key into OpenSearch.
root-ca-key.pem -> offline/encrypted CA material only
root-ca.pem -> safe to distribute as trust anchor
6.2 Expected PKI files
root-ca.pem
root-ca-key.pem
admin.pem
admin-key.pem
node-0.pem
node-0-key.pem
node-1.pem
node-1-key.pem
node-2.pem
node-2-key.pem
6.3 SAN design
Use a short certificate CN and put Kubernetes DNS identities in SANs.
Common Service DNS names:
brilliant-opensearch-cluster-master
brilliant-opensearch-cluster-master.observability
brilliant-opensearch-cluster-master.observability.svc
brilliant-opensearch-cluster-master.observability.svc.cluster.local
brilliant-opensearch-cluster-master-headless
brilliant-opensearch-cluster-master-headless.observability
brilliant-opensearch-cluster-master-headless.observability.svc
brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local
Pod-specific names should also be included where transport communication uses them:
brilliant-opensearch-cluster-master-0.brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local
brilliant-opensearch-cluster-master-1.brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local
brilliant-opensearch-cluster-master-2.brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local
6.4 Optional OpenSSL generation template
If certificates already exist, skip generation and perform verification only.
Root CA
openssl genrsa -out root-ca-key.pem 4096
openssl req -x509 -new -sha256 \
-key root-ca-key.pem \
-days 3650 \
-out root-ca.pem \
-subj '/C=BD/ST=Dhaka/L=Dhaka/O=Brilliant/OU=OpenSearch CA/CN=Brilliant OpenSearch Root CA'
Admin certificate
openssl genrsa -out admin-key-rsa.pem 2048
openssl pkcs8 -topk8 -nocrypt \
-in admin-key-rsa.pem \
-out admin-key.pem
openssl req -new \
-key admin-key.pem \
-out admin.csr \
-subj '/C=BD/ST=Dhaka/L=Dhaka/O=Brilliant/OU=OpenSearch Administrators/CN=opensearch-admin'
cat >admin.ext <<'EOF_ADMIN_EXT'
basicConstraints=CA:FALSE
keyUsage=digitalSignature,keyEncipherment
extendedKeyUsage=clientAuth
EOF_ADMIN_EXT
openssl x509 -req \
-in admin.csr \
-CA root-ca.pem \
-CAkey root-ca-key.pem \
-CAcreateserial \
-out admin.pem \
-days 825 \
-sha256 \
-extfile admin.ext
Node certificate example for pod 0
Create a SAN file containing the actual Service and pod DNS names:
cat >node-0.ext <<'EOF_NODE0_EXT'
basicConstraints=CA:FALSE
keyUsage=digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth,clientAuth
subjectAltName=@alt_names
[alt_names]
DNS.1=brilliant-opensearch-cluster-master
DNS.2=brilliant-opensearch-cluster-master.observability
DNS.3=brilliant-opensearch-cluster-master.observability.svc
DNS.4=brilliant-opensearch-cluster-master.observability.svc.cluster.local
DNS.5=brilliant-opensearch-cluster-master-headless
DNS.6=brilliant-opensearch-cluster-master-headless.observability
DNS.7=brilliant-opensearch-cluster-master-headless.observability.svc
DNS.8=brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local
DNS.9=brilliant-opensearch-cluster-master-0.brilliant-opensearch-cluster-master-headless
DNS.10=brilliant-opensearch-cluster-master-0.brilliant-opensearch-cluster-master-headless.observability.svc.cluster.local
EOF_NODE0_EXT
openssl genrsa -out node-0-key-rsa.pem 2048
openssl pkcs8 -topk8 -nocrypt \
-in node-0-key-rsa.pem \
-out node-0-key.pem
openssl req -new \
-key node-0-key.pem \
-out node-0.csr \
-subj '/C=BD/ST=Dhaka/L=Dhaka/O=Brilliant/OU=OpenSearch Nodes/CN=opensearch-master-0'
openssl x509 -req \
-in node-0.csr \
-CA root-ca.pem \
-CAkey root-ca-key.pem \
-CAcreateserial \
-out node-0.pem \
-days 825 \
-sha256 \
-extfile node-0.ext
Repeat with the correct CN and pod-specific SAN for node 1 and node 2.
6.5 Certificate verification
Verify chain:
openssl verify -CAfile root-ca.pem admin.pem
openssl verify -CAfile root-ca.pem node-0.pem
openssl verify -CAfile root-ca.pem node-1.pem
openssl verify -CAfile root-ca.pem node-2.pem
Verify subjects:
openssl x509 -in admin.pem -noout -subject -nameopt RFC2253
openssl x509 -in node-0.pem -noout -subject -nameopt RFC2253
openssl x509 -in node-1.pem -noout -subject -nameopt RFC2253
openssl x509 -in node-2.pem -noout -subject -nameopt RFC2253
Verify SANs:
openssl x509 -in node-0.pem -noout -text | sed -n '/Subject Alternative Name/,+2p'
Verify EKU:
openssl x509 -in node-0.pem -noout -text | sed -n '/Extended Key Usage/,+2p'
Verify certificate/key match:
openssl x509 -in node-0.pem -pubkey -noout | sha256sum
openssl pkey -in node-0-key.pem -pubout | sha256sum
The hashes must match.
Verify validity:
openssl x509 -in node-0.pem -noout -dates
7. Kubernetes Secrets
7.1 Initial admin password
Do not place the real password in Git.
read -rsp 'Initial OpenSearch admin password: ' OPENSEARCH_INITIAL_ADMIN_PASSWORD
echo
kubectl create secret generic opensearch-secret \
-n "$NS" \
--from-literal=OPENSEARCH_INITIAL_ADMIN_PASSWORD="$OPENSEARCH_INITIAL_ADMIN_PASSWORD" \
--dry-run=client -o yaml | kubectl apply -f -
unset OPENSEARCH_INITIAL_ADMIN_PASSWORD
7.2 Node certificate Secrets
kubectl create secret generic opensearch-node-0-cert \
-n "$NS" \
--from-file=node-0.pem \
--from-file=node-0-key.pem \
--from-file=root-ca.pem \
--dry-run=client -o yaml | kubectl apply -f -
kubectl create secret generic opensearch-node-1-cert \
-n "$NS" \
--from-file=node-1.pem \
--from-file=node-1-key.pem \
--from-file=root-ca.pem \
--dry-run=client -o yaml | kubectl apply -f -
kubectl create secret generic opensearch-node-2-cert \
-n "$NS" \
--from-file=node-2.pem \
--from-file=node-2-key.pem \
--from-file=root-ca.pem \
--dry-run=client -o yaml | kubectl apply -f -
7.3 Admin certificate Secret
kubectl create secret generic opensearch-admin-cert \
-n "$NS" \
--from-file=admin.pem \
--from-file=admin-key.pem \
--from-file=root-ca.pem \
--dry-run=client -o yaml | kubectl apply -f -
7.4 Verify Secret metadata without dumping data
kubectl get secret -n "$NS" \
opensearch-secret \
opensearch-node-0-cert \
opensearch-node-1-cert \
opensearch-node-2-cert \
opensearch-admin-cert
Avoid:
kubectl get secret <name> -o yaml
when the output would expose sensitive Base64 data in terminals, tickets, screenshots, or chat logs.
8. OpenSearch Helm Configuration
8.1 Resource design
Per OpenSearch pod:
CPU request: 2
CPU limit: 4
Memory request: 6Gi
Memory limit: 8Gi
JVM heap: 3Gi
PVC: 120Gi
8.2 Helm values baseline
The exact chart schema must always be checked with the pinned chart:
helm show values "$OS_CHART" > /tmp/opensearch-default-values.yaml
Critical values for this environment:
clusterName: brilliant-opensearch-cluster
nodeGroup: master
singleNode: false
masterService: brilliant-opensearch-cluster-master
roles:
- master
- ingest
- data
- remote_cluster_client
replicas: 3
image:
repository: opensearchproject/opensearch
tag: "3.7.0"
pullPolicy: IfNotPresent
opensearchJavaOpts: "-Xms3g -Xmx3g"
resources:
requests:
cpu: "2"
memory: 6Gi
limits:
cpu: "4"
memory: 8Gi
persistence:
enabled: true
storageClass: longhorn
accessModes:
- ReadWriteOnce
size: 120Gi
antiAffinity: hard
antiAffinityTopologyKey: kubernetes.io/hostname
podManagementPolicy: Parallel
protocol: https
httpPort: 9200
transportPort: 9300
metricsPort: 9600
podSecurityContext:
fsGroup: 1000
runAsUser: 1000
securityContext:
runAsNonRoot: true
runAsUser: 1000
capabilities:
drop:
- ALL
envFrom:
- secretRef:
name: opensearch-secret
secretMounts: []
sysctl:
enabled: false
sysctlInit:
enabled: false
Note
If the worker nodes are not preconfigured with
vm.max_map_count=262144, enable exactly one supported sysctl method. Do not enable both pod-level sysctl and a privileged sysctl init container unless you have a specific reason.
8.3 Node scheduling
Storage taint toleration:
tolerations:
- key: dedicated
operator: Equal
value: storage
effect: NoSchedule
Use labels/affinity to include only the intended storage/worker nodes. Do not rely on anti-affinity to exclude the control plane; anti-affinity controls pod co-location, not arbitrary node blacklisting.
Verify rendered affinity before install.
8.4 Per-pod certificate selection
The main OpenSearch container should see only its selected runtime certificate set.
extraVolumes:
- name: node-0-source
secret:
secretName: opensearch-node-0-cert
defaultMode: 0440
- name: node-1-source
secret:
secretName: opensearch-node-1-cert
defaultMode: 0440
- name: node-2-source
secret:
secretName: opensearch-node-2-cert
defaultMode: 0440
- name: runtime-certs
emptyDir: {}
extraVolumeMounts:
- name: runtime-certs
mountPath: /usr/share/opensearch/config/certs
readOnly: true
Init container:
extraInitContainers:
- name: select-node-certificate
image: busybox:1.36
command:
- sh
- -ec
- |
ordinal="${HOSTNAME##*-}"
case "$ordinal" in
0|1|2) ;;
*)
echo "Unexpected StatefulSet ordinal: $ordinal" >&2
exit 1
;;
esac
src="/source/node-${ordinal}"
cert="${src}/node-${ordinal}.pem"
key="${src}/node-${ordinal}-key.pem"
ca="${src}/root-ca.pem"
test -r "$cert"
test -r "$key"
test -r "$ca"
cp "$cert" /runtime/node.pem
cp "$key" /runtime/node-key.pem
cp "$ca" /runtime/root-ca.pem
chown 1000:1000 /runtime/node.pem /runtime/node-key.pem /runtime/root-ca.pem
chmod 0644 /runtime/node.pem /runtime/root-ca.pem
chmod 0600 /runtime/node-key.pem
ls -l /runtime
volumeMounts:
- name: node-0-source
mountPath: /source/node-0
readOnly: true
- name: node-1-source
mountPath: /source/node-1
readOnly: true
- name: node-2-source
mountPath: /source/node-2
readOnly: true
- name: runtime-certs
mountPath: /runtime
Important
The
chown 1000:1000matches the documented chart runtime user. Verify the actual image UID/GID before copying this into a different image or chart.
kubectl exec -n "$NS" "${OS_STS}-0" -c opensearch -- id
The init container and main container mount the same emptyDir at different paths:
init container: /runtime/node.pem
main container: /usr/share/opensearch/config/certs/node.pem
These are the same underlying file.
8.5 OpenSearch TLS configuration
Use fixed runtime paths:
config:
opensearch.yml: |
cluster.name: brilliant-opensearch-cluster
network.host: 0.0.0.0
plugins.security.ssl.transport.pemcert_filepath: certs/node.pem
plugins.security.ssl.transport.pemkey_filepath: certs/node-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: certs/root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: true
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: certs/node.pem
plugins.security.ssl.http.pemkey_filepath: certs/node-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: certs/root-ca.pem
plugins.security.nodes_dn:
- 'CN=opensearch-master-0,OU=OpenSearch Nodes,O=Brilliant,L=Dhaka,ST=Dhaka,C=BD'
- 'CN=opensearch-master-1,OU=OpenSearch Nodes,O=Brilliant,L=Dhaka,ST=Dhaka,C=BD'
- 'CN=opensearch-master-2,OU=OpenSearch Nodes,O=Brilliant,L=Dhaka,ST=Dhaka,C=BD'
plugins.security.authcz.admin_dn:
- 'CN=opensearch-admin,OU=OpenSearch Administrators,O=Brilliant,L=Dhaka,ST=Dhaka,C=BD'
Use the actual RFC2253 certificate DNs from openssl x509, not guessed strings.
8.6 Temporary transport hostname-verification workaround
If logs show:
No subject alternative DNS name matching brilliant-opensearch-cluster-master-headless found
then the certificate SAN is wrong for the DNS name used by transport communication.
Temporary diagnostic setting:
plugins.security.ssl.transport.enforce_hostname_verification: false
This is not the desired production state. Regenerate node certificates with correct SANs, deploy them, verify transport TLS, then restore:
plugins.security.ssl.transport.enforce_hostname_verification: true
9. Render, Validate, and Deploy OpenSearch
Use this order every time:
values.yaml
-> helm lint
-> helm template
-> inspect rendered YAML
-> Kubernetes server-side dry-run
-> helm upgrade/install
-> runtime validation
9.1 Helm lint
helm lint "$OS_CHART" -f values.yaml
9.2 Render
helm template "$OS_RELEASE" "$OS_CHART" \
-n "$NS" \
-f values.yaml \
> rendered-opensearch.yaml
9.3 Inspect rendered manifest
grep -nE 'kind: StatefulSet|replicas:|storageClassName:|120Gi|runtime-certs|node-[012]-source|select-node-certificate|nodes_dn|admin_dn|9200|9300' rendered-opensearch.yaml
Verify:
[ ] replicas = 3
[ ] image/version pinned
[ ] requests/limits present
[ ] 120Gi Longhorn PVC template
[ ] hard anti-affinity
[ ] expected toleration / affinity
[ ] runtime-certs emptyDir
[ ] init container selects certificate by ordinal
[ ] main container mounts runtime-certs only
[ ] node DNs and admin DN correct
[ ] REST port 9200
[ ] transport port 9300
[ ] bootstrap Secret reference present
9.4 Kubernetes API dry-run
kubectl apply --dry-run=server -f rendered-opensearch.yaml
9.5 Deploy
helm upgrade --install "$OS_RELEASE" "$OS_CHART" \
-n "$NS" \
-f values.yaml
9.6 Observe rollout
kubectl get pods -n "$NS" -w
In another terminal:
kubectl rollout status statefulset/"$OS_STS" -n "$NS"
10. OpenSearch Security Initialization and Admin Password
10.1 Important distinction
These are not the same thing:
Kubernetes Secret / OPENSEARCH_INITIAL_ADMIN_PASSWORD
!=
active OpenSearch Security index
For new demo-security initialization, OPENSEARCH_INITIAL_ADMIN_PASSWORD is used during setup. Once the Security configuration is initialized, changing an environment variable or mounted YAML file does not automatically rewrite the active security index.
10.2 Inspect effective internal user configuration
kubectl exec -n "$NS" "${OS_STS}-0" -c opensearch -- \
grep -A8 '^admin:' \
/usr/share/opensearch/config/opensearch-security/internal_users.yml
If the mounted file still contains a demo hash, it does not prove the intended password is active.
10.3 Generate a bcrypt hash
kubectl exec -it -n "$NS" "${OS_STS}-0" -c opensearch -- \
/usr/share/opensearch/plugins/opensearch-security/tools/hash.sh \
-p '<STRONG_NEW_PASSWORD>'
Avoid placing real passwords in shell history. Prefer an interactive/secure workflow where possible.
10.4 Create custom internal_users.yml
---
_meta:
type: internalusers
config_version: 2
admin:
hash: "$2y$12$<GENERATED_BCRYPT_HASH>"
reserved: true
backend_roles:
- admin
description: OpenSearch administrator
10.5 Critical Secret key name
The chart expects the exact key:
internal_users.yml
Not:
internal_users.yaml
Create/update correctly even if your local filename is .yaml:
kubectl create secret generic opensearch-internal-users \
-n "$NS" \
--from-file=internal_users.yml=./internal_users.yaml \
--dry-run=client -o yaml | kubectl apply -f -
Verify only the key names:
kubectl get secret opensearch-internal-users \
-n "$NS" \
-o json | jq -r '.data | keys[]'
Expected:
internal_users.yml
10.6 Helm security configuration
securityConfig:
enabled: true
internalUsersSecret: opensearch-internal-users
Do not duplicate securityConfig: blocks.
10.7 Admin certificate mount
The admin certificate should be separate from normal node runtime TLS material.
extraVolumes:
# existing volumes ...
- name: admin-certs
secret:
secretName: opensearch-admin-cert
defaultMode: 0440
extraVolumeMounts:
# existing mounts ...
- name: admin-certs
mountPath: /usr/share/opensearch/config/admin-certs
readOnly: true
With the chart's fsGroup: 1000, mode 0440 allows the non-root OpenSearch process to read these files through the pod group. Verify the effective UID/GID before changing this mode. A stricter production design is to mount the admin key only into a temporary administrative Job/Pod rather than every OpenSearch runtime pod.
10.8 Apply only internal users with securityadmin.sh
After the cluster is healthy and the admin certificate is trusted:
kubectl exec -it -n "$NS" "${OS_STS}-0" -c opensearch -- \
/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
-f /usr/share/opensearch/config/opensearch-security/internal_users.yml \
-t internalusers \
-icl \
-nhnv \
-cacert /usr/share/opensearch/config/admin-certs/root-ca.pem \
-cert /usr/share/opensearch/config/admin-certs/admin.pem \
-key /usr/share/opensearch/config/admin-certs/admin-key.pem
Using -f + -t internalusers is safer than unintentionally overwriting the complete Security configuration.
11. OpenSearch Cluster Verification
11.1 Kubernetes state
kubectl get sts -n "$NS"
kubectl get pods -n "$NS" -o wide
kubectl get pvc -n "$NS"
kubectl get svc -n "$NS"
Expected:
3 OpenSearch pods
3 Bound PVCs
1 PVC per pod
pods spread across intended nodes
ClusterIP Service present
headless Service present
11.2 Init container logs
kubectl logs -n "$NS" "${OS_STS}-0" -c select-node-certificate
kubectl logs -n "$NS" "${OS_STS}-1" -c select-node-certificate
kubectl logs -n "$NS" "${OS_STS}-2" -c select-node-certificate
11.3 Runtime certificate files
kubectl exec -n "$NS" "${OS_STS}-0" -c opensearch -- \
ls -l /usr/share/opensearch/config/certs
Expected:
node.pem
node-key.pem
root-ca.pem
11.4 OpenSearch logs
kubectl logs -n "$NS" "${OS_STS}-0" -c opensearch --tail=200
If a container restarted:
kubectl logs -n "$NS" "${OS_STS}-0" -c opensearch --previous --tail=200
Correct flag:
--previous
not:
--privious
11.5 Local diagnostic port-forward
kubectl port-forward -n "$NS" svc/"$OS_SVC" 9200:9200
Because the certificate may not contain localhost, curl -k is acceptable only as a local diagnostic:
curl -k -u 'admin:<PASSWORD>' \
'https://localhost:9200/_cluster/health?pretty'
curl -k -u 'admin:<PASSWORD>' \
'https://localhost:9200/_cat/nodes?v'
Do not interpret -k as production TLS validation.
11.6 Cluster health meaning
GREEN -> all primary and replica shards allocated
YELLOW -> all primaries allocated; at least one replica unassigned
RED -> at least one primary shard unavailable
11.7 Replica count
OpenSearch defaults to one replica unless changed. A 3-node cluster can use two replicas to keep three shard copies, but this is a capacity/durability choice, not an automatic requirement.
Before setting number_of_replicas: 2, account for storage, indexing cost, and failure-domain design.
12. OpenSearch Dashboards Deployment
12.1 Architecture
Browser
|
| HTTP :31599 /opensearch
v
NGINX Ingress
|
v
OpenSearch Dashboards Service :5601
|
v
Dashboards pod
|
| HTTPS :9200 + CA + hostname validation
v
OpenSearch Service
12.2 Verify OpenSearch Service certificate SAN
The hostname used by Dashboards must exist in the OpenSearch certificate SAN:
brilliant-opensearch-cluster-master
A failure against https://localhost:9200 such as:
no alternative certificate subject name matches target hostname 'localhost'
means CA trust may be correct but hostname verification failed.
12.3 Root CA Secret for Dashboards
kubectl create secret generic opensearch-dashboard-ca \
-n "$NS" \
--from-file=root-ca.pem \
--dry-run=client -o yaml | kubectl apply -f -
12.4 kibanaserver password Secret
read -rsp 'kibanaserver password: ' KIBANASERVER_PASSWORD
echo
kubectl create secret generic opensearch-kibanaserver-password \
-n "$NS" \
--from-literal=KIBANASERVER_PASSWORD="$KIBANASERVER_PASSWORD" \
--dry-run=client -o yaml | kubectl apply -f -
unset KIBANASERVER_PASSWORD
The Secret object name and the Secret key are different concepts:
secretKeyRef.name -> opensearch-kibanaserver-password
secretKeyRef.key -> KIBANASERVER_PASSWORD
12.5 Dashboards Helm values
opensearchHosts: "https://brilliant-opensearch-cluster-master:9200"
extraEnvs:
- name: KIBANASERVER_PASSWORD
valueFrom:
secretKeyRef:
name: opensearch-kibanaserver-password
key: KIBANASERVER_PASSWORD
extraVolumes:
- name: root-cert
secret:
secretName: opensearch-dashboard-ca
extraVolumeMounts:
- name: root-cert
readOnly: true
mountPath: /usr/share/opensearch-dashboards/certs
config:
opensearch_dashboards.yml: |
server:
name: dashboards
host: "0.0.0.0"
basePath: /opensearch
rewriteBasePath: true
opensearch:
username: kibanaserver
password: ${KIBANASERVER_PASSWORD}
ssl:
verificationMode: full
certificateAuthorities:
- /usr/share/opensearch-dashboards/certs/root-ca.pem
opensearch_security.multitenancy.enabled: false
data_source:
enabled: true
workspace:
enabled: true
Optional feature keys such as explore and datasetManagement must be checked against the exact OpenSearch Dashboards version before enabling them.
12.6 Workspaces and multi-tenancy
When using Workspaces together with the OpenSearch Security plugin:
opensearch_security.multitenancy.enabled: false
is required to avoid feature conflict.
12.7 server.publicBaseUrl
In the tested OpenSearch Dashboards 3.7.0 deployment, adding server.publicBaseUrl caused startup validation failure:
"publicBaseUrl" is not allowed
Therefore it is intentionally absent from this SOP.
12.8 Render Dashboards before deploy
helm template brilliant-opensearch-dashboard <DASHBOARDS_CHART_PATH> \
-n "$NS" \
-f dashboards-values.yaml \
> rendered-dashboards.yaml
Inspect:
grep -nE 'KIBANASERVER_PASSWORD|root-cert|basePath|rewriteBasePath|verificationMode|5601' rendered-dashboards.yaml
12.9 Ingress conflict check
Before creating a hostless path, inspect existing Ingress objects:
kubectl get ingress -A
Grafana already used hostless / in the implementation. A second hostless / route under the same NGINX controller conflicted even though it was in another namespace.
Use /opensearch instead.
12.10 Dashboards Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: opensearch-dashboards-ingress
namespace: observability
spec:
ingressClassName: nginx
rules:
- http:
paths:
- path: /opensearch
pathType: Prefix
backend:
service:
name: brilliant-opensearch-dashboard-opensearch-dashboards
port:
number: 5601
Apply:
kubectl apply -f opensearch-dashboards-ingress.yaml
12.11 Runtime verification
kubectl get deploy,pods,svc,ingress -n "$NS"
Current logs:
kubectl logs -n "$NS" deploy/brilliant-opensearch-dashboard-opensearch-dashboards --tail=200
Previous container logs after a restart:
kubectl logs -n "$NS" deploy/brilliant-opensearch-dashboard-opensearch-dashboards --previous --tail=200
Expected external URL pattern:
http://<KUBERNETES-NODE-IP>:31599/opensearch
12.12 401 -> 404 diagnostic meaning
During troubleshooting:
unauthenticated /opensearch -> 401 Authentication required
valid authentication -> 404 Not Found
This indicated traffic was reaching Dashboards and authentication worked, while Dashboards did not yet recognize /opensearch as its base path. server.basePath + rewriteBasePath resolved that routing layer.
13. Data Prepper — Full Deployment, Routing, Parsing, and Verification
This chapter is the canonical Data Prepper procedure for this environment. It consolidates the Data Prepper material from the cPanel ingestion, multi-source routing, and Linux Audit SOPs.
13.1 Role in the architecture
Fluent Bit
|
| HTTP POST /logs
v
NGINX Ingress / NodePort
|
v
Data Prepper HTTP source :2021
|
v
Router pipeline
|
+--> cPanel access pipeline
+--> system syslog pipeline
+--> Linux audit pipeline
+--> catch-all / unknown pipeline
|
v
OpenSearch HTTPS :9200
Data Prepper is responsible for receiving events, routing them by log_source, parsing/normalizing selected formats, and writing documents to the correct OpenSearch index.
13.2 Environment values used in this implementation
| Setting | Value used |
|---|---|
| Data Prepper version | 2.8.0 |
| Namespace | observability |
| Helm release | data-prepper |
| HTTP source port | 2021 |
| Server API / metrics port | 4900 |
| HTTP path | /logs |
| OpenSearch endpoint | https://brilliant-opensearch-cluster-master:9200 |
| Root CA path inside Data Prepper | /usr/share/data-prepper/cert/root-ca.pem |
| cPanel access index | cpanel-access-log-%{yyyy.MM.dd} |
| Syslog index | cpanel-system-syslog-%{yyyy.MM.dd} |
| Audit index | cpanel-audit-log-%{yyyy.MM.dd} |
| Catch-all index | cpanel-all-log-%{yyyy.MM.dd} |
Verify the actual release and namespace before modifying anything:
helm list -n "$NS"
kubectl get deploy,svc,pod -n "$NS" | grep -i data-prepper
Expected Helm release:
data-prepper
Warning
One troubleshooting incident installed/upgraded a second
data-prepperrelease in thedefaultnamespace. Always specify-n observability/--namespace observability.
13.3 Back up the current Data Prepper configuration before changes
mkdir -p ~/backup/data-prepper
helm get values data-prepper -n "$NS" -a \
> ~/backup/data-prepper/values-$(date +%F-%H%M%S).yaml
helm get manifest data-prepper -n "$NS" \
> ~/backup/data-prepper/manifest-$(date +%F-%H%M%S).yaml
Also export the currently running pipeline/config objects if they are stored in ConfigMaps or Secrets:
kubectl get configmap -n "$NS" -o name | grep -i prepper
kubectl get secret -n "$NS" -o name | grep -i prepper
Do not dump Secret data into Git, screenshots, or terminal transcripts.
13.4 One HTTP source per listener port
Only one pipeline in a Data Prepper process should own HTTP port 2021.
Incorrect design:
access pipeline -> HTTP :2021
syslog pipeline -> HTTP :2021
audit pipeline -> HTTP :2021
Typical failure:
bind(..) failed: Address already in use
Correct design:
ONE HTTP source :2021
|
v
router pipeline
|
+--> access pipeline
+--> syslog pipeline
+--> audit pipeline
+--> catch-all pipeline
Downstream pipelines use a Data Prepper pipeline source, not another HTTP source.
13.5 Complete router pipeline
The normalized log_source field created by Fluent Bit is the routing key.
version: "2"
cpanel-log-router:
source:
http:
path: /logs
ssl: false
port: 2021
health_check_service: true
unauthenticated_health_check: true
route:
- accessLog: '/log_source == "cpanel.access_log"'
- systemSyslog: '/log_source == "system.syslog"'
- auditLog: '/log_source == "audit.audit.log"'
- unhandled: '/log_source != "cpanel.access_log" and /log_source != "system.syslog" and /log_source != "audit.audit.log"'
sink:
- pipeline:
name: cpanel-access-pipeline
routes:
- accessLog
- pipeline:
name: cpanel-system-syslog-pipeline
routes:
- systemSyslog
- pipeline:
name: cpanel-audit-pipeline
routes:
- auditLog
- pipeline:
name: cpanel-catch-all-pipeline
routes:
- unhandled
Why the unhandled route is explicit
A pipeline sink without a routes: restriction can receive all events. If a catch-all sink is left unqualified, dedicated events may also be copied into the catch-all path. The explicit unhandled condition prevents that duplication.
13.6 Complete cPanel access pipeline
The tested raw access format is:
20.100.201.54 proxy - [08/09/2026:07:37:27 -0000] "GET /wp-content/themes/twentytwenty/404.php HTTP/1.1" 301 0 "-" "-" "-" "X-Forwarded-For: 20.100.201.54" 80
Pipeline:
cpanel-access-pipeline:
workers: 2
delay: 100
source:
pipeline:
name: cpanel-log-router
processor:
- drop_events:
drop_when: '/log_source != "cpanel.access_log"'
- grok:
break_on_match: false
match:
log:
- '^%{IP:client_ip} %{NOTSPACE:identity} %{NOTSPACE:user} \[%{GREEDYDATA:timestamp}\] "%{WORD:method} %{URIPATHPARAM:url} HTTP/%{NUMBER:http_version}" %{INT:http_status:int} %{NONNEGINT:bytes:long} "%{NOTSPACE:referrer}" "%{GREEDYDATA:user_agent}" "%{NOTSPACE:auth_method}" "%{GREEDYDATA:x_forwarded_for}" %{POSINT:service_port:int}$'
target_key: parsed
- date:
match:
- key: /parsed/timestamp
patterns:
- "MM/dd/yyyy:HH:mm:ss Z"
destination: "@timestamp"
sink:
- opensearch:
hosts:
- "https://brilliant-opensearch-cluster-master:9200"
insecure: false
username: logger
password: "<SECRET-MANAGED-VALUE>"
cert: /usr/share/data-prepper/cert/root-ca.pem
index_type: custom
index: cpanel-access-log-%{yyyy.MM.dd}
Expected parsed fields:
parsed.client_ip
parsed.identity
parsed.user
parsed.timestamp
parsed.method
parsed.url
parsed.http_version
parsed.http_status
parsed.bytes
parsed.referrer
parsed.user_agent
parsed.auth_method
parsed.x_forwarded_for
parsed.service_port
@timestamp
Important Grok compatibility issue from the implementation
The tested Data Prepper 2.8.0 setup failed when this was used:
%{PORT:service_port}
with:
No definition for key 'PORT' found, aborting
The working replacement was:
%{POSINT:service_port}
13.7 Syslog pipeline
Syslog records observed in the environment included both:
2026-08-11T10:42:39.679625+00:00 bwebhosting kernel: Firewall: ...
and:
2026-08-11T10:42:39.434068+00:00 bwebhosting fluent-bit[1241036]: ...
The process token therefore needs to tolerate both a plain process name and a process name with an optional PID.
A safe baseline is to preserve raw data while the exact parser is being tuned:
cpanel-system-syslog-pipeline:
source:
pipeline:
name: cpanel-log-router
processor:
- drop_events:
drop_when: '/log_source != "system.syslog"'
sink:
- opensearch:
hosts:
- "https://brilliant-opensearch-cluster-master:9200"
insecure: false
username: logger
password: "<SECRET-MANAGED-VALUE>"
cert: /usr/share/data-prepper/cert/root-ca.pem
index_type: custom
index: cpanel-system-syslog-%{yyyy.MM.dd}
Add Grok only after representative samples are verified. If a document reaches OpenSearch with parsed: {}, transport/routing may be healthy while the Grok expression is not matching.
13.8 Linux Audit pipeline
Audit records are multi-type and are handled in the dedicated Linux Audit chapter of this SOP. The router must send only:
log_source = audit.audit.log
to:
cpanel-audit-pipeline
The audit pipeline should additionally protect itself with:
- drop_events:
drop_when: '/log_source != "audit.audit.log"'
The full Audit Grok, date, EXECVE, SYSCALL, LOGIN, USER_ACCT, USER_START, CRED_ACQ, and CWD parsing procedure is documented in Section 17.
13.9 Catch-all pipeline
Use the catch-all to retain unknown/new sources without forcing every source into a dedicated parser on day one.
cpanel-catch-all-pipeline:
source:
pipeline:
name: cpanel-log-router
sink:
- opensearch:
hosts:
- "https://brilliant-opensearch-cluster-master:9200"
insecure: false
username: logger
password: "<SECRET-MANAGED-VALUE>"
cert: /usr/share/data-prepper/cert/root-ca.pem
index_type: custom
index: cpanel-all-log-%{yyyy.MM.dd}
Recommended model:
Known/high-value source
-> dedicated route
-> dedicated parser
-> dedicated index
Unknown/new source
-> catch-all
-> raw visibility
-> inspect
-> promote later if required
13.10 OpenSearch Root CA mount
Data Prepper must trust the CA that signed the OpenSearch HTTP certificate.
Conceptual Kubernetes volume:
volumes:
- name: opensearch-root-ca
secret:
secretName: opensearch-dashboard-ca
Conceptual container mount:
volumeMounts:
- name: opensearch-root-ca
mountPath: /usr/share/data-prepper/cert
readOnly: true
Verify inside the pod:
kubectl exec -n "$NS" deploy/data-prepper -- \
ls -l /usr/share/data-prepper/cert/root-ca.pem
Do not set insecure: true merely to bypass a certificate problem. Correct the CA/SAN/TLS configuration instead.
13.11 Data Prepper credential handling
The source SOP used an OpenSearch user named:
logger
The real password must not be committed to Git.
Required security model:
Data Prepper identity
-> authenticate to OpenSearch
-> only required write/create-index permissions
-> no admin/security-management permissions
If the Helm chart or deployment mechanism stores pipelines.yaml in a Secret, keep the sensitive pipeline configuration in that Secret or use the secret-management mechanism supported by the deployment. Do not assume arbitrary environment-variable interpolation in every Data Prepper configuration field without testing the exact deployed version.
13.12 Kubernetes Service
Expected Service path:
Ingress
-> data-prepper Service :2021
-> Data Prepper Pod :2021
Service example:
apiVersion: v1
kind: Service
metadata:
name: data-prepper
namespace: observability
spec:
type: ClusterIP
selector:
app: data-prepper
ports:
- name: ingest
port: 2021
targetPort: 2021
- name: metrics
port: 4900
targetPort: 4900
The selector is chart/deployment-specific. Verify that it matches the actual pod labels:
kubectl get pod -n "$NS" --show-labels | grep data-prepper
kubectl get svc data-prepper -n "$NS" -o yaml
Verify the backend endpoint:
kubectl get endpointslice \
-n "$NS" \
-l kubernetes.io/service-name=data-prepper \
-o wide
Expected result: the current Data Prepper pod IP and port 2021 are present.
13.13 NGINX Ingress
The implementation used:
http://118.67.213.76:31599/logs
Flow:
118.67.213.76:31599
-> ingress-nginx controller :80
-> prepper-ingress /logs
-> data-prepper:2021
Ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: prepper-ingress
namespace: observability
spec:
ingressClassName: nginx
rules:
- http:
paths:
- path: /logs
pathType: Prefix
backend:
service:
name: data-prepper
port:
number: 2021
Apply and verify:
kubectl apply -f prepper-ingress.yaml
kubectl get ingress prepper-ingress -n "$NS" -o wide
kubectl describe ingress prepper-ingress -n "$NS"
13.14 Data Prepper Helm upgrade procedure
The exact chart path depends on how Data Prepper was installed. Use the real chart reference/path instead of guessing.
export DP_RELEASE="data-prepper"
export DP_CHART="<DATA_PREPPER_CHART_OR_LOCAL_PATH>"
export DP_VALUES="<DATA_PREPPER_VALUES_FILE>"
Confirm the current release first:
helm list -n "$NS"
Render before upgrading:
helm template "$DP_RELEASE" "$DP_CHART" \
-n "$NS" \
-f "$DP_VALUES" \
> /tmp/data-prepper-rendered.yaml
Inspect the important items:
grep -nE 'kind: (Deployment|Service|ConfigMap|Secret)|2021|4900|pipelines|data-prepper' \
/tmp/data-prepper-rendered.yaml | head -200
Then upgrade the correct release in the correct namespace:
helm upgrade "$DP_RELEASE" "$DP_CHART" \
-n "$NS" \
-f "$DP_VALUES"
Watch rollout:
kubectl get pods -n "$NS" -w
13.15 Pod and listener verification
kubectl get pods -n "$NS" | grep data-prepper
kubectl logs -n "$NS" deploy/data-prepper --tail=200
Expected health:
READY: 1/1
STATUS: Running
If ss/netstat do not exist in the image, /proc/net/tcp can verify the listener.
For port 2021:
2021 decimal = 07E5 hexadecimal
0A = LISTEN
01 = ESTABLISHED
Example:
kubectl exec -n "$NS" deploy/data-prepper -- \
sh -c 'grep -i ":07E5" /proc/net/tcp /proc/net/tcp6 || true'
13.16 Health endpoint and metrics
When the HTTP source health service is enabled:
kubectl run dp-curl --rm -i --restart=Never \
-n "$NS" \
--image=curlimages/curl:latest \
-- \
curl -sS http://data-prepper:2021/health
Data Prepper server/metrics API:
kubectl exec -n "$NS" deploy/data-prepper -- \
curl -s http://127.0.0.1:4900/metrics/prometheus | head -100
Useful metric families from the audit/cPanel work include:
grokProcessingMatch
grokProcessingMismatch
grokProcessingErrors
dateProcessingMatchSuccess
dateProcessingMatchFailure
documentsSuccess
documentErrors
Pipeline/plugin names can be normalized into the full metric name.
13.17 Controlled event test inside Kubernetes
The HTTP source accepts JSON events. Use a controlled event before involving Fluent Bit:
kubectl run dp-curl --rm -i --restart=Never \
-n "$NS" \
--image=curlimages/curl:latest \
-- \
curl -sS -X POST http://data-prepper:2021/logs \
-H 'Content-Type: application/json' \
--data '[{"log":"test","source_file":"manual-test","log_source":"test.manual"}]'
Then verify that the catch-all path receives it.
13.18 Controlled event test through NGINX
From a host that can reach the NGINX NodePort:
curl -i -X POST \
'http://118.67.213.76:31599/logs' \
-H 'Content-Type: application/json' \
--data '[{"log":"ingress-test","source_file":"manual-test","log_source":"test.manual"}]'
A successful test proves:
client
-> NodePort
-> NGINX
-> Ingress rule
-> Service
-> EndpointSlice
-> Data Prepper :2021
13.19 OpenSearch sink verification
Do not use index existence alone as proof of successful ingestion.
Example:
green open cpanel-access-log-2026.08.09 ... docs.count=0
means the index exists but has no documents.
Check index list:
curl --cacert <ROOT_CA> -u '<USER>:<PASSWORD>' \
'https://<OPENSEARCH>:9200/_cat/indices/cpanel-*?v'
Check document count:
curl --cacert <ROOT_CA> -u '<USER>:<PASSWORD>' \
'https://<OPENSEARCH>:9200/cpanel-access-log-*/_count?pretty'
Check parsed fields:
curl --cacert <ROOT_CA> -u '<USER>:<PASSWORD>' \
-H 'Content-Type: application/json' \
'https://<OPENSEARCH>:9200/cpanel-access-log-*/_search?pretty' \
-d '{
"size": 5,
"sort": [{"@timestamp": "desc"}],
"_source": ["@timestamp","log_source","source_file","log","parsed"]
}'
Interpretation:
log exists + parsed exists -> parser matched
log exists + parsed missing -> parser did not match / processor issue
index exists + docs.count = 0 -> no successful document insertion
13.20 502 Bad Gateway troubleshooting
Fluent Bit previously reported:
HTTP status=502
502 Bad Gateway
nginx
This means Fluent Bit reached NGINX, but NGINX did not have a working upstream path to Data Prepper.
Check in this order:
kubectl get pod -n "$NS" | grep data-prepper
kubectl get svc data-prepper -n "$NS" -o wide
kubectl get endpointslice -n "$NS" \
-l kubernetes.io/service-name=data-prepper -o wide
kubectl logs -n "$NS" deploy/data-prepper --tail=200
kubectl describe ingress prepper-ingress -n "$NS"
Do not change Fluent Bit Grok/parsing when NGINX is returning a backend/upstream error.
13.21 OpenSearch sink failure troubleshooting
Observed sink errors included:
Failed to initialize OpenSearch sink with a retryable exception
Request execution cancelled
I/O session is closed
Troubleshoot this path independently:
Data Prepper
-> Kubernetes DNS
-> brilliant-opensearch-cluster-master
-> TCP 9200
-> TLS handshake
-> Root CA validation
-> hostname/SAN validation
-> logger authentication
-> logger authorization
-> index write
Do not modify Grok while the failure is DNS/TCP/TLS/authentication/authorization at the sink.
13.22 YAML and processor hierarchy checks
Correct Grok hierarchy:
processor:
- grok:
break_on_match: false
match:
log:
- '<GROK_PATTERN>'
target_key: parsed
A prior indentation mistake caused Data Prepper to interpret options such as false at the wrong level and fail pipeline construction.
Validate indentation and quoting before every Helm upgrade.
13.23 Production hardening for Data Prepper
The original implementation used:
ssl: false
on the Data Prepper HTTP source and exposed it through an NGINX NodePort. Before considering the ingestion endpoint production-hardened, use the controls appropriate to the environment:
TLS on the ingestion path
HTTP authentication where supported/configured
private network routing
firewall/source-IP restrictions
NetworkPolicy
Ingress authentication / mTLS where appropriate
least-privilege OpenSearch sink identity
persistent DLQ if required
resource limits
metrics and alerting
The endpoint receives security-sensitive data including IP addresses, usernames, URLs, audit records, processes, and commands. Protect access and retention accordingly.
14. Fluent Bit — Full cPanel/Linux Collection Procedure
This chapter is the canonical Fluent Bit procedure for the cPanel/Linux sources in this project. It includes the access-only baseline, the expanded multi-source configuration, source classification, persistent offset state, filesystem buffering, verification, and outage-recovery testing.
14.1 Role in the architecture
cPanel/Linux log files
|
v
Fluent Bit Tail inputs
|
+--> persistent Tail DB
+--> filesystem buffer
+--> source_file
+--> Lua log_source classification
|
v
HTTP POST /logs
|
v
NGINX -> Data Prepper
Fluent Bit must:
- Continuously tail the required log files.
- Preserve the original path as
source_file. - Maintain independent Tail DB state for each input.
- Use filesystem buffering for undelivered chunks.
- Normalize the source into a stable
log_sourcevalue. - Send JSON to Data Prepper.
- Retry transient failures.
- Resume correctly after service/server restarts.
14.2 Source inventory used in the project
| Category | Example path | Expected log_source |
|---|---|---|
| cPanel access | /usr/local/cpanel/logs/access_log |
cpanel.access_log |
| Syslog | /var/log/syslog |
system.syslog |
| Kernel | /var/log/kern.log |
system.kern.log |
| Exim | /var/log/exim_mainlog |
system.exim_mainlog |
| Auditd | /var/log/audit/audit.log |
audit.audit.log |
| Apache | /var/log/apache2/... |
apache.<filename> |
Before configuration, verify which files actually exist on the server:
for f in \
/usr/local/cpanel/logs/access_log \
/var/log/syslog \
/var/log/kern.log \
/var/log/exim_mainlog \
/var/log/audit/audit.log; do
[ -e "$f" ] && ls -lh "$f"
done
Inspect representative records before writing parsers:
tail -n 20 /usr/local/cpanel/logs/access_log
sudo tail -n 20 /var/log/audit/audit.log
14.3 Optional installation verification
The source SOPs started from an already installed Fluent Bit service. If deploying on a new cPanel host, use the package procedure for the actual OS and pin/test the version according to your change policy.
First identify the OS:
cat /etc/os-release
After installation, the service must exist:
systemctl status fluent-bit --no-pager
fluent-bit --version
Do not replace a working production package with a newer one during an ingestion troubleshooting session unless the upgrade itself is planned and tested.
14.4 Service account and permissions
Determine the service account:
systemctl cat fluent-bit | grep -E '^(User|Group)=' || true
ps -eo user,group,pid,cmd | grep '[f]luent-bit'
The Fluent Bit process needs read permission on every selected source file and write permission to the Tail DB/buffer directories.
For audit logs, verify access explicitly:
namei -l /var/log/audit/audit.log
Do not run Fluent Bit as root merely to bypass a permissions problem unless that is the deliberately approved security model.
14.5 Persistent state and buffer directories
Recommended separation:
/var/lib/fluent-bit/
├── state/
│ ├── cpanel-access.db
│ ├── syslog.db
│ ├── kern.db
│ ├── exim.db
│ └── audit.db
└── storage/
└── filesystem chunks
Create the directories using the actual Fluent Bit service account:
install -d -m 0750 -o fluent-bit -g fluent-bit /var/lib/fluent-bit/state
install -d -m 0750 -o fluent-bit -g fluent-bit /var/lib/fluent-bit/storage
If the service uses another user/group, replace fluent-bit:fluent-bit accordingly.
Keep these directories outside any broad Tail wildcard so Fluent Bit does not ingest its own SQLite/WAL or buffer files.
14.6 Access-log-only baseline configuration
Use this first when proving the cPanel access path end to end:
[SERVICE]
Flush 1
Daemon Off
Log_Level info
Parsers_File parsers.conf
Plugins_File plugins.conf
storage.metrics on
storage.path /var/lib/fluent-bit/storage
storage.sync full
storage.checksum on
storage.backlog.mem_limit 500M
[INPUT]
Name tail
Path /usr/local/cpanel/logs/access_log
Path_Key source_file
Tag access_log
DB /var/lib/fluent-bit/state/cpanel-access.db
DB.Journal_Mode wal
Refresh_Interval 5
Skip_Long_Lines true
Mem_Buf_Limit 512MB
Storage.Type filesystem
[FILTER]
Name modify
Match access_log
Add log_source cpanel.access_log
[OUTPUT]
Name http
Host 118.67.213.76
Port 31599
Format json
HTTP_Method POST
URI /logs
Match access_log
Retry_Limit no_limits
Storage.Total_Limit_Size 4G
Notes
DB stores file identity/offset state. It is not the durable event queue.
Storage.Type filesystem allows chunks to be backed by the filesystem storage layer.
Storage.Total_Limit_Size 4G caps the logical filesystem queue for this output. If the queue fills, old queued chunks can be discarded to make room for new data; monitor it.
Retry_Limit no_limits avoids abandoning a failed chunk merely because a finite retry count was reached. This still does not make local disk capacity infinite.
14.7 Historical ingestion behavior
For a file with no existing Tail DB state:
Read_from_Head true
means read from the beginning.
Without it, a new input normally begins at the end of the file and follows new records.
Use historical backfill intentionally. Deleting/replacing a Tail DB and then setting Read_from_Head true can re-ingest already indexed data.
For static backfill testing, use a copy of the source file and a dedicated DB/index whenever possible.
14.8 Multi-source Tail configuration
After the access-only path is proven, expand to the required sources. Each input should have a unique DB.
[SERVICE]
Flush 1
Daemon Off
Log_Level info
Parsers_File parsers.conf
Plugins_File plugins.conf
storage.metrics on
storage.path /var/lib/fluent-bit/storage
storage.sync full
storage.checksum on
storage.backlog.mem_limit 500M
[INPUT]
Name tail
Path /usr/local/cpanel/logs/access_log
Path_Key source_file
Tag cpanel.files
DB /var/lib/fluent-bit/state/cpanel-access.db
DB.Journal_Mode wal
Refresh_Interval 5
Skip_Long_Lines true
Mem_Buf_Limit 512MB
Storage.Type filesystem
[INPUT]
Name tail
Path /var/log/syslog
Path_Key source_file
Tag cpanel.files
DB /var/lib/fluent-bit/state/syslog.db
DB.Journal_Mode wal
Refresh_Interval 5
Skip_Long_Lines true
Mem_Buf_Limit 256MB
Storage.Type filesystem
[INPUT]
Name tail
Path /var/log/kern.log
Path_Key source_file
Tag cpanel.files
DB /var/lib/fluent-bit/state/kern.db
DB.Journal_Mode wal
Refresh_Interval 5
Skip_Long_Lines true
Mem_Buf_Limit 256MB
Storage.Type filesystem
[INPUT]
Name tail
Path /var/log/exim_mainlog
Path_Key source_file
Tag cpanel.files
DB /var/lib/fluent-bit/state/exim.db
DB.Journal_Mode wal
Refresh_Interval 5
Skip_Long_Lines true
Mem_Buf_Limit 256MB
Storage.Type filesystem
[INPUT]
Name tail
Path /var/log/audit/audit.log
Path_Key source_file
Tag cpanel.files
DB /var/lib/fluent-bit/state/audit.db
DB.Journal_Mode wal
Refresh_Interval 5
Skip_Long_Lines true
Mem_Buf_Limit 512MB
Storage.Type filesystem
[INPUT]
Name tail
Path /var/log/apache2/*.log
Path_Key source_file
Tag cpanel.files
DB /var/lib/fluent-bit/state/apache.db
DB.Journal_Mode wal
Refresh_Interval 5
Skip_Long_Lines true
Mem_Buf_Limit 512MB
Storage.Type filesystem
[FILTER]
Name lua
Match cpanel.files
Script /etc/fluent-bit/classify.lua
Call classify
[OUTPUT]
Name http
Host 118.67.213.76
Port 31599
Format json
HTTP_Method POST
URI /logs
Match cpanel.files
Retry_Limit no_limits
Storage.Total_Limit_Size 4G
Only keep input paths that actually exist and are required. If Apache/cPanel stores logs elsewhere on a specific host, use the real path.
14.9 source_file and log_source
A Tail event should preserve its file:
{
"log": "...",
"source_file": "/var/log/syslog"
}
The Lua filter converts that into a stable routing field:
{
"log": "...",
"source_file": "/var/log/syslog",
"log_source": "system.syslog"
}
Data Prepper routes on log_source, not on assumptions derived from the destination index name.
14.10 Lua source-classification script
Create:
/etc/fluent-bit/classify.lua
Content:
function classify(tag, timestamp, record)
local path = record["source_file"] or ""
if path == "/usr/local/cpanel/logs/access_log" then
record["log_source"] = "cpanel.access_log"
elseif path == "/var/log/syslog" then
record["log_source"] = "system.syslog"
elseif path == "/var/log/kern.log" then
record["log_source"] = "system.kern.log"
elseif path == "/var/log/exim_mainlog" then
record["log_source"] = "system.exim_mainlog"
elseif path == "/var/log/audit/audit.log" then
record["log_source"] = "audit.audit.log"
elseif string.match(path, "^/var/log/apache2/") then
local filename = string.match(path, "([^/]+)$") or "unknown"
record["log_source"] = "apache." .. filename
else
record["log_source"] = "unknown"
end
return 1, timestamp, record
end
Set ownership/permissions appropriate to the service:
chown root:root /etc/fluent-bit/classify.lua
chmod 0644 /etc/fluent-bit/classify.lua
The earlier implementation attempted inline Lua in the classic configuration and Fluent Bit failed with:
extra indentation level found
invalid indentation level
Keeping Lua in a separate file avoids that configuration-indentation problem and is easier to test/version.
14.11 Validate configuration before restart
If supported by the installed Fluent Bit version/build:
fluent-bit --dry-run -c /etc/fluent-bit/fluent-bit.conf
If the binary is not in PATH:
command -v fluent-bit
systemctl cat fluent-bit
Then restart:
systemctl restart fluent-bit
systemctl status fluent-bit --no-pager
journalctl -u fluent-bit --since '5 minutes ago' --no-pager
Healthy state should not show sustained:
configuration parser errors
502 Bad Gateway
connection refused
failed to flush
chunk retry exhaustion
14.12 Prove the source is generating data
For cPanel access logs, generate or wait for a request and confirm the file changes:
tail -Fn0 /usr/local/cpanel/logs/access_log
For audit:
sudo tail -Fn0 /var/log/audit/audit.log
If no fresh event exists in the source file, do not troubleshoot Data Prepper/OpenSearch yet.
14.13 Verify Tail DB creation and persistence
ls -lh /var/lib/fluent-bit/state/
If SQLite is installed and Fluent Bit is stopped or the DB can be safely inspected:
sqlite3 /var/lib/fluent-bit/state/cpanel-access.db \
'SELECT id,name,offset,inode FROM in_tail_files;'
The Tail database tells Fluent Bit where it last read each tracked file.
14.14 Verify filesystem buffering
find /var/lib/fluent-bit/storage -maxdepth 2 -type f -ls | head -50
Remember:
Tail DB
-> protects file read position
filesystem storage
-> holds chunks that have been read but are not yet successfully delivered
They are separate mechanisms.
14.15 Verify HTTP output to Data Prepper
Follow Fluent Bit logs:
journalctl -u fluent-bit -f
The target used in this implementation is:
http://118.67.213.76:31599/logs
Test the receiver directly from the cPanel server:
curl -i -X POST \
'http://118.67.213.76:31599/logs' \
-H 'Content-Type: application/json' \
--data '[{"log":"manual-cpanel-host-test","source_file":"manual-test","log_source":"test.manual"}]'
If the manual curl fails, fix network/Ingress/Data Prepper before modifying Fluent Bit parsing or Tail settings.
14.16 Interpretation of common Fluent Bit output failures
502 Bad Gateway
Fluent Bit -> NGINX = reachable
NGINX -> Data Prepper upstream = unhealthy/unreachable
Check Data Prepper pod, Service, EndpointSlice, listener, and Ingress.
connection refused
The target IP/port is reachable at the network level but nothing is accepting the connection, or a firewall is rejecting it.
failed to flush chunk
Inspect the associated HTTP/network error. With filesystem buffering enabled, confirm the queue is growing and not at the storage limit.
configuration parser error
Use the exact reported file/line. Do not assume the systemd unit failure text is the root cause; Fluent Bit's own startup output contains the useful parser error.
14.17 Durable buffering recovery test
Perform this only after normal ingestion is stable.
- Record the current OpenSearch count.
- Make Data Prepper unavailable intentionally.
- Generate several test requests/log records.
- Confirm Fluent Bit reports retries.
- Confirm filesystem backlog activity in
/var/lib/fluent-bit/storage. - Restart Fluent Bit while Data Prepper is still unavailable.
- Confirm the Tail DB and storage directories survive the restart.
- Restore Data Prepper.
- Confirm queued events are forwarded.
- Verify the expected documents appear in OpenSearch.
Useful commands:
systemctl restart fluent-bit
journalctl -u fluent-bit --since '10 minutes ago' --no-pager
find /var/lib/fluent-bit/storage -type f | head
This test validates both offset persistence and undelivered-event buffering.
14.18 Long-line behavior
The configuration uses:
Skip_Long_Lines true
This prevents a single oversized line from stopping monitoring of the entire file, but the oversized event can be skipped.
If large audit/application records matter, test representative maximum line sizes and tune Buffer_Chunk_Size / Buffer_Max_Size rather than assuming the defaults are sufficient.
14.19 Log rotation
Fluent Bit Tail tracks files using file metadata/offset state. Verify rotation behavior for each high-value log:
ls -li /usr/local/cpanel/logs/access_log*
ls -li /var/log/audit/audit.log*
Test that rotation does not cause either permanent loss or unexpected full-file replay. Do not delete Tail DB files as a routine rotation fix.
14.20 Multiple files and inotify limits
If the deployment expands to a large number of Apache/cPanel files, inspect inotify capacity:
sysctl fs.inotify.max_user_watches
sysctl fs.inotify.max_user_instances
Increase them only if monitoring scale requires it and persist the chosen values through the normal OS configuration-management process.
14.21 Security hardening for Fluent Bit
Use the minimum filesystem permissions required to read the configured logs and write its own state/buffer directories.
Protect:
/etc/fluent-bit/fluent-bit.conf
/etc/fluent-bit/classify.lua
/var/lib/fluent-bit/state/
/var/lib/fluent-bit/storage/
The original ingestion route used plain HTTP to the NGINX NodePort. For production, protect the transport with private routing, firewall restrictions, authentication and/or TLS as appropriate to the environment.
Do not store OpenSearch credentials in Fluent Bit when Fluent Bit is intentionally sending only to Data Prepper.
14.22 Fluent Bit acceptance checklist
- Required source files exist and receive new records.
- Fluent Bit service is running.
- No classic-config indentation/parser error remains.
- Each Tail input has a unique persistent DB.
- Tail DB directory is writable.
- Filesystem buffer directory is writable.
Storage.Type filesystemis enabled for protected inputs.source_fileis present.log_sourcematches the source file./usr/local/cpanel/logs/access_logbecomescpanel.access_log./var/log/syslogbecomessystem.syslog./var/log/audit/audit.logbecomesaudit.audit.log.- HTTP destination is the correct NGINX/Data Prepper endpoint.
- URI is
/logs. - Output format is JSON.
- Fluent Bit has no sustained 502/refused/flush failures.
- Restart resumes from stored offsets.
- Data Prepper outage/recovery test has been completed.
- Backlog disk usage is monitored.
15. cPanel Access Log Parsing
15.1 Example raw log
20.100.201.54 proxy - [08/09/2026:07:37:27 -0000] "GET /wp-content/themes/twentytwenty/404.php HTTP/1.1" 301 0 "-" "-" "-" "X-Forwarded-For: 20.100.201.54" 80
15.2 Recommended parser
Use one timestamp capture and let the date processor create @timestamp.
cpanel-access-pipeline:
source:
pipeline:
name: cpanel-log-router
processor:
- grok:
break_on_match: true
match:
log:
- '^%{IP:client_ip} %{NOTSPACE:identity} %{NOTSPACE:user} \[%{GREEDYDATA:timestamp}\] "%{WORD:method} %{URIPATHPARAM:url} HTTP/%{NUMBER:http_version}" %{INT:http_status:int} %{NONNEGINT:bytes:long} "%{NOTSPACE:referrer}" "%{GREEDYDATA:user_agent}" "%{NOTSPACE:auth_method}" "%{GREEDYDATA:x_forwarded_for}" %{POSINT:service_port:int}$'
target_key: parsed
- date:
match:
- key: /parsed/timestamp
patterns:
- "MM/dd/yyyy:HH:mm:ss Z"
destination: "@timestamp"
sink:
- opensearch:
hosts:
- "https://brilliant-opensearch-cluster-master:9200"
insecure: false
username: logger
password: "<SECRET-MANAGED-VALUE>"
cert: /usr/share/data-prepper/cert/root-ca.pem
index_type: custom
index: cpanel-access-log-%{yyyy.MM.dd}
15.3 Parsed fields
parsed.client_ip
parsed.identity
parsed.user
parsed.timestamp
parsed.method
parsed.url
parsed.http_version
parsed.http_status
parsed.bytes
parsed.referrer
parsed.user_agent
parsed.auth_method
parsed.x_forwarded_for
parsed.service_port
@timestamp
15.4 Important Grok issue: %{PORT}
In the tested Data Prepper 2.8.0 environment:
%{PORT:service_port}
failed with:
No definition for key 'PORT' found
Use:
%{POSINT:service_port:int}
instead.
15.5 YAML hierarchy
Correct:
processor:
- grok:
break_on_match: true
match:
log:
- '<PATTERN>'
target_key: parsed
break_on_match, match, and target_key belong under grok:.
15.6 Parser validation test
curl -sS -X POST \
'http://<INGRESS-NODE-IP>:31599/logs' \
-H 'Content-Type: application/json' \
--data '[{
"log":"20.100.201.54 proxy - [08/09/2026:07:37:27 -0000] \"GET /wp-content/themes/twentytwenty/404.php HTTP/1.1\" 301 0 \"-\" \"-\" \"-\" \"X-Forwarded-For: 20.100.201.54\" 80",
"source_file":"/usr/local/cpanel/logs/access_log",
"log_source":"cpanel.access_log"
}]'
Then inspect a newly indexed document.
16. Linux Syslog and Other Log Sources
16.1 Example syslog records
2026-08-11T10:42:39.679625+00:00 bwebhosting kernel: Firewall: ...
2026-08-11T10:42:39.434068+00:00 bwebhosting fluent-bit[1241036]: ...
A parser must tolerate both:
process:
process[pid]:
Do not assume a PID is always present.
16.2 Recommended syslog approach
Start with a small set of real samples from the actual host and build multiple Grok patterns if formats vary.
If a document contains:
{
"parsed": {}
}
then transport and indexing may be working while the Grok pattern failed to match.
16.3 Do not infer source from index name
If a document in cpanel-system-syslog-* contains:
{
"log_source": "system.exim_mainlog"
}
then routing/filtering is wrong even though the index name says syslog.
Verify:
_index
source_file
log_source
log
parsed
@timestamp
16.4 Unknown/catch-all strategy
Use:
cpanel-all-log-YYYY.MM.dd
for sources not yet assigned to a dedicated parser.
This provides a safe discovery path:
unknown log appears
-> stored raw
-> inspect format
-> create dedicated parser if operationally valuable
17. Linux Audit Log Parsing
17.1 Audit event model
A single logical Audit event can create multiple records sharing one event ID:
SYSCALL
EXECVE
CWD
PATH
PROCTITLE
Therefore:
Total Audit documents != total commands executed
Use:
audit.event_id
for correlation.
Use primarily:
audit.log_type = EXECVE
when counting command execution records.
17.2 Audit pipeline source
cpanel-audit-pipeline:
source:
pipeline:
name: cpanel-log-router
A defensive filter may be retained:
processor:
- drop_events:
drop_when: '/log_source != "audit.audit.log"'
17.3 Common Audit header
Typical:
type=EXECVE msg=audit(1787117401.120:11280588): ...
Parser:
- grok:
break_on_match: true
match:
log:
- '^type=%{WORD:log_type} msg=audit\(%{INT:timestamp_seconds}\.%{INT:timestamp_fraction}:%{INT:event_id}\): %{GREEDYDATA:msg}'
target_key: audit
Expected:
{
"audit": {
"log_type": "EXECVE",
"timestamp_seconds": "1787117401",
"timestamp_fraction": "120",
"event_id": "11280588",
"msg": "..."
}
}
17.4 Audit timestamp
- date:
match:
- key: /audit/timestamp_seconds
patterns:
- epoch_second
destination: "@timestamp"
17.5 SYSCALL
Parse only when:
/audit/log_type == "SYSCALL"
Common target fields:
syscall.syscall
syscall.pid
syscall.ppid
syscall.success
syscall.exit_code
syscall.exe
syscall.comm
syscall.AUID
syscall.UID
syscall.terminal
Audit keys may be quoted or (null), so maintain multiple patterns if necessary.
17.6 EXECVE
Stage 1 — preserve argument count and raw argument text
- grok:
break_on_match: true
grok_when: '/audit/log_type == "EXECVE"'
target_key: execve
match:
/audit/msg:
- '^argc=%{INT:arg_count} %{GREEDYDATA:args_raw}$'
Stage 2 — optional key/value split
- key_value:
key_value_when: '/audit/log_type == "EXECVE"'
source: /execve/args_raw
destination: /execve/args
field_delimiter_regex: '\s+'
value_split_characters: "="
delete_value_regex: '^"|"$'
Do not configure both field_delimiter_regex and a nonempty field_split_characters; these options are mutually exclusive.
Known limitation
Splitting on whitespace can break an argument that contains literal spaces inside quotes, for example:
a1="hello world"
Therefore always preserve:
execve.args_raw
and do not treat execve.args.* as a lossless reconstruction of every possible command line. Test representative Audit samples before relying on argument-level fields for security logic.
17.7 Mapping-growth limitation
Dynamic fields such as:
execve.args.a0
execve.args.a1
execve.args.a2
...
can create field-count growth.
Prefer long-term normalized fields such as:
execve.command
execve.arg_count
execve.args_raw
and limit dynamic argument mappings.
17.8 CWD
- grok:
break_on_match: true
grok_when: '/audit/log_type == "CWD"'
target_key: cwd
match:
/audit/msg:
- '^cwd="%{GREEDYDATA:path}"'
17.9 USER_ACCT, USER_START, CRED_ACQ
For each type, keep the full parser chain consistent:
grok_when
-> target_key
-> key_value_when
-> source
-> destination
Example nested key/value processing:
- key_value:
key_value_when: '/audit/log_type == "USER_ACCT"'
source: /user_account/msg
destination: /user_account/parsed_msg
field_split_characters: " "
value_split_characters: "="
delete_value_regex: '^"|"$'
Known limitation: simple space splitting can break values such as:
acct="John Doe"
Preserve the raw message.
17.10 LOGIN
Some Audit LOGIN records contain the control separator \u001d before human-readable identity fields. Patterns should tolerate arbitrary content between result and UID= instead of assuming normal whitespace only.
17.11 Audit parser verification
Count total EXECVE:
curl -k -u 'USER:PASSWORD' \
-H 'Content-Type: application/json' \
'https://OPENSEARCH:9200/cpanel-audit-log-YYYY.MM.DD/_count' \
-d '{
"query": {
"match": {
"audit.log_type": "EXECVE"
}
}
}'
Count parsed EXECVE:
curl -k -u 'USER:PASSWORD' \
-H 'Content-Type: application/json' \
'https://OPENSEARCH:9200/cpanel-audit-log-YYYY.MM.DD/_count' \
-d '{
"query": {
"bool": {
"filter": [
{"match": {"audit.log_type": "EXECVE"}},
{"exists": {"field": "execve.arg_count"}}
]
}
}
}'
Ideal validation target:
Total EXECVE == parsed EXECVE
Investigate mismatches with raw documents rather than only aggregate Grok metrics.
18. OpenSearch Index Templates and Mapping
18.1 Index model
cpanel-access-log-* -> parsed cPanel HTTP access logs
cpanel-system-syslog-* -> system syslog
cpanel-audit-log-* -> Linux Audit records
cpanel-all-log-* -> unknown/unhandled raw logs
18.2 cPanel access mappings
Critical types:
@timestamp -> date
parsed.client_ip -> ip
parsed.http_status -> integer
parsed.bytes -> long
parsed.service_port -> integer
Template example:
{
"index_patterns": ["cpanel-access-log-*"],
"template": {
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"parsed": {
"properties": {
"client_ip": {
"type": "ip"
},
"http_status": {
"type": "integer"
},
"bytes": {
"type": "long"
},
"service_port": {
"type": "integer"
}
}
}
}
}
}
}
Apply with Dev Tools or API:
PUT _index_template/cpanel-access-log-template
{
"index_patterns": ["cpanel-access-log-*"],
"template": {
"mappings": {
"properties": {
"@timestamp": {"type": "date"},
"parsed": {
"properties": {
"client_ip": {"type": "ip"},
"http_status": {"type": "integer"},
"bytes": {"type": "long"},
"service_port": {"type": "integer"}
}
}
}
}
}
}
18.3 Verify mapping
GET cpanel-access-log-*/_field_caps?fields=@timestamp,parsed.client_ip,parsed.http_status,parsed.bytes,parsed.service_port
18.4 Audit default search fields
A large Audit mapping can cause:
too_many_nested_clauses
maxClauseCount is set to 1024
Restrict default query fields:
PUT cpanel-audit-log-*/_settings
{
"index.query.default_field": [
"log",
"audit.log_type",
"audit.event_id",
"syscall.exe",
"syscall.comm",
"syscall.AUID",
"execve.args.a0"
]
}
Add the same setting to the matching index template so future daily indexes inherit it.
Do not immediately increase the Boolean clause limit. Control field growth and default-field expansion first.
18.5 Replica policy
Set shard/replica counts in an index template deliberately. Do not assume Helm pod replicas automatically configure index replicas.
19. End-to-End Verification
Always test left-to-right.
19.1 Source file
tail -n 5 /usr/local/cpanel/logs/access_log
stat /usr/local/cpanel/logs/access_log
19.2 Fluent Bit
systemctl status fluent-bit --no-pager
journalctl -u fluent-bit --since '10 minutes ago' --no-pager
Check Tail DB exists:
ls -lh /var/lib/fluent-bit/state/
Check buffer path:
du -sh /var/lib/fluent-bit/storage/
19.3 Ingress
From the source server:
curl -i -X POST \
'http://118.67.213.76:31599/logs' \
-H 'Content-Type: application/json' \
--data '[{"log":"manual ingress test","source_file":"manual","log_source":"test.manual"}]'
19.4 Data Prepper pod
kubectl get pods -n "$NS" -l app=data-prepper -o wide
kubectl logs -n "$NS" deploy/data-prepper --tail=200
19.5 Service and endpoints
kubectl get svc data-prepper -n "$NS" -o wide
kubectl get endpointslice -n "$NS" \
-l kubernetes.io/service-name=data-prepper -o wide
19.6 Port 2021 listener
If the image lacks ss and netstat, inspect /proc/net/tcp*.
2021 decimal = 07E5 hexadecimal
TCP state 0A = LISTEN
TCP state 01 = ESTABLISHED
Example:
kubectl exec -n "$NS" deploy/data-prepper -- \
sh -c 'grep -i ":07E5" /proc/net/tcp /proc/net/tcp6 || true'
19.7 OpenSearch indexes
GET _cat/indices/cpanel-*?v
Remember:
index exists + docs.count=0
does not prove successful ingestion.
19.8 Recent documents
GET cpanel-access-log-*/_search
{
"size": 5,
"sort": [
{"@timestamp": "desc"}
],
"_source": [
"@timestamp",
"log_source",
"source_file",
"log",
"parsed"
]
}
19.9 Last 24 hours count
GET cpanel-access-log-*/_count
{
"query": {
"range": {
"@timestamp": {
"gte": "now-24h",
"lte": "now"
}
}
}
}
This proves:
index wildcard works
+ data exists
+ @timestamp is queryable
+ time-range query works
20. OpenSearch Dashboards Data Source and Workspace
20.1 Enable data sources
data_source:
enabled: true
20.2 Workspaces
workspace:
enabled: true
opensearch_security.multitenancy.enabled: false
20.3 Data source encryption master key
The source SOP configured a data-source credential encryption master key on OpenSearch nodes.
Requirements:
same key on every OpenSearch node
valid supported key length
never commit the key to Git
store using secure secret management
Before reusing the exact configuration key in a new OpenSearch version, verify the current OpenSearch documentation for the data-source encryption setting.
20.4 Create workspace
Recommended workspace:
Cpanel
Inside the workspace:
- Create an OpenSearch data source connection.
- Use the cluster Service endpoint.
- Use the required username/password authentication method.
- Test the connection.
- Create index pattern:
cpanel-access-log-*
- Select time field:
@timestamp
21. cPanel Access Dashboard
Recommended dashboard:
cPanel Access Overview
Suggested layout:
+-------------------------------------------------------------------+
| Total Requests | Unique Clients | 2xx | 4xx | 5xx |
+-------------------------------------------------------------------+
| Requests Over Time |
+--------------------------------+----------------------------------+
| HTTP Status Distribution | Top Requested URLs |
+--------------------------------+----------------------------------+
| Top Client IPs | HTTP Methods |
+--------------------------------+----------------------------------+
| Top User Agents |
+-------------------------------------------------------------------+
21.1 Total Requests
Visualization: Metric
Aggregation: Count
Label: Total Requests
21.2 Unique Clients
Visualization: Metric
Aggregation: Unique Count / Cardinality
Field: parsed.client_ip
Label: Unique Clients
21.3 2xx
DQL:
parsed.http_status >= 200 and parsed.http_status < 300
21.4 4xx
parsed.http_status >= 400 and parsed.http_status < 500
21.5 5xx
parsed.http_status >= 500 and parsed.http_status < 600
Do not use chained comparison syntax such as:
200 <= parsed.http_status < 300
21.6 Requests over time
Visualization: Area/Line
Metric: Count
X axis: Date Histogram
Field: @timestamp
Interval: Auto
21.7 HTTP status distribution
Visualization: Pie/Donut
Metric: Count
Terms: parsed.http_status
Size: 10
21.8 Top URLs
Visualization: Horizontal Bar
Metric: Count
Terms: parsed.url.keyword
Fallback: parsed.url if aggregatable
Size: 10
21.9 Top client IPs
Terms: parsed.client_ip
Size: 10
21.10 HTTP methods
Terms: parsed.method.keyword
Fallback: parsed.method if aggregatable
21.11 Top user agents
Terms: parsed.user_agent.keyword
Fallback: parsed.user_agent if aggregatable
Useful for scanners, bots, crawlers, and automated clients.
22. Linux Audit Security Dashboard
Recommended dashboard:
Linux Server Audit & Security Overview
Layout:
+-----------------------------------------------------------+
| Commands | Logins | Failed Syscalls | Users | Audit Docs |
+-----------------------------------------------------------+
| Audit Activity Over Time |
+---------------------------+-------------------------------+
| Top Commands | Top Executables |
+---------------------------+-------------------------------+
| Top Users | Top Failed Syscalls |
+---------------------------+-------------------------------+
| Authentication & Session Activity |
+-----------------------------------------------------------+
| Recent Audit Activity |
+-----------------------------------------------------------+
22.1 Commands executed
DQL:
audit.log_type: "EXECVE"
Metric: Count.
22.2 Login events
audit.log_type: "LOGIN"
22.3 Failed syscalls
audit.log_type: "SYSCALL" AND syscall.success: "no"
22.4 Top failed syscall names
Terms: syscall.syscall.keyword
22.5 Top commands
DQL: audit.log_type: "EXECVE"
Terms: execve.args.a0.keyword
Use a normalized execve.command field instead if you implement the recommended schema improvement.
22.6 Top users
DQL: audit.log_type: "SYSCALL"
Terms: syscall.AUID.keyword
22.7 Authentication/session activity
DQL:
audit.log_type: ("LOGIN" OR "USER_ACCT" OR "USER_START" OR "CRED_ACQ")
Use Date Histogram on @timestamp and split series by audit.log_type.keyword.
22.8 Recent activity
Use separate tables initially because one logical Audit event spans multiple documents.
Recent EXECVE fields:
@timestamp
audit.event_id
execve.args.a0
execve.args.a1
execve.args.a2
Recent failed syscall fields:
@timestamp
audit.event_id
syscall.AUID
syscall.exe
syscall.syscall
syscall.success
syscall.exit_code
Future correlation should group by audit.event_id.
23. Anomaly Detection, Forecasting, and Alerting
23.1 Request-volume anomaly detector
Recommended detector:
Total-Request-Detector
Index:
cpanel-access-log-*
Timestamp:
@timestamp
Feature:
Request-Count -> Count
Starting operational settings from the implementation:
Interval: 10 minutes
Frequency: 10 minutes
Window delay: 1 minute
History: 144 intervals (~24 hours)
Shingle size: 8
Sparse data: Fill with Zeros
Do not split by high-cardinality client IP in the first detector.
Validate:
actual request count
expected range
anomaly grade
confidence
historical timeline
23.2 Forecasting
Recommended forecaster:
Total-Request-Forecast
Starting values used:
Forecast interval: 3 minutes
Window delay: 2 minutes
Horizon: 24 intervals (~72 minutes)
History: 537 intervals (~26h 51m)
These are starting parameters, not universal defaults.
23.3 Alert monitor
Flow:
Anomaly Detector
-> Monitor
-> Trigger
-> Alert
-> Action
-> Notification Channel
Starting trigger threshold:
Anomaly grade > 0.7
AND
Confidence > 0.7
Tune after reviewing false positives and real incidents.
24. SMTP Notification Configuration
Recorded sender configuration:
Sender name: cloud-operation
SMTP host: zsmtp.brilliant.com.bd
Port: 465
Method: SSL
From address: cloud.operation@brilliant.com.bd
Do not put SMTP passwords into Helm values or Git.
24.1 Create temporary credential files
printf '%s' 'cloud.operation@brilliant.com.bd' > /tmp/smtp-username
read -rsp 'SMTP password: ' SMTP_PASSWORD
echo
printf '%s' "$SMTP_PASSWORD" > /tmp/smtp-password
unset SMTP_PASSWORD
24.2 Create Secret for OpenSearch keystore
kubectl create secret generic opensearch-smtp-keystore \
-n "$NS" \
--from-file=opensearch.notifications.core.email.cloud-operation.username=/tmp/smtp-username \
--from-file=opensearch.notifications.core.email.cloud-operation.password=/tmp/smtp-password \
--dry-run=client -o yaml | kubectl apply -f -
rm -f /tmp/smtp-username /tmp/smtp-password
The key path uses the sender name cloud-operation, not the sender email address.
24.3 Helm keystore value
keystore:
- secretName: opensearch-smtp-keystore
24.4 Upgrade
helm upgrade "$OS_RELEASE" "$OS_CHART" \
-n "$NS" \
-f values.yaml
24.5 Verify keystore
kubectl exec -n "$NS" "${OS_STS}-0" -c opensearch -- \
/usr/share/opensearch/bin/opensearch-keystore list
Expected entries:
opensearch.notifications.core.email.cloud-operation.username
opensearch.notifications.core.email.cloud-operation.password
Verify all OpenSearch nodes if the chart injects keystore material per pod.
24.6 Invalid Addresses troubleshooting
Check:
From address is valid
recipient is valid
SMTP host/port are correct
SSL/TLS mode matches the server
credentials exist in keystore
credentials are present on all relevant nodes
SMTP relay permits sender
SMTP account may use configured From address
recipient is allowed
25. Reliability, Backup, and Recovery Tests
25.1 Fluent Bit buffering recovery
After normal ingestion is stable:
- Make Data Prepper temporarily unavailable.
- Generate a few cPanel requests.
- Confirm Fluent Bit retries.
- Confirm filesystem backlog grows.
- Restart Fluent Bit while Data Prepper is unavailable.
- Restore Data Prepper.
- Confirm buffered events drain.
- Confirm documents reach OpenSearch.
Monitor:
du -sh /var/lib/fluent-bit/storage/
journalctl -u fluent-bit -f
25.2 OpenSearch pod recreation
Delete one pod:
kubectl delete pod -n "$NS" "${OS_STS}-0"
Expected:
same StatefulSet ordinal recreated
same PVC reused
node rejoins cluster
Verify:
kubectl get pod,pvc -n "$NS" -o wide
25.3 Worker restart/failure test
Expected:
remaining OpenSearch nodes continue serving
replica shard can be promoted if primary is lost
failed node/pod rejoins according to scheduling/storage policy
25.4 Snapshots
Shard replicas are not backups.
Snapshot storage should be independent from the OpenSearch cluster and Longhorn volumes.
Required operational practice:
configure repository
schedule snapshots
retain snapshots
monitor failures
perform restore tests
A backup is not considered valid until restore has been tested.
26. Security Hardening
26.1 Credentials
Do not commit:
OpenSearch admin password
kibanaserver password
Data Prepper logger password
SMTP password
data-source encryption key
private keys
Kubernetes Secret manifests containing real Base64 data
26.2 Identity separation
admin -> security/cluster administration only
kibanaserver -> Dashboards backend only
logger -> Data Prepper ingestion only
human users -> read/search/dashboard permissions as required
26.3 Data Prepper ingress
The original implementation used:
ssl: false
on the Data Prepper HTTP source and exposed it through NGINX NodePort. For production, protect the path with one or more of:
TLS
HTTP basic authentication
private routing/VPN
source IP restriction
firewall ACL
Kubernetes NetworkPolicy
authenticated ingress
The Data Prepper HTTP source supports TLS and HTTP basic authentication.
26.4 OpenSearch ports
9300 -> never expose publicly
9200 -> normally private/internal
5601 -> expose through controlled reverse proxy/Ingress
2021 -> restrict to approved log senders
26.5 Certificate management
Monitor:
Not Before
Not After
SAN correctness
key/cert pair
CA trust
rotation date
root-ca-key.pem must remain offline/encrypted.
26.6 NetworkPolicy
If NetworkPolicy is used, explicitly permit:
Dashboards -> OpenSearch :9200
Data Prepper -> OpenSearch :9200
OpenSearch nodes <-> OpenSearch nodes :9300
Ingress -> Dashboards :5601
Ingress -> Data Prepper :2021
27. Operational Monitoring
27.1 Kubernetes
Monitor:
pod restarts
OOMKilled
CPU throttling
memory RSS
Pending pods
PVC usage
node pressure
scheduling failures
Commands:
kubectl get pods -n "$NS" -o wide
kubectl top pods -n "$NS"
kubectl get events -n "$NS" --sort-by=.lastTimestamp | tail -50
27.2 OpenSearch
Monitor:
cluster health
node count
JVM heap pressure
GC
shard allocation
pending tasks
indexing latency
search latency
circuit breakers
disk watermarks
Useful APIs:
GET _cluster/health
GET _cat/nodes?v
GET _cat/shards?v
GET _cat/indices?v
GET _cluster/pending_tasks
GET _nodes/stats
27.3 Fluent Bit
Monitor:
input records
output records
retries
failed chunks
storage queue usage
service restarts
27.4 Data Prepper
Monitor:
requestsReceived
successRequests
badRequests
requestsRejected
grokProcessingMismatch
grokProcessingErrors
date failures
OpenSearch document errors
27.5 Longhorn
Monitor:
volume health
replica count
replica placement
disk usage
snapshot growth
attachment failures
28. Troubleshooting Matrix
| Symptom | Layer | Likely cause | First checks |
|---|---|---|---|
Fluent Bit extra indentation level found |
Fluent Bit config | classic config indentation / inline Lua | reported line; move Lua to separate file |
Fluent Bit repeated 502 Bad Gateway |
Ingress/upstream | NGINX reachable but Data Prepper backend unhealthy | Data Prepper pod, Service, EndpointSlice, :2021 |
Fluent Bit connection refused |
Network/listener | endpoint unavailable | IP/port/firewall/listener |
Data Prepper Address already in use |
Pipeline | multiple HTTP sources on same port | keep one router HTTP source |
Data Prepper No definition for key 'PORT' |
Grok | unsupported pattern in tested 2.8 setup | use POSINT |
| Data Prepper pipeline YAML Boolean error | YAML | options at wrong hierarchy | indent under grok: correctly |
parsed: {} |
Parser | Grok did not match | compare raw line with regex |
| Wrong source in dedicated index | Routing | router/drop condition wrong | inspect log_source, route, new docs |
Index exists, docs.count=0 |
Sink/ingestion | no successful documents | Data Prepper sink logs + pipeline metrics |
Data Prepper Request execution cancelled / I/O session is closed |
OpenSearch sink | DNS/TCP/TLS/auth/sink instability | test each layer to :9200 |
OpenSearch pod Running but cluster unavailable |
OpenSearch transport | TLS/discovery/cluster manager | pod logs, :9300, headless DNS, SAN |
No subject alternative DNS name ... headless |
TLS | SAN mismatch | regenerate certs with actual DNS SANs |
Cluster manager not present |
Cluster formation | transport TLS/discovery issue | headless Service, :9300, cert SAN, nodes_dn |
Unauthorized |
Authentication | wrong active credential | active Security config / user hash |
Resource 'admin' is reserved |
Security API | reserved resource cannot be changed by that API | update YAML + securityadmin using admin cert |
Secret subPath not a directory |
Kubernetes mount | Secret key name mismatch | internal_users.yml exactly |
Dashboards publicBaseUrl is not allowed |
Dashboards config | unsupported key in tested version | remove it |
| Dashboards 401 | Authentication | auth required/missing | security config / credential flow |
| Dashboards 404 after auth | Base path/routing | application not aware of ingress path | basePath, rewriteBasePath |
Dashboards permission indices:monitor/settings/get |
Authorization | backend role missing required permission | identify plugin + target index before grant |
| NGINX duplicate host/path | Ingress | catch-all route already exists | use unique host/path |
too_many_nested_clauses |
OpenSearch mapping/query | too many default searchable fields | restrict index.query.default_field; control mappings |
SMTP Invalid Addresses |
Notification | sender/recipient/relay/auth issue | address + keystore + relay policy |
29. Troubleshooting Decision Trees
29.1 Log ingestion
Is the source file receiving new lines?
|
+-- NO --> troubleshoot cPanel/application logging
|
+-- YES
|
v
Did Fluent Bit discover/read the file?
|
+-- NO --> path, permissions, Tail DB, Read_from_Head policy
|
+-- YES
|
v
Does event contain source_file + log_source?
|
+-- NO --> Path_Key / Lua classification
|
+-- YES
|
v
Does Fluent Bit POST /logs successfully?
|
+-- 502 --> Ingress reached, upstream unhealthy
+-- refused --> listener/network/firewall issue
|
+-- success
|
v
Is Data Prepper 1/1 Running and :2021 listening?
|
+-- NO --> startup/pipeline configuration
|
+-- YES
|
v
Does the router match the expected log_source?
|
+-- NO --> route expression
|
+-- YES
|
v
Does the parser populate expected fields?
|
+-- NO --> Grok/date/key-value mismatch
|
+-- YES
|
v
Can Data Prepper reach OpenSearch :9200?
|
+-- NO --> DNS/TCP/TLS/CA
|
+-- YES
|
v
Does OpenSearch authenticate/authorize logger?
|
+-- NO --> user/role/index permissions
|
+-- YES
|
v
Does the expected index contain new documents?
29.2 OpenSearch deployment
Pod Pending?
|
+--> node affinity
+--> hard anti-affinity
+--> taints/tolerations
+--> CPU/memory requests
+--> PVC/Longhorn
Pod starts but crashes?
|
+--> vm.max_map_count
+--> JVM heap / OOM
+--> certificate file permissions
+--> TLS key format
+--> opensearch.yml syntax
+--> bootstrap security
Pods run but cluster does not form?
|
+--> headless Service DNS
+--> transport :9300
+--> SANs
+--> nodes_dn
+--> discovery settings
REST HTTPS fails?
|
+--> REST TLS enabled
+--> cert/key paths
+--> CA
+--> hostname/SAN
+--> certificate expiry
Authentication fails?
|
+--> active Security index
+--> internal_users.yml
+--> securityadmin applied?
29.3 Dashboards
Can browser reach NodePort?
|
+-- NO --> node firewall / NodePort / ingress-nginx Service
|
+-- YES
v
Does NGINX match /opensearch?
|
+-- NO --> Ingress rule / ingressClassName
|
+-- YES
v
Does Service have ready Dashboards endpoint?
|
+-- NO --> selector / pod readiness
|
+-- YES
v
Did Dashboards start without config validation errors?
|
+-- NO --> current + previous logs
|
+-- YES
v
Can Dashboards connect to OpenSearch with CA + kibanaserver?
|
+-- NO --> DNS/TLS/SAN/auth
|
+-- YES
v
Do assets/redirects stay under /opensearch?
|
+-- NO --> basePath/rewriteBasePath
|
+-- YES --> path is healthy
30. Daily / Weekly Operational Checklists
30.1 Daily
- All OpenSearch pods are
Runningand Ready. - OpenSearch cluster health is expected.
_cat/nodeslists all expected nodes.- No recurring transport TLS errors.
- Data Prepper pod is Ready.
- Data Prepper has no sustained OpenSearch sink errors.
- Fluent Bit services are running.
- No repeated Fluent Bit 5xx/connection failures.
- New cPanel access documents are arriving.
- New Audit documents are arriving.
@timestampis current and sane.- Grok processing error/mismatch rates are reviewed.
- Disk/PVC usage is below operational thresholds.
- Longhorn volumes are healthy.
- Dashboards loads and queries data.
30.2 Weekly
- Review OpenSearch JVM heap/GC trends.
- Review index growth and retention.
- Review Fluent Bit backlog utilization.
- Review Data Prepper rejected/bad requests.
- Review parsing failures/unhandled log sources.
- Review security/authentication failures.
- Review certificate expiry dates.
- Verify snapshots are succeeding.
- Periodically test restore procedure.
- Review alert quality and false positives.
31. Windows Fluent Bit Variant
This is an optional extension for downloaded Windows log files.
31.1 Recommended directory separation
C:\fluent-bit\
├── logs\
├── state\
└── buffer\
Do not place the Tail DB inside a wildcard source directory if it could be ingested accidentally.
31.2 Example concept
[SERVICE]
Flush 1
Daemon Off
Log_Level info
storage.metrics On
storage.path C:\fluent-bit\buffer
storage.sync full
storage.backlog.mem_limit 500M
[INPUT]
Name tail
Path C:\fluent-bit\logs\*.log
Path_Key source_file
Tag downloaded.logs
DB C:\fluent-bit\state\tail.db
DB.Journal_Mode wal
Read_from_Head true
Refresh_Interval 5
Skip_Long_Lines true
Mem_Buf_Limit 512MB
Storage.Type filesystem
[OUTPUT]
Name http
Host 118.67.213.76
Port 31599
Format json
HTTP_Method POST
URI /logs
Match *
Retry_Limit no_limits
Storage.Total_Limit_Size 4G
Before ingesting all historical files:
- Test one representative file.
- Confirm existing content is read.
- Restart Fluent Bit and confirm offset resume behavior.
- Verify
source_file. - Verify one test document in OpenSearch.
- Only then expand the wildcard.
32. Cheat Sheet
32.1 Complete process summary
1. Verify Kubernetes nodes, Longhorn, vm.max_map_count
2. Verify/generate Root CA, node certs, admin cert
3. Create Kubernetes Secrets
4. Configure OpenSearch Helm values
5. helm lint
6. helm template
7. kubectl --dry-run=server
8. helm upgrade --install OpenSearch
9. Verify pods/PVC/TLS/cluster health/security
10. Deploy Dashboards with CA + kibanaserver Secret
11. Configure /opensearch Ingress/basePath
12. Deploy Data Prepper with ONE HTTP :2021 router
13. Configure dedicated access/syslog/audit/catch-all pipelines
14. Configure Fluent Bit Tail DB + filesystem buffer + log_source
15. Verify /logs end-to-end
16. Create index templates/mappings
17. Create Dashboards workspace/index patterns
18. Build cPanel and Linux Audit dashboards
19. Configure anomaly detector / forecast / alerts
20. Configure SMTP keystore
21. Test failures, snapshots, and recovery
32.2 Kubernetes/OpenSearch quick commands
export NS=observability
export OS_RELEASE=brilliant-opensearch
export OS_STS=brilliant-opensearch-cluster-master
export OS_SVC=brilliant-opensearch-cluster-master
kubectl get pods -n "$NS" -o wide
kubectl get sts -n "$NS"
kubectl get svc -n "$NS"
kubectl get pvc -n "$NS"
kubectl get ingress -A
kubectl logs -n "$NS" "${OS_STS}-0" -c opensearch --tail=200
kubectl logs -n "$NS" "${OS_STS}-0" -c opensearch --previous --tail=200
kubectl port-forward -n "$NS" svc/"$OS_SVC" 9200:9200
curl -k -u 'admin:<PASSWORD>' https://localhost:9200/_cluster/health?pretty
curl -k -u 'admin:<PASSWORD>' 'https://localhost:9200/_cat/nodes?v'
32.3 Helm quick commands
helm lint "$OS_CHART" -f values.yaml
helm template "$OS_RELEASE" "$OS_CHART" \
-n "$NS" -f values.yaml > rendered-opensearch.yaml
kubectl apply --dry-run=server -f rendered-opensearch.yaml
helm upgrade --install "$OS_RELEASE" "$OS_CHART" \
-n "$NS" -f values.yaml
kubectl rollout status statefulset/"$OS_STS" -n "$NS"
32.4 TLS quick commands
openssl verify -CAfile root-ca.pem node-0.pem
openssl x509 -in node-0.pem -noout -subject -nameopt RFC2253
openssl x509 -in node-0.pem -noout -dates
openssl x509 -in node-0.pem -noout -text | sed -n '/Subject Alternative Name/,+2p'
32.5 Data Prepper quick commands
kubectl get pods -n observability -l app=data-prepper -o wide
kubectl get svc data-prepper -n observability -o wide
kubectl get endpointslice -n observability \
-l kubernetes.io/service-name=data-prepper -o wide
kubectl logs -n observability deploy/data-prepper --tail=200
kubectl exec -n observability deploy/data-prepper -- \
sh -c 'grep -i ":07E5" /proc/net/tcp /proc/net/tcp6 || true'
32.6 Fluent Bit quick commands
systemctl status fluent-bit --no-pager
journalctl -u fluent-bit --since '10 minutes ago' --no-pager
du -sh /var/lib/fluent-bit/storage/
ls -lh /var/lib/fluent-bit/state/
32.7 Test /logs
curl -i -X POST \
'http://118.67.213.76:31599/logs' \
-H 'Content-Type: application/json' \
--data '[{"log":"manual test","source_file":"manual","log_source":"test.manual"}]'
32.8 OpenSearch Dev Tools quick queries
GET _cluster/health
GET _cat/nodes?v
GET _cat/indices/cpanel-*?v
GET cpanel-access-log-*/_count
{
"query": {
"range": {
"@timestamp": {
"gte": "now-24h",
"lte": "now"
}
}
}
}
GET cpanel-access-log-*/_search
{
"size": 5,
"sort": [{"@timestamp": "desc"}]
}
32.9 DQL quick filters
2xx:
parsed.http_status >= 200 and parsed.http_status < 300
4xx:
parsed.http_status >= 400 and parsed.http_status < 500
5xx:
parsed.http_status >= 500 and parsed.http_status < 600
Audit commands:
audit.log_type: "EXECVE"
Failed syscalls:
audit.log_type: "SYSCALL" AND syscall.success: "no"
Authentication activity:
audit.log_type: ("LOGIN" OR "USER_ACCT" OR "USER_START" OR "CRED_ACQ")
32.10 Error-to-layer quick map
502 from Fluent Bit -> NGINX reachable; check Data Prepper upstream
connection refused -> listener/network
parsed: {} -> Grok mismatch
index docs.count=0 -> ingestion/sink not proven
SAN mismatch -> certificate DNS identity
Unauthorized -> authentication
security_exception -> authorization
404 after Dashboards auth-> basePath/routing
CrashLoop after config -> application validation/logs
33. Lessons Learned and Known Limitations
- Helm rendering is mandatory. Inspect final StatefulSet/Deployment, not only
values.yaml. Runningis not enough. Validate readiness, application logs, OpenSearch cluster health, and backend connectivity separately.- Transport TLS and REST TLS are different boundaries.
curl -kagainst port 9200 cannot repair transport TLS on 9300. - SANs matter. Correct CA trust is insufficient when the Service/headless DNS is missing from the certificate.
kibanaserveris a user;kibana_serveris a role. Do not confuse them.- A mounted Security YAML file is not automatically the active Security index. Apply changes appropriately with Security tooling.
- Secret
subPathnames match exact keys.internal_users.yamlandinternal_users.ymlare different keys. - One Data Prepper HTTP source per listener. Route internally to downstream pipelines.
- A sink without
routes:receives all events. Define an explicit unknown route if the catch-all must contain only unhandled sources. - Fluent Bit Tail DB is not the event queue. It remembers offsets; filesystem storage buffers undelivered chunks.
- Unlimited retries do not mean unlimited durability. A finite filesystem queue can still overflow and drop old chunks.
parsed: {}means transport can be healthy while parsing failed. Debug the raw line against the parser.- Audit records are multi-record logical events. Correlate using
audit.event_id. - Whitespace-splitting EXECVE arguments is not fully quote-aware. Preserve
args_raw. - Dynamic Audit arguments can create mapping explosion. Normalize high-value fields and restrict default searchable fields.
- Index existence does not prove document ingestion. Always inspect
docs.countand sample_source. - NGINX Ingress routing is controller-wide. Duplicate hostless
/rules can conflict across namespaces. - Dashboards path hosting requires application-aware basePath configuration.
- Version-specific settings matter. A configuration key accepted by Kibana or another OpenSearch Dashboards version may be rejected by 3.7.0.
- Least privilege first. Investigate the action and target resource before granting broad permissions.
- OpenSearch replicas and Longhorn replicas consume storage independently.
- OpenSearch shard replicas are not backups. Snapshot and restore testing remain mandatory.
34. Final Definition of Done
The platform is considered complete only when all of the following are true.
OpenSearch
- Three OpenSearch pods are Ready.
- Pods are on separate eligible Kubernetes nodes.
- Three 120Gi Longhorn PVCs are Bound.
- Transport TLS works with correct SANs.
- REST HTTPS works.
- Node certificate DNs match
nodes_dn. - Admin certificate DN matches
admin_dn. - Cluster manager is elected.
_cat/nodeslists all three nodes.- Cluster health is expected.
- Default/demo credentials are removed or changed.
- Real secrets are absent from Git.
Dashboards
- Dashboards is Ready.
- CA is mounted.
kibanaserverpassword comes from a Secret.- TLS verification is
full. /opensearchbasePath works.- Ingress does not conflict with Grafana
/. - Workspaces work with Security multi-tenancy disabled.
- Data source connection succeeds.
Fluent Bit
- Tail paths are correct.
- Tail DB is persistent and writable.
- Filesystem buffer is writable.
source_fileis present.log_sourceis correct./logsdelivery succeeds.- Retry/backlog behavior is tested.
- Queue usage is monitored.
Data Prepper
- Only one HTTP source owns port 2021.
- Service and EndpointSlice point to the current pod.
- Health endpoint is healthy.
- Router matches known log sources.
- Catch-all contains only unhandled sources.
- Access Grok populates
parsed.*. - Audit parsers populate expected type-specific fields.
- Date processors create correct
@timestamp. - OpenSearch sink has no persistent errors.
OpenSearch data
cpanel-access-log-*contains documents.cpanel-system-syslog-*contains expected syslog only.cpanel-audit-log-*contains Audit records.cpanel-all-log-*exposes unknown/unhandled sources.- cPanel access field mappings are correct.
- Audit default query fields are controlled.
- Retention/ISM strategy is defined.
Dashboards and alerting
- cPanel Access Overview exists.
- Linux Server Audit & Security Overview exists.
- Anomaly detector runs and produces sensible output.
- Forecasting runs if required.
- Monitor and trigger exist.
- SMTP credentials exist in OpenSearch keystore.
- Test notification succeeds.
- Real alert action succeeds.
Operations
- OpenSearch pod-failure test completed.
- Fluent Bit/Data Prepper interruption recovery tested.
- Snapshot repository configured.
- Snapshot schedule configured.
- Restore test completed.
- Certificate-expiry monitoring configured.
- Daily/weekly checks assigned to operations.
Verification References Used for This Consolidation
This combined SOP was cross-checked against the official documentation for:
- OpenSearch Helm installation and chart configuration.
- OpenSearch Security YAML configuration and
securityadmin.shbehavior. - OpenSearch index replica settings and cluster health semantics.
- OpenSearch Dashboards Workspaces and multi-tenancy compatibility.
- OpenSearch Dashboards multiple data sources.
- Data Prepper HTTP source, Grok processor, key-value processor, conditional routing, server API port, and OpenSearch sink.
- Fluent Bit Tail input, filesystem buffering, retry semantics, and output queue limits.
- Kubernetes StatefulSet/headless Service behavior and Kubernetes Secret/volume semantics.
When upgrading any major/minor component, rerun the same validation flow against the documentation for the exact version before applying old configuration unchanged.
Prepared by: Md. Mosiur Rahnman