commit 5f6478fc2c78b11f8dff3d5e2acd50107a5571e5 Author: mosiur Date: Wed Aug 19 09:53:24 2026 +0000 Opensearch Installation and cPanel log ingestion diff --git a/OpenSearch_Kubernetes_Helm_Production_SOP.md b/OpenSearch_Kubernetes_Helm_Production_SOP.md new file mode 100644 index 0000000..587a363 --- /dev/null +++ b/OpenSearch_Kubernetes_Helm_Production_SOP.md @@ -0,0 +1,1765 @@ +# SOP: Production OpenSearch Deployment on Kubernetes Using Helm + +## 1. Purpose + +Ei SOP-er goal holo Kubernetes cluster-e production-grade OpenSearch deploy kora using Helm, with: + +- 3-node OpenSearch cluster +- Longhorn persistent storage +- OpenSearch Dashboards +- Fluent Bit log ingestion +- HTTPS/TLS +- Node-to-node transport encryption +- Dedicated `observability` namespace +- Controlled pod scheduling +- Production-oriented security +- Nginx Proxy Manager for browser-facing HTTPS + +> **Learning-mode note:** Ei SOP-te exact executable command syntax intentionally deya hoyni. Protiti step-e command-er purpose, tool, expected output, ebong validation deya ache. Exact commands chaile `SHOW SOLUTION` type korun. + +--- + +## 2. Final Architecture + +```text +cPanel / Log Sources + | + | Fluent Bit + | HTTPS :9200 + v ++----------------------------------------+ +| Kubernetes Cluster | +| Namespace: observability | +| | +| OpenSearch Client Service | +| ClusterIP :9200 | +| | | +| v | +| +-------------------------------+ | +| | OpenSearch StatefulSet | | +| | replicas: 3 | | +| | | | +| | pod-0 -> PVC-0 | | +| | pod-1 -> PVC-1 | | +| | pod-2 -> PVC-2 | | +| +-------------------------------+ | +| ^ ^ ^ | +| | TLS Transport :9300 | +| +--------+--------+ | +| | | +| Headless Service | +| Peer discovery / stable DNS | +| | +| OpenSearch Dashboards | +| | | +| +--> HTTPS OpenSearch :9200 | ++----------------------------------------+ + ^ + | + | HTTP/HTTPS + | +Nginx Proxy Manager + ^ + | + | Public HTTPS + | + Browser +``` + +--- + +## 3. Kubernetes Cluster Topology + +Target nodes: + +- `k8s-cp-01` — control-plane +- `k8s-storage-01` — storage +- `k8s-worker-01` — worker/storage-capable +- `k8s-worker-02` — worker/storage-capable + +Observed hostname labels include: + +```text +kubernetes.io/hostname +k3s.io/hostname +``` + +Recommended topology key: + +```text +kubernetes.io/hostname +``` + +Storage node taint: + +```text +key: dedicated +value: storage +effect: NoSchedule +``` + +OpenSearch pods require a matching toleration if `k8s-storage-01` is intended as an eligible scheduling target. + +--- + +## 4. Namespace + +Use: + +```text +observability +``` + +Purpose: + +- Organizational separation +- RBAC scope +- Secret scoping +- NetworkPolicy scope +- ResourceQuota/LimitRange scope +- Easier operations and troubleshooting + +A namespace is **not automatically a security boundary**. Actual isolation requires RBAC, NetworkPolicy, Secret access control, and related controls. + +--- + +## 5. OpenSearch Node Count vs Index Replica Count + +Do not confuse Helm `replicas` with OpenSearch `number_of_replicas`. + +### Helm replicas + +Target: + +```text +replicas: 3 +``` + +Meaning: + +```text +3 OpenSearch pods +3 JVMs +3 PVCs +``` + +### OpenSearch index replicas + +For a 3-node cluster: + +```text +1 primary shard + 2 replica shards = 3 shard copies +``` + +Therefore the logical production target is: + +```text +number_of_replicas: 2 +``` + +This is configured at the OpenSearch index/template layer, not automatically from Helm replicas. + +--- + +## 6. StatefulSet Design + +Use a **StatefulSet** for OpenSearch because it provides: + +- Stable ordinal identity +- Stable pod names +- Stable network identity +- Stable PVC association +- Controlled lifecycle behavior + +Example: + +```text +opensearch-0 +opensearch-1 +opensearch-2 +``` + +Each pod must have its own PVC. + +Do not share a single data PVC across all OpenSearch nodes. + +--- + +## 7. Service Design + +OpenSearch needs two logical Service patterns. + +### 7.1 Headless Service + +Purpose: + +- Stable per-pod DNS +- Peer discovery +- StatefulSet network identity +- Node-to-node communication + +Concept: + +```text +clusterIP: None +``` + +Primary transport port: + +```text +9300 +``` + +StatefulSet `serviceName` should reference this headless Service. + +Pod DNS pattern: + +```text +...svc.cluster.local +``` + +### 7.2 Client-facing ClusterIP Service + +Purpose: + +- Dashboards → OpenSearch +- Fluent Bit → OpenSearch +- Administrative REST clients → OpenSearch + +REST/API port: + +```text +9200 +``` + +Clients should use the stable ClusterIP Service instead of targeting one specific OpenSearch pod. + +--- + +## 8. OpenSearch Ports + +### REST / HTTPS + +```text +9200 +``` + +Used by: + +- Fluent Bit +- OpenSearch Dashboards +- API clients +- Administrative requests + +### Transport + +```text +9300 +``` + +Used by: + +- Node-to-node communication +- Cluster membership +- Shard replication +- Cluster-manager communication + +Do not expose `9300` publicly. + +Keep `9200` private/internal unless there is a specific administrative requirement. + +--- + +## 9. Persistence and Storage + +Target: + +```text +StorageClass: longhorn +AccessMode: ReadWriteOnce +PVC size: 120Gi per OpenSearch pod +``` + +For 3 pods: + +```text +3 × 120Gi = 360Gi logical PVC capacity +``` + +If Longhorn volume replica count is 2: + +```text +360Gi × 2 ≈ 720Gi raw replicated capacity +``` + +OpenSearch shard replication and Longhorn block replication are different layers: + +```text +OpenSearch replicas +→ application-level redundancy + +Longhorn replicas +→ block-level redundancy +``` + +Both consume storage. + +--- + +## 10. OpenSearch Data Path + +The PVC must be mounted to the OpenSearch data directory, or `path.data` must explicitly point to the mounted location. + +Unsafe pattern: + +```text +PVC mounted to /mnt +but OpenSearch writes to default data path +``` + +Possible result: + +```text +PVC exists +but index data lives in container filesystem +→ pod recreation can lose data +``` + +--- + +## 11. Resource Sizing + +Target: + +```text +requests: + cpu: 2 + memory: 6Gi + +limits: + cpu: 4 + memory: 8Gi +``` + +For 3 pods: + +```text +Total CPU requests: 6 cores +Total memory requests: 18Gi + +Total CPU limits: 12 cores +Total memory limits: 24Gi +``` + +Scheduler uses resource requests when deciding placement. + +--- + +## 12. JVM Heap + +Use a fixed heap: + +```text +Xms = Xmx +``` + +Discussed starting point: + +```text +~3Gi heap +``` + +Remaining memory is needed for: + +- Filesystem cache +- Direct buffers +- Thread stacks +- JVM metadata +- Native libraries +- Lucene/OpenSearch overhead + +Monitor: + +- Heap utilization +- Old generation pressure +- GC frequency/duration +- Circuit breakers +- Container RSS +- OOMKilled events +- Query latency +- Indexing latency + +Heap belongs in Helm/application configuration, not in a Secret. + +--- + +## 13. Pod Scheduling Strategy + +Target placement: + +```text +k8s-storage-01 → 1 OpenSearch pod +k8s-worker-01 → 1 OpenSearch pod +k8s-worker-02 → 1 OpenSearch pod +k8s-cp-01 → no OpenSearch pod +``` + +### Toleration + +Allows a pod onto a matching tainted node. + +For the storage node, the toleration must match: + +```text +dedicated=storage:NoSchedule +``` + +Toleration allows placement but does not force it. + +### Node Affinity / Selection + +Controls which nodes are eligible. + +Positive selection using workload/storage labels is generally easier to maintain than only blacklisting the control-plane hostname. + +### Pod Anti-Affinity + +Prevents multiple OpenSearch pods from landing on the same Kubernetes node. + +Topology key: + +```text +kubernetes.io/hostname +``` + +Target: + +```text +1 OpenSearch pod per Kubernetes node +``` + +Hard anti-affinity gives stronger failure-domain isolation but can leave a pod Pending if fewer than three eligible nodes are available. + +--- + +## 14. Helm Deployment Workflow + +Do not install the chart blindly. + +Use: + +```text +Official Helm chart + + +Default values + + +Minimal custom values + ↓ +Rendered Kubernetes manifests + ↓ +Review + ↓ +Validate + ↓ +Install +``` + +Recommended workflow: + +1. Inspect available OpenSearch chart versions +2. Inspect chart defaults +3. Build a minimal custom values file +4. Render manifests locally +5. Inspect StatefulSet, Services, PVCs, Secret mounts, affinity, tolerations and probes +6. Validate the rendered YAML +7. Install only after review + +Avoid copying the full default `values.yaml` unless required. + +--- + +## 15. Helm Values to Control + +Production values should intentionally address: + +```text +clusterName +nodeGroup +replicas +singleNode +roles +image version +resources +opensearchJavaOpts +persistence +storageClass +PVC size +antiAffinity +antiAffinityTopologyKey +nodeAffinity / nodeSelector +tolerations +protocol +security configuration +secret mounts +extra volumes +extra init containers +pod management policy +probes +``` + +--- + +## 16. Image Versioning + +Avoid a broad mutable tag such as: + +```text +3 +``` + +Use a tested, pinned OpenSearch release. + +Benefits: + +- Repeatable deployment +- Predictable restarts +- Controlled upgrades +- Easier compatibility management with Dashboards + +--- + +## 17. Kernel Requirement + +Validate `vm.max_map_count` on every eligible node: + +- `k8s-storage-01` +- `k8s-worker-01` +- `k8s-worker-02` + +If Helm sysctl mechanisms are enabled, understand whether the chart uses: + +- privileged init container +- sysctl configuration +- both + +Avoid unnecessary duplicate privileged mechanisms. + +--- + +## 18. Production Security Architecture + +There are multiple TLS boundaries. + +### Browser-facing TLS + +```text +Browser → Nginx Proxy Manager +``` + +Nginx Proxy Manager manages this certificate. + +### OpenSearch REST TLS + +```text +Dashboards → OpenSearch :9200 +Fluent Bit → OpenSearch :9200 +``` + +OpenSearch manages this HTTPS/TLS configuration. + +### OpenSearch Transport TLS + +```text +node-0 ↔ node-1 ↔ node-2 :9300 +``` + +Nginx Proxy Manager does not protect node-to-node traffic. + +Transport TLS should remain enabled in production. + +--- + +## 19. Identity Separation + +Do not reuse the admin account everywhere. + +Recommended identities: + +### Admin + +Use for: + +- Security administration +- Cluster administration +- Emergency operations + +### Dashboards server account + +Use only for Dashboards backend communication. + +### Fluent Bit ingest account + +Grant only required index-write permissions. + +Do not grant: + +- Security administration +- Arbitrary index deletion +- Cluster-wide configuration access +- Read access to unrelated indices + +### Human Dashboard users + +Give only required read/search/visualization permissions. + +--- + +## 20. Bootstrap Password Secret + +The initial admin password should be stored in a Kubernetes Secret in the same namespace as OpenSearch. + +Important: + +- Base64 is encoding, not encryption +- Never commit real Secret values to Git +- Restrict Secret read permission with RBAC +- Consider Kubernetes Secret encryption at rest +- Rotate any credential exposed in chat or logs + +The bootstrap Secret should ideally contain only: + +```text +OPENSEARCH_INITIAL_ADMIN_PASSWORD +``` + +--- + +## 21. Production PKI + +Certificates must be generated by: + +- Your organization PKI +- An internal CA +- A secure OpenSSL-based private CA +- A managed PKI platform + +Minimum certificate set: + +```text +root-ca.pem +root-ca-key.pem + +admin.pem +admin-key.pem + +node-0.pem +node-0-key.pem + +node-1.pem +node-1-key.pem + +node-2.pem +node-2-key.pem +``` + +--- + +## 22. Certificate Responsibilities + +### `root-ca-key.pem` + +Purpose: + +```text +Sign certificates +``` + +Rules: + +```text +NEVER mount into OpenSearch pods +NEVER store permanently in Kubernetes +NEVER commit to Git +Store offline/encrypted +``` + +If compromised, an attacker can issue fake trusted node or admin certificates. + +### `root-ca.pem` + +Purpose: + +```text +Verify certificates issued by the CA +``` + +Can be distributed to: + +- OpenSearch nodes +- Dashboards +- Fluent Bit +- Administrative clients + +### Admin Certificate + +```text +admin.pem +admin-key.pem +``` + +Used for privileged security administration. + +Do not use it as a normal node certificate. + +Do not mount the admin private key into every OpenSearch pod. + +### Node Certificates + +Each node should have a unique identity: + +```text +node-0.pem / node-0-key.pem +node-1.pem / node-1-key.pem +node-2.pem / node-2-key.pem +``` + +Benefits: + +- Better isolation +- Easier rotation +- Easier revocation +- Better auditing + +--- + +## 23. Certificate Generation Lifecycle + +Concept: + +```text +Generate private key + ↓ +Create CSR + ↓ +Sign CSR using Root CA private key + ↓ +Receive CA-signed certificate +``` + +Example: + +```text +node-0-key.pem + ↓ +node-0 CSR + ↓ +signed by root-ca-key.pem + ↓ +node-0.pem +``` + +OpenSearch-compatible private keys should be in the correct PKCS#8 format. + +--- + +## 24. Certificate CN and SAN Design + +Do not use an excessively long Kubernetes FQDN as the certificate CN. + +X.509 CN has size constraints and very long values can cause generation errors. + +Better approach: + +```text +CN: + short node identity + +SAN: + actual Kubernetes DNS names +``` + +Use SAN entries for long pod and Service DNS names. + +--- + +## 25. Node Certificate SAN Requirements + +Each node certificate should include: + +- Its pod DNS +- Relevant short pod DNS names +- Headless Service-related DNS names if needed +- Shared OpenSearch client Service DNS + +Why shared client Service DNS matters: + +```text +Dashboards → https://client-service:9200 +``` + +The ClusterIP may route to any OpenSearch pod. + +Whichever pod responds must present a certificate valid for the requested Service hostname. + +--- + +## 26. Certificate Verification + +Before Kubernetes integration, verify: + +### CA chain + +```text +admin.pem → verifies against root-ca.pem +node-0.pem → verifies against root-ca.pem +node-1.pem → verifies against root-ca.pem +node-2.pem → verifies against root-ca.pem +``` + +### SAN + +Each node certificate contains the intended DNS names. + +### Extended Key Usage + +Node certificates support the required TLS server/client roles. + +### Private key format + +Keys are in OpenSearch-compatible PKCS#8 format. + +### Key/certificate match + +Each certificate matches its own private key. + +### Validity + +Check: + +```text +Not Before +Not After +``` + +### Distinguished Names + +Extract exact RFC2253-formatted certificate DNs for OpenSearch configuration. + +--- + +## 27. Kubernetes Secret Layout + +Recommended separation: + +### Bootstrap Secret + +```text +opensearch-secret +└── OPENSEARCH_INITIAL_ADMIN_PASSWORD +``` + +### Node TLS Secret + +```text +opensearch-node-tls +├── root-ca.pem +├── node-0.pem +├── node-0-key.pem +├── node-1.pem +├── node-1-key.pem +├── node-2.pem +└── node-2-key.pem +``` + +### Admin TLS Secret + +```text +opensearch-admin-tls +├── root-ca.pem +├── admin.pem +└── admin-key.pem +``` + +### Never place in Kubernetes + +```text +root-ca-key.pem +``` + +--- + +## 28. Per-Pod Certificate Selection + +A StatefulSet uses one common pod template. + +If the TLS Secret contains all node certificates, every pod initially sees all files. + +Production pattern: + +```text +Node TLS Secret + ↓ +read-only mount + ↓ +init container + ↓ +detect pod ordinal + ↓ +copy correct node certificate/key + ↓ +private emptyDir + ↓ +main OpenSearch container +``` + +Mapping: + +```text +pod ending in -0 +→ node-0.pem → node.pem +→ node-0-key.pem → node-key.pem + +pod ending in -1 +→ node-1.pem → node.pem +→ node-1-key.pem → node-key.pem + +pod ending in -2 +→ node-2.pem → node.pem +→ node-2-key.pem → node-key.pem +``` + +All pods can then use the same common paths: + +```text +tls/node.pem +tls/node-key.pem +tls/root-ca.pem +``` + +--- + +## 29. OpenSearch TLS Configuration + +The OpenSearch configuration must logically define: + +### Transport TLS + +```text +Node certificate +Node private key +Trusted Root CA +Hostname verification behavior +``` + +### REST TLS + +```text +HTTP TLS enabled +Node/server certificate +Node private key +Trusted Root CA +``` + +### Trusted node identities + +```text +plugins.security.nodes_dn +``` + +Must contain the exact trusted node certificate DNs. + +### Admin identity + +```text +plugins.security.authcz.admin_dn +``` + +Must contain the exact admin certificate DN. + +Do not mix node and admin identities. + +--- + +## 30. Security Configuration Files + +A production Security plugin setup may use: + +```text +config.yml +internal_users.yml +roles.yml +roles_mapping.yml +action_groups.yml +tenants.yml +``` + +Responsibilities: + +### `internal_users.yml` + +- User identities +- Password hashes +- Backend roles + +### `roles.yml` + +- Cluster permissions +- Index permissions +- Tenant permissions + +### `roles_mapping.yml` + +- Maps users/backend roles to OpenSearch roles + +### `config.yml` + +- Authentication domains +- Authorization settings +- Security plugin behavior + +--- + +## 31. OpenSearch Dashboards + +Deploy Dashboards after OpenSearch is healthy. + +Recommended flow: + +```text +Browser + ↓ +Nginx Proxy Manager + ↓ +Dashboards Service + ↓ +Dashboards pod + ↓ +HTTPS OpenSearch Client Service :9200 +``` + +Dashboards should: + +- Trust `root-ca.pem` +- Use a dedicated backend credential +- Not use the admin account +- Connect to the OpenSearch ClusterIP Service + +--- + +## 32. Fluent Bit Integration + +Fluent Bit should connect to the private OpenSearch REST endpoint over HTTPS. + +Production requirements: + +- TLS enabled +- CA verification enabled +- Trust `root-ca.pem` +- Dedicated ingest identity +- Least-privilege role +- No admin credential reuse +- Network access restricted to required paths + +--- + +## 33. Nginx Proxy Manager + +Recommended public flow: + +```text +Browser → Nginx Proxy Manager → Dashboards +``` + +Avoid: + +```text +Public → OpenSearch :9200 +Public → OpenSearch :9300 +``` + +Port `9300` should remain internal. + +Port `9200` should normally remain private. + +--- + +## 34. Probe Design + +Do not use fake endpoints such as `/healthz:8080` unless the application actually exposes them. + +### Startup Probe + +Question: + +```text +Has OpenSearch finished starting? +``` + +### Readiness Probe + +Question: + +```text +Can this pod receive Service traffic? +``` + +### Liveness Probe + +Question: + +```text +Is the process permanently stuck/dead? +``` + +Do not make liveness depend too aggressively on transient cluster health. + +Shard recovery can temporarily cause unhealthy cluster status without meaning the JVM should be restarted. + +--- + +## 35. Pod Management Policy + +Understand: + +```text +OrderedReady +``` + +vs: + +```text +Parallel +``` + +Potential multi-node bootstrap deadlock: + +```text +pod-0 waits for peers +StatefulSet waits for pod-0 Ready +pod-1 and pod-2 never start +``` + +Parallel startup may be more appropriate for a multi-node distributed cluster. + +Also consider peer discovery before readiness. + +--- + +## 36. Deployment Order + +Recommended: + +```text +1. Namespace +2. Bootstrap Secret +3. TLS Secrets +4. Security configuration +5. Helm custom values +6. Rendered-manifest review +7. OpenSearch install +8. Cluster-health validation +9. Dashboards install +10. Nginx Proxy Manager configuration +11. Fluent Bit integration +12. Index templates and replica settings +13. Snapshot backup configuration +14. Monitoring and alerting +``` + +--- + +## 37. Rendered Manifest Review + +Before Helm install, verify: + +```text +[ ] StatefulSet replicas = 3 +[ ] Correct image/version +[ ] Resources present +[ ] Longhorn PVC template present +[ ] 120Gi per pod +[ ] Correct data path +[ ] Headless Service exists +[ ] ClusterIP REST Service exists +[ ] Port 9200 mapping correct +[ ] Port 9300 mapping correct +[ ] Hard anti-affinity rendered +[ ] topology key = kubernetes.io/hostname +[ ] Storage toleration at pod level +[ ] Control-plane excluded +[ ] TLS Secret mounts present +[ ] Per-pod certificate-selection logic present +[ ] TLS paths match opensearch.yml +[ ] nodes_dn matches node certificate DNs +[ ] admin_dn matches admin certificate DN +[ ] Bootstrap Secret reference correct +[ ] Probes match HTTPS behavior +``` + +--- + +## 38. Post-Deployment Validation + +### Kubernetes Layer + +```text +[ ] All 3 pods created +[ ] Each pod on a different Kubernetes node +[ ] No OpenSearch pod on control-plane +[ ] PVCs Bound +[ ] One PVC per pod +[ ] Longhorn volumes healthy +[ ] Init containers completed +[ ] No CrashLoopBackOff +[ ] No OOMKilled +``` + +### TLS Layer + +```text +[ ] Correct certificate selected per pod +[ ] REST endpoint serves HTTPS +[ ] CA verification succeeds +[ ] Transport TLS succeeds +[ ] No SAN mismatch +[ ] No nodes_dn mismatch +[ ] No admin_dn mismatch +``` + +### OpenSearch Layer + +```text +[ ] All 3 nodes joined one cluster +[ ] Cluster-manager elected +[ ] REST API responds +[ ] Authentication succeeds +[ ] Cluster health expected +[ ] Shard allocation correct +``` + +### Storage Layer + +```text +[ ] Data survives pod recreation +[ ] Same ordinal reuses same PVC +[ ] Longhorn replicas healthy and distributed +``` + +--- + +## 39. Failure Testing + +### Delete one OpenSearch pod + +Expected: + +```text +StatefulSet recreates same ordinal +Same PVC reused +Node rejoins cluster +``` + +### Restart one Kubernetes worker + +Expected: + +```text +Remaining nodes continue serving +Shard replica promotion can occur +Missing pod returns/reschedules according to policy +``` + +### Lose the primary-shard node + +With: + +```text +number_of_replicas: 2 +``` + +a remaining replica can be promoted to primary. + +### Verify anti-affinity + +Ensure no two OpenSearch pods share the same hostname. + +--- + +## 40. Cluster Health Interpretation + +### Green + +All primaries and replicas allocated. + +### Yellow + +All primaries allocated, but at least one replica unassigned. + +Typical example: + +```text +single-node cluster +number_of_replicas: 1 +``` + +### Red + +At least one primary shard unavailable. + +--- + +## 41. Common Mistakes Found During This Deployment + +### Mistake: one node with `number_of_replicas: 1` + +Result: + +```text +1 primary + 1 replica +only 1 node +→ replica unassigned +→ yellow cluster +``` + +### Mistake: confusing Helm replicas with shard replicas + +They control different layers. + +### Mistake: using normal ClusterIP as StatefulSet `serviceName` + +StatefulSet stable network identity should use the governing headless Service. + +### Mistake: treating port 80 as OpenSearch's main port + +OpenSearch REST and transport use separate ports. + +### Mistake: mounting the PVC to `/mnt` without changing `path.data` + +The PVC can remain unused. + +### Mistake: putting tolerations under the container + +Tolerations are pod-level settings. + +### Mistake: using pod anti-affinity as a node blacklist + +Pod anti-affinity controls pod co-location, not arbitrary node exclusion. + +### Mistake: assuming Nginx Proxy Manager protects node-to-node traffic + +It does not. + +### Mistake: treating Base64 as encryption + +It is only encoding. + +### Mistake: reusing admin credentials for Fluent Bit and Dashboards + +Violates least privilege. + +### Mistake: very long FQDN as certificate CN + +Can hit X.509 CN length limits. + +Use a short CN and SAN for full DNS names. + +--- + +## 42. Operational Monitoring + +### Kubernetes + +Monitor: + +- Pod restarts +- CPU throttling +- Memory RSS +- OOMKilled events +- PVC usage +- Node pressure +- Scheduling failures + +### OpenSearch + +Monitor: + +- JVM heap pressure +- GC +- Cluster health +- Shard allocation +- Pending tasks +- Indexing latency +- Search latency +- Circuit breakers +- Disk watermarks + +### Longhorn + +Monitor: + +- Volume health +- Replica count +- Replica placement +- Disk usage +- Snapshot growth + +### Security + +Monitor: + +- Certificate expiry +- Authentication failures +- Unauthorized requests +- Admin credential usage +- TLS handshake errors + +--- + +## 43. Certificate Rotation + +Track: + +```text +Not Before +Not After +``` + +Production certificate lifecycle should include: + +- Expiry monitoring +- Renewal schedule +- Secure CA access +- Leaf-certificate replacement procedure +- Emergency replacement/revocation procedure +- Admin certificate rotation +- Protected CA backup + +--- + +## 44. Backup Strategy + +OpenSearch shard replicas are **not backups**. + +They do not protect from: + +- Accidental deletion +- Malicious deletion +- Corruption replicated across nodes +- Entire cluster loss + +Use snapshots to independent storage such as object storage/S3-compatible storage. + +Include: + +- Separate credentials +- Snapshot retention +- Snapshot lifecycle +- Restore testing + +--- + +## 45. Security Hardening Checklist + +```text +[ ] Security plugin enabled +[ ] Demo certificates removed +[ ] Custom internal Root CA used +[ ] Unique node certificates +[ ] Separate admin certificate +[ ] root-ca-key.pem stored offline +[ ] REST HTTPS enabled +[ ] Transport TLS enabled +[ ] Port 9300 not externally exposed +[ ] Port 9200 private/internal +[ ] Dashboards exposed through Nginx Proxy Manager +[ ] Dedicated Fluent Bit user +[ ] Dedicated Dashboards server user +[ ] Human users use least-privilege roles +[ ] Secret access protected by RBAC +[ ] Kubernetes Secret encryption at rest considered +[ ] NetworkPolicy considered +[ ] Certificate expiry monitored +``` + +--- + +## 46. Go-Live Checklist + +### Kubernetes + +```text +[ ] Namespace = observability +[ ] 3 eligible OpenSearch nodes +[ ] Storage taint/toleration validated +[ ] Hard anti-affinity validated +[ ] Control-plane excluded +[ ] Longhorn healthy +[ ] vm.max_map_count validated +``` + +### Helm + +```text +[ ] Chart version pinned +[ ] OpenSearch version pinned +[ ] Custom values reviewed +[ ] Rendered YAML reviewed +[ ] No unexpected resources +``` + +### Storage + +```text +[ ] 120Gi × 3 planned +[ ] Longhorn replication capacity calculated +[ ] Sufficient disk headroom +``` + +### TLS + +```text +[ ] Root CA generated +[ ] Root CA key offline +[ ] Admin certificate valid +[ ] Node certificates valid +[ ] SANs correct +[ ] Certificate/key pairs match +[ ] DNs extracted +[ ] TLS Secrets created +``` + +### OpenSearch + +```text +[ ] 3 nodes joined +[ ] Cluster-manager election healthy +[ ] HTTPS works +[ ] Authentication works +[ ] Shard replica policy set +[ ] Cluster health verified +``` + +### Dashboards / Fluent Bit + +```text +[ ] Dashboards trusts internal CA +[ ] Fluent Bit trusts internal CA +[ ] Admin password not reused +[ ] Nginx Proxy Manager exposes Dashboards only +``` + +--- + +## 47. Recommended Next Steps + +Continue in this order: + +1. Verify generated certificates +2. Extract exact admin and node DNs +3. Create Kubernetes TLS Secrets +4. Build per-pod certificate-selection logic +5. Add custom TLS configuration to Helm values +6. Render Helm manifests +7. Review StatefulSet and Services +8. Validate node placement +9. Install OpenSearch +10. Verify the 3-node HTTPS/TLS cluster +11. Deploy OpenSearch Dashboards +12. Configure Nginx Proxy Manager +13. Create Fluent Bit least-privilege ingest identity +14. Connect Fluent Bit over HTTPS +15. Configure index templates and shard replica policy +16. Configure snapshots/backups +17. Add monitoring and certificate-expiry alerts + +--- + +## 48. Troubleshooting Decision Tree + +### Pod Pending + +Investigate: + +```text +Node affinity +Pod anti-affinity +Taints +Tolerations +CPU/memory requests +PVC scheduling/topology +``` + +### Pod CrashLoopBackOff + +Investigate: + +```text +JVM heap +vm.max_map_count +TLS file paths +Private-key format +Certificate permissions +Bootstrap admin password +OpenSearch logs +``` + +### Nodes do not form one cluster + +Investigate: + +```text +Headless Service +DNS resolution +Transport port 9300 +Transport TLS +Certificate SAN +nodes_dn +Discovery settings +Initial cluster bootstrap settings +``` + +### HTTPS fails + +Investigate: + +```text +REST TLS enabled +Certificate path +Private key path +Trusted Root CA +Client hostname +SAN +Certificate expiry +``` + +### Dashboards cannot connect + +Investigate: + +```text +OpenSearch client Service +Port 9200 +HTTPS +Root CA trust +Dashboards backend credential +NetworkPolicy +``` + +### Fluent Bit cannot ingest + +Investigate: + +```text +OpenSearch endpoint +HTTPS +CA verification +Credential +Role permissions +Index permissions +Network path +``` + +--- + +## 49. Core Lessons + +```text +1. Stateful identity is not the same as pod IP. +2. StatefulSet provides stable identity; PVC stores persistent data. +3. Headless Service is for peer discovery and per-pod DNS. +4. ClusterIP Service is for normal clients. +5. Port 9200 and 9300 have different purposes. +6. Kubernetes replicas and OpenSearch shard replicas are different. +7. Redundancy works only when copies are separated across failure domains. +8. Toleration permits; affinity selects; anti-affinity separates. +9. A PVC is useless if the application writes somewhere else. +10. Helm should be rendered and reviewed before install. +11. Base64 is not encryption. +12. Reverse-proxy TLS does not protect internal OpenSearch transport. +13. Production TLS requires controlled PKI and certificate lifecycle. +14. Root CA private key never belongs in OpenSearch pods. +15. Unique node certificates are better than sharing one node key. +16. SANs are critical for DNS-based TLS verification. +17. Admin, Dashboards, and Fluent Bit should use separate identities. +18. OpenSearch replication is not a backup. +``` + +--- + +## 50. Final Target State + +```text +Kubernetes: + 3 OpenSearch pods + 3 separate Kubernetes nodes + 3 separate Longhorn PVCs + no OpenSearch pod on control-plane + +OpenSearch: + 3-node cluster + transport TLS + REST HTTPS + correct shard replication + healthy cluster-manager election + +Security: + custom internal CA + unique node certificates + separate admin certificate + dedicated Dashboards credential + dedicated Fluent Bit credential + least privilege + +Access: + Browser → Nginx Proxy Manager → Dashboards + Dashboards → HTTPS OpenSearch ClusterIP + Fluent Bit → HTTPS OpenSearch private endpoint + +Operations: + monitoring + snapshots/backups + certificate expiry tracking + restore testing +``` + +--- + +## 51. Current Project Status + +At the end of this chat, the design has reached this point: + +```text +[✓] 3-node production topology selected +[✓] Longhorn selected for PVCs +[✓] 120Gi per PVC planned +[✓] Resource requests/limits discussed +[✓] JVM heap sizing discussed +[✓] Hard anti-affinity selected +[✓] Storage taint/toleration verified +[✓] HTTPS selected +[✓] Production TLS selected +[✓] Root CA/admin/node certificates generated +[ ] Kubernetes TLS Secrets integration +[ ] Per-pod certificate selection +[ ] OpenSearch TLS settings in Helm values +[ ] Rendered manifest validation +[ ] OpenSearch deployment +[ ] Dashboards deployment +[ ] Fluent Bit integration +``` + +The immediate next phase is: + +```text +Certificates + ↓ +Kubernetes TLS Secrets + ↓ +Helm certificate mounts + ↓ +OpenSearch TLS configuration + ↓ +Render and validate + ↓ +Deploy +```