diff --git a/OpenSearch_Kubernetes_Helm_Production_SOP.md b/OpenSearch_Kubernetes_Helm_Production_SOP.md deleted file mode 100644 index 587a363..0000000 --- a/OpenSearch_Kubernetes_Helm_Production_SOP.md +++ /dev/null @@ -1,1765 +0,0 @@ -# SOP: Production OpenSearch Deployment on Kubernetes Using Helm - -## 1. Purpose - -Ei SOP-er goal holo Kubernetes cluster-e production-grade OpenSearch deploy kora using Helm, with: - -- 3-node OpenSearch cluster -- Longhorn persistent storage -- OpenSearch Dashboards -- Fluent Bit log ingestion -- HTTPS/TLS -- Node-to-node transport encryption -- Dedicated `observability` namespace -- Controlled pod scheduling -- Production-oriented security -- Nginx Proxy Manager for browser-facing HTTPS - -> **Learning-mode note:** Ei SOP-te exact executable command syntax intentionally deya hoyni. Protiti step-e command-er purpose, tool, expected output, ebong validation deya ache. Exact commands chaile `SHOW SOLUTION` type korun. - ---- - -## 2. Final Architecture - -```text -cPanel / Log Sources - | - | Fluent Bit - | HTTPS :9200 - v -+----------------------------------------+ -| Kubernetes Cluster | -| Namespace: observability | -| | -| OpenSearch Client Service | -| ClusterIP :9200 | -| | | -| v | -| +-------------------------------+ | -| | OpenSearch StatefulSet | | -| | replicas: 3 | | -| | | | -| | pod-0 -> PVC-0 | | -| | pod-1 -> PVC-1 | | -| | pod-2 -> PVC-2 | | -| +-------------------------------+ | -| ^ ^ ^ | -| | TLS Transport :9300 | -| +--------+--------+ | -| | | -| Headless Service | -| Peer discovery / stable DNS | -| | -| OpenSearch Dashboards | -| | | -| +--> HTTPS OpenSearch :9200 | -+----------------------------------------+ - ^ - | - | HTTP/HTTPS - | -Nginx Proxy Manager - ^ - | - | Public HTTPS - | - Browser -``` - ---- - -## 3. Kubernetes Cluster Topology - -Target nodes: - -- `k8s-cp-01` — control-plane -- `k8s-storage-01` — storage -- `k8s-worker-01` — worker/storage-capable -- `k8s-worker-02` — worker/storage-capable - -Observed hostname labels include: - -```text -kubernetes.io/hostname -k3s.io/hostname -``` - -Recommended topology key: - -```text -kubernetes.io/hostname -``` - -Storage node taint: - -```text -key: dedicated -value: storage -effect: NoSchedule -``` - -OpenSearch pods require a matching toleration if `k8s-storage-01` is intended as an eligible scheduling target. - ---- - -## 4. Namespace - -Use: - -```text -observability -``` - -Purpose: - -- Organizational separation -- RBAC scope -- Secret scoping -- NetworkPolicy scope -- ResourceQuota/LimitRange scope -- Easier operations and troubleshooting - -A namespace is **not automatically a security boundary**. Actual isolation requires RBAC, NetworkPolicy, Secret access control, and related controls. - ---- - -## 5. OpenSearch Node Count vs Index Replica Count - -Do not confuse Helm `replicas` with OpenSearch `number_of_replicas`. - -### Helm replicas - -Target: - -```text -replicas: 3 -``` - -Meaning: - -```text -3 OpenSearch pods -3 JVMs -3 PVCs -``` - -### OpenSearch index replicas - -For a 3-node cluster: - -```text -1 primary shard + 2 replica shards = 3 shard copies -``` - -Therefore the logical production target is: - -```text -number_of_replicas: 2 -``` - -This is configured at the OpenSearch index/template layer, not automatically from Helm replicas. - ---- - -## 6. StatefulSet Design - -Use a **StatefulSet** for OpenSearch because it provides: - -- Stable ordinal identity -- Stable pod names -- Stable network identity -- Stable PVC association -- Controlled lifecycle behavior - -Example: - -```text -opensearch-0 -opensearch-1 -opensearch-2 -``` - -Each pod must have its own PVC. - -Do not share a single data PVC across all OpenSearch nodes. - ---- - -## 7. Service Design - -OpenSearch needs two logical Service patterns. - -### 7.1 Headless Service - -Purpose: - -- Stable per-pod DNS -- Peer discovery -- StatefulSet network identity -- Node-to-node communication - -Concept: - -```text -clusterIP: None -``` - -Primary transport port: - -```text -9300 -``` - -StatefulSet `serviceName` should reference this headless Service. - -Pod DNS pattern: - -```text -...svc.cluster.local -``` - -### 7.2 Client-facing ClusterIP Service - -Purpose: - -- Dashboards → OpenSearch -- Fluent Bit → OpenSearch -- Administrative REST clients → OpenSearch - -REST/API port: - -```text -9200 -``` - -Clients should use the stable ClusterIP Service instead of targeting one specific OpenSearch pod. - ---- - -## 8. OpenSearch Ports - -### REST / HTTPS - -```text -9200 -``` - -Used by: - -- Fluent Bit -- OpenSearch Dashboards -- API clients -- Administrative requests - -### Transport - -```text -9300 -``` - -Used by: - -- Node-to-node communication -- Cluster membership -- Shard replication -- Cluster-manager communication - -Do not expose `9300` publicly. - -Keep `9200` private/internal unless there is a specific administrative requirement. - ---- - -## 9. Persistence and Storage - -Target: - -```text -StorageClass: longhorn -AccessMode: ReadWriteOnce -PVC size: 120Gi per OpenSearch pod -``` - -For 3 pods: - -```text -3 × 120Gi = 360Gi logical PVC capacity -``` - -If Longhorn volume replica count is 2: - -```text -360Gi × 2 ≈ 720Gi raw replicated capacity -``` - -OpenSearch shard replication and Longhorn block replication are different layers: - -```text -OpenSearch replicas -→ application-level redundancy - -Longhorn replicas -→ block-level redundancy -``` - -Both consume storage. - ---- - -## 10. OpenSearch Data Path - -The PVC must be mounted to the OpenSearch data directory, or `path.data` must explicitly point to the mounted location. - -Unsafe pattern: - -```text -PVC mounted to /mnt -but OpenSearch writes to default data path -``` - -Possible result: - -```text -PVC exists -but index data lives in container filesystem -→ pod recreation can lose data -``` - ---- - -## 11. Resource Sizing - -Target: - -```text -requests: - cpu: 2 - memory: 6Gi - -limits: - cpu: 4 - memory: 8Gi -``` - -For 3 pods: - -```text -Total CPU requests: 6 cores -Total memory requests: 18Gi - -Total CPU limits: 12 cores -Total memory limits: 24Gi -``` - -Scheduler uses resource requests when deciding placement. - ---- - -## 12. JVM Heap - -Use a fixed heap: - -```text -Xms = Xmx -``` - -Discussed starting point: - -```text -~3Gi heap -``` - -Remaining memory is needed for: - -- Filesystem cache -- Direct buffers -- Thread stacks -- JVM metadata -- Native libraries -- Lucene/OpenSearch overhead - -Monitor: - -- Heap utilization -- Old generation pressure -- GC frequency/duration -- Circuit breakers -- Container RSS -- OOMKilled events -- Query latency -- Indexing latency - -Heap belongs in Helm/application configuration, not in a Secret. - ---- - -## 13. Pod Scheduling Strategy - -Target placement: - -```text -k8s-storage-01 → 1 OpenSearch pod -k8s-worker-01 → 1 OpenSearch pod -k8s-worker-02 → 1 OpenSearch pod -k8s-cp-01 → no OpenSearch pod -``` - -### Toleration - -Allows a pod onto a matching tainted node. - -For the storage node, the toleration must match: - -```text -dedicated=storage:NoSchedule -``` - -Toleration allows placement but does not force it. - -### Node Affinity / Selection - -Controls which nodes are eligible. - -Positive selection using workload/storage labels is generally easier to maintain than only blacklisting the control-plane hostname. - -### Pod Anti-Affinity - -Prevents multiple OpenSearch pods from landing on the same Kubernetes node. - -Topology key: - -```text -kubernetes.io/hostname -``` - -Target: - -```text -1 OpenSearch pod per Kubernetes node -``` - -Hard anti-affinity gives stronger failure-domain isolation but can leave a pod Pending if fewer than three eligible nodes are available. - ---- - -## 14. Helm Deployment Workflow - -Do not install the chart blindly. - -Use: - -```text -Official Helm chart - + -Default values - + -Minimal custom values - ↓ -Rendered Kubernetes manifests - ↓ -Review - ↓ -Validate - ↓ -Install -``` - -Recommended workflow: - -1. Inspect available OpenSearch chart versions -2. Inspect chart defaults -3. Build a minimal custom values file -4. Render manifests locally -5. Inspect StatefulSet, Services, PVCs, Secret mounts, affinity, tolerations and probes -6. Validate the rendered YAML -7. Install only after review - -Avoid copying the full default `values.yaml` unless required. - ---- - -## 15. Helm Values to Control - -Production values should intentionally address: - -```text -clusterName -nodeGroup -replicas -singleNode -roles -image version -resources -opensearchJavaOpts -persistence -storageClass -PVC size -antiAffinity -antiAffinityTopologyKey -nodeAffinity / nodeSelector -tolerations -protocol -security configuration -secret mounts -extra volumes -extra init containers -pod management policy -probes -``` - ---- - -## 16. Image Versioning - -Avoid a broad mutable tag such as: - -```text -3 -``` - -Use a tested, pinned OpenSearch release. - -Benefits: - -- Repeatable deployment -- Predictable restarts -- Controlled upgrades -- Easier compatibility management with Dashboards - ---- - -## 17. Kernel Requirement - -Validate `vm.max_map_count` on every eligible node: - -- `k8s-storage-01` -- `k8s-worker-01` -- `k8s-worker-02` - -If Helm sysctl mechanisms are enabled, understand whether the chart uses: - -- privileged init container -- sysctl configuration -- both - -Avoid unnecessary duplicate privileged mechanisms. - ---- - -## 18. Production Security Architecture - -There are multiple TLS boundaries. - -### Browser-facing TLS - -```text -Browser → Nginx Proxy Manager -``` - -Nginx Proxy Manager manages this certificate. - -### OpenSearch REST TLS - -```text -Dashboards → OpenSearch :9200 -Fluent Bit → OpenSearch :9200 -``` - -OpenSearch manages this HTTPS/TLS configuration. - -### OpenSearch Transport TLS - -```text -node-0 ↔ node-1 ↔ node-2 :9300 -``` - -Nginx Proxy Manager does not protect node-to-node traffic. - -Transport TLS should remain enabled in production. - ---- - -## 19. Identity Separation - -Do not reuse the admin account everywhere. - -Recommended identities: - -### Admin - -Use for: - -- Security administration -- Cluster administration -- Emergency operations - -### Dashboards server account - -Use only for Dashboards backend communication. - -### Fluent Bit ingest account - -Grant only required index-write permissions. - -Do not grant: - -- Security administration -- Arbitrary index deletion -- Cluster-wide configuration access -- Read access to unrelated indices - -### Human Dashboard users - -Give only required read/search/visualization permissions. - ---- - -## 20. Bootstrap Password Secret - -The initial admin password should be stored in a Kubernetes Secret in the same namespace as OpenSearch. - -Important: - -- Base64 is encoding, not encryption -- Never commit real Secret values to Git -- Restrict Secret read permission with RBAC -- Consider Kubernetes Secret encryption at rest -- Rotate any credential exposed in chat or logs - -The bootstrap Secret should ideally contain only: - -```text -OPENSEARCH_INITIAL_ADMIN_PASSWORD -``` - ---- - -## 21. Production PKI - -Certificates must be generated by: - -- Your organization PKI -- An internal CA -- A secure OpenSSL-based private CA -- A managed PKI platform - -Minimum certificate set: - -```text -root-ca.pem -root-ca-key.pem - -admin.pem -admin-key.pem - -node-0.pem -node-0-key.pem - -node-1.pem -node-1-key.pem - -node-2.pem -node-2-key.pem -``` - ---- - -## 22. Certificate Responsibilities - -### `root-ca-key.pem` - -Purpose: - -```text -Sign certificates -``` - -Rules: - -```text -NEVER mount into OpenSearch pods -NEVER store permanently in Kubernetes -NEVER commit to Git -Store offline/encrypted -``` - -If compromised, an attacker can issue fake trusted node or admin certificates. - -### `root-ca.pem` - -Purpose: - -```text -Verify certificates issued by the CA -``` - -Can be distributed to: - -- OpenSearch nodes -- Dashboards -- Fluent Bit -- Administrative clients - -### Admin Certificate - -```text -admin.pem -admin-key.pem -``` - -Used for privileged security administration. - -Do not use it as a normal node certificate. - -Do not mount the admin private key into every OpenSearch pod. - -### Node Certificates - -Each node should have a unique identity: - -```text -node-0.pem / node-0-key.pem -node-1.pem / node-1-key.pem -node-2.pem / node-2-key.pem -``` - -Benefits: - -- Better isolation -- Easier rotation -- Easier revocation -- Better auditing - ---- - -## 23. Certificate Generation Lifecycle - -Concept: - -```text -Generate private key - ↓ -Create CSR - ↓ -Sign CSR using Root CA private key - ↓ -Receive CA-signed certificate -``` - -Example: - -```text -node-0-key.pem - ↓ -node-0 CSR - ↓ -signed by root-ca-key.pem - ↓ -node-0.pem -``` - -OpenSearch-compatible private keys should be in the correct PKCS#8 format. - ---- - -## 24. Certificate CN and SAN Design - -Do not use an excessively long Kubernetes FQDN as the certificate CN. - -X.509 CN has size constraints and very long values can cause generation errors. - -Better approach: - -```text -CN: - short node identity - -SAN: - actual Kubernetes DNS names -``` - -Use SAN entries for long pod and Service DNS names. - ---- - -## 25. Node Certificate SAN Requirements - -Each node certificate should include: - -- Its pod DNS -- Relevant short pod DNS names -- Headless Service-related DNS names if needed -- Shared OpenSearch client Service DNS - -Why shared client Service DNS matters: - -```text -Dashboards → https://client-service:9200 -``` - -The ClusterIP may route to any OpenSearch pod. - -Whichever pod responds must present a certificate valid for the requested Service hostname. - ---- - -## 26. Certificate Verification - -Before Kubernetes integration, verify: - -### CA chain - -```text -admin.pem → verifies against root-ca.pem -node-0.pem → verifies against root-ca.pem -node-1.pem → verifies against root-ca.pem -node-2.pem → verifies against root-ca.pem -``` - -### SAN - -Each node certificate contains the intended DNS names. - -### Extended Key Usage - -Node certificates support the required TLS server/client roles. - -### Private key format - -Keys are in OpenSearch-compatible PKCS#8 format. - -### Key/certificate match - -Each certificate matches its own private key. - -### Validity - -Check: - -```text -Not Before -Not After -``` - -### Distinguished Names - -Extract exact RFC2253-formatted certificate DNs for OpenSearch configuration. - ---- - -## 27. Kubernetes Secret Layout - -Recommended separation: - -### Bootstrap Secret - -```text -opensearch-secret -└── OPENSEARCH_INITIAL_ADMIN_PASSWORD -``` - -### Node TLS Secret - -```text -opensearch-node-tls -├── root-ca.pem -├── node-0.pem -├── node-0-key.pem -├── node-1.pem -├── node-1-key.pem -├── node-2.pem -└── node-2-key.pem -``` - -### Admin TLS Secret - -```text -opensearch-admin-tls -├── root-ca.pem -├── admin.pem -└── admin-key.pem -``` - -### Never place in Kubernetes - -```text -root-ca-key.pem -``` - ---- - -## 28. Per-Pod Certificate Selection - -A StatefulSet uses one common pod template. - -If the TLS Secret contains all node certificates, every pod initially sees all files. - -Production pattern: - -```text -Node TLS Secret - ↓ -read-only mount - ↓ -init container - ↓ -detect pod ordinal - ↓ -copy correct node certificate/key - ↓ -private emptyDir - ↓ -main OpenSearch container -``` - -Mapping: - -```text -pod ending in -0 -→ node-0.pem → node.pem -→ node-0-key.pem → node-key.pem - -pod ending in -1 -→ node-1.pem → node.pem -→ node-1-key.pem → node-key.pem - -pod ending in -2 -→ node-2.pem → node.pem -→ node-2-key.pem → node-key.pem -``` - -All pods can then use the same common paths: - -```text -tls/node.pem -tls/node-key.pem -tls/root-ca.pem -``` - ---- - -## 29. OpenSearch TLS Configuration - -The OpenSearch configuration must logically define: - -### Transport TLS - -```text -Node certificate -Node private key -Trusted Root CA -Hostname verification behavior -``` - -### REST TLS - -```text -HTTP TLS enabled -Node/server certificate -Node private key -Trusted Root CA -``` - -### Trusted node identities - -```text -plugins.security.nodes_dn -``` - -Must contain the exact trusted node certificate DNs. - -### Admin identity - -```text -plugins.security.authcz.admin_dn -``` - -Must contain the exact admin certificate DN. - -Do not mix node and admin identities. - ---- - -## 30. Security Configuration Files - -A production Security plugin setup may use: - -```text -config.yml -internal_users.yml -roles.yml -roles_mapping.yml -action_groups.yml -tenants.yml -``` - -Responsibilities: - -### `internal_users.yml` - -- User identities -- Password hashes -- Backend roles - -### `roles.yml` - -- Cluster permissions -- Index permissions -- Tenant permissions - -### `roles_mapping.yml` - -- Maps users/backend roles to OpenSearch roles - -### `config.yml` - -- Authentication domains -- Authorization settings -- Security plugin behavior - ---- - -## 31. OpenSearch Dashboards - -Deploy Dashboards after OpenSearch is healthy. - -Recommended flow: - -```text -Browser - ↓ -Nginx Proxy Manager - ↓ -Dashboards Service - ↓ -Dashboards pod - ↓ -HTTPS OpenSearch Client Service :9200 -``` - -Dashboards should: - -- Trust `root-ca.pem` -- Use a dedicated backend credential -- Not use the admin account -- Connect to the OpenSearch ClusterIP Service - ---- - -## 32. Fluent Bit Integration - -Fluent Bit should connect to the private OpenSearch REST endpoint over HTTPS. - -Production requirements: - -- TLS enabled -- CA verification enabled -- Trust `root-ca.pem` -- Dedicated ingest identity -- Least-privilege role -- No admin credential reuse -- Network access restricted to required paths - ---- - -## 33. Nginx Proxy Manager - -Recommended public flow: - -```text -Browser → Nginx Proxy Manager → Dashboards -``` - -Avoid: - -```text -Public → OpenSearch :9200 -Public → OpenSearch :9300 -``` - -Port `9300` should remain internal. - -Port `9200` should normally remain private. - ---- - -## 34. Probe Design - -Do not use fake endpoints such as `/healthz:8080` unless the application actually exposes them. - -### Startup Probe - -Question: - -```text -Has OpenSearch finished starting? -``` - -### Readiness Probe - -Question: - -```text -Can this pod receive Service traffic? -``` - -### Liveness Probe - -Question: - -```text -Is the process permanently stuck/dead? -``` - -Do not make liveness depend too aggressively on transient cluster health. - -Shard recovery can temporarily cause unhealthy cluster status without meaning the JVM should be restarted. - ---- - -## 35. Pod Management Policy - -Understand: - -```text -OrderedReady -``` - -vs: - -```text -Parallel -``` - -Potential multi-node bootstrap deadlock: - -```text -pod-0 waits for peers -StatefulSet waits for pod-0 Ready -pod-1 and pod-2 never start -``` - -Parallel startup may be more appropriate for a multi-node distributed cluster. - -Also consider peer discovery before readiness. - ---- - -## 36. Deployment Order - -Recommended: - -```text -1. Namespace -2. Bootstrap Secret -3. TLS Secrets -4. Security configuration -5. Helm custom values -6. Rendered-manifest review -7. OpenSearch install -8. Cluster-health validation -9. Dashboards install -10. Nginx Proxy Manager configuration -11. Fluent Bit integration -12. Index templates and replica settings -13. Snapshot backup configuration -14. Monitoring and alerting -``` - ---- - -## 37. Rendered Manifest Review - -Before Helm install, verify: - -```text -[ ] StatefulSet replicas = 3 -[ ] Correct image/version -[ ] Resources present -[ ] Longhorn PVC template present -[ ] 120Gi per pod -[ ] Correct data path -[ ] Headless Service exists -[ ] ClusterIP REST Service exists -[ ] Port 9200 mapping correct -[ ] Port 9300 mapping correct -[ ] Hard anti-affinity rendered -[ ] topology key = kubernetes.io/hostname -[ ] Storage toleration at pod level -[ ] Control-plane excluded -[ ] TLS Secret mounts present -[ ] Per-pod certificate-selection logic present -[ ] TLS paths match opensearch.yml -[ ] nodes_dn matches node certificate DNs -[ ] admin_dn matches admin certificate DN -[ ] Bootstrap Secret reference correct -[ ] Probes match HTTPS behavior -``` - ---- - -## 38. Post-Deployment Validation - -### Kubernetes Layer - -```text -[ ] All 3 pods created -[ ] Each pod on a different Kubernetes node -[ ] No OpenSearch pod on control-plane -[ ] PVCs Bound -[ ] One PVC per pod -[ ] Longhorn volumes healthy -[ ] Init containers completed -[ ] No CrashLoopBackOff -[ ] No OOMKilled -``` - -### TLS Layer - -```text -[ ] Correct certificate selected per pod -[ ] REST endpoint serves HTTPS -[ ] CA verification succeeds -[ ] Transport TLS succeeds -[ ] No SAN mismatch -[ ] No nodes_dn mismatch -[ ] No admin_dn mismatch -``` - -### OpenSearch Layer - -```text -[ ] All 3 nodes joined one cluster -[ ] Cluster-manager elected -[ ] REST API responds -[ ] Authentication succeeds -[ ] Cluster health expected -[ ] Shard allocation correct -``` - -### Storage Layer - -```text -[ ] Data survives pod recreation -[ ] Same ordinal reuses same PVC -[ ] Longhorn replicas healthy and distributed -``` - ---- - -## 39. Failure Testing - -### Delete one OpenSearch pod - -Expected: - -```text -StatefulSet recreates same ordinal -Same PVC reused -Node rejoins cluster -``` - -### Restart one Kubernetes worker - -Expected: - -```text -Remaining nodes continue serving -Shard replica promotion can occur -Missing pod returns/reschedules according to policy -``` - -### Lose the primary-shard node - -With: - -```text -number_of_replicas: 2 -``` - -a remaining replica can be promoted to primary. - -### Verify anti-affinity - -Ensure no two OpenSearch pods share the same hostname. - ---- - -## 40. Cluster Health Interpretation - -### Green - -All primaries and replicas allocated. - -### Yellow - -All primaries allocated, but at least one replica unassigned. - -Typical example: - -```text -single-node cluster -number_of_replicas: 1 -``` - -### Red - -At least one primary shard unavailable. - ---- - -## 41. Common Mistakes Found During This Deployment - -### Mistake: one node with `number_of_replicas: 1` - -Result: - -```text -1 primary + 1 replica -only 1 node -→ replica unassigned -→ yellow cluster -``` - -### Mistake: confusing Helm replicas with shard replicas - -They control different layers. - -### Mistake: using normal ClusterIP as StatefulSet `serviceName` - -StatefulSet stable network identity should use the governing headless Service. - -### Mistake: treating port 80 as OpenSearch's main port - -OpenSearch REST and transport use separate ports. - -### Mistake: mounting the PVC to `/mnt` without changing `path.data` - -The PVC can remain unused. - -### Mistake: putting tolerations under the container - -Tolerations are pod-level settings. - -### Mistake: using pod anti-affinity as a node blacklist - -Pod anti-affinity controls pod co-location, not arbitrary node exclusion. - -### Mistake: assuming Nginx Proxy Manager protects node-to-node traffic - -It does not. - -### Mistake: treating Base64 as encryption - -It is only encoding. - -### Mistake: reusing admin credentials for Fluent Bit and Dashboards - -Violates least privilege. - -### Mistake: very long FQDN as certificate CN - -Can hit X.509 CN length limits. - -Use a short CN and SAN for full DNS names. - ---- - -## 42. Operational Monitoring - -### Kubernetes - -Monitor: - -- Pod restarts -- CPU throttling -- Memory RSS -- OOMKilled events -- PVC usage -- Node pressure -- Scheduling failures - -### OpenSearch - -Monitor: - -- JVM heap pressure -- GC -- Cluster health -- Shard allocation -- Pending tasks -- Indexing latency -- Search latency -- Circuit breakers -- Disk watermarks - -### Longhorn - -Monitor: - -- Volume health -- Replica count -- Replica placement -- Disk usage -- Snapshot growth - -### Security - -Monitor: - -- Certificate expiry -- Authentication failures -- Unauthorized requests -- Admin credential usage -- TLS handshake errors - ---- - -## 43. Certificate Rotation - -Track: - -```text -Not Before -Not After -``` - -Production certificate lifecycle should include: - -- Expiry monitoring -- Renewal schedule -- Secure CA access -- Leaf-certificate replacement procedure -- Emergency replacement/revocation procedure -- Admin certificate rotation -- Protected CA backup - ---- - -## 44. Backup Strategy - -OpenSearch shard replicas are **not backups**. - -They do not protect from: - -- Accidental deletion -- Malicious deletion -- Corruption replicated across nodes -- Entire cluster loss - -Use snapshots to independent storage such as object storage/S3-compatible storage. - -Include: - -- Separate credentials -- Snapshot retention -- Snapshot lifecycle -- Restore testing - ---- - -## 45. Security Hardening Checklist - -```text -[ ] Security plugin enabled -[ ] Demo certificates removed -[ ] Custom internal Root CA used -[ ] Unique node certificates -[ ] Separate admin certificate -[ ] root-ca-key.pem stored offline -[ ] REST HTTPS enabled -[ ] Transport TLS enabled -[ ] Port 9300 not externally exposed -[ ] Port 9200 private/internal -[ ] Dashboards exposed through Nginx Proxy Manager -[ ] Dedicated Fluent Bit user -[ ] Dedicated Dashboards server user -[ ] Human users use least-privilege roles -[ ] Secret access protected by RBAC -[ ] Kubernetes Secret encryption at rest considered -[ ] NetworkPolicy considered -[ ] Certificate expiry monitored -``` - ---- - -## 46. Go-Live Checklist - -### Kubernetes - -```text -[ ] Namespace = observability -[ ] 3 eligible OpenSearch nodes -[ ] Storage taint/toleration validated -[ ] Hard anti-affinity validated -[ ] Control-plane excluded -[ ] Longhorn healthy -[ ] vm.max_map_count validated -``` - -### Helm - -```text -[ ] Chart version pinned -[ ] OpenSearch version pinned -[ ] Custom values reviewed -[ ] Rendered YAML reviewed -[ ] No unexpected resources -``` - -### Storage - -```text -[ ] 120Gi × 3 planned -[ ] Longhorn replication capacity calculated -[ ] Sufficient disk headroom -``` - -### TLS - -```text -[ ] Root CA generated -[ ] Root CA key offline -[ ] Admin certificate valid -[ ] Node certificates valid -[ ] SANs correct -[ ] Certificate/key pairs match -[ ] DNs extracted -[ ] TLS Secrets created -``` - -### OpenSearch - -```text -[ ] 3 nodes joined -[ ] Cluster-manager election healthy -[ ] HTTPS works -[ ] Authentication works -[ ] Shard replica policy set -[ ] Cluster health verified -``` - -### Dashboards / Fluent Bit - -```text -[ ] Dashboards trusts internal CA -[ ] Fluent Bit trusts internal CA -[ ] Admin password not reused -[ ] Nginx Proxy Manager exposes Dashboards only -``` - ---- - -## 47. Recommended Next Steps - -Continue in this order: - -1. Verify generated certificates -2. Extract exact admin and node DNs -3. Create Kubernetes TLS Secrets -4. Build per-pod certificate-selection logic -5. Add custom TLS configuration to Helm values -6. Render Helm manifests -7. Review StatefulSet and Services -8. Validate node placement -9. Install OpenSearch -10. Verify the 3-node HTTPS/TLS cluster -11. Deploy OpenSearch Dashboards -12. Configure Nginx Proxy Manager -13. Create Fluent Bit least-privilege ingest identity -14. Connect Fluent Bit over HTTPS -15. Configure index templates and shard replica policy -16. Configure snapshots/backups -17. Add monitoring and certificate-expiry alerts - ---- - -## 48. Troubleshooting Decision Tree - -### Pod Pending - -Investigate: - -```text -Node affinity -Pod anti-affinity -Taints -Tolerations -CPU/memory requests -PVC scheduling/topology -``` - -### Pod CrashLoopBackOff - -Investigate: - -```text -JVM heap -vm.max_map_count -TLS file paths -Private-key format -Certificate permissions -Bootstrap admin password -OpenSearch logs -``` - -### Nodes do not form one cluster - -Investigate: - -```text -Headless Service -DNS resolution -Transport port 9300 -Transport TLS -Certificate SAN -nodes_dn -Discovery settings -Initial cluster bootstrap settings -``` - -### HTTPS fails - -Investigate: - -```text -REST TLS enabled -Certificate path -Private key path -Trusted Root CA -Client hostname -SAN -Certificate expiry -``` - -### Dashboards cannot connect - -Investigate: - -```text -OpenSearch client Service -Port 9200 -HTTPS -Root CA trust -Dashboards backend credential -NetworkPolicy -``` - -### Fluent Bit cannot ingest - -Investigate: - -```text -OpenSearch endpoint -HTTPS -CA verification -Credential -Role permissions -Index permissions -Network path -``` - ---- - -## 49. Core Lessons - -```text -1. Stateful identity is not the same as pod IP. -2. StatefulSet provides stable identity; PVC stores persistent data. -3. Headless Service is for peer discovery and per-pod DNS. -4. ClusterIP Service is for normal clients. -5. Port 9200 and 9300 have different purposes. -6. Kubernetes replicas and OpenSearch shard replicas are different. -7. Redundancy works only when copies are separated across failure domains. -8. Toleration permits; affinity selects; anti-affinity separates. -9. A PVC is useless if the application writes somewhere else. -10. Helm should be rendered and reviewed before install. -11. Base64 is not encryption. -12. Reverse-proxy TLS does not protect internal OpenSearch transport. -13. Production TLS requires controlled PKI and certificate lifecycle. -14. Root CA private key never belongs in OpenSearch pods. -15. Unique node certificates are better than sharing one node key. -16. SANs are critical for DNS-based TLS verification. -17. Admin, Dashboards, and Fluent Bit should use separate identities. -18. OpenSearch replication is not a backup. -``` - ---- - -## 50. Final Target State - -```text -Kubernetes: - 3 OpenSearch pods - 3 separate Kubernetes nodes - 3 separate Longhorn PVCs - no OpenSearch pod on control-plane - -OpenSearch: - 3-node cluster - transport TLS - REST HTTPS - correct shard replication - healthy cluster-manager election - -Security: - custom internal CA - unique node certificates - separate admin certificate - dedicated Dashboards credential - dedicated Fluent Bit credential - least privilege - -Access: - Browser → Nginx Proxy Manager → Dashboards - Dashboards → HTTPS OpenSearch ClusterIP - Fluent Bit → HTTPS OpenSearch private endpoint - -Operations: - monitoring - snapshots/backups - certificate expiry tracking - restore testing -``` - ---- - -## 51. Current Project Status - -At the end of this chat, the design has reached this point: - -```text -[✓] 3-node production topology selected -[✓] Longhorn selected for PVCs -[✓] 120Gi per PVC planned -[✓] Resource requests/limits discussed -[✓] JVM heap sizing discussed -[✓] Hard anti-affinity selected -[✓] Storage taint/toleration verified -[✓] HTTPS selected -[✓] Production TLS selected -[✓] Root CA/admin/node certificates generated -[ ] Kubernetes TLS Secrets integration -[ ] Per-pod certificate selection -[ ] OpenSearch TLS settings in Helm values -[ ] Rendered manifest validation -[ ] OpenSearch deployment -[ ] Dashboards deployment -[ ] Fluent Bit integration -``` - -The immediate next phase is: - -```text -Certificates - ↓ -Kubernetes TLS Secrets - ↓ -Helm certificate mounts - ↓ -OpenSearch TLS configuration - ↓ -Render and validate - ↓ -Deploy -```