# SOP: Production OpenSearch Deployment on Kubernetes Using Helm ## 1. Purpose Ei SOP-er goal holo Kubernetes cluster-e production-grade OpenSearch deploy kora using Helm, with: - 3-node OpenSearch cluster - Longhorn persistent storage - OpenSearch Dashboards - Fluent Bit log ingestion - HTTPS/TLS - Node-to-node transport encryption - Dedicated `observability` namespace - Controlled pod scheduling - Production-oriented security - Nginx Proxy Manager for browser-facing HTTPS > **Learning-mode note:** Ei SOP-te exact executable command syntax intentionally deya hoyni. Protiti step-e command-er purpose, tool, expected output, ebong validation deya ache. Exact commands chaile `SHOW SOLUTION` type korun. --- ## 2. Final Architecture ```text cPanel / Log Sources | | Fluent Bit | HTTPS :9200 v +----------------------------------------+ | Kubernetes Cluster | | Namespace: observability | | | | OpenSearch Client Service | | ClusterIP :9200 | | | | | v | | +-------------------------------+ | | | OpenSearch StatefulSet | | | | replicas: 3 | | | | | | | | pod-0 -> PVC-0 | | | | pod-1 -> PVC-1 | | | | pod-2 -> PVC-2 | | | +-------------------------------+ | | ^ ^ ^ | | | TLS Transport :9300 | | +--------+--------+ | | | | | Headless Service | | Peer discovery / stable DNS | | | | OpenSearch Dashboards | | | | | +--> HTTPS OpenSearch :9200 | +----------------------------------------+ ^ | | HTTP/HTTPS | Nginx Proxy Manager ^ | | Public HTTPS | Browser ``` --- ## 3. Kubernetes Cluster Topology Target nodes: - `k8s-cp-01` — control-plane - `k8s-storage-01` — storage - `k8s-worker-01` — worker/storage-capable - `k8s-worker-02` — worker/storage-capable Observed hostname labels include: ```text kubernetes.io/hostname k3s.io/hostname ``` Recommended topology key: ```text kubernetes.io/hostname ``` Storage node taint: ```text key: dedicated value: storage effect: NoSchedule ``` OpenSearch pods require a matching toleration if `k8s-storage-01` is intended as an eligible scheduling target. --- ## 4. Namespace Use: ```text observability ``` Purpose: - Organizational separation - RBAC scope - Secret scoping - NetworkPolicy scope - ResourceQuota/LimitRange scope - Easier operations and troubleshooting A namespace is **not automatically a security boundary**. Actual isolation requires RBAC, NetworkPolicy, Secret access control, and related controls. --- ## 5. OpenSearch Node Count vs Index Replica Count Do not confuse Helm `replicas` with OpenSearch `number_of_replicas`. ### Helm replicas Target: ```text replicas: 3 ``` Meaning: ```text 3 OpenSearch pods 3 JVMs 3 PVCs ``` ### OpenSearch index replicas For a 3-node cluster: ```text 1 primary shard + 2 replica shards = 3 shard copies ``` Therefore the logical production target is: ```text number_of_replicas: 2 ``` This is configured at the OpenSearch index/template layer, not automatically from Helm replicas. --- ## 6. StatefulSet Design Use a **StatefulSet** for OpenSearch because it provides: - Stable ordinal identity - Stable pod names - Stable network identity - Stable PVC association - Controlled lifecycle behavior Example: ```text opensearch-0 opensearch-1 opensearch-2 ``` Each pod must have its own PVC. Do not share a single data PVC across all OpenSearch nodes. --- ## 7. Service Design OpenSearch needs two logical Service patterns. ### 7.1 Headless Service Purpose: - Stable per-pod DNS - Peer discovery - StatefulSet network identity - Node-to-node communication Concept: ```text clusterIP: None ``` Primary transport port: ```text 9300 ``` StatefulSet `serviceName` should reference this headless Service. Pod DNS pattern: ```text ...svc.cluster.local ``` ### 7.2 Client-facing ClusterIP Service Purpose: - Dashboards → OpenSearch - Fluent Bit → OpenSearch - Administrative REST clients → OpenSearch REST/API port: ```text 9200 ``` Clients should use the stable ClusterIP Service instead of targeting one specific OpenSearch pod. --- ## 8. OpenSearch Ports ### REST / HTTPS ```text 9200 ``` Used by: - Fluent Bit - OpenSearch Dashboards - API clients - Administrative requests ### Transport ```text 9300 ``` Used by: - Node-to-node communication - Cluster membership - Shard replication - Cluster-manager communication Do not expose `9300` publicly. Keep `9200` private/internal unless there is a specific administrative requirement. --- ## 9. Persistence and Storage Target: ```text StorageClass: longhorn AccessMode: ReadWriteOnce PVC size: 120Gi per OpenSearch pod ``` For 3 pods: ```text 3 × 120Gi = 360Gi logical PVC capacity ``` If Longhorn volume replica count is 2: ```text 360Gi × 2 ≈ 720Gi raw replicated capacity ``` OpenSearch shard replication and Longhorn block replication are different layers: ```text OpenSearch replicas → application-level redundancy Longhorn replicas → block-level redundancy ``` Both consume storage. --- ## 10. OpenSearch Data Path The PVC must be mounted to the OpenSearch data directory, or `path.data` must explicitly point to the mounted location. Unsafe pattern: ```text PVC mounted to /mnt but OpenSearch writes to default data path ``` Possible result: ```text PVC exists but index data lives in container filesystem → pod recreation can lose data ``` --- ## 11. Resource Sizing Target: ```text requests: cpu: 2 memory: 6Gi limits: cpu: 4 memory: 8Gi ``` For 3 pods: ```text Total CPU requests: 6 cores Total memory requests: 18Gi Total CPU limits: 12 cores Total memory limits: 24Gi ``` Scheduler uses resource requests when deciding placement. --- ## 12. JVM Heap Use a fixed heap: ```text Xms = Xmx ``` Discussed starting point: ```text ~3Gi heap ``` Remaining memory is needed for: - Filesystem cache - Direct buffers - Thread stacks - JVM metadata - Native libraries - Lucene/OpenSearch overhead Monitor: - Heap utilization - Old generation pressure - GC frequency/duration - Circuit breakers - Container RSS - OOMKilled events - Query latency - Indexing latency Heap belongs in Helm/application configuration, not in a Secret. --- ## 13. Pod Scheduling Strategy Target placement: ```text k8s-storage-01 → 1 OpenSearch pod k8s-worker-01 → 1 OpenSearch pod k8s-worker-02 → 1 OpenSearch pod k8s-cp-01 → no OpenSearch pod ``` ### Toleration Allows a pod onto a matching tainted node. For the storage node, the toleration must match: ```text dedicated=storage:NoSchedule ``` Toleration allows placement but does not force it. ### Node Affinity / Selection Controls which nodes are eligible. Positive selection using workload/storage labels is generally easier to maintain than only blacklisting the control-plane hostname. ### Pod Anti-Affinity Prevents multiple OpenSearch pods from landing on the same Kubernetes node. Topology key: ```text kubernetes.io/hostname ``` Target: ```text 1 OpenSearch pod per Kubernetes node ``` Hard anti-affinity gives stronger failure-domain isolation but can leave a pod Pending if fewer than three eligible nodes are available. --- ## 14. Helm Deployment Workflow Do not install the chart blindly. Use: ```text Official Helm chart + Default values + Minimal custom values ↓ Rendered Kubernetes manifests ↓ Review ↓ Validate ↓ Install ``` Recommended workflow: 1. Inspect available OpenSearch chart versions 2. Inspect chart defaults 3. Build a minimal custom values file 4. Render manifests locally 5. Inspect StatefulSet, Services, PVCs, Secret mounts, affinity, tolerations and probes 6. Validate the rendered YAML 7. Install only after review Avoid copying the full default `values.yaml` unless required. --- ## 15. Helm Values to Control Production values should intentionally address: ```text clusterName nodeGroup replicas singleNode roles image version resources opensearchJavaOpts persistence storageClass PVC size antiAffinity antiAffinityTopologyKey nodeAffinity / nodeSelector tolerations protocol security configuration secret mounts extra volumes extra init containers pod management policy probes ``` --- ## 16. Image Versioning Avoid a broad mutable tag such as: ```text 3 ``` Use a tested, pinned OpenSearch release. Benefits: - Repeatable deployment - Predictable restarts - Controlled upgrades - Easier compatibility management with Dashboards --- ## 17. Kernel Requirement Validate `vm.max_map_count` on every eligible node: - `k8s-storage-01` - `k8s-worker-01` - `k8s-worker-02` If Helm sysctl mechanisms are enabled, understand whether the chart uses: - privileged init container - sysctl configuration - both Avoid unnecessary duplicate privileged mechanisms. --- ## 18. Production Security Architecture There are multiple TLS boundaries. ### Browser-facing TLS ```text Browser → Nginx Proxy Manager ``` Nginx Proxy Manager manages this certificate. ### OpenSearch REST TLS ```text Dashboards → OpenSearch :9200 Fluent Bit → OpenSearch :9200 ``` OpenSearch manages this HTTPS/TLS configuration. ### OpenSearch Transport TLS ```text node-0 ↔ node-1 ↔ node-2 :9300 ``` Nginx Proxy Manager does not protect node-to-node traffic. Transport TLS should remain enabled in production. --- ## 19. Identity Separation Do not reuse the admin account everywhere. Recommended identities: ### Admin Use for: - Security administration - Cluster administration - Emergency operations ### Dashboards server account Use only for Dashboards backend communication. ### Fluent Bit ingest account Grant only required index-write permissions. Do not grant: - Security administration - Arbitrary index deletion - Cluster-wide configuration access - Read access to unrelated indices ### Human Dashboard users Give only required read/search/visualization permissions. --- ## 20. Bootstrap Password Secret The initial admin password should be stored in a Kubernetes Secret in the same namespace as OpenSearch. Important: - Base64 is encoding, not encryption - Never commit real Secret values to Git - Restrict Secret read permission with RBAC - Consider Kubernetes Secret encryption at rest - Rotate any credential exposed in chat or logs The bootstrap Secret should ideally contain only: ```text OPENSEARCH_INITIAL_ADMIN_PASSWORD ``` --- ## 21. Production PKI Certificates must be generated by: - Your organization PKI - An internal CA - A secure OpenSSL-based private CA - A managed PKI platform Minimum certificate set: ```text root-ca.pem root-ca-key.pem admin.pem admin-key.pem node-0.pem node-0-key.pem node-1.pem node-1-key.pem node-2.pem node-2-key.pem ``` --- ## 22. Certificate Responsibilities ### `root-ca-key.pem` Purpose: ```text Sign certificates ``` Rules: ```text NEVER mount into OpenSearch pods NEVER store permanently in Kubernetes NEVER commit to Git Store offline/encrypted ``` If compromised, an attacker can issue fake trusted node or admin certificates. ### `root-ca.pem` Purpose: ```text Verify certificates issued by the CA ``` Can be distributed to: - OpenSearch nodes - Dashboards - Fluent Bit - Administrative clients ### Admin Certificate ```text admin.pem admin-key.pem ``` Used for privileged security administration. Do not use it as a normal node certificate. Do not mount the admin private key into every OpenSearch pod. ### Node Certificates Each node should have a unique identity: ```text node-0.pem / node-0-key.pem node-1.pem / node-1-key.pem node-2.pem / node-2-key.pem ``` Benefits: - Better isolation - Easier rotation - Easier revocation - Better auditing --- ## 23. Certificate Generation Lifecycle Concept: ```text Generate private key ↓ Create CSR ↓ Sign CSR using Root CA private key ↓ Receive CA-signed certificate ``` Example: ```text node-0-key.pem ↓ node-0 CSR ↓ signed by root-ca-key.pem ↓ node-0.pem ``` OpenSearch-compatible private keys should be in the correct PKCS#8 format. --- ## 24. Certificate CN and SAN Design Do not use an excessively long Kubernetes FQDN as the certificate CN. X.509 CN has size constraints and very long values can cause generation errors. Better approach: ```text CN: short node identity SAN: actual Kubernetes DNS names ``` Use SAN entries for long pod and Service DNS names. --- ## 25. Node Certificate SAN Requirements Each node certificate should include: - Its pod DNS - Relevant short pod DNS names - Headless Service-related DNS names if needed - Shared OpenSearch client Service DNS Why shared client Service DNS matters: ```text Dashboards → https://client-service:9200 ``` The ClusterIP may route to any OpenSearch pod. Whichever pod responds must present a certificate valid for the requested Service hostname. --- ## 26. Certificate Verification Before Kubernetes integration, verify: ### CA chain ```text admin.pem → verifies against root-ca.pem node-0.pem → verifies against root-ca.pem node-1.pem → verifies against root-ca.pem node-2.pem → verifies against root-ca.pem ``` ### SAN Each node certificate contains the intended DNS names. ### Extended Key Usage Node certificates support the required TLS server/client roles. ### Private key format Keys are in OpenSearch-compatible PKCS#8 format. ### Key/certificate match Each certificate matches its own private key. ### Validity Check: ```text Not Before Not After ``` ### Distinguished Names Extract exact RFC2253-formatted certificate DNs for OpenSearch configuration. --- ## 27. Kubernetes Secret Layout Recommended separation: ### Bootstrap Secret ```text opensearch-secret └── OPENSEARCH_INITIAL_ADMIN_PASSWORD ``` ### Node TLS Secret ```text opensearch-node-tls ├── root-ca.pem ├── node-0.pem ├── node-0-key.pem ├── node-1.pem ├── node-1-key.pem ├── node-2.pem └── node-2-key.pem ``` ### Admin TLS Secret ```text opensearch-admin-tls ├── root-ca.pem ├── admin.pem └── admin-key.pem ``` ### Never place in Kubernetes ```text root-ca-key.pem ``` --- ## 28. Per-Pod Certificate Selection A StatefulSet uses one common pod template. If the TLS Secret contains all node certificates, every pod initially sees all files. Production pattern: ```text Node TLS Secret ↓ read-only mount ↓ init container ↓ detect pod ordinal ↓ copy correct node certificate/key ↓ private emptyDir ↓ main OpenSearch container ``` Mapping: ```text pod ending in -0 → node-0.pem → node.pem → node-0-key.pem → node-key.pem pod ending in -1 → node-1.pem → node.pem → node-1-key.pem → node-key.pem pod ending in -2 → node-2.pem → node.pem → node-2-key.pem → node-key.pem ``` All pods can then use the same common paths: ```text tls/node.pem tls/node-key.pem tls/root-ca.pem ``` --- ## 29. OpenSearch TLS Configuration The OpenSearch configuration must logically define: ### Transport TLS ```text Node certificate Node private key Trusted Root CA Hostname verification behavior ``` ### REST TLS ```text HTTP TLS enabled Node/server certificate Node private key Trusted Root CA ``` ### Trusted node identities ```text plugins.security.nodes_dn ``` Must contain the exact trusted node certificate DNs. ### Admin identity ```text plugins.security.authcz.admin_dn ``` Must contain the exact admin certificate DN. Do not mix node and admin identities. --- ## 30. Security Configuration Files A production Security plugin setup may use: ```text config.yml internal_users.yml roles.yml roles_mapping.yml action_groups.yml tenants.yml ``` Responsibilities: ### `internal_users.yml` - User identities - Password hashes - Backend roles ### `roles.yml` - Cluster permissions - Index permissions - Tenant permissions ### `roles_mapping.yml` - Maps users/backend roles to OpenSearch roles ### `config.yml` - Authentication domains - Authorization settings - Security plugin behavior --- ## 31. OpenSearch Dashboards Deploy Dashboards after OpenSearch is healthy. Recommended flow: ```text Browser ↓ Nginx Proxy Manager ↓ Dashboards Service ↓ Dashboards pod ↓ HTTPS OpenSearch Client Service :9200 ``` Dashboards should: - Trust `root-ca.pem` - Use a dedicated backend credential - Not use the admin account - Connect to the OpenSearch ClusterIP Service --- ## 32. Fluent Bit Integration Fluent Bit should connect to the private OpenSearch REST endpoint over HTTPS. Production requirements: - TLS enabled - CA verification enabled - Trust `root-ca.pem` - Dedicated ingest identity - Least-privilege role - No admin credential reuse - Network access restricted to required paths --- ## 33. Nginx Proxy Manager Recommended public flow: ```text Browser → Nginx Proxy Manager → Dashboards ``` Avoid: ```text Public → OpenSearch :9200 Public → OpenSearch :9300 ``` Port `9300` should remain internal. Port `9200` should normally remain private. --- ## 34. Probe Design Do not use fake endpoints such as `/healthz:8080` unless the application actually exposes them. ### Startup Probe Question: ```text Has OpenSearch finished starting? ``` ### Readiness Probe Question: ```text Can this pod receive Service traffic? ``` ### Liveness Probe Question: ```text Is the process permanently stuck/dead? ``` Do not make liveness depend too aggressively on transient cluster health. Shard recovery can temporarily cause unhealthy cluster status without meaning the JVM should be restarted. --- ## 35. Pod Management Policy Understand: ```text OrderedReady ``` vs: ```text Parallel ``` Potential multi-node bootstrap deadlock: ```text pod-0 waits for peers StatefulSet waits for pod-0 Ready pod-1 and pod-2 never start ``` Parallel startup may be more appropriate for a multi-node distributed cluster. Also consider peer discovery before readiness. --- ## 36. Deployment Order Recommended: ```text 1. Namespace 2. Bootstrap Secret 3. TLS Secrets 4. Security configuration 5. Helm custom values 6. Rendered-manifest review 7. OpenSearch install 8. Cluster-health validation 9. Dashboards install 10. Nginx Proxy Manager configuration 11. Fluent Bit integration 12. Index templates and replica settings 13. Snapshot backup configuration 14. Monitoring and alerting ``` --- ## 37. Rendered Manifest Review Before Helm install, verify: ```text [ ] StatefulSet replicas = 3 [ ] Correct image/version [ ] Resources present [ ] Longhorn PVC template present [ ] 120Gi per pod [ ] Correct data path [ ] Headless Service exists [ ] ClusterIP REST Service exists [ ] Port 9200 mapping correct [ ] Port 9300 mapping correct [ ] Hard anti-affinity rendered [ ] topology key = kubernetes.io/hostname [ ] Storage toleration at pod level [ ] Control-plane excluded [ ] TLS Secret mounts present [ ] Per-pod certificate-selection logic present [ ] TLS paths match opensearch.yml [ ] nodes_dn matches node certificate DNs [ ] admin_dn matches admin certificate DN [ ] Bootstrap Secret reference correct [ ] Probes match HTTPS behavior ``` --- ## 38. Post-Deployment Validation ### Kubernetes Layer ```text [ ] All 3 pods created [ ] Each pod on a different Kubernetes node [ ] No OpenSearch pod on control-plane [ ] PVCs Bound [ ] One PVC per pod [ ] Longhorn volumes healthy [ ] Init containers completed [ ] No CrashLoopBackOff [ ] No OOMKilled ``` ### TLS Layer ```text [ ] Correct certificate selected per pod [ ] REST endpoint serves HTTPS [ ] CA verification succeeds [ ] Transport TLS succeeds [ ] No SAN mismatch [ ] No nodes_dn mismatch [ ] No admin_dn mismatch ``` ### OpenSearch Layer ```text [ ] All 3 nodes joined one cluster [ ] Cluster-manager elected [ ] REST API responds [ ] Authentication succeeds [ ] Cluster health expected [ ] Shard allocation correct ``` ### Storage Layer ```text [ ] Data survives pod recreation [ ] Same ordinal reuses same PVC [ ] Longhorn replicas healthy and distributed ``` --- ## 39. Failure Testing ### Delete one OpenSearch pod Expected: ```text StatefulSet recreates same ordinal Same PVC reused Node rejoins cluster ``` ### Restart one Kubernetes worker Expected: ```text Remaining nodes continue serving Shard replica promotion can occur Missing pod returns/reschedules according to policy ``` ### Lose the primary-shard node With: ```text number_of_replicas: 2 ``` a remaining replica can be promoted to primary. ### Verify anti-affinity Ensure no two OpenSearch pods share the same hostname. --- ## 40. Cluster Health Interpretation ### Green All primaries and replicas allocated. ### Yellow All primaries allocated, but at least one replica unassigned. Typical example: ```text single-node cluster number_of_replicas: 1 ``` ### Red At least one primary shard unavailable. --- ## 41. Common Mistakes Found During This Deployment ### Mistake: one node with `number_of_replicas: 1` Result: ```text 1 primary + 1 replica only 1 node → replica unassigned → yellow cluster ``` ### Mistake: confusing Helm replicas with shard replicas They control different layers. ### Mistake: using normal ClusterIP as StatefulSet `serviceName` StatefulSet stable network identity should use the governing headless Service. ### Mistake: treating port 80 as OpenSearch's main port OpenSearch REST and transport use separate ports. ### Mistake: mounting the PVC to `/mnt` without changing `path.data` The PVC can remain unused. ### Mistake: putting tolerations under the container Tolerations are pod-level settings. ### Mistake: using pod anti-affinity as a node blacklist Pod anti-affinity controls pod co-location, not arbitrary node exclusion. ### Mistake: assuming Nginx Proxy Manager protects node-to-node traffic It does not. ### Mistake: treating Base64 as encryption It is only encoding. ### Mistake: reusing admin credentials for Fluent Bit and Dashboards Violates least privilege. ### Mistake: very long FQDN as certificate CN Can hit X.509 CN length limits. Use a short CN and SAN for full DNS names. --- ## 42. Operational Monitoring ### Kubernetes Monitor: - Pod restarts - CPU throttling - Memory RSS - OOMKilled events - PVC usage - Node pressure - Scheduling failures ### OpenSearch Monitor: - JVM heap pressure - GC - Cluster health - Shard allocation - Pending tasks - Indexing latency - Search latency - Circuit breakers - Disk watermarks ### Longhorn Monitor: - Volume health - Replica count - Replica placement - Disk usage - Snapshot growth ### Security Monitor: - Certificate expiry - Authentication failures - Unauthorized requests - Admin credential usage - TLS handshake errors --- ## 43. Certificate Rotation Track: ```text Not Before Not After ``` Production certificate lifecycle should include: - Expiry monitoring - Renewal schedule - Secure CA access - Leaf-certificate replacement procedure - Emergency replacement/revocation procedure - Admin certificate rotation - Protected CA backup --- ## 44. Backup Strategy OpenSearch shard replicas are **not backups**. They do not protect from: - Accidental deletion - Malicious deletion - Corruption replicated across nodes - Entire cluster loss Use snapshots to independent storage such as object storage/S3-compatible storage. Include: - Separate credentials - Snapshot retention - Snapshot lifecycle - Restore testing --- ## 45. Security Hardening Checklist ```text [ ] Security plugin enabled [ ] Demo certificates removed [ ] Custom internal Root CA used [ ] Unique node certificates [ ] Separate admin certificate [ ] root-ca-key.pem stored offline [ ] REST HTTPS enabled [ ] Transport TLS enabled [ ] Port 9300 not externally exposed [ ] Port 9200 private/internal [ ] Dashboards exposed through Nginx Proxy Manager [ ] Dedicated Fluent Bit user [ ] Dedicated Dashboards server user [ ] Human users use least-privilege roles [ ] Secret access protected by RBAC [ ] Kubernetes Secret encryption at rest considered [ ] NetworkPolicy considered [ ] Certificate expiry monitored ``` --- ## 46. Go-Live Checklist ### Kubernetes ```text [ ] Namespace = observability [ ] 3 eligible OpenSearch nodes [ ] Storage taint/toleration validated [ ] Hard anti-affinity validated [ ] Control-plane excluded [ ] Longhorn healthy [ ] vm.max_map_count validated ``` ### Helm ```text [ ] Chart version pinned [ ] OpenSearch version pinned [ ] Custom values reviewed [ ] Rendered YAML reviewed [ ] No unexpected resources ``` ### Storage ```text [ ] 120Gi × 3 planned [ ] Longhorn replication capacity calculated [ ] Sufficient disk headroom ``` ### TLS ```text [ ] Root CA generated [ ] Root CA key offline [ ] Admin certificate valid [ ] Node certificates valid [ ] SANs correct [ ] Certificate/key pairs match [ ] DNs extracted [ ] TLS Secrets created ``` ### OpenSearch ```text [ ] 3 nodes joined [ ] Cluster-manager election healthy [ ] HTTPS works [ ] Authentication works [ ] Shard replica policy set [ ] Cluster health verified ``` ### Dashboards / Fluent Bit ```text [ ] Dashboards trusts internal CA [ ] Fluent Bit trusts internal CA [ ] Admin password not reused [ ] Nginx Proxy Manager exposes Dashboards only ``` --- ## 47. Recommended Next Steps Continue in this order: 1. Verify generated certificates 2. Extract exact admin and node DNs 3. Create Kubernetes TLS Secrets 4. Build per-pod certificate-selection logic 5. Add custom TLS configuration to Helm values 6. Render Helm manifests 7. Review StatefulSet and Services 8. Validate node placement 9. Install OpenSearch 10. Verify the 3-node HTTPS/TLS cluster 11. Deploy OpenSearch Dashboards 12. Configure Nginx Proxy Manager 13. Create Fluent Bit least-privilege ingest identity 14. Connect Fluent Bit over HTTPS 15. Configure index templates and shard replica policy 16. Configure snapshots/backups 17. Add monitoring and certificate-expiry alerts --- ## 48. Troubleshooting Decision Tree ### Pod Pending Investigate: ```text Node affinity Pod anti-affinity Taints Tolerations CPU/memory requests PVC scheduling/topology ``` ### Pod CrashLoopBackOff Investigate: ```text JVM heap vm.max_map_count TLS file paths Private-key format Certificate permissions Bootstrap admin password OpenSearch logs ``` ### Nodes do not form one cluster Investigate: ```text Headless Service DNS resolution Transport port 9300 Transport TLS Certificate SAN nodes_dn Discovery settings Initial cluster bootstrap settings ``` ### HTTPS fails Investigate: ```text REST TLS enabled Certificate path Private key path Trusted Root CA Client hostname SAN Certificate expiry ``` ### Dashboards cannot connect Investigate: ```text OpenSearch client Service Port 9200 HTTPS Root CA trust Dashboards backend credential NetworkPolicy ``` ### Fluent Bit cannot ingest Investigate: ```text OpenSearch endpoint HTTPS CA verification Credential Role permissions Index permissions Network path ``` --- ## 49. Core Lessons ```text 1. Stateful identity is not the same as pod IP. 2. StatefulSet provides stable identity; PVC stores persistent data. 3. Headless Service is for peer discovery and per-pod DNS. 4. ClusterIP Service is for normal clients. 5. Port 9200 and 9300 have different purposes. 6. Kubernetes replicas and OpenSearch shard replicas are different. 7. Redundancy works only when copies are separated across failure domains. 8. Toleration permits; affinity selects; anti-affinity separates. 9. A PVC is useless if the application writes somewhere else. 10. Helm should be rendered and reviewed before install. 11. Base64 is not encryption. 12. Reverse-proxy TLS does not protect internal OpenSearch transport. 13. Production TLS requires controlled PKI and certificate lifecycle. 14. Root CA private key never belongs in OpenSearch pods. 15. Unique node certificates are better than sharing one node key. 16. SANs are critical for DNS-based TLS verification. 17. Admin, Dashboards, and Fluent Bit should use separate identities. 18. OpenSearch replication is not a backup. ``` --- ## 50. Final Target State ```text Kubernetes: 3 OpenSearch pods 3 separate Kubernetes nodes 3 separate Longhorn PVCs no OpenSearch pod on control-plane OpenSearch: 3-node cluster transport TLS REST HTTPS correct shard replication healthy cluster-manager election Security: custom internal CA unique node certificates separate admin certificate dedicated Dashboards credential dedicated Fluent Bit credential least privilege Access: Browser → Nginx Proxy Manager → Dashboards Dashboards → HTTPS OpenSearch ClusterIP Fluent Bit → HTTPS OpenSearch private endpoint Operations: monitoring snapshots/backups certificate expiry tracking restore testing ``` --- ## 51. Current Project Status At the end of this chat, the design has reached this point: ```text [✓] 3-node production topology selected [✓] Longhorn selected for PVCs [✓] 120Gi per PVC planned [✓] Resource requests/limits discussed [✓] JVM heap sizing discussed [✓] Hard anti-affinity selected [✓] Storage taint/toleration verified [✓] HTTPS selected [✓] Production TLS selected [✓] Root CA/admin/node certificates generated [ ] Kubernetes TLS Secrets integration [ ] Per-pod certificate selection [ ] OpenSearch TLS settings in Helm values [ ] Rendered manifest validation [ ] OpenSearch deployment [ ] Dashboards deployment [ ] Fluent Bit integration ``` The immediate next phase is: ```text Certificates ↓ Kubernetes TLS Secrets ↓ Helm certificate mounts ↓ OpenSearch TLS configuration ↓ Render and validate ↓ Deploy ```