SOP/OpenSearch_Kubernetes_Helm_Production_SOP.md

SOP: Production OpenSearch Deployment on Kubernetes Using Helm

1. Purpose

Ei SOP-er goal holo Kubernetes cluster-e production-grade OpenSearch deploy kora using Helm, with:

• 3-node OpenSearch cluster
• Longhorn persistent storage
• OpenSearch Dashboards
• Fluent Bit log ingestion
• HTTPS/TLS
• Node-to-node transport encryption
• Dedicated observability namespace
• Controlled pod scheduling
• Production-oriented security
• Nginx Proxy Manager for browser-facing HTTPS

Learning-mode note: Ei SOP-te exact executable command syntax intentionally deya hoyni. Protiti step-e command-er purpose, tool, expected output, ebong validation deya ache. Exact commands chaile SHOW SOLUTION type korun.

---

2. Final Architecture

cPanel / Log Sources
        |
        | Fluent Bit
        | HTTPS :9200
        v
+----------------------------------------+
| Kubernetes Cluster                    |
| Namespace: observability              |
|                                        |
|  OpenSearch Client Service            |
|  ClusterIP :9200                      |
|           |                            |
|           v                            |
|   +-------------------------------+    |
|   | OpenSearch StatefulSet        |    |
|   | replicas: 3                   |    |
|   |                               |    |
|   | pod-0 -> PVC-0                |    |
|   | pod-1 -> PVC-1                |    |
|   | pod-2 -> PVC-2                |    |
|   +-------------------------------+    |
|        ^        ^        ^             |
|        | TLS Transport :9300           |
|        +--------+--------+             |
|                 |                      |
|        Headless Service                |
|        Peer discovery / stable DNS     |
|                                        |
|  OpenSearch Dashboards                 |
|        |                               |
|        +--> HTTPS OpenSearch :9200     |
+----------------------------------------+
        ^
        |
        | HTTP/HTTPS
        |
Nginx Proxy Manager
        ^
        |
        | Public HTTPS
        |
      Browser

---

3. Kubernetes Cluster Topology

Target nodes:

• k8s-cp-01 — control-plane
• k8s-storage-01 — storage
• k8s-worker-01 — worker/storage-capable
• k8s-worker-02 — worker/storage-capable

Observed hostname labels include:

kubernetes.io/hostname
k3s.io/hostname

Recommended topology key:

kubernetes.io/hostname

Storage node taint:

key: dedicated
value: storage
effect: NoSchedule

OpenSearch pods require a matching toleration if k8s-storage-01 is intended as an eligible scheduling target.

---

4. Namespace

Use:

observability

Purpose:

• Organizational separation
• RBAC scope
• Secret scoping
• NetworkPolicy scope
• ResourceQuota/LimitRange scope
• Easier operations and troubleshooting

A namespace is not automatically a security boundary. Actual isolation requires RBAC, NetworkPolicy, Secret access control, and related controls.

---

5. OpenSearch Node Count vs Index Replica Count

Do not confuse Helm replicas with OpenSearch number_of_replicas.

Helm replicas

Target:

replicas: 3

Meaning:

3 OpenSearch pods
3 JVMs
3 PVCs

OpenSearch index replicas

For a 3-node cluster:

1 primary shard + 2 replica shards = 3 shard copies

Therefore the logical production target is:

number_of_replicas: 2

This is configured at the OpenSearch index/template layer, not automatically from Helm replicas.

---

6. StatefulSet Design

Use a StatefulSet for OpenSearch because it provides:

• Stable ordinal identity
• Stable pod names
• Stable network identity
• Stable PVC association
• Controlled lifecycle behavior

Example:

opensearch-0
opensearch-1
opensearch-2

Each pod must have its own PVC.

Do not share a single data PVC across all OpenSearch nodes.

---

7. Service Design

OpenSearch needs two logical Service patterns.

7.1 Headless Service

Purpose:

• Stable per-pod DNS
• Peer discovery
• StatefulSet network identity
• Node-to-node communication

Concept:

clusterIP: None

Primary transport port:

9300

StatefulSet serviceName should reference this headless Service.

Pod DNS pattern:

<pod>.<headless-service>.<namespace>.svc.cluster.local

7.2 Client-facing ClusterIP Service

Purpose:

• Dashboards → OpenSearch
• Fluent Bit → OpenSearch
• Administrative REST clients → OpenSearch

REST/API port:

9200

Clients should use the stable ClusterIP Service instead of targeting one specific OpenSearch pod.

---

8. OpenSearch Ports

REST / HTTPS

9200

Used by:

• Fluent Bit
• OpenSearch Dashboards
• API clients
• Administrative requests

Transport

9300

Used by:

• Node-to-node communication
• Cluster membership
• Shard replication
• Cluster-manager communication

Do not expose 9300 publicly.

Keep 9200 private/internal unless there is a specific administrative requirement.

---

9. Persistence and Storage

Target:

StorageClass: longhorn
AccessMode: ReadWriteOnce
PVC size: 120Gi per OpenSearch pod

For 3 pods:

3 × 120Gi = 360Gi logical PVC capacity

If Longhorn volume replica count is 2:

360Gi × 2 ≈ 720Gi raw replicated capacity

OpenSearch shard replication and Longhorn block replication are different layers:

OpenSearch replicas
→ application-level redundancy

Longhorn replicas
→ block-level redundancy

Both consume storage.

---

10. OpenSearch Data Path

The PVC must be mounted to the OpenSearch data directory, or path.data must explicitly point to the mounted location.

Unsafe pattern:

PVC mounted to /mnt
but OpenSearch writes to default data path

Possible result:

PVC exists
but index data lives in container filesystem
→ pod recreation can lose data

---

11. Resource Sizing

Target:

requests:
  cpu: 2
  memory: 6Gi

limits:
  cpu: 4
  memory: 8Gi

For 3 pods:

Total CPU requests:    6 cores
Total memory requests: 18Gi

Total CPU limits:      12 cores
Total memory limits:   24Gi

Scheduler uses resource requests when deciding placement.

---

12. JVM Heap

Use a fixed heap:

Xms = Xmx

Discussed starting point:

~3Gi heap

Remaining memory is needed for:

• Filesystem cache
• Direct buffers
• Thread stacks
• JVM metadata
• Native libraries
• Lucene/OpenSearch overhead

Monitor:

• Heap utilization
• Old generation pressure
• GC frequency/duration
• Circuit breakers
• Container RSS
• OOMKilled events
• Query latency
• Indexing latency

Heap belongs in Helm/application configuration, not in a Secret.

---

13. Pod Scheduling Strategy

Target placement:

k8s-storage-01 → 1 OpenSearch pod
k8s-worker-01  → 1 OpenSearch pod
k8s-worker-02  → 1 OpenSearch pod
k8s-cp-01      → no OpenSearch pod

Toleration

Allows a pod onto a matching tainted node.

For the storage node, the toleration must match:

dedicated=storage:NoSchedule

Toleration allows placement but does not force it.

Node Affinity / Selection

Controls which nodes are eligible.

Positive selection using workload/storage labels is generally easier to maintain than only blacklisting the control-plane hostname.

Pod Anti-Affinity

Prevents multiple OpenSearch pods from landing on the same Kubernetes node.

Topology key:

kubernetes.io/hostname

Target:

1 OpenSearch pod per Kubernetes node

Hard anti-affinity gives stronger failure-domain isolation but can leave a pod Pending if fewer than three eligible nodes are available.

---

14. Helm Deployment Workflow

Do not install the chart blindly.

Use:

Official Helm chart
        +
Default values
        +
Minimal custom values
        ↓
Rendered Kubernetes manifests
        ↓
Review
        ↓
Validate
        ↓
Install

Recommended workflow:

1. Inspect available OpenSearch chart versions
2. Inspect chart defaults
3. Build a minimal custom values file
4. Render manifests locally
5. Inspect StatefulSet, Services, PVCs, Secret mounts, affinity, tolerations and probes
6. Validate the rendered YAML
7. Install only after review

Avoid copying the full default values.yaml unless required.

---

15. Helm Values to Control

Production values should intentionally address:

clusterName
nodeGroup
replicas
singleNode
roles
image version
resources
opensearchJavaOpts
persistence
storageClass
PVC size
antiAffinity
antiAffinityTopologyKey
nodeAffinity / nodeSelector
tolerations
protocol
security configuration
secret mounts
extra volumes
extra init containers
pod management policy
probes

---

16. Image Versioning

Avoid a broad mutable tag such as:

3

Use a tested, pinned OpenSearch release.

Benefits:

• Repeatable deployment
• Predictable restarts
• Controlled upgrades
• Easier compatibility management with Dashboards

---

17. Kernel Requirement

Validate vm.max_map_count on every eligible node:

• k8s-storage-01
• k8s-worker-01
• k8s-worker-02

If Helm sysctl mechanisms are enabled, understand whether the chart uses:

• privileged init container
• sysctl configuration
• both

Avoid unnecessary duplicate privileged mechanisms.

---

18. Production Security Architecture

There are multiple TLS boundaries.

Browser-facing TLS

Browser → Nginx Proxy Manager

Nginx Proxy Manager manages this certificate.

OpenSearch REST TLS

Dashboards → OpenSearch :9200
Fluent Bit → OpenSearch :9200

OpenSearch manages this HTTPS/TLS configuration.

OpenSearch Transport TLS

node-0 ↔ node-1 ↔ node-2 :9300

Nginx Proxy Manager does not protect node-to-node traffic.

Transport TLS should remain enabled in production.

---

19. Identity Separation

Do not reuse the admin account everywhere.

Recommended identities:

Admin

Use for:

• Security administration
• Cluster administration
• Emergency operations

Dashboards server account

Use only for Dashboards backend communication.

Fluent Bit ingest account

Grant only required index-write permissions.

Do not grant:

• Security administration
• Arbitrary index deletion
• Cluster-wide configuration access
• Read access to unrelated indices

Human Dashboard users

Give only required read/search/visualization permissions.

---

20. Bootstrap Password Secret

The initial admin password should be stored in a Kubernetes Secret in the same namespace as OpenSearch.

Important:

• Base64 is encoding, not encryption
• Never commit real Secret values to Git
• Restrict Secret read permission with RBAC
• Consider Kubernetes Secret encryption at rest
• Rotate any credential exposed in chat or logs

The bootstrap Secret should ideally contain only:

OPENSEARCH_INITIAL_ADMIN_PASSWORD

---

21. Production PKI

Certificates must be generated by:

• Your organization PKI
• An internal CA
• A secure OpenSSL-based private CA
• A managed PKI platform

Minimum certificate set:

root-ca.pem
root-ca-key.pem

admin.pem
admin-key.pem

node-0.pem
node-0-key.pem

node-1.pem
node-1-key.pem

node-2.pem
node-2-key.pem

---

22. Certificate Responsibilities

root-ca-key.pem

Purpose:

Sign certificates

Rules:

NEVER mount into OpenSearch pods
NEVER store permanently in Kubernetes
NEVER commit to Git
Store offline/encrypted

If compromised, an attacker can issue fake trusted node or admin certificates.

root-ca.pem

Purpose:

Verify certificates issued by the CA

Can be distributed to:

• OpenSearch nodes
• Dashboards
• Fluent Bit
• Administrative clients

Admin Certificate

admin.pem
admin-key.pem

Used for privileged security administration.

Do not use it as a normal node certificate.

Do not mount the admin private key into every OpenSearch pod.

Node Certificates

Each node should have a unique identity:

node-0.pem / node-0-key.pem
node-1.pem / node-1-key.pem
node-2.pem / node-2-key.pem

Benefits:

• Better isolation
• Easier rotation
• Easier revocation
• Better auditing

---

23. Certificate Generation Lifecycle

Concept:

Generate private key
      ↓
Create CSR
      ↓
Sign CSR using Root CA private key
      ↓
Receive CA-signed certificate

Example:

node-0-key.pem
      ↓
node-0 CSR
      ↓
signed by root-ca-key.pem
      ↓
node-0.pem

OpenSearch-compatible private keys should be in the correct PKCS#8 format.

---

24. Certificate CN and SAN Design

Do not use an excessively long Kubernetes FQDN as the certificate CN.

X.509 CN has size constraints and very long values can cause generation errors.

Better approach:

CN:
  short node identity

SAN:
  actual Kubernetes DNS names

Use SAN entries for long pod and Service DNS names.

---

25. Node Certificate SAN Requirements

Each node certificate should include:

• Its pod DNS
• Relevant short pod DNS names
• Headless Service-related DNS names if needed
• Shared OpenSearch client Service DNS

Why shared client Service DNS matters:

Dashboards → https://client-service:9200

The ClusterIP may route to any OpenSearch pod.

Whichever pod responds must present a certificate valid for the requested Service hostname.

---

26. Certificate Verification

Before Kubernetes integration, verify:

CA chain

admin.pem  → verifies against root-ca.pem
node-0.pem → verifies against root-ca.pem
node-1.pem → verifies against root-ca.pem
node-2.pem → verifies against root-ca.pem

SAN

Each node certificate contains the intended DNS names.

Extended Key Usage

Node certificates support the required TLS server/client roles.

Private key format

Keys are in OpenSearch-compatible PKCS#8 format.

Key/certificate match

Each certificate matches its own private key.

Validity

Check:

Not Before
Not After

Distinguished Names

Extract exact RFC2253-formatted certificate DNs for OpenSearch configuration.

---

27. Kubernetes Secret Layout

Recommended separation:

Bootstrap Secret

opensearch-secret
└── OPENSEARCH_INITIAL_ADMIN_PASSWORD

Node TLS Secret

opensearch-node-tls
├── root-ca.pem
├── node-0.pem
├── node-0-key.pem
├── node-1.pem
├── node-1-key.pem
├── node-2.pem
└── node-2-key.pem

Admin TLS Secret

opensearch-admin-tls
├── root-ca.pem
├── admin.pem
└── admin-key.pem

Never place in Kubernetes

root-ca-key.pem

---

28. Per-Pod Certificate Selection

A StatefulSet uses one common pod template.

If the TLS Secret contains all node certificates, every pod initially sees all files.

Production pattern:

Node TLS Secret
      ↓
read-only mount
      ↓
init container
      ↓
detect pod ordinal
      ↓
copy correct node certificate/key
      ↓
private emptyDir
      ↓
main OpenSearch container

Mapping:

pod ending in -0
→ node-0.pem → node.pem
→ node-0-key.pem → node-key.pem

pod ending in -1
→ node-1.pem → node.pem
→ node-1-key.pem → node-key.pem

pod ending in -2
→ node-2.pem → node.pem
→ node-2-key.pem → node-key.pem

All pods can then use the same common paths:

tls/node.pem
tls/node-key.pem
tls/root-ca.pem

---

29. OpenSearch TLS Configuration

The OpenSearch configuration must logically define:

Transport TLS

Node certificate
Node private key
Trusted Root CA
Hostname verification behavior

REST TLS

HTTP TLS enabled
Node/server certificate
Node private key
Trusted Root CA

Trusted node identities

plugins.security.nodes_dn

Must contain the exact trusted node certificate DNs.

Admin identity

plugins.security.authcz.admin_dn

Must contain the exact admin certificate DN.

Do not mix node and admin identities.

---

30. Security Configuration Files

A production Security plugin setup may use:

config.yml
internal_users.yml
roles.yml
roles_mapping.yml
action_groups.yml
tenants.yml

Responsibilities:

internal_users.yml

• User identities
• Password hashes
• Backend roles

roles.yml

• Cluster permissions
• Index permissions
• Tenant permissions

roles_mapping.yml

• Maps users/backend roles to OpenSearch roles

config.yml

• Authentication domains
• Authorization settings
• Security plugin behavior

---

31. OpenSearch Dashboards

Deploy Dashboards after OpenSearch is healthy.

Recommended flow:

Browser
  ↓
Nginx Proxy Manager
  ↓
Dashboards Service
  ↓
Dashboards pod
  ↓
HTTPS OpenSearch Client Service :9200

Dashboards should:

• Trust root-ca.pem
• Use a dedicated backend credential
• Not use the admin account
• Connect to the OpenSearch ClusterIP Service

---

32. Fluent Bit Integration

Fluent Bit should connect to the private OpenSearch REST endpoint over HTTPS.

Production requirements:

• TLS enabled
• CA verification enabled
• Trust root-ca.pem
• Dedicated ingest identity
• Least-privilege role
• No admin credential reuse
• Network access restricted to required paths

---

33. Nginx Proxy Manager

Recommended public flow:

Browser → Nginx Proxy Manager → Dashboards

Avoid:

Public → OpenSearch :9200
Public → OpenSearch :9300

Port 9300 should remain internal.

Port 9200 should normally remain private.

---

34. Probe Design

Do not use fake endpoints such as /healthz:8080 unless the application actually exposes them.

Startup Probe

Question:

Has OpenSearch finished starting?

Readiness Probe

Question:

Can this pod receive Service traffic?

Liveness Probe

Question:

Is the process permanently stuck/dead?

Do not make liveness depend too aggressively on transient cluster health.

Shard recovery can temporarily cause unhealthy cluster status without meaning the JVM should be restarted.

---

35. Pod Management Policy

Understand:

OrderedReady

vs:

Parallel

Potential multi-node bootstrap deadlock:

pod-0 waits for peers
StatefulSet waits for pod-0 Ready
pod-1 and pod-2 never start

Parallel startup may be more appropriate for a multi-node distributed cluster.

Also consider peer discovery before readiness.

---

36. Deployment Order

Recommended:

1. Namespace
2. Bootstrap Secret
3. TLS Secrets
4. Security configuration
5. Helm custom values
6. Rendered-manifest review
7. OpenSearch install
8. Cluster-health validation
9. Dashboards install
10. Nginx Proxy Manager configuration
11. Fluent Bit integration
12. Index templates and replica settings
13. Snapshot backup configuration
14. Monitoring and alerting

---

37. Rendered Manifest Review

Before Helm install, verify:

[ ] StatefulSet replicas = 3
[ ] Correct image/version
[ ] Resources present
[ ] Longhorn PVC template present
[ ] 120Gi per pod
[ ] Correct data path
[ ] Headless Service exists
[ ] ClusterIP REST Service exists
[ ] Port 9200 mapping correct
[ ] Port 9300 mapping correct
[ ] Hard anti-affinity rendered
[ ] topology key = kubernetes.io/hostname
[ ] Storage toleration at pod level
[ ] Control-plane excluded
[ ] TLS Secret mounts present
[ ] Per-pod certificate-selection logic present
[ ] TLS paths match opensearch.yml
[ ] nodes_dn matches node certificate DNs
[ ] admin_dn matches admin certificate DN
[ ] Bootstrap Secret reference correct
[ ] Probes match HTTPS behavior

---

38. Post-Deployment Validation

Kubernetes Layer

[ ] All 3 pods created
[ ] Each pod on a different Kubernetes node
[ ] No OpenSearch pod on control-plane
[ ] PVCs Bound
[ ] One PVC per pod
[ ] Longhorn volumes healthy
[ ] Init containers completed
[ ] No CrashLoopBackOff
[ ] No OOMKilled

TLS Layer

[ ] Correct certificate selected per pod
[ ] REST endpoint serves HTTPS
[ ] CA verification succeeds
[ ] Transport TLS succeeds
[ ] No SAN mismatch
[ ] No nodes_dn mismatch
[ ] No admin_dn mismatch

OpenSearch Layer

[ ] All 3 nodes joined one cluster
[ ] Cluster-manager elected
[ ] REST API responds
[ ] Authentication succeeds
[ ] Cluster health expected
[ ] Shard allocation correct

Storage Layer

[ ] Data survives pod recreation
[ ] Same ordinal reuses same PVC
[ ] Longhorn replicas healthy and distributed

---

39. Failure Testing

Delete one OpenSearch pod

Expected:

StatefulSet recreates same ordinal
Same PVC reused
Node rejoins cluster

Restart one Kubernetes worker

Expected:

Remaining nodes continue serving
Shard replica promotion can occur
Missing pod returns/reschedules according to policy

Lose the primary-shard node

With:

number_of_replicas: 2

a remaining replica can be promoted to primary.

Verify anti-affinity

Ensure no two OpenSearch pods share the same hostname.

---

40. Cluster Health Interpretation

Green

All primaries and replicas allocated.

Yellow

All primaries allocated, but at least one replica unassigned.

Typical example:

single-node cluster
number_of_replicas: 1

Red

At least one primary shard unavailable.

---

41. Common Mistakes Found During This Deployment

Mistake: one node with number_of_replicas: 1

Result:

1 primary + 1 replica
only 1 node
→ replica unassigned
→ yellow cluster

Mistake: confusing Helm replicas with shard replicas

They control different layers.

Mistake: using normal ClusterIP as StatefulSet serviceName

StatefulSet stable network identity should use the governing headless Service.

Mistake: treating port 80 as OpenSearch's main port

OpenSearch REST and transport use separate ports.

Mistake: mounting the PVC to /mnt without changing path.data

The PVC can remain unused.

Mistake: putting tolerations under the container

Tolerations are pod-level settings.

Mistake: using pod anti-affinity as a node blacklist

Pod anti-affinity controls pod co-location, not arbitrary node exclusion.

Mistake: assuming Nginx Proxy Manager protects node-to-node traffic

It does not.

Mistake: treating Base64 as encryption

It is only encoding.

Mistake: reusing admin credentials for Fluent Bit and Dashboards

Violates least privilege.

Mistake: very long FQDN as certificate CN

Can hit X.509 CN length limits.

Use a short CN and SAN for full DNS names.

---

42. Operational Monitoring

Kubernetes

Monitor:

• Pod restarts
• CPU throttling
• Memory RSS
• OOMKilled events
• PVC usage
• Node pressure
• Scheduling failures

OpenSearch

Monitor:

• JVM heap pressure
• GC
• Cluster health
• Shard allocation
• Pending tasks
• Indexing latency
• Search latency
• Circuit breakers
• Disk watermarks

Longhorn

Monitor:

• Volume health
• Replica count
• Replica placement
• Disk usage
• Snapshot growth

Security

Monitor:

• Certificate expiry
• Authentication failures
• Unauthorized requests
• Admin credential usage
• TLS handshake errors

---

43. Certificate Rotation

Track:

Not Before
Not After

Production certificate lifecycle should include:

• Expiry monitoring
• Renewal schedule
• Secure CA access
• Leaf-certificate replacement procedure
• Emergency replacement/revocation procedure
• Admin certificate rotation
• Protected CA backup

---

44. Backup Strategy

OpenSearch shard replicas are not backups.

They do not protect from:

• Accidental deletion
• Malicious deletion
• Corruption replicated across nodes
• Entire cluster loss

Use snapshots to independent storage such as object storage/S3-compatible storage.

Include:

• Separate credentials
• Snapshot retention
• Snapshot lifecycle
• Restore testing

---

45. Security Hardening Checklist

[ ] Security plugin enabled
[ ] Demo certificates removed
[ ] Custom internal Root CA used
[ ] Unique node certificates
[ ] Separate admin certificate
[ ] root-ca-key.pem stored offline
[ ] REST HTTPS enabled
[ ] Transport TLS enabled
[ ] Port 9300 not externally exposed
[ ] Port 9200 private/internal
[ ] Dashboards exposed through Nginx Proxy Manager
[ ] Dedicated Fluent Bit user
[ ] Dedicated Dashboards server user
[ ] Human users use least-privilege roles
[ ] Secret access protected by RBAC
[ ] Kubernetes Secret encryption at rest considered
[ ] NetworkPolicy considered
[ ] Certificate expiry monitored

---

46. Go-Live Checklist

Kubernetes

[ ] Namespace = observability
[ ] 3 eligible OpenSearch nodes
[ ] Storage taint/toleration validated
[ ] Hard anti-affinity validated
[ ] Control-plane excluded
[ ] Longhorn healthy
[ ] vm.max_map_count validated

Helm

[ ] Chart version pinned
[ ] OpenSearch version pinned
[ ] Custom values reviewed
[ ] Rendered YAML reviewed
[ ] No unexpected resources

Storage

[ ] 120Gi × 3 planned
[ ] Longhorn replication capacity calculated
[ ] Sufficient disk headroom

TLS

[ ] Root CA generated
[ ] Root CA key offline
[ ] Admin certificate valid
[ ] Node certificates valid
[ ] SANs correct
[ ] Certificate/key pairs match
[ ] DNs extracted
[ ] TLS Secrets created

OpenSearch

[ ] 3 nodes joined
[ ] Cluster-manager election healthy
[ ] HTTPS works
[ ] Authentication works
[ ] Shard replica policy set
[ ] Cluster health verified

Dashboards / Fluent Bit

[ ] Dashboards trusts internal CA
[ ] Fluent Bit trusts internal CA
[ ] Admin password not reused
[ ] Nginx Proxy Manager exposes Dashboards only

---

47. Recommended Next Steps

Continue in this order:

1. Verify generated certificates
2. Extract exact admin and node DNs
3. Create Kubernetes TLS Secrets
4. Build per-pod certificate-selection logic
5. Add custom TLS configuration to Helm values
6. Render Helm manifests
7. Review StatefulSet and Services
8. Validate node placement
9. Install OpenSearch
10. Verify the 3-node HTTPS/TLS cluster
11. Deploy OpenSearch Dashboards
12. Configure Nginx Proxy Manager
13. Create Fluent Bit least-privilege ingest identity
14. Connect Fluent Bit over HTTPS
15. Configure index templates and shard replica policy
16. Configure snapshots/backups
17. Add monitoring and certificate-expiry alerts

---

48. Troubleshooting Decision Tree

Pod Pending

Investigate:

Node affinity
Pod anti-affinity
Taints
Tolerations
CPU/memory requests
PVC scheduling/topology

Pod CrashLoopBackOff

Investigate:

JVM heap
vm.max_map_count
TLS file paths
Private-key format
Certificate permissions
Bootstrap admin password
OpenSearch logs

Nodes do not form one cluster

Investigate:

Headless Service
DNS resolution
Transport port 9300
Transport TLS
Certificate SAN
nodes_dn
Discovery settings
Initial cluster bootstrap settings

HTTPS fails

Investigate:

REST TLS enabled
Certificate path
Private key path
Trusted Root CA
Client hostname
SAN
Certificate expiry

Dashboards cannot connect

Investigate:

OpenSearch client Service
Port 9200
HTTPS
Root CA trust
Dashboards backend credential
NetworkPolicy

Fluent Bit cannot ingest

Investigate:

OpenSearch endpoint
HTTPS
CA verification
Credential
Role permissions
Index permissions
Network path

---

49. Core Lessons

1. Stateful identity is not the same as pod IP.
2. StatefulSet provides stable identity; PVC stores persistent data.
3. Headless Service is for peer discovery and per-pod DNS.
4. ClusterIP Service is for normal clients.
5. Port 9200 and 9300 have different purposes.
6. Kubernetes replicas and OpenSearch shard replicas are different.
7. Redundancy works only when copies are separated across failure domains.
8. Toleration permits; affinity selects; anti-affinity separates.
9. A PVC is useless if the application writes somewhere else.
10. Helm should be rendered and reviewed before install.
11. Base64 is not encryption.
12. Reverse-proxy TLS does not protect internal OpenSearch transport.
13. Production TLS requires controlled PKI and certificate lifecycle.
14. Root CA private key never belongs in OpenSearch pods.
15. Unique node certificates are better than sharing one node key.
16. SANs are critical for DNS-based TLS verification.
17. Admin, Dashboards, and Fluent Bit should use separate identities.
18. OpenSearch replication is not a backup.

---

50. Final Target State

Kubernetes:
  3 OpenSearch pods
  3 separate Kubernetes nodes
  3 separate Longhorn PVCs
  no OpenSearch pod on control-plane

OpenSearch:
  3-node cluster
  transport TLS
  REST HTTPS
  correct shard replication
  healthy cluster-manager election

Security:
  custom internal CA
  unique node certificates
  separate admin certificate
  dedicated Dashboards credential
  dedicated Fluent Bit credential
  least privilege

Access:
  Browser → Nginx Proxy Manager → Dashboards
  Dashboards → HTTPS OpenSearch ClusterIP
  Fluent Bit → HTTPS OpenSearch private endpoint

Operations:
  monitoring
  snapshots/backups
  certificate expiry tracking
  restore testing

---

51. Current Project Status

At the end of this chat, the design has reached this point:

[✓] 3-node production topology selected
[✓] Longhorn selected for PVCs
[✓] 120Gi per PVC planned
[✓] Resource requests/limits discussed
[✓] JVM heap sizing discussed
[✓] Hard anti-affinity selected
[✓] Storage taint/toleration verified
[✓] HTTPS selected
[✓] Production TLS selected
[✓] Root CA/admin/node certificates generated
[ ] Kubernetes TLS Secrets integration
[ ] Per-pod certificate selection
[ ] OpenSearch TLS settings in Helm values
[ ] Rendered manifest validation
[ ] OpenSearch deployment
[ ] Dashboards deployment
[ ] Fluent Bit integration

The immediate next phase is:

Certificates
    ↓
Kubernetes TLS Secrets
    ↓
Helm certificate mounts
    ↓
OpenSearch TLS configuration
    ↓
Render and validate
    ↓
Deploy