mosiur/SOP

Fork 0

Files

mosiur 5f6478fc2c Opensearch Installation and cPanel log ingestion

2026-08-19 09:53:24 +00:00

30 KiB

Raw Blame History

SOP: Production OpenSearch Deployment on Kubernetes Using Helm

1. Purpose

Ei SOP-er goal holo Kubernetes cluster-e production-grade OpenSearch deploy kora using Helm, with:

3-node OpenSearch cluster
Longhorn persistent storage
OpenSearch Dashboards
Fluent Bit log ingestion
HTTPS/TLS
Node-to-node transport encryption
Dedicated observability namespace
Controlled pod scheduling
Production-oriented security
Nginx Proxy Manager for browser-facing HTTPS

Learning-mode note: Ei SOP-te exact executable command syntax intentionally deya hoyni. Protiti step-e command-er purpose, tool, expected output, ebong validation deya ache. Exact commands chaile SHOW SOLUTION type korun.

2. Final Architecture

cPanel / Log Sources
        |
        | Fluent Bit
        | HTTPS :9200
        v
+----------------------------------------+
| Kubernetes Cluster                    |
| Namespace: observability              |
|                                        |
|  OpenSearch Client Service            |
|  ClusterIP :9200                      |
|           |                            |
|           v                            |
|   +-------------------------------+    |
|   | OpenSearch StatefulSet        |    |
|   | replicas: 3                   |    |
|   |                               |    |
|   | pod-0 -> PVC-0                |    |
|   | pod-1 -> PVC-1                |    |
|   | pod-2 -> PVC-2                |    |
|   +-------------------------------+    |
|        ^        ^        ^             |
|        | TLS Transport :9300           |
|        +--------+--------+             |
|                 |                      |
|        Headless Service                |
|        Peer discovery / stable DNS     |
|                                        |
|  OpenSearch Dashboards                 |
|        |                               |
|        +--> HTTPS OpenSearch :9200     |
+----------------------------------------+
        ^
        |
        | HTTP/HTTPS
        |
Nginx Proxy Manager
        ^
        |
        | Public HTTPS
        |
      Browser

3. Kubernetes Cluster Topology

Target nodes:

k8s-cp-01 — control-plane
k8s-storage-01 — storage
k8s-worker-01 — worker/storage-capable
k8s-worker-02 — worker/storage-capable

Observed hostname labels include:

kubernetes.io/hostname
k3s.io/hostname

Recommended topology key:

kubernetes.io/hostname

Storage node taint:

key: dedicated
value: storage
effect: NoSchedule

OpenSearch pods require a matching toleration if k8s-storage-01 is intended as an eligible scheduling target.

4. Namespace

Use:

observability

Purpose:

Organizational separation
RBAC scope
Secret scoping
NetworkPolicy scope
ResourceQuota/LimitRange scope
Easier operations and troubleshooting

A namespace is not automatically a security boundary. Actual isolation requires RBAC, NetworkPolicy, Secret access control, and related controls.

5. OpenSearch Node Count vs Index Replica Count

Do not confuse Helm replicas with OpenSearch number_of_replicas.

Helm replicas

Target:

replicas: 3

Meaning:

3 OpenSearch pods
3 JVMs
3 PVCs

OpenSearch index replicas

For a 3-node cluster:

1 primary shard + 2 replica shards = 3 shard copies

Therefore the logical production target is:

number_of_replicas: 2

This is configured at the OpenSearch index/template layer, not automatically from Helm replicas.

6. StatefulSet Design

Use a StatefulSet for OpenSearch because it provides:

Stable ordinal identity
Stable pod names
Stable network identity
Stable PVC association
Controlled lifecycle behavior

Example:

opensearch-0
opensearch-1
opensearch-2

Each pod must have its own PVC.

Do not share a single data PVC across all OpenSearch nodes.

7. Service Design

OpenSearch needs two logical Service patterns.

7.1 Headless Service

Purpose:

Stable per-pod DNS
Peer discovery
StatefulSet network identity
Node-to-node communication

Concept:

clusterIP: None

Primary transport port:

StatefulSet serviceName should reference this headless Service.

Pod DNS pattern:

<pod>.<headless-service>.<namespace>.svc.cluster.local

7.2 Client-facing ClusterIP Service

Purpose:

Dashboards → OpenSearch
Fluent Bit → OpenSearch
Administrative REST clients → OpenSearch

REST/API port:

Clients should use the stable ClusterIP Service instead of targeting one specific OpenSearch pod.

8. OpenSearch Ports

REST / HTTPS

Used by:

Fluent Bit
OpenSearch Dashboards
API clients
Administrative requests

Transport

Used by:

Node-to-node communication
Cluster membership
Shard replication
Cluster-manager communication

Do not expose 9300 publicly.

Keep 9200 private/internal unless there is a specific administrative requirement.

9. Persistence and Storage

Target:

StorageClass: longhorn
AccessMode: ReadWriteOnce
PVC size: 120Gi per OpenSearch pod

For 3 pods:

3 × 120Gi = 360Gi logical PVC capacity

If Longhorn volume replica count is 2:

360Gi × 2 ≈ 720Gi raw replicated capacity

OpenSearch shard replication and Longhorn block replication are different layers:

OpenSearch replicas
→ application-level redundancy

Longhorn replicas
→ block-level redundancy

Both consume storage.

10. OpenSearch Data Path

The PVC must be mounted to the OpenSearch data directory, or path.data must explicitly point to the mounted location.

Unsafe pattern:

PVC mounted to /mnt
but OpenSearch writes to default data path

Possible result:

PVC exists
but index data lives in container filesystem
→ pod recreation can lose data

11. Resource Sizing

Target:

requests:
  cpu: 2
  memory: 6Gi

limits:
  cpu: 4
  memory: 8Gi

For 3 pods:

Total CPU requests:    6 cores
Total memory requests: 18Gi

Total CPU limits:      12 cores
Total memory limits:   24Gi

Scheduler uses resource requests when deciding placement.

12. JVM Heap

Use a fixed heap:

Xms = Xmx

Discussed starting point:

~3Gi heap

Remaining memory is needed for:

Filesystem cache
Direct buffers
Thread stacks
JVM metadata
Native libraries
Lucene/OpenSearch overhead

Monitor:

Heap utilization
Old generation pressure
GC frequency/duration
Circuit breakers
Container RSS
OOMKilled events
Query latency
Indexing latency

Heap belongs in Helm/application configuration, not in a Secret.

13. Pod Scheduling Strategy

Target placement:

k8s-storage-01 → 1 OpenSearch pod
k8s-worker-01  → 1 OpenSearch pod
k8s-worker-02  → 1 OpenSearch pod
k8s-cp-01      → no OpenSearch pod

Toleration

Allows a pod onto a matching tainted node.

For the storage node, the toleration must match:

dedicated=storage:NoSchedule

Toleration allows placement but does not force it.

Node Affinity / Selection

Controls which nodes are eligible.

Positive selection using workload/storage labels is generally easier to maintain than only blacklisting the control-plane hostname.

Pod Anti-Affinity

Prevents multiple OpenSearch pods from landing on the same Kubernetes node.

Topology key:

kubernetes.io/hostname

Target:

1 OpenSearch pod per Kubernetes node

Hard anti-affinity gives stronger failure-domain isolation but can leave a pod Pending if fewer than three eligible nodes are available.

14. Helm Deployment Workflow

Do not install the chart blindly.

Use:

Official Helm chart
        +
Default values
        +
Minimal custom values
        ↓
Rendered Kubernetes manifests
        ↓
Review
        ↓
Validate
        ↓
Install

Recommended workflow:

Inspect available OpenSearch chart versions
Inspect chart defaults
Build a minimal custom values file
Render manifests locally
Inspect StatefulSet, Services, PVCs, Secret mounts, affinity, tolerations and probes
Validate the rendered YAML
Install only after review

Avoid copying the full default values.yaml unless required.

15. Helm Values to Control

Production values should intentionally address:

clusterName
nodeGroup
replicas
singleNode
roles
image version
resources
opensearchJavaOpts
persistence
storageClass
PVC size
antiAffinity
antiAffinityTopologyKey
nodeAffinity / nodeSelector
tolerations
protocol
security configuration
secret mounts
extra volumes
extra init containers
pod management policy
probes

16. Image Versioning

Avoid a broad mutable tag such as:

Use a tested, pinned OpenSearch release.

Benefits:

Repeatable deployment
Predictable restarts
Controlled upgrades
Easier compatibility management with Dashboards

17. Kernel Requirement

Validate vm.max_map_count on every eligible node:

k8s-storage-01
k8s-worker-01
k8s-worker-02

If Helm sysctl mechanisms are enabled, understand whether the chart uses:

privileged init container
sysctl configuration
both

Avoid unnecessary duplicate privileged mechanisms.

18. Production Security Architecture

There are multiple TLS boundaries.

Browser-facing TLS

Browser → Nginx Proxy Manager

Nginx Proxy Manager manages this certificate.

OpenSearch REST TLS

Dashboards → OpenSearch :9200
Fluent Bit → OpenSearch :9200

OpenSearch manages this HTTPS/TLS configuration.

OpenSearch Transport TLS

node-0 ↔ node-1 ↔ node-2 :9300

Nginx Proxy Manager does not protect node-to-node traffic.

Transport TLS should remain enabled in production.

19. Identity Separation

Do not reuse the admin account everywhere.

Recommended identities:

Admin

Use for:

Security administration
Cluster administration
Emergency operations

Dashboards server account

Use only for Dashboards backend communication.

Fluent Bit ingest account

Grant only required index-write permissions.

Do not grant:

Security administration
Arbitrary index deletion
Cluster-wide configuration access
Read access to unrelated indices

Human Dashboard users

Give only required read/search/visualization permissions.

20. Bootstrap Password Secret

The initial admin password should be stored in a Kubernetes Secret in the same namespace as OpenSearch.

Important:

Base64 is encoding, not encryption
Never commit real Secret values to Git
Restrict Secret read permission with RBAC
Consider Kubernetes Secret encryption at rest
Rotate any credential exposed in chat or logs

The bootstrap Secret should ideally contain only:

OPENSEARCH_INITIAL_ADMIN_PASSWORD

21. Production PKI

Certificates must be generated by:

Your organization PKI
An internal CA
A secure OpenSSL-based private CA
A managed PKI platform

Minimum certificate set:

root-ca.pem
root-ca-key.pem

admin.pem
admin-key.pem

node-0.pem
node-0-key.pem

node-1.pem
node-1-key.pem

node-2.pem
node-2-key.pem

22. Certificate Responsibilities

`root-ca-key.pem`

Purpose:

Sign certificates

Rules:

NEVER mount into OpenSearch pods
NEVER store permanently in Kubernetes
NEVER commit to Git
Store offline/encrypted

If compromised, an attacker can issue fake trusted node or admin certificates.

`root-ca.pem`

Purpose:

Verify certificates issued by the CA

Can be distributed to:

OpenSearch nodes
Dashboards
Fluent Bit
Administrative clients

Admin Certificate

admin.pem
admin-key.pem

Used for privileged security administration.

Do not use it as a normal node certificate.

Do not mount the admin private key into every OpenSearch pod.

Node Certificates

Each node should have a unique identity:

node-0.pem / node-0-key.pem
node-1.pem / node-1-key.pem
node-2.pem / node-2-key.pem

Benefits:

Better isolation
Easier rotation
Easier revocation
Better auditing

23. Certificate Generation Lifecycle

Concept:

Generate private key
      ↓
Create CSR
      ↓
Sign CSR using Root CA private key
      ↓
Receive CA-signed certificate

Example:

node-0-key.pem
      ↓
node-0 CSR
      ↓
signed by root-ca-key.pem
      ↓
node-0.pem

OpenSearch-compatible private keys should be in the correct PKCS#8 format.

24. Certificate CN and SAN Design

Do not use an excessively long Kubernetes FQDN as the certificate CN.

X.509 CN has size constraints and very long values can cause generation errors.

Better approach:

CN:
  short node identity

SAN:
  actual Kubernetes DNS names

Use SAN entries for long pod and Service DNS names.

25. Node Certificate SAN Requirements

Each node certificate should include:

Its pod DNS
Relevant short pod DNS names
Headless Service-related DNS names if needed
Shared OpenSearch client Service DNS

Why shared client Service DNS matters:

Dashboards → https://client-service:9200

The ClusterIP may route to any OpenSearch pod.

Whichever pod responds must present a certificate valid for the requested Service hostname.

26. Certificate Verification

Before Kubernetes integration, verify:

CA chain

admin.pem  → verifies against root-ca.pem
node-0.pem → verifies against root-ca.pem
node-1.pem → verifies against root-ca.pem
node-2.pem → verifies against root-ca.pem

SAN

Each node certificate contains the intended DNS names.

Extended Key Usage

Node certificates support the required TLS server/client roles.

Private key format

Keys are in OpenSearch-compatible PKCS#8 format.

Key/certificate match

Each certificate matches its own private key.

Validity

Check:

Not Before
Not After

Distinguished Names

Extract exact RFC2253-formatted certificate DNs for OpenSearch configuration.

27. Kubernetes Secret Layout

Recommended separation:

Bootstrap Secret

opensearch-secret
└── OPENSEARCH_INITIAL_ADMIN_PASSWORD

Node TLS Secret

opensearch-node-tls
├── root-ca.pem
├── node-0.pem
├── node-0-key.pem
├── node-1.pem
├── node-1-key.pem
├── node-2.pem
└── node-2-key.pem

Admin TLS Secret

opensearch-admin-tls
├── root-ca.pem
├── admin.pem
└── admin-key.pem

Never place in Kubernetes

root-ca-key.pem

28. Per-Pod Certificate Selection

A StatefulSet uses one common pod template.

If the TLS Secret contains all node certificates, every pod initially sees all files.

Production pattern:

Node TLS Secret
      ↓
read-only mount
      ↓
init container
      ↓
detect pod ordinal
      ↓
copy correct node certificate/key
      ↓
private emptyDir
      ↓
main OpenSearch container

Mapping:

pod ending in -0
→ node-0.pem → node.pem
→ node-0-key.pem → node-key.pem

pod ending in -1
→ node-1.pem → node.pem
→ node-1-key.pem → node-key.pem

pod ending in -2
→ node-2.pem → node.pem
→ node-2-key.pem → node-key.pem

All pods can then use the same common paths:

tls/node.pem
tls/node-key.pem
tls/root-ca.pem

29. OpenSearch TLS Configuration

The OpenSearch configuration must logically define:

Transport TLS

Node certificate
Node private key
Trusted Root CA
Hostname verification behavior

REST TLS

HTTP TLS enabled
Node/server certificate
Node private key
Trusted Root CA

Trusted node identities

plugins.security.nodes_dn

Must contain the exact trusted node certificate DNs.

Admin identity

plugins.security.authcz.admin_dn

Must contain the exact admin certificate DN.

Do not mix node and admin identities.

30. Security Configuration Files

A production Security plugin setup may use:

config.yml
internal_users.yml
roles.yml
roles_mapping.yml
action_groups.yml
tenants.yml

Responsibilities:

`internal_users.yml`

User identities
Password hashes
Backend roles

`roles.yml`

Cluster permissions
Index permissions
Tenant permissions

`roles_mapping.yml`

Maps users/backend roles to OpenSearch roles

`config.yml`

Authentication domains
Authorization settings
Security plugin behavior

31. OpenSearch Dashboards

Deploy Dashboards after OpenSearch is healthy.

Recommended flow:

Browser
  ↓
Nginx Proxy Manager
  ↓
Dashboards Service
  ↓
Dashboards pod
  ↓
HTTPS OpenSearch Client Service :9200

Dashboards should:

Trust root-ca.pem
Use a dedicated backend credential
Not use the admin account
Connect to the OpenSearch ClusterIP Service

32. Fluent Bit Integration

Fluent Bit should connect to the private OpenSearch REST endpoint over HTTPS.

Production requirements:

TLS enabled
CA verification enabled
Trust root-ca.pem
Dedicated ingest identity
Least-privilege role
No admin credential reuse
Network access restricted to required paths

33. Nginx Proxy Manager

Recommended public flow:

Browser → Nginx Proxy Manager → Dashboards

Avoid:

Public → OpenSearch :9200
Public → OpenSearch :9300

Port 9300 should remain internal.

Port 9200 should normally remain private.

34. Probe Design

Do not use fake endpoints such as /healthz:8080 unless the application actually exposes them.

Startup Probe

Question:

Has OpenSearch finished starting?

Readiness Probe

Question:

Can this pod receive Service traffic?

Liveness Probe

Question:

Is the process permanently stuck/dead?

Do not make liveness depend too aggressively on transient cluster health.

Shard recovery can temporarily cause unhealthy cluster status without meaning the JVM should be restarted.

35. Pod Management Policy

Understand:

OrderedReady

vs:

Parallel

Potential multi-node bootstrap deadlock:

pod-0 waits for peers
StatefulSet waits for pod-0 Ready
pod-1 and pod-2 never start

Parallel startup may be more appropriate for a multi-node distributed cluster.

Also consider peer discovery before readiness.

36. Deployment Order

Recommended:

1. Namespace
2. Bootstrap Secret
3. TLS Secrets
4. Security configuration
5. Helm custom values
6. Rendered-manifest review
7. OpenSearch install
8. Cluster-health validation
9. Dashboards install
10. Nginx Proxy Manager configuration
11. Fluent Bit integration
12. Index templates and replica settings
13. Snapshot backup configuration
14. Monitoring and alerting

37. Rendered Manifest Review

Before Helm install, verify:

[ ] StatefulSet replicas = 3
[ ] Correct image/version
[ ] Resources present
[ ] Longhorn PVC template present
[ ] 120Gi per pod
[ ] Correct data path
[ ] Headless Service exists
[ ] ClusterIP REST Service exists
[ ] Port 9200 mapping correct
[ ] Port 9300 mapping correct
[ ] Hard anti-affinity rendered
[ ] topology key = kubernetes.io/hostname
[ ] Storage toleration at pod level
[ ] Control-plane excluded
[ ] TLS Secret mounts present
[ ] Per-pod certificate-selection logic present
[ ] TLS paths match opensearch.yml
[ ] nodes_dn matches node certificate DNs
[ ] admin_dn matches admin certificate DN
[ ] Bootstrap Secret reference correct
[ ] Probes match HTTPS behavior

38. Post-Deployment Validation

Kubernetes Layer

[ ] All 3 pods created
[ ] Each pod on a different Kubernetes node
[ ] No OpenSearch pod on control-plane
[ ] PVCs Bound
[ ] One PVC per pod
[ ] Longhorn volumes healthy
[ ] Init containers completed
[ ] No CrashLoopBackOff
[ ] No OOMKilled

TLS Layer

[ ] Correct certificate selected per pod
[ ] REST endpoint serves HTTPS
[ ] CA verification succeeds
[ ] Transport TLS succeeds
[ ] No SAN mismatch
[ ] No nodes_dn mismatch
[ ] No admin_dn mismatch

OpenSearch Layer

[ ] All 3 nodes joined one cluster
[ ] Cluster-manager elected
[ ] REST API responds
[ ] Authentication succeeds
[ ] Cluster health expected
[ ] Shard allocation correct

Storage Layer

[ ] Data survives pod recreation
[ ] Same ordinal reuses same PVC
[ ] Longhorn replicas healthy and distributed

39. Failure Testing

Delete one OpenSearch pod

Expected:

StatefulSet recreates same ordinal
Same PVC reused
Node rejoins cluster

Restart one Kubernetes worker

Expected:

Remaining nodes continue serving
Shard replica promotion can occur
Missing pod returns/reschedules according to policy

Lose the primary-shard node

With:

number_of_replicas: 2

a remaining replica can be promoted to primary.

Verify anti-affinity

Ensure no two OpenSearch pods share the same hostname.

40. Cluster Health Interpretation

Green

All primaries and replicas allocated.

Yellow

All primaries allocated, but at least one replica unassigned.

Typical example:

single-node cluster
number_of_replicas: 1

Red

At least one primary shard unavailable.

41. Common Mistakes Found During This Deployment

Mistake: one node with `number_of_replicas: 1`

Result:

1 primary + 1 replica
only 1 node
→ replica unassigned
→ yellow cluster

Mistake: confusing Helm replicas with shard replicas

They control different layers.

Mistake: using normal ClusterIP as StatefulSet `serviceName`

StatefulSet stable network identity should use the governing headless Service.

Mistake: treating port 80 as OpenSearch's main port

OpenSearch REST and transport use separate ports.

Mistake: mounting the PVC to `/mnt` without changing `path.data`

The PVC can remain unused.

Mistake: putting tolerations under the container

Tolerations are pod-level settings.

Mistake: using pod anti-affinity as a node blacklist

Pod anti-affinity controls pod co-location, not arbitrary node exclusion.

Mistake: assuming Nginx Proxy Manager protects node-to-node traffic

It does not.

Mistake: treating Base64 as encryption

It is only encoding.

Mistake: reusing admin credentials for Fluent Bit and Dashboards

Violates least privilege.

Mistake: very long FQDN as certificate CN

Can hit X.509 CN length limits.

Use a short CN and SAN for full DNS names.

42. Operational Monitoring

Kubernetes

Monitor:

Pod restarts
CPU throttling
Memory RSS
OOMKilled events
PVC usage
Node pressure
Scheduling failures

OpenSearch

Monitor:

JVM heap pressure
GC
Cluster health
Shard allocation
Pending tasks
Indexing latency
Search latency
Circuit breakers
Disk watermarks

Longhorn

Monitor:

Volume health
Replica count
Replica placement
Disk usage
Snapshot growth

Security

Monitor:

Certificate expiry
Authentication failures
Unauthorized requests
Admin credential usage
TLS handshake errors

43. Certificate Rotation

Track:

Not Before
Not After

Production certificate lifecycle should include:

Expiry monitoring
Renewal schedule
Secure CA access
Leaf-certificate replacement procedure
Emergency replacement/revocation procedure
Admin certificate rotation
Protected CA backup

44. Backup Strategy

OpenSearch shard replicas are not backups.

They do not protect from:

Accidental deletion
Malicious deletion
Corruption replicated across nodes
Entire cluster loss

Use snapshots to independent storage such as object storage/S3-compatible storage.

Include:

Separate credentials
Snapshot retention
Snapshot lifecycle
Restore testing

45. Security Hardening Checklist

[ ] Security plugin enabled
[ ] Demo certificates removed
[ ] Custom internal Root CA used
[ ] Unique node certificates
[ ] Separate admin certificate
[ ] root-ca-key.pem stored offline
[ ] REST HTTPS enabled
[ ] Transport TLS enabled
[ ] Port 9300 not externally exposed
[ ] Port 9200 private/internal
[ ] Dashboards exposed through Nginx Proxy Manager
[ ] Dedicated Fluent Bit user
[ ] Dedicated Dashboards server user
[ ] Human users use least-privilege roles
[ ] Secret access protected by RBAC
[ ] Kubernetes Secret encryption at rest considered
[ ] NetworkPolicy considered
[ ] Certificate expiry monitored

46. Go-Live Checklist

Kubernetes

[ ] Namespace = observability
[ ] 3 eligible OpenSearch nodes
[ ] Storage taint/toleration validated
[ ] Hard anti-affinity validated
[ ] Control-plane excluded
[ ] Longhorn healthy
[ ] vm.max_map_count validated

Helm

[ ] Chart version pinned
[ ] OpenSearch version pinned
[ ] Custom values reviewed
[ ] Rendered YAML reviewed
[ ] No unexpected resources

Storage

[ ] 120Gi × 3 planned
[ ] Longhorn replication capacity calculated
[ ] Sufficient disk headroom

TLS

[ ] Root CA generated
[ ] Root CA key offline
[ ] Admin certificate valid
[ ] Node certificates valid
[ ] SANs correct
[ ] Certificate/key pairs match
[ ] DNs extracted
[ ] TLS Secrets created

OpenSearch

[ ] 3 nodes joined
[ ] Cluster-manager election healthy
[ ] HTTPS works
[ ] Authentication works
[ ] Shard replica policy set
[ ] Cluster health verified

Dashboards / Fluent Bit

[ ] Dashboards trusts internal CA
[ ] Fluent Bit trusts internal CA
[ ] Admin password not reused
[ ] Nginx Proxy Manager exposes Dashboards only

47. Recommended Next Steps

Continue in this order:

Verify generated certificates
Extract exact admin and node DNs
Create Kubernetes TLS Secrets
Build per-pod certificate-selection logic
Add custom TLS configuration to Helm values
Render Helm manifests
Review StatefulSet and Services
Validate node placement
Install OpenSearch
Verify the 3-node HTTPS/TLS cluster
Deploy OpenSearch Dashboards
Configure Nginx Proxy Manager
Create Fluent Bit least-privilege ingest identity
Connect Fluent Bit over HTTPS
Configure index templates and shard replica policy
Configure snapshots/backups
Add monitoring and certificate-expiry alerts

48. Troubleshooting Decision Tree

Pod Pending

Investigate:

Node affinity
Pod anti-affinity
Taints
Tolerations
CPU/memory requests
PVC scheduling/topology

Pod CrashLoopBackOff

Investigate:

JVM heap
vm.max_map_count
TLS file paths
Private-key format
Certificate permissions
Bootstrap admin password
OpenSearch logs

Nodes do not form one cluster

Investigate:

Headless Service
DNS resolution
Transport port 9300
Transport TLS
Certificate SAN
nodes_dn
Discovery settings
Initial cluster bootstrap settings

HTTPS fails

Investigate:

REST TLS enabled
Certificate path
Private key path
Trusted Root CA
Client hostname
SAN
Certificate expiry

Dashboards cannot connect

Investigate:

OpenSearch client Service
Port 9200
HTTPS
Root CA trust
Dashboards backend credential
NetworkPolicy

Fluent Bit cannot ingest

Investigate:

OpenSearch endpoint
HTTPS
CA verification
Credential
Role permissions
Index permissions
Network path

49. Core Lessons

1. Stateful identity is not the same as pod IP.
2. StatefulSet provides stable identity; PVC stores persistent data.
3. Headless Service is for peer discovery and per-pod DNS.
4. ClusterIP Service is for normal clients.
5. Port 9200 and 9300 have different purposes.
6. Kubernetes replicas and OpenSearch shard replicas are different.
7. Redundancy works only when copies are separated across failure domains.
8. Toleration permits; affinity selects; anti-affinity separates.
9. A PVC is useless if the application writes somewhere else.
10. Helm should be rendered and reviewed before install.
11. Base64 is not encryption.
12. Reverse-proxy TLS does not protect internal OpenSearch transport.
13. Production TLS requires controlled PKI and certificate lifecycle.
14. Root CA private key never belongs in OpenSearch pods.
15. Unique node certificates are better than sharing one node key.
16. SANs are critical for DNS-based TLS verification.
17. Admin, Dashboards, and Fluent Bit should use separate identities.
18. OpenSearch replication is not a backup.

50. Final Target State

Kubernetes:
  3 OpenSearch pods
  3 separate Kubernetes nodes
  3 separate Longhorn PVCs
  no OpenSearch pod on control-plane

OpenSearch:
  3-node cluster
  transport TLS
  REST HTTPS
  correct shard replication
  healthy cluster-manager election

Security:
  custom internal CA
  unique node certificates
  separate admin certificate
  dedicated Dashboards credential
  dedicated Fluent Bit credential
  least privilege

Access:
  Browser → Nginx Proxy Manager → Dashboards
  Dashboards → HTTPS OpenSearch ClusterIP
  Fluent Bit → HTTPS OpenSearch private endpoint

Operations:
  monitoring
  snapshots/backups
  certificate expiry tracking
  restore testing

51. Current Project Status

At the end of this chat, the design has reached this point:

[✓] 3-node production topology selected
[✓] Longhorn selected for PVCs
[✓] 120Gi per PVC planned
[✓] Resource requests/limits discussed
[✓] JVM heap sizing discussed
[✓] Hard anti-affinity selected
[✓] Storage taint/toleration verified
[✓] HTTPS selected
[✓] Production TLS selected
[✓] Root CA/admin/node certificates generated
[ ] Kubernetes TLS Secrets integration
[ ] Per-pod certificate selection
[ ] OpenSearch TLS settings in Helm values
[ ] Rendered manifest validation
[ ] OpenSearch deployment
[ ] Dashboards deployment
[ ] Fluent Bit integration

The immediate next phase is:

Certificates
    ↓
Kubernetes TLS Secrets
    ↓
Helm certificate mounts
    ↓
OpenSearch TLS configuration
    ↓
Render and validate
    ↓
Deploy

30 KiB Raw Blame History Unescape Escape

SOP: Production OpenSearch Deployment on Kubernetes Using Helm

1. Purpose

2. Final Architecture

3. Kubernetes Cluster Topology

4. Namespace

5. OpenSearch Node Count vs Index Replica Count

Helm replicas

OpenSearch index replicas

6. StatefulSet Design

7. Service Design

7.1 Headless Service

7.2 Client-facing ClusterIP Service

8. OpenSearch Ports

REST / HTTPS

Transport

9. Persistence and Storage

10. OpenSearch Data Path

11. Resource Sizing

12. JVM Heap

13. Pod Scheduling Strategy

Toleration

Node Affinity / Selection

Pod Anti-Affinity

14. Helm Deployment Workflow

15. Helm Values to Control

16. Image Versioning

17. Kernel Requirement

18. Production Security Architecture

Browser-facing TLS

OpenSearch REST TLS

OpenSearch Transport TLS

19. Identity Separation

Admin

Dashboards server account

Fluent Bit ingest account

Human Dashboard users

20. Bootstrap Password Secret

21. Production PKI

22. Certificate Responsibilities

root-ca-key.pem

root-ca.pem

Admin Certificate

Node Certificates

23. Certificate Generation Lifecycle

24. Certificate CN and SAN Design

25. Node Certificate SAN Requirements

26. Certificate Verification

CA chain

SAN

Extended Key Usage

Private key format

Key/certificate match

Validity

Distinguished Names

27. Kubernetes Secret Layout

Bootstrap Secret

Node TLS Secret

Admin TLS Secret

Never place in Kubernetes

28. Per-Pod Certificate Selection

29. OpenSearch TLS Configuration

Transport TLS

REST TLS

Trusted node identities

Admin identity

30. Security Configuration Files

internal_users.yml

roles.yml

roles_mapping.yml

config.yml

31. OpenSearch Dashboards

32. Fluent Bit Integration

33. Nginx Proxy Manager

34. Probe Design

Startup Probe

Readiness Probe

Liveness Probe

35. Pod Management Policy

36. Deployment Order

30 KiB

Raw Blame History

`root-ca-key.pem`

`root-ca.pem`

`internal_users.yml`

`roles.yml`

`roles_mapping.yml`

`config.yml`

Mistake: one node with `number_of_replicas: 1`

Mistake: using normal ClusterIP as StatefulSet `serviceName`

Mistake: mounting the PVC to `/mnt` without changing `path.data`