1766 lines
30 KiB
Markdown
1766 lines
30 KiB
Markdown
# SOP: Production OpenSearch Deployment on Kubernetes Using Helm
|
||
|
||
## 1. Purpose
|
||
|
||
Ei SOP-er goal holo Kubernetes cluster-e production-grade OpenSearch deploy kora using Helm, with:
|
||
|
||
- 3-node OpenSearch cluster
|
||
- Longhorn persistent storage
|
||
- OpenSearch Dashboards
|
||
- Fluent Bit log ingestion
|
||
- HTTPS/TLS
|
||
- Node-to-node transport encryption
|
||
- Dedicated `observability` namespace
|
||
- Controlled pod scheduling
|
||
- Production-oriented security
|
||
- Nginx Proxy Manager for browser-facing HTTPS
|
||
|
||
> **Learning-mode note:** Ei SOP-te exact executable command syntax intentionally deya hoyni. Protiti step-e command-er purpose, tool, expected output, ebong validation deya ache. Exact commands chaile `SHOW SOLUTION` type korun.
|
||
|
||
---
|
||
|
||
## 2. Final Architecture
|
||
|
||
```text
|
||
cPanel / Log Sources
|
||
|
|
||
| Fluent Bit
|
||
| HTTPS :9200
|
||
v
|
||
+----------------------------------------+
|
||
| Kubernetes Cluster |
|
||
| Namespace: observability |
|
||
| |
|
||
| OpenSearch Client Service |
|
||
| ClusterIP :9200 |
|
||
| | |
|
||
| v |
|
||
| +-------------------------------+ |
|
||
| | OpenSearch StatefulSet | |
|
||
| | replicas: 3 | |
|
||
| | | |
|
||
| | pod-0 -> PVC-0 | |
|
||
| | pod-1 -> PVC-1 | |
|
||
| | pod-2 -> PVC-2 | |
|
||
| +-------------------------------+ |
|
||
| ^ ^ ^ |
|
||
| | TLS Transport :9300 |
|
||
| +--------+--------+ |
|
||
| | |
|
||
| Headless Service |
|
||
| Peer discovery / stable DNS |
|
||
| |
|
||
| OpenSearch Dashboards |
|
||
| | |
|
||
| +--> HTTPS OpenSearch :9200 |
|
||
+----------------------------------------+
|
||
^
|
||
|
|
||
| HTTP/HTTPS
|
||
|
|
||
Nginx Proxy Manager
|
||
^
|
||
|
|
||
| Public HTTPS
|
||
|
|
||
Browser
|
||
```
|
||
|
||
---
|
||
|
||
## 3. Kubernetes Cluster Topology
|
||
|
||
Target nodes:
|
||
|
||
- `k8s-cp-01` — control-plane
|
||
- `k8s-storage-01` — storage
|
||
- `k8s-worker-01` — worker/storage-capable
|
||
- `k8s-worker-02` — worker/storage-capable
|
||
|
||
Observed hostname labels include:
|
||
|
||
```text
|
||
kubernetes.io/hostname
|
||
k3s.io/hostname
|
||
```
|
||
|
||
Recommended topology key:
|
||
|
||
```text
|
||
kubernetes.io/hostname
|
||
```
|
||
|
||
Storage node taint:
|
||
|
||
```text
|
||
key: dedicated
|
||
value: storage
|
||
effect: NoSchedule
|
||
```
|
||
|
||
OpenSearch pods require a matching toleration if `k8s-storage-01` is intended as an eligible scheduling target.
|
||
|
||
---
|
||
|
||
## 4. Namespace
|
||
|
||
Use:
|
||
|
||
```text
|
||
observability
|
||
```
|
||
|
||
Purpose:
|
||
|
||
- Organizational separation
|
||
- RBAC scope
|
||
- Secret scoping
|
||
- NetworkPolicy scope
|
||
- ResourceQuota/LimitRange scope
|
||
- Easier operations and troubleshooting
|
||
|
||
A namespace is **not automatically a security boundary**. Actual isolation requires RBAC, NetworkPolicy, Secret access control, and related controls.
|
||
|
||
---
|
||
|
||
## 5. OpenSearch Node Count vs Index Replica Count
|
||
|
||
Do not confuse Helm `replicas` with OpenSearch `number_of_replicas`.
|
||
|
||
### Helm replicas
|
||
|
||
Target:
|
||
|
||
```text
|
||
replicas: 3
|
||
```
|
||
|
||
Meaning:
|
||
|
||
```text
|
||
3 OpenSearch pods
|
||
3 JVMs
|
||
3 PVCs
|
||
```
|
||
|
||
### OpenSearch index replicas
|
||
|
||
For a 3-node cluster:
|
||
|
||
```text
|
||
1 primary shard + 2 replica shards = 3 shard copies
|
||
```
|
||
|
||
Therefore the logical production target is:
|
||
|
||
```text
|
||
number_of_replicas: 2
|
||
```
|
||
|
||
This is configured at the OpenSearch index/template layer, not automatically from Helm replicas.
|
||
|
||
---
|
||
|
||
## 6. StatefulSet Design
|
||
|
||
Use a **StatefulSet** for OpenSearch because it provides:
|
||
|
||
- Stable ordinal identity
|
||
- Stable pod names
|
||
- Stable network identity
|
||
- Stable PVC association
|
||
- Controlled lifecycle behavior
|
||
|
||
Example:
|
||
|
||
```text
|
||
opensearch-0
|
||
opensearch-1
|
||
opensearch-2
|
||
```
|
||
|
||
Each pod must have its own PVC.
|
||
|
||
Do not share a single data PVC across all OpenSearch nodes.
|
||
|
||
---
|
||
|
||
## 7. Service Design
|
||
|
||
OpenSearch needs two logical Service patterns.
|
||
|
||
### 7.1 Headless Service
|
||
|
||
Purpose:
|
||
|
||
- Stable per-pod DNS
|
||
- Peer discovery
|
||
- StatefulSet network identity
|
||
- Node-to-node communication
|
||
|
||
Concept:
|
||
|
||
```text
|
||
clusterIP: None
|
||
```
|
||
|
||
Primary transport port:
|
||
|
||
```text
|
||
9300
|
||
```
|
||
|
||
StatefulSet `serviceName` should reference this headless Service.
|
||
|
||
Pod DNS pattern:
|
||
|
||
```text
|
||
<pod>.<headless-service>.<namespace>.svc.cluster.local
|
||
```
|
||
|
||
### 7.2 Client-facing ClusterIP Service
|
||
|
||
Purpose:
|
||
|
||
- Dashboards → OpenSearch
|
||
- Fluent Bit → OpenSearch
|
||
- Administrative REST clients → OpenSearch
|
||
|
||
REST/API port:
|
||
|
||
```text
|
||
9200
|
||
```
|
||
|
||
Clients should use the stable ClusterIP Service instead of targeting one specific OpenSearch pod.
|
||
|
||
---
|
||
|
||
## 8. OpenSearch Ports
|
||
|
||
### REST / HTTPS
|
||
|
||
```text
|
||
9200
|
||
```
|
||
|
||
Used by:
|
||
|
||
- Fluent Bit
|
||
- OpenSearch Dashboards
|
||
- API clients
|
||
- Administrative requests
|
||
|
||
### Transport
|
||
|
||
```text
|
||
9300
|
||
```
|
||
|
||
Used by:
|
||
|
||
- Node-to-node communication
|
||
- Cluster membership
|
||
- Shard replication
|
||
- Cluster-manager communication
|
||
|
||
Do not expose `9300` publicly.
|
||
|
||
Keep `9200` private/internal unless there is a specific administrative requirement.
|
||
|
||
---
|
||
|
||
## 9. Persistence and Storage
|
||
|
||
Target:
|
||
|
||
```text
|
||
StorageClass: longhorn
|
||
AccessMode: ReadWriteOnce
|
||
PVC size: 120Gi per OpenSearch pod
|
||
```
|
||
|
||
For 3 pods:
|
||
|
||
```text
|
||
3 × 120Gi = 360Gi logical PVC capacity
|
||
```
|
||
|
||
If Longhorn volume replica count is 2:
|
||
|
||
```text
|
||
360Gi × 2 ≈ 720Gi raw replicated capacity
|
||
```
|
||
|
||
OpenSearch shard replication and Longhorn block replication are different layers:
|
||
|
||
```text
|
||
OpenSearch replicas
|
||
→ application-level redundancy
|
||
|
||
Longhorn replicas
|
||
→ block-level redundancy
|
||
```
|
||
|
||
Both consume storage.
|
||
|
||
---
|
||
|
||
## 10. OpenSearch Data Path
|
||
|
||
The PVC must be mounted to the OpenSearch data directory, or `path.data` must explicitly point to the mounted location.
|
||
|
||
Unsafe pattern:
|
||
|
||
```text
|
||
PVC mounted to /mnt
|
||
but OpenSearch writes to default data path
|
||
```
|
||
|
||
Possible result:
|
||
|
||
```text
|
||
PVC exists
|
||
but index data lives in container filesystem
|
||
→ pod recreation can lose data
|
||
```
|
||
|
||
---
|
||
|
||
## 11. Resource Sizing
|
||
|
||
Target:
|
||
|
||
```text
|
||
requests:
|
||
cpu: 2
|
||
memory: 6Gi
|
||
|
||
limits:
|
||
cpu: 4
|
||
memory: 8Gi
|
||
```
|
||
|
||
For 3 pods:
|
||
|
||
```text
|
||
Total CPU requests: 6 cores
|
||
Total memory requests: 18Gi
|
||
|
||
Total CPU limits: 12 cores
|
||
Total memory limits: 24Gi
|
||
```
|
||
|
||
Scheduler uses resource requests when deciding placement.
|
||
|
||
---
|
||
|
||
## 12. JVM Heap
|
||
|
||
Use a fixed heap:
|
||
|
||
```text
|
||
Xms = Xmx
|
||
```
|
||
|
||
Discussed starting point:
|
||
|
||
```text
|
||
~3Gi heap
|
||
```
|
||
|
||
Remaining memory is needed for:
|
||
|
||
- Filesystem cache
|
||
- Direct buffers
|
||
- Thread stacks
|
||
- JVM metadata
|
||
- Native libraries
|
||
- Lucene/OpenSearch overhead
|
||
|
||
Monitor:
|
||
|
||
- Heap utilization
|
||
- Old generation pressure
|
||
- GC frequency/duration
|
||
- Circuit breakers
|
||
- Container RSS
|
||
- OOMKilled events
|
||
- Query latency
|
||
- Indexing latency
|
||
|
||
Heap belongs in Helm/application configuration, not in a Secret.
|
||
|
||
---
|
||
|
||
## 13. Pod Scheduling Strategy
|
||
|
||
Target placement:
|
||
|
||
```text
|
||
k8s-storage-01 → 1 OpenSearch pod
|
||
k8s-worker-01 → 1 OpenSearch pod
|
||
k8s-worker-02 → 1 OpenSearch pod
|
||
k8s-cp-01 → no OpenSearch pod
|
||
```
|
||
|
||
### Toleration
|
||
|
||
Allows a pod onto a matching tainted node.
|
||
|
||
For the storage node, the toleration must match:
|
||
|
||
```text
|
||
dedicated=storage:NoSchedule
|
||
```
|
||
|
||
Toleration allows placement but does not force it.
|
||
|
||
### Node Affinity / Selection
|
||
|
||
Controls which nodes are eligible.
|
||
|
||
Positive selection using workload/storage labels is generally easier to maintain than only blacklisting the control-plane hostname.
|
||
|
||
### Pod Anti-Affinity
|
||
|
||
Prevents multiple OpenSearch pods from landing on the same Kubernetes node.
|
||
|
||
Topology key:
|
||
|
||
```text
|
||
kubernetes.io/hostname
|
||
```
|
||
|
||
Target:
|
||
|
||
```text
|
||
1 OpenSearch pod per Kubernetes node
|
||
```
|
||
|
||
Hard anti-affinity gives stronger failure-domain isolation but can leave a pod Pending if fewer than three eligible nodes are available.
|
||
|
||
---
|
||
|
||
## 14. Helm Deployment Workflow
|
||
|
||
Do not install the chart blindly.
|
||
|
||
Use:
|
||
|
||
```text
|
||
Official Helm chart
|
||
+
|
||
Default values
|
||
+
|
||
Minimal custom values
|
||
↓
|
||
Rendered Kubernetes manifests
|
||
↓
|
||
Review
|
||
↓
|
||
Validate
|
||
↓
|
||
Install
|
||
```
|
||
|
||
Recommended workflow:
|
||
|
||
1. Inspect available OpenSearch chart versions
|
||
2. Inspect chart defaults
|
||
3. Build a minimal custom values file
|
||
4. Render manifests locally
|
||
5. Inspect StatefulSet, Services, PVCs, Secret mounts, affinity, tolerations and probes
|
||
6. Validate the rendered YAML
|
||
7. Install only after review
|
||
|
||
Avoid copying the full default `values.yaml` unless required.
|
||
|
||
---
|
||
|
||
## 15. Helm Values to Control
|
||
|
||
Production values should intentionally address:
|
||
|
||
```text
|
||
clusterName
|
||
nodeGroup
|
||
replicas
|
||
singleNode
|
||
roles
|
||
image version
|
||
resources
|
||
opensearchJavaOpts
|
||
persistence
|
||
storageClass
|
||
PVC size
|
||
antiAffinity
|
||
antiAffinityTopologyKey
|
||
nodeAffinity / nodeSelector
|
||
tolerations
|
||
protocol
|
||
security configuration
|
||
secret mounts
|
||
extra volumes
|
||
extra init containers
|
||
pod management policy
|
||
probes
|
||
```
|
||
|
||
---
|
||
|
||
## 16. Image Versioning
|
||
|
||
Avoid a broad mutable tag such as:
|
||
|
||
```text
|
||
3
|
||
```
|
||
|
||
Use a tested, pinned OpenSearch release.
|
||
|
||
Benefits:
|
||
|
||
- Repeatable deployment
|
||
- Predictable restarts
|
||
- Controlled upgrades
|
||
- Easier compatibility management with Dashboards
|
||
|
||
---
|
||
|
||
## 17. Kernel Requirement
|
||
|
||
Validate `vm.max_map_count` on every eligible node:
|
||
|
||
- `k8s-storage-01`
|
||
- `k8s-worker-01`
|
||
- `k8s-worker-02`
|
||
|
||
If Helm sysctl mechanisms are enabled, understand whether the chart uses:
|
||
|
||
- privileged init container
|
||
- sysctl configuration
|
||
- both
|
||
|
||
Avoid unnecessary duplicate privileged mechanisms.
|
||
|
||
---
|
||
|
||
## 18. Production Security Architecture
|
||
|
||
There are multiple TLS boundaries.
|
||
|
||
### Browser-facing TLS
|
||
|
||
```text
|
||
Browser → Nginx Proxy Manager
|
||
```
|
||
|
||
Nginx Proxy Manager manages this certificate.
|
||
|
||
### OpenSearch REST TLS
|
||
|
||
```text
|
||
Dashboards → OpenSearch :9200
|
||
Fluent Bit → OpenSearch :9200
|
||
```
|
||
|
||
OpenSearch manages this HTTPS/TLS configuration.
|
||
|
||
### OpenSearch Transport TLS
|
||
|
||
```text
|
||
node-0 ↔ node-1 ↔ node-2 :9300
|
||
```
|
||
|
||
Nginx Proxy Manager does not protect node-to-node traffic.
|
||
|
||
Transport TLS should remain enabled in production.
|
||
|
||
---
|
||
|
||
## 19. Identity Separation
|
||
|
||
Do not reuse the admin account everywhere.
|
||
|
||
Recommended identities:
|
||
|
||
### Admin
|
||
|
||
Use for:
|
||
|
||
- Security administration
|
||
- Cluster administration
|
||
- Emergency operations
|
||
|
||
### Dashboards server account
|
||
|
||
Use only for Dashboards backend communication.
|
||
|
||
### Fluent Bit ingest account
|
||
|
||
Grant only required index-write permissions.
|
||
|
||
Do not grant:
|
||
|
||
- Security administration
|
||
- Arbitrary index deletion
|
||
- Cluster-wide configuration access
|
||
- Read access to unrelated indices
|
||
|
||
### Human Dashboard users
|
||
|
||
Give only required read/search/visualization permissions.
|
||
|
||
---
|
||
|
||
## 20. Bootstrap Password Secret
|
||
|
||
The initial admin password should be stored in a Kubernetes Secret in the same namespace as OpenSearch.
|
||
|
||
Important:
|
||
|
||
- Base64 is encoding, not encryption
|
||
- Never commit real Secret values to Git
|
||
- Restrict Secret read permission with RBAC
|
||
- Consider Kubernetes Secret encryption at rest
|
||
- Rotate any credential exposed in chat or logs
|
||
|
||
The bootstrap Secret should ideally contain only:
|
||
|
||
```text
|
||
OPENSEARCH_INITIAL_ADMIN_PASSWORD
|
||
```
|
||
|
||
---
|
||
|
||
## 21. Production PKI
|
||
|
||
Certificates must be generated by:
|
||
|
||
- Your organization PKI
|
||
- An internal CA
|
||
- A secure OpenSSL-based private CA
|
||
- A managed PKI platform
|
||
|
||
Minimum certificate set:
|
||
|
||
```text
|
||
root-ca.pem
|
||
root-ca-key.pem
|
||
|
||
admin.pem
|
||
admin-key.pem
|
||
|
||
node-0.pem
|
||
node-0-key.pem
|
||
|
||
node-1.pem
|
||
node-1-key.pem
|
||
|
||
node-2.pem
|
||
node-2-key.pem
|
||
```
|
||
|
||
---
|
||
|
||
## 22. Certificate Responsibilities
|
||
|
||
### `root-ca-key.pem`
|
||
|
||
Purpose:
|
||
|
||
```text
|
||
Sign certificates
|
||
```
|
||
|
||
Rules:
|
||
|
||
```text
|
||
NEVER mount into OpenSearch pods
|
||
NEVER store permanently in Kubernetes
|
||
NEVER commit to Git
|
||
Store offline/encrypted
|
||
```
|
||
|
||
If compromised, an attacker can issue fake trusted node or admin certificates.
|
||
|
||
### `root-ca.pem`
|
||
|
||
Purpose:
|
||
|
||
```text
|
||
Verify certificates issued by the CA
|
||
```
|
||
|
||
Can be distributed to:
|
||
|
||
- OpenSearch nodes
|
||
- Dashboards
|
||
- Fluent Bit
|
||
- Administrative clients
|
||
|
||
### Admin Certificate
|
||
|
||
```text
|
||
admin.pem
|
||
admin-key.pem
|
||
```
|
||
|
||
Used for privileged security administration.
|
||
|
||
Do not use it as a normal node certificate.
|
||
|
||
Do not mount the admin private key into every OpenSearch pod.
|
||
|
||
### Node Certificates
|
||
|
||
Each node should have a unique identity:
|
||
|
||
```text
|
||
node-0.pem / node-0-key.pem
|
||
node-1.pem / node-1-key.pem
|
||
node-2.pem / node-2-key.pem
|
||
```
|
||
|
||
Benefits:
|
||
|
||
- Better isolation
|
||
- Easier rotation
|
||
- Easier revocation
|
||
- Better auditing
|
||
|
||
---
|
||
|
||
## 23. Certificate Generation Lifecycle
|
||
|
||
Concept:
|
||
|
||
```text
|
||
Generate private key
|
||
↓
|
||
Create CSR
|
||
↓
|
||
Sign CSR using Root CA private key
|
||
↓
|
||
Receive CA-signed certificate
|
||
```
|
||
|
||
Example:
|
||
|
||
```text
|
||
node-0-key.pem
|
||
↓
|
||
node-0 CSR
|
||
↓
|
||
signed by root-ca-key.pem
|
||
↓
|
||
node-0.pem
|
||
```
|
||
|
||
OpenSearch-compatible private keys should be in the correct PKCS#8 format.
|
||
|
||
---
|
||
|
||
## 24. Certificate CN and SAN Design
|
||
|
||
Do not use an excessively long Kubernetes FQDN as the certificate CN.
|
||
|
||
X.509 CN has size constraints and very long values can cause generation errors.
|
||
|
||
Better approach:
|
||
|
||
```text
|
||
CN:
|
||
short node identity
|
||
|
||
SAN:
|
||
actual Kubernetes DNS names
|
||
```
|
||
|
||
Use SAN entries for long pod and Service DNS names.
|
||
|
||
---
|
||
|
||
## 25. Node Certificate SAN Requirements
|
||
|
||
Each node certificate should include:
|
||
|
||
- Its pod DNS
|
||
- Relevant short pod DNS names
|
||
- Headless Service-related DNS names if needed
|
||
- Shared OpenSearch client Service DNS
|
||
|
||
Why shared client Service DNS matters:
|
||
|
||
```text
|
||
Dashboards → https://client-service:9200
|
||
```
|
||
|
||
The ClusterIP may route to any OpenSearch pod.
|
||
|
||
Whichever pod responds must present a certificate valid for the requested Service hostname.
|
||
|
||
---
|
||
|
||
## 26. Certificate Verification
|
||
|
||
Before Kubernetes integration, verify:
|
||
|
||
### CA chain
|
||
|
||
```text
|
||
admin.pem → verifies against root-ca.pem
|
||
node-0.pem → verifies against root-ca.pem
|
||
node-1.pem → verifies against root-ca.pem
|
||
node-2.pem → verifies against root-ca.pem
|
||
```
|
||
|
||
### SAN
|
||
|
||
Each node certificate contains the intended DNS names.
|
||
|
||
### Extended Key Usage
|
||
|
||
Node certificates support the required TLS server/client roles.
|
||
|
||
### Private key format
|
||
|
||
Keys are in OpenSearch-compatible PKCS#8 format.
|
||
|
||
### Key/certificate match
|
||
|
||
Each certificate matches its own private key.
|
||
|
||
### Validity
|
||
|
||
Check:
|
||
|
||
```text
|
||
Not Before
|
||
Not After
|
||
```
|
||
|
||
### Distinguished Names
|
||
|
||
Extract exact RFC2253-formatted certificate DNs for OpenSearch configuration.
|
||
|
||
---
|
||
|
||
## 27. Kubernetes Secret Layout
|
||
|
||
Recommended separation:
|
||
|
||
### Bootstrap Secret
|
||
|
||
```text
|
||
opensearch-secret
|
||
└── OPENSEARCH_INITIAL_ADMIN_PASSWORD
|
||
```
|
||
|
||
### Node TLS Secret
|
||
|
||
```text
|
||
opensearch-node-tls
|
||
├── root-ca.pem
|
||
├── node-0.pem
|
||
├── node-0-key.pem
|
||
├── node-1.pem
|
||
├── node-1-key.pem
|
||
├── node-2.pem
|
||
└── node-2-key.pem
|
||
```
|
||
|
||
### Admin TLS Secret
|
||
|
||
```text
|
||
opensearch-admin-tls
|
||
├── root-ca.pem
|
||
├── admin.pem
|
||
└── admin-key.pem
|
||
```
|
||
|
||
### Never place in Kubernetes
|
||
|
||
```text
|
||
root-ca-key.pem
|
||
```
|
||
|
||
---
|
||
|
||
## 28. Per-Pod Certificate Selection
|
||
|
||
A StatefulSet uses one common pod template.
|
||
|
||
If the TLS Secret contains all node certificates, every pod initially sees all files.
|
||
|
||
Production pattern:
|
||
|
||
```text
|
||
Node TLS Secret
|
||
↓
|
||
read-only mount
|
||
↓
|
||
init container
|
||
↓
|
||
detect pod ordinal
|
||
↓
|
||
copy correct node certificate/key
|
||
↓
|
||
private emptyDir
|
||
↓
|
||
main OpenSearch container
|
||
```
|
||
|
||
Mapping:
|
||
|
||
```text
|
||
pod ending in -0
|
||
→ node-0.pem → node.pem
|
||
→ node-0-key.pem → node-key.pem
|
||
|
||
pod ending in -1
|
||
→ node-1.pem → node.pem
|
||
→ node-1-key.pem → node-key.pem
|
||
|
||
pod ending in -2
|
||
→ node-2.pem → node.pem
|
||
→ node-2-key.pem → node-key.pem
|
||
```
|
||
|
||
All pods can then use the same common paths:
|
||
|
||
```text
|
||
tls/node.pem
|
||
tls/node-key.pem
|
||
tls/root-ca.pem
|
||
```
|
||
|
||
---
|
||
|
||
## 29. OpenSearch TLS Configuration
|
||
|
||
The OpenSearch configuration must logically define:
|
||
|
||
### Transport TLS
|
||
|
||
```text
|
||
Node certificate
|
||
Node private key
|
||
Trusted Root CA
|
||
Hostname verification behavior
|
||
```
|
||
|
||
### REST TLS
|
||
|
||
```text
|
||
HTTP TLS enabled
|
||
Node/server certificate
|
||
Node private key
|
||
Trusted Root CA
|
||
```
|
||
|
||
### Trusted node identities
|
||
|
||
```text
|
||
plugins.security.nodes_dn
|
||
```
|
||
|
||
Must contain the exact trusted node certificate DNs.
|
||
|
||
### Admin identity
|
||
|
||
```text
|
||
plugins.security.authcz.admin_dn
|
||
```
|
||
|
||
Must contain the exact admin certificate DN.
|
||
|
||
Do not mix node and admin identities.
|
||
|
||
---
|
||
|
||
## 30. Security Configuration Files
|
||
|
||
A production Security plugin setup may use:
|
||
|
||
```text
|
||
config.yml
|
||
internal_users.yml
|
||
roles.yml
|
||
roles_mapping.yml
|
||
action_groups.yml
|
||
tenants.yml
|
||
```
|
||
|
||
Responsibilities:
|
||
|
||
### `internal_users.yml`
|
||
|
||
- User identities
|
||
- Password hashes
|
||
- Backend roles
|
||
|
||
### `roles.yml`
|
||
|
||
- Cluster permissions
|
||
- Index permissions
|
||
- Tenant permissions
|
||
|
||
### `roles_mapping.yml`
|
||
|
||
- Maps users/backend roles to OpenSearch roles
|
||
|
||
### `config.yml`
|
||
|
||
- Authentication domains
|
||
- Authorization settings
|
||
- Security plugin behavior
|
||
|
||
---
|
||
|
||
## 31. OpenSearch Dashboards
|
||
|
||
Deploy Dashboards after OpenSearch is healthy.
|
||
|
||
Recommended flow:
|
||
|
||
```text
|
||
Browser
|
||
↓
|
||
Nginx Proxy Manager
|
||
↓
|
||
Dashboards Service
|
||
↓
|
||
Dashboards pod
|
||
↓
|
||
HTTPS OpenSearch Client Service :9200
|
||
```
|
||
|
||
Dashboards should:
|
||
|
||
- Trust `root-ca.pem`
|
||
- Use a dedicated backend credential
|
||
- Not use the admin account
|
||
- Connect to the OpenSearch ClusterIP Service
|
||
|
||
---
|
||
|
||
## 32. Fluent Bit Integration
|
||
|
||
Fluent Bit should connect to the private OpenSearch REST endpoint over HTTPS.
|
||
|
||
Production requirements:
|
||
|
||
- TLS enabled
|
||
- CA verification enabled
|
||
- Trust `root-ca.pem`
|
||
- Dedicated ingest identity
|
||
- Least-privilege role
|
||
- No admin credential reuse
|
||
- Network access restricted to required paths
|
||
|
||
---
|
||
|
||
## 33. Nginx Proxy Manager
|
||
|
||
Recommended public flow:
|
||
|
||
```text
|
||
Browser → Nginx Proxy Manager → Dashboards
|
||
```
|
||
|
||
Avoid:
|
||
|
||
```text
|
||
Public → OpenSearch :9200
|
||
Public → OpenSearch :9300
|
||
```
|
||
|
||
Port `9300` should remain internal.
|
||
|
||
Port `9200` should normally remain private.
|
||
|
||
---
|
||
|
||
## 34. Probe Design
|
||
|
||
Do not use fake endpoints such as `/healthz:8080` unless the application actually exposes them.
|
||
|
||
### Startup Probe
|
||
|
||
Question:
|
||
|
||
```text
|
||
Has OpenSearch finished starting?
|
||
```
|
||
|
||
### Readiness Probe
|
||
|
||
Question:
|
||
|
||
```text
|
||
Can this pod receive Service traffic?
|
||
```
|
||
|
||
### Liveness Probe
|
||
|
||
Question:
|
||
|
||
```text
|
||
Is the process permanently stuck/dead?
|
||
```
|
||
|
||
Do not make liveness depend too aggressively on transient cluster health.
|
||
|
||
Shard recovery can temporarily cause unhealthy cluster status without meaning the JVM should be restarted.
|
||
|
||
---
|
||
|
||
## 35. Pod Management Policy
|
||
|
||
Understand:
|
||
|
||
```text
|
||
OrderedReady
|
||
```
|
||
|
||
vs:
|
||
|
||
```text
|
||
Parallel
|
||
```
|
||
|
||
Potential multi-node bootstrap deadlock:
|
||
|
||
```text
|
||
pod-0 waits for peers
|
||
StatefulSet waits for pod-0 Ready
|
||
pod-1 and pod-2 never start
|
||
```
|
||
|
||
Parallel startup may be more appropriate for a multi-node distributed cluster.
|
||
|
||
Also consider peer discovery before readiness.
|
||
|
||
---
|
||
|
||
## 36. Deployment Order
|
||
|
||
Recommended:
|
||
|
||
```text
|
||
1. Namespace
|
||
2. Bootstrap Secret
|
||
3. TLS Secrets
|
||
4. Security configuration
|
||
5. Helm custom values
|
||
6. Rendered-manifest review
|
||
7. OpenSearch install
|
||
8. Cluster-health validation
|
||
9. Dashboards install
|
||
10. Nginx Proxy Manager configuration
|
||
11. Fluent Bit integration
|
||
12. Index templates and replica settings
|
||
13. Snapshot backup configuration
|
||
14. Monitoring and alerting
|
||
```
|
||
|
||
---
|
||
|
||
## 37. Rendered Manifest Review
|
||
|
||
Before Helm install, verify:
|
||
|
||
```text
|
||
[ ] StatefulSet replicas = 3
|
||
[ ] Correct image/version
|
||
[ ] Resources present
|
||
[ ] Longhorn PVC template present
|
||
[ ] 120Gi per pod
|
||
[ ] Correct data path
|
||
[ ] Headless Service exists
|
||
[ ] ClusterIP REST Service exists
|
||
[ ] Port 9200 mapping correct
|
||
[ ] Port 9300 mapping correct
|
||
[ ] Hard anti-affinity rendered
|
||
[ ] topology key = kubernetes.io/hostname
|
||
[ ] Storage toleration at pod level
|
||
[ ] Control-plane excluded
|
||
[ ] TLS Secret mounts present
|
||
[ ] Per-pod certificate-selection logic present
|
||
[ ] TLS paths match opensearch.yml
|
||
[ ] nodes_dn matches node certificate DNs
|
||
[ ] admin_dn matches admin certificate DN
|
||
[ ] Bootstrap Secret reference correct
|
||
[ ] Probes match HTTPS behavior
|
||
```
|
||
|
||
---
|
||
|
||
## 38. Post-Deployment Validation
|
||
|
||
### Kubernetes Layer
|
||
|
||
```text
|
||
[ ] All 3 pods created
|
||
[ ] Each pod on a different Kubernetes node
|
||
[ ] No OpenSearch pod on control-plane
|
||
[ ] PVCs Bound
|
||
[ ] One PVC per pod
|
||
[ ] Longhorn volumes healthy
|
||
[ ] Init containers completed
|
||
[ ] No CrashLoopBackOff
|
||
[ ] No OOMKilled
|
||
```
|
||
|
||
### TLS Layer
|
||
|
||
```text
|
||
[ ] Correct certificate selected per pod
|
||
[ ] REST endpoint serves HTTPS
|
||
[ ] CA verification succeeds
|
||
[ ] Transport TLS succeeds
|
||
[ ] No SAN mismatch
|
||
[ ] No nodes_dn mismatch
|
||
[ ] No admin_dn mismatch
|
||
```
|
||
|
||
### OpenSearch Layer
|
||
|
||
```text
|
||
[ ] All 3 nodes joined one cluster
|
||
[ ] Cluster-manager elected
|
||
[ ] REST API responds
|
||
[ ] Authentication succeeds
|
||
[ ] Cluster health expected
|
||
[ ] Shard allocation correct
|
||
```
|
||
|
||
### Storage Layer
|
||
|
||
```text
|
||
[ ] Data survives pod recreation
|
||
[ ] Same ordinal reuses same PVC
|
||
[ ] Longhorn replicas healthy and distributed
|
||
```
|
||
|
||
---
|
||
|
||
## 39. Failure Testing
|
||
|
||
### Delete one OpenSearch pod
|
||
|
||
Expected:
|
||
|
||
```text
|
||
StatefulSet recreates same ordinal
|
||
Same PVC reused
|
||
Node rejoins cluster
|
||
```
|
||
|
||
### Restart one Kubernetes worker
|
||
|
||
Expected:
|
||
|
||
```text
|
||
Remaining nodes continue serving
|
||
Shard replica promotion can occur
|
||
Missing pod returns/reschedules according to policy
|
||
```
|
||
|
||
### Lose the primary-shard node
|
||
|
||
With:
|
||
|
||
```text
|
||
number_of_replicas: 2
|
||
```
|
||
|
||
a remaining replica can be promoted to primary.
|
||
|
||
### Verify anti-affinity
|
||
|
||
Ensure no two OpenSearch pods share the same hostname.
|
||
|
||
---
|
||
|
||
## 40. Cluster Health Interpretation
|
||
|
||
### Green
|
||
|
||
All primaries and replicas allocated.
|
||
|
||
### Yellow
|
||
|
||
All primaries allocated, but at least one replica unassigned.
|
||
|
||
Typical example:
|
||
|
||
```text
|
||
single-node cluster
|
||
number_of_replicas: 1
|
||
```
|
||
|
||
### Red
|
||
|
||
At least one primary shard unavailable.
|
||
|
||
---
|
||
|
||
## 41. Common Mistakes Found During This Deployment
|
||
|
||
### Mistake: one node with `number_of_replicas: 1`
|
||
|
||
Result:
|
||
|
||
```text
|
||
1 primary + 1 replica
|
||
only 1 node
|
||
→ replica unassigned
|
||
→ yellow cluster
|
||
```
|
||
|
||
### Mistake: confusing Helm replicas with shard replicas
|
||
|
||
They control different layers.
|
||
|
||
### Mistake: using normal ClusterIP as StatefulSet `serviceName`
|
||
|
||
StatefulSet stable network identity should use the governing headless Service.
|
||
|
||
### Mistake: treating port 80 as OpenSearch's main port
|
||
|
||
OpenSearch REST and transport use separate ports.
|
||
|
||
### Mistake: mounting the PVC to `/mnt` without changing `path.data`
|
||
|
||
The PVC can remain unused.
|
||
|
||
### Mistake: putting tolerations under the container
|
||
|
||
Tolerations are pod-level settings.
|
||
|
||
### Mistake: using pod anti-affinity as a node blacklist
|
||
|
||
Pod anti-affinity controls pod co-location, not arbitrary node exclusion.
|
||
|
||
### Mistake: assuming Nginx Proxy Manager protects node-to-node traffic
|
||
|
||
It does not.
|
||
|
||
### Mistake: treating Base64 as encryption
|
||
|
||
It is only encoding.
|
||
|
||
### Mistake: reusing admin credentials for Fluent Bit and Dashboards
|
||
|
||
Violates least privilege.
|
||
|
||
### Mistake: very long FQDN as certificate CN
|
||
|
||
Can hit X.509 CN length limits.
|
||
|
||
Use a short CN and SAN for full DNS names.
|
||
|
||
---
|
||
|
||
## 42. Operational Monitoring
|
||
|
||
### Kubernetes
|
||
|
||
Monitor:
|
||
|
||
- Pod restarts
|
||
- CPU throttling
|
||
- Memory RSS
|
||
- OOMKilled events
|
||
- PVC usage
|
||
- Node pressure
|
||
- Scheduling failures
|
||
|
||
### OpenSearch
|
||
|
||
Monitor:
|
||
|
||
- JVM heap pressure
|
||
- GC
|
||
- Cluster health
|
||
- Shard allocation
|
||
- Pending tasks
|
||
- Indexing latency
|
||
- Search latency
|
||
- Circuit breakers
|
||
- Disk watermarks
|
||
|
||
### Longhorn
|
||
|
||
Monitor:
|
||
|
||
- Volume health
|
||
- Replica count
|
||
- Replica placement
|
||
- Disk usage
|
||
- Snapshot growth
|
||
|
||
### Security
|
||
|
||
Monitor:
|
||
|
||
- Certificate expiry
|
||
- Authentication failures
|
||
- Unauthorized requests
|
||
- Admin credential usage
|
||
- TLS handshake errors
|
||
|
||
---
|
||
|
||
## 43. Certificate Rotation
|
||
|
||
Track:
|
||
|
||
```text
|
||
Not Before
|
||
Not After
|
||
```
|
||
|
||
Production certificate lifecycle should include:
|
||
|
||
- Expiry monitoring
|
||
- Renewal schedule
|
||
- Secure CA access
|
||
- Leaf-certificate replacement procedure
|
||
- Emergency replacement/revocation procedure
|
||
- Admin certificate rotation
|
||
- Protected CA backup
|
||
|
||
---
|
||
|
||
## 44. Backup Strategy
|
||
|
||
OpenSearch shard replicas are **not backups**.
|
||
|
||
They do not protect from:
|
||
|
||
- Accidental deletion
|
||
- Malicious deletion
|
||
- Corruption replicated across nodes
|
||
- Entire cluster loss
|
||
|
||
Use snapshots to independent storage such as object storage/S3-compatible storage.
|
||
|
||
Include:
|
||
|
||
- Separate credentials
|
||
- Snapshot retention
|
||
- Snapshot lifecycle
|
||
- Restore testing
|
||
|
||
---
|
||
|
||
## 45. Security Hardening Checklist
|
||
|
||
```text
|
||
[ ] Security plugin enabled
|
||
[ ] Demo certificates removed
|
||
[ ] Custom internal Root CA used
|
||
[ ] Unique node certificates
|
||
[ ] Separate admin certificate
|
||
[ ] root-ca-key.pem stored offline
|
||
[ ] REST HTTPS enabled
|
||
[ ] Transport TLS enabled
|
||
[ ] Port 9300 not externally exposed
|
||
[ ] Port 9200 private/internal
|
||
[ ] Dashboards exposed through Nginx Proxy Manager
|
||
[ ] Dedicated Fluent Bit user
|
||
[ ] Dedicated Dashboards server user
|
||
[ ] Human users use least-privilege roles
|
||
[ ] Secret access protected by RBAC
|
||
[ ] Kubernetes Secret encryption at rest considered
|
||
[ ] NetworkPolicy considered
|
||
[ ] Certificate expiry monitored
|
||
```
|
||
|
||
---
|
||
|
||
## 46. Go-Live Checklist
|
||
|
||
### Kubernetes
|
||
|
||
```text
|
||
[ ] Namespace = observability
|
||
[ ] 3 eligible OpenSearch nodes
|
||
[ ] Storage taint/toleration validated
|
||
[ ] Hard anti-affinity validated
|
||
[ ] Control-plane excluded
|
||
[ ] Longhorn healthy
|
||
[ ] vm.max_map_count validated
|
||
```
|
||
|
||
### Helm
|
||
|
||
```text
|
||
[ ] Chart version pinned
|
||
[ ] OpenSearch version pinned
|
||
[ ] Custom values reviewed
|
||
[ ] Rendered YAML reviewed
|
||
[ ] No unexpected resources
|
||
```
|
||
|
||
### Storage
|
||
|
||
```text
|
||
[ ] 120Gi × 3 planned
|
||
[ ] Longhorn replication capacity calculated
|
||
[ ] Sufficient disk headroom
|
||
```
|
||
|
||
### TLS
|
||
|
||
```text
|
||
[ ] Root CA generated
|
||
[ ] Root CA key offline
|
||
[ ] Admin certificate valid
|
||
[ ] Node certificates valid
|
||
[ ] SANs correct
|
||
[ ] Certificate/key pairs match
|
||
[ ] DNs extracted
|
||
[ ] TLS Secrets created
|
||
```
|
||
|
||
### OpenSearch
|
||
|
||
```text
|
||
[ ] 3 nodes joined
|
||
[ ] Cluster-manager election healthy
|
||
[ ] HTTPS works
|
||
[ ] Authentication works
|
||
[ ] Shard replica policy set
|
||
[ ] Cluster health verified
|
||
```
|
||
|
||
### Dashboards / Fluent Bit
|
||
|
||
```text
|
||
[ ] Dashboards trusts internal CA
|
||
[ ] Fluent Bit trusts internal CA
|
||
[ ] Admin password not reused
|
||
[ ] Nginx Proxy Manager exposes Dashboards only
|
||
```
|
||
|
||
---
|
||
|
||
## 47. Recommended Next Steps
|
||
|
||
Continue in this order:
|
||
|
||
1. Verify generated certificates
|
||
2. Extract exact admin and node DNs
|
||
3. Create Kubernetes TLS Secrets
|
||
4. Build per-pod certificate-selection logic
|
||
5. Add custom TLS configuration to Helm values
|
||
6. Render Helm manifests
|
||
7. Review StatefulSet and Services
|
||
8. Validate node placement
|
||
9. Install OpenSearch
|
||
10. Verify the 3-node HTTPS/TLS cluster
|
||
11. Deploy OpenSearch Dashboards
|
||
12. Configure Nginx Proxy Manager
|
||
13. Create Fluent Bit least-privilege ingest identity
|
||
14. Connect Fluent Bit over HTTPS
|
||
15. Configure index templates and shard replica policy
|
||
16. Configure snapshots/backups
|
||
17. Add monitoring and certificate-expiry alerts
|
||
|
||
---
|
||
|
||
## 48. Troubleshooting Decision Tree
|
||
|
||
### Pod Pending
|
||
|
||
Investigate:
|
||
|
||
```text
|
||
Node affinity
|
||
Pod anti-affinity
|
||
Taints
|
||
Tolerations
|
||
CPU/memory requests
|
||
PVC scheduling/topology
|
||
```
|
||
|
||
### Pod CrashLoopBackOff
|
||
|
||
Investigate:
|
||
|
||
```text
|
||
JVM heap
|
||
vm.max_map_count
|
||
TLS file paths
|
||
Private-key format
|
||
Certificate permissions
|
||
Bootstrap admin password
|
||
OpenSearch logs
|
||
```
|
||
|
||
### Nodes do not form one cluster
|
||
|
||
Investigate:
|
||
|
||
```text
|
||
Headless Service
|
||
DNS resolution
|
||
Transport port 9300
|
||
Transport TLS
|
||
Certificate SAN
|
||
nodes_dn
|
||
Discovery settings
|
||
Initial cluster bootstrap settings
|
||
```
|
||
|
||
### HTTPS fails
|
||
|
||
Investigate:
|
||
|
||
```text
|
||
REST TLS enabled
|
||
Certificate path
|
||
Private key path
|
||
Trusted Root CA
|
||
Client hostname
|
||
SAN
|
||
Certificate expiry
|
||
```
|
||
|
||
### Dashboards cannot connect
|
||
|
||
Investigate:
|
||
|
||
```text
|
||
OpenSearch client Service
|
||
Port 9200
|
||
HTTPS
|
||
Root CA trust
|
||
Dashboards backend credential
|
||
NetworkPolicy
|
||
```
|
||
|
||
### Fluent Bit cannot ingest
|
||
|
||
Investigate:
|
||
|
||
```text
|
||
OpenSearch endpoint
|
||
HTTPS
|
||
CA verification
|
||
Credential
|
||
Role permissions
|
||
Index permissions
|
||
Network path
|
||
```
|
||
|
||
---
|
||
|
||
## 49. Core Lessons
|
||
|
||
```text
|
||
1. Stateful identity is not the same as pod IP.
|
||
2. StatefulSet provides stable identity; PVC stores persistent data.
|
||
3. Headless Service is for peer discovery and per-pod DNS.
|
||
4. ClusterIP Service is for normal clients.
|
||
5. Port 9200 and 9300 have different purposes.
|
||
6. Kubernetes replicas and OpenSearch shard replicas are different.
|
||
7. Redundancy works only when copies are separated across failure domains.
|
||
8. Toleration permits; affinity selects; anti-affinity separates.
|
||
9. A PVC is useless if the application writes somewhere else.
|
||
10. Helm should be rendered and reviewed before install.
|
||
11. Base64 is not encryption.
|
||
12. Reverse-proxy TLS does not protect internal OpenSearch transport.
|
||
13. Production TLS requires controlled PKI and certificate lifecycle.
|
||
14. Root CA private key never belongs in OpenSearch pods.
|
||
15. Unique node certificates are better than sharing one node key.
|
||
16. SANs are critical for DNS-based TLS verification.
|
||
17. Admin, Dashboards, and Fluent Bit should use separate identities.
|
||
18. OpenSearch replication is not a backup.
|
||
```
|
||
|
||
---
|
||
|
||
## 50. Final Target State
|
||
|
||
```text
|
||
Kubernetes:
|
||
3 OpenSearch pods
|
||
3 separate Kubernetes nodes
|
||
3 separate Longhorn PVCs
|
||
no OpenSearch pod on control-plane
|
||
|
||
OpenSearch:
|
||
3-node cluster
|
||
transport TLS
|
||
REST HTTPS
|
||
correct shard replication
|
||
healthy cluster-manager election
|
||
|
||
Security:
|
||
custom internal CA
|
||
unique node certificates
|
||
separate admin certificate
|
||
dedicated Dashboards credential
|
||
dedicated Fluent Bit credential
|
||
least privilege
|
||
|
||
Access:
|
||
Browser → Nginx Proxy Manager → Dashboards
|
||
Dashboards → HTTPS OpenSearch ClusterIP
|
||
Fluent Bit → HTTPS OpenSearch private endpoint
|
||
|
||
Operations:
|
||
monitoring
|
||
snapshots/backups
|
||
certificate expiry tracking
|
||
restore testing
|
||
```
|
||
|
||
---
|
||
|
||
## 51. Current Project Status
|
||
|
||
At the end of this chat, the design has reached this point:
|
||
|
||
```text
|
||
[✓] 3-node production topology selected
|
||
[✓] Longhorn selected for PVCs
|
||
[✓] 120Gi per PVC planned
|
||
[✓] Resource requests/limits discussed
|
||
[✓] JVM heap sizing discussed
|
||
[✓] Hard anti-affinity selected
|
||
[✓] Storage taint/toleration verified
|
||
[✓] HTTPS selected
|
||
[✓] Production TLS selected
|
||
[✓] Root CA/admin/node certificates generated
|
||
[ ] Kubernetes TLS Secrets integration
|
||
[ ] Per-pod certificate selection
|
||
[ ] OpenSearch TLS settings in Helm values
|
||
[ ] Rendered manifest validation
|
||
[ ] OpenSearch deployment
|
||
[ ] Dashboards deployment
|
||
[ ] Fluent Bit integration
|
||
```
|
||
|
||
The immediate next phase is:
|
||
|
||
```text
|
||
Certificates
|
||
↓
|
||
Kubernetes TLS Secrets
|
||
↓
|
||
Helm certificate mounts
|
||
↓
|
||
OpenSearch TLS configuration
|
||
↓
|
||
Render and validate
|
||
↓
|
||
Deploy
|
||
```
|