mosiur/brilliant-monitoring
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Convert helm-charts-observability to regular directory

Browse Source
This commit is contained in:
Mosiur Rahman
2026-08-13 07:38:55 +00:00
parent 0362f95667
commit beae59543c
93 changed files with 7648 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
helm-charts-observability/charts/data-prepper/.helmignore Normal file
View File

@@ -0,0 +1,23 @@
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*.orig
*~
# Various IDEs
.project
.idea/
*.tmproj
.vscode/

22
helm-charts-observability/charts/data-prepper/CHANGELOG.md Normal file
View File

@@ -0,0 +1,22 @@
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [0.3.1]
### Added
- Added configurable `initContainers`
## [0.3.0]
### Added
- Demo pipeline not part of values.yaml. New value `pipelineConfig.demoPipeline` controls creation of demo pipeline.
## [0.2.0]
### Added
- Added configurable `global.dockerRegistry`
## [0.1.0]
### Added
- Create initial version of data-prepper helm chart

50
helm-charts-observability/charts/data-prepper/Chart.yaml Normal file
View File

@@ -0,0 +1,50 @@
apiVersion: v2
name: data-prepper
description: A Helm chart for Data Prepper
# A chart can be either an 'application' or a 'library' chart.
#
# Application charts are a collection of templates that can be packaged into versioned archives
# to be deployed.
#
# Library charts provide useful utilities or functions for the chart developer. They're included as
# a dependency of application charts to inject those utilities and functions into the rendering
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.3.1
# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
# follow Semantic Versioning. They should reflect the version the application is using.
# It is recommended to use it with quotes.
appVersion: "2.8.0"
maintainers:
- name: gaiksaya
url: https://github.com/gaiksaya
- name: peterzhuamazon
url: https://github.com/peterzhuamazon
- name: prudhvigodithi
url: https://github.com/prudhvigodithi
- name: sergk
url: https://github.com/sergk
- name: TheAlgo
url: https://github.com/TheAlgo
home: https://opensearch.org/docs/latest/data-prepper/
sources:
- https://github.com/opensearch-project/data-prepper
- https://github.com/opensearch-project/helm-charts
annotations:
artifacthub.io/category: monitoring-logging
artifacthub.io/license: Apache-2.0
artifacthub.io/links: |
- name: Data Prepper Documentation
url: https://opensearch.org/docs/latest/data-prepper/
- name: OpenSearch Project
url: https://opensearch.org

130
helm-charts-observability/charts/data-prepper/README.md Normal file
View File

@@ -0,0 +1,130 @@
# Data Prepper Helm Chart
![Version: 0.1.0](https://img.shields.io/badge/Version-0.1.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 2.8.0](https://img.shields.io/badge/AppVersion-2.8.0-informational?style=flat-square)
A Helm chart for Data Prepper
**Homepage:** <https://opensearch.org/docs/latest/data-prepper/>
Data Prepper is an essential component of the OpenSearch project, designed for high-volume data transformation and ingestion into OpenSearch. This Helm chart simplifies deploying Data Prepper on Kubernetes environments, ensuring you can easily set up your data processing pipelines.
## Requirements
Before installing the Data Prepper Helm chart, ensure your environment meets the following requirements:
* Kubernetes >= 1.14
* Helm >= 2.17.0
* We recommend having at least 4 GiB of memory available for this deployment. A minimum of 2 GiB may suffice, but less than that could lead to deployment failures.
## Installation
To install the Data Prepper Helm chart, follow these steps:
* Add the OpenSearch Helm repository if you haven't already:
```bash
helm repo add opensearch https://opensearch-project.github.io/helm-charts/
helm repo update
```
* Install the Data Prepper chart with:
```bash
helm install my-data-prepper-release opensearch/data-prepper
```
Replace my-data-prepper-release with your desired release name. When no explicit pipeline is defined, this will configure a demo pipeline using a random source and stdout sink.
## Configuration
The Data Prepper Helm chart comes with a variety of configuration options to tailor the deployment to your needs.
The default values are specified in the [values.yaml](values.yaml) file. You can override these values by providing your own `values.yaml` file during installation or by specifying configuration options with --set flags.
For a detailed list of configuration options, refer to the values.yaml file or the [Data Prepper documentation](https://opensearch.org/docs/latest/data-prepper/managing-data-prepper/configuring-data-prepper/).
## Uninstalling the Chart
To uninstall/delete the my-data-prepper deployment:
```bash
helm delete my-data-prepper
```
This command removes all the Kubernetes components associated with the chart and deletes the release.
## Contributing
We welcome contributions! Please read our [CONTRIBUTING.md](../../CONTRIBUTING.md) for details on how to submit contributions to this project.
## Maintainers
| Name | Email | Url |
| ---- | ------ | --- |
| gaiksaya | | <https://github.com/gaiksaya> |
| peterzhuamazon | | <https://github.com/peterzhuamazon> |
| prudhvigodithi | | <https://github.com/prudhvigodithi> |
| sergk | | <https://github.com/sergk> |
| TheAlgo | | <https://github.com/TheAlgo> |
## Source Code
* <https://github.com/opensearch-project/data-prepper>
* <https://github.com/opensearch-project/helm-charts>
## Values
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| affinity | object | `{}` | |
| autoscaling.enabled | bool | `false` | |
| autoscaling.maxReplicas | int | `100` | |
| autoscaling.minReplicas | int | `1` | |
| autoscaling.targetCPUUtilizationPercentage | int | `80` | |
| config | object | `{"data-prepper-config.yaml":"ssl: false\n# circuit_breakers:\n# heap:\n# usage: 2gb\n# reset: 30s\n# check_interval: 5s\n","log4j2-rolling.properties":"#\n# Copyright OpenSearch Contributors\n# SPDX-License-Identifier: Apache-2.0\n#\n\nstatus = error\ndest = err\nname = PropertiesConfig\n\nproperty.filename = log/data-prepper/data-prepper.log\n\nappender.console.type = Console\nappender.console.name = STDOUT\nappender.console.layout.type = PatternLayout\nappender.console.layout.pattern = %d{ISO8601} [%t] %-5p %40C - %m%n\n\nappender.rolling.type = RollingFile\nappender.rolling.name = RollingFile\nappender.rolling.fileName = ${filename}\nappender.rolling.filePattern = logs/data-prepper.log.%d{MM-dd-yy-HH}-%i.gz\nappender.rolling.layout.type = PatternLayout\nappender.rolling.layout.pattern = %d{ISO8601} [%t] %-5p %40C - %m%n\nappender.rolling.policies.type = Policies\nappender.rolling.policies.time.type = TimeBasedTriggeringPolicy\nappender.rolling.policies.time.interval = 1\nappender.rolling.policies.time.modulate = true\nappender.rolling.policies.size.type = SizeBasedTriggeringPolicy\nappender.rolling.policies.size.size=100MB\nappender.rolling.strategy.type = DefaultRolloverStrategy\nappender.rolling.strategy.max = 168\n\nrootLogger.level = warn\nrootLogger.appenderRef.stdout.ref = STDOUT\nrootLogger.appenderRef.file.ref = RollingFile\n\nlogger.pipeline.name = org.opensearch.dataprepper.pipeline\nlogger.pipeline.level = info\n\nlogger.parser.name = org.opensearch.dataprepper.parser\nlogger.parser.level = info\n\nlogger.plugins.name = org.opensearch.dataprepper.plugins\nlogger.plugins.level = info\n"}` | Data Prepper configuration |
| config."data-prepper-config.yaml" | string | `"ssl: false\n# circuit_breakers:\n# heap:\n# usage: 2gb\n# reset: 30s\n# check_interval: 5s\n"` | Main Data Prepper configuration file content |
| config."log4j2-rolling.properties" | string | `"#\n# Copyright OpenSearch Contributors\n# SPDX-License-Identifier: Apache-2.0\n#\n\nstatus = error\ndest = err\nname = PropertiesConfig\n\nproperty.filename = log/data-prepper/data-prepper.log\n\nappender.console.type = Console\nappender.console.name = STDOUT\nappender.console.layout.type = PatternLayout\nappender.console.layout.pattern = %d{ISO8601} [%t] %-5p %40C - %m%n\n\nappender.rolling.type = RollingFile\nappender.rolling.name = RollingFile\nappender.rolling.fileName = ${filename}\nappender.rolling.filePattern = logs/data-prepper.log.%d{MM-dd-yy-HH}-%i.gz\nappender.rolling.layout.type = PatternLayout\nappender.rolling.layout.pattern = %d{ISO8601} [%t] %-5p %40C - %m%n\nappender.rolling.policies.type = Policies\nappender.rolling.policies.time.type = TimeBasedTriggeringPolicy\nappender.rolling.policies.time.interval = 1\nappender.rolling.policies.time.modulate = true\nappender.rolling.policies.size.type = SizeBasedTriggeringPolicy\nappender.rolling.policies.size.size=100MB\nappender.rolling.strategy.type = DefaultRolloverStrategy\nappender.rolling.strategy.max = 168\n\nrootLogger.level = warn\nrootLogger.appenderRef.stdout.ref = STDOUT\nrootLogger.appenderRef.file.ref = RollingFile\n\nlogger.pipeline.name = org.opensearch.dataprepper.pipeline\nlogger.pipeline.level = info\n\nlogger.parser.name = org.opensearch.dataprepper.parser\nlogger.parser.level = info\n\nlogger.plugins.name = org.opensearch.dataprepper.plugins\nlogger.plugins.level = info\n"` | Log4j2 configuration for Data Prepper logging |
| extraEnvs | list | `[]` | Extra environment variables to pass to the Data Prepper container |
| fullnameOverride | string | `""` | Override the default fullname for the deployment |
| global.dockerRegistry | string | `""` | Set if you want to change the default docker registry, e.g. a private one. |
| image.pullPolicy | string | `"IfNotPresent"` | The image tag to pull. Default: IfNotPresent |
| image.repository | string | `"opensearchproject/data-prepper"` | The image repository from which to pull the Data Prepper image |
| image.tag | string | `""` | Overrides the image tag whose default is the chart appVersion. |
| imagePullSecrets | list | `[]` | List of imagePullSecrets to use if the Docker image is stored in a private registry |
| ingress.annotations | object | `{}` | |
| ingress.className | string | `""` | |
| ingress.enabled | bool | `false` | |
| ingress.hosts[0].host | string | `"chart-example.local"` | |
| ingress.hosts[0].paths[0].path | string | `"/"` | |
| ingress.hosts[0].paths[0].pathType | string | `"ImplementationSpecific"` | |
| ingress.tls | list | `[]` | |
| nameOverride | string | `""` | Override the default name for the deployment |
| nodeSelector | object | `{}` | |
| pipelineConfig | object | (See below) | Pipeline configuration |
| pipelineConfig.enabled | boolean | `false` | Enable inline configuration in `config` sub key. |
| pipelineConfig.config | object | `{}` | Pipeline configuration file inline if `enabled` is set to true |
| pipelineConfig.demoPipeline | boolean | "" | If set, a demo pipeline will be provisioned with source `random` and sink `stdout`. |
| pipelineConfig.existingSecret | string | `""` | The name of an existing secret containing the pipeline configuration. If enabled is false existingSecret is used. The existingSecret must have a key named `pipelines.yaml`. |
| podAnnotations | object | `{}` | |
| podLabels | object | `{}` | |
| podSecurityContext | object | `{}` | |
| initContainers | list | `[]` | Optional list of init containers for the pod |
| ports | list | `[{"name":"http-source","port":2021},{"name":"otel-traces","port":21890},{"name":"otel-metrics","port":21891},{"name":"otel-logs","port":21892}]` | Data Prepper ports |
| ports[0] | object | `{"name":"http-source","port":2021}` | The port that the source is running on. Default value is 2021. Valid options are between 0 and 65535. https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/http-source/ |
| ports[1] | object | `{"name":"otel-traces","port":21890}` | The port that the otel_trace_source source runs on. Default value is 21890. https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/otel-trace-source/ |
| ports[2] | object | `{"name":"otel-metrics","port":21891}` | The port that the OpenTelemtry metrics source runs on. Default value is 21891. https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/otel-metrics-source/ |
| ports[3] | object | `{"name":"otel-logs","port":21892}` | Represents the port that the otel_logs_source source is running on. Default value is 21892. https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/otel-logs-source/ |
| replicaCount | int | `1` | |
| resources | object | `{}` | |
| securityContext | object | `{}` | |
| service.type | string | `"ClusterIP"` | |
| serviceAccount.annotations | object | `{}` | Annotations to add to the service account |
| serviceAccount.automount | bool | `true` | Automatically mount a ServiceAccount's API credentials? |
| serviceAccount.create | bool | `true` | Specifies whether a service account should be created |
| serviceAccount.name | string | `""` | The name of the service account to use. If not set and create is true, a name is generated using the fullname template |
| tolerations | list | `[]` | |
| volumeMounts | list | `[]` | |
| volumes | list | `[]` | |
## License
This project is licensed under the Apache License 2.0 - see the [LICENSE.txt](../../LICENSE.txt) file for details.

71
helm-charts-observability/charts/data-prepper/README.md.gotmpl Normal file
View File

@@ -0,0 +1,71 @@
# Data Prepper Helm Chart
{{ template "chart.deprecationWarning" . }}
{{ template "chart.badgesSection" . }}
{{ template "chart.description" . }}
{{ template "chart.homepageLine" . }}
Data Prepper is an essential component of the OpenSearch project, designed for high-volume data transformation and ingestion into OpenSearch. This Helm chart simplifies deploying Data Prepper on Kubernetes environments, ensuring you can easily set up your data processing pipelines.
## Requirements
Before installing the Data Prepper Helm chart, ensure your environment meets the following requirements:
* Kubernetes >= 1.14
* Helm >= 2.17.0
* We recommend having at least 4 GiB of memory available for this deployment. A minimum of 2 GiB may suffice, but less than that could lead to deployment failures.
## Installation
To install the Data Prepper Helm chart, follow these steps:
* Add the OpenSearch Helm repository if you haven't already:
```bash
helm repo add opensearch https://opensearch-project.github.io/helm-charts/
helm repo update
```
* Install the Data Prepper chart with:
```bash
helm install my-data-prepper-release opensearch/data-prepper
```
Replace my-data-prepper-release with your desired release name.
## Configuration
The Data Prepper Helm chart comes with a variety of configuration options to tailor the deployment to your needs.
The default values are specified in the [values.yaml](values.yaml) file. You can override these values by providing your own values.yaml file during installation or by specifying configuration options with --set flags.
For a detailed list of configuration options, refer to the values.yaml file or the [Data Prepper documentation](https://opensearch.org/docs/latest/data-prepper/managing-data-prepper/configuring-data-prepper/).
## Uninstalling the Chart
To uninstall/delete the my-data-prepper deployment:
```bash
helm delete my-data-prepper
```
This command removes all the Kubernetes components associated with the chart and deletes the release.
## Contributing
We welcome contributions! Please read our [CONTRIBUTING.md](../../CONTRIBUTING.md) for details on how to submit contributions to this project.
{{ template "chart.maintainersSection" . }}
{{ template "chart.sourcesSection" . }}
{{ template "chart.requirementsSection" . }}
{{ template "chart.valuesSection" . }}
## License
This project is licensed under the Apache License 2.0 - see the [LICENSE.txt](../../LICENSE.txt) file for details.

23
helm-charts-observability/charts/data-prepper/certificate_renew_sop.md Normal file
View File

@@ -0,0 +1,23 @@
renew certificate using this command
openssl x509 -req -in data-prepper-ingress.csr -CA $ROOT_CA_CERT -CAkey $ROOT_CA_KEY -CAcreateserial -out data-prepper-ingress.crt -days 365 -sha256 -extfile data-prepper-ingress.cnf -extensions server_extensions
k delete secret -n observability dataprepper-ingress-tls
then create kubernetes secret
k create secret --namespace observability tls dataprepper-ingress-tls --cert=data-prepper-ingress.crt --key=data-prepper-ingress.key --dry-run=client -o yaml | k apply -f -
verify
k get secret -n observability dataprepper-ingress-tls -o jsonpath='{.type}{"\n"}'
is should say kubernetes.io/tls
kubectl get secret dataprepper-ingress-tls \
-n observability \
-o jsonpath='{.data.tls\.crt}' |
base64 -d |
openssl x509 -noout -subject -issuer -ext subjectAltName

126
helm-charts-observability/charts/data-prepper/cpanel_path_structure.txt Normal file
View File

@@ -0,0 +1,126 @@
# System logs (/var/log)
/var/log/agent360-install.log
/var/log/agent360.log
/var/log/alternatives.log
/var/log/alternatives.log.1
/var/log/apache2/
/var/log/apport.log
/var/log/apt/
/var/log/audit/
/var/log/auth.log
/var/log/auth.log.1
/var/log/auth.log.2.gz
/var/log/auth.log.3.gz
/var/log/auth.log.4.gz
/var/log/bootstrap.log
/var/log/btmp
/var/log/btmp.1
/var/log/chkservd.log
/var/log/clamav/
/var/log/clam-update.log
/var/log/cloud-init.log
/var/log/cloud-init-output.log
/var/log/comet_backup_jobs_helper.log
/var/log/comet_backup_jobs_helper.log.1
/var/log/cpanel-install.log
/var/log/cpanel-server-traffic/
/var/log/dcpumon/
/var/log/dist-upgrade/
/var/log/dmesg
/var/log/dmesg.0
/var/log/dmesg.1.gz
/var/log/dpkg.log
/var/log/dpkg.log.1
/var/log/ea-passenger-analytics/
/var/log/exim_mainlog
/var/log/exim_mainlog.1.gz
/var/log/exim_mainlog.2.gz
/var/log/exim_mainlog.3.gz
/var/log/exim_mainlog.4.gz
/var/log/exim_paniclog
/var/log/exim_paniclog.1.gz
/var/log/exim_paniclog.2.gz
/var/log/exim_paniclog.3.gz
/var/log/exim_paniclog.4.gz
/var/log/exim_rejectlog
/var/log/exim_rejectlog.1.gz
/var/log/exim_rejectlog.2.gz
/var/log/exim_rejectlog.3.gz
/var/log/exim_rejectlog.4.gz
/var/log/fail2ban.log
/var/log/fail2ban.log.1
/var/log/fail2ban.log.2.gz
/var/log/fail2ban.log.3.gz
/var/log/fail2ban.log.4.gz
/var/log/faillog
/var/log/fontconfig.log
/var/log/imav-deploy.log
/var/log/imunify360/
/var/log/imunify360_user_logs/
/var/log/installer/
/var/log/journal/
/var/log/kern.log
/var/log/kern.log.1
/var/log/kern.log.2.gz
/var/log/kern.log.3.gz
/var/log/kern.log.4.gz
/var/log/landscape/
/var/log/lastlog
/var/log/lfd.log
/var/log/lfd.log.1.gz
/var/log/mail.err
/var/log/mail.log
/var/log/mail.log.1
/var/log/mail.log.2.gz
/var/log/mail.log.3.gz
/var/log/mail.log.4.gz
/var/log/munin/
/var/log/mysql/
/var/log/named/
/var/log/private/
/var/log/quota_enable.log
/var/log/README
/var/log/sw-cp-server/
/var/log/syslog
/var/log/syslog.1
/var/log/syslog.2.gz
/var/log/syslog.3.gz
/var/log/syslog.4.gz
/var/log/sysstat/
/var/log/ubuntu-advantage-apt-hook.log
/var/log/unattended-upgrades/
/var/log/wp-toolkit/
/var/log/wtmp
/var/log/xferlog.offsetftpsep
# cPanel logs (/usr/local/cpanel/logs)
/usr/local/cpanel/logs/access_log
/usr/local/cpanel/logs/api_tokens_log
/usr/local/cpanel/logs/backup_restore_manager_error_log
/usr/local/cpanel/logs/backup_restore_manager_log
/usr/local/cpanel/logs/build_locale_databases_log
/usr/local/cpanel/logs/cpbackup/
/usr/local/cpanel/logs/cpbackup_transport_history.log
/usr/local/cpanel/logs/cpbackup_transporter/
/usr/local/cpanel/logs/cpbackup_transporter.log
/usr/local/cpanel/logs/cpdavd_error_log
/usr/local/cpanel/logs/cpdavd_session_log
/usr/local/cpanel/logs/cphulkd.log
/usr/local/cpanel/logs/cphulkd_errors.log
/usr/local/cpanel/logs/cpwrapd_log
/usr/local/cpanel/logs/dnsadmin_log
/usr/local/cpanel/logs/error_log
/usr/local/cpanel/logs/incoming_http_requests.log
/usr/local/cpanel/logs/license_log
/usr/local/cpanel/logs/login_log
/usr/local/cpanel/logs/php-fpm/
/usr/local/cpanel/logs/queueprocd.log
/usr/local/cpanel/logs/secpol_log
/usr/local/cpanel/logs/session_log
/usr/local/cpanel/logs/spamd_error_log
/usr/local/cpanel/logs/splitlogs_log
/usr/local/cpanel/logs/stats_log
/usr/local/cpanel/logs/tailwatchd_log
/usr/local/cpanel/logs/update_analysis

17
helm-charts-observability/charts/data-prepper/prepper-ingress.yaml Normal file
View File

@@ -0,0 +1,17 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: prepper-ingress
namespace: observability
spec:
ingressClassName: nginx
rules:
- http:
paths:
- path: /logs
pathType: Prefix
backend:
service:
name: data-prepper
port:
number: 2021

160
helm-charts-observability/charts/data-prepper/prerequisit_inforamtion.md Normal file
View File

@@ -0,0 +1,160 @@
# Phase 1
## Infromation for preeper
- OpenSearch Kubernetes Service name
```
root@k8s-cp-01:~# k get svc -n observability
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
brilliant-opensearch-cluster-master ClusterIP 10.43.44.32 <none> 9200/TCP,9300/TCP,9600/TCP 4d18h
brilliant-opensearch-cluster-master-headless ClusterIP None <none> 9200/TCP,9300/TCP,9600/TCP 4d18h
brilliant-opensearch-dashboard-opensearch-dashboards ClusterIP 10.43.225.4 <none> 5601/TCP,9601/TCP 15h
root@k8s-cp-01:~# k get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.43.211.10 <pending> 80:31599/TCP,443:31186/TCP 26d
ingress-nginx-controller-admission ClusterIP 10.43.204.151 <none> 443/TCP 26d
ingress-nginx-controller-metrics ClusterIP 10.43.119.183 <none> 10254/TCP 26d
root@k8s-cp-01:~#
```
- OpenSearch HTTPS port
*Mentioned above*
- CA certificate
*CA Certificate is in the server monitoring/opensearch-pki*
*There is another certificate
```
apiVersion: v1
data:
root-ca.pem: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUY3VENDQTlXZ0F3SUJBZ0lVTC9hTytnYS9YY3lzakFYbGh1WG92d0xLWkJjd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2dZVXhDekFKQmdOVkJBWVRBa0pFTVE0d0RBWURWUVFJREFWRWFHRnJZVEVPTUF3R0ExVUVCd3dGUkdoaAphMkV4RWpBUUJnTlZCQW9NQ1VKeWFXeHNhV0Z1ZERFYk1Ca0dBMVVFQ3d3U1QzQmxibE5sWVhKamFDQlNiMjkwCklFTkJNU1V3SXdZRFZRUUREQnhDY21sc2JHbGhiblFnVDNCbGJsTmxZWEpqYUNCU2IyOTBJRU5CTUI0WERUSTIKTURjeU16RTFNREV4T1ZvWERUTTJNRGN5TURFMU1ERXhPVm93Z1lVeEN6QUpCZ05WQkFZVEFrSkVNUTR3REFZRApWUVFJREFWRWFHRnJZVEVPTUF3R0ExVUVCd3dGUkdoaGEyRXhFakFRQmdOVkJBb01DVUp5YVd4c2FXRnVkREViCk1Ca0dBMVVFQ3d3U1QzQmxibE5sWVhKamFDQlNiMjkwSUVOQk1TVXdJd1lEVlFRRERCeENjbWxzYkdsaGJuUWcKVDNCbGJsTmxZWEpqYUNCU2IyOTBJRU5CTUlJQ0lqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FnOEFNSUlDQ2dLQwpBZ0VBdXlEYmRaMUFycWd6K1l3SkNUcWlxREtuUzNDRGhHekNxK0h5WDN3clBnV0JvZFBraHJCV3dkNzQ4ZjdqClhZWDNQditQS0F0ZzVGYjdxa2ZZUFZJR0FPZUNiUWxkRlJnZld0S2F2MkdxcHZseUlKcy9OZW5PZzFZQ3IydVUKbDE3UnFPNk1BSGt6d1pNTVFDV1JXMlFiUy9weVY0VHd6dkNxSTR1ZVpzOFJ4MndwbERJdWFPVXdJMEVTMG8zdQpWdDJHMVlCTWlFOUNVTDliQWlYbWQ0UTR4a1NFRFRvOVY1SVk3Z2dHYjFTU0xWU2x6RDJHZElaMGN3VlE2LzcwCnUrQ1FPa005ZXhIRVR2eGpGTmZ4azI2YlRZVUJmNnBNS2VpSDdMbC9ucjZFVVVzTW5DSUR5VGtnK3Zta1M5ZWoKUmY1R0QwQ00rcnl1V1BiRHlHOCtzcDh4d1N1ZkcydDdkUGZRSTUzd1Zzc1BKSkk5b0lsZ0lJWXkyMkFUMlo4RwpKL2dZbmYwMU5LREhSV1dBU2FQaTBLT2dURldqRG5xam94Q0FHZjVGVFRJSHdBQjVsMS8rMXZFbEkyNjQ2M2d4CnBWcTk4Vyt4U3FQQ0xaTW9sWlE5TVkwbjVHeGgvZjhkK0RPb1BVT0lXazJqWXdvS1JGQXJiTlNTNnkySnVrS3YKSWNvMnN4YUhqZkhrdDdqdDBkckRLcjUrelI5bndUWDhVN3hBaWlTYkVTekN6OWlvdU9QSEcxMFFtdzl6VVRNMgoxdzJmMmlQRWpvbFpBVjFBaFU3dzFWNHBzc3JpWDIvNTlDR1dYT1VEcWNnd2MwdWh4djRnWFRndXBCeGcrYU5yCllybWZwNndtejdzN3lmblh3T2xVaVRDRW1Bc2NBemNkMmdCaGwvN1dpcmFRZFVzQ0F3RUFBYU5UTUZFd0hRWUQKVlIwT0JCWUVGSGJhbmo2RkdNVWZ3bEMwSytqY081TTlHd0FwTUI4R0ExVWRJd1FZTUJhQUZIYmFuajZGR01VZgp3bEMwSytqY081TTlHd0FwTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3RFFZSktvWklodmNOQVFFTEJRQURnZ0lCCkFBZVVVVUFMTk5FNDM2UEwvZXl6V0c3bmlPdndXODF3UkdVdGdEUnJpY2xFWjh0K3MzUG9MOXRibyszUGN1ekgKakxsUFArYmVLY2tUTEZKVGZZWUN3Z0dWZkdIRXE1ejV1SUt3ZWIyMG9GSEg3ek9DSzF3VVJBSUlPZnFycncyMApjcWNzdTRnK3NENnl0bllWN2Y1T0RFT2RNaXpjVXRrbGpMSjVieXJLSU9OV2FrSTBpd0xLeXFPQWlGeUVSY2xiCjB0ZzhoTDR6dkRLVUZ6dVJycXhxS0padzNWUWUvM1JWUFYwZ0RLUW96VEQ0M3lnQUdGVlVjckx1dHZOK2V5bEEKVDdYMGtNZXJlVVRKd1N6RDVvenhGRXFkaDlLNHZkZTA2a3BuSXpQS2tEanNFZiszeW9WeFVxRGJCeGV3ai9aOApodzhmT3lBWC9jV2UvZlhmWVdPV3RpTlhiQVJRd0pvUWlLb3lacHRuNGpvRXhLWVB2Qzg0RitINEZnZnpidXFkCnNnL2R2SjBxcldGaUlZeWZBRS9NNGM3VTBDUnVsZ1lETXJhbktIUHhxTllVT21pRzM4T0xLTDlFMmlvV002UHYKek82djVvUk5pazFFN3BrRUVMSUFPczR3U2MvdG45MU01TmtpSjgyRXNEQ2ZaVG5FVU9Cd3VsRzZSbW8vYWF0agpCWUJEaHo0dnhmN0ZjNG13WEdXNVZXc1Qzd2ZVckErZ3l3VWo5dlFqTTV6SjEybDl4NU9Dd3pNMTJuQWdSZ1dFCkh4ZDh5TFhkNE9rTWRaRWNVckY5eWtSMGJUZ3AyWnowL3lPR21KbVpieHRNdHlGaTVMcmt6VElodWJNS1l2bncKcWsrbWdaUmNMcXBWclRSYTNiaGdEVzJrSWpFT2I5SHB6NG5MZ3loUVpCMFAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
kind: Secret
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"root-ca.pem":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUY3VENDQTlXZ0F3SUJBZ0lVTC9hTytnYS9YY3lzakFYbGh1WG92d0xLWkJjd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2dZVXhDekFKQmdOVkJBWVRBa0pFTVE0d0RBWURWUVFJREFWRWFHRnJZVEVPTUF3R0ExVUVCd3dGUkdoaAphMkV4RWpBUUJnTlZCQW9NQ1VKeWFXeHNhV0Z1ZERFYk1Ca0dBMVVFQ3d3U1QzQmxibE5sWVhKamFDQlNiMjkwCklFTkJNU1V3SXdZRFZRUUREQnhDY21sc2JHbGhiblFnVDNCbGJsTmxZWEpqYUNCU2IyOTBJRU5CTUI0WERUSTIKTURjeU16RTFNREV4T1ZvWERUTTJNRGN5TURFMU1ERXhPVm93Z1lVeEN6QUpCZ05WQkFZVEFrSkVNUTR3REFZRApWUVFJREFWRWFHRnJZVEVPTUF3R0ExVUVCd3dGUkdoaGEyRXhFakFRQmdOVkJBb01DVUp5YVd4c2FXRnVkREViCk1Ca0dBMVVFQ3d3U1QzQmxibE5sWVhKamFDQlNiMjkwSUVOQk1TVXdJd1lEVlFRRERCeENjbWxzYkdsaGJuUWcKVDNCbGJsTmxZWEpqYUNCU2IyOTBJRU5CTUlJQ0lqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FnOEFNSUlDQ2dLQwpBZ0VBdXlEYmRaMUFycWd6K1l3SkNUcWlxREtuUzNDRGhHekNxK0h5WDN3clBnV0JvZFBraHJCV3dkNzQ4ZjdqClhZWDNQditQS0F0ZzVGYjdxa2ZZUFZJR0FPZUNiUWxkRlJnZld0S2F2MkdxcHZseUlKcy9OZW5PZzFZQ3IydVUKbDE3UnFPNk1BSGt6d1pNTVFDV1JXMlFiUy9weVY0VHd6dkNxSTR1ZVpzOFJ4MndwbERJdWFPVXdJMEVTMG8zdQpWdDJHMVlCTWlFOUNVTDliQWlYbWQ0UTR4a1NFRFRvOVY1SVk3Z2dHYjFTU0xWU2x6RDJHZElaMGN3VlE2LzcwCnUrQ1FPa005ZXhIRVR2eGpGTmZ4azI2YlRZVUJmNnBNS2VpSDdMbC9ucjZFVVVzTW5DSUR5VGtnK3Zta1M5ZWoKUmY1R0QwQ00rcnl1V1BiRHlHOCtzcDh4d1N1ZkcydDdkUGZRSTUzd1Zzc1BKSkk5b0lsZ0lJWXkyMkFUMlo4RwpKL2dZbmYwMU5LREhSV1dBU2FQaTBLT2dURldqRG5xam94Q0FHZjVGVFRJSHdBQjVsMS8rMXZFbEkyNjQ2M2d4CnBWcTk4Vyt4U3FQQ0xaTW9sWlE5TVkwbjVHeGgvZjhkK0RPb1BVT0lXazJqWXdvS1JGQXJiTlNTNnkySnVrS3YKSWNvMnN4YUhqZkhrdDdqdDBkckRLcjUrelI5bndUWDhVN3hBaWlTYkVTekN6OWlvdU9QSEcxMFFtdzl6VVRNMgoxdzJmMmlQRWpvbFpBVjFBaFU3dzFWNHBzc3JpWDIvNTlDR1dYT1VEcWNnd2MwdWh4djRnWFRndXBCeGcrYU5yCllybWZwNndtejdzN3lmblh3T2xVaVRDRW1Bc2NBemNkMmdCaGwvN1dpcmFRZFVzQ0F3RUFBYU5UTUZFd0hRWUQKVlIwT0JCWUVGSGJhbmo2RkdNVWZ3bEMwSytqY081TTlHd0FwTUI4R0ExVWRJd1FZTUJhQUZIYmFuajZGR01VZgp3bEMwSytqY081TTlHd0FwTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3RFFZSktvWklodmNOQVFFTEJRQURnZ0lCCkFBZVVVVUFMTk5FNDM2UEwvZXl6V0c3bmlPdndXODF3UkdVdGdEUnJpY2xFWjh0K3MzUG9MOXRibyszUGN1ekgKakxsUFArYmVLY2tUTEZKVGZZWUN3Z0dWZkdIRXE1ejV1SUt3ZWIyMG9GSEg3ek9DSzF3VVJBSUlPZnFycncyMApjcWNzdTRnK3NENnl0bllWN2Y1T0RFT2RNaXpjVXRrbGpMSjVieXJLSU9OV2FrSTBpd0xLeXFPQWlGeUVSY2xiCjB0ZzhoTDR6dkRLVUZ6dVJycXhxS0padzNWUWUvM1JWUFYwZ0RLUW96VEQ0M3lnQUdGVlVjckx1dHZOK2V5bEEKVDdYMGtNZXJlVVRKd1N6RDVvenhGRXFkaDlLNHZkZTA2a3BuSXpQS2tEanNFZiszeW9WeFVxRGJCeGV3ai9aOApodzhmT3lBWC9jV2UvZlhmWVdPV3RpTlhiQVJRd0pvUWlLb3lacHRuNGpvRXhLWVB2Qzg0RitINEZnZnpidXFkCnNnL2R2SjBxcldGaUlZeWZBRS9NNGM3VTBDUnVsZ1lETXJhbktIUHhxTllVT21pRzM4T0xLTDlFMmlvV002UHYKek82djVvUk5pazFFN3BrRUVMSUFPczR3U2MvdG45MU01TmtpSjgyRXNEQ2ZaVG5FVU9Cd3VsRzZSbW8vYWF0agpCWUJEaHo0dnhmN0ZjNG13WEdXNVZXc1Qzd2ZVckErZ3l3VWo5dlFqTTV6SjEybDl4NU9Dd3pNMTJuQWdSZ1dFCkh4ZDh5TFhkNE9rTWRaRWNVckY5eWtSMGJUZ3AyWnowL3lPR21KbVpieHRNdHlGaTVMcmt6VElodWJNS1l2bncKcWsrbWdaUmNMcXBWclRSYTNiaGdEVzJrSWpFT2I5SHB6NG5MZ3loUVpCMFAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo="},"kind":"Secret","metadata":{"annotations":{},"name":"opensearch-dashboard-ca","namespace":"observability"}}
creationTimestamp: "2026-08-03T13:45:31Z"
name: opensearch-dashboard-ca
namespace: observability
resourceVersion: "4485295"
uid: 7fba4407-7eb1-456a-a10c-e8500630df2e
type: Opaque
```
- Ingestion username/password
*username: logger*
*password: Enp129s0f0-l0gin*
```
root@k8s-cp-01:~# k get secret -n observability prepper-creds -o yaml
apiVersion: v1
data:
logger: RW5wMTI5czBmMC1sMGdnM3I=
kind: Secret
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"logger":"RW5wMTI5czBmMC1sMGdnM3I="},"kind":"Secret","metadata":{"annotations":{},"name":"prepper-creds","namespace":"observability"}}
creationTimestamp: "2026-08-04T06:17:58Z"
name: prepper-creds
namespace: observability
resourceVersion: "4604982"
uid: 6592aff6-3e6b-45a1-9953-348c647ca8d3
type: Opaque
root@k8s-cp-01:~#
```
- Target index name
*I am not understanding this*
We will use dedicated ingestion user
# Phase 2
### Data Prepper pipeline:
```
cpanel-log-pipeline:
source:
http:
path: /logs
ssl: true
sink:
- opensearch:
hosts: ["brilliant-opensearch-cluster-master:9200"]
insecure: false
username: logger
password: ${LOGGER_PASSWORD}
index_type: custom
index: cpanel-test-%{yyyy.MM.dd}
ssl: true
```
log paths of cpanel:
##### System logs (/var/log)
/var/log/apache2/
/var/log/audit/
/var/log/auth.log
/var/log/bootstrap.log
/var/log/chkservd.log
/var/log/clamav/
/var/log/cpanel-install.log
/var/log/cpanel-server-traffic/
/var/log/dmesg
/var/log/dpkg.log
/var/log/exim_mainlog
/var/log/exim_paniclog
/var/log/exim_rejectlog
/var/log/fail2ban.log
/var/log/kern.log
/var/log/lfd.log
/var/log/mail.err
/var/log/mail.log
/var/log/munin/
/var/log/mysql/
/var/log/named/
/var/log/private/
/var/log/quota_enable.log
/var/log/sw-cp-server/
/var/log/syslog
##### cPanel logs (/usr/local/cpanel/logs)
/usr/local/cpanel/logs/access_log
/usr/local/cpanel/logs/api_tokens_log
/usr/local/cpanel/logs/backup_restore_manager_error_log
/usr/local/cpanel/logs/backup_restore_manager_log
/usr/local/cpanel/logs/build_locale_databases_log
/usr/local/cpanel/logs/cpbackup/
/usr/local/cpanel/logs/cpbackup_transport_history.log
/usr/local/cpanel/logs/cpbackup_transporter/
/usr/local/cpanel/logs/cpbackup_transporter.log
/usr/local/cpanel/logs/cpdavd_error_log
/usr/local/cpanel/logs/cpdavd_session_log
/usr/local/cpanel/logs/cphulkd.log
/usr/local/cpanel/logs/cphulkd_errors.log
/usr/local/cpanel/logs/cpwrapd_log
/usr/local/cpanel/logs/dnsadmin_log
/usr/local/cpanel/logs/error_log
/usr/local/cpanel/logs/incoming_http_requests.log
/usr/local/cpanel/logs/license_log
/usr/local/cpanel/logs/login_log
/usr/local/cpanel/logs/php-fpm/
/usr/local/cpanel/logs/queueprocd.log
/usr/local/cpanel/logs/secpol_log
/usr/local/cpanel/logs/session_log
/usr/local/cpanel/logs/spamd_error_log
/usr/local/cpanel/logs/splitlogs_log
/usr/local/cpanel/logs/stats_log
/usr/local/cpanel/logs/tailwatchd_log
/usr/local/cpanel/logs/update_analysis
# Phase 3
I used values.yaml for creating the resources

226
helm-charts-observability/charts/data-prepper/template.yaml Normal file
View File

@@ -0,0 +1,226 @@
---
# Source: data-prepper/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: data-prepper
labels:
helm.sh/chart: data-prepper-0.3.1
app.kubernetes.io/name: data-prepper
app.kubernetes.io/instance: data-prepper
app.kubernetes.io/version: "2.8.0"
app.kubernetes.io/managed-by: Helm
automountServiceAccountToken: true
---
# Source: data-prepper/templates/secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: data-prepper-pipeline
labels:
helm.sh/chart: data-prepper-0.3.1
app.kubernetes.io/name: data-prepper
app.kubernetes.io/instance: data-prepper
app.kubernetes.io/version: "2.8.0"
app.kubernetes.io/managed-by: Helm
type: Opaque
stringData:
pipelines.yaml: |
cpanel-log-pipeline:
buffer:
bounded_blocking:
batch_size: 128
buffer_size: 2048
delay: 10
sink:
- opensearch:
cert: /usr/share/dataprepper/cert/root-ca.pem
hosts:
- https://brilliant-opensearch-cluster-master:9200
index: cpanel-test-%{yyyy.MM.dd}
index_type: custom
insecure: false
password: ${LOGGER_PASSWORD}
username: logger
source:
http:
path: /logs
ssl: false
workers: 2
---
# Source: data-prepper/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: data-prepper-config
labels:
helm.sh/chart: data-prepper-0.3.1
app.kubernetes.io/name: data-prepper
app.kubernetes.io/instance: data-prepper
app.kubernetes.io/version: "2.8.0"
app.kubernetes.io/managed-by: Helm
data:
data-prepper-config.yaml: |
ssl: false
# circuit_breakers:
# heap:
# usage: 2gb
# reset: 30s
# check_interval: 5s
log4j2-rolling.properties: |
#
# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0
#
status = error
dest = err
name = PropertiesConfig
property.filename = log/data-prepper/data-prepper.log
appender.console.type = Console
appender.console.name = STDOUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{ISO8601} [%t] %-5p %40C - %m%n
appender.rolling.type = RollingFile
appender.rolling.name = RollingFile
appender.rolling.fileName = ${filename}
appender.rolling.filePattern = logs/data-prepper.log.%d{MM-dd-yy-HH}-%i.gz
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = %d{ISO8601} [%t] %-5p %40C - %m%n
appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.policies.time.interval = 1
appender.rolling.policies.time.modulate = true
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling.policies.size.size=100MB
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.max = 168
rootLogger.level = warn
rootLogger.appenderRef.stdout.ref = STDOUT
rootLogger.appenderRef.file.ref = RollingFile
logger.pipeline.name = org.opensearch.dataprepper.pipeline
logger.pipeline.level = info
logger.parser.name = org.opensearch.dataprepper.parser
logger.parser.level = info
logger.plugins.name = org.opensearch.dataprepper.plugins
logger.plugins.level = info
---
# Source: data-prepper/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: data-prepper
labels:
helm.sh/chart: data-prepper-0.3.1
app.kubernetes.io/name: data-prepper
app.kubernetes.io/instance: data-prepper
app.kubernetes.io/version: "2.8.0"
app.kubernetes.io/managed-by: Helm
spec:
type: ClusterIP
ports:
- name: http-source
port: 2021
targetPort: 2021
protocol: TCP
selector:
app.kubernetes.io/name: data-prepper
app.kubernetes.io/instance: data-prepper
---
# Source: data-prepper/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: data-prepper
labels:
helm.sh/chart: data-prepper-0.3.1
app.kubernetes.io/name: data-prepper
app.kubernetes.io/instance: data-prepper
app.kubernetes.io/version: "2.8.0"
app.kubernetes.io/managed-by: Helm
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: data-prepper
app.kubernetes.io/instance: data-prepper
template:
metadata:
annotations:
checksum/config: d9f25a6756b9ae068c1ec208e538bfd4708057961af4ababf7abb949f2458ac
checksum/pipelineconfig: 2166ddb3315ca7f85e5ee3ab3ebee0ede10992fb0882d771a2f9b9d9188f5df
labels:
helm.sh/chart: data-prepper-0.3.1
app.kubernetes.io/name: data-prepper
app.kubernetes.io/instance: data-prepper
app.kubernetes.io/version: "2.8.0"
app.kubernetes.io/managed-by: Helm
spec:
serviceAccountName: data-prepper
securityContext:
{}
containers:
- name: data-prepper
env:
- name: LOGGER_PASSWORD
valueFrom:
secretKeyRef:
key: logger
secretName: prepper-creds
securityContext:
{}
image: "opensearchproject/data-prepper:2.8.0"
imagePullPolicy: IfNotPresent
ports:
- name: http-source
containerPort: 2021
protocol: TCP
- name: server
containerPort: 4900
protocol: TCP
livenessProbe:
httpGet:
path: /list
port: server
periodSeconds: 10
initialDelaySeconds: 60
failureThreshold: 2
readinessProbe:
httpGet:
path: /list
port: server
resources:
limits:
cpu: 4
memory: 6Gi
requests:
cpu: 2
memory: 4Gi
volumeMounts:
- name: data-prepper-config
mountPath: /usr/share/data-prepper/config
readOnly: true
- name: data-prepper-pipelines
mountPath: /usr/share/data-prepper/pipelines
readOnly: true
- mountPath: /usr/share/dataprepper/cert
name: root-ca
readOnly: true
volumes:
- name: data-prepper-config
configMap:
name: data-prepper-config
- name: data-prepper-pipelines
secret:
secretName: data-prepper-pipeline
- name: root-ca
secret:
secretName: opensearch-dashboard-ca

30
helm-charts-observability/charts/data-prepper/templates/NOTES.txt Normal file
View File

@@ -0,0 +1,30 @@
1. Get the application URL by running these commands:
{{- if .Values.ingress.enabled }}
{{- range $host := .Values.ingress.hosts }}
{{- range .paths }}
http{{ if $.Values.ingress.tls }}s{{ end }}://{{ $host.host }}{{ .path }}
{{- end }}
{{- end }}
{{- else if contains "NodePort" .Values.service.type }}
export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "data-prepper.fullname" . }})
export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT
{{- else if contains "LoadBalancer" .Values.service.type }}
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
You can watch its status by running 'kubectl get --namespace {{ .Release.Namespace }} svc -w {{ include "data-prepper.fullname" . }}'
export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "data-prepper.fullname" . }} --template "{{"{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}"}}")
echo http://$SERVICE_IP:{{ .Values.service.port }}
{{- else if contains "ClusterIP" .Values.service.type }}
export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "data-prepper.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
export CONTAINER_PORT=$(kubectl get pod --namespace {{ .Release.Namespace }} $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8080:$CONTAINER_PORT
{{- end }}
{{- if eq "true" (include "data-prepper.demoPipeline" .) }}
The demo pipeline configuration is enabled, using the `random` source and `stdout` sink.
{{- else if .Values.pipelineConfig.enabled }}
Inline pipeline configuration is enabled. Please refer to the values.yaml file for the configuration.
{{- else }}
Pipeline configuration from secret {{ .Values.pipelineConfig.existingSecret }} is enabled.
{{- end }}

95
helm-charts-observability/charts/data-prepper/templates/_helpers.tpl Normal file
View File

@@ -0,0 +1,95 @@
{{/*
Expand the name of the chart.
*/}}
{{- define "data-prepper.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "data-prepper.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "data-prepper.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Common labels
*/}}
{{- define "data-prepper.labels" -}}
helm.sh/chart: {{ include "data-prepper.chart" . }}
{{ include "data-prepper.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}
{{/*
Selector labels
*/}}
{{- define "data-prepper.selectorLabels" -}}
app.kubernetes.io/name: {{ include "data-prepper.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
{{/*
Create the name of the service account to use
*/}}
{{- define "data-prepper.serviceAccountName" -}}
{{- if .Values.serviceAccount.create }}
{{- default (include "data-prepper.fullname" .) .Values.serviceAccount.name }}
{{- else }}
{{- default "default" .Values.serviceAccount.name }}
{{- end }}
{{- end }}
{{/*
Create the dockerRegistry prefix if defined
*/}}
{{- define "data-prepper.dockerRegistry" -}}
{{- if eq .Values.global.dockerRegistry "" -}}
{{- .Values.global.dockerRegistry -}}
{{- else -}}
{{- .Values.global.dockerRegistry | trimSuffix "/" | printf "%s/" -}}
{{- end -}}
{{- end -}}
{{/*
Utility to decide whether demoPipeline should be enabled.
If pipelineConfig.demoPipeline is true/false, return that value.
If pipelineConfig.demoPipeline is undefined, return true if no other pipeline is configured.
*/}}
{{- define "data-prepper.demoPipeline" -}}
{{- $demoPipeline := .Values.pipelineConfig.demoPipeline | toString -}}
{{- if eq $demoPipeline "true" -}}
{{ true }}
{{- else if eq $demoPipeline "false" -}}
{{ false }}
{{- else if or (not $demoPipeline) (eq $demoPipeline "") -}}
{{- if and (not .Values.pipelineConfig.enabled) (not .Values.pipelineConfig.existingSecret) -}}
{{ true }}
{{- else -}}
{{ false }}
{{- end -}}
{{- else -}}
{{ false }}
{{- end -}}
{{- end -}}

15
helm-charts-observability/charts/data-prepper/templates/configmap.yaml Normal file
View File

@@ -0,0 +1,15 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "data-prepper.fullname" . }}-config
labels:
{{- include "data-prepper.labels" . | nindent 4 }}
data:
{{- range $configName, $configYaml := .Values.config }}
{{ $configName }}: |
{{- if eq (kindOf $configYaml) "map" }}
{{- tpl (toYaml $configYaml) $ | nindent 4 }}
{{- else }}
{{- tpl $configYaml $ | nindent 4 -}}
{{- end -}}
{{- end -}}

25
helm-charts-observability/charts/data-prepper/templates/demoPipeline.yaml Normal file
View File

@@ -0,0 +1,25 @@
{{- if eq "true" (include "data-prepper.demoPipeline" .) }}
apiVersion: v1
kind: Secret
metadata:
name: {{ include "data-prepper.fullname" . }}-demo-pipeline
labels:
{{- include "data-prepper.labels" . | nindent 4 }}
type: Opaque
stringData:
pipelines.yaml: |
simple-sample-pipeline:
workers: 2 # the number of workers
delay: 5000 # in milliseconds, how long workers wait between read attempts
source:
random: {}
buffer:
bounded_blocking:
buffer_size: 1024 # max number of records the buffer accepts
batch_size: 256 # max number of records the buffer drains after each read
processor:
- string_converter:
upper_case: true
sink:
- stdout: {}
{{- end }}

118
helm-charts-observability/charts/data-prepper/templates/deployment.yaml Normal file
View File

@@ -0,0 +1,118 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "data-prepper.fullname" . }}
labels:
{{- include "data-prepper.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "data-prepper.selectorLabels" . | nindent 6 }}
template:
metadata:
annotations:
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
{{- if .Values.config }}
checksum/config: {{ include (print .Template.BasePath "/configmap.yaml") . | sha256sum | trunc 63 }}
{{- end }}
{{- if .Values.pipelineConfig.enabled }}
checksum/pipelineconfig: {{ include (print .Template.BasePath "/secret.yaml") . | sha256sum | trunc 63 }}
{{- end }}
labels:
{{- include "data-prepper.labels" . | nindent 8 }}
{{- with .Values.podLabels }}
{{- toYaml . | nindent 8 }}
{{- end }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "data-prepper.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
- name: {{ .Chart.Name }}
env:
{{- with .Values.extraEnvs }}
{{- toYaml . | nindent 12 }}
{{- end }}
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ template "data-prepper.dockerRegistry" . }}{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
{{- range .Values.ports }}
- name: {{ .name }}
containerPort: {{ .port }}
protocol: TCP
{{- end }}
- name: server
containerPort: {{ (.Values.config).serverPort | default 4900 }}
protocol: TCP
livenessProbe:
httpGet:
path: /list
port: server
{{- if not ( empty (.Values.config).ssl ) }}
scheme: HTTPS
{{- end }}
periodSeconds: 10
initialDelaySeconds: 60
failureThreshold: 2
readinessProbe:
httpGet:
path: /list
port: server
{{- if not ( empty (.Values.config).ssl ) }}
scheme: HTTPS
{{- end }}
resources:
{{- toYaml .Values.resources | nindent 12 }}
volumeMounts:
- name: data-prepper-config
mountPath: /usr/share/data-prepper/config
readOnly: true
- name: data-prepper-pipelines
mountPath: /usr/share/data-prepper/pipelines
readOnly: true
{{- with .Values.volumeMounts }}
{{- toYaml . | nindent 12 }}
{{- end }}
volumes:
- name: data-prepper-config
configMap:
name: {{ include "data-prepper.fullname" . }}-config
- name: data-prepper-pipelines
secret:
{{- if eq "true" (include "data-prepper.demoPipeline" .) }}
secretName: {{ include "data-prepper.fullname" . }}-demo-pipeline
{{- else if .Values.pipelineConfig.enabled }}
secretName: {{ include "data-prepper.fullname" . }}-pipeline
{{- else }}
secretName: {{ required "A valid pipeline configuration is required! Please set one of:\npipelineConfig.existingSecret - for your custom pipeline secret\npipelineConfig.enabled - for inline configuration in values.yaml or\npipelineConfig.demoPipeline=true - for a sample demo pipeline using random source and stdout sink." .Values.pipelineConfig.existingSecret }}
{{- end }}
{{- with .Values.volumes }}
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.initContainers }}
initContainers:
{{- toYaml . | nindent 8 }}
{{- end }}

32
helm-charts-observability/charts/data-prepper/templates/hpa.yaml Normal file
View File

@@ -0,0 +1,32 @@
{{- if .Values.autoscaling.enabled }}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: {{ include "data-prepper.fullname" . }}
labels:
{{- include "data-prepper.labels" . | nindent 4 }}
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{ include "data-prepper.fullname" . }}
minReplicas: {{ .Values.autoscaling.minReplicas }}
maxReplicas: {{ .Values.autoscaling.maxReplicas }}
metrics:
{{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
{{- end }}
{{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
{{- end }}
{{- end }}

61
helm-charts-observability/charts/data-prepper/templates/ingress.yaml Normal file
View File

@@ -0,0 +1,61 @@
{{- if .Values.ingress.enabled -}}
{{- $fullName := include "data-prepper.fullname" . -}}
{{- $svcPort := .Values.service.port -}}
{{- if and .Values.ingress.className (not (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion)) }}
{{- if not (hasKey .Values.ingress.annotations "kubernetes.io/ingress.class") }}
{{- $_ := set .Values.ingress.annotations "kubernetes.io/ingress.class" .Values.ingress.className}}
{{- end }}
{{- end }}
{{- if semverCompare ">=1.19-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1
{{- else if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1beta1
{{- else -}}
apiVersion: extensions/v1beta1
{{- end }}
kind: Ingress
metadata:
name: {{ $fullName }}
labels:
{{- include "data-prepper.labels" . | nindent 4 }}
{{- with .Values.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
{{- if and .Values.ingress.className (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion) }}
ingressClassName: {{ .Values.ingress.className }}
{{- end }}
{{- if .Values.ingress.tls }}
tls:
{{- range .Values.ingress.tls }}
- hosts:
{{- range .hosts }}
- {{ . | quote }}
{{- end }}
secretName: {{ .secretName }}
{{- end }}
{{- end }}
rules:
{{- range .Values.ingress.hosts }}
- host: {{ .host | quote }}
http:
paths:
{{- range .paths }}
- path: {{ .path }}
{{- if and .pathType (semverCompare ">=1.18-0" $.Capabilities.KubeVersion.GitVersion) }}
pathType: {{ .pathType }}
{{- end }}
backend:
{{- if semverCompare ">=1.19-0" $.Capabilities.KubeVersion.GitVersion }}
service:
name: {{ $fullName }}
port:
number: {{ $svcPort }}
{{- else }}
serviceName: {{ $fullName }}
servicePort: {{ $svcPort }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}

12
helm-charts-observability/charts/data-prepper/templates/secret.yaml Normal file
View File

@@ -0,0 +1,12 @@
{{- if .Values.pipelineConfig.enabled }}
apiVersion: v1
kind: Secret
metadata:
name: {{ include "data-prepper.fullname" . }}-pipeline
labels:
{{- include "data-prepper.labels" . | nindent 4 }}
type: Opaque
stringData:
pipelines.yaml: |
{{- toYaml .Values.pipelineConfig.config | nindent 4 }}
{{- end }}

17
helm-charts-observability/charts/data-prepper/templates/service.yaml Normal file
View File

@@ -0,0 +1,17 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "data-prepper.fullname" . }}
labels:
{{- include "data-prepper.labels" . | nindent 4 }}
spec:
type: {{ .Values.service.type }}
ports:
{{- range .Values.ports }}
- name: {{ .name }}
port: {{ .port }}
targetPort: {{ .port }}
protocol: TCP
{{- end }}
selector:
{{- include "data-prepper.selectorLabels" . | nindent 4 }}

13
helm-charts-observability/charts/data-prepper/templates/serviceaccount.yaml Normal file
View File

@@ -0,0 +1,13 @@
{{- if .Values.serviceAccount.create -}}
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "data-prepper.serviceAccountName" . }}
labels:
{{- include "data-prepper.labels" . | nindent 4 }}
{{- with .Values.serviceAccount.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
automountServiceAccountToken: {{ .Values.serviceAccount.automount }}
{{- end }}

360
helm-charts-observability/charts/data-prepper/values.yaml Normal file
View File

@@ -0,0 +1,360 @@
# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0
# Default values for data-prepper.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
global:
# Set if you want to change the default docker registry, e.g. a private one.
dockerRegistry: ""
image:
# -- The image repository from which to pull the Data Prepper image
repository: opensearchproject/data-prepper
# -- The image tag to pull. Default: IfNotPresent
pullPolicy: IfNotPresent
# -- Overrides the image tag whose default is the chart appVersion.
tag: ""
# -- List of imagePullSecrets to use if the Docker image is stored in a private registry
imagePullSecrets: []
# -- Override the default name for the deployment
nameOverride: ""
# -- Override the default fullname for the deployment
fullnameOverride: ""
# -- Extra environment variables to pass to the Data Prepper container
extraEnvs:
# - name: "JAVA_OPTS"
# value: "-Dlog4j2.debug=true"
- name: "LOGGER_PASSWORD"
valueFrom:
secretKeyRef:
name: prepper-creds
key: logger
# Check https://opensearch.org/docs/latest/data-prepper/managing-data-prepper/configuring-data-prepper/
# for more information on the configuration options
# -- Data Prepper configuration
config:
# -- Main Data Prepper configuration file content
data-prepper-config.yaml: |
ssl: false
# circuit_breakers:
# heap:
# usage: 2gb
# reset: 30s
# check_interval: 5s
# -- Log4j2 configuration for Data Prepper logging
log4j2-rolling.properties: |
#
# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0
#
status = error
dest = err
name = PropertiesConfig
property.filename = log/data-prepper/data-prepper.log
appender.console.type = Console
appender.console.name = STDOUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{ISO8601} [%t] %-5p %40C - %m%n
appender.rolling.type = RollingFile
appender.rolling.name = RollingFile
appender.rolling.fileName = ${filename}
appender.rolling.filePattern = logs/data-prepper.log.%d{MM-dd-yy-HH}-%i.gz
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = %d{ISO8601} [%t] %-5p %40C - %m%n
appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.policies.time.interval = 1
appender.rolling.policies.time.modulate = true
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling.policies.size.size=100MB
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.max = 168
rootLogger.level = warn
rootLogger.appenderRef.stdout.ref = STDOUT
rootLogger.appenderRef.file.ref = RollingFile
logger.pipeline.name = org.opensearch.dataprepper.pipeline
logger.pipeline.level = info
logger.parser.name = org.opensearch.dataprepper.parser
logger.parser.level = info
logger.plugins.name = org.opensearch.dataprepper.plugins
logger.plugins.level = info
# For OpenSearch Data Prepper is crucial for defining the behavior and structure of your data processing pipelines.
# Each pipeline is defined with a unique name and can include `source`, `processor`, and `sink` components to ingest,
# process, and output data respectively. This flexible configuration allows for the creation of complex data processing
# flows, including the routing of data between pipelines.
# For detailed information on the available options and to get the most up-to-date guidance on configuring `pipeline.yaml`,
# please consult the [OpenSearch Documentation on Pipelines](https://opensearch.org/docs/2.4/data-prepper/pipelines/pipelines/).
# This resource provides comprehensive examples and explanations of each component, ensuring you can tailor your Data Prepper
# deployment to meet your specific data processing needs.
# -- Pipeline configuration
pipelineConfig:
# If 'true', a secret containing a demo pipeline configuration with random source and stdout sink will be created.
# If left undefined, the demo pipeline will be used only when no other pipeline is configured below
demoPipeline: ""
# -- The name of the existing secret containing the pipeline configuration.
# If enabled is false existingSecret is used. The existingSecret must have a key named `pipelines.yaml`.
existingSecret: ""
# If enabled, a secret containing the pipeline configuration will be created based on the 'config' section below.
enabled: true
# The configuration of the pipeline see https://opensearch.org/docs/2.4/data-prepper/pipelines/pipelines/
config:
## Provide your pipeline configuration here if 'enabled' is set to true. See documentation for more advanced pipelines
# simple-sample-pipeline:
# workers: 2 # the number of workers
# delay: 5000 # in milliseconds, how long workers wait between read attempt
cpanel-log-router:
workers: 2
delay: 100
source:
http:
path: /logs
ssl: false
port: 2021
buffer:
bounded_blocking:
buffer_size: 2048
batch_size: 128
route:
- systemLog : '/log_source == "system.syslog"'
- accessLog : '/log_source == "cpanel.access_log"'
- auditLog : '/log_source == "audit.audit.log"'
sink:
- pipeline:
name: cpanel-access-pipeline
routes:
- accessLog
- pipeline:
name: cpanel-system-syslog-pipeline
routes:
- systemLog
- pipeline:
name: cpanel-audit-pipeline
- pipeline:
name: cpanel-all-log
cpanel-access-pipeline:
source:
pipeline:
name: cpanel-log-router
processor:
- grok:
break_on_match: false
match:
log:
- '^%{IP:client_ip} %{NOTSPACE:identity} %{NOTSPACE:user} \[%{GREEDYDATA:timestamp}\] "%{WORD:method} %{URIPATHPARAM:url} HTTP/%{NUMBER:http_version}" %{INT:http_status:int} %{NONNEGINT:bytes:long} "%{NOTSPACE:referrer}" "%{GREEDYDATA:user_agent}" "%{NOTSPACE:auth_method}" "%{GREEDYDATA:x_forwarded_for}" %{POSINT:service_port:int}$'
target_key: parsed
- date:
match:
- key: /parsed/timestamp
patterns:
- "MM/dd/yyyy:HH:mm:ss Z"
destination: "@timestamp"
sink:
- opensearch:
hosts:
- "https://brilliant-opensearch-cluster-master:9200"
insecure: false
username: logger
password: "Enp129s0f0-l0gin"
index_type: custom
cert: /usr/share/dataprepper/cert/root-ca.pem
index: cpanel-access-log-%{yyyy.MM.dd}
cpanel-system-syslog-pipeline:
source:
pipeline:
name: cpanel-log-router
processor:
- drop_events:
drop_when: '/log_source != "system.syslog"'
- grok:
break_on_match: false
match:
log:
- '^%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:hostname} %{DATA:process}\[(?:%{POSINT:pid})\]: %{GREEDYDATA:message}$'
target_key: parsed
sink:
- opensearch:
hosts:
- "https://brilliant-opensearch-cluster-master:9200"
insecure: false
username: logger
password: "Enp129s0f0-l0gin"
index_type: custom
cert: /usr/share/dataprepper/cert/root-ca.pem
index: cpanel-system-syslog-%{yyyy.MM.dd}
cpanel-audit-pipeline:
source:
pipeline:
name: cpanel-log-router
processor:
- drop_events:
drop_when: '/log_source != "audit.audit.log"'
sink:
- opensearch:
hosts:
- "https://brilliant-opensearch-cluster-master:9200"
insecure: false
username: logger
password: "Enp129s0f0-l0gin"
index_type: custom
cert: /usr/share/dataprepper/cert/root-ca.pem
index: cpanel-audit-log-%{yyyy.MM.dd}
cpanel-all-log:
source:
pipeline:
name: cpanel-log-router
sink:
- opensearch:
hosts:
- "https://brilliant-opensearch-cluster-master:9200"
insecure: false
username: logger
password: "Enp129s0f0-l0gin"
index_type: custom
cert: /usr/share/dataprepper/cert/root-ca.pem
index: cpanel-all-log-%{yyyy.MM.dd}
# -- Data Prepper ports
ports:
# -- The port that the source is running on. Default value is 2021. Valid options are between 0 and 65535.
# https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/http-source/
- name: http-source
port: 2021
# # -- The port that the otel_trace_source source runs on. Default value is 21890.
# # https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/otel-trace-source/
# - name: otel-traces
# port: 21890
# # -- The port that the OpenTelemtry metrics source runs on. Default value is 21891.
# # https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/otel-metrics-source/
# - name: otel-metrics
# port: 21891
# # -- Represents the port that the otel_logs_source source is running on. Default value is 21892.
# # https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/otel-logs-source/
# - name: otel-logs
# port: 21892
serviceAccount:
# -- Specifies whether a service account should be created
create: true
# -- Automatically mount a ServiceAccount's API credentials?
automount: true
# -- Annotations to add to the service account
annotations: {}
# -- The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name: ""
podAnnotations: {}
podLabels: {}
podSecurityContext: {}
# fsGroup: 2000
securityContext: {}
# capabilities:
# drop:
# - ALL
# readOnlyRootFilesystem: true
# runAsNonRoot: true
# runAsUser: 1000
initContainers: []
service:
type: ClusterIP
port: 2021
ingress:
enabled: false
className: "nginx"
annotations:
kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
hosts:
- paths:
- path: /logs
pathType: Prefix
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
resources:
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
limits:
cpu: 4
memory: 6Gi
requests:
cpu: 2
memory: 4Gi
autoscaling:
# done for testing, in production this will be true
enabled: false
minReplicas: 1
maxReplicas: 100
targetCPUUtilizationPercentage: 75
targetMemoryUtilizationPercentage: 75
# Additional volumes on the output Deployment definition.
volumes:
- name: root-ca
secret:
secretName: opensearch-dashboard-ca
# - name: foo
# secret:
# secretName: mysecret
# optional: false
# Additional volumeMounts on the output Deployment definition.
volumeMounts:
- name: root-ca
readOnly: true
mountPath: /usr/share/dataprepper/cert
# - name: foo
# mountPath: "/etc/foo"
# readOnly: true
nodeSelector: {}
tolerations: []
affinity: {}