diff --git a/opensearch-cpanel-audit-mapping-explosion-index-template-sop.md b/opensearch-cpanel-audit-mapping-explosion-index-template-sop.md new file mode 100644 index 0000000..4aafac2 --- /dev/null +++ b/opensearch-cpanel-audit-mapping-explosion-index-template-sop.md @@ -0,0 +1,1647 @@ +# OpenSearch `too_many_nested_clauses` and cPanel Audit Log Mapping Explosion: Diagnosis, Index Template, and Permanent-Fix Plan + +## 1. Overview / TL;DR + +A search against the daily cPanel audit index +`cpanel-audit-log-2026.08.20` partially failed with +`too_many_nested_clauses` because the query exceeded OpenSearch's +configured `maxClauseCount` of `1024`. Inspection of the field +capabilities showed that the audit ingestion pipeline had created +hundreds of independent EXECVE argument fields such as `execve.args.a0` +through `execve.args.a309`, often with additional `.keyword` +multifields. This creates a mapping-explosion problem and makes broad +Dashboard queries capable of expanding into more Boolean clauses than +OpenSearch allows. The immediate workaround discussed was to raise +`indices.query.bool.max_clause_count` to `4096`, while the durable plan +is to create an index template for future `cpanel-audit-log-*` indexes +and modify Data Prepper so EXECVE arguments are normalized into a stable +field structure instead of dynamically creating `a0`, `a1`, `a2`, and so +on. + +The index template described here will automatically apply to future +indexes matching `cpanel-audit-log-*`; it will not retroactively change +`cpanel-audit-log-2026.08.20`. The Data Prepper normalization was +identified as required for the permanent fix but was not implemented or +verified during this conversation. + +------------------------------------------------------------------------ + +## 2. Pass 1 --- Inventory + +The following inventory captures every distinct issue, question, +command, error, workaround, observation, decision, and unresolved item +discussed in the conversation. + +1. A search returned `too_many_nested_clauses` on shard `0` of + `cpanel-audit-log-2026.08.20`. +2. The reported Boolean clause limit was + `maxClauseCount is set to 1024`. +3. The search involved 8 shards: 7 successful and 1 failed. +4. The response showed `hits.total: 0` and an empty aggregation despite + the partial shard failure. +5. The failure was identified as a query-expansion problem rather than + proof of an indexing failure. +6. A command was provided to inspect cluster settings, including + defaults. +7. A temporary workaround was proposed: increase + `indices.query.bool.max_clause_count` to `4096`. +8. A command was provided to verify the cluster setting after changing + it. +9. A warning was given not to immediately use very large values such as + `10000` or `65536` because larger query limits can increase CPU/heap + pressure. +10. Broad queries across all fields, including `fields: ["*"]` or + unspecified fields, were identified as possible clause-expansion + triggers. +11. An example `query_string` query without explicit fields was shown. +12. An improved `query_string` example restricted searches to `message`, + `user`, `source.ip`, and `event.action`. +13. Wildcard, prefix, regexp, fuzzy, and large OR-list queries were + identified as possible contributors. +14. Example risky Dashboard/query patterns included `*`, `error*`, + `user:*`, and `field:(value1 OR value2 OR value3 OR ...)`. +15. A command was provided to inspect the mapping of + `cpanel-audit-log-2026.08.20`. +16. A Linux `curl | jq | wc -l` command was provided as a quick + field-count technique. +17. A `_field_caps?fields=*` command was provided to inspect all field + capabilities. +18. The uploaded `_field_caps` output showed hundreds of + `execve.args.aN` fields. +19. Many of those EXECVE fields also had `.keyword` multifields. +20. Other fields, such as `syscall.comm`, `audit.msg`, and `login.user`, + also appeared as text plus keyword pairs. +21. The mapping problem was identified as mapping explosion, primarily + under `execve.args`. +22. The current incoming structure was described as an object with + fields such as `a0`, `a1`, `a2`, etc. +23. A normalized target structure was proposed using `execve.args` as an + array, `execve.args_raw` as text, and `execve.arg_count` as a + numeric field. +24. A sample explicit mapping was proposed for `@timestamp`, `syscall`, + and `execve`. +25. The recommendation was not to repair the existing daily index + mapping in place. +26. The intended ingestion flow was documented as + `auditd -> Fluent Bit -> Data Prepper -> normalize EXECVE arguments -> OpenSearch index template -> cpanel-audit-log-YYYY.MM.DD`. +27. The user asked how to create an index template. +28. The command `GET /_index_template` was provided to inspect existing + composable templates. +29. The command + `POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21` + was provided to determine what would apply to the next daily index. +30. A proposed `cpanel-audit-log-template` was supplied with + `index_patterns: ["cpanel-audit-log-*"]`, priority `200`, version + `1`, a field limit of `1000`, dynamic mapping, disabled date + detection, a `strings_as_keyword` dynamic template, and explicit + audit/syscall/execve fields. +31. The template was designed to map generic dynamically discovered + strings as `keyword` rather than default text plus keyword. +32. The command `GET /_index_template/cpanel-audit-log-template` was + provided to verify that template. +33. The simulation response's `overlapping` section was identified as + something to inspect for conflicts. +34. It was emphasized that an index template does not transform incoming + documents. +35. It was emphasized that if Data Prepper continues sending + `execve.args.a0`, `a1`, etc., those fields can still be dynamically + created. +36. A warning was given not to solve the problem by increasing + `index.mapping.total_fields.limit` to `5000`. +37. The existing template output showed only one template named + `cpanel_access_template`. +38. `cpanel_access_template` matches `cpanel-access-log-*`, not + `cpanel-audit-log-*`. +39. Therefore, the existing access-log template does not apply to or + conflict with the audit-log index pattern. +40. A separate final proposed audit template named + `cpanel_audit_template` was supplied. +41. The final proposed audit template used + `index_patterns: ["cpanel-audit-log-*"]`, priority `100`, version + `1`, `_meta.flow: "audit"`, `_meta.description`, + `index.mapping.total_fields.limit: 1000`, `dynamic: true`, + `date_detection: false`, and a `strings_as_keyword` dynamic + template. +42. The final audit template explicitly mapped `@timestamp`, + `log_source`, `source_file`, `log`, `audit`, `execve`, and + `syscall`. +43. The final template verification command was + `GET /_index_template/cpanel_audit_template`. +44. The final simulation command was + `POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21`. +45. The user asked whether the template would automatically apply to + future audit indexes. +46. It was confirmed that future indexes matching `cpanel-audit-log-*` + will automatically receive the template when they are created. +47. `cpanel-audit-log-2026.08.20` was explicitly identified as an + existing index that will not be retroactively changed. +48. Example future indexes `cpanel-audit-log-2026.08.21`, `.22`, and + `.23` were identified as matching the template. +49. It was clarified that the template does not need to be recreated + every day. +50. Data Prepper normalization remained the required permanent fix and + was unresolved at the end of the conversation. +51. No successful output from creating `cpanel_audit_template`, + simulating it, changing `max_clause_count`, or modifying Data + Prepper was shown in the conversation. + +------------------------------------------------------------------------ + +## 3. Prerequisites & Required Knowledge + +### Tools and access + +You need: + +- Access to **OpenSearch Dashboards → Dev Tools** with permission to + read cluster settings, inspect mappings/templates, create index + templates, and---if using the temporary workaround---change + persistent cluster settings. +- Access to the OpenSearch REST API if running the supplied `curl` + command from Linux. +- `curl`, `jq`, and `wc` for the optional Linux field-count command. +- Administrative access to the **Data Prepper** configuration for the + permanent normalization fix. The exact Data Prepper configuration + file/path was not provided in this conversation. +- Access to the audit ingestion path involving **auditd**, **Fluent + Bit**, **Data Prepper**, and **OpenSearch**. + +### Concepts + + ----------------------------------------------------------------------- + Concept Meaning in this procedure + ----------------------------------- ----------------------------------- + OpenSearch index A collection of documents with + mappings and settings. The affected + index was + `cpanel-audit-log-2026.08.20`. + + Mapping Defines how document fields are + stored and queried. Uncontrolled + dynamic fields can cause mapping + explosion. + + Dynamic mapping OpenSearch automatically creates + mappings for fields it sees for the + first time. + + Dynamic template A rule controlling how dynamically + discovered fields are mapped. + + Index template Settings and mappings automatically + applied when a new index name + matches an index pattern. + + Composable index template The `_index_template` API form used + in this conversation. + + Mapping explosion Uncontrolled growth in the number + of mapped fields. Here, EXECVE + arguments were a major source. + + Boolean clause An internal query component. The + observed query exceeded a limit of + 1024 clauses. + + EXECVE Linux audit event information + containing executable arguments. + + Multifield One logical value indexed in + multiple ways, such as a `text` + field plus `.keyword`. + + Field capabilities OpenSearch metadata showing field + types and whether fields are + searchable/aggregatable. + + Data Prepper The pipeline stage identified as + the correct place to normalize + incoming audit records before + indexing. + ----------------------------------------------------------------------- + +### Environment and assumptions actually established in the conversation + +The conversation established the following and no more: + +- OpenSearch is receiving cPanel audit logs in daily indexes named + like `cpanel-audit-log-2026.08.20`. +- A separate cPanel access-log template already exists for + `cpanel-access-log-*`. +- Audit ingestion involves auditd, Fluent Bit, Data Prepper, and + OpenSearch. +- The affected search reported a Boolean clause limit of `1024`. +- The audit mapping contains EXECVE argument fields extending at least + through `execve.args.a309`, plus `execve.args.a310[0]`. +- No OpenSearch version, operating-system version, Data Prepper + version, node count, heap size, or exact cluster topology was + stated. Do not assume those values. + +------------------------------------------------------------------------ + +## 4. Background / Why This Happens + +The original search did not fail because OpenSearch necessarily had no +audit data. It failed on one shard because the generated query contained +more Boolean clauses than the configured maximum. + +The actual error was: + +``` json +{ + "type": "too_many_nested_clauses", + "reason": "too_many_nested_clauses: Query contains too many nested clauses; maxClauseCount is set to 1024" +} +``` + +The response also showed: + +``` json +{ + "_shards": { + "total": 8, + "successful": 7, + "skipped": 0, + "failed": 1 + } +} +``` + +This is important because the same response contained: + +``` json +{ + "hits": { + "total": 0, + "max_score": null, + "hits": [] + } +} +``` + +and an empty aggregation. Because a shard failed, those empty results +must not automatically be interpreted as proof that no logs exist. + +### How the field mapping contributed + +The field-capabilities output for `cpanel-audit-log-2026.08.20` showed a +very large family of fields: + +``` text +execve.args.a0 +execve.args.a1 +execve.args.a2 +... +execve.args.a100 +... +execve.args.a200 +... +execve.args.a309 +execve.args.a310[0] +``` + +Many also had keyword multifields: + +``` text +execve.args.a100 +execve.args.a100.keyword +``` + +The same text-plus-keyword pattern appeared elsewhere, including fields +such as: + +``` text +syscall.comm +syscall.comm.keyword + +audit.msg +audit.msg.keyword + +login.user +login.user.keyword +``` + +This means a command with many Linux audit EXECVE arguments can create +many new field names. A broad Dashboard query that searches many or all +searchable fields can then expand into a very large internal Boolean +query. Once the number of generated clauses exceeds `1024`, the search +can fail with `too_many_nested_clauses`. + +### Why simply raising limits is not the permanent fix + +Raising `indices.query.bool.max_clause_count` gives queries more room, +but it does not stop the mapping from growing. Likewise, increasing +`index.mapping.total_fields.limit` only allows more fields to +accumulate. + +The durable design is to normalize command arguments before OpenSearch +receives them. Instead of documents shaped like: + +``` json +{ + "execve": { + "args": { + "a0": "curl", + "a1": "-k", + "a2": "https://example.com", + "a3": "-o", + "a4": "file" + } + } +} +``` + +the target discussed in the conversation was: + +``` json +{ + "execve": { + "command": "curl", + "args": [ + "-k", + "https://example.com", + "-o", + "file" + ], + "args_raw": "curl -k https://example.com -o file", + "arg_count": 5 + } +} +``` + +A later example showed the same principle with the executable included +in the `args` array: + +``` json +{ + "execve": { + "arg_count": 3, + "args": [ + "/usr/bin/curl", + "-k", + "https://example.com" + ], + "args_raw": "/usr/bin/curl -k https://example.com" + } +} +``` + +These two examples are slightly different in whether the executable is +represented separately as `command` or as the first element of `args`. +The conversation did not resolve which representation should be adopted. +The important invariant is that argument position must no longer become +a field name such as `a0`, `a1`, or `a309`. + +------------------------------------------------------------------------ + +## 5. Step-by-Step Procedure + +### Step 1 --- Confirm the original failure pattern + +If the search response contains the following error, treat it as a +query-clause problem: + +``` text +too_many_nested_clauses +Query contains too many nested clauses; maxClauseCount is set to 1024 +``` + +The actual response in this incident reported: + +``` json +{ + "_shards": { + "total": 8, + "successful": 7, + "skipped": 0, + "failed": 1, + "failures": [ + { + "shard": 0, + "index": "cpanel-audit-log-2026.08.20", + "node": "Wbt_J4spT3OWQg2CiDBk3g", + "reason": { + "type": "too_many_nested_clauses", + "reason": "too_many_nested_clauses: Query contains too many nested clauses; maxClauseCount is set to 1024" + } + } + ] + } +} +``` + +Why: this distinguishes a partially failed search from a simple zero-hit +result. + +### Step 2 --- Inspect the current Boolean clause setting + +In OpenSearch Dashboards Dev Tools, run: + +``` http +GET /_cluster/settings?include_defaults=true&flat_settings=true +``` + +Look for: + +``` text +indices.query.bool.max_clause_count +``` + +No output from this command was shown in the conversation. + +### Step 3 --- Inspect the affected mapping and field capabilities + +Run: + +``` http +GET /cpanel-audit-log-2026.08.20/_mapping +``` + +Then inspect all field capabilities: + +``` http +GET /cpanel-audit-log-2026.08.20/_field_caps?fields=* +``` + +The field-capabilities result in the conversation confirmed extensive +EXECVE field proliferation, including `execve.args.a0` through at least +`execve.args.a309`, plus `execve.args.a310[0]`, and many corresponding +`.keyword` fields. + +An optional Linux quick-count command provided in the conversation was: + +``` bash +curl -sk -u admin:'PASSWORD' \ +'https://OPENSEARCH:9200/cpanel-audit-log-2026.08.20/_mapping' \ +| jq '.. | objects | select(has("type")) | .type' \ +| wc -l +``` + +`PASSWORD` and `OPENSEARCH` are placeholders exactly as used in the +conversation. No result from this command was shown. + +### Step 4 --- Review Dashboard/query patterns that can amplify the problem + +Check whether the failing query searches all fields, for example: + +``` json +{ + "query": { + "query_string": { + "query": "login" + } + } +} +``` + +or explicitly uses: + +``` json +"fields": ["*"] +``` + +Also inspect Dashboard filters or queries resembling: + +``` text +* +``` + +``` text +error* +``` + +``` text +user:* +``` + +``` text +field:(value1 OR value2 OR value3 OR ...) +``` + +Where possible, constrain queries to the fields actually required. An +example from the conversation was: + +``` json +{ + "query": { + "query_string": { + "query": "login", + "fields": [ + "message", + "user", + "source.ip", + "event.action" + ] + } + } +} +``` + +This was an illustrative example. The conversation did not establish +that these exact four fields exist in the cPanel audit schema or that +this exact query was used in the failing visualization. + +### Step 5 --- Apply the temporary query-limit workaround only if needed + +The temporary workaround discussed was: + +``` http +PUT /_cluster/settings +{ + "persistent": { + "indices.query.bool.max_clause_count": 4096 + } +} +``` + +Then verify: + +``` http +GET /_cluster/settings?include_defaults=true&flat_settings=true +``` + +Why: `4096` can allow the current broad query to execute while the +schema is being corrected. + +**Operational warning:** This changes a persistent cluster setting. The +conversation warned against immediately using very large values such as +`10000` or `65536`, because allowing larger Boolean queries can increase +CPU and heap consumption. + +No successful output from setting `4096` was shown in the conversation. + +### Step 6 --- Inspect existing index templates + +Run: + +``` http +GET /_index_template +``` + +The actual result shown in the conversation contained one existing +template: + +``` json +{ + "index_templates": [ + { + "name": "cpanel_access_template", + "index_template": { + "index_patterns": [ + "cpanel-access-log-*" + ], + "template": { + "settings": {}, + "mappings": { + "properties": { + "parsed": { + "type": "object", + "properties": { + "client_ip": { + "type": "ip" + } + } + } + } + } + }, + "composed_of": [], + "priority": 0, + "_meta": { + "flow": "simple" + } + } + } + ] +} +``` + +This template applies to: + +``` text +cpanel-access-log-* +``` + +It does not match: + +``` text +cpanel-audit-log-* +``` + +Therefore, based on the templates shown in the conversation, it does not +conflict with the audit-log template. + +### Step 7 --- Simulate what would apply to the next audit index + +Before creating the audit template, the following simulation command was +proposed: + +``` http +POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21 +``` + +Why: this shows which template would apply and helps identify +overlaps/conflicts. + +No simulation output was shown. + +### Step 8 --- Create the audit index template + +The final template proposed after reviewing the existing access template +was named `cpanel_audit_template`: + +``` http +PUT /_index_template/cpanel_audit_template +{ + "index_patterns": [ + "cpanel-audit-log-*" + ], + "priority": 100, + "version": 1, + "_meta": { + "flow": "audit", + "description": "Template for cPanel Linux audit logs" + }, + "template": { + "settings": { + "index.mapping.total_fields.limit": 1000 + }, + "mappings": { + "dynamic": true, + "date_detection": false, + "dynamic_templates": [ + { + "strings_as_keyword": { + "match_mapping_type": "string", + "mapping": { + "type": "keyword", + "ignore_above": 1024 + } + } + } + ], + "properties": { + "@timestamp": { + "type": "date" + }, + "log_source": { + "type": "keyword" + }, + "source_file": { + "type": "keyword" + }, + "log": { + "type": "text" + }, + "audit": { + "properties": { + "event_id": { + "type": "keyword" + }, + "log_type": { + "type": "keyword" + }, + "msg": { + "type": "text" + }, + "timestamp_seconds": { + "type": "long" + }, + "timestamp_fraction": { + "type": "keyword" + } + } + }, + "execve": { + "properties": { + "arg_count": { + "type": "integer" + }, + "args_raw": { + "type": "text" + } + } + }, + "syscall": { + "properties": { + "pid": { + "type": "long" + }, + "ppid": { + "type": "long" + }, + "uid": { + "type": "long" + }, + "gid": { + "type": "long" + }, + "auid": { + "type": "long" + }, + "comm": { + "type": "keyword" + }, + "exe": { + "type": "keyword" + }, + "architecture": { + "type": "keyword" + }, + "syscall": { + "type": "keyword" + }, + "success": { + "type": "keyword" + }, + "exit_code": { + "type": "long" + }, + "terminal": { + "type": "keyword" + }, + "key": { + "type": "keyword" + } + } + } + } + } + } +} +``` + +Why: + +- `index_patterns` ensures future indexes named `cpanel-audit-log-*` + match automatically. +- `dynamic_templates` maps otherwise-unmapped strings as a single + `keyword` field rather than automatically creating a text field plus + `.keyword`. +- Explicit mappings are supplied for important audit/syscall fields. +- `index.mapping.total_fields.limit` remains at `1000` rather than + being raised to accommodate uncontrolled field growth. + +**Important limitation:** This template does not transform incoming +documents. With `"dynamic": true`, incoming `execve.args.a0`, +`execve.args.a1`, etc. can still become fields. Data Prepper +normalization is still required. + +### Step 9 --- Verify the audit template exists + +Run: + +``` http +GET /_index_template/cpanel_audit_template +``` + +No actual output from this command was shown in the conversation. + +### Step 10 --- Simulate the future daily index again + +Run: + +``` http +POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21 +``` + +Inspect the resolved mappings/settings and any: + +``` json +"overlapping": [...] +``` + +The expected goal is for `cpanel_audit_template` to apply to +`cpanel-audit-log-2026.08.21`. No actual simulation result was shown, so +this must be verified in the environment. + +### Step 11 --- Understand automatic application to future indexes + +Because the template matches: + +``` json +"index_patterns": [ + "cpanel-audit-log-*" +] +``` + +it is intended to apply automatically when new matching indexes are +created. + +The conversation explicitly distinguished: + +``` text +cpanel-audit-log-2026.08.20 <- existing index; NOT changed +cpanel-audit-log-2026.08.21 <- template should apply automatically +cpanel-audit-log-2026.08.22 <- template should apply automatically +cpanel-audit-log-2026.08.23 <- template should apply automatically +``` + +The template does not need to be recreated every day. + +### Step 12 --- Modify Data Prepper to normalize EXECVE arguments + +This is the permanent-fix stage. + +Current problematic shape: + +``` json +{ + "execve": { + "args": { + "a0": "/usr/bin/curl", + "a1": "-k", + "a2": "https://example.com" + } + } +} +``` + +Target shape discussed: + +``` json +{ + "execve": { + "arg_count": 3, + "args": [ + "/usr/bin/curl", + "-k", + "https://example.com" + ], + "args_raw": "/usr/bin/curl -k https://example.com" + } +} +``` + +The intended stable mapping would then include: + +``` json +"execve": { + "properties": { + "arg_count": { + "type": "integer" + }, + "args": { + "type": "keyword" + }, + "args_raw": { + "type": "text" + } + } +} +``` + +Why: whether a process has 3 arguments or 300, the OpenSearch mapping +remains centered on: + +``` text +execve.arg_count +execve.args +execve.args_raw +``` + +instead of creating hundreds of position-specific fields. + +**Unresolved:** No exact Data Prepper processor/configuration was +supplied or tested in this conversation. Do not invent one from this +document. The Data Prepper pipeline must be inspected before +implementing this transformation. + +### Step 13 --- Allow the next daily index to be created with the corrected design + +The intended end-to-end flow is: + +``` text +auditd + ↓ +Fluent Bit + ↓ +Data Prepper + ↓ +normalize EXECVE arguments + ↓ +OpenSearch index template + ↓ +cpanel-audit-log-YYYY.MM.DD +``` + +Do not assume the existing `cpanel-audit-log-2026.08.20` mapping will be +corrected by creating the template. Existing mappings are not +retroactively rewritten by the new template. + +------------------------------------------------------------------------ + +## 6. Verification + +### Verify cluster clause limit + +``` http +GET /_cluster/settings?include_defaults=true&flat_settings=true +``` + +Confirm the intended value for: + +``` text +indices.query.bool.max_clause_count +``` + +If the temporary workaround was applied, confirm it resolves to `4096`. + +### Verify the template + +``` http +GET /_index_template/cpanel_audit_template +``` + +Confirm that its pattern is: + +``` json +"index_patterns": [ + "cpanel-audit-log-*" +] +``` + +### Verify template resolution before index creation + +``` http +POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21 +``` + +Confirm the intended audit template is selected and review `overlapping` +for conflicts. + +### Verify a future index after it is created + +The conversation did not provide a post-creation verification +command/output beyond the template APIs and mapping/field-capabilities +APIs already used. Reuse: + +``` http +GET /cpanel-audit-log-2026.08.21/_mapping +``` + +and: + +``` http +GET /cpanel-audit-log-2026.08.21/_field_caps?fields=* +``` + +The critical design check is that the permanent Data Prepper fix +eventually prevents continued creation of fields such as: + +``` text +execve.args.a0 +execve.args.a1 +... +execve.args.a309 +``` + +No successful future-index verification was shown in the conversation. + +### Verify the search itself + +Retry the Dashboard/search that originally produced: + +``` text +too_many_nested_clauses +``` + +Success means the search no longer reports a failed shard for this +reason. No successful retry output was shown in the conversation. + +------------------------------------------------------------------------ + +## 7. Troubleshooting Checklist + +- [ ] Confirm the error is `too_many_nested_clauses` and note the + reported `maxClauseCount`. +- [ ] Check whether the response has failed shards rather than + treating `hits.total: 0` as definitive. +- [ ] Run + `GET /_cluster/settings?include_defaults=true&flat_settings=true`. +- [ ] Inspect `GET /cpanel-audit-log-2026.08.20/_mapping`. +- [ ] Inspect `GET /cpanel-audit-log-2026.08.20/_field_caps?fields=*`. +- [ ] Look for large field families such as `execve.args.a0`, `a1`, + `a2`, ... . +- [ ] Check whether those fields also have `.keyword` multifields. +- [ ] Check Dashboard queries for `fields: ["*"]`, unspecified fields, + `*`, wildcard/prefix/regexp/fuzzy searches, or large OR lists. +- [ ] Restrict broad `query_string` searches to required fields where + possible. +- [ ] If an immediate workaround is required, consider the discussed + `4096` clause limit rather than jumping to extremely large values. +- [ ] Run `GET /_index_template` and identify every template matching + the audit index pattern. +- [ ] Confirm `cpanel_access_template` only matches + `cpanel-access-log-*`. +- [ ] Create/verify `cpanel_audit_template` for `cpanel-audit-log-*`. +- [ ] Simulate `cpanel-audit-log-2026.08.21` and inspect overlaps. +- [ ] Do not expect the new template to rewrite + `cpanel-audit-log-2026.08.20`. +- [ ] Do not raise `index.mapping.total_fields.limit` merely to allow + the exploding schema to grow. +- [ ] Inspect the Data Prepper pipeline and normalize EXECVE arguments + before indexing. +- [ ] Verify future mappings no longer proliferate `execve.args.aN` + fields. +- [ ] Retry the original search and confirm all shards complete + successfully. + +------------------------------------------------------------------------ + +## 8. Dead Ends / What Didn't Work + +### Treating the empty hit count as proof there are no logs + +The search showed zero hits and an empty aggregation, but it also had +one failed shard. Therefore, the empty result was not a reliable +indication that the audit index contained no relevant logs. + +### Raising only `max_clause_count` + +The proposed change: + +``` http +PUT /_cluster/settings +{ + "persistent": { + "indices.query.bool.max_clause_count": 4096 + } +} +``` + +was explicitly treated as a workaround, not the root fix. It does not +stop field proliferation. + +### Raising the total field limit + +The conversation specifically warned against solving the issue with: + +``` json +"index.mapping.total_fields.limit": 5000 +``` + +That would mainly allow the bad schema to grow further. + +### Creating only an index template + +The index template improves mappings for future indexes, but it does not +rewrite incoming documents. If Data Prepper continues emitting: + +``` text +execve.args.a0 +execve.args.a1 +execve.args.a2 +... +``` + +OpenSearch can still dynamically create those fields because the +proposed template keeps `"dynamic": true`. + +### Trying to repair the current mapping in place + +The recommendation was not to attempt to turn the already-existing `a0`, +`a1`, etc. fields into a single array field in the current index. The +clean path is to correct the ingestion schema/template for new indexes. + +### No implemented Data Prepper transformation yet + +The permanent transformation was identified, but no exact Data Prepper +configuration was produced or tested. This remains unfinished rather +than a failed command. + +------------------------------------------------------------------------ + +## 9. Common Pitfalls / Gotchas + +1. **Templates are not retroactive.** `cpanel-audit-log-2026.08.20` + will not be remapped by creating `cpanel_audit_template`. +2. **Templates do not transform documents.** They control + settings/mappings. Data Prepper must normalize the incoming EXECVE + structure. +3. **`dynamic: true` still permits new fields.** The dynamic string + template reduces text-plus-keyword duplication, but it does not + prevent `a0`, `a1`, etc. from becoming distinct fields. +4. **Do not confuse access and audit patterns.** + `cpanel_access_template` matches `cpanel-access-log-*`; the audit + template matches `cpanel-audit-log-*`. +5. **Do not recreate the audit template daily.** Once present, its + wildcard pattern is intended to match future daily audit indexes + automatically. +6. **Do not assume a simulation succeeded.** No simulation output was + provided in the conversation. +7. **Do not assume the template was created successfully.** No + `PUT /_index_template/cpanel_audit_template` response was shown. +8. **Do not assume `4096` was applied.** The conversation supplied the + command but did not show its result. +9. **Avoid arbitrarily huge clause limits.** Values such as `10000` or + `65536` were specifically discouraged as an immediate response. +10. **Avoid increasing the field limit to hide mapping explosion.** The + discussed template intentionally kept + `index.mapping.total_fields.limit` at `1000`. +11. **The normalized EXECVE representation needs a final schema + decision.** One example used a separate `command` field; another + included the executable in `args`. The conversation did not resolve + that difference. +12. **The illustrative restricted query fields were not verified against + this audit schema.** `message`, `user`, `source.ip`, and + `event.action` were examples of explicit field targeting, not + confirmed audit-index field names. + +------------------------------------------------------------------------ + +## 10. Root Cause Summary + +The identified root cause was uncontrolled dynamic field creation in the +cPanel Linux audit-log schema, especially the representation of EXECVE +command arguments as position-specific object keys such as +`execve.args.a0`, `execve.args.a1`, and continuing through hundreds of +possible argument positions. Many dynamically mapped strings also +received `.keyword` multifields, further increasing the field count. +Broad OpenSearch Dashboard/query-string searches can expand across this +large set of searchable fields and generate more than the configured +`1024` Boolean clauses, causing `too_many_nested_clauses`. The permanent +fix is to normalize EXECVE arguments in Data Prepper into a stable +array/raw-command/count structure before indexing and use a controlled +index template for future `cpanel-audit-log-*` indexes. + +------------------------------------------------------------------------ + +## 11. Appendix + +### A. Full reference commands + +``` http +GET /_cluster/settings?include_defaults=true&flat_settings=true +``` + +``` http +PUT /_cluster/settings +{ + "persistent": { + "indices.query.bool.max_clause_count": 4096 + } +} +``` + +``` http +GET /_cluster/settings?include_defaults=true&flat_settings=true +``` + +``` http +GET /cpanel-audit-log-2026.08.20/_mapping +``` + +``` http +GET /cpanel-audit-log-2026.08.20/_field_caps?fields=* +``` + +``` bash +curl -sk -u admin:'PASSWORD' \ +'https://OPENSEARCH:9200/cpanel-audit-log-2026.08.20/_mapping' \ +| jq '.. | objects | select(has("type")) | .type' \ +| wc -l +``` + +``` http +GET /_index_template +``` + +``` http +POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21 +``` + +``` http +PUT /_index_template/cpanel_audit_template +{ + "index_patterns": [ + "cpanel-audit-log-*" + ], + "priority": 100, + "version": 1, + "_meta": { + "flow": "audit", + "description": "Template for cPanel Linux audit logs" + }, + "template": { + "settings": { + "index.mapping.total_fields.limit": 1000 + }, + "mappings": { + "dynamic": true, + "date_detection": false, + "dynamic_templates": [ + { + "strings_as_keyword": { + "match_mapping_type": "string", + "mapping": { + "type": "keyword", + "ignore_above": 1024 + } + } + } + ], + "properties": { + "@timestamp": { + "type": "date" + }, + "log_source": { + "type": "keyword" + }, + "source_file": { + "type": "keyword" + }, + "log": { + "type": "text" + }, + "audit": { + "properties": { + "event_id": { + "type": "keyword" + }, + "log_type": { + "type": "keyword" + }, + "msg": { + "type": "text" + }, + "timestamp_seconds": { + "type": "long" + }, + "timestamp_fraction": { + "type": "keyword" + } + } + }, + "execve": { + "properties": { + "arg_count": { + "type": "integer" + }, + "args_raw": { + "type": "text" + } + } + }, + "syscall": { + "properties": { + "pid": { + "type": "long" + }, + "ppid": { + "type": "long" + }, + "uid": { + "type": "long" + }, + "gid": { + "type": "long" + }, + "auid": { + "type": "long" + }, + "comm": { + "type": "keyword" + }, + "exe": { + "type": "keyword" + }, + "architecture": { + "type": "keyword" + }, + "syscall": { + "type": "keyword" + }, + "success": { + "type": "keyword" + }, + "exit_code": { + "type": "long" + }, + "terminal": { + "type": "keyword" + }, + "key": { + "type": "keyword" + } + } + } + } + } + } +} +``` + +``` http +GET /_index_template/cpanel_audit_template +``` + +``` http +POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21 +``` + +``` http +GET /cpanel-audit-log-2026.08.21/_mapping +``` + +``` http +GET /cpanel-audit-log-2026.08.21/_field_caps?fields=* +``` + +### B. Earlier proposed template variant + +Before the existing access template was shown, an earlier template was +proposed under the name `cpanel-audit-log-template` with priority `200`. +This was later superseded in the conversation by `cpanel_audit_template` +with priority `100`. + +The earlier creation endpoint was: + +``` http +PUT /_index_template/cpanel-audit-log-template +``` + +and its verification endpoint was: + +``` http +GET /_index_template/cpanel-audit-log-template +``` + +This difference is retained here because it appeared in the +conversation. For the reconstructed procedure, use the later +`cpanel_audit_template` unless your environment already contains one of +these names and you deliberately choose otherwise. + +### C. Existing access-log template observed + +``` json +{ + "index_templates": [ + { + "name": "cpanel_access_template", + "index_template": { + "index_patterns": [ + "cpanel-access-log-*" + ], + "template": { + "settings": {}, + "mappings": { + "properties": { + "parsed": { + "type": "object", + "properties": { + "client_ip": { + "type": "ip" + } + } + } + } + } + }, + "composed_of": [], + "priority": 0, + "_meta": { + "flow": "simple" + } + } + } + ] +} +``` + +### D. Query examples referenced + +Broad query: + +``` json +{ + "query": { + "query_string": { + "query": "login" + } + } +} +``` + +Explicit-field example: + +``` json +{ + "query": { + "query_string": { + "query": "login", + "fields": [ + "message", + "user", + "source.ip", + "event.action" + ] + } + } +} +``` + +Potentially problematic query/filter patterns: + +``` text +* +error* +user:* +field:(value1 OR value2 OR value3 OR ...) +``` + +### E. EXECVE structures referenced + +Problematic current concept: + +``` json +{ + "execve": { + "args": { + "a0": "curl", + "a1": "-k", + "a2": "https://example.com", + "a3": "-o", + "a4": "file" + } + } +} +``` + +Normalized example with separate command: + +``` json +{ + "execve": { + "command": "curl", + "args": [ + "-k", + "https://example.com", + "-o", + "file" + ], + "args_raw": "curl -k https://example.com -o file", + "arg_count": 5 + } +} +``` + +Normalized example with executable included in `args`: + +``` json +{ + "execve": { + "arg_count": 3, + "args": [ + "/usr/bin/curl", + "-k", + "https://example.com" + ], + "args_raw": "/usr/bin/curl -k https://example.com" + } +} +``` + +Stable mapping concept: + +``` json +"execve": { + "properties": { + "arg_count": { + "type": "integer" + }, + "args": { + "type": "keyword" + }, + "args_raw": { + "type": "text" + } + } +} +``` + +### F. Referenced index names and patterns + +``` text +cpanel-audit-log-2026.08.20 +cpanel-audit-log-2026.08.21 +cpanel-audit-log-2026.08.22 +cpanel-audit-log-2026.08.23 +cpanel-audit-log-* +cpanel-access-log-* +cpanel-audit-log-YYYY.MM.DD +``` + +### G. Ingestion flow + +``` text +auditd + ↓ +Fluent Bit + ↓ +Data Prepper + ↓ +normalize EXECVE arguments + ↓ +OpenSearch index template + ↓ +cpanel-audit-log-YYYY.MM.DD +``` + +### H. Glossary + +- **auditd** --- Linux Audit daemon, the source of the audit events + discussed. +- **Boolean clause** --- An internal query component used by + Lucene/OpenSearch Boolean queries. +- **cPanel** --- The server/control-panel environment whose access and + audit logs are being ingested. +- **Data Prepper** --- The processing stage identified for normalizing + audit event fields before OpenSearch indexing. +- **Dev Tools** --- OpenSearch Dashboards interface used to issue REST + API requests. +- **Dynamic mapping** --- Automatic field mapping based on incoming + document values. +- **Dynamic template** --- Rule controlling how dynamically discovered + fields are mapped. +- **EXECVE** --- Linux audit record information describing program + execution and arguments. +- **Field capabilities (`_field_caps`)** --- API showing field types + and search/aggregation capabilities. +- **Fluent Bit** --- Log forwarder in the stated ingestion flow. +- **Index pattern** --- Wildcard expression used by an index template + to determine which new index names it applies to. +- **Index template** --- Reusable mappings/settings automatically + applied when a matching index is created. +- **Mapping explosion** --- Excessive growth in mapped fields, usually + caused by uncontrolled dynamic keys. +- **Multifield** --- A field indexed in multiple forms, such as `text` + and `.keyword`. +- **OpenSearch** --- Search/analytics datastore receiving the cPanel + logs. +- **Shard** --- Partition of an OpenSearch index. The incident had 8 + searched shards, with 1 failure. +- **`maxClauseCount`** --- Maximum number of Boolean query clauses; + the observed value was `1024`. + +------------------------------------------------------------------------ + +## 12. Coverage Check + +Every Pass 1 inventory item is accounted for below. + +1. Included --- original `too_many_nested_clauses` shard failure. +2. Included --- `maxClauseCount` of `1024`. +3. Included --- 8 total / 7 successful / 1 failed shard. +4. Included --- zero hits and empty aggregation. +5. Included --- query-expansion diagnosis versus indexing failure. +6. Included --- cluster-settings inspection command. +7. Included --- temporary `4096` workaround. +8. Included --- post-change verification command. +9. Included --- warning against `10000`/`65536`. +10. Included --- all-field query expansion. +11. Included --- broad `query_string` example. +12. Included --- explicit-field `query_string` example. +13. Included --- wildcard/prefix/regexp/fuzzy/OR-list risk. +14. Included --- `*`, `error*`, `user:*`, and OR-list examples. +15. Included --- mapping inspection command. +16. Included --- Linux `curl | jq | wc -l` field-count command. +17. Included --- `_field_caps?fields=*`. +18. Included --- `execve.args.aN` proliferation. +19. Included --- `.keyword` EXECVE multifields. +20. Included --- other text/keyword pairs. +21. Included --- mapping-explosion diagnosis. +22. Included --- current `a0/a1/a2` object shape. +23. Included --- normalized args/args_raw/arg_count target. +24. Included --- explicit mapping concept. +25. Included --- recommendation not to repair current mapping in place. +26. Included --- auditd/Fluent Bit/Data Prepper/OpenSearch flow. +27. Included --- index-template creation question. +28. Included --- `GET /_index_template`. +29. Included --- simulation command. +30. Included --- earlier `cpanel-audit-log-template` proposal. +31. Included --- dynamic strings-as-keyword rationale. +32. Included --- earlier template verification command. +33. Included --- `overlapping` conflict check. +34. Included --- template does not transform documents. +35. Included --- Data Prepper can continue creating `aN` fields. +36. Included --- warning against field limit `5000`. +37. Included --- observed `cpanel_access_template`. +38. Included --- access template pattern differs from audit pattern. +39. Included --- no demonstrated access/audit template conflict. +40. Included --- final `cpanel_audit_template`. +41. Included --- final template priority/version/meta/settings/dynamic + configuration. +42. Included --- explicit mapped field groups. +43. Included --- final template verification command. +44. Included --- final simulation command. +45. Included --- automatic-future-application question. +46. Included --- automatic application to matching new indexes. +47. Included --- existing `.2026.08.20` not retroactively changed. +48. Included --- `.21`, `.22`, `.23` future examples. +49. Included --- template does not need daily recreation. +50. Included --- Data Prepper normalization remains unresolved permanent + fix. +51. Included --- explicit statement that no successful outputs were + shown for template creation, simulation, clause-limit change, or + Data Prepper modification.