# OpenSearch `too_many_nested_clauses` and cPanel Audit Log Mapping Explosion: Diagnosis, Index Template, and Permanent-Fix Plan ## 1. Overview / TL;DR A search against the daily cPanel audit index `cpanel-audit-log-2026.08.20` partially failed with `too_many_nested_clauses` because the query exceeded OpenSearch's configured `maxClauseCount` of `1024`. Inspection of the field capabilities showed that the audit ingestion pipeline had created hundreds of independent EXECVE argument fields such as `execve.args.a0` through `execve.args.a309`, often with additional `.keyword` multifields. This creates a mapping-explosion problem and makes broad Dashboard queries capable of expanding into more Boolean clauses than OpenSearch allows. The immediate workaround discussed was to raise `indices.query.bool.max_clause_count` to `4096`, while the durable plan is to create an index template for future `cpanel-audit-log-*` indexes and modify Data Prepper so EXECVE arguments are normalized into a stable field structure instead of dynamically creating `a0`, `a1`, `a2`, and so on. The index template described here will automatically apply to future indexes matching `cpanel-audit-log-*`; it will not retroactively change `cpanel-audit-log-2026.08.20`. The Data Prepper normalization was identified as required for the permanent fix but was not implemented or verified during this conversation. ------------------------------------------------------------------------ ## 2. Pass 1 --- Inventory The following inventory captures every distinct issue, question, command, error, workaround, observation, decision, and unresolved item discussed in the conversation. 1. A search returned `too_many_nested_clauses` on shard `0` of `cpanel-audit-log-2026.08.20`. 2. The reported Boolean clause limit was `maxClauseCount is set to 1024`. 3. The search involved 8 shards: 7 successful and 1 failed. 4. The response showed `hits.total: 0` and an empty aggregation despite the partial shard failure. 5. The failure was identified as a query-expansion problem rather than proof of an indexing failure. 6. A command was provided to inspect cluster settings, including defaults. 7. A temporary workaround was proposed: increase `indices.query.bool.max_clause_count` to `4096`. 8. A command was provided to verify the cluster setting after changing it. 9. A warning was given not to immediately use very large values such as `10000` or `65536` because larger query limits can increase CPU/heap pressure. 10. Broad queries across all fields, including `fields: ["*"]` or unspecified fields, were identified as possible clause-expansion triggers. 11. An example `query_string` query without explicit fields was shown. 12. An improved `query_string` example restricted searches to `message`, `user`, `source.ip`, and `event.action`. 13. Wildcard, prefix, regexp, fuzzy, and large OR-list queries were identified as possible contributors. 14. Example risky Dashboard/query patterns included `*`, `error*`, `user:*`, and `field:(value1 OR value2 OR value3 OR ...)`. 15. A command was provided to inspect the mapping of `cpanel-audit-log-2026.08.20`. 16. A Linux `curl | jq | wc -l` command was provided as a quick field-count technique. 17. A `_field_caps?fields=*` command was provided to inspect all field capabilities. 18. The uploaded `_field_caps` output showed hundreds of `execve.args.aN` fields. 19. Many of those EXECVE fields also had `.keyword` multifields. 20. Other fields, such as `syscall.comm`, `audit.msg`, and `login.user`, also appeared as text plus keyword pairs. 21. The mapping problem was identified as mapping explosion, primarily under `execve.args`. 22. The current incoming structure was described as an object with fields such as `a0`, `a1`, `a2`, etc. 23. A normalized target structure was proposed using `execve.args` as an array, `execve.args_raw` as text, and `execve.arg_count` as a numeric field. 24. A sample explicit mapping was proposed for `@timestamp`, `syscall`, and `execve`. 25. The recommendation was not to repair the existing daily index mapping in place. 26. The intended ingestion flow was documented as `auditd -> Fluent Bit -> Data Prepper -> normalize EXECVE arguments -> OpenSearch index template -> cpanel-audit-log-YYYY.MM.DD`. 27. The user asked how to create an index template. 28. The command `GET /_index_template` was provided to inspect existing composable templates. 29. The command `POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21` was provided to determine what would apply to the next daily index. 30. A proposed `cpanel-audit-log-template` was supplied with `index_patterns: ["cpanel-audit-log-*"]`, priority `200`, version `1`, a field limit of `1000`, dynamic mapping, disabled date detection, a `strings_as_keyword` dynamic template, and explicit audit/syscall/execve fields. 31. The template was designed to map generic dynamically discovered strings as `keyword` rather than default text plus keyword. 32. The command `GET /_index_template/cpanel-audit-log-template` was provided to verify that template. 33. The simulation response's `overlapping` section was identified as something to inspect for conflicts. 34. It was emphasized that an index template does not transform incoming documents. 35. It was emphasized that if Data Prepper continues sending `execve.args.a0`, `a1`, etc., those fields can still be dynamically created. 36. A warning was given not to solve the problem by increasing `index.mapping.total_fields.limit` to `5000`. 37. The existing template output showed only one template named `cpanel_access_template`. 38. `cpanel_access_template` matches `cpanel-access-log-*`, not `cpanel-audit-log-*`. 39. Therefore, the existing access-log template does not apply to or conflict with the audit-log index pattern. 40. A separate final proposed audit template named `cpanel_audit_template` was supplied. 41. The final proposed audit template used `index_patterns: ["cpanel-audit-log-*"]`, priority `100`, version `1`, `_meta.flow: "audit"`, `_meta.description`, `index.mapping.total_fields.limit: 1000`, `dynamic: true`, `date_detection: false`, and a `strings_as_keyword` dynamic template. 42. The final audit template explicitly mapped `@timestamp`, `log_source`, `source_file`, `log`, `audit`, `execve`, and `syscall`. 43. The final template verification command was `GET /_index_template/cpanel_audit_template`. 44. The final simulation command was `POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21`. 45. The user asked whether the template would automatically apply to future audit indexes. 46. It was confirmed that future indexes matching `cpanel-audit-log-*` will automatically receive the template when they are created. 47. `cpanel-audit-log-2026.08.20` was explicitly identified as an existing index that will not be retroactively changed. 48. Example future indexes `cpanel-audit-log-2026.08.21`, `.22`, and `.23` were identified as matching the template. 49. It was clarified that the template does not need to be recreated every day. 50. Data Prepper normalization remained the required permanent fix and was unresolved at the end of the conversation. 51. No successful output from creating `cpanel_audit_template`, simulating it, changing `max_clause_count`, or modifying Data Prepper was shown in the conversation. ------------------------------------------------------------------------ ## 3. Prerequisites & Required Knowledge ### Tools and access You need: - Access to **OpenSearch Dashboards → Dev Tools** with permission to read cluster settings, inspect mappings/templates, create index templates, and---if using the temporary workaround---change persistent cluster settings. - Access to the OpenSearch REST API if running the supplied `curl` command from Linux. - `curl`, `jq`, and `wc` for the optional Linux field-count command. - Administrative access to the **Data Prepper** configuration for the permanent normalization fix. The exact Data Prepper configuration file/path was not provided in this conversation. - Access to the audit ingestion path involving **auditd**, **Fluent Bit**, **Data Prepper**, and **OpenSearch**. ### Concepts ----------------------------------------------------------------------- Concept Meaning in this procedure ----------------------------------- ----------------------------------- OpenSearch index A collection of documents with mappings and settings. The affected index was `cpanel-audit-log-2026.08.20`. Mapping Defines how document fields are stored and queried. Uncontrolled dynamic fields can cause mapping explosion. Dynamic mapping OpenSearch automatically creates mappings for fields it sees for the first time. Dynamic template A rule controlling how dynamically discovered fields are mapped. Index template Settings and mappings automatically applied when a new index name matches an index pattern. Composable index template The `_index_template` API form used in this conversation. Mapping explosion Uncontrolled growth in the number of mapped fields. Here, EXECVE arguments were a major source. Boolean clause An internal query component. The observed query exceeded a limit of 1024 clauses. EXECVE Linux audit event information containing executable arguments. Multifield One logical value indexed in multiple ways, such as a `text` field plus `.keyword`. Field capabilities OpenSearch metadata showing field types and whether fields are searchable/aggregatable. Data Prepper The pipeline stage identified as the correct place to normalize incoming audit records before indexing. ----------------------------------------------------------------------- ### Environment and assumptions actually established in the conversation The conversation established the following and no more: - OpenSearch is receiving cPanel audit logs in daily indexes named like `cpanel-audit-log-2026.08.20`. - A separate cPanel access-log template already exists for `cpanel-access-log-*`. - Audit ingestion involves auditd, Fluent Bit, Data Prepper, and OpenSearch. - The affected search reported a Boolean clause limit of `1024`. - The audit mapping contains EXECVE argument fields extending at least through `execve.args.a309`, plus `execve.args.a310[0]`. - No OpenSearch version, operating-system version, Data Prepper version, node count, heap size, or exact cluster topology was stated. Do not assume those values. ------------------------------------------------------------------------ ## 4. Background / Why This Happens The original search did not fail because OpenSearch necessarily had no audit data. It failed on one shard because the generated query contained more Boolean clauses than the configured maximum. The actual error was: ``` json { "type": "too_many_nested_clauses", "reason": "too_many_nested_clauses: Query contains too many nested clauses; maxClauseCount is set to 1024" } ``` The response also showed: ``` json { "_shards": { "total": 8, "successful": 7, "skipped": 0, "failed": 1 } } ``` This is important because the same response contained: ``` json { "hits": { "total": 0, "max_score": null, "hits": [] } } ``` and an empty aggregation. Because a shard failed, those empty results must not automatically be interpreted as proof that no logs exist. ### How the field mapping contributed The field-capabilities output for `cpanel-audit-log-2026.08.20` showed a very large family of fields: ``` text execve.args.a0 execve.args.a1 execve.args.a2 ... execve.args.a100 ... execve.args.a200 ... execve.args.a309 execve.args.a310[0] ``` Many also had keyword multifields: ``` text execve.args.a100 execve.args.a100.keyword ``` The same text-plus-keyword pattern appeared elsewhere, including fields such as: ``` text syscall.comm syscall.comm.keyword audit.msg audit.msg.keyword login.user login.user.keyword ``` This means a command with many Linux audit EXECVE arguments can create many new field names. A broad Dashboard query that searches many or all searchable fields can then expand into a very large internal Boolean query. Once the number of generated clauses exceeds `1024`, the search can fail with `too_many_nested_clauses`. ### Why simply raising limits is not the permanent fix Raising `indices.query.bool.max_clause_count` gives queries more room, but it does not stop the mapping from growing. Likewise, increasing `index.mapping.total_fields.limit` only allows more fields to accumulate. The durable design is to normalize command arguments before OpenSearch receives them. Instead of documents shaped like: ``` json { "execve": { "args": { "a0": "curl", "a1": "-k", "a2": "https://example.com", "a3": "-o", "a4": "file" } } } ``` the target discussed in the conversation was: ``` json { "execve": { "command": "curl", "args": [ "-k", "https://example.com", "-o", "file" ], "args_raw": "curl -k https://example.com -o file", "arg_count": 5 } } ``` A later example showed the same principle with the executable included in the `args` array: ``` json { "execve": { "arg_count": 3, "args": [ "/usr/bin/curl", "-k", "https://example.com" ], "args_raw": "/usr/bin/curl -k https://example.com" } } ``` These two examples are slightly different in whether the executable is represented separately as `command` or as the first element of `args`. The conversation did not resolve which representation should be adopted. The important invariant is that argument position must no longer become a field name such as `a0`, `a1`, or `a309`. ------------------------------------------------------------------------ ## 5. Step-by-Step Procedure ### Step 1 --- Confirm the original failure pattern If the search response contains the following error, treat it as a query-clause problem: ``` text too_many_nested_clauses Query contains too many nested clauses; maxClauseCount is set to 1024 ``` The actual response in this incident reported: ``` json { "_shards": { "total": 8, "successful": 7, "skipped": 0, "failed": 1, "failures": [ { "shard": 0, "index": "cpanel-audit-log-2026.08.20", "node": "Wbt_J4spT3OWQg2CiDBk3g", "reason": { "type": "too_many_nested_clauses", "reason": "too_many_nested_clauses: Query contains too many nested clauses; maxClauseCount is set to 1024" } } ] } } ``` Why: this distinguishes a partially failed search from a simple zero-hit result. ### Step 2 --- Inspect the current Boolean clause setting In OpenSearch Dashboards Dev Tools, run: ``` http GET /_cluster/settings?include_defaults=true&flat_settings=true ``` Look for: ``` text indices.query.bool.max_clause_count ``` No output from this command was shown in the conversation. ### Step 3 --- Inspect the affected mapping and field capabilities Run: ``` http GET /cpanel-audit-log-2026.08.20/_mapping ``` Then inspect all field capabilities: ``` http GET /cpanel-audit-log-2026.08.20/_field_caps?fields=* ``` The field-capabilities result in the conversation confirmed extensive EXECVE field proliferation, including `execve.args.a0` through at least `execve.args.a309`, plus `execve.args.a310[0]`, and many corresponding `.keyword` fields. An optional Linux quick-count command provided in the conversation was: ``` bash curl -sk -u admin:'PASSWORD' \ 'https://OPENSEARCH:9200/cpanel-audit-log-2026.08.20/_mapping' \ | jq '.. | objects | select(has("type")) | .type' \ | wc -l ``` `PASSWORD` and `OPENSEARCH` are placeholders exactly as used in the conversation. No result from this command was shown. ### Step 4 --- Review Dashboard/query patterns that can amplify the problem Check whether the failing query searches all fields, for example: ``` json { "query": { "query_string": { "query": "login" } } } ``` or explicitly uses: ``` json "fields": ["*"] ``` Also inspect Dashboard filters or queries resembling: ``` text * ``` ``` text error* ``` ``` text user:* ``` ``` text field:(value1 OR value2 OR value3 OR ...) ``` Where possible, constrain queries to the fields actually required. An example from the conversation was: ``` json { "query": { "query_string": { "query": "login", "fields": [ "message", "user", "source.ip", "event.action" ] } } } ``` This was an illustrative example. The conversation did not establish that these exact four fields exist in the cPanel audit schema or that this exact query was used in the failing visualization. ### Step 5 --- Apply the temporary query-limit workaround only if needed The temporary workaround discussed was: ``` http PUT /_cluster/settings { "persistent": { "indices.query.bool.max_clause_count": 4096 } } ``` Then verify: ``` http GET /_cluster/settings?include_defaults=true&flat_settings=true ``` Why: `4096` can allow the current broad query to execute while the schema is being corrected. **Operational warning:** This changes a persistent cluster setting. The conversation warned against immediately using very large values such as `10000` or `65536`, because allowing larger Boolean queries can increase CPU and heap consumption. No successful output from setting `4096` was shown in the conversation. ### Step 6 --- Inspect existing index templates Run: ``` http GET /_index_template ``` The actual result shown in the conversation contained one existing template: ``` json { "index_templates": [ { "name": "cpanel_access_template", "index_template": { "index_patterns": [ "cpanel-access-log-*" ], "template": { "settings": {}, "mappings": { "properties": { "parsed": { "type": "object", "properties": { "client_ip": { "type": "ip" } } } } } }, "composed_of": [], "priority": 0, "_meta": { "flow": "simple" } } } ] } ``` This template applies to: ``` text cpanel-access-log-* ``` It does not match: ``` text cpanel-audit-log-* ``` Therefore, based on the templates shown in the conversation, it does not conflict with the audit-log template. ### Step 7 --- Simulate what would apply to the next audit index Before creating the audit template, the following simulation command was proposed: ``` http POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21 ``` Why: this shows which template would apply and helps identify overlaps/conflicts. No simulation output was shown. ### Step 8 --- Create the audit index template The final template proposed after reviewing the existing access template was named `cpanel_audit_template`: ``` http PUT /_index_template/cpanel_audit_template { "index_patterns": [ "cpanel-audit-log-*" ], "priority": 100, "version": 1, "_meta": { "flow": "audit", "description": "Template for cPanel Linux audit logs" }, "template": { "settings": { "index.mapping.total_fields.limit": 1000 }, "mappings": { "dynamic": true, "date_detection": false, "dynamic_templates": [ { "strings_as_keyword": { "match_mapping_type": "string", "mapping": { "type": "keyword", "ignore_above": 1024 } } } ], "properties": { "@timestamp": { "type": "date" }, "log_source": { "type": "keyword" }, "source_file": { "type": "keyword" }, "log": { "type": "text" }, "audit": { "properties": { "event_id": { "type": "keyword" }, "log_type": { "type": "keyword" }, "msg": { "type": "text" }, "timestamp_seconds": { "type": "long" }, "timestamp_fraction": { "type": "keyword" } } }, "execve": { "properties": { "arg_count": { "type": "integer" }, "args_raw": { "type": "text" } } }, "syscall": { "properties": { "pid": { "type": "long" }, "ppid": { "type": "long" }, "uid": { "type": "long" }, "gid": { "type": "long" }, "auid": { "type": "long" }, "comm": { "type": "keyword" }, "exe": { "type": "keyword" }, "architecture": { "type": "keyword" }, "syscall": { "type": "keyword" }, "success": { "type": "keyword" }, "exit_code": { "type": "long" }, "terminal": { "type": "keyword" }, "key": { "type": "keyword" } } } } } } } ``` Why: - `index_patterns` ensures future indexes named `cpanel-audit-log-*` match automatically. - `dynamic_templates` maps otherwise-unmapped strings as a single `keyword` field rather than automatically creating a text field plus `.keyword`. - Explicit mappings are supplied for important audit/syscall fields. - `index.mapping.total_fields.limit` remains at `1000` rather than being raised to accommodate uncontrolled field growth. **Important limitation:** This template does not transform incoming documents. With `"dynamic": true`, incoming `execve.args.a0`, `execve.args.a1`, etc. can still become fields. Data Prepper normalization is still required. ### Step 9 --- Verify the audit template exists Run: ``` http GET /_index_template/cpanel_audit_template ``` No actual output from this command was shown in the conversation. ### Step 10 --- Simulate the future daily index again Run: ``` http POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21 ``` Inspect the resolved mappings/settings and any: ``` json "overlapping": [...] ``` The expected goal is for `cpanel_audit_template` to apply to `cpanel-audit-log-2026.08.21`. No actual simulation result was shown, so this must be verified in the environment. ### Step 11 --- Understand automatic application to future indexes Because the template matches: ``` json "index_patterns": [ "cpanel-audit-log-*" ] ``` it is intended to apply automatically when new matching indexes are created. The conversation explicitly distinguished: ``` text cpanel-audit-log-2026.08.20 <- existing index; NOT changed cpanel-audit-log-2026.08.21 <- template should apply automatically cpanel-audit-log-2026.08.22 <- template should apply automatically cpanel-audit-log-2026.08.23 <- template should apply automatically ``` The template does not need to be recreated every day. ### Step 12 --- Modify Data Prepper to normalize EXECVE arguments This is the permanent-fix stage. Current problematic shape: ``` json { "execve": { "args": { "a0": "/usr/bin/curl", "a1": "-k", "a2": "https://example.com" } } } ``` Target shape discussed: ``` json { "execve": { "arg_count": 3, "args": [ "/usr/bin/curl", "-k", "https://example.com" ], "args_raw": "/usr/bin/curl -k https://example.com" } } ``` The intended stable mapping would then include: ``` json "execve": { "properties": { "arg_count": { "type": "integer" }, "args": { "type": "keyword" }, "args_raw": { "type": "text" } } } ``` Why: whether a process has 3 arguments or 300, the OpenSearch mapping remains centered on: ``` text execve.arg_count execve.args execve.args_raw ``` instead of creating hundreds of position-specific fields. **Unresolved:** No exact Data Prepper processor/configuration was supplied or tested in this conversation. Do not invent one from this document. The Data Prepper pipeline must be inspected before implementing this transformation. ### Step 13 --- Allow the next daily index to be created with the corrected design The intended end-to-end flow is: ``` text auditd ↓ Fluent Bit ↓ Data Prepper ↓ normalize EXECVE arguments ↓ OpenSearch index template ↓ cpanel-audit-log-YYYY.MM.DD ``` Do not assume the existing `cpanel-audit-log-2026.08.20` mapping will be corrected by creating the template. Existing mappings are not retroactively rewritten by the new template. ------------------------------------------------------------------------ ## 6. Verification ### Verify cluster clause limit ``` http GET /_cluster/settings?include_defaults=true&flat_settings=true ``` Confirm the intended value for: ``` text indices.query.bool.max_clause_count ``` If the temporary workaround was applied, confirm it resolves to `4096`. ### Verify the template ``` http GET /_index_template/cpanel_audit_template ``` Confirm that its pattern is: ``` json "index_patterns": [ "cpanel-audit-log-*" ] ``` ### Verify template resolution before index creation ``` http POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21 ``` Confirm the intended audit template is selected and review `overlapping` for conflicts. ### Verify a future index after it is created The conversation did not provide a post-creation verification command/output beyond the template APIs and mapping/field-capabilities APIs already used. Reuse: ``` http GET /cpanel-audit-log-2026.08.21/_mapping ``` and: ``` http GET /cpanel-audit-log-2026.08.21/_field_caps?fields=* ``` The critical design check is that the permanent Data Prepper fix eventually prevents continued creation of fields such as: ``` text execve.args.a0 execve.args.a1 ... execve.args.a309 ``` No successful future-index verification was shown in the conversation. ### Verify the search itself Retry the Dashboard/search that originally produced: ``` text too_many_nested_clauses ``` Success means the search no longer reports a failed shard for this reason. No successful retry output was shown in the conversation. ------------------------------------------------------------------------ ## 7. Troubleshooting Checklist - [ ] Confirm the error is `too_many_nested_clauses` and note the reported `maxClauseCount`. - [ ] Check whether the response has failed shards rather than treating `hits.total: 0` as definitive. - [ ] Run `GET /_cluster/settings?include_defaults=true&flat_settings=true`. - [ ] Inspect `GET /cpanel-audit-log-2026.08.20/_mapping`. - [ ] Inspect `GET /cpanel-audit-log-2026.08.20/_field_caps?fields=*`. - [ ] Look for large field families such as `execve.args.a0`, `a1`, `a2`, ... . - [ ] Check whether those fields also have `.keyword` multifields. - [ ] Check Dashboard queries for `fields: ["*"]`, unspecified fields, `*`, wildcard/prefix/regexp/fuzzy searches, or large OR lists. - [ ] Restrict broad `query_string` searches to required fields where possible. - [ ] If an immediate workaround is required, consider the discussed `4096` clause limit rather than jumping to extremely large values. - [ ] Run `GET /_index_template` and identify every template matching the audit index pattern. - [ ] Confirm `cpanel_access_template` only matches `cpanel-access-log-*`. - [ ] Create/verify `cpanel_audit_template` for `cpanel-audit-log-*`. - [ ] Simulate `cpanel-audit-log-2026.08.21` and inspect overlaps. - [ ] Do not expect the new template to rewrite `cpanel-audit-log-2026.08.20`. - [ ] Do not raise `index.mapping.total_fields.limit` merely to allow the exploding schema to grow. - [ ] Inspect the Data Prepper pipeline and normalize EXECVE arguments before indexing. - [ ] Verify future mappings no longer proliferate `execve.args.aN` fields. - [ ] Retry the original search and confirm all shards complete successfully. ------------------------------------------------------------------------ ## 8. Dead Ends / What Didn't Work ### Treating the empty hit count as proof there are no logs The search showed zero hits and an empty aggregation, but it also had one failed shard. Therefore, the empty result was not a reliable indication that the audit index contained no relevant logs. ### Raising only `max_clause_count` The proposed change: ``` http PUT /_cluster/settings { "persistent": { "indices.query.bool.max_clause_count": 4096 } } ``` was explicitly treated as a workaround, not the root fix. It does not stop field proliferation. ### Raising the total field limit The conversation specifically warned against solving the issue with: ``` json "index.mapping.total_fields.limit": 5000 ``` That would mainly allow the bad schema to grow further. ### Creating only an index template The index template improves mappings for future indexes, but it does not rewrite incoming documents. If Data Prepper continues emitting: ``` text execve.args.a0 execve.args.a1 execve.args.a2 ... ``` OpenSearch can still dynamically create those fields because the proposed template keeps `"dynamic": true`. ### Trying to repair the current mapping in place The recommendation was not to attempt to turn the already-existing `a0`, `a1`, etc. fields into a single array field in the current index. The clean path is to correct the ingestion schema/template for new indexes. ### No implemented Data Prepper transformation yet The permanent transformation was identified, but no exact Data Prepper configuration was produced or tested. This remains unfinished rather than a failed command. ------------------------------------------------------------------------ ## 9. Common Pitfalls / Gotchas 1. **Templates are not retroactive.** `cpanel-audit-log-2026.08.20` will not be remapped by creating `cpanel_audit_template`. 2. **Templates do not transform documents.** They control settings/mappings. Data Prepper must normalize the incoming EXECVE structure. 3. **`dynamic: true` still permits new fields.** The dynamic string template reduces text-plus-keyword duplication, but it does not prevent `a0`, `a1`, etc. from becoming distinct fields. 4. **Do not confuse access and audit patterns.** `cpanel_access_template` matches `cpanel-access-log-*`; the audit template matches `cpanel-audit-log-*`. 5. **Do not recreate the audit template daily.** Once present, its wildcard pattern is intended to match future daily audit indexes automatically. 6. **Do not assume a simulation succeeded.** No simulation output was provided in the conversation. 7. **Do not assume the template was created successfully.** No `PUT /_index_template/cpanel_audit_template` response was shown. 8. **Do not assume `4096` was applied.** The conversation supplied the command but did not show its result. 9. **Avoid arbitrarily huge clause limits.** Values such as `10000` or `65536` were specifically discouraged as an immediate response. 10. **Avoid increasing the field limit to hide mapping explosion.** The discussed template intentionally kept `index.mapping.total_fields.limit` at `1000`. 11. **The normalized EXECVE representation needs a final schema decision.** One example used a separate `command` field; another included the executable in `args`. The conversation did not resolve that difference. 12. **The illustrative restricted query fields were not verified against this audit schema.** `message`, `user`, `source.ip`, and `event.action` were examples of explicit field targeting, not confirmed audit-index field names. ------------------------------------------------------------------------ ## 10. Root Cause Summary The identified root cause was uncontrolled dynamic field creation in the cPanel Linux audit-log schema, especially the representation of EXECVE command arguments as position-specific object keys such as `execve.args.a0`, `execve.args.a1`, and continuing through hundreds of possible argument positions. Many dynamically mapped strings also received `.keyword` multifields, further increasing the field count. Broad OpenSearch Dashboard/query-string searches can expand across this large set of searchable fields and generate more than the configured `1024` Boolean clauses, causing `too_many_nested_clauses`. The permanent fix is to normalize EXECVE arguments in Data Prepper into a stable array/raw-command/count structure before indexing and use a controlled index template for future `cpanel-audit-log-*` indexes. ------------------------------------------------------------------------ ## 11. Appendix ### A. Full reference commands ``` http GET /_cluster/settings?include_defaults=true&flat_settings=true ``` ``` http PUT /_cluster/settings { "persistent": { "indices.query.bool.max_clause_count": 4096 } } ``` ``` http GET /_cluster/settings?include_defaults=true&flat_settings=true ``` ``` http GET /cpanel-audit-log-2026.08.20/_mapping ``` ``` http GET /cpanel-audit-log-2026.08.20/_field_caps?fields=* ``` ``` bash curl -sk -u admin:'PASSWORD' \ 'https://OPENSEARCH:9200/cpanel-audit-log-2026.08.20/_mapping' \ | jq '.. | objects | select(has("type")) | .type' \ | wc -l ``` ``` http GET /_index_template ``` ``` http POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21 ``` ``` http PUT /_index_template/cpanel_audit_template { "index_patterns": [ "cpanel-audit-log-*" ], "priority": 100, "version": 1, "_meta": { "flow": "audit", "description": "Template for cPanel Linux audit logs" }, "template": { "settings": { "index.mapping.total_fields.limit": 1000 }, "mappings": { "dynamic": true, "date_detection": false, "dynamic_templates": [ { "strings_as_keyword": { "match_mapping_type": "string", "mapping": { "type": "keyword", "ignore_above": 1024 } } } ], "properties": { "@timestamp": { "type": "date" }, "log_source": { "type": "keyword" }, "source_file": { "type": "keyword" }, "log": { "type": "text" }, "audit": { "properties": { "event_id": { "type": "keyword" }, "log_type": { "type": "keyword" }, "msg": { "type": "text" }, "timestamp_seconds": { "type": "long" }, "timestamp_fraction": { "type": "keyword" } } }, "execve": { "properties": { "arg_count": { "type": "integer" }, "args_raw": { "type": "text" } } }, "syscall": { "properties": { "pid": { "type": "long" }, "ppid": { "type": "long" }, "uid": { "type": "long" }, "gid": { "type": "long" }, "auid": { "type": "long" }, "comm": { "type": "keyword" }, "exe": { "type": "keyword" }, "architecture": { "type": "keyword" }, "syscall": { "type": "keyword" }, "success": { "type": "keyword" }, "exit_code": { "type": "long" }, "terminal": { "type": "keyword" }, "key": { "type": "keyword" } } } } } } } ``` ``` http GET /_index_template/cpanel_audit_template ``` ``` http POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21 ``` ``` http GET /cpanel-audit-log-2026.08.21/_mapping ``` ``` http GET /cpanel-audit-log-2026.08.21/_field_caps?fields=* ``` ### B. Earlier proposed template variant Before the existing access template was shown, an earlier template was proposed under the name `cpanel-audit-log-template` with priority `200`. This was later superseded in the conversation by `cpanel_audit_template` with priority `100`. The earlier creation endpoint was: ``` http PUT /_index_template/cpanel-audit-log-template ``` and its verification endpoint was: ``` http GET /_index_template/cpanel-audit-log-template ``` This difference is retained here because it appeared in the conversation. For the reconstructed procedure, use the later `cpanel_audit_template` unless your environment already contains one of these names and you deliberately choose otherwise. ### C. Existing access-log template observed ``` json { "index_templates": [ { "name": "cpanel_access_template", "index_template": { "index_patterns": [ "cpanel-access-log-*" ], "template": { "settings": {}, "mappings": { "properties": { "parsed": { "type": "object", "properties": { "client_ip": { "type": "ip" } } } } } }, "composed_of": [], "priority": 0, "_meta": { "flow": "simple" } } } ] } ``` ### D. Query examples referenced Broad query: ``` json { "query": { "query_string": { "query": "login" } } } ``` Explicit-field example: ``` json { "query": { "query_string": { "query": "login", "fields": [ "message", "user", "source.ip", "event.action" ] } } } ``` Potentially problematic query/filter patterns: ``` text * error* user:* field:(value1 OR value2 OR value3 OR ...) ``` ### E. EXECVE structures referenced Problematic current concept: ``` json { "execve": { "args": { "a0": "curl", "a1": "-k", "a2": "https://example.com", "a3": "-o", "a4": "file" } } } ``` Normalized example with separate command: ``` json { "execve": { "command": "curl", "args": [ "-k", "https://example.com", "-o", "file" ], "args_raw": "curl -k https://example.com -o file", "arg_count": 5 } } ``` Normalized example with executable included in `args`: ``` json { "execve": { "arg_count": 3, "args": [ "/usr/bin/curl", "-k", "https://example.com" ], "args_raw": "/usr/bin/curl -k https://example.com" } } ``` Stable mapping concept: ``` json "execve": { "properties": { "arg_count": { "type": "integer" }, "args": { "type": "keyword" }, "args_raw": { "type": "text" } } } ``` ### F. Referenced index names and patterns ``` text cpanel-audit-log-2026.08.20 cpanel-audit-log-2026.08.21 cpanel-audit-log-2026.08.22 cpanel-audit-log-2026.08.23 cpanel-audit-log-* cpanel-access-log-* cpanel-audit-log-YYYY.MM.DD ``` ### G. Ingestion flow ``` text auditd ↓ Fluent Bit ↓ Data Prepper ↓ normalize EXECVE arguments ↓ OpenSearch index template ↓ cpanel-audit-log-YYYY.MM.DD ``` ### H. Glossary - **auditd** --- Linux Audit daemon, the source of the audit events discussed. - **Boolean clause** --- An internal query component used by Lucene/OpenSearch Boolean queries. - **cPanel** --- The server/control-panel environment whose access and audit logs are being ingested. - **Data Prepper** --- The processing stage identified for normalizing audit event fields before OpenSearch indexing. - **Dev Tools** --- OpenSearch Dashboards interface used to issue REST API requests. - **Dynamic mapping** --- Automatic field mapping based on incoming document values. - **Dynamic template** --- Rule controlling how dynamically discovered fields are mapped. - **EXECVE** --- Linux audit record information describing program execution and arguments. - **Field capabilities (`_field_caps`)** --- API showing field types and search/aggregation capabilities. - **Fluent Bit** --- Log forwarder in the stated ingestion flow. - **Index pattern** --- Wildcard expression used by an index template to determine which new index names it applies to. - **Index template** --- Reusable mappings/settings automatically applied when a matching index is created. - **Mapping explosion** --- Excessive growth in mapped fields, usually caused by uncontrolled dynamic keys. - **Multifield** --- A field indexed in multiple forms, such as `text` and `.keyword`. - **OpenSearch** --- Search/analytics datastore receiving the cPanel logs. - **Shard** --- Partition of an OpenSearch index. The incident had 8 searched shards, with 1 failure. - **`maxClauseCount`** --- Maximum number of Boolean query clauses; the observed value was `1024`. ------------------------------------------------------------------------ ## 12. Coverage Check Every Pass 1 inventory item is accounted for below. 1. Included --- original `too_many_nested_clauses` shard failure. 2. Included --- `maxClauseCount` of `1024`. 3. Included --- 8 total / 7 successful / 1 failed shard. 4. Included --- zero hits and empty aggregation. 5. Included --- query-expansion diagnosis versus indexing failure. 6. Included --- cluster-settings inspection command. 7. Included --- temporary `4096` workaround. 8. Included --- post-change verification command. 9. Included --- warning against `10000`/`65536`. 10. Included --- all-field query expansion. 11. Included --- broad `query_string` example. 12. Included --- explicit-field `query_string` example. 13. Included --- wildcard/prefix/regexp/fuzzy/OR-list risk. 14. Included --- `*`, `error*`, `user:*`, and OR-list examples. 15. Included --- mapping inspection command. 16. Included --- Linux `curl | jq | wc -l` field-count command. 17. Included --- `_field_caps?fields=*`. 18. Included --- `execve.args.aN` proliferation. 19. Included --- `.keyword` EXECVE multifields. 20. Included --- other text/keyword pairs. 21. Included --- mapping-explosion diagnosis. 22. Included --- current `a0/a1/a2` object shape. 23. Included --- normalized args/args_raw/arg_count target. 24. Included --- explicit mapping concept. 25. Included --- recommendation not to repair current mapping in place. 26. Included --- auditd/Fluent Bit/Data Prepper/OpenSearch flow. 27. Included --- index-template creation question. 28. Included --- `GET /_index_template`. 29. Included --- simulation command. 30. Included --- earlier `cpanel-audit-log-template` proposal. 31. Included --- dynamic strings-as-keyword rationale. 32. Included --- earlier template verification command. 33. Included --- `overlapping` conflict check. 34. Included --- template does not transform documents. 35. Included --- Data Prepper can continue creating `aN` fields. 36. Included --- warning against field limit `5000`. 37. Included --- observed `cpanel_access_template`. 38. Included --- access template pattern differs from audit pattern. 39. Included --- no demonstrated access/audit template conflict. 40. Included --- final `cpanel_audit_template`. 41. Included --- final template priority/version/meta/settings/dynamic configuration. 42. Included --- explicit mapped field groups. 43. Included --- final template verification command. 44. Included --- final simulation command. 45. Included --- automatic-future-application question. 46. Included --- automatic application to matching new indexes. 47. Included --- existing `.2026.08.20` not retroactively changed. 48. Included --- `.21`, `.22`, `.23` future examples. 49. Included --- template does not need daily recreation. 50. Included --- Data Prepper normalization remains unresolved permanent fix. 51. Included --- explicit statement that no successful outputs were shown for template creation, simulation, clause-limit change, or Data Prepper modification.