mosiur/disaster-reports
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
disaster-reports/opensearch-cpanel-audit-mapping-explosion-index-template-sop.md
mosiur d535d2b2c1 Upload files to "/"
2026-08-20 04:54:33 +00:00

45 KiB
Raw Permalink Blame History

OpenSearch too_many_nested_clauses and cPanel Audit Log Mapping Explosion: Diagnosis, Index Template, and Permanent-Fix Plan

1. Overview / TL;DR

A search against the daily cPanel audit index cpanel-audit-log-2026.08.20 partially failed with too_many_nested_clauses because the query exceeded OpenSearch's configured maxClauseCount of 1024. Inspection of the field capabilities showed that the audit ingestion pipeline had created hundreds of independent EXECVE argument fields such as execve.args.a0 through execve.args.a309, often with additional .keyword multifields. This creates a mapping-explosion problem and makes broad Dashboard queries capable of expanding into more Boolean clauses than OpenSearch allows. The immediate workaround discussed was to raise indices.query.bool.max_clause_count to 4096, while the durable plan is to create an index template for future cpanel-audit-log-* indexes and modify Data Prepper so EXECVE arguments are normalized into a stable field structure instead of dynamically creating a0, a1, a2, and so on.

The index template described here will automatically apply to future indexes matching cpanel-audit-log-*; it will not retroactively change cpanel-audit-log-2026.08.20. The Data Prepper normalization was identified as required for the permanent fix but was not implemented or verified during this conversation.

2. Pass 1 --- Inventory

The following inventory captures every distinct issue, question, command, error, workaround, observation, decision, and unresolved item discussed in the conversation.

  1. A search returned too_many_nested_clauses on shard 0 of cpanel-audit-log-2026.08.20.
  2. The reported Boolean clause limit was maxClauseCount is set to 1024.
  3. The search involved 8 shards: 7 successful and 1 failed.
  4. The response showed hits.total: 0 and an empty aggregation despite the partial shard failure.
  5. The failure was identified as a query-expansion problem rather than proof of an indexing failure.
  6. A command was provided to inspect cluster settings, including defaults.
  7. A temporary workaround was proposed: increase indices.query.bool.max_clause_count to 4096.
  8. A command was provided to verify the cluster setting after changing it.
  9. A warning was given not to immediately use very large values such as 10000 or 65536 because larger query limits can increase CPU/heap pressure.
  10. Broad queries across all fields, including fields: ["*"] or unspecified fields, were identified as possible clause-expansion triggers.
  11. An example query_string query without explicit fields was shown.
  12. An improved query_string example restricted searches to message, user, source.ip, and event.action.
  13. Wildcard, prefix, regexp, fuzzy, and large OR-list queries were identified as possible contributors.
  14. Example risky Dashboard/query patterns included *, error*, user:*, and field:(value1 OR value2 OR value3 OR ...).
  15. A command was provided to inspect the mapping of cpanel-audit-log-2026.08.20.
  16. A Linux curl | jq | wc -l command was provided as a quick field-count technique.
  17. A _field_caps?fields=* command was provided to inspect all field capabilities.
  18. The uploaded _field_caps output showed hundreds of execve.args.aN fields.
  19. Many of those EXECVE fields also had .keyword multifields.
  20. Other fields, such as syscall.comm, audit.msg, and login.user, also appeared as text plus keyword pairs.
  21. The mapping problem was identified as mapping explosion, primarily under execve.args.
  22. The current incoming structure was described as an object with fields such as a0, a1, a2, etc.
  23. A normalized target structure was proposed using execve.args as an array, execve.args_raw as text, and execve.arg_count as a numeric field.
  24. A sample explicit mapping was proposed for @timestamp, syscall, and execve.
  25. The recommendation was not to repair the existing daily index mapping in place.
  26. The intended ingestion flow was documented as auditd -> Fluent Bit -> Data Prepper -> normalize EXECVE arguments -> OpenSearch index template -> cpanel-audit-log-YYYY.MM.DD.
  27. The user asked how to create an index template.
  28. The command GET /_index_template was provided to inspect existing composable templates.
  29. The command POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21 was provided to determine what would apply to the next daily index.
  30. A proposed cpanel-audit-log-template was supplied with index_patterns: ["cpanel-audit-log-*"], priority 200, version 1, a field limit of 1000, dynamic mapping, disabled date detection, a strings_as_keyword dynamic template, and explicit audit/syscall/execve fields.
  31. The template was designed to map generic dynamically discovered strings as keyword rather than default text plus keyword.
  32. The command GET /_index_template/cpanel-audit-log-template was provided to verify that template.
  33. The simulation response's overlapping section was identified as something to inspect for conflicts.
  34. It was emphasized that an index template does not transform incoming documents.
  35. It was emphasized that if Data Prepper continues sending execve.args.a0, a1, etc., those fields can still be dynamically created.
  36. A warning was given not to solve the problem by increasing index.mapping.total_fields.limit to 5000.
  37. The existing template output showed only one template named cpanel_access_template.
  38. cpanel_access_template matches cpanel-access-log-*, not cpanel-audit-log-*.
  39. Therefore, the existing access-log template does not apply to or conflict with the audit-log index pattern.
  40. A separate final proposed audit template named cpanel_audit_template was supplied.
  41. The final proposed audit template used index_patterns: ["cpanel-audit-log-*"], priority 100, version 1, _meta.flow: "audit", _meta.description, index.mapping.total_fields.limit: 1000, dynamic: true, date_detection: false, and a strings_as_keyword dynamic template.
  42. The final audit template explicitly mapped @timestamp, log_source, source_file, log, audit, execve, and syscall.
  43. The final template verification command was GET /_index_template/cpanel_audit_template.
  44. The final simulation command was POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21.
  45. The user asked whether the template would automatically apply to future audit indexes.
  46. It was confirmed that future indexes matching cpanel-audit-log-* will automatically receive the template when they are created.
  47. cpanel-audit-log-2026.08.20 was explicitly identified as an existing index that will not be retroactively changed.
  48. Example future indexes cpanel-audit-log-2026.08.21, .22, and .23 were identified as matching the template.
  49. It was clarified that the template does not need to be recreated every day.
  50. Data Prepper normalization remained the required permanent fix and was unresolved at the end of the conversation.
  51. No successful output from creating cpanel_audit_template, simulating it, changing max_clause_count, or modifying Data Prepper was shown in the conversation.

3. Prerequisites & Required Knowledge

Tools and access

You need:

  • Access to OpenSearch Dashboards → Dev Tools with permission to read cluster settings, inspect mappings/templates, create index templates, and---if using the temporary workaround---change persistent cluster settings.
  • Access to the OpenSearch REST API if running the supplied curl command from Linux.
  • curl, jq, and wc for the optional Linux field-count command.
  • Administrative access to the Data Prepper configuration for the permanent normalization fix. The exact Data Prepper configuration file/path was not provided in this conversation.
  • Access to the audit ingestion path involving auditd, Fluent Bit, Data Prepper, and OpenSearch.

Concepts

Concept Meaning in this procedure

OpenSearch index A collection of documents with mappings and settings. The affected index was cpanel-audit-log-2026.08.20.

Mapping Defines how document fields are stored and queried. Uncontrolled dynamic fields can cause mapping explosion.

Dynamic mapping OpenSearch automatically creates mappings for fields it sees for the first time.

Dynamic template A rule controlling how dynamically discovered fields are mapped.

Index template Settings and mappings automatically applied when a new index name matches an index pattern.

Composable index template The _index_template API form used in this conversation.

Mapping explosion Uncontrolled growth in the number of mapped fields. Here, EXECVE arguments were a major source.

Boolean clause An internal query component. The observed query exceeded a limit of 1024 clauses.

EXECVE Linux audit event information containing executable arguments.

Multifield One logical value indexed in multiple ways, such as a text field plus .keyword.

Field capabilities OpenSearch metadata showing field types and whether fields are searchable/aggregatable.

Data Prepper The pipeline stage identified as the correct place to normalize incoming audit records before indexing.

Environment and assumptions actually established in the conversation

The conversation established the following and no more:

  • OpenSearch is receiving cPanel audit logs in daily indexes named like cpanel-audit-log-2026.08.20.
  • A separate cPanel access-log template already exists for cpanel-access-log-*.
  • Audit ingestion involves auditd, Fluent Bit, Data Prepper, and OpenSearch.
  • The affected search reported a Boolean clause limit of 1024.
  • The audit mapping contains EXECVE argument fields extending at least through execve.args.a309, plus execve.args.a310[0].
  • No OpenSearch version, operating-system version, Data Prepper version, node count, heap size, or exact cluster topology was stated. Do not assume those values.

4. Background / Why This Happens

The original search did not fail because OpenSearch necessarily had no audit data. It failed on one shard because the generated query contained more Boolean clauses than the configured maximum.

The actual error was:

{
  "type": "too_many_nested_clauses",
  "reason": "too_many_nested_clauses: Query contains too many nested clauses; maxClauseCount is set to 1024"
}

The response also showed:

{
  "_shards": {
    "total": 8,
    "successful": 7,
    "skipped": 0,
    "failed": 1
  }
}

This is important because the same response contained:

{
  "hits": {
    "total": 0,
    "max_score": null,
    "hits": []
  }
}

and an empty aggregation. Because a shard failed, those empty results must not automatically be interpreted as proof that no logs exist.

How the field mapping contributed

The field-capabilities output for cpanel-audit-log-2026.08.20 showed a very large family of fields:

execve.args.a0
execve.args.a1
execve.args.a2
...
execve.args.a100
...
execve.args.a200
...
execve.args.a309
execve.args.a310[0]

Many also had keyword multifields:

execve.args.a100
execve.args.a100.keyword

The same text-plus-keyword pattern appeared elsewhere, including fields such as:

syscall.comm
syscall.comm.keyword

audit.msg
audit.msg.keyword

login.user
login.user.keyword

This means a command with many Linux audit EXECVE arguments can create many new field names. A broad Dashboard query that searches many or all searchable fields can then expand into a very large internal Boolean query. Once the number of generated clauses exceeds 1024, the search can fail with too_many_nested_clauses.

Why simply raising limits is not the permanent fix

Raising indices.query.bool.max_clause_count gives queries more room, but it does not stop the mapping from growing. Likewise, increasing index.mapping.total_fields.limit only allows more fields to accumulate.

The durable design is to normalize command arguments before OpenSearch receives them. Instead of documents shaped like:

{
  "execve": {
    "args": {
      "a0": "curl",
      "a1": "-k",
      "a2": "https://example.com",
      "a3": "-o",
      "a4": "file"
    }
  }
}

the target discussed in the conversation was:

{
  "execve": {
    "command": "curl",
    "args": [
      "-k",
      "https://example.com",
      "-o",
      "file"
    ],
    "args_raw": "curl -k https://example.com -o file",
    "arg_count": 5
  }
}

A later example showed the same principle with the executable included in the args array:

{
  "execve": {
    "arg_count": 3,
    "args": [
      "/usr/bin/curl",
      "-k",
      "https://example.com"
    ],
    "args_raw": "/usr/bin/curl -k https://example.com"
  }
}

These two examples are slightly different in whether the executable is represented separately as command or as the first element of args. The conversation did not resolve which representation should be adopted. The important invariant is that argument position must no longer become a field name such as a0, a1, or a309.

5. Step-by-Step Procedure

Step 1 --- Confirm the original failure pattern

If the search response contains the following error, treat it as a query-clause problem:

too_many_nested_clauses
Query contains too many nested clauses; maxClauseCount is set to 1024

The actual response in this incident reported:

{
  "_shards": {
    "total": 8,
    "successful": 7,
    "skipped": 0,
    "failed": 1,
    "failures": [
      {
        "shard": 0,
        "index": "cpanel-audit-log-2026.08.20",
        "node": "Wbt_J4spT3OWQg2CiDBk3g",
        "reason": {
          "type": "too_many_nested_clauses",
          "reason": "too_many_nested_clauses: Query contains too many nested clauses; maxClauseCount is set to 1024"
        }
      }
    ]
  }
}

Why: this distinguishes a partially failed search from a simple zero-hit result.

Step 2 --- Inspect the current Boolean clause setting

In OpenSearch Dashboards Dev Tools, run:

GET /_cluster/settings?include_defaults=true&flat_settings=true

Look for:

indices.query.bool.max_clause_count

No output from this command was shown in the conversation.

Step 3 --- Inspect the affected mapping and field capabilities

Run:

GET /cpanel-audit-log-2026.08.20/_mapping

Then inspect all field capabilities:

GET /cpanel-audit-log-2026.08.20/_field_caps?fields=*

The field-capabilities result in the conversation confirmed extensive EXECVE field proliferation, including execve.args.a0 through at least execve.args.a309, plus execve.args.a310[0], and many corresponding .keyword fields.

An optional Linux quick-count command provided in the conversation was:

curl -sk -u admin:'PASSWORD' \
'https://OPENSEARCH:9200/cpanel-audit-log-2026.08.20/_mapping' \
| jq '.. | objects | select(has("type")) | .type' \
| wc -l

PASSWORD and OPENSEARCH are placeholders exactly as used in the conversation. No result from this command was shown.

Step 4 --- Review Dashboard/query patterns that can amplify the problem

Check whether the failing query searches all fields, for example:

{
  "query": {
    "query_string": {
      "query": "login"
    }
  }
}

or explicitly uses:

"fields": ["*"]

Also inspect Dashboard filters or queries resembling:

*

error*

user:*

field:(value1 OR value2 OR value3 OR ...)

Where possible, constrain queries to the fields actually required. An example from the conversation was:

{
  "query": {
    "query_string": {
      "query": "login",
      "fields": [
        "message",
        "user",
        "source.ip",
        "event.action"
      ]
    }
  }
}

This was an illustrative example. The conversation did not establish that these exact four fields exist in the cPanel audit schema or that this exact query was used in the failing visualization.

Step 5 --- Apply the temporary query-limit workaround only if needed

The temporary workaround discussed was:

PUT /_cluster/settings
{
  "persistent": {
    "indices.query.bool.max_clause_count": 4096
  }
}

Then verify:

GET /_cluster/settings?include_defaults=true&flat_settings=true

Why: 4096 can allow the current broad query to execute while the schema is being corrected.

Operational warning: This changes a persistent cluster setting. The conversation warned against immediately using very large values such as 10000 or 65536, because allowing larger Boolean queries can increase CPU and heap consumption.

No successful output from setting 4096 was shown in the conversation.

Step 6 --- Inspect existing index templates

Run:

GET /_index_template

The actual result shown in the conversation contained one existing template:

{
  "index_templates": [
    {
      "name": "cpanel_access_template",
      "index_template": {
        "index_patterns": [
          "cpanel-access-log-*"
        ],
        "template": {
          "settings": {},
          "mappings": {
            "properties": {
              "parsed": {
                "type": "object",
                "properties": {
                  "client_ip": {
                    "type": "ip"
                  }
                }
              }
            }
          }
        },
        "composed_of": [],
        "priority": 0,
        "_meta": {
          "flow": "simple"
        }
      }
    }
  ]
}

This template applies to:

cpanel-access-log-*

It does not match:

cpanel-audit-log-*

Therefore, based on the templates shown in the conversation, it does not conflict with the audit-log template.

Step 7 --- Simulate what would apply to the next audit index

Before creating the audit template, the following simulation command was proposed:

POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21

Why: this shows which template would apply and helps identify overlaps/conflicts.

No simulation output was shown.

Step 8 --- Create the audit index template

The final template proposed after reviewing the existing access template was named cpanel_audit_template:

PUT /_index_template/cpanel_audit_template
{
  "index_patterns": [
    "cpanel-audit-log-*"
  ],
  "priority": 100,
  "version": 1,
  "_meta": {
    "flow": "audit",
    "description": "Template for cPanel Linux audit logs"
  },
  "template": {
    "settings": {
      "index.mapping.total_fields.limit": 1000
    },
    "mappings": {
      "dynamic": true,
      "date_detection": false,
      "dynamic_templates": [
        {
          "strings_as_keyword": {
            "match_mapping_type": "string",
            "mapping": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "log_source": {
          "type": "keyword"
        },
        "source_file": {
          "type": "keyword"
        },
        "log": {
          "type": "text"
        },
        "audit": {
          "properties": {
            "event_id": {
              "type": "keyword"
            },
            "log_type": {
              "type": "keyword"
            },
            "msg": {
              "type": "text"
            },
            "timestamp_seconds": {
              "type": "long"
            },
            "timestamp_fraction": {
              "type": "keyword"
            }
          }
        },
        "execve": {
          "properties": {
            "arg_count": {
              "type": "integer"
            },
            "args_raw": {
              "type": "text"
            }
          }
        },
        "syscall": {
          "properties": {
            "pid": {
              "type": "long"
            },
            "ppid": {
              "type": "long"
            },
            "uid": {
              "type": "long"
            },
            "gid": {
              "type": "long"
            },
            "auid": {
              "type": "long"
            },
            "comm": {
              "type": "keyword"
            },
            "exe": {
              "type": "keyword"
            },
            "architecture": {
              "type": "keyword"
            },
            "syscall": {
              "type": "keyword"
            },
            "success": {
              "type": "keyword"
            },
            "exit_code": {
              "type": "long"
            },
            "terminal": {
              "type": "keyword"
            },
            "key": {
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}

Why:

  • index_patterns ensures future indexes named cpanel-audit-log-* match automatically.
  • dynamic_templates maps otherwise-unmapped strings as a single keyword field rather than automatically creating a text field plus .keyword.
  • Explicit mappings are supplied for important audit/syscall fields.
  • index.mapping.total_fields.limit remains at 1000 rather than being raised to accommodate uncontrolled field growth.

Important limitation: This template does not transform incoming documents. With "dynamic": true, incoming execve.args.a0, execve.args.a1, etc. can still become fields. Data Prepper normalization is still required.

Step 9 --- Verify the audit template exists

Run:

GET /_index_template/cpanel_audit_template

No actual output from this command was shown in the conversation.

Step 10 --- Simulate the future daily index again

Run:

POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21

Inspect the resolved mappings/settings and any:

"overlapping": [...]

The expected goal is for cpanel_audit_template to apply to cpanel-audit-log-2026.08.21. No actual simulation result was shown, so this must be verified in the environment.

Step 11 --- Understand automatic application to future indexes

Because the template matches:

"index_patterns": [
  "cpanel-audit-log-*"
]

it is intended to apply automatically when new matching indexes are created.

The conversation explicitly distinguished:

cpanel-audit-log-2026.08.20   <- existing index; NOT changed
cpanel-audit-log-2026.08.21   <- template should apply automatically
cpanel-audit-log-2026.08.22   <- template should apply automatically
cpanel-audit-log-2026.08.23   <- template should apply automatically

The template does not need to be recreated every day.

Step 12 --- Modify Data Prepper to normalize EXECVE arguments

This is the permanent-fix stage.

Current problematic shape:

{
  "execve": {
    "args": {
      "a0": "/usr/bin/curl",
      "a1": "-k",
      "a2": "https://example.com"
    }
  }
}

Target shape discussed:

{
  "execve": {
    "arg_count": 3,
    "args": [
      "/usr/bin/curl",
      "-k",
      "https://example.com"
    ],
    "args_raw": "/usr/bin/curl -k https://example.com"
  }
}

The intended stable mapping would then include:

"execve": {
  "properties": {
    "arg_count": {
      "type": "integer"
    },
    "args": {
      "type": "keyword"
    },
    "args_raw": {
      "type": "text"
    }
  }
}

Why: whether a process has 3 arguments or 300, the OpenSearch mapping remains centered on:

execve.arg_count
execve.args
execve.args_raw

instead of creating hundreds of position-specific fields.

Unresolved: No exact Data Prepper processor/configuration was supplied or tested in this conversation. Do not invent one from this document. The Data Prepper pipeline must be inspected before implementing this transformation.

Step 13 --- Allow the next daily index to be created with the corrected design

The intended end-to-end flow is:

auditd
   ↓
Fluent Bit
   ↓
Data Prepper
   ↓
normalize EXECVE arguments
   ↓
OpenSearch index template
   ↓
cpanel-audit-log-YYYY.MM.DD

Do not assume the existing cpanel-audit-log-2026.08.20 mapping will be corrected by creating the template. Existing mappings are not retroactively rewritten by the new template.

6. Verification

Verify cluster clause limit

GET /_cluster/settings?include_defaults=true&flat_settings=true

Confirm the intended value for:

indices.query.bool.max_clause_count

If the temporary workaround was applied, confirm it resolves to 4096.

Verify the template

GET /_index_template/cpanel_audit_template

Confirm that its pattern is:

"index_patterns": [
  "cpanel-audit-log-*"
]

Verify template resolution before index creation

POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21

Confirm the intended audit template is selected and review overlapping for conflicts.

Verify a future index after it is created

The conversation did not provide a post-creation verification command/output beyond the template APIs and mapping/field-capabilities APIs already used. Reuse:

GET /cpanel-audit-log-2026.08.21/_mapping

and:

GET /cpanel-audit-log-2026.08.21/_field_caps?fields=*

The critical design check is that the permanent Data Prepper fix eventually prevents continued creation of fields such as:

execve.args.a0
execve.args.a1
...
execve.args.a309

No successful future-index verification was shown in the conversation.

Verify the search itself

Retry the Dashboard/search that originally produced:

too_many_nested_clauses

Success means the search no longer reports a failed shard for this reason. No successful retry output was shown in the conversation.

7. Troubleshooting Checklist

  • Confirm the error is too_many_nested_clauses and note the reported maxClauseCount.
  • Check whether the response has failed shards rather than treating hits.total: 0 as definitive.
  • Run GET /_cluster/settings?include_defaults=true&flat_settings=true.
  • Inspect GET /cpanel-audit-log-2026.08.20/_mapping.
  • Inspect GET /cpanel-audit-log-2026.08.20/_field_caps?fields=*.
  • Look for large field families such as execve.args.a0, a1, a2, ... .
  • Check whether those fields also have .keyword multifields.
  • Check Dashboard queries for fields: ["*"], unspecified fields, *, wildcard/prefix/regexp/fuzzy searches, or large OR lists.
  • Restrict broad query_string searches to required fields where possible.
  • If an immediate workaround is required, consider the discussed 4096 clause limit rather than jumping to extremely large values.
  • Run GET /_index_template and identify every template matching the audit index pattern.
  • Confirm cpanel_access_template only matches cpanel-access-log-*.
  • Create/verify cpanel_audit_template for cpanel-audit-log-*.
  • Simulate cpanel-audit-log-2026.08.21 and inspect overlaps.
  • Do not expect the new template to rewrite cpanel-audit-log-2026.08.20.
  • Do not raise index.mapping.total_fields.limit merely to allow the exploding schema to grow.
  • Inspect the Data Prepper pipeline and normalize EXECVE arguments before indexing.
  • Verify future mappings no longer proliferate execve.args.aN fields.
  • Retry the original search and confirm all shards complete successfully.

8. Dead Ends / What Didn't Work

Treating the empty hit count as proof there are no logs

The search showed zero hits and an empty aggregation, but it also had one failed shard. Therefore, the empty result was not a reliable indication that the audit index contained no relevant logs.

Raising only max_clause_count

The proposed change:

PUT /_cluster/settings
{
  "persistent": {
    "indices.query.bool.max_clause_count": 4096
  }
}

was explicitly treated as a workaround, not the root fix. It does not stop field proliferation.

Raising the total field limit

The conversation specifically warned against solving the issue with:

"index.mapping.total_fields.limit": 5000

That would mainly allow the bad schema to grow further.

Creating only an index template

The index template improves mappings for future indexes, but it does not rewrite incoming documents. If Data Prepper continues emitting:

execve.args.a0
execve.args.a1
execve.args.a2
...

OpenSearch can still dynamically create those fields because the proposed template keeps "dynamic": true.

Trying to repair the current mapping in place

The recommendation was not to attempt to turn the already-existing a0, a1, etc. fields into a single array field in the current index. The clean path is to correct the ingestion schema/template for new indexes.

No implemented Data Prepper transformation yet

The permanent transformation was identified, but no exact Data Prepper configuration was produced or tested. This remains unfinished rather than a failed command.

9. Common Pitfalls / Gotchas

  1. Templates are not retroactive. cpanel-audit-log-2026.08.20 will not be remapped by creating cpanel_audit_template.
  2. Templates do not transform documents. They control settings/mappings. Data Prepper must normalize the incoming EXECVE structure.
  3. dynamic: true still permits new fields. The dynamic string template reduces text-plus-keyword duplication, but it does not prevent a0, a1, etc. from becoming distinct fields.
  4. Do not confuse access and audit patterns. cpanel_access_template matches cpanel-access-log-*; the audit template matches cpanel-audit-log-*.
  5. Do not recreate the audit template daily. Once present, its wildcard pattern is intended to match future daily audit indexes automatically.
  6. Do not assume a simulation succeeded. No simulation output was provided in the conversation.
  7. Do not assume the template was created successfully. No PUT /_index_template/cpanel_audit_template response was shown.
  8. Do not assume 4096 was applied. The conversation supplied the command but did not show its result.
  9. Avoid arbitrarily huge clause limits. Values such as 10000 or 65536 were specifically discouraged as an immediate response.
  10. Avoid increasing the field limit to hide mapping explosion. The discussed template intentionally kept index.mapping.total_fields.limit at 1000.
  11. The normalized EXECVE representation needs a final schema decision. One example used a separate command field; another included the executable in args. The conversation did not resolve that difference.
  12. The illustrative restricted query fields were not verified against this audit schema. message, user, source.ip, and event.action were examples of explicit field targeting, not confirmed audit-index field names.

10. Root Cause Summary

The identified root cause was uncontrolled dynamic field creation in the cPanel Linux audit-log schema, especially the representation of EXECVE command arguments as position-specific object keys such as execve.args.a0, execve.args.a1, and continuing through hundreds of possible argument positions. Many dynamically mapped strings also received .keyword multifields, further increasing the field count. Broad OpenSearch Dashboard/query-string searches can expand across this large set of searchable fields and generate more than the configured 1024 Boolean clauses, causing too_many_nested_clauses. The permanent fix is to normalize EXECVE arguments in Data Prepper into a stable array/raw-command/count structure before indexing and use a controlled index template for future cpanel-audit-log-* indexes.

11. Appendix

A. Full reference commands

GET /_cluster/settings?include_defaults=true&flat_settings=true

PUT /_cluster/settings
{
  "persistent": {
    "indices.query.bool.max_clause_count": 4096
  }
}

GET /_cluster/settings?include_defaults=true&flat_settings=true

GET /cpanel-audit-log-2026.08.20/_mapping

GET /cpanel-audit-log-2026.08.20/_field_caps?fields=*

curl -sk -u admin:'PASSWORD' \
'https://OPENSEARCH:9200/cpanel-audit-log-2026.08.20/_mapping' \
| jq '.. | objects | select(has("type")) | .type' \
| wc -l

GET /_index_template

POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21

PUT /_index_template/cpanel_audit_template
{
  "index_patterns": [
    "cpanel-audit-log-*"
  ],
  "priority": 100,
  "version": 1,
  "_meta": {
    "flow": "audit",
    "description": "Template for cPanel Linux audit logs"
  },
  "template": {
    "settings": {
      "index.mapping.total_fields.limit": 1000
    },
    "mappings": {
      "dynamic": true,
      "date_detection": false,
      "dynamic_templates": [
        {
          "strings_as_keyword": {
            "match_mapping_type": "string",
            "mapping": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "log_source": {
          "type": "keyword"
        },
        "source_file": {
          "type": "keyword"
        },
        "log": {
          "type": "text"
        },
        "audit": {
          "properties": {
            "event_id": {
              "type": "keyword"
            },
            "log_type": {
              "type": "keyword"
            },
            "msg": {
              "type": "text"
            },
            "timestamp_seconds": {
              "type": "long"
            },
            "timestamp_fraction": {
              "type": "keyword"
            }
          }
        },
        "execve": {
          "properties": {
            "arg_count": {
              "type": "integer"
            },
            "args_raw": {
              "type": "text"
            }
          }
        },
        "syscall": {
          "properties": {
            "pid": {
              "type": "long"
            },
            "ppid": {
              "type": "long"
            },
            "uid": {
              "type": "long"
            },
            "gid": {
              "type": "long"
            },
            "auid": {
              "type": "long"
            },
            "comm": {
              "type": "keyword"
            },
            "exe": {
              "type": "keyword"
            },
            "architecture": {
              "type": "keyword"
            },
            "syscall": {
              "type": "keyword"
            },
            "success": {
              "type": "keyword"
            },
            "exit_code": {
              "type": "long"
            },
            "terminal": {
              "type": "keyword"
            },
            "key": {
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}

GET /_index_template/cpanel_audit_template

POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21

GET /cpanel-audit-log-2026.08.21/_mapping

GET /cpanel-audit-log-2026.08.21/_field_caps?fields=*

B. Earlier proposed template variant

Before the existing access template was shown, an earlier template was proposed under the name cpanel-audit-log-template with priority 200. This was later superseded in the conversation by cpanel_audit_template with priority 100.

The earlier creation endpoint was:

PUT /_index_template/cpanel-audit-log-template

and its verification endpoint was:

GET /_index_template/cpanel-audit-log-template

This difference is retained here because it appeared in the conversation. For the reconstructed procedure, use the later cpanel_audit_template unless your environment already contains one of these names and you deliberately choose otherwise.

C. Existing access-log template observed

{
  "index_templates": [
    {
      "name": "cpanel_access_template",
      "index_template": {
        "index_patterns": [
          "cpanel-access-log-*"
        ],
        "template": {
          "settings": {},
          "mappings": {
            "properties": {
              "parsed": {
                "type": "object",
                "properties": {
                  "client_ip": {
                    "type": "ip"
                  }
                }
              }
            }
          }
        },
        "composed_of": [],
        "priority": 0,
        "_meta": {
          "flow": "simple"
        }
      }
    }
  ]
}

D. Query examples referenced

Broad query:

{
  "query": {
    "query_string": {
      "query": "login"
    }
  }
}

Explicit-field example:

{
  "query": {
    "query_string": {
      "query": "login",
      "fields": [
        "message",
        "user",
        "source.ip",
        "event.action"
      ]
    }
  }
}

Potentially problematic query/filter patterns:

*
error*
user:*
field:(value1 OR value2 OR value3 OR ...)

E. EXECVE structures referenced

Problematic current concept:

{
  "execve": {
    "args": {
      "a0": "curl",
      "a1": "-k",
      "a2": "https://example.com",
      "a3": "-o",
      "a4": "file"
    }
  }
}

Normalized example with separate command:

{
  "execve": {
    "command": "curl",
    "args": [
      "-k",
      "https://example.com",
      "-o",
      "file"
    ],
    "args_raw": "curl -k https://example.com -o file",
    "arg_count": 5
  }
}

Normalized example with executable included in args:

{
  "execve": {
    "arg_count": 3,
    "args": [
      "/usr/bin/curl",
      "-k",
      "https://example.com"
    ],
    "args_raw": "/usr/bin/curl -k https://example.com"
  }
}

Stable mapping concept:

"execve": {
  "properties": {
    "arg_count": {
      "type": "integer"
    },
    "args": {
      "type": "keyword"
    },
    "args_raw": {
      "type": "text"
    }
  }
}

F. Referenced index names and patterns

cpanel-audit-log-2026.08.20
cpanel-audit-log-2026.08.21
cpanel-audit-log-2026.08.22
cpanel-audit-log-2026.08.23
cpanel-audit-log-*
cpanel-access-log-*
cpanel-audit-log-YYYY.MM.DD

G. Ingestion flow

auditd
   ↓
Fluent Bit
   ↓
Data Prepper
   ↓
normalize EXECVE arguments
   ↓
OpenSearch index template
   ↓
cpanel-audit-log-YYYY.MM.DD

H. Glossary

  • auditd --- Linux Audit daemon, the source of the audit events discussed.
  • Boolean clause --- An internal query component used by Lucene/OpenSearch Boolean queries.
  • cPanel --- The server/control-panel environment whose access and audit logs are being ingested.
  • Data Prepper --- The processing stage identified for normalizing audit event fields before OpenSearch indexing.
  • Dev Tools --- OpenSearch Dashboards interface used to issue REST API requests.
  • Dynamic mapping --- Automatic field mapping based on incoming document values.
  • Dynamic template --- Rule controlling how dynamically discovered fields are mapped.
  • EXECVE --- Linux audit record information describing program execution and arguments.
  • Field capabilities (_field_caps) --- API showing field types and search/aggregation capabilities.
  • Fluent Bit --- Log forwarder in the stated ingestion flow.
  • Index pattern --- Wildcard expression used by an index template to determine which new index names it applies to.
  • Index template --- Reusable mappings/settings automatically applied when a matching index is created.
  • Mapping explosion --- Excessive growth in mapped fields, usually caused by uncontrolled dynamic keys.
  • Multifield --- A field indexed in multiple forms, such as text and .keyword.
  • OpenSearch --- Search/analytics datastore receiving the cPanel logs.
  • Shard --- Partition of an OpenSearch index. The incident had 8 searched shards, with 1 failure.
  • maxClauseCount --- Maximum number of Boolean query clauses; the observed value was 1024.

12. Coverage Check

Every Pass 1 inventory item is accounted for below.

  1. Included --- original too_many_nested_clauses shard failure.
  2. Included --- maxClauseCount of 1024.
  3. Included --- 8 total / 7 successful / 1 failed shard.
  4. Included --- zero hits and empty aggregation.
  5. Included --- query-expansion diagnosis versus indexing failure.
  6. Included --- cluster-settings inspection command.
  7. Included --- temporary 4096 workaround.
  8. Included --- post-change verification command.
  9. Included --- warning against 10000/65536.
  10. Included --- all-field query expansion.
  11. Included --- broad query_string example.
  12. Included --- explicit-field query_string example.
  13. Included --- wildcard/prefix/regexp/fuzzy/OR-list risk.
  14. Included --- *, error*, user:*, and OR-list examples.
  15. Included --- mapping inspection command.
  16. Included --- Linux curl | jq | wc -l field-count command.
  17. Included --- _field_caps?fields=*.
  18. Included --- execve.args.aN proliferation.
  19. Included --- .keyword EXECVE multifields.
  20. Included --- other text/keyword pairs.
  21. Included --- mapping-explosion diagnosis.
  22. Included --- current a0/a1/a2 object shape.
  23. Included --- normalized args/args_raw/arg_count target.
  24. Included --- explicit mapping concept.
  25. Included --- recommendation not to repair current mapping in place.
  26. Included --- auditd/Fluent Bit/Data Prepper/OpenSearch flow.
  27. Included --- index-template creation question.
  28. Included --- GET /_index_template.
  29. Included --- simulation command.
  30. Included --- earlier cpanel-audit-log-template proposal.
  31. Included --- dynamic strings-as-keyword rationale.
  32. Included --- earlier template verification command.
  33. Included --- overlapping conflict check.
  34. Included --- template does not transform documents.
  35. Included --- Data Prepper can continue creating aN fields.
  36. Included --- warning against field limit 5000.
  37. Included --- observed cpanel_access_template.
  38. Included --- access template pattern differs from audit pattern.
  39. Included --- no demonstrated access/audit template conflict.
  40. Included --- final cpanel_audit_template.
  41. Included --- final template priority/version/meta/settings/dynamic configuration.
  42. Included --- explicit mapped field groups.
  43. Included --- final template verification command.
  44. Included --- final simulation command.
  45. Included --- automatic-future-application question.
  46. Included --- automatic application to matching new indexes.
  47. Included --- existing .2026.08.20 not retroactively changed.
  48. Included --- .21, .22, .23 future examples.
  49. Included --- template does not need daily recreation.
  50. Included --- Data Prepper normalization remains unresolved permanent fix.
  51. Included --- explicit statement that no successful outputs were shown for template creation, simulation, clause-limit change, or Data Prepper modification.
Reference in New Issue View Git Blame Copy Permalink