mosiur/disaster-reports
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
disaster-reports/opensearch-cpanel-audit-mapping-explosion-index-template-sop.md
mosiur d535d2b2c1 Upload files to "/"
2026-08-20 04:54:33 +00:00

1648 lines
45 KiB
Markdown
Raw Permalink Blame History

# OpenSearch `too_many_nested_clauses` and cPanel Audit Log Mapping Explosion: Diagnosis, Index Template, and Permanent-Fix Plan
## 1. Overview / TL;DR
A search against the daily cPanel audit index
`cpanel-audit-log-2026.08.20` partially failed with
`too_many_nested_clauses` because the query exceeded OpenSearch's
configured `maxClauseCount` of `1024`. Inspection of the field
capabilities showed that the audit ingestion pipeline had created
hundreds of independent EXECVE argument fields such as `execve.args.a0`
through `execve.args.a309`, often with additional `.keyword`
multifields. This creates a mapping-explosion problem and makes broad
Dashboard queries capable of expanding into more Boolean clauses than
OpenSearch allows. The immediate workaround discussed was to raise
`indices.query.bool.max_clause_count` to `4096`, while the durable plan
is to create an index template for future `cpanel-audit-log-*` indexes
and modify Data Prepper so EXECVE arguments are normalized into a stable
field structure instead of dynamically creating `a0`, `a1`, `a2`, and so
on.
The index template described here will automatically apply to future
indexes matching `cpanel-audit-log-*`; it will not retroactively change
`cpanel-audit-log-2026.08.20`. The Data Prepper normalization was
identified as required for the permanent fix but was not implemented or
verified during this conversation.
------------------------------------------------------------------------
## 2. Pass 1 --- Inventory
The following inventory captures every distinct issue, question,
command, error, workaround, observation, decision, and unresolved item
discussed in the conversation.
1. A search returned `too_many_nested_clauses` on shard `0` of
`cpanel-audit-log-2026.08.20`.
2. The reported Boolean clause limit was
`maxClauseCount is set to 1024`.
3. The search involved 8 shards: 7 successful and 1 failed.
4. The response showed `hits.total: 0` and an empty aggregation despite
the partial shard failure.
5. The failure was identified as a query-expansion problem rather than
proof of an indexing failure.
6. A command was provided to inspect cluster settings, including
defaults.
7. A temporary workaround was proposed: increase
`indices.query.bool.max_clause_count` to `4096`.
8. A command was provided to verify the cluster setting after changing
it.
9. A warning was given not to immediately use very large values such as
`10000` or `65536` because larger query limits can increase CPU/heap
pressure.
10. Broad queries across all fields, including `fields: ["*"]` or
unspecified fields, were identified as possible clause-expansion
triggers.
11. An example `query_string` query without explicit fields was shown.
12. An improved `query_string` example restricted searches to `message`,
`user`, `source.ip`, and `event.action`.
13. Wildcard, prefix, regexp, fuzzy, and large OR-list queries were
identified as possible contributors.
14. Example risky Dashboard/query patterns included `*`, `error*`,
`user:*`, and `field:(value1 OR value2 OR value3 OR ...)`.
15. A command was provided to inspect the mapping of
`cpanel-audit-log-2026.08.20`.
16. A Linux `curl | jq | wc -l` command was provided as a quick
field-count technique.
17. A `_field_caps?fields=*` command was provided to inspect all field
capabilities.
18. The uploaded `_field_caps` output showed hundreds of
`execve.args.aN` fields.
19. Many of those EXECVE fields also had `.keyword` multifields.
20. Other fields, such as `syscall.comm`, `audit.msg`, and `login.user`,
also appeared as text plus keyword pairs.
21. The mapping problem was identified as mapping explosion, primarily
under `execve.args`.
22. The current incoming structure was described as an object with
fields such as `a0`, `a1`, `a2`, etc.
23. A normalized target structure was proposed using `execve.args` as an
array, `execve.args_raw` as text, and `execve.arg_count` as a
numeric field.
24. A sample explicit mapping was proposed for `@timestamp`, `syscall`,
and `execve`.
25. The recommendation was not to repair the existing daily index
mapping in place.
26. The intended ingestion flow was documented as
`auditd -> Fluent Bit -> Data Prepper -> normalize EXECVE arguments -> OpenSearch index template -> cpanel-audit-log-YYYY.MM.DD`.
27. The user asked how to create an index template.
28. The command `GET /_index_template` was provided to inspect existing
composable templates.
29. The command
`POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21`
was provided to determine what would apply to the next daily index.
30. A proposed `cpanel-audit-log-template` was supplied with
`index_patterns: ["cpanel-audit-log-*"]`, priority `200`, version
`1`, a field limit of `1000`, dynamic mapping, disabled date
detection, a `strings_as_keyword` dynamic template, and explicit
audit/syscall/execve fields.
31. The template was designed to map generic dynamically discovered
strings as `keyword` rather than default text plus keyword.
32. The command `GET /_index_template/cpanel-audit-log-template` was
provided to verify that template.
33. The simulation response's `overlapping` section was identified as
something to inspect for conflicts.
34. It was emphasized that an index template does not transform incoming
documents.
35. It was emphasized that if Data Prepper continues sending
`execve.args.a0`, `a1`, etc., those fields can still be dynamically
created.
36. A warning was given not to solve the problem by increasing
`index.mapping.total_fields.limit` to `5000`.
37. The existing template output showed only one template named
`cpanel_access_template`.
38. `cpanel_access_template` matches `cpanel-access-log-*`, not
`cpanel-audit-log-*`.
39. Therefore, the existing access-log template does not apply to or
conflict with the audit-log index pattern.
40. A separate final proposed audit template named
`cpanel_audit_template` was supplied.
41. The final proposed audit template used
`index_patterns: ["cpanel-audit-log-*"]`, priority `100`, version
`1`, `_meta.flow: "audit"`, `_meta.description`,
`index.mapping.total_fields.limit: 1000`, `dynamic: true`,
`date_detection: false`, and a `strings_as_keyword` dynamic
template.
42. The final audit template explicitly mapped `@timestamp`,
`log_source`, `source_file`, `log`, `audit`, `execve`, and
`syscall`.
43. The final template verification command was
`GET /_index_template/cpanel_audit_template`.
44. The final simulation command was
`POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21`.
45. The user asked whether the template would automatically apply to
future audit indexes.
46. It was confirmed that future indexes matching `cpanel-audit-log-*`
will automatically receive the template when they are created.
47. `cpanel-audit-log-2026.08.20` was explicitly identified as an
existing index that will not be retroactively changed.
48. Example future indexes `cpanel-audit-log-2026.08.21`, `.22`, and
`.23` were identified as matching the template.
49. It was clarified that the template does not need to be recreated
every day.
50. Data Prepper normalization remained the required permanent fix and
was unresolved at the end of the conversation.
51. No successful output from creating `cpanel_audit_template`,
simulating it, changing `max_clause_count`, or modifying Data
Prepper was shown in the conversation.
------------------------------------------------------------------------
## 3. Prerequisites & Required Knowledge
### Tools and access
You need:
- Access to **OpenSearch Dashboards → Dev Tools** with permission to
read cluster settings, inspect mappings/templates, create index
templates, and---if using the temporary workaround---change
persistent cluster settings.
- Access to the OpenSearch REST API if running the supplied `curl`
command from Linux.
- `curl`, `jq`, and `wc` for the optional Linux field-count command.
- Administrative access to the **Data Prepper** configuration for the
permanent normalization fix. The exact Data Prepper configuration
file/path was not provided in this conversation.
- Access to the audit ingestion path involving **auditd**, **Fluent
Bit**, **Data Prepper**, and **OpenSearch**.
### Concepts
-----------------------------------------------------------------------
Concept Meaning in this procedure
----------------------------------- -----------------------------------
OpenSearch index A collection of documents with
mappings and settings. The affected
index was
`cpanel-audit-log-2026.08.20`.
Mapping Defines how document fields are
stored and queried. Uncontrolled
dynamic fields can cause mapping
explosion.
Dynamic mapping OpenSearch automatically creates
mappings for fields it sees for the
first time.
Dynamic template A rule controlling how dynamically
discovered fields are mapped.
Index template Settings and mappings automatically
applied when a new index name
matches an index pattern.
Composable index template The `_index_template` API form used
in this conversation.
Mapping explosion Uncontrolled growth in the number
of mapped fields. Here, EXECVE
arguments were a major source.
Boolean clause An internal query component. The
observed query exceeded a limit of
1024 clauses.
EXECVE Linux audit event information
containing executable arguments.
Multifield One logical value indexed in
multiple ways, such as a `text`
field plus `.keyword`.
Field capabilities OpenSearch metadata showing field
types and whether fields are
searchable/aggregatable.
Data Prepper The pipeline stage identified as
the correct place to normalize
incoming audit records before
indexing.
-----------------------------------------------------------------------
### Environment and assumptions actually established in the conversation
The conversation established the following and no more:
- OpenSearch is receiving cPanel audit logs in daily indexes named
like `cpanel-audit-log-2026.08.20`.
- A separate cPanel access-log template already exists for
`cpanel-access-log-*`.
- Audit ingestion involves auditd, Fluent Bit, Data Prepper, and
OpenSearch.
- The affected search reported a Boolean clause limit of `1024`.
- The audit mapping contains EXECVE argument fields extending at least
through `execve.args.a309`, plus `execve.args.a310[0]`.
- No OpenSearch version, operating-system version, Data Prepper
version, node count, heap size, or exact cluster topology was
stated. Do not assume those values.
------------------------------------------------------------------------
## 4. Background / Why This Happens
The original search did not fail because OpenSearch necessarily had no
audit data. It failed on one shard because the generated query contained
more Boolean clauses than the configured maximum.
The actual error was:
``` json
{
"type": "too_many_nested_clauses",
"reason": "too_many_nested_clauses: Query contains too many nested clauses; maxClauseCount is set to 1024"
}
```
The response also showed:
``` json
{
"_shards": {
"total": 8,
"successful": 7,
"skipped": 0,
"failed": 1
}
}
```
This is important because the same response contained:
``` json
{
"hits": {
"total": 0,
"max_score": null,
"hits": []
}
}
```
and an empty aggregation. Because a shard failed, those empty results
must not automatically be interpreted as proof that no logs exist.
### How the field mapping contributed
The field-capabilities output for `cpanel-audit-log-2026.08.20` showed a
very large family of fields:
``` text
execve.args.a0
execve.args.a1
execve.args.a2
...
execve.args.a100
...
execve.args.a200
...
execve.args.a309
execve.args.a310[0]
```
Many also had keyword multifields:
``` text
execve.args.a100
execve.args.a100.keyword
```
The same text-plus-keyword pattern appeared elsewhere, including fields
such as:
``` text
syscall.comm
syscall.comm.keyword
audit.msg
audit.msg.keyword
login.user
login.user.keyword
```
This means a command with many Linux audit EXECVE arguments can create
many new field names. A broad Dashboard query that searches many or all
searchable fields can then expand into a very large internal Boolean
query. Once the number of generated clauses exceeds `1024`, the search
can fail with `too_many_nested_clauses`.
### Why simply raising limits is not the permanent fix
Raising `indices.query.bool.max_clause_count` gives queries more room,
but it does not stop the mapping from growing. Likewise, increasing
`index.mapping.total_fields.limit` only allows more fields to
accumulate.
The durable design is to normalize command arguments before OpenSearch
receives them. Instead of documents shaped like:
``` json
{
"execve": {
"args": {
"a0": "curl",
"a1": "-k",
"a2": "https://example.com",
"a3": "-o",
"a4": "file"
}
}
}
```
the target discussed in the conversation was:
``` json
{
"execve": {
"command": "curl",
"args": [
"-k",
"https://example.com",
"-o",
"file"
],
"args_raw": "curl -k https://example.com -o file",
"arg_count": 5
}
}
```
A later example showed the same principle with the executable included
in the `args` array:
``` json
{
"execve": {
"arg_count": 3,
"args": [
"/usr/bin/curl",
"-k",
"https://example.com"
],
"args_raw": "/usr/bin/curl -k https://example.com"
}
}
```
These two examples are slightly different in whether the executable is
represented separately as `command` or as the first element of `args`.
The conversation did not resolve which representation should be adopted.
The important invariant is that argument position must no longer become
a field name such as `a0`, `a1`, or `a309`.
------------------------------------------------------------------------
## 5. Step-by-Step Procedure
### Step 1 --- Confirm the original failure pattern
If the search response contains the following error, treat it as a
query-clause problem:
``` text
too_many_nested_clauses
Query contains too many nested clauses; maxClauseCount is set to 1024
```
The actual response in this incident reported:
``` json
{
"_shards": {
"total": 8,
"successful": 7,
"skipped": 0,
"failed": 1,
"failures": [
{
"shard": 0,
"index": "cpanel-audit-log-2026.08.20",
"node": "Wbt_J4spT3OWQg2CiDBk3g",
"reason": {
"type": "too_many_nested_clauses",
"reason": "too_many_nested_clauses: Query contains too many nested clauses; maxClauseCount is set to 1024"
}
}
]
}
}
```
Why: this distinguishes a partially failed search from a simple zero-hit
result.
### Step 2 --- Inspect the current Boolean clause setting
In OpenSearch Dashboards Dev Tools, run:
``` http
GET /_cluster/settings?include_defaults=true&flat_settings=true
```
Look for:
``` text
indices.query.bool.max_clause_count
```
No output from this command was shown in the conversation.
### Step 3 --- Inspect the affected mapping and field capabilities
Run:
``` http
GET /cpanel-audit-log-2026.08.20/_mapping
```
Then inspect all field capabilities:
``` http
GET /cpanel-audit-log-2026.08.20/_field_caps?fields=*
```
The field-capabilities result in the conversation confirmed extensive
EXECVE field proliferation, including `execve.args.a0` through at least
`execve.args.a309`, plus `execve.args.a310[0]`, and many corresponding
`.keyword` fields.
An optional Linux quick-count command provided in the conversation was:
``` bash
curl -sk -u admin:'PASSWORD' \
'https://OPENSEARCH:9200/cpanel-audit-log-2026.08.20/_mapping' \
| jq '.. | objects | select(has("type")) | .type' \
| wc -l
```
`PASSWORD` and `OPENSEARCH` are placeholders exactly as used in the
conversation. No result from this command was shown.
### Step 4 --- Review Dashboard/query patterns that can amplify the problem
Check whether the failing query searches all fields, for example:
``` json
{
"query": {
"query_string": {
"query": "login"
}
}
}
```
or explicitly uses:
``` json
"fields": ["*"]
```
Also inspect Dashboard filters or queries resembling:
``` text
*
```
``` text
error*
```
``` text
user:*
```
``` text
field:(value1 OR value2 OR value3 OR ...)
```
Where possible, constrain queries to the fields actually required. An
example from the conversation was:
``` json
{
"query": {
"query_string": {
"query": "login",
"fields": [
"message",
"user",
"source.ip",
"event.action"
]
}
}
}
```
This was an illustrative example. The conversation did not establish
that these exact four fields exist in the cPanel audit schema or that
this exact query was used in the failing visualization.
### Step 5 --- Apply the temporary query-limit workaround only if needed
The temporary workaround discussed was:
``` http
PUT /_cluster/settings
{
"persistent": {
"indices.query.bool.max_clause_count": 4096
}
}
```
Then verify:
``` http
GET /_cluster/settings?include_defaults=true&flat_settings=true
```
Why: `4096` can allow the current broad query to execute while the
schema is being corrected.
**Operational warning:** This changes a persistent cluster setting. The
conversation warned against immediately using very large values such as
`10000` or `65536`, because allowing larger Boolean queries can increase
CPU and heap consumption.
No successful output from setting `4096` was shown in the conversation.
### Step 6 --- Inspect existing index templates
Run:
``` http
GET /_index_template
```
The actual result shown in the conversation contained one existing
template:
``` json
{
"index_templates": [
{
"name": "cpanel_access_template",
"index_template": {
"index_patterns": [
"cpanel-access-log-*"
],
"template": {
"settings": {},
"mappings": {
"properties": {
"parsed": {
"type": "object",
"properties": {
"client_ip": {
"type": "ip"
}
}
}
}
}
},
"composed_of": [],
"priority": 0,
"_meta": {
"flow": "simple"
}
}
}
]
}
```
This template applies to:
``` text
cpanel-access-log-*
```
It does not match:
``` text
cpanel-audit-log-*
```
Therefore, based on the templates shown in the conversation, it does not
conflict with the audit-log template.
### Step 7 --- Simulate what would apply to the next audit index
Before creating the audit template, the following simulation command was
proposed:
``` http
POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21
```
Why: this shows which template would apply and helps identify
overlaps/conflicts.
No simulation output was shown.
### Step 8 --- Create the audit index template
The final template proposed after reviewing the existing access template
was named `cpanel_audit_template`:
``` http
PUT /_index_template/cpanel_audit_template
{
"index_patterns": [
"cpanel-audit-log-*"
],
"priority": 100,
"version": 1,
"_meta": {
"flow": "audit",
"description": "Template for cPanel Linux audit logs"
},
"template": {
"settings": {
"index.mapping.total_fields.limit": 1000
},
"mappings": {
"dynamic": true,
"date_detection": false,
"dynamic_templates": [
{
"strings_as_keyword": {
"match_mapping_type": "string",
"mapping": {
"type": "keyword",
"ignore_above": 1024
}
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"log_source": {
"type": "keyword"
},
"source_file": {
"type": "keyword"
},
"log": {
"type": "text"
},
"audit": {
"properties": {
"event_id": {
"type": "keyword"
},
"log_type": {
"type": "keyword"
},
"msg": {
"type": "text"
},
"timestamp_seconds": {
"type": "long"
},
"timestamp_fraction": {
"type": "keyword"
}
}
},
"execve": {
"properties": {
"arg_count": {
"type": "integer"
},
"args_raw": {
"type": "text"
}
}
},
"syscall": {
"properties": {
"pid": {
"type": "long"
},
"ppid": {
"type": "long"
},
"uid": {
"type": "long"
},
"gid": {
"type": "long"
},
"auid": {
"type": "long"
},
"comm": {
"type": "keyword"
},
"exe": {
"type": "keyword"
},
"architecture": {
"type": "keyword"
},
"syscall": {
"type": "keyword"
},
"success": {
"type": "keyword"
},
"exit_code": {
"type": "long"
},
"terminal": {
"type": "keyword"
},
"key": {
"type": "keyword"
}
}
}
}
}
}
}
```
Why:
- `index_patterns` ensures future indexes named `cpanel-audit-log-*`
match automatically.
- `dynamic_templates` maps otherwise-unmapped strings as a single
`keyword` field rather than automatically creating a text field plus
`.keyword`.
- Explicit mappings are supplied for important audit/syscall fields.
- `index.mapping.total_fields.limit` remains at `1000` rather than
being raised to accommodate uncontrolled field growth.
**Important limitation:** This template does not transform incoming
documents. With `"dynamic": true`, incoming `execve.args.a0`,
`execve.args.a1`, etc. can still become fields. Data Prepper
normalization is still required.
### Step 9 --- Verify the audit template exists
Run:
``` http
GET /_index_template/cpanel_audit_template
```
No actual output from this command was shown in the conversation.
### Step 10 --- Simulate the future daily index again
Run:
``` http
POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21
```
Inspect the resolved mappings/settings and any:
``` json
"overlapping": [...]
```
The expected goal is for `cpanel_audit_template` to apply to
`cpanel-audit-log-2026.08.21`. No actual simulation result was shown, so
this must be verified in the environment.
### Step 11 --- Understand automatic application to future indexes
Because the template matches:
``` json
"index_patterns": [
"cpanel-audit-log-*"
]
```
it is intended to apply automatically when new matching indexes are
created.
The conversation explicitly distinguished:
``` text
cpanel-audit-log-2026.08.20 <- existing index; NOT changed
cpanel-audit-log-2026.08.21 <- template should apply automatically
cpanel-audit-log-2026.08.22 <- template should apply automatically
cpanel-audit-log-2026.08.23 <- template should apply automatically
```
The template does not need to be recreated every day.
### Step 12 --- Modify Data Prepper to normalize EXECVE arguments
This is the permanent-fix stage.
Current problematic shape:
``` json
{
"execve": {
"args": {
"a0": "/usr/bin/curl",
"a1": "-k",
"a2": "https://example.com"
}
}
}
```
Target shape discussed:
``` json
{
"execve": {
"arg_count": 3,
"args": [
"/usr/bin/curl",
"-k",
"https://example.com"
],
"args_raw": "/usr/bin/curl -k https://example.com"
}
}
```
The intended stable mapping would then include:
``` json
"execve": {
"properties": {
"arg_count": {
"type": "integer"
},
"args": {
"type": "keyword"
},
"args_raw": {
"type": "text"
}
}
}
```
Why: whether a process has 3 arguments or 300, the OpenSearch mapping
remains centered on:
``` text
execve.arg_count
execve.args
execve.args_raw
```
instead of creating hundreds of position-specific fields.
**Unresolved:** No exact Data Prepper processor/configuration was
supplied or tested in this conversation. Do not invent one from this
document. The Data Prepper pipeline must be inspected before
implementing this transformation.
### Step 13 --- Allow the next daily index to be created with the corrected design
The intended end-to-end flow is:
``` text
auditd
↓
Fluent Bit
↓
Data Prepper
↓
normalize EXECVE arguments
↓
OpenSearch index template
↓
cpanel-audit-log-YYYY.MM.DD
```
Do not assume the existing `cpanel-audit-log-2026.08.20` mapping will be
corrected by creating the template. Existing mappings are not
retroactively rewritten by the new template.
------------------------------------------------------------------------
## 6. Verification
### Verify cluster clause limit
``` http
GET /_cluster/settings?include_defaults=true&flat_settings=true
```
Confirm the intended value for:
``` text
indices.query.bool.max_clause_count
```
If the temporary workaround was applied, confirm it resolves to `4096`.
### Verify the template
``` http
GET /_index_template/cpanel_audit_template
```
Confirm that its pattern is:
``` json
"index_patterns": [
"cpanel-audit-log-*"
]
```
### Verify template resolution before index creation
``` http
POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21
```
Confirm the intended audit template is selected and review `overlapping`
for conflicts.
### Verify a future index after it is created
The conversation did not provide a post-creation verification
command/output beyond the template APIs and mapping/field-capabilities
APIs already used. Reuse:
``` http
GET /cpanel-audit-log-2026.08.21/_mapping
```
and:
``` http
GET /cpanel-audit-log-2026.08.21/_field_caps?fields=*
```
The critical design check is that the permanent Data Prepper fix
eventually prevents continued creation of fields such as:
``` text
execve.args.a0
execve.args.a1
...
execve.args.a309
```
No successful future-index verification was shown in the conversation.
### Verify the search itself
Retry the Dashboard/search that originally produced:
``` text
too_many_nested_clauses
```
Success means the search no longer reports a failed shard for this
reason. No successful retry output was shown in the conversation.
------------------------------------------------------------------------
## 7. Troubleshooting Checklist
- [ ] Confirm the error is `too_many_nested_clauses` and note the
reported `maxClauseCount`.
- [ ] Check whether the response has failed shards rather than
treating `hits.total: 0` as definitive.
- [ ] Run
`GET /_cluster/settings?include_defaults=true&flat_settings=true`.
- [ ] Inspect `GET /cpanel-audit-log-2026.08.20/_mapping`.
- [ ] Inspect `GET /cpanel-audit-log-2026.08.20/_field_caps?fields=*`.
- [ ] Look for large field families such as `execve.args.a0`, `a1`,
`a2`, ... .
- [ ] Check whether those fields also have `.keyword` multifields.
- [ ] Check Dashboard queries for `fields: ["*"]`, unspecified fields,
`*`, wildcard/prefix/regexp/fuzzy searches, or large OR lists.
- [ ] Restrict broad `query_string` searches to required fields where
possible.
- [ ] If an immediate workaround is required, consider the discussed
`4096` clause limit rather than jumping to extremely large values.
- [ ] Run `GET /_index_template` and identify every template matching
the audit index pattern.
- [ ] Confirm `cpanel_access_template` only matches
`cpanel-access-log-*`.
- [ ] Create/verify `cpanel_audit_template` for `cpanel-audit-log-*`.
- [ ] Simulate `cpanel-audit-log-2026.08.21` and inspect overlaps.
- [ ] Do not expect the new template to rewrite
`cpanel-audit-log-2026.08.20`.
- [ ] Do not raise `index.mapping.total_fields.limit` merely to allow
the exploding schema to grow.
- [ ] Inspect the Data Prepper pipeline and normalize EXECVE arguments
before indexing.
- [ ] Verify future mappings no longer proliferate `execve.args.aN`
fields.
- [ ] Retry the original search and confirm all shards complete
successfully.
------------------------------------------------------------------------
## 8. Dead Ends / What Didn't Work
### Treating the empty hit count as proof there are no logs
The search showed zero hits and an empty aggregation, but it also had
one failed shard. Therefore, the empty result was not a reliable
indication that the audit index contained no relevant logs.
### Raising only `max_clause_count`
The proposed change:
``` http
PUT /_cluster/settings
{
"persistent": {
"indices.query.bool.max_clause_count": 4096
}
}
```
was explicitly treated as a workaround, not the root fix. It does not
stop field proliferation.
### Raising the total field limit
The conversation specifically warned against solving the issue with:
``` json
"index.mapping.total_fields.limit": 5000
```
That would mainly allow the bad schema to grow further.
### Creating only an index template
The index template improves mappings for future indexes, but it does not
rewrite incoming documents. If Data Prepper continues emitting:
``` text
execve.args.a0
execve.args.a1
execve.args.a2
...
```
OpenSearch can still dynamically create those fields because the
proposed template keeps `"dynamic": true`.
### Trying to repair the current mapping in place
The recommendation was not to attempt to turn the already-existing `a0`,
`a1`, etc. fields into a single array field in the current index. The
clean path is to correct the ingestion schema/template for new indexes.
### No implemented Data Prepper transformation yet
The permanent transformation was identified, but no exact Data Prepper
configuration was produced or tested. This remains unfinished rather
than a failed command.
------------------------------------------------------------------------
## 9. Common Pitfalls / Gotchas
1. **Templates are not retroactive.** `cpanel-audit-log-2026.08.20`
will not be remapped by creating `cpanel_audit_template`.
2. **Templates do not transform documents.** They control
settings/mappings. Data Prepper must normalize the incoming EXECVE
structure.
3. **`dynamic: true` still permits new fields.** The dynamic string
template reduces text-plus-keyword duplication, but it does not
prevent `a0`, `a1`, etc. from becoming distinct fields.
4. **Do not confuse access and audit patterns.**
`cpanel_access_template` matches `cpanel-access-log-*`; the audit
template matches `cpanel-audit-log-*`.
5. **Do not recreate the audit template daily.** Once present, its
wildcard pattern is intended to match future daily audit indexes
automatically.
6. **Do not assume a simulation succeeded.** No simulation output was
provided in the conversation.
7. **Do not assume the template was created successfully.** No
`PUT /_index_template/cpanel_audit_template` response was shown.
8. **Do not assume `4096` was applied.** The conversation supplied the
command but did not show its result.
9. **Avoid arbitrarily huge clause limits.** Values such as `10000` or
`65536` were specifically discouraged as an immediate response.
10. **Avoid increasing the field limit to hide mapping explosion.** The
discussed template intentionally kept
`index.mapping.total_fields.limit` at `1000`.
11. **The normalized EXECVE representation needs a final schema
decision.** One example used a separate `command` field; another
included the executable in `args`. The conversation did not resolve
that difference.
12. **The illustrative restricted query fields were not verified against
this audit schema.** `message`, `user`, `source.ip`, and
`event.action` were examples of explicit field targeting, not
confirmed audit-index field names.
------------------------------------------------------------------------
## 10. Root Cause Summary
The identified root cause was uncontrolled dynamic field creation in the
cPanel Linux audit-log schema, especially the representation of EXECVE
command arguments as position-specific object keys such as
`execve.args.a0`, `execve.args.a1`, and continuing through hundreds of
possible argument positions. Many dynamically mapped strings also
received `.keyword` multifields, further increasing the field count.
Broad OpenSearch Dashboard/query-string searches can expand across this
large set of searchable fields and generate more than the configured
`1024` Boolean clauses, causing `too_many_nested_clauses`. The permanent
fix is to normalize EXECVE arguments in Data Prepper into a stable
array/raw-command/count structure before indexing and use a controlled
index template for future `cpanel-audit-log-*` indexes.
------------------------------------------------------------------------
## 11. Appendix
### A. Full reference commands
``` http
GET /_cluster/settings?include_defaults=true&flat_settings=true
```
``` http
PUT /_cluster/settings
{
"persistent": {
"indices.query.bool.max_clause_count": 4096
}
}
```
``` http
GET /_cluster/settings?include_defaults=true&flat_settings=true
```
``` http
GET /cpanel-audit-log-2026.08.20/_mapping
```
``` http
GET /cpanel-audit-log-2026.08.20/_field_caps?fields=*
```
``` bash
curl -sk -u admin:'PASSWORD' \
'https://OPENSEARCH:9200/cpanel-audit-log-2026.08.20/_mapping' \
| jq '.. | objects | select(has("type")) | .type' \
| wc -l
```
``` http
GET /_index_template
```
``` http
POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21
```
``` http
PUT /_index_template/cpanel_audit_template
{
"index_patterns": [
"cpanel-audit-log-*"
],
"priority": 100,
"version": 1,
"_meta": {
"flow": "audit",
"description": "Template for cPanel Linux audit logs"
},
"template": {
"settings": {
"index.mapping.total_fields.limit": 1000
},
"mappings": {
"dynamic": true,
"date_detection": false,
"dynamic_templates": [
{
"strings_as_keyword": {
"match_mapping_type": "string",
"mapping": {
"type": "keyword",
"ignore_above": 1024
}
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"log_source": {
"type": "keyword"
},
"source_file": {
"type": "keyword"
},
"log": {
"type": "text"
},
"audit": {
"properties": {
"event_id": {
"type": "keyword"
},
"log_type": {
"type": "keyword"
},
"msg": {
"type": "text"
},
"timestamp_seconds": {
"type": "long"
},
"timestamp_fraction": {
"type": "keyword"
}
}
},
"execve": {
"properties": {
"arg_count": {
"type": "integer"
},
"args_raw": {
"type": "text"
}
}
},
"syscall": {
"properties": {
"pid": {
"type": "long"
},
"ppid": {
"type": "long"
},
"uid": {
"type": "long"
},
"gid": {
"type": "long"
},
"auid": {
"type": "long"
},
"comm": {
"type": "keyword"
},
"exe": {
"type": "keyword"
},
"architecture": {
"type": "keyword"
},
"syscall": {
"type": "keyword"
},
"success": {
"type": "keyword"
},
"exit_code": {
"type": "long"
},
"terminal": {
"type": "keyword"
},
"key": {
"type": "keyword"
}
}
}
}
}
}
}
```
``` http
GET /_index_template/cpanel_audit_template
```
``` http
POST /_index_template/_simulate_index/cpanel-audit-log-2026.08.21
```
``` http
GET /cpanel-audit-log-2026.08.21/_mapping
```
``` http
GET /cpanel-audit-log-2026.08.21/_field_caps?fields=*
```
### B. Earlier proposed template variant
Before the existing access template was shown, an earlier template was
proposed under the name `cpanel-audit-log-template` with priority `200`.
This was later superseded in the conversation by `cpanel_audit_template`
with priority `100`.
The earlier creation endpoint was:
``` http
PUT /_index_template/cpanel-audit-log-template
```
and its verification endpoint was:
``` http
GET /_index_template/cpanel-audit-log-template
```
This difference is retained here because it appeared in the
conversation. For the reconstructed procedure, use the later
`cpanel_audit_template` unless your environment already contains one of
these names and you deliberately choose otherwise.
### C. Existing access-log template observed
``` json
{
"index_templates": [
{
"name": "cpanel_access_template",
"index_template": {
"index_patterns": [
"cpanel-access-log-*"
],
"template": {
"settings": {},
"mappings": {
"properties": {
"parsed": {
"type": "object",
"properties": {
"client_ip": {
"type": "ip"
}
}
}
}
}
},
"composed_of": [],
"priority": 0,
"_meta": {
"flow": "simple"
}
}
}
]
}
```
### D. Query examples referenced
Broad query:
``` json
{
"query": {
"query_string": {
"query": "login"
}
}
}
```
Explicit-field example:
``` json
{
"query": {
"query_string": {
"query": "login",
"fields": [
"message",
"user",
"source.ip",
"event.action"
]
}
}
}
```
Potentially problematic query/filter patterns:
``` text
*
error*
user:*
field:(value1 OR value2 OR value3 OR ...)
```
### E. EXECVE structures referenced
Problematic current concept:
``` json
{
"execve": {
"args": {
"a0": "curl",
"a1": "-k",
"a2": "https://example.com",
"a3": "-o",
"a4": "file"
}
}
}
```
Normalized example with separate command:
``` json
{
"execve": {
"command": "curl",
"args": [
"-k",
"https://example.com",
"-o",
"file"
],
"args_raw": "curl -k https://example.com -o file",
"arg_count": 5
}
}
```
Normalized example with executable included in `args`:
``` json
{
"execve": {
"arg_count": 3,
"args": [
"/usr/bin/curl",
"-k",
"https://example.com"
],
"args_raw": "/usr/bin/curl -k https://example.com"
}
}
```
Stable mapping concept:
``` json
"execve": {
"properties": {
"arg_count": {
"type": "integer"
},
"args": {
"type": "keyword"
},
"args_raw": {
"type": "text"
}
}
}
```
### F. Referenced index names and patterns
``` text
cpanel-audit-log-2026.08.20
cpanel-audit-log-2026.08.21
cpanel-audit-log-2026.08.22
cpanel-audit-log-2026.08.23
cpanel-audit-log-*
cpanel-access-log-*
cpanel-audit-log-YYYY.MM.DD
```
### G. Ingestion flow
``` text
auditd
↓
Fluent Bit
↓
Data Prepper
↓
normalize EXECVE arguments
↓
OpenSearch index template
↓
cpanel-audit-log-YYYY.MM.DD
```
### H. Glossary
- **auditd** --- Linux Audit daemon, the source of the audit events
discussed.
- **Boolean clause** --- An internal query component used by
Lucene/OpenSearch Boolean queries.
- **cPanel** --- The server/control-panel environment whose access and
audit logs are being ingested.
- **Data Prepper** --- The processing stage identified for normalizing
audit event fields before OpenSearch indexing.
- **Dev Tools** --- OpenSearch Dashboards interface used to issue REST
API requests.
- **Dynamic mapping** --- Automatic field mapping based on incoming
document values.
- **Dynamic template** --- Rule controlling how dynamically discovered
fields are mapped.
- **EXECVE** --- Linux audit record information describing program
execution and arguments.
- **Field capabilities (`_field_caps`)** --- API showing field types
and search/aggregation capabilities.
- **Fluent Bit** --- Log forwarder in the stated ingestion flow.
- **Index pattern** --- Wildcard expression used by an index template
to determine which new index names it applies to.
- **Index template** --- Reusable mappings/settings automatically
applied when a matching index is created.
- **Mapping explosion** --- Excessive growth in mapped fields, usually
caused by uncontrolled dynamic keys.
- **Multifield** --- A field indexed in multiple forms, such as `text`
and `.keyword`.
- **OpenSearch** --- Search/analytics datastore receiving the cPanel
logs.
- **Shard** --- Partition of an OpenSearch index. The incident had 8
searched shards, with 1 failure.
- **`maxClauseCount`** --- Maximum number of Boolean query clauses;
the observed value was `1024`.
------------------------------------------------------------------------
## 12. Coverage Check
Every Pass 1 inventory item is accounted for below.
1. Included --- original `too_many_nested_clauses` shard failure.
2. Included --- `maxClauseCount` of `1024`.
3. Included --- 8 total / 7 successful / 1 failed shard.
4. Included --- zero hits and empty aggregation.
5. Included --- query-expansion diagnosis versus indexing failure.
6. Included --- cluster-settings inspection command.
7. Included --- temporary `4096` workaround.
8. Included --- post-change verification command.
9. Included --- warning against `10000`/`65536`.
10. Included --- all-field query expansion.
11. Included --- broad `query_string` example.
12. Included --- explicit-field `query_string` example.
13. Included --- wildcard/prefix/regexp/fuzzy/OR-list risk.
14. Included --- `*`, `error*`, `user:*`, and OR-list examples.
15. Included --- mapping inspection command.
16. Included --- Linux `curl | jq | wc -l` field-count command.
17. Included --- `_field_caps?fields=*`.
18. Included --- `execve.args.aN` proliferation.
19. Included --- `.keyword` EXECVE multifields.
20. Included --- other text/keyword pairs.
21. Included --- mapping-explosion diagnosis.
22. Included --- current `a0/a1/a2` object shape.
23. Included --- normalized args/args_raw/arg_count target.
24. Included --- explicit mapping concept.
25. Included --- recommendation not to repair current mapping in place.
26. Included --- auditd/Fluent Bit/Data Prepper/OpenSearch flow.
27. Included --- index-template creation question.
28. Included --- `GET /_index_template`.
29. Included --- simulation command.
30. Included --- earlier `cpanel-audit-log-template` proposal.
31. Included --- dynamic strings-as-keyword rationale.
32. Included --- earlier template verification command.
33. Included --- `overlapping` conflict check.
34. Included --- template does not transform documents.
35. Included --- Data Prepper can continue creating `aN` fields.
36. Included --- warning against field limit `5000`.
37. Included --- observed `cpanel_access_template`.
38. Included --- access template pattern differs from audit pattern.
39. Included --- no demonstrated access/audit template conflict.
40. Included --- final `cpanel_audit_template`.
41. Included --- final template priority/version/meta/settings/dynamic
configuration.
42. Included --- explicit mapped field groups.
43. Included --- final template verification command.
44. Included --- final simulation command.
45. Included --- automatic-future-application question.
46. Included --- automatic application to matching new indexes.
47. Included --- existing `.2026.08.20` not retroactively changed.
48. Included --- `.21`, `.22`, `.23` future examples.
49. Included --- template does not need daily recreation.
50. Included --- Data Prepper normalization remains unresolved permanent
fix.
51. Included --- explicit statement that no successful outputs were
shown for template creation, simulation, clause-limit change, or
Data Prepper modification.
Reference in New Issue View Git Blame Copy Permalink