ssi: add processor tags and preserve_original_event on failure (4/4) by kcreddy · Pull Request #20575 · elastic/integrations

kcreddy · 2026-08-06T14:08:49Z

Proposed commit message

ssi: add processor tags and preserve_original_event on failure (4/4)

Tag every ingest pipeline processor across the 15 SSI-owned packages in
this final batch with a `<name>_<content hash>` identifier, so that
failure telemetry can attribute an error to the step that produced it
rather than collapsing same-type processors into one bucket. All 3724
processors across the 42 pipeline files now carry a tag, including
those nested inside `on_failure` handlers and `foreach` bodies, and
those in the pipeline-level `on_failure` block.

The hash is taken over the processor's content together with every
processor enclosing it, with `tag` keys stripped. This makes uniqueness
mechanical rather than positional, which matters: an earlier revision
kept the plain name on whichever duplicate came first, and that handed
`set_event_start_from_device_first_seen` to a processor setting
`host.type`, `set_host_hostname_from_health_appliance_host` to one
setting `host.name`, and `vul_id_is_long` to the script converting
`qid`. Hashing over the enclosing chain also gives identical constructs
the same identity in every package, so a single query aggregates
failures for one step across the whole estate; the hash carries that
identity, the readable prefix is for humans. Tags that said nothing --
`script_<hash>`, `drop_<hash>` -- are re-seeded from the processor's
`description`.

Unlike the other three batches, no package here needed
preserve_original_event adding to its pipeline-level on_failure; all
fifteen already had it.

Four fixes ride along, each recorded in the affected package's
changelog:

  - bitsight: event.original was removed at processor 7 of 23, before
    everything that records an error, so pipeline_error documents never
    retained the payload the preserve_original_event tagging exists to
    keep. It was the only package here still carrying that pattern, and
    pipeline tests missed it because the test config presets the tag.
  - withsecure_elements: the pipeline-level on_failure handlers set
    error.message instead of appending, discarding what processor-level
    handlers had already recorded.
  - extrahop: the participants.id on_failure handler removed the
    unrelated object_id field. Unreachable, as convert to string cannot
    fail, but the tag would have entrenched it.
  - swimlane: a duplicated no-op rename of swimlane.audit_log.category
    is removed rather than given a disambiguating tag.

These are enhancements, so the packages take a minor version bump.

Updates #20558

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

Relates SSI: Standardize processor tagging and event.original preservation across owned pipelines #20558

Add unique tags to every ingest pipeline processor and add `preserve_original_event` to pipeline-level `on_failure` handlers for the 15 GA packages owned by elastic/security-service-integrations. This is the final group of four PRs standardising all SSI-owned ingest pipelines. Updates elastic#20558 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-08-06T14:10:25Z

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

Update the link field in each new changelog entry from the tracking issue to the merged pull request. Updates elastic#20558

elastic-vault-github-plugin-prod · 2026-08-06T14:45:56Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

infra-vault-gh-plugin-prod · 2026-08-11T13:15:07Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6

There are valid Vera comments that should be addressed.

I have additional queries:

Some tags lack the content hash. Why is this?
All the changes have at least type: enhancement, but all bump patch. We should be bumping minor.

Tag every ingest pipeline processor across the 15 SSI-owned packages in this final batch with a `<name>_<content hash>` identifier, so that failure telemetry can attribute an error to the step that produced it rather than collapsing same-type processors into one bucket. All 3725 processors across the 42 pipeline files now carry a tag, including those nested inside `on_failure` handlers and `foreach` bodies, and those in the pipeline-level `on_failure` block. The hash is taken over the processor's content together with every processor enclosing it, with `tag` keys stripped. This makes uniqueness mechanical rather than positional, which matters: an earlier revision of this PR kept the plain name on whichever duplicate came first, and that handed `set_event_start_from_device_first_seen` to a processor setting `host.type`, `set_host_hostname_from_health_appliance_host` to one setting `host.name`, and `vul_id_is_long` to the script converting `qid`. Position is the wrong tiebreaker when the pre-existing duplicate name is wrong for one of the processors. Hashing over the enclosing chain also gives identical constructs the same identity in every package, so a single query aggregates failures for one step across the whole estate; the hash carries that identity, the readable prefix is for humans. Add preserve_original_event to pipeline-level on_failure handlers that were missing it, ensuring the raw payload is retained when a pipeline error document is indexed. Three fixes ride along, each recorded in the affected package's changelog: - withsecure_elements: the pipeline-level `on_failure` handlers set `error.message` instead of appending to it, discarding anything the processor-level handlers had already recorded. - extrahop: the `participants.id` `on_failure` handler removed the unrelated `object_id` field. Present since the package's initial release, but unreachable, as `convert` to `string` cannot fail. - swimlane: a duplicated no-op `rename` of `swimlane.audit_log.category` is removed rather than given a disambiguating tag. These are enhancements, so the packages take a minor version bump. Updates elastic#20558

Copilot

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

kcreddy · 2026-08-12T13:20:23Z

There are valid Vera comments that should be addressed.

@efd6, all comments addressed in f100baa.

I have additional queries:

Some tags lack the content hash. Why is this?

All the changes have at least type: enhancement, but all bump patch. We should be bumping minor.

Originally, I used the elastic-package modify to add tags and shifted away from it (to use claude). The content hashing was missed in between. Also bumped minor.

Copilot

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

Copilot

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

elastic-vault-github-plugin-prod · 2026-08-12T14:49:10Z

✅ All changelog entries have the correct PR link.

vera-review-bot · 2026-08-12T14:50:31Z

🟢 No issues across the latest commits a03734e.

Review summary

Issues found across earlier commits 017d84c — 1 high

🟠 event.original is deleted at processor 6, before every processor that can fail, so error documents never retain the original payload despite the preserve_original_event tagging. Drop this remove processor to match the other 14 packages in this batch. (link) (Unresolved)

Issues found across earlier commits f100baa — 5 low

🔵 This tag is named rename_webApp_url_... but the processor renames webApp.id, not the URL. Rename the tag to rename_webApp_id_d9470b51. (link) (Resolved)
🔵 This tag claims the processor converts a mobile boolean, but it converts threat.indicator.as.number to a long. Rename it to convert_threat_indicator_as_number_to_long_05e467e1. (link) (Resolved)
🔵 This tag starts with rename_ but the processor is a remove. Change it to remove_policy_details_groupid_eac3e052. (link) (Resolved)
🔵 This tag says node_cve but it is in the variable data stream pipeline. Rename it to fingerprint_nozomi_networks_variable_497947fd. (link) (Resolved)
🔵 Several newly generated tags are just the processor type plus a hash even though the processor carries a description: that could have named it. Give these tags a descriptive stem, e.g. script_to_drop_null_values_82f51bf1 here. (link) (Resolved)

Issues found across earlier commits 5315c70 — 3 medium, 4 low

🟡 This on_failure handler removes _ingest._value.object_id, but the processor it guards converts _ingest._value.id — the new tag locks in the wrong field name. Retarget both the tag and the field to id. (link) (Resolved)
🟡 The pipeline-level on_failure uses set on error.message, which overwrites any messages the processor-level handlers already appended (link) (Resolved)
🟡 Same as the incidents pipeline: this pipeline-level on_failure uses set on error.message and overwrites messages the processor-level handlers appended (link) (Resolved)
🔵 This tag says url but the processor renames webApp.id (link) (Resolved)
🔵 This tag says vul_id but the script converts qualys_was.vulnerability.qid (link) (Resolved)
🔵 This tag says host.hostname but the processor sets host.name (link) (Resolved)
🔵 The hash suffix landed on the correctly-named processor while the mis-named one kept the clean tag (link) (Resolved)

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

infra-vault-gh-plugin-prod · 2026-08-12T15:11:48Z

💚 Build Succeeded

Buildkite Build
Commit: a03734e

History

💚 Build #47728 succeeded 017d84c
💚 Build #47720 succeeded f100baa
💚 Build #47431 succeeded 5315c70
💚 Build #47364 succeeded e1483cc

cc @kcreddy

mergify · 2026-08-12T22:26:13Z

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

Queue this pull request

elastic-vault-github-plugin-prod · 2026-08-13T07:16:04Z

Package airlock_digital - 0.6.0 containing this change is available at https://epr.elastic.co/package/airlock_digital/0.6.0/

elastic-vault-github-plugin-prod · 2026-08-13T07:16:04Z

Package armis - 0.7.0 containing this change is available at https://epr.elastic.co/package/armis/0.7.0/

elastic-vault-github-plugin-prod · 2026-08-13T07:16:05Z

Package bitsight - 0.5.0 containing this change is available at https://epr.elastic.co/package/bitsight/0.5.0/

elastic-vault-github-plugin-prod · 2026-08-13T07:16:06Z

Package cyera - 0.11.0 containing this change is available at https://epr.elastic.co/package/cyera/0.11.0/

elastic-vault-github-plugin-prod · 2026-08-13T07:16:06Z

Package entro - 0.5.0 containing this change is available at https://epr.elastic.co/package/entro/0.5.0/

elastic-vault-github-plugin-prod · 2026-08-13T07:16:07Z

Package extrahop - 0.4.0 containing this change is available at https://epr.elastic.co/package/extrahop/0.4.0/

elastic-vault-github-plugin-prod · 2026-08-13T07:16:08Z

Package neon_cyber - 0.3.0 containing this change is available at https://epr.elastic.co/package/neon_cyber/0.3.0/

elastic-vault-github-plugin-prod · 2026-08-13T07:16:09Z

Package nozomi_networks - 0.4.0 containing this change is available at https://epr.elastic.co/package/nozomi_networks/0.4.0/

elastic-vault-github-plugin-prod · 2026-08-13T07:16:09Z

Package qualys_gav - 0.10.0 containing this change is available at https://epr.elastic.co/package/qualys_gav/0.10.0/

elastic-vault-github-plugin-prod · 2026-08-13T07:16:10Z

Package qualys_was - 0.6.0 containing this change is available at https://epr.elastic.co/package/qualys_was/0.6.0/

elastic-vault-github-plugin-prod · 2026-08-13T07:16:11Z

Package swimlane - 0.6.0 containing this change is available at https://epr.elastic.co/package/swimlane/0.6.0/

elastic-vault-github-plugin-prod · 2026-08-13T07:16:11Z

Package ti_cyware_intel_exchange - 0.5.0 containing this change is available at https://epr.elastic.co/package/ti_cyware_intel_exchange/0.5.0/

elastic-vault-github-plugin-prod · 2026-08-13T07:16:12Z

Package ti_greynoise - 0.10.0 containing this change is available at https://epr.elastic.co/package/ti_greynoise/0.10.0/

elastic-vault-github-plugin-prod · 2026-08-13T07:16:13Z

Package vectra_rux - 0.6.0 containing this change is available at https://epr.elastic.co/package/vectra_rux/0.6.0/

elastic-vault-github-plugin-prod · 2026-08-13T07:16:14Z

Package withsecure_elements - 0.4.0 containing this change is available at https://epr.elastic.co/package/withsecure_elements/0.4.0/

ssi: update changelog links to PR elastic#20575

e1483cc

Update the link field in each new changelog entry from the tracking issue to the merged pull request. Updates elastic#20558

kcreddy self-assigned this Aug 6, 2026

kcreddy added enhancement New feature or request Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Aug 6, 2026

Rewrite tags to avoid renaming existing ones

5315c70

kcreddy marked this pull request as ready for review August 11, 2026 13:15

kcreddy requested review from a team as code owners August 11, 2026 13:15

vera-review-bot Bot reviewed Aug 11, 2026

View reviewed changes

efd6 reviewed Aug 11, 2026

View reviewed changes

Comment thread packages/qualys_was/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml Outdated

Copilot AI lite review requested due to automatic review settings August 12, 2026 13:16

Copilot AI reviewed Aug 12, 2026

vera-review-bot Bot reviewed Aug 12, 2026

View reviewed changes

Address more comments

017d84c

Copilot AI review requested due to automatic review settings August 12, 2026 13:57

Copilot AI reviewed Aug 12, 2026

vera-review-bot Bot reviewed Aug 12, 2026

View reviewed changes

Comment thread packages/bitsight/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml Outdated

Address more review comments

a03734e

Copilot AI review requested due to automatic review settings August 12, 2026 14:38

Copilot AI reviewed Aug 12, 2026

efd6 approved these changes Aug 12, 2026

View reviewed changes

kcreddy merged commit 8287d81 into elastic:main Aug 13, 2026
9 checks passed

kcreddy mentioned this pull request Aug 19, 2026

SSI: Standardize processor tagging and event.original preservation across owned pipelines #20558

Open

Uh oh!

Conversation

kcreddy commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Related issues

Uh oh!

github-actions Bot commented Aug 6, 2026

✅ Elastic Docs Style Checker (Vale)

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 6, 2026

🚀 Benchmarks report

Uh oh!

infra-vault-gh-plugin-prod Bot commented Aug 11, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

kcreddy commented Aug 12, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 12, 2026

Uh oh!

vera-review-bot Bot commented Aug 12, 2026

Review summary

Uh oh!

infra-vault-gh-plugin-prod Bot commented Aug 12, 2026

💚 Build Succeeded

History

Uh oh!

mergify Bot commented Aug 12, 2026

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 13, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 13, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 13, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 13, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 13, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 13, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 13, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 13, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 13, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 13, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 13, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 13, 2026

Uh oh!

kcreddy commented Aug 6, 2026 •

edited

Loading