Skip to content

Update ML transform source index to scope to the relevant data streams#19998

Merged
sodhikirti07 merged 4 commits into
mainfrom
update-transform-queried-indices
Jul 7, 2026
Merged

Update ML transform source index to scope to the relevant data streams#19998
sodhikirti07 merged 4 commits into
mainfrom
update-transform-queried-indices

Conversation

@sodhikirti07

@sodhikirti07 sodhikirti07 commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Proposed commit message

Update the source indices to scope relevant data streams

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

Using ITP

Related issues

Screenshots

@sodhikirti07 sodhikirti07 added the enhancement New feature or request label Jul 6, 2026
@sodhikirti07 sodhikirti07 requested a review from a team as a code owner July 6, 2026 16:30
@sodhikirti07 sodhikirti07 added the Integration:lmd Lateral Movement Detection label Jul 6, 2026
@sodhikirti07 sodhikirti07 requested a review from a team as a code owner July 6, 2026 16:30
@sodhikirti07 sodhikirti07 added Integration:beaconing Network Beaconing Identification Integration:ded Data Exfiltration Detection Team:Security-Applied ML Elastic Security Protections Machine Learning (ML) team [elastic/sec-applied-ml] Integration:pad Privileged Access Detection labels Jul 6, 2026
@infra-vault-gh-plugin-prod

infra-vault-gh-plugin-prod Bot commented Jul 6, 2026

Copy link
Copy Markdown

Pinging @elastic/sec-applied-ml (Team:Security-Applied ML)

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

susan-shu-c
susan-shu-c reviewed Jul 6, 2026
View reviewed changes

@susan-shu-c susan-shu-c left a comment •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you provide some more info on the PR on the logs-endpoint.events.network-*,logs-network_traffic.* sources (like link to doc on readme?) in case users are wondering where they're from

@jmcarlock

jmcarlock commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Looks good! I suggest updating the transform tables in the readmes to reflect the new transform source indices.

@mergify

mergify Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

@andrewkroh andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jul 6, 2026
@kapral18 kapral18 requested a review from Copilot July 6, 2026 20:34
Copilot AI reviewed Jul 6, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates several ML packages’ Elasticsearch transforms to read from more specific source index patterns (data streams) instead of logs-*, and bumps package versions + changelogs accordingly.

Changes:

  • Scoped transform source.index patterns to integration-specific data streams (Okta, Elastic Defend endpoint events, System/Windows security sources, Network Packet Capture).
  • Bumped package versions for pad, lmd, ded, and beaconing, and updated transform destination/pipeline versioned IDs where applicable.
  • Updated package READMEs and changelog.yml entries to reflect the new source index patterns.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
packages/pad/manifest.yml Bumps PAD package version to 2.3.0.
packages/pad/elasticsearch/transform/pivot_transform_win_privilege_list_ea/transform.yml Narrows Windows privilege transform source index; updates dest index + fleet transform version.
packages/pad/elasticsearch/transform/pivot_transform_okta_sessions_ea/transform.yml Narrows Okta sessions transform source index; updates dest index + fleet transform version.
packages/pad/docs/README.md Documents updated PAD transform source index patterns.
packages/pad/changelog.yml Adds PAD 2.3.0 changelog entry.
packages/lmd/manifest.yml Bumps LMD package version to 3.3.0.
packages/lmd/elasticsearch/transform/pivot_transform_ea/transform.yml Narrows LMD transform source index; updates fleet transform version.
packages/lmd/docs/README.md Documents updated LMD transform source index pattern.
packages/lmd/changelog.yml Adds LMD 3.3.0 changelog entry.
packages/ded/manifest.yml Bumps DED package version to 3.3.0.
packages/ded/elasticsearch/transform/pivot_transform_ea/transform.yml Narrows DED transform source index; updates dest index, pipeline, and fleet transform version.
packages/ded/docs/README.md Documents updated DED transform source index patterns.
packages/ded/changelog.yml Adds DED 3.3.0 changelog entry.
packages/beaconing/manifest.yml Bumps Beaconing package version to 1.7.0.
packages/beaconing/elasticsearch/transform/pivot_transform/transform.yml Narrows Beaconing transform source index; updates dest index, pipeline, and fleet transform version.
packages/beaconing/docs/README.md Documents updated Beaconing transform source index patterns.
packages/beaconing/changelog.yml Adds Beaconing 1.7.0 changelog entry.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread packages/pad/elasticsearch/transform/pivot_transform_win_privilege_list_ea/transform.yml
Comment thread packages/pad/docs/README.md
Comment thread packages/beaconing/elasticsearch/transform/pivot_transform/transform.yml Outdated
@sodhikirti07 sodhikirti07 requested a review from kapral18 July 6, 2026 20:58
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

✅ All changelog entries have the correct PR link.

@infra-vault-gh-plugin-prod

infra-vault-gh-plugin-prod Bot commented Jul 6, 2026

Copy link
Copy Markdown

💚 Build Succeeded

History

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Security-ML Integrations Test Pipeline — Integrations CI (run #3)

🟢 All packages passed

Package Stack version Result
beaconing 9.4.3 🟢
ded 9.4.3 🟢
lmd 9.4.3 🟢
pad 9.4.3 🟢

@sodhikirti07 sodhikirti07 merged commit b0870eb into main Jul 7, 2026
11 checks passed
@sodhikirti07 sodhikirti07 deleted the update-transform-queried-indices branch July 7, 2026 01:08
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Package beaconing - 1.7.0 containing this change is available at https://epr.elastic.co/package/beaconing/1.7.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Package ded - 3.3.0 containing this change is available at https://epr.elastic.co/package/ded/3.3.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Package lmd - 3.3.0 containing this change is available at https://epr.elastic.co/package/lmd/3.3.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Package pad - 2.3.0 containing this change is available at https://epr.elastic.co/package/pad/2.3.0/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments
@kapral18 kapral18 kapral18 approved these changes
@jmcarlock jmcarlock jmcarlock approved these changes
@susan-shu-c susan-shu-c susan-shu-c approved these changes

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. enhancement New feature or request Integration:beaconing Network Beaconing Identification Integration:ded Data Exfiltration Detection Integration:lmd Lateral Movement Detection Integration:pad Privileged Access Detection Team:Security-Applied ML Elastic Security Protections Machine Learning (ML) team [elastic/sec-applied-ml]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@sodhikirti07 @jmcarlock @kapral18 @susan-shu-c @andrewkroh