Skip to content

Update security ML jobs to include MITRE ATT&CK framework tactics and technique metdata#19220

Merged
jmcarlock merged 6 commits into
elastic:mainfrom
ymao1:update-ml-jobs-with-threat-categories
Jun 2, 2026
Merged

Update security ML jobs to include MITRE ATT&CK framework tactics and technique metdata#19220
jmcarlock merged 6 commits into
elastic:mainfrom
ymao1:update-ml-jobs-with-threat-categories

Conversation

@ymao1

@ymao1 ymao1 commented May 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed commit message

Adding MITRE ATT&CK framework tactic and technique codes to the custom_settings field for all security ML jobs. These mappings were derived from the prebuilt detection ML rules that include the mapping within the rule definitions. Techniques and subtechniques are stored in the same array.

Kibana PR here: elastic/kibana#271300

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@github-actions

This comment has been minimized.

Sign in to view
@github-actions

This comment has been minimized.

Sign in to view
@andrewkroh andrewkroh added Integration:problemchild Living off the Land Attack Detection Integration:ded Data Exfiltration Detection Integration:dga Domain Generation Algorithm Detection Integration:lmd Lateral Movement Detection Integration:pad Privileged Access Detection labels May 26, 2026
@github-actions

This comment has been minimized.

Sign in to view
@github-actions

github-actions Bot commented May 26, 2026

Copy link
Copy Markdown
Contributor

TL;DR

Check integrations ded failed while building the package because the transform manifest references a versioned ingest pipeline name that does not exist in the built package (3.1.0-ml_ded_ingest_pipeline inside package version 3.1.1). Update the transform’s versioned references to match the package version bump.

Remediation

  • In packages/ded/elasticsearch/transform/pivot_transform_ea/transform.yml, update all version-coupled values from 3.1.0 to 3.1.1 (at minimum dest.pipeline; also keep dest.index and _meta.fleet_transform_version aligned with the package version convention).
  • Re-run .buildkite/scripts/test_one_package.sh packages/ded origin/main <commit_sha> (or the equivalent local elastic-package check) to confirm package build resolves transform manifests successfully.
Investigation details

Root Cause

The transform definition uses a hard-coded versioned pipeline name:

  • packages/ded/elasticsearch/transform/pivot_transform_ea/transform.yml:9 → pipeline: 3.1.0-ml_ded_ingest_pipeline

Buildkite attempted to build package ded/3.1.1, so it looked for 3.1.0-ml_ded_ingest_pipeline.yml in that build output and failed because that filename/version does not exist for the bumped package version.

Evidence

Error: checking package failed: building package failed: resolving transform manifests failed:
failed reading transform definition file ".../packages/ded/3.1.1/elasticsearch/transform/pivot_transform_ea/transform.yml":
destination ingest pipeline file 3.1.0-ml_ded_ingest_pipeline.yml not found: incorrect version used in pipeline or unknown pipeline

Verification

  • Not run in this environment; diagnosis based on Buildkite failure log and repository source inspection.

Follow-up

  • After fixing transform.yml, scan packages/ded/ for any remaining 3.1.0 literals that should track the package version to avoid the same class of failure on the next patch bump.

Note

🔒 Integrity filter blocked 3 items

The following items were blocked because they don't meet the GitHub integrity level.

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

@ymao1

ymao1 commented Jun 1, 2026

Copy link
Copy Markdown
Contributor Author

@elasticmachine merge upstream

@ymao1 ymao1 mentioned this pull request Jun 1, 2026
Merged
@elasticmachine

elasticmachine commented Jun 1, 2026

Copy link
Copy Markdown

💚 Build Succeeded

History

@ymao1 ymao1 marked this pull request as ready for review June 1, 2026 18:21
@ymao1 ymao1 requested review from a team as code owners June 1, 2026 18:21
sodhikirti07
sodhikirti07 approved these changes Jun 1, 2026
View reviewed changes

@sodhikirti07 sodhikirti07 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ymao1 Changes looks good to me

@jmcarlock

jmcarlock commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

LGTM! All ITP tests pass.

@andrewkroh andrewkroh added the Team:Security-Applied ML Elastic Security Protections Machine Learning (ML) team [elastic/sec-applied-ml] label Jun 1, 2026
@infra-vault-gh-plugin-prod

infra-vault-gh-plugin-prod Bot commented Jun 1, 2026

Copy link
Copy Markdown

Pinging @elastic/sec-applied-ml (Team:Security-Applied ML)

ymao1 added a commit to elastic/kibana that referenced this pull request Jun 2, 2026
@ymao1 @elasticmachine
Update security ML jobs to include MITRE ATT&CK framework tactics and…
b124bc0 
… technique metdata (#271300)

## Summary

Adding MITRE ATT&CK framework tactic and technique codes to the
`custom_settings` field for all security ML jobs. These mappings were
derived from the prebuilt detection ML rules that include the mapping
within the rule definitions. Techniques and subtechniques are stored in
the same array.

Integrations PR here: elastic/integrations#19220

---------

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
@jmcarlock jmcarlock merged commit 23935b1 into elastic:main Jun 2, 2026
10 checks passed
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Package ded - 3.1.1 containing this change is available at https://epr.elastic.co/package/ded/3.1.1/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Package dga - 3.0.1 containing this change is available at https://epr.elastic.co/package/dga/3.0.1/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Package lmd - 3.1.2 containing this change is available at https://epr.elastic.co/package/lmd/3.1.2/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Package pad - 2.1.1 containing this change is available at https://epr.elastic.co/package/pad/2.1.1/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Package problemchild - 3.0.2 containing this change is available at https://epr.elastic.co/package/problemchild/3.0.2/

@ymao1 ymao1 mentioned this pull request Jul 8, 2026
Merged
ymao1 added a commit to elastic/kibana that referenced this pull request Jul 9, 2026
@ymao1 @kibanamachine
[Entity Analytics] Showing warning for missing MITRE tactics in anoma…
08e163a 
…ly section on entity flyout (#277095)

## Summary

We updated all of the Kibana EA ML jobs to include MITRE ATT&CK mappings
in #271300 and
elastic/integrations#19220. For the jobs in the
integrations repo, the user needs the latest version of the integration
in order to have the mappings. This adds a check to determine whether
any integrations ML configs are missing those mappings and shows a
warning with a link to the integrations page.

 
<img width="936" height="242" alt="Screenshot 2026-07-08 at 4 31 29 PM"
src="https://github.com/user-attachments/assets/a6e1ad7a-b867-467c-9f57-f8b9f85c59a7"
/>

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
kibanamachine added a commit to elastic/kibana that referenced this pull request Jul 9, 2026
@kibanamachine @ymao1 @elasticmachine
[9.5] [Entity Analytics] Showing warning for missing MITRE tactics in…
4e4f470 
… anomaly section on entity flyout (#277095) (#277192)

# Backport

This will backport the following commits from `main` to `9.5`:
- [[Entity Analytics] Showing warning for missing MITRE tactics in
anomaly section on entity flyout
(#277095)](#277095)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Ying
Mao","email":"ying.mao@elastic.co"},"sourceCommit":{"committedDate":"2026-07-09T13:16:36Z","message":"[Entity
Analytics] Showing warning for missing MITRE tactics in anomaly section
on entity flyout (#277095)\n\n## Summary\n\nWe updated all of the Kibana
EA ML jobs to include MITRE ATT&CK mappings\nin
#271300
and\nhttps://github.com/elastic/integrations/pull/19220. For the jobs in
the\nintegrations repo, the user needs the latest version of the
integration\nin order to have the mappings. This adds a check to
determine whether\nany integrations ML configs are missing those
mappings and shows a\nwarning with a link to the integrations page.\n\n
\n<img width=\"936\" height=\"242\" alt=\"Screenshot 2026-07-08 at 4 31
29 PM\"\nsrc=\"https://github.com/user-attachments/assets/a6e1ad7a-b867-467c-9f57-f8b9f85c59a7\"\n/>\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"08e163a2166ad369d1e0a128c4ab5ee6020cd801","branchLabelMapping":{"^v9.6.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Entity
Analytics","backport:version","v9.5.0","v9.6.0"],"title":"[Entity
Analytics] Showing warning for missing MITRE tactics in anomaly section
on entity
flyout","number":277095,"url":"https://github.com/elastic/kibana/pull/277095","mergeCommit":{"message":"[Entity
Analytics] Showing warning for missing MITRE tactics in anomaly section
on entity flyout (#277095)\n\n## Summary\n\nWe updated all of the Kibana
EA ML jobs to include MITRE ATT&CK mappings\nin
#271300
and\nhttps://github.com/elastic/integrations/pull/19220. For the jobs in
the\nintegrations repo, the user needs the latest version of the
integration\nin order to have the mappings. This adds a check to
determine whether\nany integrations ML configs are missing those
mappings and shows a\nwarning with a link to the integrations page.\n\n
\n<img width=\"936\" height=\"242\" alt=\"Screenshot 2026-07-08 at 4 31
29 PM\"\nsrc=\"https://github.com/user-attachments/assets/a6e1ad7a-b867-467c-9f57-f8b9f85c59a7\"\n/>\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"08e163a2166ad369d1e0a128c4ab5ee6020cd801"}},"sourceBranch":"main","suggestedTargetBranches":["9.5"],"targetPullRequestStates":[{"branch":"9.5","label":"v9.5.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.6.0","branchLabelMappingKey":"^v9.6.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/277095","number":277095,"mergeCommit":{"message":"[Entity
Analytics] Showing warning for missing MITRE tactics in anomaly section
on entity flyout (#277095)\n\n## Summary\n\nWe updated all of the Kibana
EA ML jobs to include MITRE ATT&CK mappings\nin
#271300
and\nhttps://github.com/elastic/integrations/pull/19220. For the jobs in
the\nintegrations repo, the user needs the latest version of the
integration\nin order to have the mappings. This adds a check to
determine whether\nany integrations ML configs are missing those
mappings and shows a\nwarning with a link to the integrations page.\n\n
\n<img width=\"936\" height=\"242\" alt=\"Screenshot 2026-07-08 at 4 31
29 PM\"\nsrc=\"https://github.com/user-attachments/assets/a6e1ad7a-b867-467c-9f57-f8b9f85c59a7\"\n/>\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"08e163a2166ad369d1e0a128c4ab5ee6020cd801"}}]}]
BACKPORT-->

---------

Co-authored-by: Ying Mao <ying.mao@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jmcarlock jmcarlock jmcarlock approved these changes
@sodhikirti07 sodhikirti07 sodhikirti07 approved these changes
+1 more reviewer
@yuriShafet yuriShafet yuriShafet approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

Integration:ded Data Exfiltration Detection Integration:dga Domain Generation Algorithm Detection Integration:lmd Lateral Movement Detection Integration:pad Privileged Access Detection Integration:problemchild Living off the Land Attack Detection Team:Security-Applied ML Elastic Security Protections Machine Learning (ML) team [elastic/sec-applied-ml]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@ymao1 @elasticmachine @jmcarlock @yuriShafet @sodhikirti07 @andrewkroh