Skip to content

[Recorded Future] Add Identity Detections Support #19406

Description

@jamiehynds

Summary

Extend the existing Recorded Future integration to ingest identity exposure detections from the Recorded Future Identity API, and add a dedicated dashboard for analyst triage and monitoring.

Background

The current Recorded Future integration ingests threat intelligence (alerts, indicators, etc.) but does not pull data from the Identity API. Recorded Future's Identity module surfaces compromised credentials discovered in breach dumps and stealer malware logs — a high-signal data source for detecting account compromise before attackers exploit stolen credentials.

Identity detections come in three types:

  • Workforce — credentials for the organization's own login portals (e.g., SSO, VPN, webmail)
  • External — employee credentials found on third-party websites and services outside the organization
  • VIP — detections scoped to high-value identities

Each detection includes the exposed identity (email/username), password metadata (hash algorithm, complexity properties, cleartext hint), the targeted authorization service (URL, domain, technology stack such as Okta SSO or VPN), any stolen browser cookies, malware family attribution, and breach/dump source metadata including exfiltration date.

We have a Recorded Future instance populated with identity detections available for development and testing.

API Details

Endpoint: POST https://api.recordedfuture.com/identity/detections

Authentication: API key via X-RFToken header (existing integration auth mechanism)

Key request parameters:

Parameter Type Description
organization_id string[] (≤100) Scope to specific org IDs (uhash format)
include_enterprise_level boolean Include enterprise-wide detections across all orgs
filter.novel_only boolean Only return newly discovered exposures
filter.malware_only boolean Only return stealer malware detections
filter.domains string[] (≤100) Filter by target domain
filter.detection_types enum[] Workforce, External, Vip
filter.source_type enum[] MalwareLogs, DatabaseDumps, MalwareCombolists, DatabaseCombolists
filter.authorization_technology object Filter by auth tech (e.g., Okta SSO) by ID or name
filter.cookies enum Cookies or UnexpiredCookies
filter.created.gte / .lt date-time Date range filter
limit int (0–1000) Max results per page
offset string Cursor token for pagination

Key response fields per detection:

Field Description
id Unique detection ID
novel Whether this is a newly discovered exposure
type Workforce, External, or Vip
source_type MalwareLogs, DatabaseDumps, MalwareCombolists, DatabaseCombolists
subject Compromised identity (email or username)
password.type e.g., Cleartext
password.hashes[] Array of {algorithm, hash} or {algorithm, hash_prefix}
password.properties[] Complexity flags (e.g., UpperCase, AtLeast12Characters)
password.cleartext_hint Partial cleartext hint (e.g., Pass)
authorization_service.url Login URL where credential was exposed
authorization_service.domain Target domain
authorization_service.technology[] Auth tech stack (id, name, category)
cookies[] Stolen browser cookies (dns, name, expiration, secure, http)
malware_family.id / .name Responsible malware (e.g., Vidar Stealer)
dump.name Breach dump name
dump.source Dump source ID
dump.downloaded When RF downloaded the dump
dump.compromise.exfiltration_date When credentials were exfiltrated
dump.compromise.os / .os_username Victim machine details
dump.location Geographic location of compromised machine
created When the detection was created in RF

Related endpoints to be aware of (not in scope for this issue):

  • GET /identity/detections/{id} — lookup a single detection by ID
  • POST /identity/search — search exposures by specific identity attributes

Proposed Solution

1. New data stream: recorded_future.identity

Add a new data stream to pull identity detections from the RF Identity API on a configurable polling interval (default: every hour). Use filter.created.gte with a checkpoint timestamp to only fetch new detections since the last run, falling back to novel_only: true as an option for users who want deduplication by novelty.

ECS field mappings (key fields):

RF Field ECS / Custom Field
subject user.email or user.name
type recordedfuture.identity.detection_type
source_type recordedfuture.identity.source_type
novel recordedfuture.identity.novel
authorization_service.url url.full
authorization_service.domain url.domain
authorization_service.technology[].name recordedfuture.identity.authorization_technology
malware_family.name threat.software.name
malware_family.id threat.software.id
dump.compromise.exfiltration_date recordedfuture.identity.exfiltration_date
dump.compromise.os host.os.name
dump.name recordedfuture.identity.dump.name
dump.source recordedfuture.identity.dump.source
cookies[].expiration recordedfuture.identity.cookies.expiration
dump.location.country.alpha2Code source.geo.country_iso_code
created @timestamp

Password hash fields should be mapped under recordedfuture.identity.password.*. Do not store password.cleartext in plaintext.

2. Integration config options

Add the following new config parameters to the integration:

  • organization_id (optional, array) — limit to specific org IDs
  • include_enterprise_level (boolean, default: false)
  • novel_only (boolean, default: false) — only ingest newly discovered exposures
  • detection_types (multi-select: Workforce, External, Vip; default: all)
  • source_types (multi-select; default: all)
  • domains (optional, array) — filter by target domain

3. Dashboard: Recorded Future Identity Exposure Overview

Add a new dashboard with the following panels:

Panel Visualization Description
Total detections Metric Count of detections in selected time range
Novel detections Metric Count where novel = true
Detections over time Bar/area chart Bucketed by @timestamp
Detection type breakdown Donut Workforce vs External vs VIP
Source type breakdown Donut MalwareLogs vs DatabaseDumps vs Combolists
Top malware families Horizontal bar By threat.software.name
Top targeted domains Data table By url.domain, with detection count
Top targeted auth technologies Data table Okta, VPN, webmail, etc.
Detections with unexpired cookies Metric High-urgency subset
Detections by country Map or table By source.geo.country_iso_code
Recent detections Events table Latest detections with subject, type, domain, malware, exfil date

The dashboard should support standard time picker and a detection_type filter control.

Acceptance Criteria

  • New data stream recorded_future.identity ingests detections from POST /identity/detections
  • Polling uses cursor-based pagination to handle responses >1000 results
  • Checkpoint-based incremental ingestion (no duplicate documents on re-run)
  • All required ECS fields mapped; custom recordedfuture.identity.* fields documented
  • Password cleartext is not stored; only hash metadata and complexity properties
  • New config parameters exposed in the integration UI with appropriate descriptions
  • Integration tested against the development RF instance with real identity detections
  • Unit tests cover pagination logic and field mapping
  • New dashboard deployed with all panels described above
  • Dashboard passes the standard Elastic dashboard review checklist
  • CHANGELOG.md updated
  • Docs updated to describe the new data stream and config options

References

Metadata

Metadata

Labels

Integration:ti_recordedfutureRecorded FutureTeam:SDE-CrestCrest developers on the Security Integrations team [elastic/sit-crest-contractors]Team:Security-Service IntegrationsSecurity Service Integrations team [elastic/security-service-integrations]enhancementNew feature or request

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions