-
Notifications
You must be signed in to change notification settings - Fork 90
create SOL detection rule documentation #2989
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
alexane-bougeardbebin-sekoia
wants to merge
14
commits into
main
Choose a base branch
from
Sol-detection-rules
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
3acae7b
create SOL detection rule documentation
alexane-bougeardbebin-sekoia d4ecd51
Add sol_detection_rule.md documentation file
alexane-bougeardbebin-sekoia 58ef783
Add guide for creating SOL detection and Event Drop rules
alexane-bougeardbebin-sekoia 338df7c
Add intake/asset event drop. Document similarity, lag management and …
Sengthay 7ca355d
Fix link
Sengthay 1147365
Invalid path
Sengthay 4535391
Path fixes
Sengthay 4a56a28
Revise SOL detection rule creation documentation
alexane-bougeardbebin-sekoia c95ec9d
Rename similarity strategy section to event grouping
alexane-bougeardbebin-sekoia b1ae553
Enhance documentation on SOL detection similarity strategies
alexane-bougeardbebin-sekoia 8275ccf
Add SOL detection rules and related documentation
alexane-bougeardbebin-sekoia 7a8d33c
Remove duplicate SOL detection rules and Event Drop rule
alexane-bougeardbebin-sekoia 2ea608a
Refactor SOL detection rules section in mkdocs.yml
alexane-bougeardbebin-sekoia c0c43a0
Remove irrelevant parts
Sengthay File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,83 @@ | ||
| # Create an Event Drop rule from an intake or asset | ||
|
|
||
| You can create an Event Drop detection rule directly from an intake or an asset, without starting from a blank rule in the catalog. The rule creation panel opens with the Event Drop pattern selected and the SOL pattern pre-filled with a condition on the source, so you only set the schedule and create the rule. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - You have permissions to create detection rules. | ||
| - You are on the details page of the intake or asset you want to monitor for missing data. | ||
|
|
||
| ## Start from an intake | ||
|
|
||
| To create an Event Drop rule that monitors a specific intake: | ||
|
|
||
| 1. Open the details page of the intake. | ||
| 2. Open the more options menu (the `...` button) in the intake header. | ||
| 3. Select **Create event drop alerting**. | ||
|
|
||
| The **Create new rule** panel opens with the **Event Drop** pattern selected. The rule name is set to `Event drop for intake "<intake name>"`, and the pattern targets the intake UUID: | ||
|
|
||
| ``` | ||
| event_telemetry | ||
| | where timestamp between (?time.start .. ?time.end) | ||
| | aggregate count() by bin(timestamp, 2h) | ||
| | where intake_uuid == "<intake uuid>" | ||
| | where count < 500 | ||
| ``` | ||
|
|
||
| 4. Adjust the time bin and the threshold if the defaults do not match the intake's expected volume. | ||
| 5. Set the schedule and complete any remaining fields. | ||
| 6. Click **Create**. | ||
|
|
||
| {: style="max-width:100%"} | ||
|
|
||
| {: style="max-width:100%"} | ||
|
|
||
| ## Start from an asset | ||
|
|
||
| To create an Event Drop rule that monitors a specific asset: | ||
|
|
||
| 1. Open the details page of the asset. | ||
| 2. Click **Configure an event drop alert**. | ||
|
|
||
| The **Create new rule** panel opens with the **Event Drop** pattern selected. The rule name is set to `Event drop for asset "<asset name>"`, and the pattern targets the asset UUID: | ||
|
|
||
| ``` | ||
| asset_telemetry | ||
| | where timestamp between (?time.start .. ?time.end) | ||
| | aggregate count() by bin(timestamp, 1d) | ||
| | where asset_uuid == "<asset uuid>" | ||
| | where count < 1000000 | ||
| ``` | ||
|
|
||
| 3. Adjust the time bin and the threshold if the defaults do not match the asset's expected volume. | ||
| 4. Set the schedule and complete any remaining fields. | ||
| 5. Click **Create**. | ||
|
|
||
| {: style="max-width:100%"} | ||
|
|
||
| {: style="max-width:100%"} | ||
|
|
||
| ## What gets pre-filled | ||
|
|
||
| The entry point determines the rule name, the datasource and the field used in the condition: | ||
|
|
||
| | Source | Entry point | Rule name | Datasource | Condition field | | ||
| |---|---|---|---|---| | ||
| | Intake | **Create event drop alerting** in the intake more options menu | `Event drop for intake "<name>"` | `event_telemetry` | `intake_uuid` | | ||
| | Asset | **Configure an event drop alert** button on the asset page | `Event drop for asset "<name>"` | `asset_telemetry` | `asset_uuid` | | ||
|
|
||
| Both pre-filled patterns also bound the query with `?time.start` and `?time.end`, the time conditions required by every SOL rule. | ||
|
|
||
| !!! note "Tune the defaults" | ||
| The pre-filled threshold and time bin are starting points. Set them to values that reflect the normal telemetry volume of the intake or asset, so the rule alerts only when the data drop is meaningful. | ||
|
|
||
| ## Complete and create the rule | ||
|
|
||
| Once the panel is open, scheduling and the remaining fields work exactly as in the standard creation flow. For the full description of each field and the run schedule, see [Create a SOL detection rule](/xdr/features/detect/create_sol_detection_rule.md). | ||
|
|
||
| ## Related articles | ||
|
|
||
| * [Create a SOL detection rule](/xdr/features/detect/create_sol_detection_rule.md): Step-by-step instructions to configure and activate a SOL or Event Drop detection rule from the catalog. | ||
| * [SOL detection rules](/xdr/features/detect/sol_detection_rule.md): Conceptual overview of the SOL and Event Drop detection patterns, how they work, and when to use them. | ||
| * [Lag management for SOL detection rules](/xdr/features/detect/sol_detection_lag_management.md): How rules handle late-arriving events through a fixed run overlap and deduplication. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,109 @@ | ||
| # Create a SOL detection rule | ||
|
|
||
| SOL detection rules let you operationalize your SOL investigation queries as standing detections. This article walks you through creating a SOL detection rule or an Event Drop rule from the rule catalog. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - You have an active Sekoia.io account with permissions to create detection rules. | ||
| - You are familiar with SOL query syntax or have an existing query you want to reuse. | ||
| - For Event Drop rules: you have identified the source to monitor (all incoming events, a specific intake, or a specific asset). | ||
|
|
||
| ## Create a SOL detection rule | ||
|
|
||
| ### Step 1: Open the rule creation panel | ||
|
|
||
| Navigate to **Detection > Rules catalog**, then click **+ New rule**. | ||
|
|
||
| ### Step 2: Select the SOL detection pattern | ||
|
|
||
| In the **Specify your detection patterns** section, select the **SOL** tab. SOL sits alongside the Sigma, CTI, Anomaly and Event Drop patterns. | ||
|
|
||
| {: style="max-width:100%"} | ||
|
|
||
| ### Step 3: Write your SOL query | ||
|
|
||
| In the **Query** field, enter your SOL query. This is the query that will execute on each scheduled run. | ||
|
|
||
| !!! warning "Time conditions are required" | ||
| Whatever the datasource, your SOL pattern must bound the query with a start and an end time using the `?time.start` and `?time.end` placeholders. The platform replaces them with each run's time window. A rule whose pattern omits these conditions cannot be saved or executed. | ||
|
|
||
| ``` | ||
| events | ||
| | where timestamp between (?time.start .. ?time.end) | ||
| | limit 100 | ||
| ``` | ||
|
|
||
| !!! tip "Reuse an existing investigation" | ||
| You can copy a SOL query directly from a Notebook or investigation. Before saving, make sure it bounds the time range with `?time.start` and `?time.end`. | ||
|
|
||
| !!! note "Result cap" | ||
| The platform automatically appends a 10,000-row limit to your query. You do not need to add it manually. This is a temporary protective limit and may be raised in the future. | ||
|
|
||
| ### Step 4: Set the run schedule | ||
|
|
||
| In the **Run query every** field, enter a value and select the time unit: minutes, hours, or days. The rule then runs at that frequency. | ||
|
|
||
| !!! warning "Minimum schedule interval" | ||
| The minimum accepted schedule is every 5 minutes. | ||
|
|
||
| !!! note "Late-arriving events" | ||
| You do not need to widen the schedule to account for ingestion lag. The platform automatically extends each run 5 minutes into the past to catch events that arrive after their timestamp, and deduplicates overlapping results so the same alert is not raised twice. See [Lag management for SOL detection rules](/xdr/features/detect/sol_detection_lag_management.md). | ||
|
|
||
| ### Step 5: Configure the similarity strategy (optional) | ||
|
|
||
| By default, all of a rule's matches are grouped into a single alert: each new match increments that alert's occurrence counter instead of creating a separate alert. You do not need to configure anything to get this behavior. | ||
|
|
||
| To split matches into separate alerts, configure a similarity strategy by selecting one or more fields. Matches that share the same values for those fields are grouped into one alert, and matches with different values are raised as separate alerts. You can also group interchangeable fields, written as `[source.ip, destination.ip]`, so their values can be swapped when matching. | ||
|
|
||
| !!! note "Similarity for SOL rules" | ||
| The fields available in the **Similarity strategy** selector depend on your query. SOL detection rules apply no default similarity strategy based on the datasource, because a SOL rule can query any datasource. See [Alert similarity for SOL detection rules](/xdr/features/detect/sol_detection_similarity.md) to understand the default grouping and when to configure a strategy. | ||
|
|
||
| ### Step 6: Complete the rule metadata | ||
|
|
||
| Fill in the remaining rule fields: | ||
|
|
||
| | Field | Description | | ||
| |---|---| | ||
| | **Name** | A descriptive name for the rule | | ||
| | **Severity** | The severity level assigned to generated alerts | | ||
| | **Effort** | Estimated analyst effort to triage an alert from this rule | | ||
| | **Threats** | Associated threat categories or techniques (MITRE ATT&CK) | | ||
| | **Description** | A summary of what the rule detects and why | | ||
| | **Tags** | Optional labels to help organize and filter rules | | ||
|
|
||
| ### Step 7: Save and activate the rule | ||
|
|
||
| Click **Create** and enable the rule to start executing on the defined schedule. | ||
|
|
||
| ## Create an Event Drop rule | ||
|
|
||
| Event Drop rules follow the same creation flow as SOL rules, with the following differences: | ||
|
|
||
| - Select the **Event Drop** detection pattern at step 2. | ||
| - In the **Query** field, enter the SOL pattern targeting the events, intake or asset you want to monitor for missing data. Like any SOL pattern, it must bound the query with `?time.start` and `?time.end`: | ||
|
|
||
| ``` | ||
| events | ||
| | where timestamp between (?time.start .. ?time.end) | ||
| | aggregate count() by bin(timestamp, 1d) | ||
| | where count < 1000000 | ||
| ``` | ||
|
|
||
| - The **Threats** and security alert classification fields are not available for Event Drop rules, as they apply to threat detection rather than availability monitoring. The **Effort** field remains available. | ||
|
|
||
| !!! tip "Create from an intake or asset" | ||
| You can start Event Drop rule creation directly from an intake or an asset, with the pattern pre-filled on the source UUID. See [Create an Event Drop rule from an intake or asset](/xdr/features/detect/create_event_drop_from_source.md). | ||
|
|
||
| ## What happens next | ||
|
|
||
| Once saved, your rule begins executing immediately according to its schedule. When the query returns results, an alert is created. You can open any alert generated by a SOL or Event Drop rule to access the **Results** tab, which shows the raw query results for that execution. | ||
|
|
||
| {: style="max-width:100%"} | ||
|
|
||
| ## Related articles | ||
|
|
||
| * [SOL detection rules](/xdr/features/detect/sol_detection_rule.md): Conceptual overview of the SOL and Event Drop detection patterns, how they work, and when to use them. | ||
| * [Create an Event Drop rule from an intake or asset](/xdr/features/detect/create_event_drop_from_source.md): Create a pre-filled Event Drop rule directly from an intake or asset details page. | ||
| * [Alert similarity for SOL detection rules](/xdr/features/detect/sol_detection_similarity.md): How SOL rules group matches into alerts, the group-all default, and how to configure a similarity strategy. | ||
| * [Lag management for SOL detection rules](/xdr/features/detect/sol_detection_lag_management.md): How rules handle late-arriving events through a fixed run overlap and deduplication. | ||
| * [Write queries in SOL](/xdr/features/investigate/sol_how_to_guides.md): Reference guide for SOL syntax, operators, and functions. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| # Lag management for SOL detection rules | ||
|
|
||
| Lag management is the built-in behavior that lets SOL detection rules evaluate events that reach the platform after their own timestamp. It combines a fixed overlap on every run with deduplication of overlapping results, so late events are still detected and the same alert is never raised twice. | ||
|
|
||
| ## Why ingestion lag affects detection | ||
|
|
||
| SOL detection rules run on a schedule and evaluate the events that fall inside each run's time window. Events do not always arrive in real time: depending on the intake, an event timestamped at a given moment can land in Sekoia several minutes later. When this ingestion lag pushes an event past the moment its window has already been evaluated, the rule never sees it and the alert is never raised, even though the rule logic is correct. | ||
|
|
||
| For example, a rule that runs every 15 minutes evaluates a window that closes at the run time. An event timestamped shortly before the window closes, but delivered to the platform a few minutes after the run, would fall outside that already-evaluated window and go undetected. | ||
|
|
||
| ## How the overlap window works | ||
|
|
||
| To catch these late events, every SOL detection run reaches back into the period the previous run already covered. Each run overlaps the previous run's window by a fixed 5 minutes, extending the start of the window 5 minutes earlier than where the last run ended. | ||
|
|
||
| Consider a rule that runs every 15 minutes: | ||
|
|
||
| | Run time | Window evaluated | | ||
| |---|---| | ||
| | 10:15 | Events timestamped 10:00 to 10:15 | | ||
| | 10:30 | Events timestamped 10:10 to 10:30 | | ||
| | 10:45 | Events timestamped 10:25 to 10:45 | | ||
|
|
||
| The 10:30 run reaches back to 10:10, so it re-covers the final 5 minutes of the 10:15 run. An event timestamped 10:14 that only arrives in Sekoia at 10:21 is missed by the 10:15 run, but the 10:30 run still evaluates it because its window starts at 10:10. The detection fires. | ||
|
|
||
| !!! note "Automatic behavior" | ||
| The 5-minute overlap is applied to every SOL detection run. It is managed by the platform and is not configurable on the rule. | ||
|
|
||
| ## Deduplication in the overlap window | ||
|
|
||
| The overlap that recovers late events also re-reads events that arrived on time and were already evaluated by the previous run. Left unchecked, those events would be evaluated twice and could raise the same alert twice. | ||
|
|
||
| To prevent this, the engine deduplicates the overlapping results before any alert is created. Results that the previous run already processed are dropped, so only genuinely new matches reach alert creation. | ||
|
|
||
| Using the same example, an event timestamped 10:12 that arrived on time was already evaluated by the 10:15 run. It falls inside the 10:10 to 10:15 overlap that the 10:30 run re-reads, so the 10:30 run discards it during deduplication and does not raise a second alert for it. | ||
|
|
||
| !!! note "Different from alert similarity" | ||
| Overlap deduplication is a row-level safeguard internal to rule execution: it removes results the rule has already evaluated. It is not the same as alert similarity, which groups similar events from different matches into a single alert as occurrences. The two work at different layers. For how matches are grouped into an alert, see [Alert similarity](/xdr/features/investigate/alerts.md#alert-similarity). | ||
|
|
||
| ## Related articles | ||
|
|
||
| * [SOL detection rules](/xdr/features/detect/sol_detection_rule.md): Conceptual overview of the SOL and Event Drop detection patterns, how they work, and when to use them. | ||
| * [Create a SOL detection rule](/xdr/features/detect/create_sol_detection_rule.md): Step-by-step instructions to configure and activate a SOL or Event Drop detection rule. | ||
| * [Alert similarity](/xdr/features/investigate/alerts.md#similarity-strategies): How the platform groups similar events into a single alert as occurrences. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| # SOL detection rules | ||
|
|
||
| SOL detection rules let you write detection logic directly in Sekoia Operating Language (SOL), turning your investigations into active detections without any conversion step. This article explains what SOL detection rules are, how they work, and when to use them. | ||
|
|
||
| !!! note "Early Access" | ||
| This feature is currently in Early Access and is only available for Beta testers. Sekoia.io plans to roll out this functionality to all environments soon. | ||
|
|
||
| ## What are SOL detection rules | ||
|
|
||
| SOL detection rules are detection rules that execute a SOL query on a defined schedule and generate alerts when the query returns results. Unlike Sigma-based rules, they let you reuse your existing SOL investigation queries as-is, closing the gap between investigation and detection in a single workflow. | ||
|
|
||
| A detection rule uses a single detection pattern. When you create a rule, you choose one pattern type in the **Specify your detection patterns** area, where SOL and Event Drop sit alongside Sigma, CTI and Anomaly. SOL and Event Drop are the two SOL-powered patterns. | ||
|
|
||
| ## SOL-powered detection patterns | ||
|
|
||
| Two detection patterns run on SOL: | ||
|
|
||
| | Detection pattern | Purpose | | ||
| | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | ||
| | **SOL** | Executes a SOL query on a schedule and raises a security alert when results are returned | | ||
| | **Event Drop** | Monitors events, intakes or assets for missing or dropped events and alerts you when expected data stops flowing (relies on SOL query) | | ||
|
|
||
| {: style="max-width:100%"} | ||
|
|
||
| ## How SOL detection rules work | ||
|
|
||
| When you create a SOL detection rule, the platform executes your query at the frequency you define. Each execution is logged. Each run injects its time window into the query through the `?time.start` and `?time.end` placeholders, which your SOL pattern must include. See [Create a SOL detection rule](/xdr/features/detect/create_sol_detection_rule.md) for the query requirements. | ||
|
|
||
| By default, all of a rule's matches are grouped into a single alert, and each new match increments that alert's occurrence counter. You can configure a similarity strategy to split matches into separate alerts by field value instead. For details, see [Alert similarity for SOL detection rules](/xdr/features/detect/sol_detection_similarity.md). | ||
|
|
||
| !!! note "Result cap" | ||
| SOL detection rules automatically cap query results at 10,000 rows per execution. Queries that would return more rows are truncated at that limit. This is a temporary protective limit and may be raised in the future. | ||
|
|
||
| Rules start executing immediately after creation. Results are accessible directly from the alert view. | ||
|
|
||
| To handle events that reach the platform after their timestamp, each run overlaps the previous run's window by a fixed 5 minutes, and overlapping results are deduplicated so the same alert is not raised twice. For details, see [Lag management for SOL detection rules](/xdr/features/detect/sol_detection_lag_management.md). | ||
|
|
||
| ## Alerts generated by SOL detection rules | ||
|
|
||
| Alerts generated by SOL and Event Drop rules include a dedicated **Results** tab that displays the raw, paginated results returned by the query for that execution. | ||
|
|
||
| !!! note "Event Drop alerts" | ||
| Alerts generated by Event Drop rules do not include threat intelligence enrichment or a Kill Chain phase, as these do not apply to availability monitoring. | ||
|
|
||
| ## When to use SOL detection rules | ||
|
|
||
| - You have already written a SOL query during an investigation and want to operationalize it as a standing detection without rewriting it in Sigma. | ||
| - You want to detect patterns or conditions that are difficult to express in standard Sigma syntax. | ||
| - You need to monitor events, intakes or assets for data availability issues using an Event Drop pattern. | ||
|
|
||
| ## Related articles | ||
|
|
||
| * [Create a SOL detection rule](/xdr/features/detect/create_sol_detection_rule.md): Step-by-step instructions to configure and activate a SOL or Event Drop detection rule. | ||
| * [Create an Event Drop rule from an intake or asset](/xdr/features/detect/create_event_drop_from_source.md): Create a pre-filled Event Drop rule directly from an intake or asset details page. | ||
| * [Alert similarity for SOL detection rules](/xdr/features/detect/sol_detection_similarity.md): How SOL rules group matches into alerts, the group-all default, and how to configure a similarity strategy. | ||
| * [Lag management for SOL detection rules](/xdr/features/detect/sol_detection_lag_management.md): How rules handle late-arriving events through a fixed run overlap and deduplication. | ||
| * [Write queries in SOL](/xdr/features/investigate/sol_how_to_guides.md): Reference guide for SOL syntax, operators, and functions. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.