Describe the bug
We are encountering a consistent timeout issue when running PPL monitors in Alerting. The monitor executes successfully when triggered manually (via Dev Tools), but fails when executed on schedule with a ReceiveTimeoutTransportException.
This suggests a transport-level timeout (default ~60s) that cannot be configured via cluster or PPL settings.
Error
{
"error_message": "Failed to run PPL Monitor Npy68p0Bs4ku8IPEc884, PPL Trigger m5y58p0Bs4ku8IPEPrOS: ReceiveTimeoutTransportException[[opensearch-cluster-warm-nodes-3][x.x.x.x:9300][cluster:admin/opensearch/ppl] request_id [58894985] timed out after [60040ms]]"
}
Observed Behavior
Related component
No response
To Reproduce
Create a scheduled ppl monitor that takes longer than 60 sec to execute
Expected behavior
- Scheduled executions should behave the same as manual executions
- The transport timeout should either:
- Be configurable, or
- Be aligned with PPL/query execution expectations
- Long-running PPL queries should not fail due to transport-layer timeout while still executing
Additional Details
Investigation Summary
We traced this to the use of:
org.opensearch.sql.plugin.transport.TransportPPLQueryRequest
Key findings:
TransportPPLQueryRequest does not include any timeout fieldActionRequest (parent class) also does not support timeouts- The timeout comes from the transport layer (
TransportService.sendRequest)
- The failure occurs at ~60s → likely default
TransportRequestOptions timeout
We attempted:
- Increasing all PPL-related timeouts
- Adjusting cluster settings (
search.default_search_timeout, search.cancel_after_time_interval)
- Tuning query complexity
None affected the issue.
Root Cause (Hypothesis)
The Alerting plugin likely executes PPL queries internally using a transport call similar to:
transportService.sendRequest(...)
without overriding timeout:
TransportRequestOptions.builder().withTimeout(...)
This results in:
- Default timeout (~60s)
- Hard failure even if query continues running
Why this is problematic
- Users cannot configure this timeout
- Behavior differs between:
- Manual execution
- Scheduled monitor execution
- Leads to:
- False alert failures
- Inconsistent system behavior
Suggested Fix
One of the following:
Option 1 (preferred)
Allow configuring transport timeout for PPL execution in Alerting:
- e.g. monitor-level setting:
"transport_timeout": "120s"
Option 2
Increase default timeout for PPL execution in Alerting internally
Option 3
Use a different execution mechanism that:
- Avoids strict transport timeout coupling
- Or supports async execution with polling
Option 4
Propagate timeout from monitor configuration → PPL execution layer
Impact
- Makes PPL monitors unreliable for large datasets or longer queries
- Forces users to:
- Reduce query scope artificially
- Or modify OpenSearch source code to increase timeout
Environment
- OpenSearch version: 3.5
- Alerting plugin version: 3.5
- SQL/PPL plugin version: 3.5
Request
Please provide:
- A way to configure transport timeout for PPL execution in Alerting
OR
- Increase/remove the hardcoded/default timeout behavior
Describe the bug
We are encountering a consistent timeout issue when running PPL monitors in Alerting. The monitor executes successfully when triggered manually (via Dev Tools), but fails when executed on schedule with a
ReceiveTimeoutTransportException.This suggests a transport-level timeout (default ~60s) that cannot be configured via cluster or PPL settings.
Error
{ "error_message": "Failed to run PPL Monitor Npy68p0Bs4ku8IPEc884, PPL Trigger m5y58p0Bs4ku8IPEPrOS: ReceiveTimeoutTransportException[[opensearch-cluster-warm-nodes-3][x.x.x.x:9300][cluster:admin/opensearch/ppl] request_id [58894985] timed out after [60040ms]]" }Observed Behavior
✅ Manual execution via Dev Tools:
❌ Scheduled execution (Alerting):
ReceiveTimeoutTransportExceptionRelated component
No response
To Reproduce
Create a scheduled ppl monitor that takes longer than 60 sec to execute
Expected behavior
Additional Details
Investigation Summary
We traced this to the use of:
Key findings:
TransportPPLQueryRequestdoes not include any timeout fieldActionRequest(parent class) also does not support timeoutsTransportService.sendRequest)TransportRequestOptionstimeoutWe attempted:
search.default_search_timeout,search.cancel_after_time_interval)None affected the issue.
Root Cause (Hypothesis)
The Alerting plugin likely executes PPL queries internally using a transport call similar to:
without overriding timeout:
This results in:
Why this is problematic
Suggested Fix
One of the following:
Option 1 (preferred)
Allow configuring transport timeout for PPL execution in Alerting:
Option 2
Increase default timeout for PPL execution in Alerting internally
Option 3
Use a different execution mechanism that:
Option 4
Propagate timeout from monitor configuration → PPL execution layer
Impact
Environment
Request
Please provide:
OR