Skip to content

fix(harness): raise idle watchdog default to 600s so long compaction survives - #4013

Merged
TomeHirata merged 2 commits into
mainfrom
fix/compaction-idle-watchdog
Aug 5, 2026
Merged

fix(harness): raise idle watchdog default to 600s so long compaction survives#4013
TomeHirata merged 2 commits into
mainfrom
fix/compaction-idle-watchdog

Conversation

@TomeHirata

Copy link
Copy Markdown
Contributor

Related issue

N/A

Summary

  • The per-turn idle watchdog fails a turn that emits no non-heartbeat events for its window (default 240s). It resets on every real progress event, so it's meant to catch a wedged turn — not a slow-but-healthy one.
  • Context compaction's summarizing LLM call (runtime/compaction.pysummarize_history) runs as a single long await that emits nothing until it returns. On a near-full context that call can take longer than 240s, so the idle watchdog trips it.
  • This wedges the session in a loop reported in Slack: Prompt is too long → auto-compaction → turn exceeded the 240s harness idle watchdog (run_turn emitted no events for 240s; likely a wedged LLM or tool call) → retry re-triggers the same slow compaction → repeat.
  • Fix: raise the idle watchdog default from 240s to 600s in omnigent/runtime/harnesses/_scaffold.py, giving a healthy long compaction room to finish. The HARNESS_TURN_TIMEOUT_S env knob and the absolute per-turn ceiling (HARNESS_TURN_ABSOLUTE_TIMEOUT_S, 3600s) are unchanged.

Test Plan

  • pytest tests/runtime/harnesses/test_scaffold.py -k "watchdog or idle or wedged or busy or absolute or heartbeat" — 6 passed. These drive the watchdog via HARNESS_TURN_TIMEOUT_S overrides, so they're independent of the default and confirm idle/absolute/heartbeat behavior still holds.
  • No test asserts the 240 default; the only other 240 literals in the tree are unrelated (MCP startup settle, pi-native park timeout).

Demo

N/A

Type of change

  • Bug fix
  • Feature
  • UI / frontend change
  • Refactor / chore
  • Docs
  • Test / CI
  • Breaking change

Test coverage

  • Unit tests added / updated
  • Integration tests added / updated
  • E2E tests added / updated
  • Manual verification completed
  • Existing tests cover this change
  • Not applicable

Coverage notes

Behavior is unchanged except for the default window length. Existing scaffold watchdog tests exercise idle-timeout, absolute-cap, and heartbeat-ignoring paths via explicit HARNESS_TURN_TIMEOUT_S overrides, so they cover the mechanism regardless of the default. I ran that subset locally (6 passed). No new test is added because the change is a single default constant with no new branch.

Changelog

Long context compaction no longer fails with a 240s idle-watchdog timeout on near-full sessions


This pull request and its description were written by Isaac.

…survives

The per-turn idle watchdog fails a turn that emits no non-heartbeat
events for the window. Context compaction's summarizing LLM call runs
as a single long await that emits nothing until it returns, so on a
near-full context it can exceed the 240s default and trip the watchdog.
That wedges the session in a "Prompt is too long" -> compaction ->
240s-timeout loop, since every retry re-triggers the same slow compaction.

Raise the default from 240s to 600s so a healthy long compaction has
room to finish. The HARNESS_TURN_TIMEOUT_S env knob and the absolute
ceiling are unchanged.

Co-authored-by: Isaac
Copilot AI lite review requested due to automatic review settings August 4, 2026 04:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions github-actions Bot added the size/XS Pull request size: XS label Aug 4, 2026
@omnigent-ci

omnigent-ci Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Polly AI Review

Summary

Low-risk, single-constant change: raises the idle-watchdog default from 240s to 600s in _scaffold.py, plus an explanatory comment. The reasoning is sound and matches the code — every non-heartbeat ctx.emit resets the idle deadline, so the watchdog is meant to catch a wedged turn, and a long single await (context compaction's summarize_history) genuinely emits nothing until it returns, so it can trip a too-tight window. Raising the default gives healthy compaction room without touching the absolute per-turn ceiling.

Blocking issues

None. Verified:

  • _TURN_IDLE_TIMEOUT_S is the idle watchdog; the new default 600 remains well under the unchanged absolute cap _TURN_ABSOLUTE_TIMEOUT_S = 3600, so the backstop still bounds a runaway-but-active turn.
  • The HARNESS_TURN_TIMEOUT_S env knob and <= 0 disable path are untouched — ops can still override.
  • No test asserts the 240 literal; the scaffold watchdog tests drive the mechanism via explicit HARNESS_TURN_TIMEOUT_S overrides ("2", "10") and a direct _TURN_IDLE_TIMEOUT_S monkeypatch (0.2), so they remain valid regardless of the default.

Security vulnerabilities

None. No lockfile, dependency, extras, or trust-boundary changes.

Non-blocking notes

  • The stated failure mode is specifically the in-process LLM compaction path (_summarize_history_uncachedllm_client.responses.create), which has no inner timeout and is what the raised window protects. The runner-delegated path (_summarize_via_runner_uncached) already caps its HTTP call at timeout=120.0, so it would fail its own way before ever reaching even the old 240s watchdog — worth being aware of, but not something this PR needs to address.
  • 600s is a reasonable heuristic, but it's a fixed guess at "longest healthy compaction." If near-full compactions routinely approach that, a follow-up could consider emitting a periodic non-heartbeat progress event from the long compaction await (the actual root cause) rather than continuing to widen the window. Not required here — the default bump is the pragmatic fix and the env knob remains for tuning.

Overall: correct, minimal, and safe to merge. The change does exactly what it claims and the existing tests still cover the watchdog mechanism.


Automated review by Polly · workflow run

Copilot AI review requested due to automatic review settings August 4, 2026 06:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@TomeHirata
TomeHirata enabled auto-merge (squash) August 4, 2026 06:26
@TomeHirata
TomeHirata disabled auto-merge August 5, 2026 12:44
@TomeHirata
TomeHirata merged commit a0f50d2 into main Aug 5, 2026
62 of 63 checks passed
@TomeHirata
TomeHirata deleted the fix/compaction-idle-watchdog branch August 5, 2026 12:44
@github-actions github-actions Bot added the no-doc-update Merged PR does not need a docs update label Aug 5, 2026
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

🏷️ Doc impact: no-doc-update

Only changes an internal ops env-var default (HARNESS_TURN_TIMEOUT_S 240s→600s) for the turn idle watchdog, which is not a documented user-facing surface or default.

Auto-classified on merge. Set the label manually before merging to override. · run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-doc-update Merged PR does not need a docs update size/XS Pull request size: XS

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants