Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/resolve-follow-up-inputs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"eve": patch
---

Follow-up messages now resolve pending input before the conversation continues: unanswered tool approvals are denied and unanswered questions are dismissed instead of leaving the message queued.
6 changes: 3 additions & 3 deletions docs/concepts/sessions-runs-and-streaming.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,13 +85,13 @@ curl -X POST http://127.0.0.1:2000/eve/v1/session/<sessionId> \

The follow-up reuses the same durable session: same history, same state.

If the session is waiting on a human-in-the-loop approval, a matching text reply such as `approve` or `deny` answers the approval. Other follow-up text is held until the approval is answered, so an unrelated message does not implicitly deny the pending tool call.
If the session is waiting on a human-in-the-loop approval, send an `inputResponses` entry to approve or deny it. A follow-up `message` denies each unanswered approval, then starts the follow-up turn.

If the session is waiting on `ask_question`, a follow-up message clears that pending request before the model continues. An exact option match or permitted freeform response answers the question; any other message marks the question unanswered and starts the follow-up turn.
If the session is waiting on `ask_question`, send an `inputResponses` entry to answer it. A follow-up `message` dismisses each unanswered question, then starts the follow-up turn.

A response is stale when its request is no longer pending: the question or approval was already answered, cleared by a follow-up message, or cancelled. eve delivers a stale response to the model as a new user message, and the model decides whether the old selection still matters. A stale approval never authorizes the earlier tool call; the model must request the action and approval again if they are still needed.

Responses match pending requests by request ID, so a response to an older request stays a plain user message even while a different question or approval is pending. Like any follow-up message, a stale response clears a pending question and is held while an approval is pending.
Responses match pending requests by request ID, so a response to an older request stays a plain user message even while a different question or approval is pending. Like any follow-up message, a stale response clears a pending question and denies a pending approval.

For deterministic ordering, send one follow-up at a time and wait for the next `session.waiting` event before sending another message to the same session. See [message delivery and queueing](./execution-model-and-durability#message-delivery-and-queueing) for the current runtime contract.

Expand Down
4 changes: 2 additions & 2 deletions docs/tools/human-in-the-loop.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,11 +100,11 @@ Approvals and questions share one protocol:
1. The model requests input (an approval, or an `ask_question`).
2. eve emits an `input.requested` stream event carrying the pending requests.
3. The turn parks at `session.waiting`, durably, for as long as it takes.
4. The client answers with `inputResponses` (structured, keyed by `requestId`) or a normal follow-up `message`. A follow-up whose text matches an option ID, option label, or numeric option index resolves automatically, including approval options such as `approve` and `deny`.
4. The client answers with `inputResponses`, structured and keyed by `requestId`. A normal follow-up `message` supersedes the pending batch instead: unanswered approvals are denied and unanswered questions are dismissed before the message continues the conversation.

The run picks back up exactly where it parked. Because the pause is durable, nothing is held in memory while it waits — the process can restart and the parked turn survives.

For approval requests, unrelated follow-up text does not deny the tool call. eve keeps the approval pending and holds that text until the approval is answered, then replays it as the next message in the session.
A follow-up message never remains queued behind pending input. eve resolves unanswered approvals as denied and unanswered questions as ignored, then continues with the message. To answer a pending request, send an explicit `inputResponses` entry.

See [Sessions, runs & streaming](/docs/concepts/sessions-runs-and-streaming) for the full event and resume contract that this builds on.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,44 +3,25 @@ import { defineEval } from "eve/evals";
const MARKER = "authored-always-unrelated-input-P7M2";
const TOOL_NAME = "gate";

/** Regression reproduction for https://github.com/vercel/eve/issues/533. */
/** Regression reproduction for https://github.com/vercel/eve/issues/1224. */
export default defineEval({
description:
"HITL repro (#533): unrelated input does not replay an unresolved authored tool call.",
description: "HITL repro (#1224): unrelated input denies the authored tool call and continues.",
async test(t) {
const parked = await t.send(`Call the \`${TOOL_NAME}\` tool with marker "${MARKER}".`);
parked.calledTool(TOOL_NAME, { status: "pending", count: 1 });
const approval = t.requireInputRequest({
t.requireInputRequest({
display: "confirmation",
toolName: TOOL_NAME,
});

const unrelated = await t.send("Queue this unrelated note: ORBITAL-PINE-6C3R.");
const unrelated = await t.send("Reply with exactly ORBITAL-PINE-6C3R.");

unrelated.expectOk();
unrelated.notEvent("action.result", {
data: { result: { toolName: TOOL_NAME } },
});
unrelated.notEvent("step.started");
unrelated.event("session.waiting", { count: 1 });

const approved = await t.respond({
optionId: "approve",
requestId: approval.requestId,
});

approved.expectOk();
approved.event("action.result", {
data: {
result: {
kind: "tool-result",
output: new RegExp(MARKER),
toolName: TOOL_NAME,
},
status: "completed",
},
unrelated.event("action.result", {
data: { result: { toolName: TOOL_NAME }, status: "rejected" },
count: 1,
});
unrelated.messageIncludes(/ORBITAL-PINE-6C3R/i);
t.succeeded();
},
});
28 changes: 7 additions & 21 deletions e2e/fixtures/agent-tools-hitl/evals/hitl/text-approve.eval.ts
Original file line number Diff line number Diff line change
@@ -1,37 +1,23 @@
import { defineEval } from "eve/evals";

import { GUARDED_ECHO_TOKEN } from "./shared";

/**
* HITL flow: a plain follow-up message whose text matches an approval option
* resolves the pending approval the same way as structured inputResponses.
* HITL flow: a plain follow-up message is not an approval response, even when
* its text matches the approve option.
*/
export default defineEval({
description: "HITL smoke: text approve resolves a pending tool approval.",
description: "HITL smoke: text approve denies a pending tool approval.",
async test(t) {
const parked = await t.send('Call the guarded-echo tool with note "text-approve".');
parked.calledTool("guarded-echo", { status: "pending", count: 1 });
t.requireInputRequest({ display: "confirmation", toolName: "guarded-echo" });

const approved = await t.send("approve");
approved.expectOk();
approved.event("action.result", {
data: {
result: {
kind: "tool-result",
output: new RegExp(GUARDED_ECHO_TOKEN),
toolName: "guarded-echo",
},
status: "completed",
},
const resumed = await t.send("approve");
resumed.expectOk();
resumed.event("action.result", {
data: { result: { toolName: "guarded-echo" }, status: "rejected" },
count: 1,
});

t.succeeded();
t.calledTool("guarded-echo", {
output: new RegExp(GUARDED_ECHO_TOKEN),
status: "completed",
count: 1,
});
},
});
Original file line number Diff line number Diff line change
@@ -1,48 +1,28 @@
import { defineEval } from "eve/evals";

import { GUARDED_ECHO_TOKEN } from "./shared";

/**
* HITL flow: unrelated text sent while an approval is pending must not deny
* the approval or disappear. The message is replayed after the approval
* receives an explicit terminal answer.
* HITL flow: unrelated text sent while an approval is pending denies the
* approval and replays the message immediately.
*/
export default defineEval({
description: "HITL smoke: unrelated message during approval is queued.",
description: "HITL smoke: unrelated message denies approval and resumes.",
async test(t) {
const parked = await t.send('Call the guarded-echo tool with note "queued-approval".');
parked.calledTool("guarded-echo", { status: "pending", count: 1 });
const request = t.requireInputRequest({
t.requireInputRequest({
display: "confirmation",
toolName: "guarded-echo",
});

const queued = await t.send(
const resumed = await t.send(
"After the pending approval is resolved, reply with exactly QUEUED-HITL-OK.",
);
queued.expectOk();
queued.notEvent("action.result", {
resumed.expectOk();
resumed.event("action.result", {
data: { result: { toolName: "guarded-echo" }, status: "rejected" },
});
queued.event("session.waiting", { count: 1 });

const approved = await t.respond({
requestId: request.requestId,
optionId: "approve",
});
approved.expectOk();
approved.event("action.result", {
data: {
result: {
kind: "tool-result",
output: new RegExp(GUARDED_ECHO_TOKEN),
toolName: "guarded-echo",
},
status: "completed",
},
count: 1,
});
approved.messageIncludes(/QUEUED-HITL-OK/i);
resumed.messageIncludes(/QUEUED-HITL-OK/i);

t.succeeded();
},
Expand Down
Loading
Loading