Vally harness and CI workflow updates#380
Merged
Merged
Conversation
Split from #375: harness output capture and runner updates, new copilot-client tests, and CI workflow updates (rename vally-evaluation to run-vally-evaluations, test-harness tweaks, actions-lock). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9429ad33-6f31-4ec5-ba00-0f11b5afb82e
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Vally test harness to capture richer generation output (including raw model responses) and to better handle Copilot generation failures, while also adjusting CI workflows to run Vally evaluations under updated tooling/permissions.
Changes:
- Extend harness result types and reporting to include
rawResponsefrom generation. - Add Copilot generation retry/error-classification logic and improve runner error output behavior (especially for JSON output).
- Update GitHub Actions workflows for harness link-sync checks and Vally evaluation execution (renamed workflow, updated Node/Vally versions, permissions, and PR scoping).
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/harness/types.ts | Adds rawResponse to evaluation results and updates defaults/signatures. |
| tests/harness/runner.ts | Captures rawResponse, adds generation failure findings/suggestions, and adds JSON error output helpers. |
| tests/harness/ralph-loop.test.ts | Updates mocks/tests to satisfy the new rawResponse field and formatting changes. |
| tests/harness/copilot-client.ts | Adds typed Copilot generation errors, retry/backoff, configurable timeout/retries, and improved code extraction heuristics. |
| tests/harness/copilot-client.test.ts | Adds unit tests for extractCode and classifyCopilotError. |
| .github/workflows/test-harness.yml | Expands skill link sync verification to include Python in addition to Rust. |
| .github/workflows/run-vally-evaluations.yml | Renames/updates the Vally workflow, updates tool versions, permissions, and PR scoping behavior. |
| .github/aw/actions-lock.json | Adds a locked entry for github/gh-aw-actions/setup@v0.81.6. |
Comment on lines
+347
to
351
| if (s && typeof s.disconnect === "function") { | ||
| await s.disconnect(); | ||
| } else if (typeof s.destroy === "function") { | ||
| } else if (s && typeof s.destroy === "function") { | ||
| await s.destroy(); | ||
| } |
Comment on lines
+879
to
+899
| function classifyCliError(error: unknown): CliErrorKind { | ||
| if (error instanceof CopilotGenerationError) { | ||
| return error.kind; | ||
| } | ||
|
|
||
| const message = error instanceof Error ? error.message.toLowerCase() : ""; | ||
| if ( | ||
| message.includes("authentication failed") || | ||
| message.includes("bad credentials") || | ||
| message.includes("401") | ||
| ) { | ||
| return "auth"; | ||
| } | ||
| if (message.includes("timeout") || message.includes("session.idle")) { | ||
| return "timeout"; | ||
| } | ||
| if (message.includes("rate limit") || message.includes("429")) { | ||
| return "transient"; | ||
| } | ||
| return "fatal"; | ||
| } |
This was referenced Jul 15, 2026
Member
Author
This was referenced Jul 15, 2026
richardpark-msft
left a comment
richardpark-msft
left a comment
Member
There was a problem hiding this comment.
Bringing over some comments to this PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Split from #375 (2 of 5).
Updates the Vally test harness code and CI workflows:
tests/harness/runner/output-capture changes, newcopilot-client.test.ts,ralph-loopandtypesupdates.github/workflows/— renamedvally-evaluation.yml->run-vally-evaluations.yml,test-harness.ymlupdate.github/aw/actions-lock.json