Standardize Rust Vally scenarios and remove azure-servicebus-rust by LarryOsterman · Pull Request #381 · microsoft/skills

LarryOsterman · 2026-07-15T17:26:16Z

Split from #375 (3 of 5).

Rust scenario standardization and cleanup:

Adds eval.yaml / skill_effectiveness_eval.yaml and removes older eval-experiment.yaml / eval-nobuild*.yaml variants across *-rust scenarios
Updates shared Rust experiment config and the rust-cargo-build-failure grader plugin
Removes the azure-servicebus-rust skill (plugin + symlink) and its scenarios

Split from #375: restandardizes Rust scenario eval files (adds eval.yaml and skill_effectiveness_eval.yaml, removes older eval-experiment/nobuild variants), updates shared Rust experiment config and cargo-build-failure grader plugin, and removes the azure-servicebus-rust skill and scenarios. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9429ad33-6f31-4ec5-ba00-0f11b5afb82e

Copilot

Pull request overview

Standardizes Rust Vally evaluations, removes obsolete variants, and retires the Azure Service Bus Rust skill.

Changes:

Normalizes eight Rust evaluation suites and grader configuration.
Removes legacy experiment and no-build definitions.
Deletes the Service Bus Rust skill and scenarios.

Reviewed changes

Copilot reviewed 39 out of 39 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
`.github/plugins/azure-sdk-rust/skills/azure-servicebus-rust/SKILL.md`	Removes the Service Bus skill.
`.github/skills/azure-servicebus-rust`	Removes the compatibility symlink.
`tests/scenarios/_shared/vally/azure-sdk-rust-skill-experiment.yaml`	Drops the Service Bus evaluation.
`tests/scenarios/_shared/vally/grader-plugins/rust-cargo-build-failure/index.ts`	Updates Vally grader metadata.
`tests/scenarios/_shared/vally/grader-plugins/rust-cargo-build-failure/package.json`	Updates the Vally dependency.
`tests/scenarios/azure-cosmos-rust/vally/eval-experiment.yaml`	Removes the legacy experiment.
`tests/scenarios/azure-cosmos-rust/vally/eval.yaml`	Standardizes Cosmos evaluation checks.
`tests/scenarios/azure-cosmos-rust/vally/skill_effectiveness_eval.yaml`	Updates Cosmos effectiveness evaluation.
`tests/scenarios/azure-eventhub-rust/vally/eval-experiment.yaml`	Removes the legacy experiment.
`tests/scenarios/azure-eventhub-rust/vally/eval.yaml`	Standardizes Event Hubs evaluation checks.
`tests/scenarios/azure-eventhub-rust/vally/skill_effectiveness_eval.yaml`	Updates Event Hubs effectiveness evaluation.
`tests/scenarios/azure-identity-rust/vally/eval-experiment.yaml`	Removes the legacy experiment.
`tests/scenarios/azure-identity-rust/vally/eval.yaml`	Standardizes Identity evaluation checks.
`tests/scenarios/azure-identity-rust/vally/skill_effectiveness_eval.yaml`	Updates Identity effectiveness evaluation.
`tests/scenarios/azure-keyvault-certificates-rust/vally/eval-experiment.yaml`	Removes the legacy experiment.
`tests/scenarios/azure-keyvault-certificates-rust/vally/eval.yaml`	Standardizes certificate evaluation checks.
`tests/scenarios/azure-keyvault-certificates-rust/vally/skill_effectiveness_eval.yaml`	Updates certificate effectiveness evaluation.
`tests/scenarios/azure-keyvault-keys-rust/vally/eval-experiment.yaml`	Removes the legacy experiment.
`tests/scenarios/azure-keyvault-keys-rust/vally/eval.yaml`	Standardizes key evaluation checks.
`tests/scenarios/azure-keyvault-keys-rust/vally/skill_effectiveness_eval.yaml`	Updates key effectiveness evaluation.
`tests/scenarios/azure-keyvault-secrets-rust/vally/eval-experiment.yaml`	Removes the legacy experiment.
`tests/scenarios/azure-keyvault-secrets-rust/vally/eval.yaml`	Standardizes secret evaluation checks.
`tests/scenarios/azure-keyvault-secrets-rust/vally/skill_effectiveness_eval.yaml`	Updates secret effectiveness evaluation.
`tests/scenarios/azure-servicebus-rust/acceptance-criteria.md`	Removes Service Bus acceptance criteria.
`tests/scenarios/azure-servicebus-rust/scenarios.yaml`	Removes Service Bus scenarios.
`tests/scenarios/azure-servicebus-rust/vally/eval-experiment.yaml`	Removes the Service Bus experiment.
`tests/scenarios/azure-servicebus-rust/vally/eval.yaml`	Removes the Service Bus evaluation.
`tests/scenarios/azure-servicebus-rust/vally/skill_effectiveness_eval.yaml`	Removes its effectiveness evaluation.
`tests/scenarios/azure-servicebus-rust/vally/skill_effectiveness_experiment.yaml`	Removes its effectiveness experiment.
`tests/scenarios/azure-servicebus-rust/vally/workspace/cargo.toml`	Removes its test workspace.
`tests/scenarios/azure-storage-blob-rust/vally/eval-keyvault-encryption.yaml`	Standardizes the encryption evaluation.
`tests/scenarios/azure-storage-blob-rust/vally/eval-nobuild-experiment.yaml`	Removes a legacy no-build experiment.
`tests/scenarios/azure-storage-blob-rust/vally/eval-nobuild.yaml`	Removes a legacy no-build evaluation.
`tests/scenarios/azure-storage-blob-rust/vally/eval.yaml`	Standardizes Blob evaluation checks.
`tests/scenarios/azure-storage-blob-rust/vally/skill_effectiveness_eval.yaml`	Updates Blob effectiveness evaluation.
`tests/scenarios/azure-storage-blob-rust/vally/skill_effectiveness_experiment.yaml`	Drops obsolete evaluation variants.
`tests/scenarios/azure-storage-queue-rust/vally/eval-experiment.yaml`	Removes the legacy experiment.
`tests/scenarios/azure-storage-queue-rust/vally/eval.yaml`	Standardizes Queue evaluation checks.
`tests/scenarios/azure-storage-queue-rust/vally/skill_effectiveness_eval.yaml`	Updates Queue effectiveness evaluation.

LarryOsterman · 2026-07-15T17:35:19Z

Split from #375 (see #375 for the full split plan). Do not merge #375 until this and the sibling split PRs (#379, #380, #381, #382, #383) are merged.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

…oft/vally ^0.9.0

Copilot

Pull request overview

Copilot reviewed 43 out of 44 changed files in this pull request and generated 2 comments.

Files not reviewed (1)

tests/pnpm-lock.yaml: Generated file

Copilot

Pull request overview

Copilot reviewed 44 out of 45 changed files in this pull request and generated 2 comments.

Files not reviewed (1)

tests/pnpm-lock.yaml: Generated file

Copilot · 2026-07-21T23:09:59Z

    description:
      "Rust-specific grader: checks trajectory for failed cargo build tool calls, extracts Rust compiler errors (E0XXX), and scales score by failure ratio",
-    behavior: { execution: "single", requiresWorkspace: false },
+    behavior: { requiresWorkspace: false },


Fixed in bd1d8a8 — behavior: { requiresWorkspace: true } is now set in CargoBuildTrajectoryGrader's metadata, ensuring Vally will not silently pass without a workspace path.

Copilot · 2026-07-21T23:03:09Z

 defaults:
  runs: 1
-  timeout: 30m
+  timeout: 60m


The gpt-5.3-codex prompt grader override was removed and grader_model: gpt-5.5 added to defaults in bd1d8a8. The subsequent rename to judge_model was applied in 1475e00.

Copilot

Pull request overview

Copilot reviewed 44 out of 45 changed files in this pull request and generated no new comments.

Files not reviewed (1)

tests/pnpm-lock.yaml: Generated file

Comments suppressed due to low confidence (1)

tests/scenarios/_shared/vally/grader-plugins/rust-cargo-build-failure/index.ts:427

The grader now runs configured cargo commands against the final workspace, but its metadata still declares that no workspace is required. Vally can therefore invoke it without workspace context; resolveWorkspaceDir() then returns undefined, no command is executed, and the grader treats zero results as passing (index.ts:573-615). Mark the workspace as required so the newly migrated build/check/clippy graders cannot silently pass without running.

    behavior: { requiresWorkspace: false },

Copilot

Pull request overview

Copilot reviewed 44 out of 45 changed files in this pull request and generated 17 comments.

Files not reviewed (1)

tests/pnpm-lock.yaml: Generated file

-  timeout: 30m
+  timeout: 60m
  model: claude-sonnet-4.6
+  grader_model: gpt-5.5


-  timeout: 30m
+  timeout: 60m
  model: claude-sonnet-4.6
+  grader_model: gpt-5.5


…enarios

Copilot

Pull request overview

Copilot reviewed 44 out of 45 changed files in this pull request and generated no new comments.

Files not reviewed (1)

tests/pnpm-lock.yaml: Generated file

Comments suppressed due to low confidence (2)

tests/scenarios/azure-cosmos-rust/vally/eval.yaml:12

Vally 0.9 uses judge_model for the default prompt-judge model; grader_model is not a valid defaults key. This leaves the Cosmos capability eval inconsistent with every other standardized Rust eval and causes strict config validation to reject it. Rename this key to judge_model.

  grader_model: gpt-5.5

tests/scenarios/azure-cosmos-rust/vally/skill_effectiveness_eval.yaml:12

Vally 0.9 uses judge_model for the default prompt-judge model; grader_model is not a valid defaults key. This leaves the Cosmos skill-effectiveness eval inconsistent with every other standardized Rust eval and causes strict config validation to reject it. Rename this key to judge_model.

  grader_model: gpt-5.5

Copilot AI review requested due to automatic review settings July 15, 2026 17:26

LarryOsterman requested a review from thegovind as a code owner July 15, 2026 17:26

Copilot started reviewing on behalf of LarryOsterman July 15, 2026 17:26 View session

LarryOsterman mentioned this pull request Jul 15, 2026

Add Vally skill-effectiveness experiments and Python test scenarios #383

Open

Copilot AI reviewed Jul 15, 2026

View reviewed changes

This was referenced Jul 15, 2026

Add python skills and test. #375

Draft

skill-creator: lifecycle guidance and Azure SDK patterns reference #379

Merged

This was referenced Jul 15, 2026

Vally harness and CI workflow updates #380

Merged

Add capability and non-hero references for Python skills #382

Merged

Copilot started work on behalf of LarryOsterman July 16, 2026 22:12 View session

Fix Rust Vally review feedback in PR #381

b5ede07

Copilot AI review requested due to automatic review settings July 16, 2026 22:16

LarryOsterman removed the request for review from Copilot July 16, 2026 22:16

Copilot finished work on behalf of LarryOsterman July 16, 2026 22:18

LarryOsterman requested a review from Copilot July 16, 2026 22:25

Copilot started reviewing on behalf of LarryOsterman July 16, 2026 22:25 View session

Copilot AI reviewed Jul 16, 2026

LarryOsterman requested review from a team and Copilot July 16, 2026 22:48

Copilot started reviewing on behalf of LarryOsterman July 16, 2026 22:58 View session

Copilot AI reviewed Jul 16, 2026

Copilot started work on behalf of LarryOsterman July 16, 2026 23:07 View session

Fix CI: update pnpm-lock.yaml and add pnpm-workspace.yaml for @micros…

0d4babe

…oft/vally ^0.9.0

Copilot AI review requested due to automatic review settings July 16, 2026 23:09

LarryOsterman removed the request for review from Copilot July 16, 2026 23:09

Copilot finished work on behalf of LarryOsterman July 16, 2026 23:10

LarryOsterman requested a review from Copilot July 16, 2026 23:26

Copilot started reviewing on behalf of LarryOsterman July 16, 2026 23:28 View session

Copilot AI review requested due to automatic review settings July 21, 2026 21:45

Copilot started reviewing on behalf of LarryOsterman July 21, 2026 21:46 View session

Use vally version 0.10.0

c0d30ac

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread .github/workflows/run-vally-evaluations.yml Outdated

Comment thread tests/harness/copilot-client.ts

Copilot AI review requested due to automatic review settings July 21, 2026 21:49

Copilot started reviewing on behalf of LarryOsterman July 21, 2026 21:49 View session

Copilot started work on behalf of LarryOsterman July 21, 2026 21:50 View session

Fix review comments for workflow and nested skill tests

3f7eb20

Copilot finished work on behalf of LarryOsterman July 21, 2026 21:54

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 21, 2026 21:55

Copilot started reviewing on behalf of LarryOsterman July 21, 2026 21:55 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

vhvb1989 approved these changes Jul 21, 2026

View reviewed changes

Copilot started work on behalf of LarryOsterman July 21, 2026 22:47 View session

fix review comments from review 4749328063

bd1d8a8

Copilot AI review requested due to automatic review settings July 21, 2026 22:49

Copilot finished work on behalf of LarryOsterman July 21, 2026 22:50

Copilot started reviewing on behalf of LarryOsterman July 21, 2026 22:50 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Copilot started work on behalf of LarryOsterman July 21, 2026 22:58 View session

vhvb1989 approved these changes Jul 21, 2026

View reviewed changes

Rename grader_model to judge_model in Vally eval defaults for Rust sc…

1475e00

…enarios

Copilot AI review requested due to automatic review settings July 21, 2026 23:02

Copilot started reviewing on behalf of LarryOsterman July 21, 2026 23:02 View session

Copilot finished work on behalf of LarryOsterman July 21, 2026 23:03

Copilot started work on behalf of LarryOsterman July 21, 2026 23:05 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Copilot finished work on behalf of LarryOsterman July 21, 2026 23:10

LarryOsterman merged commit cc93536 into main Jul 21, 2026
6 checks passed

Uh oh!

Conversation

LarryOsterman commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LarryOsterman commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants