Add Vally skill-effectiveness experiments and Python test scenarios by LarryOsterman · Pull Request #383 · microsoft/skills

LarryOsterman · 2026-07-15T17:29:11Z

Split from #375 (5 of 5).

Adds the Vally skill-effectiveness experiment framework and Python test scenarios:

New scripts: run-skill-experiments.ps1, run-all-evals.ps1, generate-skill-experiments.ps1, compare-experiment.mjs
Standardized vally/ eval + experiment yaml and scenarios.yaml for ~40 Python skills
Shared Python grader tooling under tests/scenarios/_shared/
Updates to run-harness-by-language.ps1 and test dependency manifests

178 files under tests/ (excludes tests/harness/ and *-rust scenarios, covered by #380 and #381).

Split from #375: adds the Vally skill-effectiveness experiment framework (run-skill-experiments.ps1, run-all-evals.ps1, generate-skill-experiments.ps1, compare-experiment.mjs), standardized vally eval/experiment yaml and scenarios.yaml for ~40 Python skills, shared Python grader tooling, and harness runner script/dependency updates. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9429ad33-6f31-4ec5-ba00-0f11b5afb82e

Copilot

Pull request overview

Adds a standardized Vally “skill effectiveness” experiment setup for Python skills under tests/scenarios/, including shared Python grader tooling and updated test-runner dependencies/ignores. This PR is part of the split from #375 and focuses on the Python scenario/eval content (not harness/CI).

Changes:

Adds shared Node-based graders for Python syntax + basic idiomatic checks under tests/scenarios/_shared/vally/tools/.
Introduces per-skill Vally eval.yaml + skill_effectiveness_experiment.yaml (and, for some, skill_effectiveness_eval.yaml) plus updates many scenarios.yaml model defaults.
Updates tests/package.json (Vally version + package manager metadata) and expands tests/.gitignore to ignore evaluation output folders.

Reviewed changes

Copilot reviewed 176 out of 178 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/package.json	Bumps `@microsoft/vally` and adds `packageManager` metadata for tests tooling.
tests/.gitignore	Ignores additional eval output directories under `tests/`.
tests/scenarios/_shared/vally/tools/check-python-syntax.mjs	New shared Python syntax validation grader (supports optional exclude patterns config).
tests/scenarios/_shared/vally/tools/check-python-idiomatic.mjs	New shared heuristic Python “idiomatic” checks grader.
tests/scenarios/_shared/vally/azure-sdk-python-skill-evaluations.yaml	Aggregates many Python skill evals into a single suite file.
tests/scenarios/agent-framework-azure-ai-py/vally/skill_effectiveness_experiment.yaml	Adds skill-vs-baseline experiment definition for this skill.
tests/scenarios/agent-framework-azure-ai-py/vally/eval.yaml	Adds Vally eval suite for this skill’s scenarios.
tests/scenarios/agent-framework-azure-ai-py/vally/skill_effectiveness_eval.yaml	Adds the skill-effectiveness eval suite definition for this skill.
tests/scenarios/agent-framework-azure-ai-py/scenarios.yaml	Sets scenario model config for this skill.
tests/scenarios/agent-framework-azure-ai-py/test-output.txt	Adds a captured output artifact for this scenario directory.
tests/scenarios/azure-search-documents-py/vally/skill_effectiveness_experiment.yaml	Adds skill-vs-baseline experiment definition for this skill.
tests/scenarios/azure-search-documents-py/vally/eval.yaml	Adds Vally eval suite for this skill’s scenarios.
tests/scenarios/azure-search-documents-py/scenarios.yaml	Updates scenario model config for this skill.
tests/scenarios/azure-eventgrid-py/vally/skill_effectiveness_experiment.yaml	Adds skill-vs-baseline experiment definition for this skill.
tests/scenarios/azure-eventgrid-py/vally/eval.yaml	Adds Vally eval suite for this skill’s scenarios.
tests/scenarios/azure-eventgrid-py/scenarios.yaml	Updates scenario model config for this skill.
tests/scenarios/azure-messaging-webpubsubservice-py/vally/skill_effectiveness_experiment.yaml	Adds skill-vs-baseline experiment definition for this skill.
tests/scenarios/azure-messaging-webpubsubservice-py/vally/eval.yaml	Adds Vally eval suite for this skill’s scenarios.
tests/scenarios/azure-messaging-webpubsubservice-py/scenarios.yaml	Updates scenario model config for this skill.
tests/scenarios/azure-monitor-opentelemetry-py/vally/skill_effectiveness_experiment.yaml	Adds skill-vs-baseline experiment definition for this skill.
tests/scenarios/azure-monitor-opentelemetry-py/vally/eval.yaml	Adds Vally eval suite for this skill’s scenarios.
tests/scenarios/azure-monitor-opentelemetry-py/scenarios.yaml	Updates scenario model config for this skill.
tests/scenarios/pydantic-models-py/vally/syntax-check-config.json	Adds per-scenario syntax-check exclusion config.
tests/scenarios/pydantic-models-py/vally/skill_effectiveness_experiment.yaml	Adds skill-vs-baseline experiment definition for this skill.
tests/scenarios/pydantic-models-py/scenarios.yaml	Updates scenario model config for this skill.
tests/scenarios/fastapi-router-py/vally/syntax-check-config.json	Adds per-scenario syntax-check exclusion config.
tests/scenarios/fastapi-router-py/vally/skill_effectiveness_experiment.yaml	Adds skill-vs-baseline experiment definition for this skill.
tests/scenarios/azure-storage-queue-py/vally/skill_effectiveness_experiment.yaml	Adds skill-vs-baseline experiment definition for this skill.
tests/scenarios/azure-storage-queue-py/vally/eval.yaml	Adds Vally eval suite for this skill’s scenarios.
tests/scenarios/azure-storage-queue-py/vally/skill_effectiveness_eval.yaml	Adds the skill-effectiveness eval suite definition for this skill.
tests/scenarios/azure-storage-queue-py/scenarios.yaml	Updates scenario model config for this skill.
tests/scenarios/m365-agents-py/vally/skill_effectiveness_experiment.yaml	Adds skill-vs-baseline experiment definition for this skill.
tests/scenarios/m365-agents-py/vally/eval.yaml	Adds Vally eval suite for this skill’s scenarios.
tests/scenarios/m365-agents-py/vally/skill_effectiveness_eval.yaml	Adds the skill-effectiveness eval suite definition for this skill.
tests/scenarios/m365-agents-py/scenarios.yaml	Updates scenario model config for this skill.
tests/scenarios/azure-ai-language-conversations-py/scenarios.yaml	Adds scenario definitions and mock responses for this skill.
tests/scenarios/azure-ai-language-conversations-py/acceptance-criteria.md	Expands/clarifies acceptance criteria with correct/incorrect examples.

Files not reviewed (1)

tests/pnpm-lock.yaml: Generated file

LarryOsterman · 2026-07-15T17:35:23Z

Split from #375 (see #375 for the full split plan). Do not merge #375 until this and the sibling split PRs (#379, #380, #381, #382, #383) are merged.

Copilot

Pull request overview

Copilot reviewed 176 out of 178 changed files in this pull request and generated 7 comments.

Files not reviewed (1)

tests/pnpm-lock.yaml: Generated file

@@ -0,0 +1 @@
+Error: Experiment config file not found: Q:\src\skills\tests\scenarios\agent-framework-azure-ai-py\skill_effectiveness_experiment.yaml


+      - CRITICAL: Include explicit exception handling:
+        * Wrap client initialization and send/receive operations in try/except blocks
+        * Handle: ServiceBusError, EventHubError, OperationTimeoutError, AzureError
+        * Example: try: sender.send_batch(batch) except ServiceBusError as e: print(f'Error: {e}')


+      - CRITICAL: Include explicit exception handling:
+        * Wrap client initialization and send/receive operations in try/except blocks
+        * Handle: ServiceBusError, EventHubError, OperationTimeoutError, AzureError
+        * Example: try: sender.send_batch(batch) except ServiceBusError as e: print(f'Error: {e}')


+      - CRITICAL: Include explicit exception handling:
+        * Wrap client initialization and send/receive operations in try/except blocks
+        * Handle: ServiceBusError, EventHubError, OperationTimeoutError, AzureError
+        * Example: try: sender.send_batch(batch) except ServiceBusError as e: print(f'Error: {e}')


+      - CRITICAL: Include explicit exception handling:
+        * Wrap client initialization and send/receive operations in try/except blocks
+        * Handle: ServiceBusError, EventHubError, OperationTimeoutError, AzureError
+        * Example: try: sender.send_batch(batch) except ServiceBusError as e: print(f'Error: {e}')


Copilot

Pull request overview

Copilot reviewed 176 out of 178 changed files in this pull request and generated 16 comments.

Files not reviewed (1)

tests/pnpm-lock.yaml: Generated file

+    # Run compare-experiment.mjs for the experiment directory
+    $compareScript = Join-Path $PSScriptRoot "compare-experiment.mjs"
+    if (Test-Path $compareScript) {
+        & node $compareScript $experimentResultsDir


@@ -0,0 +1 @@
+Error: Experiment config file not found: Q:\src\skills\tests\scenarios\agent-framework-azure-ai-py\skill_effectiveness_experiment.yaml


+        * Wrap client initialization and send/receive operations in try/except blocks
+        * Handle: ServiceBusError, EventHubError, OperationTimeoutError, AzureError
+        * Example: try: sender.send_batch(batch) except ServiceBusError as e: print(f'Error: {e}')


+        * Wrap client initialization and send/receive operations in try/except blocks
+        * Handle: ServiceBusError, EventHubError, OperationTimeoutError, AzureError
+        * Example: try: sender.send_batch(batch) except ServiceBusError as e: print(f'Error: {e}')


+        $jsonlPath = Join-Path $runDir.FullName "results.jsonl"
+        if (Test-Path -LiteralPath $jsonlPath) {
+            $line = Get-Content -Path $jsonlPath |
+            Where-Object { $_ -match '"type"\s*:\s*"run-summary"' } |
+            Select-Object -Last 1
+            if ($line) {
+                try { $summary = $line | ConvertFrom-Json -Depth 20 } catch {}
+            }
+        }


+  }
+}
+
+function matchGlobPattern(filePath, pattern) {


+  ".vally",
+]);
+
+function collectPythonFiles(dir, acc) {


…lude-pattern support

richardpark-msft

Some stuff I think is worth fixing:

Let's use a real python linter (pylint?). Our skills should produce clean code anyways.
Can you add in a README.md too? There's some handy little workflows in here, hard to discover without a small roadmap.

richardpark-msft · 2026-07-16T23:48:11Z

+  const regex = new RegExp(`^${regexPattern}$`);
+  return regex.test(filePath);
+}
+


It's probably better to just use pylint or something. I'm assuming you have, or could have, a python env here.

richardpark-msft · 2026-07-16T23:48:11Z

+const EXCLUDED_DIRS = new Set([
+  ".git",
+  ".hg",
+  ".svn",


What environment are we running this check in, that we need to exclude some of these things?

richardpark-msft · 2026-07-16T23:48:11Z

+          scoring: scale_1_5
+  - name: persistent_agent_thread
+    prompt: |-
+      Show how to retrieve an existing agent and continue a conversation


I think you're just porting over some existing tests, but I feel like a more realistic prompt would say "write a program that can" or something of that nature. I'm curious if it's a style difference between us though.

richardpark-msft · 2026-07-16T23:48:11Z

+defaults:
+  runs: 1
+  timeout: 30m
+  model: claude-sonnet-4.6


I'd add a comment that mentions you're purposefully picking models from two different families (I believe you've mentioned this before)

richardpark-msft · 2026-07-16T23:48:11Z

+        weight: 1
+        config:
+          model: claude-sonnet-4.6
+          prompt: |-


If you can, I think it'd be better to use pylint. Someone on the Python team probably has a realistic config we can use (or maybe even a different linter)

…k, PS1 fixes

Copilot · 2026-07-17T21:07:47Z

Pull request overview

Copilot reviewed 176 out of 178 changed files in this pull request and generated 16 comments....

Fixed in commit f7b82c6. Here's a summary of all changes made:

pydantic-models-py eval.yaml and skill_effectiveness_eval.yaml: Removed the duplicate syntax-check-config.json file mapping in the environment.files section.
azure-eventgrid-py/vally/skill_effectiveness_eval.yaml: Replaced Service Bus/Event Hubs exception types (ServiceBusError, EventHubError, OperationTimeoutError) and sender.send_batch example with correct Event Grid exceptions (HttpResponseError, AzureError, Exception) and client.send(...) example.
azure-messaging-webpubsubservice-py/vally/skill_effectiveness_eval.yaml: Same fix — replaced Service Bus exceptions with Web PubSub-appropriate exceptions and client.send_to_all(...) example.
azure-monitor-opentelemetry-py/vally/eval.yaml and skill_effectiveness_eval.yaml: Changed the explicit_connection_string stimulus from asking for a hardcoded literal to asking for an environment-provided value, aligning the prompt with the rubric's credential security requirement. Also corrected the one_line_setup exception example from client.query_workspace(...) to configure_azure_monitor().
check-python-idiomatic.mjs: Changed tab detection from text.includes("\t") (flags any tab in the file) to /^\t|^ *\t/m.test(text) (flags only tabs in leading whitespace), avoiding false positives on separator = "\t" string literals.
run-all-evals.ps1: Three fixes: (1) replaced npx vally with pnpm --dir $PSScriptRoot exec vally to use the pinned Vally version from the lockfile; (2) guarded $runDir discovery behind $exitCode -eq 0 so a failed run cannot surface a stale previous result; (3) changed summary lookup from scanning results.jsonl for a run-summary type line to reading run-summary.jsonl directly.
run-skill-experiments.ps1: Changed -Compare to invoke compare-experiment.mjs once per $result.ExperimentDir (the per-skill variant directory) instead of once with $experimentResultsDir (the batch root).

Copilot

Pull request overview

Copilot reviewed 175 out of 177 changed files in this pull request and generated 8 comments.

Files not reviewed (1)

tests/pnpm-lock.yaml: Generated file

+Running experiment 'azure-monitor-ingestion-py-skill-experiment'
+Output: Q:\src\skills\tests\scenarios\azure-monitor-ingestion-py\vally\vally-experiment-results\2026-06-30T23-35-17-490Z


+Running experiment 'azure-monitor-ingestion-py-skill-experiment'
+Output: Q:\src\skills\tests\scenarios\azure-monitor-ingestion-py\vally\vally-experiment-results\2026-06-30T23-40-22-287Z


+# Validate vally is available
+$vallyCmd = Get-Command vally -ErrorAction SilentlyContinue
+if (-not $vallyCmd) {
+    Write-Error "vally command not found. Please install Vally to run experiments."
+    exit 1
+}


+$results = @()
+
+foreach ($evalFile in $evalFiles) {


+      - CRITICAL: Include explicit exception handling:
+        * Wrap client initialization and send/receive operations in try/except blocks
+        * Handle: ServiceBusError, EventHubError, OperationTimeoutError, AzureError
+        * Example: try: sender.send_batch(batch) except ServiceBusError as e: print(f'Error: {e}')


+      - CRITICAL: Include explicit exception handling:
+        * Wrap client initialization and send/receive operations in try/except blocks
+        * Handle: ServiceBusError, EventHubError, OperationTimeoutError, AzureError
+        * Example: try: sender.send_batch(batch) except ServiceBusError as e: print(f'Error: {e}')


+      - Return the final answer as Python code only.
+      - CRITICAL: Include explicit exception handling:
+        * Wrap client initialization and query/ingestion operations in try/except blocks
+        * Handle: ServiceError, HttpResponseError, ValidationError, AzureError


+      - Return the final answer as Python code only.
+      - CRITICAL: Include explicit exception handling:
+        * Wrap client initialization and query/ingestion operations in try/except blocks
+        * Handle: ServiceError, HttpResponseError, ValidationError, AzureError


Copilot AI review requested due to automatic review settings July 15, 2026 17:29

LarryOsterman requested a review from thegovind as a code owner July 15, 2026 17:29

Copilot started reviewing on behalf of LarryOsterman July 15, 2026 17:29 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Update vally version

3f410d7

Copilot AI review requested due to automatic review settings July 15, 2026 19:21

Copilot started reviewing on behalf of LarryOsterman July 15, 2026 19:22 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Update package info

e573b16

Copilot AI review requested due to automatic review settings July 15, 2026 21:25

Copilot started reviewing on behalf of LarryOsterman July 15, 2026 21:26 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Copilot started work on behalf of LarryOsterman July 16, 2026 23:01 View session

Fix review comments: delete artifact, fix exception handling, add exc…

6e8b2e8

…lude-pattern support

Copilot AI review requested due to automatic review settings July 16, 2026 23:20

LarryOsterman removed the request for review from Copilot July 16, 2026 23:20

Copilot finished work on behalf of LarryOsterman July 16, 2026 23:23

richardpark-msft approved these changes Jul 16, 2026

View reviewed changes

Copilot started work on behalf of LarryOsterman July 17, 2026 20:58 View session

Copilot finished work on behalf of LarryOsterman July 17, 2026 21:00

Copilot started work on behalf of LarryOsterman July 17, 2026 21:01 View session

Fix review 4708288806 comments: exceptions, duplicate files, tab chec…

f7b82c6

…k, PS1 fixes

Copilot AI review requested due to automatic review settings July 17, 2026 21:06

Copilot started reviewing on behalf of LarryOsterman July 17, 2026 21:07 View session

Copilot finished work on behalf of LarryOsterman July 17, 2026 21:08

Copilot AI reviewed Jul 17, 2026

View reviewed changes

		@@ -0,0 +1 @@
		Error: Experiment config file not found: Q:\src\skills\tests\scenarios\agent-framework-azure-ai-py\skill_effectiveness_experiment.yaml

		Running experiment 'azure-monitor-ingestion-py-skill-experiment'
		Output: Q:\src\skills\tests\scenarios\azure-monitor-ingestion-py\vally\vally-experiment-results\2026-06-30T23-35-17-490Z

Uh oh!

Conversation

LarryOsterman commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LarryOsterman commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

richardpark-msft left a comment

Choose a reason for hiding this comment

Uh oh!

richardpark-msft Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

richardpark-msft Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

richardpark-msft Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

richardpark-msft Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

richardpark-msft Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Jul 17, 2026

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants