Add azd ai eval extension for Foundry evaluations

.github/workflows/lint-ext-azure-ai-evaluations.yml

@@ -0,0 +1,28 @@
+name: ext-azure-ai-evaluations-ci
+
+on:
+  pull_request:
+    paths:
+      - "cli/azd/extensions/azure.ai.evaluations/**"
+      - ".github/workflows/lint-ext-azure-ai-evaluations.yml"
+      - ".github/workflows/verify-ext-providers.yml"
+    branches: [main]
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+  pull-requests: write # required by reusable workflow lint-go.yml
+
+jobs:
+  lint:
+    uses: ./.github/workflows/lint-go.yml
+    with:
+      working-directory: cli/azd/extensions/azure.ai.evaluations
+
+  verify-providers:
+    uses: ./.github/workflows/verify-ext-providers.yml
+    with:
+      working-directory: cli/azd/extensions/azure.ai.evaluations

cli/azd/extensions/azure.ai.evaluations/.gitignore

@@ -0,0 +1,5 @@
+# Test report written by ci-test.ps1 for the pipeline to publish.
+junitTestReport.xml
+
+# Debug log written when --debug or AZD_EXT_DEBUG is set.
+azd-ai-eval-*.log

cli/azd/extensions/azure.ai.evaluations/.golangci.yaml

@@ -0,0 +1,21 @@
+version: "2"
+
+linters:
+  default: none
+  enable:
+    - gosec
+    - lll
+    - unused
+    - errorlint
+  settings:
+    lll:
+      line-length: 220
+      tab-width: 4
+    gosec:
+      excludes:
+        - G204 # Subprocess launched with variable (bicep build invoked in tests)
+        - G304 # Potential file inclusion via variable
+
+formatters:
+  enable:
+    - gofmt

cli/azd/extensions/azure.ai.evaluations/CHANGELOG.md

@@ -0,0 +1,29 @@
+# Release History
+
+## 1.0.4-beta (Unreleased)
+
+First release of the Foundry evaluations extension.
+
+### Features Added
+
+- Initial release of the Foundry evaluations extension, `azd ai eval`.
+- `init` scaffolds `evals/azure.eval.yaml` next to an agent and adds the service
+  entry that `$ref`s it to `azure.yaml`, making no service calls.
+- `generate` synthesizes a rubric and dataset from the agent's context, writes
+  them under `evals/`, and merges `source:` references into the deployment spec
+  while preserving comments, ordering and neighboring entries.
+- `run` creates the eval group when it does not exist, starts a run, and
+  summarizes the result.
+- `azure.ai.eval` service-target provider deploys datasets, evaluators and eval
+  groups during a deploy, reconciling them in dependency order.
+- Change detection so a repeated deploy publishes no redundant versions:
+  datasets are fingerprinted locally, evaluator definitions are compared on the
+  keys the author wrote, and eval groups are recreated only when their own
+  declaration changes.
+- Atomic commands for every operation: `dataset`, `evaluator`, `run` and
+  `results` subcommands, all supporting `-o json` and `--no-prompt`.
+- Testing criteria are shaped from each evaluator's published contract, so
+  evaluators requiring inputs beyond the agent shape — `ground_truth`,
+  `context`, `instruction_id_list` — work by binding them to dataset columns.
+  A required column the dataset does not carry is reported before the request
+  is sent, naming the column.

cli/azd/extensions/azure.ai.evaluations/README.md

@@ -0,0 +1,173 @@
+# Azure Developer CLI (azd) Evaluations Extension
+
+Define Foundry evaluations alongside your agent in `azure.yaml`, deploy them
+with `azd up`, and run them from the terminal.
+
+```bash
+azd ai eval init          # scaffold evals/ next to your agent
+azd ai eval generate      # synthesize a rubric and dataset from the agent
+azd up                    # register datasets and evaluators, create the eval group
+azd ai eval run           # run the evaluation and summarize the results
+```
+
+## What gets deployed
+
+Eval resources are one service entry in `azure.yaml`, normally a `$ref` to a
+file under `evals/`:
+
+```yaml
+# azure.yaml
+services:
+  ai-project:
+    host: azure.ai.project
+  evals:
+    host: azure.ai.eval
+    uses: [ai-project]
+    $ref: ./evals/azure.yaml
+```
+
+```yaml
+# evals/azure.yaml
+datasets:
+  - name: support-golden
+    source: ./datasets/support-golden.jsonl
+
+evaluators:
+  - name: support-quality
+    source: ./evaluators/support-quality.json
+
+evalGroups:
+  - name: support-quality
+    dataset: support-golden
+    evaluators:
+      - builtin.task_adherence
+      - support-quality
+    target:
+      type: agent
+      name: support-agent
+    options:
+      eval_model: gpt-4.1-nano
+```
+
+`azd up` reconciles **datasets → evaluators → eval groups**, in that order,
+because a group references the versions the first two resolve to.
+
+Relative paths inside a `$ref`'d file resolve against **that file's**
+directory, so `./datasets/x.jsonl` above means `evals/datasets/x.jsonl`.
+
+### Repeated deploys do not create redundant versions
+
+Datasets are fingerprinted locally, because the dataset API exposes no content
+hash and comparing against the service would mean downloading the blob on every
+deploy. Evaluator definitions are compared against the service, but only on the
+keys you authored — the service adds `data_schema`, `init_parameters` and
+`metrics` of its own.
+
+Eval groups are immutable, so a change to a group's evaluators, target or
+options creates a new group and a new id. The id is cached in the azd
+environment so repeat runs stay comparable.
+
+## Commands
+
+| Group | Commands |
+|---|---|
+| `azd ai eval` | `init` · `generate` · `run` |
+| `azd ai eval dataset` | `create` · `list` · `show` · `update` · `delete` |
+| `azd ai eval evaluator` | `upload` · `list` · `show` · `update` · `delete` · `builtins` |
+| `azd ai eval run` | `start` · `list` · `show` · `cancel` |
+| `azd ai eval results` | `show` · `export` |
+
+`create` and `update` both publish a new immutable version; the server
+auto-increments and nothing mutates in place.
+
+Every command supports `-o json` and `--no-prompt`, so the whole surface is
+usable from CI.
+
+## Evaluators
+
+Built-ins need no declaration — reference them as `builtin.<name>` and list
+them with `azd ai eval evaluator builtins`.
+
+Evaluators do not share an input contract, so the CLI reads each one's
+published contract and shapes the request to match. An evaluator needing an
+input your dataset does not carry is reported before the request is sent, with
+the column named, rather than as a service-side rejection.
+
+A custom rubric is a JSON list of weighted dimensions:
+
+```json
+{
+  "dimensions": [
+    { "id": "accuracy", "description": "The answer is factually correct.", "weight": 5 },
+    { "id": "tone", "description": "The answer is polite and professional.", "weight": 2 }
+  ]
+}
+```
+
+`weight` is an **integer from 1 to 10**. Weights do not need to sum to
+anything.
+
+## Choosing a project
+
+The project endpoint is resolved in this order:
+
+1. `--project-endpoint`
+2. `FOUNDRY_PROJECT_ENDPOINT` in the active azd environment
+3. the host environment variable of the same name
+
+## Local development
+
+### Prerequisites
+
+- Go (the version in `go.mod`; `GOTOOLCHAIN=auto` fetches it)
+- [azd](https://aka.ms/azd) and the extension developer kit:
+  `azd ext install microsoft.azd.extensions`
+
+### Build, test, install
+
+```bash
+azd x build          # compile and install into the local azd
+azd x pack           # package the artifacts
+azd x publish        # register in the local extension source
+azd ext install azure.ai.evaluations --source local
+```
+
+```bash
+go test ./internal/...   # unit tests
+```
+
+### Live integration tests
+
+These talk to a real Foundry project, so they are excluded from the default
+build by the `live` tag and additionally gated on an environment variable:
+
+```bash
+export AZURE_AI_EVAL_E2E_LIVE=1
+export FOUNDRY_PROJECT_ENDPOINT=https://<account>.services.ai.azure.com/api/projects/<project>
+export AZURE_AI_EVAL_MODEL=gpt-4.1-nano       # optional judge model
+export AZURE_AI_EVAL_AGENT=<agent-name>       # optional, enables the run phase
+
+go test -tags live ./internal/cmd/ ./tests/live/
+```
+
+They clean up every resource they create.
+
+### Debug logging
+
+Request tracing is off by default. `--debug`, or `AZD_EXT_DEBUG=true`, writes
+it to a dated log file rather than the terminal.
+
+## TODO before release
+
+Both are files the azd extensions team owns, so they are not changed here:
+
+- [ ] **`cli/azd/extensions/registry.json`** — add the `azure.ai.evaluations`
+  entry. Until it exists `azd extension install azure.ai.evaluations` cannot
+  resolve, so the extension is only reachable through `azd x pack` +
+  `azd x publish` into the local source registry.
+- [ ] **`.github/CODEOWNERS`** — add `/cli/azd/extensions/azure.ai.evaluations/`.
+  Every sibling Foundry extension has an entry; without one, PRs here get no
+  reviewer routing.
+- [ ] **`microsoft.foundry/extension.yaml`** — add the dependency, but only
+  after the registry entry lands. Declaring a dependency that cannot resolve
+  breaks installing the bundle.

cli/azd/extensions/azure.ai.evaluations/build.ps1

@@ -0,0 +1,78 @@
+# Ensure script fails on any error
+$ErrorActionPreference = 'Stop'
+
+# Get the directory of the script
+$EXTENSION_DIR = Split-Path -Parent $MyInvocation.MyCommand.Path
+
+# Change to the script directory
+Set-Location -Path $EXTENSION_DIR
+
+# Create a safe version of EXTENSION_ID replacing dots with dashes
+$EXTENSION_ID_SAFE = $env:EXTENSION_ID -replace '\.', '-'
+
+# Define output directory
+$OUTPUT_DIR = if ($env:OUTPUT_DIR) { $env:OUTPUT_DIR } else { Join-Path $EXTENSION_DIR "bin" }
+
+# Create output directory if it doesn't exist
+if (-not (Test-Path -Path $OUTPUT_DIR)) {
+    New-Item -ItemType Directory -Path $OUTPUT_DIR | Out-Null
+}
+
+# Get Git commit hash and build date
+$COMMIT = git rev-parse HEAD
+if ($LASTEXITCODE -ne 0) {
+    Write-Host "Error: Failed to get git commit hash"
+    exit 1
+}
+$BUILD_DATE = (Get-Date -Format "yyyy-MM-ddTHH:mm:ssZ")
+
+# List of OS and architecture combinations
+if ($env:EXTENSION_PLATFORM) {
+    $PLATFORMS = @($env:EXTENSION_PLATFORM)
+}
+else {
+    $PLATFORMS = @(
+        "windows/amd64",
+        "windows/arm64",
+        "darwin/amd64",
+        "darwin/arm64",
+        "linux/amd64",
+        "linux/arm64"
+    )
+}
+
+$VERSION_PATH = "azureaieval/internal/version"
+
+# Loop through platforms and build
+foreach ($PLATFORM in $PLATFORMS) {
+    $OS, $ARCH = $PLATFORM -split '/'
+
+    $OUTPUT_NAME = Join-Path $OUTPUT_DIR "$EXTENSION_ID_SAFE-$OS-$ARCH"
+
+    if ($OS -eq "windows") {
+        $OUTPUT_NAME += ".exe"
+    }
+
+    Write-Host "Building for $OS/$ARCH..."
+
+    # Delete the output file if it already exists
+    if (Test-Path -Path $OUTPUT_NAME) {
+        Remove-Item -Path $OUTPUT_NAME -Force
+    }
+
+    # Set environment variables for Go build
+    $env:GOOS = $OS
+    $env:GOARCH = $ARCH
+
+    go build `
+        -ldflags="-X '$VERSION_PATH.Version=$env:EXTENSION_VERSION' -X '$VERSION_PATH.Commit=$COMMIT' -X '$VERSION_PATH.BuildDate=$BUILD_DATE'" `
+        -o $OUTPUT_NAME
+
+    if ($LASTEXITCODE -ne 0) {
+        Write-Host "An error occurred while building for $OS/$ARCH"
+        exit 1
+    }
+}
+
+Write-Host "Build completed successfully!"
+Write-Host "Binaries are located in the $OUTPUT_DIR directory."