Add azd ai eval extension for Foundry evaluations - #9500
Open
Mohamed Hessien (m7md7sien) wants to merge 223 commits into
Open
Add azd ai eval extension for Foundry evaluations#9500Mohamed Hessien (m7md7sien) wants to merge 223 commits into
Mohamed Hessien (m7md7sien) wants to merge 223 commits into
Conversation
New azd extension exposing azd ai eval, registering the azure.ai.eval service-target provider. Scaffold only: manifest, entrypoint, root command.
Lifted eval_api (models, operations, poller, generation, portal_urls) and dataset_api from azure.ai.agents, de-agent-scoped. Added evalcore with IsTransientError and an EvaluatorList that accepts either a bare string or a mapping with a threshold. Skipped artifacts.go and eval_config.go since the config model differs.
… commands - EvalConfig and GenerateConfig types with validation covering cross-references, duplicate names, unsupported target types, and evaluation levels - ResolveGroup picks the only group or errors with the available names - ArtifactPath accepts a directory or an explicit file path for local_dir - evalContext resolves the project endpoint (flag, azd env, host env) and builds both clients against the azd developer CLI credential - dataset create/update/list/show/delete with -o json - Tier-0 tests for parsing, validation, group resolution, and path handling
- buildEvalGroupRequest maps evaluators to testing criteria, keeping the builtin prefix on evaluator_name while stripping it from name, and carries per-evaluator thresholds in initialization_parameters - run resolves the group from --eval-id, a pinned id, or the azd environment, creating it when absent, then binds the dataset to the run since the group has no dataset binding today - Local datasets are sent inline with optional truncation; registered datasets are referenced by id
- evaluator upload/update/list/show/builtins/delete; rubric evaluators only in M1, code evaluators deferred to M2 with the folder walk and RBAC they require - normalizeRubricBody accepts a bare definition or a full document - results show/export with per-criteria pass and fail counts, --failed-only, and JSON or CSV output, replacing the counts-only view - Added ListEvaluators, ListEvaluatorVersions, DeleteEvaluatorVersion, and CancelOpenAIEvalRun to the eval client
init scaffolds both YAML files and the artifact directories without any service call, so it works offline and unauthenticated. Built-ins are referenced from the group but never declared as custom evaluators. A dataset flag containing a path becomes a local source; a bare name references a registered dataset. Tests assert the scaffold loads and validates, and that paths are used verbatim rather than re-rooted.
…tion Registers the azure.ai.eval service target so azd up and azd deploy reach this extension; the extension ships no deploy command of its own. - Reads the eval config from the service entry's inline properties, the same AdditionalProperties channel the agents extension uses - Deploy reconciles datasets, then evaluators, then eval groups, since a group references the versions the first two resolve to; it fails fast and the next deploy resumes - Datasets are change-detected with a local SHA-256 digest kept in the azd environment, because the dataset API returns no content hash and comparing against the service would mean downloading the blob every deploy - Evaluator definitions come back inline, so those are compared directly - Package and Publish are no-ops; eval artifacts are plain files already on disk
- generate submits the rubric and dataset generation jobs, downloads the artifacts locally, and writes source references into the deployment spec - MergeArtifactRefs edits through the yaml Node API so comments, key order, and hand-edited sibling keys survive; matching is by name and merging is idempotent - Raised the client poll budget from 2s x 300 to 5s x 720. The old 10 minute limit gave up while the service was still working, which is the timeout that forced a second command - A supplied --evaluator or a local --dataset is honored and its generation is skipped - Tests cover comment preservation, section creation, idempotence, and fingerprinting
Adds the templated release pipeline for the new extension and lists it as a dependency of the microsoft.foundry meta-package. The registry.json artifact entries are generated by the release, so they are not hand-authored here.
… tests Live testing against a real project found two issues. 1. The dataset model only bound snake_case URIs (data_uri, blob_uri), but the project endpoint returns camelCase (dataUri). ResolvedBlobURI therefore returned empty, which would have failed the generate download much later with no useful error. Both spellings are now accepted. 2. Built-in evaluators do not share one input contract. builtin.ifeval requires instruction_id_list and is rejected under the agent-target data mapping with MissingRequiredDataMapping. The live tests now select an evaluator whose inputs match, and the helper documents why. Live tests are gated behind the live build tag and AZURE_AI_EVAL_E2E_LIVE, and clean up every resource they create. Verified: builtin listing, the full dataset pending-upload lifecycle with version auto-increment, and eval group creation returning 201.
…published contract The builder sent one fixed data mapping and one fixed set of initialization parameters to every evaluator. That only suited agent-target quality evaluators, and the service rejected the rest. The evaluator listing publishes a full contract per evaluator: definition.data_schema (accepted and required inputs), definition.init_parameters, and supported_evaluation_levels. The builder now reads it and shapes each testing criterion accordingly. This fixes four concrete defects. - Required inputs were never honoured, so builtin.ifeval (instruction_id_list), builtin.similarity (ground_truth) and builtin.retrieval (context) all failed with MissingRequiredDataMapping. Fields not supplied by the agent target are now bound to dataset columns, and the item schema declares them. - Inputs an evaluator does not accept were sent anyway. - initialization_parameters always carried model, deployment_name and threshold. No evaluator accepts 'model', and builtin.ifeval accepts nothing at all. Parameters are now filtered to the declared properties, and a required one that is missing is reported locally. - evaluation_level was sent as run metadata, where it has no effect. It is an initialization parameter on the evaluators that declare it. It also encodes an exclusivity rule the service enforces: 'messages' and 'query'/'response' cannot both appear in a mapping, so the evaluation level selects between the conversation and turn shapes. A missing dataset column is now caught before the request is sent and names the column, rather than surfacing as a 400 pointing at testing_criteria[0].data_mapping. An evaluator with no published contract keeps the previous agent-target shape, so custom evaluators are unaffected. Verified against a live project: all ten built-ins are accepted, where two previously failed. The live test exercises the shipping builder rather than a hand-rolled request, so a regression in this logic fails the suite. Adds DeleteOpenAIEval so those tests clean up after themselves.
…invokes Without build.ps1 and build.sh the dev kit reported a successful build in under a second and produced no binary, so azd x pack had nothing to package and azd x publish failed with 'Artifacts not found'. Copied from azure.ai.agents with the version package path retargeted, plus its golangci config. Verified end to end: build, pack, publish, install from the local registry, and 'azd ai eval --help' listing every command.
…ly returns The listing spells it evaluator_type, so the TYPE column in 'evaluator list' and 'evaluator builtins' was always blank. Both spellings are now accepted.
Running a real 'azd deploy' against the service-target provider surfaced two failures that no unit test covered. Evaluator references only decoded from YAML. azd hands the service entry to the extension as JSON, so a group written as '- builtin.task_adherence' -- the form the CLI's own init command writes -- failed with 'cannot unmarshal string into EvaluatorRef'. EvaluatorList now decodes and encodes the mixed string-or-mapping form through JSON as well, and a test asserts the two decoders agree. The dataset reconciler passed the declared version straight to UploadNewVersion, which derives the next version from it. A declaration without an explicit version passed empty, so every deploy retried 1.0 and the service returned 409 TemporaryDataReferencesForExistingAsset once that version existed. It now looks up the latest registered version first. Verified against a live project: first deploy publishes the dataset at 1.0 and creates the group; an unchanged redeploy reports 'unchanged at version 1.0' and uploads nothing; and editing the dataset publishes 2.0 and recreates the group, since groups are immutable.
azd core does not resolve $ref for extensions. It strips the ServiceConfig fields it owns and leaves $ref at the top of the map for the owning extension to resolve, so a service authored the way the spec documents it -- host: azure.ai.eval plus $ref: ./evals/azure.yaml -- parsed to an empty config. azd deploy then reported success in three seconds having created nothing, which is worse than failing. The provider now calls foundry.ResolveFileRefs with the project root from the azd project client. Relative source paths inside an included file are written against that file, but ResolveFileRefs inlines content without rebasing them, so the include's own directory is now the base for source resolution. Verified against a live project: the $ref form deploys, and the dataset fingerprint matches the one from the equivalent inline config, confirming both forms resolve to the same file.
…ation changes Change detection only covered upstream artifacts, so retargeting a group at a different agent, swapping an evaluator, or changing the judge model left the old group in place. Groups are immutable, so the edit silently had no effect and later runs kept evaluating the previous definition. The group's declaration is now fingerprinted alongside the dataset and evaluator artifacts. The id and description are excluded: one is server assigned and the other is cosmetic, so neither should force a recreate. The digest is recorded when an existing group is reused as well as when one is created. Recording it only on create meant a group deployed before this change never established a baseline, and the first edit after it would still go undetected. Verified against a live project: changing the target produced a new group id, and two further deploys with no change reused it.
The data-plane clients trace every request and response through log.Printf, which Go writes to stderr by default, so a plain command interleaved raw URLs and status lines with its own output. A long generate run was mostly HTTP traces. Ports the debug setup from the agents extension: the standard logger is discarded unless --debug or AZD_EXT_DEBUG is set, and debug output goes to a dated file rather than the terminal. The hook chains the SDK PersistentPreRunE instead of replacing it. Assigning PersistentPreRun has no effect once the E variant is set, and overwriting the E variant would drop the SDK own setup. Also reports jobs as submitted when generate is given --no-wait, which is a successful submission rather than an empty result.
…fails Data generation with an agent source is accepted and then fails within seconds with DataGenerationJobSystemError, whose message says only that something went wrong and to try again. It is not transient: it reproduces for every agent tried, while the identical request without the agent source runs normally. The CLI now names the agent, says a retry will not help, and points at the two workarounds, instead of relaying advice that cannot succeed.
The spec lists run start, list, show and cancel, and M1 requires every operation to be reachable atomically, but run was a single composite command with no subcommands. Listing runs, inspecting one, and cancelling an in-flight run were unreachable, even though the client already had the calls. Adds run list, run show and run cancel. Each takes the eval group id as an optional argument and otherwise falls back to the id recorded in the azd environment, matching results show. Cancelling a run that already reached a terminal state is refused locally, because the service reports success either way and the CLI would otherwise claim to have cancelled a finished run. Two related fixes. Passing --project-endpoint disabled the azd environment cache entirely: the environment name was only resolved when the endpoint came from azd, so every cached eval group and run id lookup returned empty. The name is now resolved independently of where the endpoint came from. The spec documents --wait and --no-wait, but cobra does not derive the negative form from a bool, so --no-wait was rejected as an unknown flag. Verified live: start with --no-wait, list, show, cancel, and the terminal-state guard on a second cancel. JSON output checked on the new subcommands.
The spec lists run start alongside list, show and cancel. The behaviour existed only as the composite `azd ai eval run`, so the atomic name in the spec did not resolve. Both forms are now built by one constructor, so their flags cannot drift apart, and a test asserts that.
… work Exercising the atomic write commands against a live project found three failures. None were covered by tests, because none of these paths had been run end to end. dataset update always collided. It passed the --version flag straight to UploadNewVersion, which derives the next version from what it is given, so an omitted flag restarted at 1.0 and the service returned 409 TemporaryDataReferencesForExistingAsset. The flag help promised the opposite, that omitting it would take the next version. This is the same defect that was fixed in the deploy reconciler earlier, so the discovery is now centralised in DatasetClient.UploadNextVersion and both callers use it, rather than being fixed twice and available to be missed a third time. evaluator upload rejected every hand-authored rubric. The service needs a type discriminator on the definition, and without it fails the whole request with "The request field is required", which names a field that is present. Generated rubrics carry the type, so only the hand-authored path documented in the spec was affected. The type is now filled in when absent and left alone when set. evaluator show returned 404. It omitted the version segment from the path, but the service has no route for an unversioned evaluator, despite the doc comment claiming the latest would be fetched. The latest version is now resolved first, comparing numerically because versions are integers as strings and a lexical compare ranks "9" above "15" -- the service already publishes evaluators at version 15 and 17. Verified live: dataset create, show, update to 2.0, list and delete; evaluator upload, show resolving the latest, update to version 2, list and delete. Both suites leave nothing behind.
…nfig Deploying a config that declares a custom evaluator, rather than only built-in ones, failed in two ways. Every earlier test used built-ins, so neither showed up. The evaluator was republished on every deploy. The service enriches a definition when it stores it, so a rubric consisting of nothing but type and dimensions comes back carrying data_schema, init_parameters and metrics it was never given. Comparing whole documents therefore never matched. Only the keys the author actually wrote are compared now, structurally, so key order and formatting are not changes either. This is what the spec means by repeated azd up creating no redundant versions. The eval group was then rejected with a request for a model that had been set. Evaluators disagree on what the judge model is called: built-ins declare deployment_name, and a custom rubric declares model. The builder sent only deployment_name, so the custom evaluator saw its required parameter missing. The judge model is now bound under whichever name the evaluator declares. Verified live: first deploy publishes the evaluator and creates the group, two redeploys report it unchanged and publish nothing, editing the rubric publishes the next version and recreates the group, and a further redeploy is a no-op again.
…d dataset The flag is documented as taking a path or the name of a registered dataset, and means use this one instead of generating. It only suppressed generation when the value looked like a local path, so passing the name of an existing dataset still submitted a generation job and, since agent-seeded generation is currently broken server-side, failed the whole command. --evaluator already skipped unconditionally, so the two flags disagreed. Both the skip and the default-spec synthesis now key off whether the flag was supplied at all. This was the last thing standing between a generated config and the documented end-to-end flow. Verified live: init scaffolds a group referencing its own rubric, generate writes that rubric and merges the reference into the same file while preserving comments and ordering, azd up registers the dataset and evaluator and creates the group, and the run completes and scores against the generated rubric.
…listing
GET /datasets/{name}/versions returns nothing for a second or two after a
version is created, even though the version itself reads back immediately.
Measured: empty at 0s, populated at 2s.
That undermines the version discovery added for dataset update, which reads the
listing to decide what to increment from. An empty listing is ambiguous -- it
means either a new dataset or a stale read -- so back-to-back create and update
could still restart at 1.0 and take a 409.
Rather than delaying every first upload to wait for the index, a conflict is
now treated as the stale read it is: re-read the listing, which by then
reflects reality, and retry once. The common path is unchanged.
The live test asserted on the first listing response and was failing for the
same reason. It now polls, and says why.
Verified: create immediately followed by update produces 2.0 rather than a
conflict, and the full live suite passes.
…requires The shared extension build template invokes ci-build.ps1 and ci-test.ps1 from the extension directory. Neither existed, so the release pipeline added alongside this extension would have failed on its first run. Both are modelled on the agents extension with two deliberate differences. ci-build.ps1 reads version.txt from the extension directory rather than its parent, where no such file exists, so the default works when the pipeline is not supplying -Version. It accepts -BuildRecordMode, which the template always passes, but builds nothing extra: this extension has no record/playback mode and no pipeline step consumes a record binary. ci-test.ps1 passes --junitfile explicitly. The pipeline publishes **/junitTestReport.xml from the extension directory, and the extension template does not set GOTESTSUM_JUNITFILE the way the CLI build does, so without this no test results would surface in the build. Verified locally with gotestsum installed: 97 tests across 7 suites reported. Also adds the README and CHANGELOG that 17 of the 21 extensions ship. The README documents the deployed shape, the command surface, the rubric weight constraint, and how to run the live tests.
The manifest declared two capabilities the extension did not back. metadata was declared but the command was never registered, so azd could not discover the command tree: azd ai eval metadata failed with unknown command while the same call against a peer extension returned its full tree. azd uses this for discovery, so the declaration was actively misleading. The command is now registered and reports nine commands. lifecycle-events was declared but no event handlers exist. The SDK only starts its event manager when handlers are registered, so the capability was an unused permission rather than a broken promise. It is removed; the listen command is still invoked because the service-target-provider capability triggers it, which a deploy after the change confirms. Adds tests over the manifest so neither can drift again: every declared capability must be backed by the command that implements it, the declared provider name must match the host the code registers, and version.txt must agree with the manifest version, which until now was only a comment asking for it.
…ction from a file Auditing every flag and API sequence the spec documents against the running extension turned up two gaps. The spec describes a drift check that was never implemented. It matters because of how change detection works: when local content is unchanged, the version recorded at the last deploy is reused, so a version published outside the repo would be silently ignored and the eval group pinned to older data. A deploy now fails when the service holds a newer version than the recorded one, naming both versions. An explicit version: on the declaration skips the check, because that is the author stating which version they want. This was added after testing the remedy the error message suggests and finding it did not work -- the message now describes something that does. --gen-instruction-file was documented but absent. A useful generation instruction is usually longer than fits on a command line, and putting it in a file makes it reviewable with the rest of the config. Verified live: publishing a version out-of-band fails the next deploy, and pinning that version lets it through.
… edits M1 exits on all the spec examples running end to end, so I ran them verbatim. Two did not. --eval-id could never work. It is meant to run an existing group ignoring the config, and appears in both the CI/CD example and the recovery advice, but a run needs a target and a dataset and an eval group carries neither: the group holds only its testing criteria, and the dataset travels on the run. Every --eval-id invocation failed asking for a target. The pairing survives in the group's previous run, so re-running a group now repeats what it last ran, and a group that has never run says so and points at the config-based path. The failure-and-recovery example promised an error that did not exist. A run sends a local dataset inline, so unregistered local edits were evaluated silently and the results could not be traced to any dataset version. That now fails with the message the spec documents, once a deploy has recorded a fingerprint to compare against. Before that there is nothing to have drifted from, and running is how a group first comes into existence. Verified live: the CI/CD example returns JSON with a run id, and with unregistered edits the config-based run fails while --eval-id succeeds, which is exactly the recovery the spec describes.
…ploy spec init --dataset ./tests/golden.jsonl wrote that path into evals/azure.yaml unchanged, but source: is resolved relative to the file it appears in, so the deploy looked for evals/tests/golden.jsonl and failed on a file the user had just pointed at. This is the spec's bring-your-own-data example exactly as written, so that example could never have worked. The path is now rebased onto the output directory, with forward slashes so the config reads the same on every platform, and absolute paths left alone. With this the documented examples all run end to end, which is what M1 exits on: bring-your-own-data through init, azd up and run; results show --failed-only -O writing its file; and the CI/CD sequence of dataset create, run start -o json and results export --format csv.
agent.context.traces accepts source, window and sample, but the generation API takes a day count and nothing else, so source and sample were parsed and dropped without a word. An author who set sample: 500 believed they had narrowed the trace selection when nothing had changed. Both fields are documented in the spec, so this was reachable by following it. They are now reported as having no effect, naming each one, with the verb agreeing so one field reads "has" and two read "have". The warning goes to stdout rather than stderr because azd does not surface an extension's stderr -- written to stderr it was invisible in a real run even though the unit test passed -- and is suppressed under -o json so the output stays parseable.
azidentity gives the azd CLI a fixed 10 second timeout (cliTimeout in developer_credential_util.go, a const with no option to raise it) and discards the subprocess's stderr. An azd that overruns therefore surfaces as "AzureDeveloperCLICredential: exit status 1" with no cause, and the command fails. Measured previously: most token calls are fast, but one took 72s. The next call usually finds a warm token, which is why "try again" has been the standing advice. This makes the retry automatic, so a slow token costs a pause rather than a failed command. Hit during a bug bash run of `azd ai eval generate`, where it failed the dataset half after the evaluator had succeeded. Scope is deliberately narrow: one retry, the original error is returned if it also fails, and a cancelled context is not retried. Tests cover all four: recovery on the second attempt, no retry on success, the cause surviving a double failure, and a cancelled context short-circuiting.
Two fixes have landed since 1.0.3-beta was published to the bug bash feed, both found by running the scenarios end to end: - the multi-eval error pointed at --eval, a flag `create` does not have - a slow azd token failed the command rather than being retried Bumps version.txt and extension.yaml and adds the changelog section, so the build published to the feed reports the version that carries them.
b74972a assigned the retrying credential to evalContext.cred and then built both data-plane clients from the unwrapped one, so every token still came from the bare AzureDeveloperCLICredential and nothing retried. The commit had no production effect, and 1.0.4-beta shipped a changelog entry saying otherwise. Found by an adversarial review pass, not by the tests: all four existing tests construct azdTokenRetry directly, so they passed with the wiring absent. Rather than correcting the two call sites, newAzdTokenCredential now returns the wrapper, so the raw credential is never in scope and the mistake cannot recur. Adds a test asserting the constructor returns the wrapper; verified it fails when the wrapping is reverted.
The dataset extension's redaction did not cover everything, and the eval extension had none of it at all. Dataset: DownloadDataset was a fourth SAS-backed Do() that returned the raw *url.Error, and its caller logged that error. The earlier commit said it had covered "all three"; there are four. Eval: this extension carries a near-duplicate dataset_api that still logged a container SAS through url.URL.Redacted, which masks a userinfo password and leaves the query, where sig lives, intact. Four SAS-backed Do() calls also returned the SAS to the user. Two more Redacted() log sites in the paging code used the same unsafe idiom on project-endpoint links. Rather than copy the two helpers a third time, they now live in internal/urlsafe and both packages use them. Tests pin the premise as well as the fix, so Redacted() cannot quietly come back.
…change The previous commit was pushed with a failing test and cspell errors: urlsafe was not in the dictionary, and the conformance test still asserted the help names exit code 2.
Under azd up and azd deploy the eval line read the same whether the eval had just been created or was being reused, so a deploy could not answer "did this publish anything?" -- the question the change detection exists to answer. The spec shows the created/unchanged distinction for these steps, so the spec and the binary disagreed. EnsureEval now reports whether it created, and the service target words its line accordingly: "Created eval X (id)" or "Eval X is unchanged (id)". That also removes a heuristic from the direct command, which had been inferring the same thing by comparing the returned id against the recorded one. It now uses the fact. Verified against the shared project: first deploy prints Created, second prints unchanged with the same id, and azd ai eval create agrees with both.
Ported from azure.ai.dataset, where a review caught the same bug. This extension carries a near-duplicate dataset_api that still returned an empty version for every listing error, so a 403, throttle or timeout made an existing dataset restart at 1.0 and publish over or collide with a version that was already there. Only a 404 and an empty listing now mean versionless; everything else propagates. The conflict retry keeps best-effort semantics, since the version it just had refused is already a correct next step. Tests cover both, and were checked by reverting the fix.
A side-by-side diff of the duplicated packages, after the same class of bug had been fixed twice, found the pagination had diverged in three places. A relative nextLink was refused: url.Parse leaves host and scheme empty, so the origin check rejected a legitimate link. It is now resolved against the endpoint first, which keeps the guarantee. Only an immediately self-referencing link ended the walk, so a two-page cycle ran to maxPages; a seen set ends any repeat. And both truncation exits were silent, which is how a stale latest-version gets chosen without anyone knowing.
Four follow-ups from the review and bug bash rounds. checkDatasetDrift tolerated a failed listing. That guard exists to catch a version someone else published, so a 403 or a timeout silently skipping it defeats the point; a listing we could not read is not evidence there was none. latestDatasetVersion now returns an error, and only a 404 or an empty listing still mean "nothing registered". This reverses a documented decision. The old test was named ToleratesAFailedListing and its comment argued an unreachable project should not fail the deploy. That trade is now the other way: fail closed. The test and its comment say so rather than being left stale. init reported "Created evals/azure.eval.yaml" over a file that already existed and was only appended to, which reads like it was overwritten. It now says "Updated" and marks the config line "(eval added)". run output list showed pass/fail per sample and no scores, so a bare pass and a perfect one looked identical and the only way to tell was the portal. It now carries a SCORE column with the sample's mean.
Surface a failed drift check, and say what the output means
The pagination fixes went in unguarded, which is the same asymmetry that let the two copies drift in the first place. Three tests, each checked by reverting its fix: a relative nextLink is followed rather than refused; a link resolving to another host is refused and that host is never contacted; and a two-page cycle ends the walk. Without the cycle fix the last one runs long enough to blow a 60s test timeout, so it was a hang, not a slow path.
This is the paginator that lists evals, runs and evaluators, so it is the more exercised of the two and it had no cover for the three fixes. Same three cases as dataset_api, each checked by removing the guard it tests: a relative nextLink is followed; a protocol-relative link resolving to another host is refused and that host is never contacted, so resolving cannot become a bypass; and a two-page cycle ends the walk rather than running to maxPages, which without the guard blows a 45s timeout.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #9549
Internal work item: https://msdata.visualstudio.com/Vienna/_workitems/edit/5363979/
Adds
azd ai eval— define and run Foundry evaluations from the terminal.Replaces #9339, which pointed at a branch missing ~3,400 lines of eval work that had only ever landed downstream, including a fix for a bug that 404'd any dataset whose name contained a space.
Surface
Notable
One
generate, with a selector.generatesubmits both generation jobs in parallel;--dataset/--evaluatornarrows to one. Each reports its own outcome, the command exits non-zero if either failed, and whatever succeeded is still recorded.--no-waitresumes throughazd ai eval job, where the selector is required because a job id alone does not say which collection to poll.Runs know what they are scoring. The run data source is decided by configuration: a
source:block hands gathering to the service (traces, stored responses), otherwise rows come from the dataset andtarget:says what to invoke — an agent, a model, or nothing at all. Previously only agent targets existed, soinit --source traceswrote a configuration its ownrunrefused, and a model target was validated as legal then sent as an agent.initdetects rather than demands.--targetdefaults to the project's only agent, prompts when there are several, and names the flag under--no-prompt.Every user-facing string lives in one file (
internal/messages/messages.go) — so the whole voice of the CLI can be reviewed in one sitting. The move surfaced real duplicates, including three copies of one error and two contradictory--formatmessages where one was unreachable.The configuration is
azure.eval.yaml, prefixed for azd the wayazure.yamlis. An existingeval.yamlis still read and written back to, so nothing breaks and no project grows a second file. A directory holding both is refused:azure.yamlreferences one by name, so silently preferring one would mean editing one configuration whileazd updeployed the other.Waiting is bounded. A run that never reaches a terminal state used to hold the terminal open indefinitely. The wait now stops after two hours and is treated the way
--no-waitis — the run is still going server-side, so the caller gets the reattach line and exit 0 rather than an error about a failure that did not happen.Every
showrenders a detail view, with the machine-readable document behind-o json.run output showwas the last one emitting raw JSON whatever was asked for, which made the command a person reaches for after a failing listing the hardest one in the CLI to read.The end-to-end suites were not running
go test ./...reported a clean run for this extension while compiling none oftests/cli,tests/liveortests/hero— all three carry build tags, andgo list ./tests/...answers "matched no packages". CI type-checks them but never executes them, so 1,822 lines of end-to-end tests had gone unrun and were holding seven failures.Running them found
--format jsonlbeing refused by a guard that named only json and csv while the flag's own help offered jsonl;evaluator showmissing the--output-filethat the reconciliation error tells the reader to adopt a remote change with; and a missing PASS THRESHOLD column. The other four were stale assertions, re-pinned to the spec rather than to the implementation.Verified live
Against a real Foundry project, end to end:
azd ai eval init --source traceswith no other flags, then a run scoring 20 real conversations from Application Insights.--no-wait -o json, gate pass exit 0, gate breach exit 1).azd, which is where the remaining not-found and--output-filedefects came from.Green: 524 unit tests,
tests/cli66,tests/live5,tests/hero8,go vet -tags live,hero,gofmt, andci-test.ps1exit 0.Reviewed
Nine model-assisted review passes (Sonnet 4.6, GPT-5.3-Codex, Gemini 3.1 Pro, Grok 4.5, GPT-5.6 Sol, GPT-5.6 Terra, GPT-5.6 Luna) found real defects that are fixed here — among them dataset job commands calling their collection on the wrong API version,
--no-wait -o jsonlosing the job ids it exists to report, a declared target with no name being silently scored as no target, two nil-deref panics in the endpoint cascade, andevaluator show --output-filetruncating the local definition it was meant to update if the write failed.Not included, deliberately
No
registry.jsonentry and nomicrosoft.foundrybundle dependency — both need published artifacts, and listing an unpublished extension in the bundle breaksazd extension install microsoft.foundryfor everyone.Known gaps
initdoes not yet prompt for evaluators or judge model; it detects sensible values instead. The spec calls for prompts with an error under--no-prompt. Not changed here because the--evaluatordefault also schedules rubric generation, which is what makes theinit→generateflow work; reshaping that is broader than this PR.dataset versions liston an unknown name lists nothing and exits 0 here, while the dataset extension (Add azd ai dataset extension for Foundry datasets #9499) answers the same question with an error. A list is a filter rather than a lookup, so empty-and-successful is defensible and two tests depend on it — but the two extensions should agree. Raised for reviewers rather than settled unilaterally.main.goreports 1 unless the error chain carries an*internal.ExitCodeError, which is constructed only on the hooks path. Spec Scenario 5 wants 2 for "regressed" versus 1 for "could not run". The fix is in azd core, not here.Draft — not requesting reviewers yet.