Skip to content

fix: validate unified agent definitions in doctor#9254

Merged
huimiu merged 1 commit into
mainfrom
hui/fix-doctor-agent-definitions
Jul 23, 2026
Merged

fix: validate unified agent definitions in doctor#9254
huimiu merged 1 commit into
mainfrom
hui/fix-doctor-agent-definitions

Conversation

@huimiu

@huimiu huimiu commented Jul 22, 2026

Copy link
Copy Markdown
Member

The problem

Modern agent projects keep their agent definition inline in azure.yaml. Deploy and run already understand this, but azd ai agent doctor didn't — it still went looking for a standalone agent.yaml file on disk and failed the check when it couldn't find one.

The result: a perfectly valid project that deploys successfully would still get a red FAIL from doctor. That's misleading and makes people distrust the command (see #9223).

The fix

Point doctor's definition check at the same resolver that run and deploy use — project.LoadAgentDefinition, added in #9149 — instead of reading agent.yaml off disk itself.

The check now passes when a definition is found in any legitimate place:

  • inline in azure.yaml
  • referenced from azure.yaml
  • a legacy agent.yaml / agent.yml on disk

If a service has no definition anywhere, doctor still fails — but the message and suggestion no longer assume the old agent.yaml file.

Everything else about the check is unchanged: same check ID (local.agent-yaml-valid), same --local-only behavior, and it still aggregates failures across services in a stable, name-sorted order.

Why this approach

Reusing the runtime resolver keeps doctor honest: "doctor passes" now means "deploy will resolve the same definition," so the two can't quietly drift apart. It also avoids maintaining a second, doctor-only YAML parser that we'd have to keep in sync by hand.

One side effect worth calling out (manual-env check)

Making the definition check pass for inline-only projects has a knock-on effect on the next check, local.manual-env-vars:

  • Before: inline-only projects failed the definition check, which made the manual-env check skip.
  • Now: the definition check passes, so the manual-env check actually runs — but its extractor still only reads a standalone <service>/agent.yaml from disk. For inline projects it finds nothing and reports "no manual env vars are missing."

This is expected, not a regression introduced here. Teaching manual-env to read the resolved definition is separate work tracked in #8710. The existing skip-cascade guard is kept, so when the definition check itself fails or skips, manual-env still skips as before (now with matching wording).

Testing

Unit tests: go test ./internal/cmd/doctor ./internal/cmd pass, with new coverage for inline definitions, inline-over-stale-file precedence, and the service-name sort helper.

I also built the extension and ran azd ai agent doctor --local-only end-to-end against sample projects covering each definition source, to confirm azd really surfaces the inline definition to the check (not just hand-built fixtures):

Scenario agent definition valid result
Unified azure.yaml, inline definition, no agent.yaml on disk (the #9223 case) Pass
Inline definition + a broken legacy agent.yaml on disk Pass (inline wins)
Legacy agent.yaml-only service (no inline definition) Pass (disk fallback)
Agent service with no definition anywhere Fail, source-neutral message + suggestion
Two agent services both missing definitions Single aggregated Fail, listed in service-name order
Non-agent service alongside an inline agent Pass, only the agent service validated

On the running extension I also confirmed the check ID and --local-only behavior are unchanged, the failure suggestion no longer tells modern projects to create an agent.yaml, and the manual-env behavior change above is observable.

Closes #9223.
Part of #8710.

@azure-pipelines

This comment was marked as outdated.

@huimiu
huimiu marked this pull request as ready for review July 22, 2026 09:50
Copilot AI review requested due to automatic review settings July 22, 2026 09:50
@huimiu
huimiu requested a review from hund030 as a code owner July 22, 2026 09:50
@azure-pipelines

This comment was marked as outdated.

@github-actions

Copy link
Copy Markdown

📋 Prioritization Note

Thanks for the contribution! The linked issue isn't in the current milestone yet.
Thank you for logging this issue; our team is reviewing it. If you need urgent prioritization, tag @RickWinter and @kristenwomack to let us know.

@github-actions github-actions Bot added the ext-agents azure.ai.agents extension label Jul 22, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates doctor validation to use the same unified agent-definition resolver as runtime and deployment.

Changes:

  • Supports inline, referenced, and legacy definitions.
  • Uses source-neutral diagnostic wording.
  • Expands tests for resolution precedence and multi-service failures.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
checks_project.go Resolves and validates unified definitions.
checks_project_test.go Adds definition-resolution test coverage.
checks_manual_env.go Updates skip wording.
checks_manual_env_test.go Updates skip assertion.
checks_local.go Registers the renamed check.
checks_local_test.go Updates check metadata assertion.
doctor_format_test.go Updates capitalization coverage.

@jongio jongio left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The switch to project.LoadAgentDefinition here only inspects the returned error, so an inline definition whose kind is anything other than hosted passes this check without being validated. In agentDefinitionFromStruct (internal/project/agent_definition.go:596-598) a non-hosted kind returns (zero, false, nil) before the validation block runs. That covers both workflow (a valid kind, whose name check is then skipped) and unknown values like nonsense.

The retained disk fallback still validates strictly through ValidateAgentDefinition, which rejects unknown kinds, so an invalid kind now fails when it lives in a legacy agent.yaml on disk but passes when it is inline in azure.yaml. Since doctor is a validation command, worth deciding whether it should reject a resolved-but-unvalidated non-hosted inline definition, or whether validating unknown/workflow kinds belongs in the shared resolver (which would also cover run and deploy). Not blocking, and the fuller fix is arguably out of scope for this alignment change.

@huimiu
huimiu merged commit af791ac into main Jul 23, 2026
38 checks passed
@huimiu
huimiu deleted the hui/fix-doctor-agent-definitions branch July 23, 2026 02:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ext-agents azure.ai.agents extension

Projects

None yet

Development

Successfully merging this pull request may close these issues.

azd ai agent doctor requires deprecated standalone agent.yaml

4 participants