Expand ai-scoping auth gate: external URL shape, public/confidential client, IdP authorize-params notes

[lorr1]

Context

Follow-up from PR #99 review (see #99 (comment) and #99 (comment)).

The current ai-scoping auth gate (Step 6.1 in skills/ai-scoping/SKILL.md) only captures information about the upstream OAuth provider being wrapped. Two classes of information that the user usually knows at scoping time — and that silently bake PoC-specific assumptions into the generated manifests when not asked — are missing:

1. External URL shape — the URL under which the ToolHive-issued auth endpoint will be reachable in the target cluster. This value is used in five interlocked places in the generated manifests: mcpexternalauthconfig.yaml (embeddedAuthServer.issuer and upstreamProviders[0].*.redirectUri), mcpserver.yaml (oidcConfigRef.audience and oidcConfigRef.resourceUrl), ingress.yaml (rules[0].host + path), and mcpoidcconfig.yaml (spec.inline.issuer). Templates currently emit https://mcp.REPLACE_ME_DOMAIN/{server_name} — a PoC-cluster-specific shape that cannot be corrected by domain substitution alone, because other clusters use patterns like https://<domain>/<server_name>/mcp, https://<server_name>.<domain>/mcp, etc. See PR Align mcp-scope auth schema with OAuth2/OIDC reality #99 comment for context.

2. Public vs. confidential OAuth client — whether the upstream OAuth app uses PKCE alone (public client) or requires a client_secret (confidential client). Templates currently always emit clientSecretRef as a commented-out example block and instruct the user to uncomment it if confidential. This is conservative but wrong for the common case: Google (web apps), GitHub, Slack, and most enterprise OAuth apps are confidential. The user knows this at scoping time; asking lets us render the clientSecretRef block live.

Additionally, two notes-only captures would reduce foot-guns in Phase 2 review without touching the generated CRDs:

3. IdP-specific authorize-endpoint parameters beyond scopes — some upstreams require extra parameters in the /authorize call that OpenAPI does not model. Examples: Atlassian requires audience=api.atlassian.com or the response is ID-token-only; Slack requires user_scope for user-context tokens. These do not map to CRD fields but are critical to document so the Phase 2 reviewer catches them.

4. Redirect-URI registration reminder — the emitted redirectUri in mcpexternalauthconfig.yaml must be registered with the upstream IdP (OAuth consent screen on Google, app registration on GitHub, etc.) before the first login works. Scoping could surface this as: "Before deploy, register <emitted redirectUri> with <provider> at <console URL>."

Scope

In scope:

• Add two new gates to skills/ai-scoping/SKILL.md Step 6.1 covering items 1 and 2 above.
• Extend the OAuth2Auth and OIDCAuth pydantic variants in src/mcp_builder/schema/models.py with two new optional fields (see field names below).
• Thread the new fields through src/mcp_builder/generate/plan.py (pass-through, same as existing auth fields).
• Update the four Jinja templates (authconfig_embedded_oauth2.yaml.jinja2, authconfig_embedded_oidc.yaml.jinja2, mcpserver.yaml.jinja2, mcpoidcconfig.yaml.jinja2, ingress.yaml.jinja2) to honor the external URL shape when present, and to render clientSecretRef live (not commented) when client_type == confidential.
• Update skills/ai-scoping/assets/scoping-summary-template.md to surface the new fields in the summary.
• Update skills/deploy-assist/SKILL.md Step 4 substitution rules to recognize the new placeholder(s) OR to detect when the scope already baked in a concrete URL and skip domain substitution.
• Update existing e2e/fixtures/real/*.yaml fixtures (and the unit test fixtures test_scope_oauth.yaml, test_scope_oidc.yaml) to include the new optional fields where reasonable defaults apply. Where the target deployment is unknown, the field stays absent.
• Update tests: unit tests for the new schema fields (valid + cross-variant rejection stays working), integration tests verifying the generated manifests render correctly with and without the new fields.

Out of scope (explicitly):

• Items 3 and 4 (notes-only captures). Those are prompt-only additions to the auth gate — the schema already has an auth.notes: str | None free-text field. They should be folded into the SKILL.md gate prose in the same PR but do not need schema or template changes.
• Changing the ingress controller annotations (AWS ALB-specific alb.ingress.kubernetes.io/*, ingressClassName: alb, group.name: mcp-servers). Those are a separate ingress-abstraction concern and should be addressed by a dedicated issue — leave them for now.
• Per-tool scope matrix (tool → scope mapping) — YAGNI; the OpenAPI security: [...] operation-level overrides already model this at the spec layer.
• Configurable token lifespans (accessTokenLifespan, refreshTokenLifespan, authCodeLifespan in the auth-config templates). Hardcoded 1h/168h/10m works for now.
• Multi-tenant deployment model (one server per tenant vs. shared). That is a deploy-time decision, not a scoping one.
• audience/resourceUrl overrides in mcpserver.yaml that differ from the external URL. Rarely needed; add only if a use case materializes.

Detailed design

Schema additions

Add two optional fields to both OAuth2Auth and OIDCAuth in src/mcp_builder/schema/models.py. Do not add them to APIKeyAuth or NoAuth (neither has an OAuth client registration concept). extra="forbid" on the variants means a user who writes external_url_template under api_key gets a validation error — that is the desired behavior.

external_url_template: str | None = None
# The full URL template under which this ToolHive-issued auth endpoint
# will be reachable from OAuth clients, using the literal placeholder
# `<server_name>` (no braces) where the server name should be inserted.
# Examples:
#   "https://mcp.example.com/<server_name>"
#   "https://example.com/<server_name>/mcp"
#   "https://<server_name>.example.com/mcp"
# Optional; if absent, templates render a REPLACE_ME placeholder and the
# deploy-assist skill (or the user manually) rewrites at deploy time.

client_type: Literal["public", "confidential"] | None = None
# Whether the OAuth client registered with the upstream identity
# provider is a public client (PKCE only, no client_secret) or a
# confidential client (uses client_secret). When "confidential", the
# generated MCPExternalAuthConfig renders clientSecretRef live (pointing
# at a Secret the user must create). When "public" or None, the block
# is rendered as a commented example.

Rationale for the placeholder literal <server_name> (angle brackets, not {}): pydantic does not treat it as a template, and it survives yaml.safe_load round-trips without quoting concerns. The renderer substitutes it with plan.server_name at render time.

Validator: when external_url_template is present, require it to (a) start with https://, (b) contain the literal substring <server_name> exactly once. Implement as a model_validator(mode="after") on each variant.

Scoping prompt additions

In skills/ai-scoping/SKILL.md Step 6.1, after the existing auth USER GATE and before "Scopes — always populate scopes_available from the spec", add two sub-gates. Both should be optional (user can press enter to defer). Exact prose:

**External URL shape (optional).** The generated MCPExternalAuthConfig, MCPServer, and Ingress manifests all reference the same external URL — the URL under which the ToolHive-issued auth endpoint will be reachable in the target cluster. If you know where this server will deploy, give the URL template using `<server_name>` (angle brackets, no substitution) as a placeholder. Examples:

- `https://mcp.example.com/<server_name>`  — shared host, path-per-server
- `https://example.com/<server_name>/mcp`  — path-at-root
- `https://<server_name>.example.com/mcp`  — subdomain-per-server

If you don't know the target cluster yet, press enter. Generated manifests will emit a `REPLACE_ME` placeholder and the `deploy-assist` skill (or a manual edit pass) fills it in from the target cluster's conventions.

Write the concrete template to `auth.external_url_template` in the scope YAML. Skip this field entirely when deferred — do not emit an empty string.

**OAuth client type (optional).** Does the OAuth app you registered (or plan to register) with the upstream identity provider use PKCE alone (public client, no client secret) or require a client secret (confidential client)?

- Most web-app and server-side OAuth clients are **confidential** (Google web apps, GitHub, Slack, Okta, Keycloak). Choose this if the IdP issued you a client_secret when you registered the app.
- SPAs, mobile apps, and some modern SDKs use **public** clients with PKCE only.

If you don't know, press enter. Generated manifests render `clientSecretRef` as a commented example; the deployer uncomments if needed.

Write to `auth.client_type` as either `"public"` or `"confidential"`. Skip the field when deferred.

**Note on IdP-specific authorize parameters.** Some upstreams require extra parameters in the /authorize call that are not modeled by OpenAPI (Atlassian: `audience=api.atlassian.com`; Slack: `user_scope`; etc.). If you know of any, add them to `auth.notes` — they become breadcrumbs for the Phase 2 human reviewer.

**Note on redirect URI registration.** The emitted `redirectUri` in `mcpexternalauthconfig.yaml` must be registered with the upstream IdP before the first login works (OAuth consent screen on Google, app registration page on GitHub/Atlassian/etc.). If you picked an `external_url_template` above, compute the redirect URI as `{substituted_external_url}/oauth/callback` and include a one-line "before deploy, register X with Y at Z" reminder in `auth.notes`.

Template updates

Thread new context variables through src/mcp_builder/generate/renderers/manifests.py:

• _render_embedded_oauth2 and _render_embedded_oidc take auth.external_url_template and auth.client_type, substitute <server_name> with plan.server_name to produce a concrete external_url, and pass both external_url (or None) and client_type into the Jinja rendering context. Add a new helper _substitute_external_url(template: str | None, server_name: str) -> str | None colocated with _derive_provider_name.
• Similarly wire external_url through render_mcpserver (used for audience/resourceUrl), render_mcpoidc_config (used for issuer), and render_ingress (parse the template to derive host + path, which is more involved — see below).

In the four templates that currently emit https://mcp.REPLACE_ME_DOMAIN/{{ server_name }}, replace that literal with a Jinja conditional:

{% if external_url %}{{ external_url }}{% else %}https://mcp.REPLACE_ME_DOMAIN/{{ server_name }}{% endif %}

Factor this into a Jinja macro ({% macro external_url_or_placeholder(url, server_name) %}...{% endmacro %}) in a shared _macros.jinja2 file to avoid duplicating the conditional five times.

For ingress.yaml.jinja2, the host and path come from parsing the concrete external URL. Implement a helper in manifests.py:

def _split_external_url(url: str) -> tuple[str, str]:
    """Split an external URL into (ingress_host, ingress_path).

    Examples:
        "https://mcp.example.com/google-drive" -> ("mcp.example.com", "/google-drive")
        "https://example.com/google-drive/mcp" -> ("example.com", "/google-drive/mcp")
        "https://google-drive.example.com/mcp" -> ("google-drive.example.com", "/mcp")
    """

Then pass both host and path into render_ingress. If external_url is None, fall back to host="mcp.REPLACE_ME_DOMAIN" and path="/{{ server_name }}" (current behavior).

For clientSecretRef: currently rendered as a commented block. When client_type == "confidential", render live:

{% if client_type == "confidential" %}
        clientSecretRef:
          name: {{ server_name }}-oauth-secret
          key: client-secret
{% else %}
#   - clientSecretRef: uncomment ...
{% endif %}

When client_type == "confidential", also emit a new deploy/secret-oauth.yaml (a K8s Secret template with name: {{ server_name }}-oauth-secret, stringData: { client-secret: REPLACE_ME }). Add a new Jinja template secret_oauth.yaml.jinja2 and plumb it through render_manifests. Update deploy-assist SKILL.md to flag this file for manual fill-in.

Fixture updates

• tests/unit/fixtures/test_scope_oauth.yaml and test_scope_oidc.yaml: add auth.external_url_template: "https://mcp.example.com/<server_name>" and auth.client_type: "confidential" so the new code paths are exercised.
• Add a second pair of fixtures (test_scope_oauth_public.yaml, test_scope_oidc_defer.yaml) that exercise the client_type: public and missing-external_url_template paths.
• e2e/fixtures/real/*.yaml: add client_type for every OAuth2/OIDC fixture based on the real IdP's documentation (Google: confidential; GitHub: confidential; Slack: confidential; Spotify: confidential; Zoom: confidential; BambooHR: confidential; Jira: confidential — almost all enterprise OAuth apps are). Do not add external_url_template to e2e fixtures — those are deployment-neutral.

Test updates

• tests/unit/test_schema.py: add a TestExternalUrlTemplate class with cases for valid template, missing <server_name>, double <server_name>, non-https scheme, and rejection when set on APIKeyAuth/NoAuth. Add a TestClientType class with cases for "public", "confidential", None, and invalid strings.
• tests/integration/test_cli.py: extend TestRunPipelineOAuth and TestRunPipelineOIDC with assertions that (a) when external_url_template is set, the generated mcpexternalauthconfig.yaml has the concrete issuer and redirectUri, the mcpserver.yaml has the concrete audience/resourceUrl, the mcpoidcconfig.yaml has the concrete issuer, and the ingress.yaml has the concrete host+path; (b) when client_type: confidential, the clientSecretRef block is rendered live and deploy/secret-oauth.yaml is emitted.

Deploy-assist updates

In skills/deploy-assist/SKILL.md:

• Step 4 mcpexternalauthconfig.yaml / mcpserver.yaml / mcpoidcconfig.yaml / ingress.yaml handling: before substituting REPLACE_ME_DOMAIN, check whether the file already has a concrete URL (no REPLACE_ME_* token in the relevant fields). If it does, skip the domain substitution for that field and treat it as user-authoritative.
• Step 4 secret-oauth.yaml (new): if this file is present, note it in the "What the user still needs to do" checklist as a secret requiring a real client-secret value from the upstream IdP.
• Step 5 placeholder assertions: extend the allowed-exceptions list to cover client-secret: REPLACE_ME in secret-oauth.yaml (parallel to token: REPLACE_ME in secret.yaml and clientId: REPLACE_ME in mcpexternalauthconfig.yaml).

Acceptance criteria

• task check green with new schema + templates + tests.
• Scoping a real OIDC fixture (e.g., google_drive) with external_url_template: "https://mcp.example.com/<server_name>" and client_type: "confidential" produces generated manifests where:
  • mcpexternalauthconfig.yaml issuer = https://mcp.example.com/google-drive, redirectUri = https://mcp.example.com/google-drive/oauth/callback, clientSecretRef block is live.
  • mcpoidcconfig.yaml issuer = https://mcp.example.com/google-drive.
  • mcpserver.yaml audience = resourceUrl = https://mcp.example.com/google-drive.
  • ingress.yaml host = mcp.example.com, path = /google-drive.
  • deploy/secret-oauth.yaml exists with stringData.client-secret: REPLACE_ME.
• Scoping the same fixture with neither field set produces the current PoC-flavored default URLs (unchanged behavior, backward-compatible).
• Cross-variant rejection: external_url_template under api_key auth is a validation error at scope-load time.
• Scoping summary (scoping-summary.md) surfaces both new fields under the "Auth Detection" section when present.

Non-goals

See "Out of scope" above. Explicitly calling out that this issue does not touch ingress controller annotations, token lifespans, or multi-tenant deployment models — those are tracked separately (or not tracked, if not justified by use cases yet).

References

• PR Align mcp-scope auth schema with OAuth2/OIDC reality #99: Align mcp-scope auth schema with OAuth2/OIDC reality #99
• Review thread that prompted this: Align mcp-scope auth schema with OAuth2/OIDC reality #99 (comment)
• PR Align mcp-scope auth schema with OAuth2/OIDC reality #99 reply discussing the scoping extension idea: (will be added once this issue is created and linked back)