refactor(deploy): use durable jobs and CLI polling

[bnema]

Incident and root cause

gordon push currently performs an image upload and then holds POST /admin/deploy/:domain open while Gordon pulls, prepares attachments/network/env/volumes, starts a candidate, waits for readiness, activates it, stabilizes it, and schedules old-container cleanup.

A production deployment exceeded the smart-TCP HTTP server's 10-second write timeout. The request context was canceled during readiness. Failed-candidate cleanup reused that canceled context, leaving a healthy -new container; the next push failed because the candidate name was occupied.

Increasing HTTP timeouts is not a sufficient fix. Deployment is a durable server-side operation and must not be owned by a client connection.

Decision

Replace synchronous remote deployment with persisted server-side deployment jobs and short polling requests.

• A CLI disconnect stops observation only; the server job continues.
• Jobs survive Gordon restart and are reconciled against runtime state.
• Repeated requests for the same domain and immutable digest coalesce onto the active job.
• A different digest for a busy domain returns 409 with the active job ID.
• All deployment producers use one admission path; no event handler or adapter may call ContainerService.Deploy directly.

Required architecture

1. Deployment job model

Add a versioned DeploymentJob entity containing:

• UUID and revision
• authenticated requester subject
• domain, repository, requested manifest digest, selected platform and resolved image ID
• trigger/source (cli-push, manual, direct-registry-push, auto-route, startup, config-change)
• namespaced idempotency key
• config generation captured at admission
• status: queued, running, succeeded, failed, canceled, interrupted
• durable stage/checkpoint: queued, preparing, candidate-created, candidate-started, ready, activating, activated, stabilizing, cleanup-pending, complete
• timestamps, attempt count, candidate/old/result container IDs
• allowlisted terminal error code plus redacted {summary,cause,hint}

Never persist raw logs, credentials, environment values, secret values, or arbitrary wrapped errors.

2. One deployment-admission path

Introduce a DeploymentManager use case. Every source must submit through it:

• CLI gordon deploy
• CLI-managed gordon push
• direct registry image.pushed
• auto-route image handling
• startup/autostart recovery
• config/secret-triggered redeploys
• local control-plane commands

ImagePushedHandler, AutoRouteHandler, HTTP handlers, and startup recovery must not call ContainerService.Deploy directly after this change. Keep the existing per-domain deploy lock as defense in depth, not as admission control.

Use a bounded queue and worker pool (default concurrency 1), one active reservation per domain, explicit queue capacity, and 429 when admission is full. Different domains are scheduled fairly; same-domain work never overlaps.

3. Persisted manager/store

Add a narrow deployment-job store boundary and a filesystem adapter under server.data_dir.

The store must provide one process-atomic transaction/CAS for:

• idempotency lookup
• per-domain reservation
• job creation
• state/checkpoint transition
• queued cancellation

Persist queued state before waking workers. Use versioned JSON, process locking, temp write, file fsync, atomic rename, and parent-directory fsync. Define corruption handling: keep the last valid backup, fail closed for deployment admission, and keep already-running production containers untouched.

Terminal retention: 7 days and at most 1,000 records. Prune only terminal jobs. Idempotency keys expire with their retained job; a request after pruning creates a new job.

4. Verified immutable image identity

Every admitted job must contain a server-verified repository@sha256:... target. Never defer mutable-tag resolution until a queued job executes.

Required pipeline changes:

• Registry manifest storage emits repository, tag, manifest digest, media type, and platform metadata.
• CLI/native push returns the digest accepted by the server.
• The server verifies that a submitted digest exists in its registry and belongs to the configured repository.
• OCI index jobs record both the requested index digest and the selected platform manifest/image ID.
• Worker execution uses a digest-pinned route/image reference.
• A digest that is absent, belongs to another repository, or cannot resolve for the requested platform is rejected before job creation.

5. Replace timer-based deploy suppression with persisted push intents

The current image-name timer suppression is not durable and races unrelated pushes. Replace it with persisted push intents:

1. Before native upload, CLI creates a push intent scoped to authenticated subject + repository, with mode pending-confirmation or no-deploy.
2. Registry upload/manifest requests carry the intent ID.
3. Manifest PUT validates intent ownership/repository and atomically records the accepted digest on that intent.
4. --no-deploy consumes the intent without creating a job and cannot accidentally auto-deploy.
5. After interactive confirmation, CLI asks the server to deploy the digest recorded on its intent; the server creates/coalesces the deployment job.
6. If CLI exits after upload but before confirmation, the persisted intent remains non-deploying and expires after a bounded TTL.
7. A direct Docker/OCI push without an intent submits its stored digest through DeploymentManager according to auto-deploy policy.
8. Intent consumption and job admission are atomic so a crash cannot both suppress and lose a deploy or create duplicate jobs.

Remove repository-wide timer suppression after all producers use this correlation model.

6. HTTP contract

Gordon is v0.x; replace the synchronous deployment contract rather than maintaining two paths.

POST /admin/deploy/:domain

• requires config:write (or a dedicated deployments:write introduced consistently)
• accepts a server-verifiable digest and Idempotency-Key
• namespaces the key by authenticated subject + domain + repository
• rejects reuse of the same key with different immutable input
• returns 202 {id,status,domain,target_digest,status_url} immediately
• returns the original job for an identical repeated request
• returns 409 {active_job_id} for a different digest while the domain is reserved

GET /admin/deployments/:id

• requires an explicit deployment read permission
• requester can read its own job
• cross-requester visibility is denied unless the subject has an explicit admin-wide permission
• returns stage/status/timestamps/result and allowlisted redacted failure details

GET /admin/deployments?domain=:domain&active=true

• uses the same ownership policy
• lets a client rediscover a job when submit response was lost

DELETE /admin/deployments/:id

• requester plus write permission required
• queued-only cancellation via CAS
• running/terminal cancellation returns 409

Do not expose repository names, image digests, causes, or another requester's job to generic status-only callers unless explicitly authorized.

7. CLI submit + poll

Refactor gordon deploy and the post-push phase:

1. submit once with immutable input and idempotency key;
2. print the job ID immediately;
3. poll short GET requests with bounded backoff (for example 500 ms to 2 s);
4. update spinner text from durable job stage;
5. retry temporary status transport failures without resubmitting;
6. preserve --json with stable job and terminal result shapes;
7. on Ctrl-C/network loss, print the job ID and state that server work continues;
8. add gordon deployments status <job-id> --watch to resume observation;
9. add --no-wait only after the base contract is complete.

8. Rollout executor and durable checkpoints

Do not attempt restart recovery from temporary container names alone. Extract a rollout executor/recovery boundary from the current synchronous container service.

The executor must:

• keep existing pull/readiness/zero-downtime/rollback behavior
• attach job ID, requested digest, selected image ID, domain, and rollout role labels to candidates and promoted containers
• durably checkpoint before candidate create/start, before activation, after activation, and after old-container ownership is resolved
• expose stage callbacks to DeploymentManager
• use an authoritative activation proof (canonical container identity plus proxy target/cache generation), not only container name or SyncContainers
• make each recovery step idempotent

No job becomes succeeded while cleanup/ownership is unresolved. Use cleanup-pending and retry bounded detached cleanup.

9. Restart reconciliation and startup order

Required startup order:

1. load and validate deployment store;
2. sync runtime containers without making deployment decisions;
3. reconcile every non-terminal job from durable checkpoints and job/container labels;
4. start deployment workers and rescan persisted queued jobs;
5. run AutoStart only for domains without a non-terminal job; AutoStart itself submits through DeploymentManager.

Recovery rules:

• canonical healthy container already on target digest with matching activation proof → mark succeeded or continue cleanup checkpoint;
• old canonical active plus owned healthy candidate → resume activation only when checkpoint/labels prove safety;
• old canonical active plus missing/unusable candidate → clean owned candidate state with detached context and requeue same job;
• no safe unambiguous ownership/activation proof → mark interrupted, preserve active production container, require explicit retry;
• never choose among containers using -new/-next names alone;
• never destroy the currently active production container while resolving ambiguity.

10. Configuration generation semantics

Jobs do not persist secret values. Add/derive a monotonic deployment-config generation covering route, attachment, volume, environment-key/secret generation, and relevant runtime config.

• Record generation at admission.
• Resolve values only inside the worker.
• If generation changed before execution/recovery, fail with stale_configuration and require a new job; do not silently deploy different configuration under the original idempotency key.
• Changes that themselves require deployment submit a new manager job and obey per-domain admission.

11. Cancellation, shutdown, and cleanup

• Client cancellation never cancels server work.
• Running user cancellation is not supported in v0.
• On graceful Gordon shutdown, stop admission, persist state, allow a bounded worker drain, then mark unresolved running jobs interrupted for startup reconciliation.
• Failed-readiness and orphan cleanup use a detached lifecycle context with a fixed timeout, not the request/job context.
• Persist cleanup-pending and retry; cleanup is idempotent.
• Capture/redact diagnostics before cancellation when possible.

Alternatives rejected

1. Increase HTTP timeouts: still client-owned and proxy/disconnect-sensitive.
2. In-memory jobs: lose status, idempotency, and recovery state on restart.
3. General in-memory event bus as queue: no durable correlation/results and existing handlers can bypass admission.
4. Automatic replacement of an active same-domain job: cancellation/rollback semantics are too risky for v0.

Acceptance criteria

Admission and identity

• Every deployment producer submits through DeploymentManager; no direct ContainerService.Deploy calls remain outside the rollout worker.
• Direct registry push and CLI push integration tests prove they cannot bypass per-domain admission.
• Every job uses a server-verified digest-pinned target with OCI index/platform behavior tested.
• Same subject/domain/repository/key/input is idempotent; changed input with same key is rejected.
• Same domain+digest coalesces; different digest while busy returns 409 with active job ID.
• Queue capacity/overload and fairness are tested.

Client/API

• Submit returns 202 before pull/readiness completes.
• CLI disconnect does not cancel deployment and --watch can resume it.
• Temporary poll failures do not resubmit.
• Ownership/read/cancel permissions and cross-requester denial have tests.
• Persisted/API errors use allowlisted codes and redacted fields only.

Push correlation

• --no-deploy cannot emit an auto-deploy job.
• CLI confirmation, direct push, duplicate manifest delivery, intent expiry, and crash between manifest storage/intent consumption/job admission are covered.
• Timer-based repository suppression is removed.

Recovery and safety

• Startup follows store → sync → reconcile → workers → filtered AutoStart ordering.
• Restart is tested during prepare, candidate start, readiness, pre-activation, post-activation, and cleanup-pending.
• Promoted target is recognized without redeploy.
• Ambiguous state becomes interrupted without touching the active production container.
• Recovery uses durable checkpoints and job/container labels, not temporary names alone.
• Config-generation changes produce stale_configuration.
• Canceled request/readiness/shutdown cleanup leaves no unowned -new/-next candidate.

Regression gates

• Existing zero-downtime, readiness, rollback, per-domain lock, redundant-deploy, registry, and startup-recovery tests remain green.
• New manager/store/API/CLI/push-intent/recovery tests pass under go test -race.
• golangci-lint run ./... and go vet ./... pass.

Likely implementation areas

• internal/domain: jobs, stages, checkpoints, push intents, error codes
• internal/boundaries: manager, rollout executor, job/push-intent store ports and generated mocks
• internal/usecase: admission, worker, recovery, retention, push correlation
• internal/adapters/out/filesystem: versioned atomic store
• internal/adapters/in/http/admin: async job and intent API
• internal/adapters/in/cli/remote: submit/status/list/cancel/intent methods
• internal/adapters/in/cli: deploy/push polling, watch command, JSON output
• internal/usecase/container: executor extraction, labels, checkpoints, detached cleanup
• registry manifest/push paths: digest/platform metadata and intent correlation
• startup/app wiring: recovery order, lifecycle context, drain