Design Reconciliation Engine for Deployment Lifecycle

[ilackarms]

Purpose

Replace the imperative deployment lifecycle with a KRT-driven reconciliation engine. The system moves from "API handler calls adapter synchronously on the request path" to "API writes intent to DB, async reconciler converges actual state toward desired state."

Architecture Overview

Three layers

┌─────────────────────────────────────────────────────┐
│  API Server — dumb CRUD + watch                     │
│  Writes intent to DB, reads state, streams changes. │
│  Zero business logic.                               │
└──────────────────────┬──────────────────────────────┘
                       │ reads/writes
                       ▼
┌─────────────────────────────────────────────────────┐
│  PostgreSQL — source of truth + event bus            │
│  pg LISTEN/NOTIFY triggers on all watched tables.   │
│  DB is the ONLY communication channel between       │
│  API server and reconciler.                         │
└──────────┬───────────────────────────┬──────────────┘
           │ NOTIFY                    │ NOTIFY
           ▼                           ▼
┌──────────────────────┐  ┌───────────────────────────┐
│  KRT Reconciler      │  │  Discovery Writers        │
│  All logic here.     │  │  Per-platform, async.     │
│  DB watches feed     │  │  Write raw actual state   │
│  KRT collections.    │  │  to platform tables.      │
│  Transformations     │  │  No normalization.        │
│  derive reconcile    │  │                           │
│  requests. Handlers  │  │                           │
│  execute adapters.   │  │                           │
└──────────────────────┘  └───────────────────────────┘

Data flow

User intent (API)
    │
    ▼
deployments table (desired_state: deployed/undeployed)
    │
    │ pg NOTIFY
    ▼
KRT StaticCollection[Deployment]
    │
    │ + StaticCollection[Provider]
    │ + StaticCollection[Agent/Server]  (dependency resolution)
    │ + StaticCollection[Discovered*]   (per-platform actual state)
    │
    ▼
KRT Transformation: desired vs actual → ReconcileRequest
    │
    │ (only items where desired ≠ actual)
    ▼
KRT Leaf Handler (Register)
    │
    ├─ Execute adapter.Deploy() / adapter.Undeploy()
    ├─ Write ReconcileEvent to DB
    ├─ Update deployment status in DB
    │
    │ pg NOTIFY (from DB writes)
    ▼
API watch clients see updated status

System diagram

flowchart TB
    subgraph api [API Server — dumb CRUD + watch]
        apiWrite[Write intent to DB]
        apiRead[Read state from DB]
        apiWatch[Watch via pg LISTEN/NOTIFY]
    end

    subgraph db [PostgreSQL — source of truth + event bus]
        deployments[(deployments)]
        reconcileEvents[(reconcile_events)]
        providers[(providers)]
        artifacts[(agents / servers)]
        discoveredLocal[(discovered_local)]
        discoveredK8s[(discovered_kubernetes)]
        pgNotify[pg LISTEN/NOTIFY]
    end

    subgraph reconciler [KRT Reconciler]
        collections[StaticCollections\nfed by DB watches]
        transforms[Transformations\ndesired vs actual]
        handlers[Leaf handlers\nexecute adapters]
    end

    subgraph discovery [Discovery Writers]
        localDisc[Local]
        k8sDisc[Kubernetes]
    end

    subgraph adapters [Platform Adapters]
        localAdapt[Local]
        k8sAdapt[Kubernetes]
    end

    apiWrite --> deployments
    apiRead --> deployments
    apiWatch --> pgNotify

    deployments --> pgNotify
    reconcileEvents --> pgNotify
    providers --> pgNotify
    artifacts --> pgNotify
    discoveredLocal --> pgNotify
    discoveredK8s --> pgNotify

    pgNotify --> collections
    collections --> transforms
    transforms --> handlers

    handlers --> localAdapt
    handlers --> k8sAdapt
    handlers --> reconcileEvents
    handlers --> deployments

    localDisc --> discoveredLocal
    k8sDisc --> discoveredK8s

Loading

---

What Exists Today

Current deploy flow (imperative, synchronous)

POST /v0/deployments
  → Handler validates request
  → Service.LaunchDeployment()
    → CreateManagedDeploymentRecord() → DB status="deploying"
    → ResolveDeploymentAdapterByProviderID()
    → adapter.Deploy(ctx, deployment)         ← BLOCKS on request path
    → ApplyDeploymentActionResult() → DB status="deployed"
  → Return deployment to client

Problems: no retry on failure, restart loses in-flight state, adapter failure = permanent failure, no separation of intent from execution.

Current discovery flow (implicit, on-demand)

GET /v0/deployments
  → Query DB for managed deployments
  → For each provider:
      → adapter.Discover(ctx, providerID)     ← BLOCKS on request path
  → Merge managed + discovered
  → Return to client

Problems: discovery is request-scoped (blocks every list call), discovered deployments are ephemeral (not persisted), no caching.

What Changes vs What Stays

Changes

Component	Before	After
Deploy request path	Synchronous adapter execution	Write intent, return 202
Deployment status	Set directly from adapter result	Derived from reconcile events
Discovery	On-demand per ListDeployments call	Async writers → per-platform tables
Discovered deployments	Ephemeral (not persisted)	Persisted in platform tables
Cancelation	adapter.Cancel() on request path	desired_state: undeployed
Business logic location	Service layer (deployment/service.go)	KRT reconciler
Event backbone	None	pg LISTEN/NOTIFY

Stays the same

Component	Why
DeploymentPlatformAdapter interface	Adapters still implement Deploy/Undeploy/Discover. Reconciler is the new caller instead of service layer.
ProviderPlatformAdapter interface	Provider CRUD unchanged. Health check is additive.
Platform adapter implementations (local, k8s)	Internal logic unchanged. Called by reconciler instead of service.
Deployment DB schema (mostly)	Adding desired_state column. Existing columns stay.
API endpoints (mostly)	Same routes, deploy returns 202 instead of blocking. Watch mode is additive.
Static resource services (agent, server, skill, prompt)	No reconciliation needed. CRUD only.

[liuyingxuvka]

I have run into a similar architecture issue: moving from imperative request-path calls to an intent + async reconciler model makes the design cleaner, but it also introduces many failure/retry states that are easy to miss. I built a small free and open-source tool called Flow-Guard to simulate this kind of workflow before implementation.

Flow-Guard uses a finite-state mathematical model. It is not prompt-only and not an LLM wrapper; an AI agent can help install it and create/run the first model, but the thing being checked is the actual application/infrastructure logic.

For this design, I would model:

• API writes intent to DB
• notify delivered or missed
• reconciler observes state
• adapter action succeeds/fails/retries
• platform actual state changes
• discovery writer records actual state
• status update converges back into DB

The counterexamples I would look for are DB intent says deployed but platform did not deploy, platform deployed but status was never written back, retry creates duplicates, or discovery writes accidentally override desired intent.

Repo: https://github.com/liuyingxuvka/FlowGuard
Feedback: liuyingxuvka/FlowGuard#1

It may be a useful design-time guardrail for this reconciliation-engine transition.