Sensitive Data Management (SDM)

SDM is a Go toolset for separating sensitive data (PII) from append-only chain history using Protobuf annotations. From a single annotated .proto file it generates Go GORM structs, PostgreSQL DDL, a type-safe repository, and an optional audit trail — keeping the what changed (chain) physically separate from the who-they-are (PII) so each can be retained, encrypted, or purged independently.

Why SDM?

Most line-of-business systems eventually grow three intertwined concerns:

PII that needs to be queryable but should be isolated, encrypt-at-rest, and erasable on request (GDPR / KYC retention rules).
Auditable history of who-changed-what-when, separate from the live row so deletes don't take the trail with them.
Append-only versioned data (regulatory records, ledger entries, compliance documents) that must survive even if the PII anchor is later forgotten.

The conventional shape — one wide table with updated_at and a sidecar audit log written by application code — couples all three onto a single row, hides schema invariants in business logic, and makes "drop this user's PII" non-trivial.

SDM compiles those concerns into three first-class tables backed by triggers and a generated repository:

pii_<name>s — current PII (sensitive, single row per record, soft-deletable, erasable in place)
chain_<name>s — append-only per-field versioned history with a stable (key, field_name, version) PK, independent of the PII row's lifetime
audit_pii_<name>s (optional) — DB-trigger-written OLD-row snapshots capturing every UPDATE/DELETE on the PII table

Plus a <name>s view that joins the three so reads stay a single SELECT.

You write .proto annotations to describe shape; the generator handles table layout, triggers, repository methods, and serialization. Two config knobs (create-audit-tables, chain-drafts) opt in to additional behavior.

A runnable end-to-end demo lives at sdm-tool/sdm-example/demo.

How it fits together

For each annotated message, the generator emits three tables and one (or two) views:

   ┌──────────────────────────────────────────────────────────────────────┐
   │                                                                      │
   │  pii_<name>s                                                         │
   │    └── one row per record. PII / FK / query-indexed columns,         │
   │        plus audit cols (created_at, updated_at, deleted_at, created_by)
   │        Soft-deletable via gorm.DeletedAt.                            │
   │                                                                      │
   │  chain_<name>s                                                       │
   │    └── append-only history. PK (key, field_name, version).           │
   │        Each chain-stored field gets its own row per change;          │
   │        version bumps via a BEFORE INSERT trigger.                    │
   │                                                                      │
   │  audit_pii_<name>s     [optional: create-audit-tables: true]         │
   │    └── per-mutation snapshot of OLD row + change_type + changed_by.  │
   │        Written by an AFTER UPDATE/DELETE trigger on pii_<name>s.     │
   │                                                                      │
   │  <name>s     (view)                                                  │
   │    └── joins pii + latest chain row per field. Single SELECT for     │
   │        Fetch. Surfaces HasPendingDrafts when chain-drafts is on.     │
   │                                                                      │
   │  <name>s_with_drafts  (view, chain-drafts mode only)                 │
   │    └── overlay view that surfaces DRAFTED chain values too.          │
   │                                                                      │
   └──────────────────────────────────────────────────────────────────────┘

Application code talks to a generated repository that wraps GORM and threads a ctx-carried actor (sdm.CtxWithActor) into all three sinks (pii.created_by, chain.created_by, audit.changed_by).

                  Application
                       │
                       ▼
              ┌─────────────────┐
              │   <Name>Repo    │   ◄── sdm.CtxWithActor(ctx, "alice")
              └────────┬────────┘
                       │ Create / Upsert / Update / SaveAll
       ┌───────────────┼─────────────────────┐
       ▼               ▼                     ▼
   INSERT/UPSERT   APPEND CHAIN           SELECT FROM
   pii_<name>s     chain_<name>s          <name>s (view)
       │                                       ▲
       │  AFTER UPDATE/DELETE  trigger         │
       ▼                                       │
   audit_pii_<name>s ◄── reads sdm.actor       │
   (snapshot of OLD)     session variable      │
                                               │
                Fetch / FetchBy* / Exists ─────┘

Two opt-in knobs in sdm.cfg.yaml change the generated surface:

Knob	Default	Effect when true
`create-audit-tables`	`true`	Emits `audit_pii_<name>s`, the AFTER trigger, the `<Name>PiiAudit` struct, and `Repo.AuditLog`. See PII audit log.
`chain-drafts`	`false`	Each chain row carries a `status` (DRAFTED / CREATED / DROPPED). Repo emits `DraftChain` / `CommitChain` / `DropChain` + `Upsert` / `Update` (SaveAll / SaveChain are not emitted). Fetch gains a `drafted bool` parameter. See Chain drafts.

Quick start

# 1. Install the CLI
go install github.com/kapow-tech/sdm/cmd/sdm@latest

# 2. In your project: set up config + tooling
sdm config       # writes sdm.cfg.yaml
sdm setup        # installs protoc-gen-go, buf, protoc-gen-sdm; exports SDM protos

# 3. Annotate your .proto (see "Field annotations" below)
# 4. Generate
sdm generate

# 5. Use the generated repo
go run .

A complete worked example — proto files, generator config, application code, SQL schema, and an integration test suite — lives at sdm-example/demo.

Features

Field annotations (`sdmprotos/annotations.proto`)

Annotation	Effect
`(sdm.primary_key) = true`	Column is the PII table primary key.
`(sdm.auto_increment) = true`	Generates `BIGSERIAL` in SQL and `autoIncrement` GORM tag; assigned value is copied back to the model on Create.
`(sdm.chain_identifier_key) = true`	Field's value is used as the chain table key (defaults to the PK if absent). Lets you use an opaque `user_id` string while the PK stays a numeric `id`.
`(sdm.pii) = true`	Column lives in `pii_{name}s` (sensitive, single row per record).
`(sdm.query_index) = true`	Column lives in PII for indexed lookups (no `pii` flag needed).
`(sdm.hashed) = true`	Adds a `hashed_{field}` chain row containing `sha256(value)`. Combines freely with `pii`.
`(sdm.unique) = true`	Emits a SQL `UNIQUE` constraint and generates `FetchBy{Field}` / `ExistsBy{Field}` methods.
`(sdm.references) = "Type.field"`	Emits a foreign key. The referenced field must be `UNIQUE` or `PRIMARY KEY`. Reference fields are placed in the PII table.
`(sdm.json) = true`	String field stored as Postgres `JSONB`; Go side uses `datatypes.JSON`.

Where does each field land?

Annotations present	PII column	Chain column	View column
`primary_key`, `pii`, `query_index`, `references`	✅	—	✅ (from PII)
neither of the above	—	✅ (versioned)	✅ (latest chain value)
`hashed`	(PII column also)	✅ extra `hashed_<field>` row of `sha256(value)`	✅ extra `Hashed<Field>` column
`auto_increment` on PK	✅ as `BIGSERIAL`	—	✅

Fields with no annotation default to chain-stored.

Type handling

Proto type	PII Go type	Chain serialization	View Go type
`string`, `int32`, `int64`, `bool`	Native	`fmt.Sprintf("%v", …)`	Native
`enum`	Typed enum	`fmt.Sprintf("%v", …)` (enum's registered name)	`string` (recovered to typed enum by `AsBaseModel` via `EnumType_value`)
`string` + `(sdm.json) = true`	`datatypes.JSON`	Raw JSON text	`datatypes.JSON`
`google.protobuf.Timestamp`	`time.Time` (via `.AsTime()`)	`time.RFC3339Nano` text (view casts back via `::timestamptz`)	`time.Time`
Nested `MessageType`	`*MessageType` (with `serializer:protojson`)	`protojson.Marshal(...)`	`*MessageType` (auto-decoded by serializer)
`repeated string`	`pq.StringArray` (`text[]`)	`pgArrayLiteral` → `{a,b,c}`	`pq.StringArray` (`text[]`)
`repeated MessageType`	`[]*MessageType` (with `serializer:protojsonArray`, stored `jsonb`)	JSON array, element-wise protojson	`[]*MessageType` (auto-decoded by serializer)

Postgres timestamptz has microsecond precision (6 fractional digits) — time.Time values with nanosecond precision get truncated on round-trip.

The protojsonArray serializer (auto-emitted in sdm_helpers.go when any recorded message has a repeated MessageType field) handles []*MessageType ↔ JSON array bytes for both the PII column and the View column. Empty / nil slices round-trip as the literal [].

Baked-in audit + soft-delete

Every PII table receives four columns by default:

created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
deleted_at TIMESTAMP WITH TIME ZONE NULL,
created_by TEXT NOT NULL DEFAULT '',

created_by is set at INSERT from the actor on the ctx (see Actor attribution below) and is preserved across upserts. There is no updated_by column — "who last updated this row" lives per-change in audit_pii_{name}s.changed_by (see PII audit log). Dropping the column avoids drift between row-level state and the audit trail and removes a redundant write on every upsert.

The generated PII and View structs carry the timestamps as time.Time / gorm.DeletedAt. db.Delete(&xPii) performs soft-delete (sets deleted_at = NOW()); all generated Fetch / FetchBy{X} / Exists / ExistsBy{X} methods append an explicit WHERE deleted_at IS NULL filter so soft-deleted rows stay hidden.

Hard-delete via db.Unscoped().Delete(&xPii) bypasses the soft-delete scope and removes the row; chain history persists either way.

Actor attribution

Pass the actor identifier on ctx with sdm.CtxWithActor(ctx, actorID) (from github.com/kapow-tech/sdm/pkg/sdm) once at a request boundary (HTTP/gRPC middleware, batch job startup, CLI entrypoint). All downstream Create / Upsert / Update / SaveAll / Draft* / Commit* calls inherit it — and crucially, the helper lives in a single shared package so one context propagates across every generated repo, not just the one whose package emitted it:

import "github.com/kapow-tech/sdm/pkg/sdm"

ctx := sdm.CtxWithActor(ctx, "alice@example.com")
userRepo.Create(ctx, u)        // alice attributed
invoiceRepo.Create(ctx, i)     // alice attributed — same ctx, different repo
// → pii.created_by   = "alice@example.com"   (at INSERT only)
// → chain.created_by = "alice@example.com"   (every appended chain row)
// → audit.changed_by = "alice@example.com"   (every UPDATE / DELETE)

The actor lands in three sinks:

Sink	When written	Notes
`pii_{name}s.created_by`	INSERT only	Immutable across upserts — first writer wins.
`chain_{name}s.created_by`	Every new chain row	Append-only — each version records who appended it.
`audit_pii_{name}s.changed_by`	Every UPDATE / DELETE	Written by the AFTER trigger from the `sdm.actor` Postgres session variable.

Repos that aren't wrapped with sdm.CtxWithActor (bare context.Background()) record "" for all three. Direct GORM calls that bypass the generated write methods (db.Exec("UPDATE …"), db.Unscoped().Delete(...)) still fire the audit trigger but record "" because they don't set the session variable.

How the audit trigger reads the actor. Inside every generated write method's transaction, the repo runs SELECT set_config('sdm.actor', $1, true) (the true makes it SET LOCAL — transaction-scoped). The audit trigger then reads current_setting('sdm.actor', true) and writes that value to audit.changed_by. Because the session var is transaction-scoped, two concurrent requests with different actors can't cross-contaminate each other's audit rows.

Chain versioning

The chain table's (key, field_name, version) is a composite primary key. A BEFORE INSERT trigger sets version = MAX(version) + 1 scoped to (key, field_name), so each field's history is a per-(record, field) sequence — globally 1, 2, 3 … per field, not a single sequence across the whole table. Chain history is append-only and never rewritten.

Skip-if-unchanged. Every chain-writing method (SaveAll(_, true) in OFF mode; DraftChain / Create / Upsert / Update in ON mode) first reads the latest stored value per chain field for the key (one SELECT DISTINCT ON (field_name)), then appends a new version only when the byte-form differs. A re-save with identical data produces zero new chain rows; partial changes bump only the affected fields. Chain is a history of changes, not of saves. In chain-drafts mode the baseline is the latest CREATED row only, so re-drafting a value identical to the last committed state is a no-op even if intervening drafts were dropped.

Chain row timestamps (created_at) are stored as TIMESTAMP WITH TIME ZONE so version history surfaces with the correct offset regardless of host / server tz drift. Each chain row also carries a created_by TEXT column populated from the same actor that wrote it (see Actor attribution).

Chain drafts (opt-in)

Enabled by setting chain-drafts: true in sdm.cfg.yaml (or --sdm_opt=chain-drafts=true for direct protoc-gen-sdm usage). Stages chain changes as drafts that callers can later commit or drop, with the state machine enforced by a Postgres trigger.

Schema additions (ON mode):

-- New column on every chain_<name>s table
status TEXT NOT NULL DEFAULT 'CREATED'
  CHECK (status IN ('DRAFTED', 'CREATED', 'DROPPED'))

-- At-most-one DRAFTED row per (key, field_name) — DB-enforced
CREATE UNIQUE INDEX chain_<name>s_one_draft
  ON chain_<name>s (key, field_name)
  WHERE status = 'DRAFTED';

-- BEFORE UPDATE trigger that allows DRAFTED→CREATED, DRAFTED→DROPPED, or
-- "status unchanged"; everything else raises an exception.

State machine:

                  DraftChain / Create / Upsert / Update
                            │
                            ▼
                       ┌─────────┐
                       │ DRAFTED │ ◄── at-most-one per (key, field_name)
                       └────┬────┘
              CommitChain   │   DropChain
                            │
              ┌─────────────┴─────────────┐
              ▼                           ▼
        ┌─────────┐                 ┌─────────┐
        │ CREATED │  (committed     │ DROPPED │  (discarded
        └─────────┘   history)      └─────────┘   audit trail)
              │                           │
              ▼                           ▼
       visible in committed         visible in NEITHER
       view + overlay view          view (history only)

Other transitions raise illegal chain status transition (SQLSTATE 23514) via the trigger.

Two views are emitted instead of one:

<name>s — committed view (filters chain JOINs to status='CREATED'). What Fetch(_, _, drafted=false) reads.
<name>s_with_drafts — overlay view (filters chain JOINs to status<>'DROPPED', so a DRAFTED value supersedes the prior CREATED). What Fetch(_, _, drafted=true) reads.

Both views additionally expose has_pending_drafts bool — an EXISTS subquery against chain_<name>s for any DRAFTED row at this key. Surfaced on the View struct as HasPendingDrafts, independent of which view you queried. Use it as a signal that "this record has uncommitted changes" without paying a second round-trip.

Repository surface changes when ON:

Replaces	New
`SaveAll(ctx, m, true)`	`Upsert(ctx, m)` + `CommitChain(ctx, key, txHash)`
`SaveAll(ctx, m, false)`	`Upsert(ctx, m)` (drafts the chain side; commit later or drop)
`SaveChain(ctx, m)`	`DraftChain(ctx, m)` + `CommitChain(ctx, key, txHash)`
—	`Update(ctx, m)` — strict UPDATE (errors with `gorm.ErrRecordNotFound` if missing) + `DraftChain`
—	`DropChain(ctx, key)` — promote DRAFTED → DROPPED for that key
`Fetch(ctx, pk)`	`Fetch(ctx, pk, drafted bool)` — `false` reads `<name>s`; `true` reads `<name>s_with_drafts`

Create (PII strict INSERT) is emitted in both modes; in ON mode it also chains into DraftChain after the PII INSERT, all in the same transaction. Exists / ExistsBy* / ChangeLog / AuditLog are unchanged.

Workflow:

ctx := sdm.CtxWithActor(ctx, "alice@example.com")

// 1. Create: PII committed; chain rows staged as DRAFTED.
_ = repo.Create(ctx, &invoice.Invoice{
    InvoiceId: "inv_1", SellerId: "u_1", BuyerId: "u_2",
    Amount: 10000, Tags: []string{"draft"},
})

// 2. Committed view doesn't show the chain values yet…
v, _ := repo.Fetch(ctx, "inv_1", false)
fmt.Println(v.Amount, v.HasPendingDrafts) // 0 true

// 3. …but the overlay does.
v, _ = repo.Fetch(ctx, "inv_1", true)
fmt.Println(v.Amount, v.Tags)             // 10000 [draft]

// 4. Decide: commit (with optional tx_hash) or drop.
_ = repo.CommitChain(ctx, "inv_1", "tx-abc-123")
//   …or:
// _ = repo.DropChain(ctx, "inv_1")

// 5. Committed view now reflects the change; HasPendingDrafts flips false.
v, _ = repo.Fetch(ctx, "inv_1", false)
fmt.Println(v.Amount, v.TxHash, v.HasPendingDrafts) // 10000 tx-abc-123 false

Sentinel error. DraftChain (and by extension Create / Upsert / Update when chain-drafts is ON) returns ErrPendingDraftExists if any chain field of the record already has a pending DRAFTED row. The caller's recourse is to commit (CommitChain) or drop (DropChain) the existing draft first.

if err := repo.Upsert(ctx, inv); err != nil {
    if errors.Is(err, invoice.ErrPendingDraftExists) {
        // surface to caller — they need to resolve the pending draft
    }
    return err
}

Known caveat — half-state visibility. Because Create / Upsert / Update commit the PII row immediately but stage chain rows as DRAFTED, the committed view will show the PII columns updated and the chain columns NULL (or stale) until CommitChain runs. HasPendingDrafts flags this on every read; pass drafted=true to read the overlay. Atomic "PII + chain commit" is on the roadmap — for now the explicit two-step flow is the contract.

Testing pattern. Because the repo surface differs across modes, existing demo tests that use SaveAll are tagged //go:build !chaindrafts and chain-drafts tests are tagged //go:build chaindrafts. Run against an ON-mode generation with go test -tags chaindrafts ./integration/....

Change log

The generated Repo.ChangeLog(ctx, key) returns the full per-field version history for one record:

type ChangeLogEntry struct {
    Value     string    // raw chain field_value (TEXT)
    Timestamp time.Time // chain row created_at
}
type ChangeLog map[string]map[int64]ChangeLogEntry // field_name → version → entry

Soft-deleted PII rows do not mask chain history — chain entries persist independently. Returns gorm.ErrRecordNotFound when the key has no chain rows.

In chain-drafts mode the returned ChangeLog includes rows of every status (DRAFTED / CREATED / DROPPED). Filter on Value semantics if you only want committed history.

PII audit log

When enabled (default), every PII table gets a sibling audit_pii_{name}s table plus an AFTER UPDATE OR DELETE trigger that captures the row as it existed BEFORE each change. INSERT is not audited — the chain table already records the newly-introduced values.

Schema:

CREATE TABLE audit_pii_users (
  id BIGSERIAL PRIMARY KEY,
  ref_id TEXT NOT NULL,                          -- PK as text (composite PKs joined with ':')
  last_value JSONB NOT NULL,                     -- row_to_json(OLD)
  change_type TEXT NOT NULL,                     -- 'UPDATE' or 'DELETE' (TG_OP)
  changed_by TEXT NOT NULL DEFAULT '',           -- from session var sdm.actor
  changed_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP
);

Attribution flows through the ctx-based actor described above. The generated Save / SaveAll methods run SELECT set_config('sdm.actor', $1, true) inside the same transaction as the PII write; the trigger reads it via current_setting('sdm.actor', true). Because SET LOCAL (via set_config(..., true)) is transaction-scoped, the attribution does not leak between requests.

Reading the history — repo.AuditLog(ctx, pk) returns []{Name}PiiAudit in chronological order:

rows, _ := userRepo.AuditLog(ctx, u.Id)
for _, r := range rows {
    fmt.Printf("%s by %s at %s: %s\n",
        r.ChangeType, r.ChangedBy, r.ChangedAt, string(r.LastValue))
}

GORM soft-deletes (db.Delete(&pii) when the struct has gorm.DeletedAt) appear as change_type = 'UPDATE' because the underlying SQL is UPDATE … SET deleted_at = NOW(). Hard deletes (db.Unscoped().Delete(...)) appear as 'DELETE'.

Disabling audit tables — set create-audit-tables: false in sdm.cfg.yaml (or pass --sdm_opt=create-audit-tables=false when invoking protoc-gen-sdm directly from buf). When off, the generator skips:

the audit_pii_{name}s table and its trigger function
the {Name}PiiAudit Go struct
the Repo.AuditLog method
the SELECT set_config('sdm.actor', …) call inside Save/SaveAll (no trigger to read it)

The actor still populates pii.created_by and chain.created_by; only the per-change history disappears. If your test suite includes audit assertions, tag them with //go:build !noaudit so they're skipped by go test -tags noaudit against an audit-off generation. The demo integration suite uses this pattern — see sdm-example/demo/integration/audit_test.go.

View → base model

Every {Name}View carries an AsBaseModel() method that returns a fresh *{Name} proto. Use it when you've fetched the view and want to mutate + re-save without manually mapping fields. The converter handles:

scalar fields — direct assignment
enum fields — string → typed enum via the proto-generated {EnumType}_value map lookup
*Message fields — direct assignment (serializer:protojson already decoded on read)
google.protobuf.Timestamp — time.Time → timestamppb.New(...) (skipped if zero)
(sdm.json)=true strings — datatypes.JSON → string
repeated scalar — pq.StringArray → []string
repeated *Message — direct assignment ([]*Message already decoded by the protojsonArray serializer)

Audit columns (CreatedAt / UpdatedAt / DeletedAt / CreatedBy / TxHash) and hashed_* sidecar columns have no counterpart on the base proto and are dropped.

Installation

go install github.com/kapow-tech/sdm/cmd/sdm@latest

Two more steps put the project on a clean footing:

sdm config   # writes sdm.cfg.yaml in the current directory
sdm setup    # installs protoc-gen-go, buf, protoc-gen-sdm; exports SDM protos

For local development against an sdm checkout, build directly:

cd /path/to/sdm
go install ./cmd/sdm ./cmd/protoc-gen-sdm

The Makefile target make build does the same.

Configuration (`sdm.cfg.yaml`)

# Version of the sdm to use
sdm: "dev"

# Where the SDM annotation protos were exported by `sdm setup` (relative to this file)
sdm-proto: "proto/"

# Protos to compile and generate from (relative to this file)
user-protos:
  - "proto/user/user.proto"
  - "proto/invoice/invoice.proto"

# Where to write generated Go files
output: "models/"

# Where to write generated SQL files (defaults to `output` when omitted)
output-sql: "models/sql/"

# Emit audit_pii_{name}s tables + AFTER UPDATE/DELETE trigger +
# {Name}PiiAudit struct + Repo.AuditLog method. Defaults to true.
# When false, the actor still flows into pii.created_by /
# chain.created_by — those columns are independent.
create-audit-tables: true

# Opt-in chain draft/commit workflow. When true the generator swaps
# SaveAll for Upsert/Update + DraftChain/CommitChain/DropChain, emits
# a status column + partial unique index + state-machine trigger on
# every chain table, emits two views (committed + with-drafts), and
# Fetch / FetchBy* gain a trailing `drafted bool` parameter. Defaults
# to false. See the "Chain drafts (opt-in)" section below.
chain-drafts: false

All paths are resolved relative to the directory containing sdm.cfg.yaml. Both knobs are also exposed to direct buf/protoc usage: --sdm_opt=create-audit-tables=false, --sdm_opt=chain-drafts=true.

Usage

A complete runnable example is at sdm-example/demo.

1. Annotate your `.proto`

syntax = "proto3";
package invoice;

import "proto/sdmprotos/annotations.proto";

option go_package = "demo/models/invoice";

message Invoice {
  string invoice_id = 1 [(sdm.primary_key) = true, (sdm.chain_identifier_key) = true];
  string seller_gst = 2 [(sdm.pii) = true, (sdm.hashed) = true];
  string buyer_gst  = 3 [(sdm.pii) = true, (sdm.hashed) = true];
  string seller_id  = 4 [(sdm.references) = "User.user_id"];
  string buyer_id   = 5 [(sdm.references) = "User.user_id"];
  int64  amount     = 6;
  string metadata   = 7 [(sdm.json) = true];
  Money  price      = 8 [(sdm.pii) = true];
  repeated string tags  = 9;
  repeated Money  items = 10;
}

message Money {
  int64  value = 1;
  string unit  = 2;
}

User lives in a sibling user.proto:

message User {
  int64  id      = 1 [(sdm.primary_key) = true, (sdm.auto_increment) = true];
  string user_id = 2 [(sdm.pii) = true, (sdm.chain_identifier_key) = true, (sdm.unique) = true];
  string email   = 3 [(sdm.pii) = true, (sdm.hashed) = true, (sdm.unique) = true];
  string name    = 5 [(sdm.pii) = true];
  string pan     = 6 [(sdm.unique) = true];
  string country = 7;
}

2. Generate

A minimal sdm.cfg.yaml (see Configuration for the full reference, including create-audit-tables):

sdm: "dev"
sdm-proto: "proto/"
user-protos:
  - "proto/user/user.proto"
  - "proto/invoice/invoice.proto"
output: "models/"
output-sql: "models/sql/"

sdm generate

Per proto, four files are emitted:

{name}.pb.go — standard protobuf code
{name}_sdm_model.go — {Name}Pii, {Name}Chain, {Name}View structs (plus {Name}PiiAudit when create-audit-tables: true)
{name}_sdm_schema.sql — CREATE TABLEs, the version trigger, the view (plus the audit table + trigger when enabled)
{name}_sdm_repo.go — GORM repository

A single sdm_helpers.go per package holds pgArrayLiteral (for repeated scalar fields), the ChangeLog / ChangeLogEntry types, ErrPendingDraftExists (when chain-drafts: true), and — when nested messages are present — the protojson / protojsonArray GORM serializers.

The actor-context helpers (sdm.CtxWithActor / sdm.ActorFromContext) live in the shared package github.com/kapow-tech/sdm/pkg/sdm so a ctx created by one helper propagates across every generated repo (not just the package it was created in).

3. Use in Go

The snippet below assumes the OFF-mode API (chain-drafts: false, the default). For the draft/commit workflow, see Chain drafts.

import (
    "context"
    "github.com/kapow-tech/sdm/pkg/sdm"
    "gorm.io/driver/postgres"
    "gorm.io/gorm"
    "demo/models/invoice"
    "demo/models/user"
)

db, _ := gorm.Open(postgres.Open(dsn), &gorm.Config{})

// Attribute every downstream write to a single actor. Set once at a
// request / job boundary; all repo calls on this ctx inherit it
// (regardless of which repo's package they belong to).
ctx := sdm.CtxWithActor(context.Background(), "alice@example.com")

userRepo := user.NewUserRepo(db)
// SaveAll(_, true) upserts PII + appends a chain version per changed field.
// pii.created_by + chain.created_by are populated from the ctx actor.
_ = userRepo.SaveAll(ctx, &user.User{
    UserId: "u_001", Email: "alice@example.com", Name: "Alice", Pan: "ABCDE1234F", Country: "IN",
}, true)

repo := invoice.NewInvoiceRepo(db)
inv := &invoice.Invoice{
    InvoiceId: "inv_001",
    SellerGst: "27AAA…", BuyerGst: "29BBB…",
    SellerId:  "u_001", BuyerId: "u_002",
    Amount:    10000,
    Metadata:  `{"source":"api"}`,
    Price:     &invoice.Money{Value: 10000, Unit: "INR"},
    Tags:      []string{"urgent", "paid"},
    Items:     []*invoice.Money{{Value: 9000, Unit: "INR"}, {Value: 1000, Unit: "INR"}},
}
_ = repo.SaveAll(ctx, inv, true)

// Create is a strict INSERT on the PII row — errors on PK / unique conflict.
// Use it when you want the conflict to surface as an error rather than an upsert.
err := repo.Create(ctx, &invoice.Invoice{InvoiceId: "inv_001", /* … */})
// err is a Postgres unique-violation since inv_001 already exists.
_ = err

view, _ := repo.Fetch(ctx, "inv_001")
// view.Price is *Money, view.Items is datatypes.JSON, view.Tags is pq.StringArray.
// view.CreatedAt / view.UpdatedAt / view.DeletedAt / view.CreatedBy are populated automatically.
// (For "who last updated this row", call repo.AuditLog and read the latest row's ChangedBy.)

// AsBaseModel: convert the view back to the base proto for re-saves.
roundTrip := view.AsBaseModel()
roundTrip.Amount = 12000
_ = repo.SaveAll(ctx, roundTrip, true) // chain v2 for amount; other fields unchanged → no-op

// SaveAll(_, false): upsert PII only, leave chain alone.
_ = repo.SaveAll(ctx, roundTrip, false)

// Full per-field version history.
log, _ := repo.ChangeLog(ctx, "inv_001")
// log["amount"][1].Value == "10000"
// log["amount"][2].Value == "12000"
// log["amount"][2].Timestamp is the chain row's timestamptz created_at

// Soft-delete via GORM:
_ = db.Delete(&invoice.InvoicePii{InvoiceId: "inv_001"}).Error
// Subsequent Fetch / Exists return ErrRecordNotFound / false.
// ChangeLog still returns the chain history — soft-delete does not mask it.

// Per-change audit history (audit-on only).
audit, _ := repo.AuditLog(ctx, "inv_001")
for _, r := range audit {
    fmt.Printf("%s by %q at %s\n", r.ChangeType, r.ChangedBy, r.ChangedAt)
}

CLI reference

Command	Description
`sdm setup`	Installs `protoc-gen-go`, `buf`, `protoc-gen-sdm`; exports SDM annotation protos to a local directory.
`sdm config`	Writes a default `sdm.cfg.yaml`.
`sdm generate`	Compiles user protos and writes the four generated files per message.
`sdm --version` / `sdm -v`	Prints the CLI version (built into the binary at install time; `dev` for source builds).

sdm generate flags:

Flag	Default	Description
`--cfg`	`sdm.cfg.yaml`	Path to config file (relative paths resolve from the CWD).
`--proto`	(config)	Override the input proto path.
`--out`	(config)	Override the output directory for Go files.

The Options struct (create-audit-tables, chain-drafts) is read from sdm.cfg.yaml only — there is no per-invocation flag for them on the sdm generate side, but protoc-gen-sdm accepts them via --sdm_opt=key=value when invoked through buf / protoc.

Generated artifacts

Per recorded message (per .proto file), four Go/SQL files are emitted:

File	Purpose
`<name>.pb.go`	Standard `protoc-gen-go` output (proto message types).
`<name>_sdm_model.go`	`{Name}Pii`, `{Name}Chain`, `{Name}View` GORM structs + `TableName()` overrides; `{Name}PiiAudit` (when audit on); `EnsureUnique` chain probe; `AsBaseModel` view-to-proto converter.
`<name>_sdm_schema.sql`	DDL: `pii_<name>s`, `chain_<name>s` (+ chain-drafts status column / partial unique index / state-machine trigger when ON), `audit_pii_<name>s` (+ AFTER trigger when audit ON), version trigger, one or two views.
`<name>_sdm_repo.go`	`{Name}Repo` with `NewXxxRepo` constructor + write methods (`Create` / `Upsert` / `Update` / `SaveAll` / `DraftChain` / `CommitChain` / `DropChain`) + read methods (`Fetch` / `FetchBy` / `Exists` / `ExistsBy` / `AuditLog` / `ChangeLog`).

One package-level file is also emitted (once per Go package, not per proto file):

File	Purpose
`sdm_helpers.go`	`pgArrayLiteral`, `ChangeLog` / `ChangeLogEntry` types, `ErrPendingDraftExists` (chain-drafts ON), `protojson` / `protojsonArray` serializers.

The actor-context helpers (sdm.CtxWithActor / sdm.ActorFromContext) are not emitted per package — they live in the shared runtime package github.com/kapow-tech/sdm/pkg/sdm so a ctx works across every generated repo, regardless of which package the repo was generated into.

Generated schema layout (one-line reference):

pii_{name}s — primary key, PII / query-index / FK columns, plus the three timestamp audit columns (created_at, updated_at, deleted_at, all TIMESTAMP WITH TIME ZONE) and created_by TEXT. Soft-deleted rows have non-NULL deleted_at. created_by is set at INSERT from the ctx actor and preserved across upserts.
audit_pii_{name}s (emitted when create-audit-tables: true) — (id, ref_id, last_value, change_type, changed_by, changed_at). Populated by the audit_pii_{name}s_log_trigger AFTER UPDATE OR DELETE trigger on pii_{name}s. last_value is row_to_json(OLD)::jsonb; change_type is the trigger's TG_OP; changed_by reads from the sdm.actor Postgres session variable (transaction-scoped via SET LOCAL).
chain_{name}s — (key, field_name, version, tx_hash, field_value, created_at, created_by). created_at is TIMESTAMP WITH TIME ZONE; created_by TEXT records the actor that appended the row. The version is auto-assigned per (key, field_name) by the chain_{name}s_set_version_trigger BEFORE INSERT trigger; field values are TEXT, with the view casting back to ::jsonb, ::timestamptz, or text[] where appropriate. When chain-drafts is enabled: an additional status TEXT NOT NULL DEFAULT 'CREATED' CHECK (status IN ('DRAFTED', 'CREATED', 'DROPPED')) column, a partial unique index chain_{name}s_one_draft (key, field_name) WHERE status='DRAFTED', and a BEFORE UPDATE trigger chain_{name}s_status_guard_trigger enforcing legal transitions (DRAFTED → CREATED, DRAFTED → DROPPED, status unchanged).
{name}s (view) — joins pii_{name}s p with one LEFT JOIN per chain-stored field (DISTINCT ON (key, field_name) … ORDER BY version DESC to pick the latest version). PII audit columns including created_by are surfaced; per-update history lives in audit_pii_{name}s (when enabled) rather than on the row. When chain-drafts is enabled: each chain-side JOIN subquery filters to status='CREATED' (committed only); a sibling view {name}s_with_drafts filters to status<>'DROPPED' instead (overlay). Both views expose has_pending_drafts bool via an EXISTS subquery, surfaced on the View struct as HasPendingDrafts.

Repository surface (per message)

The exact method set depends on the chain-drafts knob — OFF mode emits the familiar Save / SaveAll pair; ON mode emits the draft-workflow trio (Upsert / Update / DraftChain / CommitChain / DropChain) instead. Read-side methods (Fetch, Exists, ChangeLog, AuditLog) exist in both modes; Fetch gains a trailing drafted bool parameter in ON mode.

Common to both modes

Method	Notes
`Create(ctx, *T)`	Strict INSERT of the PII row. Returns the driver-native error on PK / unique conflict. In OFF mode does not touch the chain table; in ON mode also calls `DraftChain` in the same transaction (chain rows staged as DRAFTED). Honors `sdm.CtxWithActor` (writes `pii.created_by`).
`Exists(ctx, pk)` / `ExistsBy{Unique}`	Counts on the PII table with the `deleted_at IS NULL` filter. Not draft-aware — existence is answered by committed state.
`ChangeLog(ctx, key)`	Returns the full per-field version history as `map[field_name]map[version]{Value, Timestamp}`. Returns `gorm.ErrRecordNotFound` if no chain rows exist.
`AuditLog(ctx, pk)` (audit-on only)	Returns `[]{Name}PiiAudit` rows for one PII record, oldest first. Each row carries `LastValue` (OLD as JSONB), `ChangeType` (`'UPDATE'`/`'DELETE'`), `ChangedBy` (from the `sdm.CtxWithActor` ctx), and `ChangedAt`. Not emitted when `create-audit-tables: false`.

OFF mode (`chain-drafts: false`, default)

Method	Notes
`SaveAll(ctx, *T, withChain bool)`	Upserts the PII row (`ON CONFLICT … DO UPDATE` on the chain identifier key); when `withChain=true`, also appends new chain versions for every field whose value changed (skip-if-unchanged). Honors `sdm.CtxWithActor`.
`Fetch(ctx, pk)`	Reads `*TView` from the view, filtered by `deleted_at IS NULL`.
`FetchBy{Unique}(ctx, val)`	Generated for every `(sdm.unique)` field.

ON mode (`chain-drafts: true`)

Method	Notes
`Upsert(ctx, *T)`	PII upsert + `DraftChain` (chain rows staged as DRAFTED). Honors `sdm.CtxWithActor`.
`Update(ctx, *T)`	Strict PII UPDATE — returns `gorm.ErrRecordNotFound` when the row doesn't exist (no insert). Followed by `DraftChain`. Honors `sdm.CtxWithActor`.
`DraftChain(ctx, *T)`	Standalone draft entry — appends DRAFTED chain rows for fields differing from the latest CREATED. Returns `ErrPendingDraftExists` if a draft is already pending for any field of this record (resolve via `CommitChain` or `DropChain` first). Honors `sdm.CtxWithActor` (writes `chain.created_by`).
`CommitChain(ctx, key…, txHash string)`	Promotes every DRAFTED row for this key to CREATED in a single UPDATE, stamping `txHash` on the promoted rows (pass `""` if not applicable). Idempotent — no-op when no drafts exist. Trigger-enforced transition.
`DropChain(ctx, key…)`	Promotes every DRAFTED row for this key to DROPPED in a single UPDATE. Idempotent.
`Fetch(ctx, pk, drafted bool)`	`drafted=false` reads the committed view `<name>s`; `drafted=true` reads the overlay view `<name>s_with_drafts`. Both filtered by `deleted_at IS NULL`. View struct's `HasPendingDrafts` is populated regardless of which is queried.
`FetchBy{Unique}(ctx, val, drafted bool)`	Same `drafted` semantics.

SaveAll and SaveChain are not emitted in ON mode — their atomic "PII + chain commit" semantics live in Upsert followed by an explicit CommitChain.

View methods

Method	Notes
`View.AsBaseModel() *T`	Converts the view row back to the base proto model (Timestamp → `timestamppb`, repeated message JSON → `[]*Message`, datatypes.JSON → string, pq.StringArray → []string). Audit columns and hashed sidecars are dropped.
`Chain.EnsureUnique(tx) bool`	Probe used by callers that want to enforce global uniqueness of a chain value before staging or committing it. Returns false if the query errors.

Migration guide

The two config knobs can be flipped at any time; the generated code is self-contained, but DB schema changes are not auto-migrated — you need to apply the diff against <name>_sdm_schema.sql against your live database yourself.

Flipping `create-audit-tables: false → true`

New artifacts: audit_pii_<name>s table + audit_pii_<name>s_log_trigger
- {Name}PiiAudit struct + Repo.AuditLog.
Existing PII rows are unaffected. The trigger only fires from this point forward; historical mutations are not back-filled.
Application code needs no change — sdm.CtxWithActor(ctx, …) was already populating pii.created_by / chain.created_by; turning on audit just adds audit.changed_by to the mix.

Flipping `create-audit-tables: true → false`

The generator stops emitting the audit table, trigger, struct, and AuditLog method.
The existing audit_pii_<name>s rows stay in the DB until you drop them manually; the trigger function definition remains too (drop with DROP FUNCTION IF EXISTS audit_pii_<name>s_log();). The repo simply stops writing to / reading from it.
Application code that calls repo.AuditLog(...) will fail to compile. Replace with calls to repo.ChangeLog(...) for per-field history (no attribution column though) or migrate to a different sink.

Flipping `chain-drafts: false → true`

This is the bigger jump: the repo surface changes (SaveAll / SaveChain are no longer emitted; Upsert / Update / DraftChain / CommitChain / DropChain appear instead) and the view signatures gain a drafted bool parameter.

DDL diff:

Adds status TEXT NOT NULL DEFAULT 'CREATED' CHECK … column on each chain_<name>s table.
Adds partial unique index chain_<name>s_one_draft.
Adds state-machine trigger chain_<name>s_status_guard.
Replaces single view <name>s with two views (<name>s + <name>s_with_drafts).

To migrate live data:

ALTER TABLE chain_<name>s
  ADD COLUMN status TEXT NOT NULL DEFAULT 'CREATED'
    CHECK (status IN ('DRAFTED', 'CREATED', 'DROPPED'));
-- index + trigger DDL: copy from the regenerated <name>_sdm_schema.sql
-- views: DROP VIEW <name>s; then run the new CREATE VIEW statements

Application code changes:

repo.SaveAll(ctx, m, true) → repo.Upsert(ctx, m); repo.CommitChain(ctx, key, txHash)
repo.SaveAll(ctx, m, false) → repo.Upsert(ctx, m) (chain stays in DRAFTED until you commit or drop)
repo.SaveChain(ctx, m) → repo.DraftChain(ctx, m); repo.CommitChain(ctx, key, txHash)
repo.Fetch(ctx, pk) → repo.Fetch(ctx, pk, false) (or true for overlay)

Flipping `chain-drafts: true → false`

DDL: drop the status column, the partial unique index, the state-machine trigger, and the <name>s_with_drafts view. Re-create <name>s from the regenerated DDL (the join semantics change — it drops the status='CREATED' filter from each chain-side JOIN).
All in-flight DRAFTED rows will be silently promoted to CREATED when the column is dropped — back up the table before migrating if those rows represent uncommitted intent.
Application code: revert Upsert/CommitChain calls to SaveAll(_, true); drop drafted parameters from Fetch/FetchBy* calls.

Recipes

Idempotent saves

SaveAll(_, true) is already idempotent: the PII upsert is a no-op (or just bumps updated_at) when nothing changed, and the chain skip-if-unchanged guard prevents new chain versions when every field value matches the latest stored one. Re-running an import job is safe.

"Who last updated this row?"

Query AuditLog and take the latest row:

rows, _ := repo.AuditLog(ctx, pk)
if len(rows) > 0 {
    last := rows[len(rows)-1]
    fmt.Printf("last updated by %s at %s (%s)\n",
        last.ChangedBy, last.ChangedAt, last.ChangeType)
}

There is intentionally no pii.updated_by column — keeping the answer in the audit table prevents drift between row state and the trail.

Showing pending edits in the UI

In chain-drafts mode, View.HasPendingDrafts is populated regardless of which view you query, so a single committed-view read can drive both the "current values" display and a "you have pending changes" badge:

v, _ := repo.Fetch(ctx, pk, false)
if v.HasPendingDrafts {
    overlay, _ := repo.Fetch(ctx, pk, true)
    // Render v.* as the committed values + overlay.* as the pending preview.
}

Per-field version diff

ChangeLog returns the full history. To compute a diff between two versions of a field:

log, _ := repo.ChangeLog(ctx, key)
amount := log["amount"]
prev := amount[2].Value
curr := amount[3].Value
fmt.Printf("amount: %s → %s\n", prev, curr)

Round-trip mutate

view, _ := repo.Fetch(ctx, pk)         // OFF mode
base := view.AsBaseModel()             // *T proto, ready to mutate
base.Country = "DE"
_ = repo.SaveAll(ctx, base, true)      // skip-if-unchanged for everything else

In ON mode the equivalent is view.AsBaseModel() → mutate → repo.Upsert(ctx, base) → repo.CommitChain(ctx, pk, txHash).

Erasing a user's PII (right-to-forget)

_ = db.Unscoped().Delete(&user.UserPii{Id: u.Id}).Error

Hard-delete drops the PII row entirely; chain history persists (it never held the PII, only chain-stored fields). AuditLog captures a change_type='DELETE' row with last_value = row_to_json(OLD)::jsonb — that includes the deleted PII columns. To erase the audit trail too:

DELETE FROM audit_pii_users WHERE ref_id = $1;

(Chain rows for that key contain no PII — (sdm.pii) fields are routed to PII, not chain — so they're safe to retain.)

Bulk import from a non-Go pipeline

Insert chain rows directly via SQL. The trigger fills in version and the view picks the latest:

INSERT INTO chain_users (key, field_name, field_value, created_by)
VALUES ('u_001', 'country', 'IN', 'importer-svc');
-- version is set by chain_users_set_version_trigger

For chain-drafts ON the row will default to status='CREATED' (the column default), so external loaders that don't know about the draft workflow will write directly committed rows — the safer fallback.

Testing patterns

The generated repo + GORM combo tests cleanly against a real Postgres via testcontainers-go. The demo at sdm-example/demo/integration is the canonical reference; the patterns below summarize how it's organized.

Build tags per mode

Because the repo surface differs across config combinations, tests are split across files using Go build tags so that one source tree can verify all generations:

Tag combo	Run when
(no tags)	`chain-drafts: false` + `create-audit-tables: true` (default)
`noaudit`	`create-audit-tables: false` (any chain-drafts)
`chaindrafts`	`chain-drafts: true` + `create-audit-tables: true`
`chaindrafts noaudit`	both off

Tests using SaveAll are tagged //go:build !chaindrafts; tests using Upsert / CommitChain are tagged //go:build chaindrafts. Audit tests add && !noaudit. Mode-agnostic helpers (TestMain, schema loading) live in untagged setup files.

TestMain that spins a container per package

func TestMain(m *testing.M) {
    ctx := context.Background()
    container, _ := tcpostgres.Run(ctx,
        "postgres:16-alpine",
        tcpostgres.WithDatabase("test"),
        tcpostgres.WithUsername("u"),
        tcpostgres.WithPassword("p"),
        testcontainers.WithWaitStrategy(
            wait.ForLog("database system is ready").WithOccurrence(2),
        ),
    )
    defer container.Terminate(ctx)

    dsn, _ := container.ConnectionString(ctx, "sslmode=disable")
    db, _ := gorm.Open(gormpg.Open(dsn), &gorm.Config{})
    testDB = db
    applySchemaFile(db, "../models/sql/user_sdm_schema.sql")
    applySchemaFile(db, "../models/sql/invoice_sdm_schema.sql")
    os.Exit(m.Run())
}

Reset between tests

A resetTables(t) helper truncates everything and resets BIGSERIAL counters; call it at the top of each test. Two variants — one that also truncates audit_pii_* (tagged !noaudit), one that doesn't.

Audit assertions

Decode LastValue and assert on it:

rows, _ := repo.AuditLog(ctx, u.Id)
var snap map[string]any
json.Unmarshal(rows[0].LastValue, &snap)
require.Equal(t, "alice@example.com", snap["email"])  // OLD row's value
require.Equal(t, "UPDATE", rows[0].ChangeType)
require.Equal(t, "bob", rows[0].ChangedBy)            // from sdm.CtxWithActor in the second SaveAll

Operational notes

Chain table growth

Chain is append-only. Every field change adds a row. Skip-if-unchanged prevents no-op writes, but rows for the same (key, field_name) accumulate in proportion to how often that field changes. For high-cardinality fields (e.g., a last_seen_at updated per request), prefer storing on the PII table (via (sdm.query_index)=true) so it lives as a normal column rather than as a versioned chain field.

A (key, field_name) partial index makes "latest version" lookups fast, but full chain scans (e.g., ChangeLog) scale with row count. If you need to retain history for years, consider periodic archival to a partitioned shadow table outside the live chain_<name>s.

Transaction semantics

Every generated write method opens a single db.Transaction(func(tx) ...). PII write + chain inserts + (when applicable) the sdm.actor session-var install run in one transaction. The audit trigger fires from the same transaction, so audit attribution is consistent with the mutating write — there's no window in which pii.updated_at advances but the audit row is missing.

In chain-drafts mode, Create/Upsert/Update commit the PII row and stage chain rows as DRAFTED in one transaction; CommitChain is a second transaction (PII is already committed at this point). If atomicity across the PII+chain boundary matters, hold both writes inside your own outer transaction:

err := db.Transaction(func(tx *gorm.DB) error {
    r := invoice.NewInvoiceRepo(tx)            // re-wrap for the outer tx
    if err := r.Upsert(ctx, inv); err != nil { return err }
    return r.CommitChain(ctx, inv.InvoiceId, txHash)
})

Read-after-write through the view

The view joins pii_<name>s p with subqueries against chain_<name>s. Within the same transaction as a write, the view sees the new rows immediately (Postgres MVCC). Across transactions, standard read-committed semantics apply.

Concurrency

The chain_<name>s_set_version_trigger reads MAX(version) under each INSERT — two concurrent inserts for the same (key, field_name) will race the read and one will violate the composite PK. Wrap your write methods in a retry loop if you expect concurrent drafts for the same field; the retry can simply re-run the same write (skip-if-unchanged makes it idempotent).

The partial unique index on (key, field_name) WHERE status='DRAFTED' prevents two concurrent DraftChain calls for the same field — one wins, the other gets ErrPendingDraftExists.

Postgres version

DDL relies on CREATE OR REPLACE TRIGGER, partial unique indexes, JSONB, row_to_json(OLD)::jsonb, and current_setting(name, true). All of these are stable in Postgres ≥ 12. The CI demo runs against postgres:16-alpine.

Limitations and roadmap

Atomic PII + chain commit in chain-drafts mode. Today Create/Upsert/Update commit the PII row and stage chain rows as DRAFTED in one transaction; CommitChain is a second transaction. The contract is "PII reflects immediately; chain values lag until you commit or drop". Wrap both calls in your own outer transaction (above) when you need true atomicity. Generator-side support for one-step commit is on the roadmap.
Composite primary keys are supported, but the chain key is composed with ':' as the separator (OLD.a::text || ':' || OLD.b::text). Don't use ':' in PK values without escaping.
Repeated message in chain history. Chain stores repeated MessageType as a single JSON array column; per-element diffs aren't surfaced. If you need element-level history, model each element as its own message.
(sdm.json)=true strings are stored verbatim and JSON-validated by Postgres on cast. Malformed JSON surfaces as a Postgres error on Fetch (the view casts field_value::jsonb).
No PII-side history versioning. PII is a single-row snapshot; per-change history lives in audit_pii_* (when enabled). There is no per-field version log for PII fields the way there is for chain fields.
ExistsBy<X> on chain-only fields fails. The method probes the PII table, but a chain-only field (e.g., pan in the demo) has no PII column — the SQL errors. Documented in the demo as a "known SDM quirk"; a future generator pass should route the probe to the view instead.
No native enum mapping in DDL. Enum fields are stored as TEXT in PII / chain. AsBaseModel recovers the typed enum via the proto-generated <EnumType>_value map.

Using with buf directly

go install github.com/kapow-tech/sdm/cmd/protoc-gen-sdm@latest

# buf.gen.yaml
version: v1
plugins:
  - plugin: go
    out: .
    opt: paths=source_relative
  - plugin: sdm
    out: .
    opt:
      - paths=source_relative
      - create-audit-tables=true   # default; omit or set false
      - chain-drafts=false         # default; omit or set true

buf generate

protoc-gen-sdm reads its options from --sdm_opt=key=value (one per line in buf.gen.yaml as shown above, or comma-separated when invoked through protoc).

Project layout

sdm/
├── cmd/
│   ├── sdm/                 # CLI (config / setup / generate)
│   └── protoc-gen-sdm/      # protoc plugin entry point
├── pkg/
│   ├── config/              # sdm.cfg.yaml load / write
│   ├── sdm/                 # shared runtime — CtxWithActor / ActorFromContext
│   │                        #   (imported by every generated repo)
│   └── generator/           # code generator (split by artifact)
│       ├── generator.go     # entry points + Options
│       ├── helpers.go       # sdm_helpers.go emission
│       ├── models.go        # <name>_sdm_model.go emission
│       ├── sql.go           # <name>_sdm_schema.sql emission
│       ├── repo.go          # <name>_sdm_repo.go emission
│       └── fields.go        # per-field / per-message introspection
├── sdmprotos/
│   └── annotations.proto    # field option extensions
├── buf.yaml / buf.gen.yaml  # for regenerating annotations.pb.go
├── Makefile                 # proto-gen, build
└── README.md                # this file

sdm-example/demo (a separate repository checkout) holds the end-to-end proto + generated + tested example.

Contributing

Local development loop:

# Rebuild the CLI + plugin
go install ./cmd/sdm ./cmd/protoc-gen-sdm

# Regenerate the SDM annotation proto (only when sdmprotos/annotations.proto changes)
make proto-gen

# Exercise the demo against the in-progress generator
cd ../sdm-example/demo
sdm generate
go test ./integration/...           # OFF + audit
go test -tags chaindrafts ./integration/...   # ON + audit
go test -tags 'chaindrafts noaudit' ./integration/...
go test -tags noaudit ./integration/...

The demo's integration tests are the contract for generator behavior — adding a feature means adding tests there first, then making the generator emit code that passes them. When changing generated code, spot-check the demo's models/ tree (git diff models/) so unintended emission changes are visible.

License

Apache 2.0. See LICENSE in the repository root.

Name		Name	Last commit message	Last commit date
Latest commit History 46 Commits
cmd		cmd
pkg		pkg
sdmprotos		sdmprotos
.gitignore		.gitignore
Makefile		Makefile
README.md		README.md
buf.gen.yaml		buf.gen.yaml
buf.yaml		buf.yaml
go.mod		go.mod
go.sum		go.sum

Folders and files

Latest commit

History

Repository files navigation

Sensitive Data Management (SDM)

Why SDM?

Table of contents

How it fits together

Quick start

Features

Field annotations (sdmprotos/annotations.proto)

Type handling

Baked-in audit + soft-delete

Actor attribution

Chain versioning

Chain drafts (opt-in)

Change log

PII audit log

View → base model

Installation

Configuration (sdm.cfg.yaml)

Usage

1. Annotate your .proto

2. Generate

3. Use in Go

CLI reference

Generated artifacts

Repository surface (per message)

Common to both modes

OFF mode (chain-drafts: false, default)

ON mode (chain-drafts: true)

View methods

Migration guide

Flipping create-audit-tables: false → true

Flipping create-audit-tables: true → false

Flipping chain-drafts: false → true

Flipping chain-drafts: true → false

Recipes

Idempotent saves

"Who last updated this row?"

Showing pending edits in the UI

Per-field version diff

Round-trip mutate

Erasing a user's PII (right-to-forget)

Bulk import from a non-Go pipeline

Testing patterns

Build tags per mode

TestMain that spins a container per package

Reset between tests

Audit assertions

Operational notes

Chain table growth

Transaction semantics

Read-after-write through the view

Concurrency

Postgres version

Limitations and roadmap

Using with buf directly

Project layout

Contributing

License

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases 16

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Field annotations (`sdmprotos/annotations.proto`)

Configuration (`sdm.cfg.yaml`)

1. Annotate your `.proto`

OFF mode (`chain-drafts: false`, default)

ON mode (`chain-drafts: true`)

Flipping `create-audit-tables: false → true`

Flipping `create-audit-tables: true → false`

Flipping `chain-drafts: false → true`

Flipping `chain-drafts: true → false`

Packages