Session transcript indexing: design discussion

[jwade83]

Following up on @spokV's note in #23 that session transcript indexing would be worth exploring as a standalone feature — I've been running a session persistence layer on top of SQLite for a while and wanted to share some design questions before jumping into code.

What I've found through practice

The biggest lesson from building session persistence: state, log, and index are three distinct concerns with different persistence semantics. Conflating them produces a system that's either too verbose to search or too compressed to cold-start from.

In my setup, these ended up as separate structures:

• State snapshot — overwritten each session close; self-contained enough for cold-start (decisions made, open questions, what to do next)
• Append-only log — full archive of every snapshot, never edited
• Session index — one line per session, scannable reference (date, objective, key outcome, files touched)

The snapshot-not-log pattern was the most counterintuitive part. Early versions appended everything, which made the session record grow until it was unsearchable. Overwriting forces you to decide what actually matters for the next session's cold-start.

Design questions for Memora

1. Separate table vs. memories table?
Session transcripts have different access patterns from standalone memories — they're sequential, chunked, and usually searched as a group. Would a session_transcripts table with its own FTS index make more sense than storing chunks as tagged memories? Or does Memora's existing tag/metadata filtering handle this well enough?

2. Chunking strategy
Transcripts need overlapping chunks for coherent semantic search. Is there a preferred chunking approach in the codebase, or would this be the first implementation of it? Configurable window size + overlap seems right, but I don't want to over-engineer the first pass.

3. Session metadata and cross-references
Memora already has memories_crossrefs. Should session chunks cross-reference each other (sequential linking within a session) and also link to any standalone memories created during that session? That would let you traverse from a memory back to the conversation that produced it.

4. Search ranking
Session content is noisier than curated memories. Should session search results be weighted differently in memory_search, or should session search be a separate tool?

Happy to take a crack at implementation if the scope feels right. Aiming for focused and minimal — session indexing only, no broader changes.

[justinTM]

@jwade83 i'd be curious how you're prompting/skilling Memora

i'm telling Codex/Claude to invoke a memory skill when relevant, and the skill records filepaths, line refs, and repo info during sessions. it was a nice first attempt, but it's not token efficient, fast, or accurate -- becoming too much like a transcript log than a memory store.

my biggest use case is to disentangle/combine parallel work-in-progress, keep track of original intent behind changes, and avoid mistakes already made before.

[jwade83]

@justinTM the transcript-log-vs-memory-store problem is exactly what pushed me toward splitting things up. The skill-based recording approach (filepaths, line refs, repo info per session) produces a growing log that's too noisy to search and too expensive to load — you end up paying tokens to re-read things that don't matter for the current session.

What ended up working for my setup was separating into three structures with different persistence rules:

State snapshot — overwritten every session close. Contains only what the next session needs to cold-start: decisions made, open questions, what to work on next. Forces a compression decision at session end — you have to decide what actually carries forward vs. what was just working noise.

Append-only log — full archive of every snapshot, never edited. When you need to recover a decision from 3 weeks ago, grep into this. Not loaded by default.

Session index — one line per session: date, objective, key outcome, files touched. Scannable without loading any content. Answers "which session introduced X?" without reading session bodies.

For your specific cases:

• Parallel WIP: The state snapshot helps because it tracks decisions and intent per work context, not file locations. Filepaths and line refs are too granular — what you actually need across sessions is "what was I trying to do and what did I decide," not "where was the cursor."

• Original intent: Lives in the append-only log. Each snapshot captures why alongside what. When current state drifts from original intent, you can diff the snapshots chronologically.

• Avoiding repeated mistakes: Session index lets you scan past sessions by topic. "Did I already try X?" is a search over one-liners, not a keyword hunt through thousands of lines of transcript.

The counterintuitive part was overwrite, not append for working state. Early versions appended everything and the record grew until it was unsearchable. Forcing an overwrite at session close compresses experience into decision residue rather than accumulating raw history. Token efficiency comes from the separation — you load the snapshot (small) and the index (one-liners) at session start. Full archives only get accessed on demand.

I'm not using Memora for this yet — I built the persistence layer in SQLite and it runs at session close through a skill. But the design questions in the original issue are about whether Memora's existing structure could carry this pattern natively, which would be cleaner than external tooling. Happy to share the snapshot format if it's useful for thinking through Memora's session model.

[spokV]

Great discussion. The way I see it, there are three fundamental questions before any implementation:

1. What's the balance between context pollution and making the model smarter? Every memory loaded is a token spent. Load too much and you drown the context window with irrelevant history. Load too little and the agent repeats past mistakes. The answer we're converging on is a memory_kind hierarchy -- state (compact project facts, always loaded), episodic (session summaries, on-demand), artifact (raw logs, suppressed by default). Retrieval widens progressively only when local results are insufficient.
2. What are the optimal collection/injection points? You can't pause the agent to "think about memory" on every turn -- it kills flow. Our current thinking: structured deltas (branch, todos, files touched) appended on each turn end via hooks -- cheap, no LLM. Full compression (rationale, decisions, dead ends) only at session close. Injection at session start loads the state snapshot automatically. This keeps the hot path fast and the capture comprehensive.
3. How do you evaluate it? This is the hardest one. Token efficiency is measurable (snapshot size vs raw transcript size). But "did the memory actually help?" is subjective. Possible signals: fewer repeated mistakes across sessions, faster cold-start (agent reaches productive state sooner), successful cross-project pattern reuse. Would love to hear if anyone has a concrete evaluation framework for this.

On the implementation side, we're planning: hybrid compression (structured + LLM), cross-project search via stable project_key (git remote hash, not cwd basename), session lifecycle MCP tools, and backward-compatible schema changes.

@jwade83 -- would love to see your snapshot format. The structured facts we're planning for the hot path probably overlap with what you're already capturing.

[jwade83]

@spokV Your state/episodic/artifact hierarchy maps closely to what I've ended up with through iteration:

Your memory_kind	My equivalent	Persistence rule
state (compact, always loaded)	State snapshot	Overwritten at session close
episodic (session summaries, on-demand)	Session index	One line appended per session
artifact (raw logs, suppressed)	Append-only log	Full snapshot archived, never edited

Snapshot format (what you'd call "state"):

# Overwritten every session close. 30-40 lines max.

## What Just Happened
# 1-2 sentences. What was accomplished, any reframe.

## In Progress
# Active items with status. Only what needs continuation.

## Decisions Made This Session
# Bullet list — the "why" alongside the "what."
# Highest-value field: recovers reasoning, not just state.

## Next Session
# "Start with:" verb + object (not "review X" but "draft Y for Z")
# "Also ready:" other unblocked items

## Open Questions
# Unresolved items carrying forward.

The compression rule: if it exceeds ~40 lines, you're writing a summary, not a handoff. The overwrite forces triage — what matters for cold-start vs. what was working noise.

Session index (what you'd call "episodic") is one line per session, scannable without loading content:

2026-03-25 | Draft spokV reply | Pivoted from table to metadata approach | draft_38_spokv_reply.md

On hooks vs. close-only:

Your hooks approach — structured deltas per-turn, no LLM — solves the biggest failure mode of close-only: if a session ends abruptly (context limit, crash, user closes), everything is lost. Structured deltas are insurance.

I've been running close-only for ~80 sessions, and the complementary case for it: LLM compression at close produces better decision residue because the model has full session context when it compresses. Close-time triage is more accurate than real-time capture for deciding what carries forward. I'd want both: hooks for structured facts (branch, files, todos) as a safety net, LLM compression at close as the primary snapshot.

On implementation:

Your metadata-based approach — memory_kind as a field rather than a new table — makes more sense than what I originally suggested. A session state is a memory with memory_kind: "state" and a project-scoped tag. The important thing is the persistence semantics (overwrite for state, append for episodic, archive for artifact), not the storage structure. The separate table idea was me projecting file-based architecture onto a database.

On evaluation — the harder question:

Your third question is the one I keep circling back to. I have ~80 sessions of data running this architecture, but I'm wary of designing an evaluation framework that just confirms the system I already built. What you test for shapes what you optimize, and the dimensions I can see are limited by the vantage point of the architecture I'm inside.

With that caveat, the ones I've been thinking about:

• Compression efficiency — snapshot size vs. raw transcript. Easy to measure, but does a high ratio mean good compression or just information loss?
• Decision survival — do decisions captured at session N actually appear in session N+1's context? Tracks whether compression retains reasoning or just state.
• Question continuity — open questions carried forward until resolved vs. silently dropped. Measures whether the system prevents thread loss across context windows.
• Graceful degradation — what happens when a session terminates abruptly before close-time compression? This is a dimension my architecture handles poorly, and where hooks-based capture would produce different evaluation constraints entirely.
• Drift detection — sessions since last external signal as a proxy for productive procrastination. Correlates with actual outcomes, but the correlation might be circumstantial.

Those are all system-side metrics — they measure what the persistence layer does. What I can't yet measure is what it enables: fewer repeated mistakes across sessions, faster pattern reuse across projects, whether the operator actually made a better decision because the snapshot carried the reasoning forward. Your framing of those outcome signals is where the harder evaluation work lives, and where I'd most want to hear what you'd prioritize.

I have enough session data to backtest against whatever framework makes sense, and happy to run new sessions against a shared set of dimensions going forward if that's useful.

[spokV]

@jwade83 Your snapshot format landed almost verbatim in our implementation. Please check dev branch — here's where things stand.

What's live:

• Hooks on SessionStart/Stop/UserPromptSubmit write structured deltas per turn (~1ms, no LLM)
• Auto-close on next session start triggers LLM compression (grok-3-mini via OpenRouter)
• LLM produces two outputs: a ~40 line branch state snapshot using your template and an uncapped episodic narrative stored as a searchable memory
• Branch state overwrites on close, episodic appends. Next session cold-starts from the snapshot automatically via hook context injection
• Hooks as insurance, LLM at close as primary — exactly your recommendation

Snapshot schema (what the LLM is prompted to produce):

What Just Happened - 1-2 sentences. What was accomplished, any reframe.
In Progress - Active items with status. Only what needs continuation.
Decisions Made - Bullet list — the "why" alongside the "what".
Next Session - "Start with:" verb + object. "Also ready:" other unblocked items.
Open Questions - Unresolved items carrying forward.

Stored in branch_state.snapshot as JSON with a narrative field (as above) plus structured fields (open_todos, touched_files, decisions, errors_encountered). The structured fields are extracted separately so they're queryable without parsing markdown.

On evaluation:
Your "decision survival" metric is the most actionable — did decisions from session N appear in session N+1's context? We can measure this mechanically by diffing the branch state snapshot against the next session's injected context.
"Graceful degradation" is where hooks-based capture pays off. We tested killing Claude mid-session and the deltas survived. Close-only would have lost everything.
The outcome signals (fewer repeated mistakes, faster cold-start) need a baseline. One approach: paired sessions on the same task, with and without session memory, measuring turns-to-productive-state.

Still validating the structured snapshot output end-to-end — the compression prompt and episodic storage just landed. Will share results once we have a few real sessions through it.

[spokV]

I'm maybe flying high here, but eventually i think we need decentralized p2p memory sharing with auth/enc to sync multiple claude nodes in remote/local machines with context injection. This is what memora can be.

[jwade83]

@spokV The JSON+narrative split makes sense for the Memora use case — programmatic queries without requiring LLM parsing, which makes sense for a multi-user product. For scenarios running markdown with a single operator, human and LLM reading the same files directly, the architectures are each solving their own problems. Whether both are testable against shared evaluation dimensions would be an interesting question — compression efficiency, decision survival, question continuity.

[hegu-1]

The state / episodic / artifact split in this thread is the right shape. The next boundary I would add is ownership and promotion. A compressed session narrative is still model-generated; it should not automatically become the same class of truth as a user-authored decision or preference.

One workable schema is:

• artifact: raw transcript chunks, private by default, append-only;
• episode: generated session narrative with source chunk ids and compressor version;
• state: current branch snapshot, replaceable, with based_on_episode_ids;
• memory: explicitly promoted durable item, with author/reviewer and supersession state.

Search can span all four, but injection policy should differ. State may cold-start a session, episodes are retrieved on demand, artifacts are evidence, and promoted memory can influence planning. That prevents a lossy summary from silently hardening into identity.

For the p2p direction mentioned above, sync promoted items and reviewed snapshots first; keep raw artifacts local unless explicitly shared. Provenance ids should survive sync so a receiving node can show why an item exists without needing the full transcript.

This maps closely to the continuity/snapshot/trace distinction documented in Personal AI OS: https://github.com/hegu-1/personal-ai-os/blob/main/docs/continuity-snapshot-trace.md