Skip to content

feat(kb): memory ⇄ KB links — inline titles, link-style memory, autolink (K-D)#282

Merged
oratis merged 6 commits into
mainfrom
feat/kb-memory-links
Jul 24, 2026
Merged

feat(kb): memory ⇄ KB links — inline titles, link-style memory, autolink (K-D)#282
oratis merged 6 commits into
mainfrom
feat/kb-memory-links

Conversation

@oratis

@oratis oratis commented Jul 23, 2026

Copy link
Copy Markdown
Owner

Part of the knowledge-base v2.0 stack (plan: #278, handoff: #281). Stacked on #281 — merge order: #278 → #279 → #280 → #281 → this.

差距

v2.0 用户需求 ③：memory 里存 link 而不是内容。K-C 之后 [[链接]] 已经是真实的图，但 memory 侧没有闭环——memory 里的 [[kb:slug]] 在提示里只是一个不透明的哈希，Lisa 不知道背后是什么、值不值得 kb_read；也没有任何机制引导"知识进 KB、memory 只留指针"。

做了什么

  1. src/kb/memory-links.ts — 组装系统提示时，MEMORY.md / USER.md 里可解析的 [[kb:slug]] 就地补标题：[[kb:oauth]] → [[kb:oauth]](OAuth 与 PKCE)。解析不到的原样保留（那是"该写这页"的信号）。
  2. memory 工具描述 — 明确指引：知识本体进 KB（kb_add / kb_write），memory 只存 [[kb:slug]] 指针（memory 上限 4KB）。
  3. idle/runner.ts 蒸馏提示 — 写完 wiki 页后，若 memory 有相关条目却没 link，用 memory 工具补一条 [[kb:slug]]。这是"memory 调取 link"自动闭环的落点。
  4. src/kb/autolink.ts — 保守自动互链，kb_write 时应用：已有 wiki 页标题的全词匹配转 [[链接]]。只匹配标题、每个标题只链首次出现、长标题优先、跳过代码块（含缩进代码）/已有链接/MD 链接、短标题（ASCII<3、CJK<2）直接不链、upsert 排除自身。

为什么这么选

  • 标题解析走 kb/index.json + stat 指纹缓存，不读全量 KB——交接文档给了两个选项（kbFingerprint 式缓存 vs 直接读 index.json），选了推荐的后者：buildSystemPromptSnapshot 每轮都跑，这条路径常态成本必须是一次 stat()。index.json 在每次 KB 变更时由 store 重新生成，新鲜度有保证；损坏/写入中则本轮跳过注入，不炸提示。
  • autolink 宁可漏不可错：index.md 按（反链 × 新鲜度）排序，错误边直接污染 always-on 提示的排序信号。
  • 文档没写的一个拍板：标题≠slug 时 autolink 产出 [[slug|Title]] 而不是裸 [[slug]]——中文标题的 slug 是 date-hash（D6），裸 slug 会把正文变成不可读的哈希；links.ts 本来就支持 display text，图的边不受影响。同理提示注入格式 [[kb:slug]](Title) 保持链接本体不动，只追加标题。

测了什么

  • autolink.test.ts（11 例）：全词/首次/最长优先/代码围栏（fenced、inline、缩进、未闭合 fence）/已有链接/边界/CJK/短标题。
  • memory-links.test.ts：纯函数注解各形态 + 对着 store 生成的真实 index.json 的集成（含缓存失效）。
  • prompt.test.ts 新增：memory 指针在快照里带标题、解析不到的原样保留。
  • tool.test.ts 新增：kb_write 自动互链 + upsert 不自链。
  • npm run typecheck 干净；npm test 1206 pass / 0 fail（基线 1184）。

🤖 Generated with Claude Code

oratis and others added 5 commits July 23, 2026 14:12
@oratis @claude
docs: knowledge base v2.0 plan (ingest · daily brief · link graph)
8683914 
Design of record for three KB capabilities on top of v1.0 (#235-#242):

- K1 paste any link (web / 公众号 / B站 / YouTube) -> Markdown source
- K2 a watchlist -> daily fetched, classified, summarized brief
- K3 [[slug]] parsed into a real link graph; index.md becomes a ranked
  map-of-content; memory stores [[kb:slug]] links instead of content

Review of the shipped v1.0 code found five gaps this plan closes, one of
them a confirmed bug: both TF-IDF tokenizers (kb/search.ts:45 and
memory/vector.ts:212) split on whitespace, so a whole CJK run collapses
into a single token and Chinese content only matches an exact
whole-string query. Also: normalizeSlug strips non-ASCII so every Chinese
title falls back to entry-<timestamp>; web_fetch is lossy plain text with
no metadata and no store; [[slug]] is documented in SCHEMA.md but nothing
parses it; and there is no KB scheduler at all.

Includes the pro/con debate for each decision (self-built extractor vs
deps vs remote reader, immutable sources vs re-ingest, autonomous ingest
as a prompt-injection path, consent, ASCII vs CJK slugs, brief storage,
and RSSHub for feeds without one) and a nine-PR ship order.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@oratis @claude
fix(kb): CJK-safe search + slugs, and provenance frontmatter
21ea9ca 
Three foundation fixes the rest of the KB v2 work stands on
(docs/PLAN_KNOWLEDGE_BASE_v2.0.md K-B). All three are things that would
have corrupted or lost data once link ingest starts pouring Chinese
content in, so they land before the firehose.

1. CJK search was broken. Both TF-IDF tokenizers (kb/search.ts,
   memory/vector.ts) carried the same three lines: lowercase, strip
   non-word chars, split on whitespace. Chinese has no whitespace, so a
   whole clause survived as ONE token:

     doc   "这篇公众号文章讲的是知识库的设计" -> one token
     query "知识库"                          -> one token
     intersection = empty  ->  zero hits

   i.e. CJK content was only findable by an exact whole-clause query.
   Replaced by a shared src/tokenize.ts that emits character bigrams for
   CJK runs (plus the run itself, so an exact phrase still ranks
   highest), segments at latin/CJK boundaries so "gpt模型" indexes as
   "gpt" + CJK bigrams, and now keeps kana and CJK Ext-A, which the old
   character class dropped entirely. Latin tokenization is unchanged.

2. Chinese titles produced throwaway slugs. normalizeSlug strips every
   non-[a-z0-9] char, so a CJK title normalized to "" and addSource fell
   back to entry-<Date.now()> — opaque, unstable, un-dedupable. New
   kb/slug.ts mints <date>-<sha256/8> for titles with no usable latin,
   keeping slugs ASCII on purpose (macOS NFD vs git NFC filenames, URLs,
   git paths) with the real title in frontmatter. canonicalUrl() drops
   utm_*/known tracking params so the same article shared twice
   canonicalizes — and hashes — identically.

3. Entries can now carry arbitrary frontmatter via KbEntry.extra,
   round-tripped on read/write. This is where ingest provenance goes
   (url/site/author/published/hash/via) without the store knowing about
   any of it, and it means keys a future version adds survive a
   round-trip through an older one. Values are whitespace-collapsed on
   write so a newline can't forge frontmatter lines on the next read.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@oratis @claude
feat(kb): parse [[links]] into a real graph; index.md becomes a ranke…
1cdb6b4 
…d MOC

v1.0 told Lisa to cross-link wiki pages with [[slug]] (SCHEMA.md) and
recorded each page's sources: in frontmatter — but nothing ever read
either one. The KB was a pile of files with decorative links.

src/kb/links.ts builds the graph from three edge sources ([[slug]],
[[slug|alias]], [[kb:slug]] in bodies; a wiki page's sources:
frontmatter) and derives what you can't get from search: backlinks,
hubs, orphans, and links pointing at pages that don't exist yet.

index.md is now a ranked map of content rather than a flat listing.
It is injected into EVERY system prompt and hard-capped there
(prompt.ts:165), so with a flat list the truncation lands at an
arbitrary point once the KB grows and whatever fell below the line stops
existing as far as Lisa is concerned. Ordering wiki pages by
(1 + backlinks) x recency means the cut always eats the least-connected
tail. Orphans and broken links are listed as the tending to-do list, and
every truncation says how much it dropped instead of trailing off.

Sources are now listed by TITLE ONLY. Layer 1 is raw captured material —
after link ingest lands that means whole web pages — and index.md is
always-on prompt text, so an excerpt there is a direct "any page on the
internet -> Lisa's system prompt" path. This is a stronger form of the
containment the plan assigned to K-I (it covers every source, not just
origin: web), and it costs nothing: a title is all a map needs, and the
body is one kb_read away. Wiki gists stay — Lisa wrote those herself.

Also: kb/index.json (the machine-readable graph, for the web UI and
tools), a kb_links tool (links to / linked from / shares tags with), and
kb_read now takes a bare slug or a [[wikilink]] with the layer optional,
and appends the pages that link to what you just read.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@oratis @claude
docs(kb): handoff for the remaining knowledge-base v2 work (K-D … K-I)
72c1488 
K-A/K-B/K-C are open as #278/#279/#280. This is the pickup document for
whoever continues: the branch stack and how not to break it, the APIs the
first three PRs landed, a file-by-file spec for each remaining PR, the
repo conventions that are easy to trip over (LISA_HOME must be set before
import, the tool-subset triple, the giant template literals in
lisa-client.ts), and the eight decisions that are already settled so they
don't get re-litigated.

Two things it corrects against the plan: the index-side injection
containment assigned to K-I already shipped in K-C, and in a stronger
form (every source is title-only in index.md, not just origin: web), so
K-I is down to the watchlist restriction and the kb_read fence.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@oratis @claude
feat(kb): memory ⇄ KB links — inline titles, link-style memory, conse…
fe700da 
…rvative autolink (K-D)

Closes the "memory stores links, not content" loop from the v2.0 plan (user
need ③). Four pieces:

- prompt.ts inlines page titles for [[kb:slug]] pointers found in MEMORY.md /
  USER.md at assembly time ([[kb:oauth]] → [[kb:oauth]](OAuth 与 PKCE)), so a
  pointer is meaningful without a kb_read. Titles come from the already-
  generated kb/index.json behind a stat-fingerprint cache — the prompt builds
  every turn, so no full-KB read on this path. Unresolvable links are kept
  verbatim: a dangling pointer is a "write this page" signal, not noise.
- memory tool description now steers knowledge OUT of memory (4KB cap) and
  into the KB with a [[kb:slug]] pointer.
- The idle tending prompt closes the loop from the other side: after
  distilling a wiki page, back-link it from any related memory entry.
- kb/autolink.ts: conservative title → [[link]] pass applied on kb_write.
  Whole-word title matches only, first occurrence, longest-first, never in
  code / existing links, short titles skipped — a wrong edge pollutes the
  (backlinks × recency) hub ranking the always-on index sorts by, so misses
  are preferred over false links. Emits [[slug|Title]] when the slug differs
  so CJK prose keeps reading as prose.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@oratis oratis mentioned this pull request Jul 23, 2026
Merged
@oratis

oratis commented Jul 24, 2026

Copy link
Copy Markdown
Owner Author

Solid, well-tested, and the graph semantics are right: autolink only links existing wiki pages (no dangling edges), excludes the page being written (no self-link on upsert), longest-title-wins, and skips code/existing links; cycles are harmless. Fast-path + stat-fingerprint cache on index.json is the right cost model for a per-turn path.

One hardening request before this lands on top of #283: the inlined title goes verbatim into the always-on prompt (memory-links.ts:392, ${match}(${title})) and into [[slug|title]] (autolink.ts:254). Titles are user/Lisa-authored today, but #283 (stacked directly above) feeds untrusted web og:title/<title> into that exact field. Please collapse whitespace / strip newlines on the inlined title (one-liner) so a crafted page title can't inject extra lines into the memory section of the prompt.

Nit: cache key is mtimeMs+size — a same-millisecond, same-size rewrite can serve a stale title. Matches the existing kbFingerprint pattern, so fine to leave.

@oratis
oratis changed the base branch from feat/kb-handoff to main July 24, 2026 07:21
@oratis
Merge remote-tracking branch 'origin/main' into feat/kb-memory-links
14268c0 
# Conflicts:
#	src/kb/links.test.ts
#	src/kb/links.ts
#	src/kb/store.test.ts
#	src/kb/store.ts
#	src/kb/tool.ts
@oratis

oratis commented Jul 24, 2026

Copy link
Copy Markdown
Owner Author

Reconciled onto main (resolved links/store/tool against the merged K-B/K-C, kept #282's autolink/memory-links additions). typecheck + 93 KB/idle tests green. Merging. (The inlined-title hardening is already covered by the store.ts title-collapse landed in #279.)

@oratis
oratis merged commit a2c82e9 into main Jul 24, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@oratis