feat(kb): parse [[links]] into a real graph; index.md becomes a ranked MOC (K-C) by oratis · Pull Request #280 · oratis/LISA

oratis · 2026-07-23T06:24:08Z

Plan §K-C. Stacked on #279.

The gap

v1.0 told Lisa to cross-link wiki pages with [[slug]] (src/kb/schema.ts:20) and recorded each page's sources: in frontmatter — but nothing in the repo ever read either one. The KB was a pile of files with decorative links: no backlinks, no way to see what a page connects to, no way to tell a hub from an orphan.

What lands

src/kb/links.ts builds the graph from three edge sources — [[slug]], [[slug|alias]], [[kb:slug]] in bodies, plus a wiki page's sources: frontmatter — and derives what search can't give you: backlinks, hubs, orphans, and links pointing at pages that don't exist yet. A bare [[slug]] resolves wiki-first, since a link written in prose means the concept page, not a raw capture that happens to share the slug.

index.md is now a ranked map of content, not a flat listing. It's injected into every system prompt and hard-capped there (src/prompt.ts:165), so with a flat list the truncation lands at an arbitrary point once the KB grows — and whatever fell below the line stops existing as far as Lisa is concerned. Ordering by (1 + backlinks) × recency means the cut always eats the least-connected tail. Orphans and broken links are listed as the wiki's to-do list (what the idle tending pass acts on), and every truncation states how much it dropped rather than trailing off silently.

Sources are listed by title only. Layer 1 is raw captured material — once link ingest lands, that means whole web pages — and index.md is always-on prompt text, so an excerpt there is a direct any page on the internet → Lisa's system prompt path. This is a stronger form of the containment the plan assigned to K-I (it covers every source, not just origin: web), and it costs nothing: a title is all a map needs and the body is one kb_read away. Wiki gists stay — Lisa wrote those herself. There's a test that feeds an injection string through a source body and asserts it never reaches the index.

Also:

kb/index.json — the machine-readable graph, for the web UI and tools.
kb_links(slug) — links to / linked from / shares tags with. Read-only, remote-safe.
kb_read now takes a bare slug or a [[wikilink]] with layer optional (the model routinely passes wikilinks verbatim), and appends the pages that link to what you just read.

Tests

+27 tests. Full suite 1184 pass / 0 fail, typecheck clean. Notable: hub ordering puts a stale orphan last, an injection string in a source body never reaches index.md, and a 40-page/300-source KB still renders under 8 KB.

🤖 Generated with Claude Code

Design of record for three KB capabilities on top of v1.0 (#235-#242): - K1 paste any link (web / 公众号 / B站 / YouTube) -> Markdown source - K2 a watchlist -> daily fetched, classified, summarized brief - K3 [[slug]] parsed into a real link graph; index.md becomes a ranked map-of-content; memory stores [[kb:slug]] links instead of content Review of the shipped v1.0 code found five gaps this plan closes, one of them a confirmed bug: both TF-IDF tokenizers (kb/search.ts:45 and memory/vector.ts:212) split on whitespace, so a whole CJK run collapses into a single token and Chinese content only matches an exact whole-string query. Also: normalizeSlug strips non-ASCII so every Chinese title falls back to entry-<timestamp>; web_fetch is lossy plain text with no metadata and no store; [[slug]] is documented in SCHEMA.md but nothing parses it; and there is no KB scheduler at all. Includes the pro/con debate for each decision (self-built extractor vs deps vs remote reader, immutable sources vs re-ingest, autonomous ingest as a prompt-injection path, consent, ASCII vs CJK slugs, brief storage, and RSSHub for feeds without one) and a nine-PR ship order. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Three foundation fixes the rest of the KB v2 work stands on (docs/PLAN_KNOWLEDGE_BASE_v2.0.md K-B). All three are things that would have corrupted or lost data once link ingest starts pouring Chinese content in, so they land before the firehose. 1. CJK search was broken. Both TF-IDF tokenizers (kb/search.ts, memory/vector.ts) carried the same three lines: lowercase, strip non-word chars, split on whitespace. Chinese has no whitespace, so a whole clause survived as ONE token: doc "这篇公众号文章讲的是知识库的设计" -> one token query "知识库" -> one token intersection = empty -> zero hits i.e. CJK content was only findable by an exact whole-clause query. Replaced by a shared src/tokenize.ts that emits character bigrams for CJK runs (plus the run itself, so an exact phrase still ranks highest), segments at latin/CJK boundaries so "gpt模型" indexes as "gpt" + CJK bigrams, and now keeps kana and CJK Ext-A, which the old character class dropped entirely. Latin tokenization is unchanged. 2. Chinese titles produced throwaway slugs. normalizeSlug strips every non-[a-z0-9] char, so a CJK title normalized to "" and addSource fell back to entry-<Date.now()> — opaque, unstable, un-dedupable. New kb/slug.ts mints <date>-<sha256/8> for titles with no usable latin, keeping slugs ASCII on purpose (macOS NFD vs git NFC filenames, URLs, git paths) with the real title in frontmatter. canonicalUrl() drops utm_*/known tracking params so the same article shared twice canonicalizes — and hashes — identically. 3. Entries can now carry arbitrary frontmatter via KbEntry.extra, round-tripped on read/write. This is where ingest provenance goes (url/site/author/published/hash/via) without the store knowing about any of it, and it means keys a future version adds survive a round-trip through an older one. Values are whitespace-collapsed on write so a newline can't forge frontmatter lines on the next read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…d MOC v1.0 told Lisa to cross-link wiki pages with [[slug]] (SCHEMA.md) and recorded each page's sources: in frontmatter — but nothing ever read either one. The KB was a pile of files with decorative links. src/kb/links.ts builds the graph from three edge sources ([[slug]], [[slug|alias]], [[kb:slug]] in bodies; a wiki page's sources: frontmatter) and derives what you can't get from search: backlinks, hubs, orphans, and links pointing at pages that don't exist yet. index.md is now a ranked map of content rather than a flat listing. It is injected into EVERY system prompt and hard-capped there (prompt.ts:165), so with a flat list the truncation lands at an arbitrary point once the KB grows and whatever fell below the line stops existing as far as Lisa is concerned. Ordering wiki pages by (1 + backlinks) x recency means the cut always eats the least-connected tail. Orphans and broken links are listed as the tending to-do list, and every truncation says how much it dropped instead of trailing off. Sources are now listed by TITLE ONLY. Layer 1 is raw captured material — after link ingest lands that means whole web pages — and index.md is always-on prompt text, so an excerpt there is a direct "any page on the internet -> Lisa's system prompt" path. This is a stronger form of the containment the plan assigned to K-I (it covers every source, not just origin: web), and it costs nothing: a title is all a map needs, and the body is one kb_read away. Wiki gists stay — Lisa wrote those herself. Also: kb/index.json (the machine-readable graph, for the web UI and tools), a kb_links tool (links to / linked from / shares tags with), and kb_read now takes a bare slug or a [[wikilink]] with the layer optional, and appends the pages that link to what you just read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

oratis · 2026-07-24T07:02:34Z

The graph model is clean (pure buildGraph, no traversal so cycles are inert) and the title-only source containment is the right call — nicely tested. One blocker plus a couple of wiring issues.

Blocker — ReDoS in WIKILINK (src/kb/links.ts). Catastrophic backtracking, measured:

"[[" + " ".repeat(3000)  -> 6.1 s
"[[" + " ".repeat(5000)  -> 25.6 s
"[[" + " ".repeat(10000) -> did not finish (>2 min)

(tabs trigger it too; "a"*3000 is 0 ms.) The greedy leading/trailing \s* overlap the whitespace-including lazy class [^\]|#\n]+?, so a [[ with a long whitespace run and no closing ]] blows up. This is not hypothetical: parseWikilinks runs over every entry body, including sources, inside buildGraph, which fires on every KB mutation and every kb_read/kb_links. addSource bodies are already arbitrary captures today and become whole web pages after K1 — one crafted or whitespace-heavy page bricks the KB subsystem. Note: bounding the class alone is not enough (I tested it — the trailing \s* still hangs on 100 K). Drop the surrounding \s* (you already .trim() the capture) and bound the lengths:

const WIKILINK = /\[\[(?:kb:)?([^\]|#\n]{1,200}?)(?:\|[^\]\n]{0,200})?\]\]/g;

Verified: 0 ms on 100 K spaces, and identical output on every existing case (dedup, |alias, kb:, non-spanning, [[ spaced ]] → spaced). Please add a pathological-input test.

Perf — kb_read is now O(N). It calls listFullEntries() + buildGraph() on every read (all files, full bodies, target read twice) purely to show backlinks — and re-runs the parser above over everything. Cache the graph by KB-dir fingerprint the way search.ts caches its index. regenerateIndexLocked also builds the graph twice (once in renderIndex, once for index.json) — build once, pass both.

Wiring — the to-do list falls off the bottom. Orphans/broken render after up to 40 wiki + 24 tags + 25 sources. renderIndex targets ~8 KB, but the idle pass caps index.md at 1200 chars (idle/runner.ts:149) and the live prompt at 2600 (prompt.ts:163), so orphans/broken — the exact thing the idle "tend the wiki" pass is told to act on — get sliced off in any real KB. Move those sections above sources, or align renderIndex's budget with the caps.

Minor: [[slug#anchor]] currently matches nothing (class excludes #, but nothing consumes the anchor) — decide whether to resolve to the base slug or report it. And source titles (attacker-controlled for web ingests) still reach index.md verbatim even though bodies don't — worth length-capping them.

The WIKILINK regex `/\[\[\s*(?:kb:)?([^\]|#\n]+?)\s*(?:\|[^\]\n]*)?\]\]/g` backtracked catastrophically on a `[[` followed by a long whitespace run with no closing `]]` (~6s at 3k chars, >2min at 10k; tabs too). parseWikilinks runs over every entry body — including untrusted `sources` (pasted / web_fetch / chat captures, and whole web pages once ingest lands) — inside buildGraph, on every KB mutation and every kb_read/kb_links, so one crafted or whitespace-heavy body could hang the KB subsystem. Drop the surrounding `\s*` (the capture is already .trim()ed at the call site) and length-bound target/alias. Verified 0ms on 100k chars with byte-identical output on all existing cases. +1 pathological-input regression test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

oratis · 2026-07-24T07:20:53Z

Pushed the ReDoS fix (8e10da0): dropped the surrounding \s* and length-bounded the target/alias — 0ms on 100k chars, byte-identical output on all existing cases, +regression test. typecheck + 18 links tests green. Merging. (The O(N) kb_read/double-graph-build and orphans/broken index ordering remain as noted — good follow-ups, not blockers.)

# Conflicts: # src/kb/store.test.ts # src/kb/store.ts

#281) * docs: knowledge base v2.0 plan (ingest · daily brief · link graph) Design of record for three KB capabilities on top of v1.0 (#235-#242): - K1 paste any link (web / 公众号 / B站 / YouTube) -> Markdown source - K2 a watchlist -> daily fetched, classified, summarized brief - K3 [[slug]] parsed into a real link graph; index.md becomes a ranked map-of-content; memory stores [[kb:slug]] links instead of content Review of the shipped v1.0 code found five gaps this plan closes, one of them a confirmed bug: both TF-IDF tokenizers (kb/search.ts:45 and memory/vector.ts:212) split on whitespace, so a whole CJK run collapses into a single token and Chinese content only matches an exact whole-string query. Also: normalizeSlug strips non-ASCII so every Chinese title falls back to entry-<timestamp>; web_fetch is lossy plain text with no metadata and no store; [[slug]] is documented in SCHEMA.md but nothing parses it; and there is no KB scheduler at all. Includes the pro/con debate for each decision (self-built extractor vs deps vs remote reader, immutable sources vs re-ingest, autonomous ingest as a prompt-injection path, consent, ASCII vs CJK slugs, brief storage, and RSSHub for feeds without one) and a nine-PR ship order. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(kb): CJK-safe search + slugs, and provenance frontmatter Three foundation fixes the rest of the KB v2 work stands on (docs/PLAN_KNOWLEDGE_BASE_v2.0.md K-B). All three are things that would have corrupted or lost data once link ingest starts pouring Chinese content in, so they land before the firehose. 1. CJK search was broken. Both TF-IDF tokenizers (kb/search.ts, memory/vector.ts) carried the same three lines: lowercase, strip non-word chars, split on whitespace. Chinese has no whitespace, so a whole clause survived as ONE token: doc "这篇公众号文章讲的是知识库的设计" -> one token query "知识库" -> one token intersection = empty -> zero hits i.e. CJK content was only findable by an exact whole-clause query. Replaced by a shared src/tokenize.ts that emits character bigrams for CJK runs (plus the run itself, so an exact phrase still ranks highest), segments at latin/CJK boundaries so "gpt模型" indexes as "gpt" + CJK bigrams, and now keeps kana and CJK Ext-A, which the old character class dropped entirely. Latin tokenization is unchanged. 2. Chinese titles produced throwaway slugs. normalizeSlug strips every non-[a-z0-9] char, so a CJK title normalized to "" and addSource fell back to entry-<Date.now()> — opaque, unstable, un-dedupable. New kb/slug.ts mints <date>-<sha256/8> for titles with no usable latin, keeping slugs ASCII on purpose (macOS NFD vs git NFC filenames, URLs, git paths) with the real title in frontmatter. canonicalUrl() drops utm_*/known tracking params so the same article shared twice canonicalizes — and hashes — identically. 3. Entries can now carry arbitrary frontmatter via KbEntry.extra, round-tripped on read/write. This is where ingest provenance goes (url/site/author/published/hash/via) without the store knowing about any of it, and it means keys a future version adds survive a round-trip through an older one. Values are whitespace-collapsed on write so a newline can't forge frontmatter lines on the next read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(kb): parse [[links]] into a real graph; index.md becomes a ranked MOC v1.0 told Lisa to cross-link wiki pages with [[slug]] (SCHEMA.md) and recorded each page's sources: in frontmatter — but nothing ever read either one. The KB was a pile of files with decorative links. src/kb/links.ts builds the graph from three edge sources ([[slug]], [[slug|alias]], [[kb:slug]] in bodies; a wiki page's sources: frontmatter) and derives what you can't get from search: backlinks, hubs, orphans, and links pointing at pages that don't exist yet. index.md is now a ranked map of content rather than a flat listing. It is injected into EVERY system prompt and hard-capped there (prompt.ts:165), so with a flat list the truncation lands at an arbitrary point once the KB grows and whatever fell below the line stops existing as far as Lisa is concerned. Ordering wiki pages by (1 + backlinks) x recency means the cut always eats the least-connected tail. Orphans and broken links are listed as the tending to-do list, and every truncation says how much it dropped instead of trailing off. Sources are now listed by TITLE ONLY. Layer 1 is raw captured material — after link ingest lands that means whole web pages — and index.md is always-on prompt text, so an excerpt there is a direct "any page on the internet -> Lisa's system prompt" path. This is a stronger form of the containment the plan assigned to K-I (it covers every source, not just origin: web), and it costs nothing: a title is all a map needs, and the body is one kb_read away. Wiki gists stay — Lisa wrote those herself. Also: kb/index.json (the machine-readable graph, for the web UI and tools), a kb_links tool (links to / linked from / shares tags with), and kb_read now takes a bare slug or a [[wikilink]] with the layer optional, and appends the pages that link to what you just read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(kb): handoff for the remaining knowledge-base v2 work (K-D … K-I) K-A/K-B/K-C are open as #278/#279/#280. This is the pickup document for whoever continues: the branch stack and how not to break it, the APIs the first three PRs landed, a file-by-file spec for each remaining PR, the repo conventions that are easy to trip over (LISA_HOME must be set before import, the tool-subset triple, the giant template literals in lisa-client.ts), and the eight decisions that are already settled so they don't get re-litigated. Two things it corrects against the plan: the index-side injection containment assigned to K-I already shipped in K-C, and in a stronger form (every source is title-only in index.md, not just origin: web), so K-I is down to the watchlist restriction and the kb_read fence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

…#281 A tooling worktree symlinked node_modules to share deps; `.gitignore` matched only the directory (`node_modules/`), so the symlink file slipped past ignore and into #280/#281. Remove it from tracking and tighten the ignore to `node_modules` (matches dir and symlink). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ink (K-D) (#282) * docs: knowledge base v2.0 plan (ingest · daily brief · link graph) Design of record for three KB capabilities on top of v1.0 (#235-#242): - K1 paste any link (web / 公众号 / B站 / YouTube) -> Markdown source - K2 a watchlist -> daily fetched, classified, summarized brief - K3 [[slug]] parsed into a real link graph; index.md becomes a ranked map-of-content; memory stores [[kb:slug]] links instead of content Review of the shipped v1.0 code found five gaps this plan closes, one of them a confirmed bug: both TF-IDF tokenizers (kb/search.ts:45 and memory/vector.ts:212) split on whitespace, so a whole CJK run collapses into a single token and Chinese content only matches an exact whole-string query. Also: normalizeSlug strips non-ASCII so every Chinese title falls back to entry-<timestamp>; web_fetch is lossy plain text with no metadata and no store; [[slug]] is documented in SCHEMA.md but nothing parses it; and there is no KB scheduler at all. Includes the pro/con debate for each decision (self-built extractor vs deps vs remote reader, immutable sources vs re-ingest, autonomous ingest as a prompt-injection path, consent, ASCII vs CJK slugs, brief storage, and RSSHub for feeds without one) and a nine-PR ship order. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(kb): CJK-safe search + slugs, and provenance frontmatter Three foundation fixes the rest of the KB v2 work stands on (docs/PLAN_KNOWLEDGE_BASE_v2.0.md K-B). All three are things that would have corrupted or lost data once link ingest starts pouring Chinese content in, so they land before the firehose. 1. CJK search was broken. Both TF-IDF tokenizers (kb/search.ts, memory/vector.ts) carried the same three lines: lowercase, strip non-word chars, split on whitespace. Chinese has no whitespace, so a whole clause survived as ONE token: doc "这篇公众号文章讲的是知识库的设计" -> one token query "知识库" -> one token intersection = empty -> zero hits i.e. CJK content was only findable by an exact whole-clause query. Replaced by a shared src/tokenize.ts that emits character bigrams for CJK runs (plus the run itself, so an exact phrase still ranks highest), segments at latin/CJK boundaries so "gpt模型" indexes as "gpt" + CJK bigrams, and now keeps kana and CJK Ext-A, which the old character class dropped entirely. Latin tokenization is unchanged. 2. Chinese titles produced throwaway slugs. normalizeSlug strips every non-[a-z0-9] char, so a CJK title normalized to "" and addSource fell back to entry-<Date.now()> — opaque, unstable, un-dedupable. New kb/slug.ts mints <date>-<sha256/8> for titles with no usable latin, keeping slugs ASCII on purpose (macOS NFD vs git NFC filenames, URLs, git paths) with the real title in frontmatter. canonicalUrl() drops utm_*/known tracking params so the same article shared twice canonicalizes — and hashes — identically. 3. Entries can now carry arbitrary frontmatter via KbEntry.extra, round-tripped on read/write. This is where ingest provenance goes (url/site/author/published/hash/via) without the store knowing about any of it, and it means keys a future version adds survive a round-trip through an older one. Values are whitespace-collapsed on write so a newline can't forge frontmatter lines on the next read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(kb): parse [[links]] into a real graph; index.md becomes a ranked MOC v1.0 told Lisa to cross-link wiki pages with [[slug]] (SCHEMA.md) and recorded each page's sources: in frontmatter — but nothing ever read either one. The KB was a pile of files with decorative links. src/kb/links.ts builds the graph from three edge sources ([[slug]], [[slug|alias]], [[kb:slug]] in bodies; a wiki page's sources: frontmatter) and derives what you can't get from search: backlinks, hubs, orphans, and links pointing at pages that don't exist yet. index.md is now a ranked map of content rather than a flat listing. It is injected into EVERY system prompt and hard-capped there (prompt.ts:165), so with a flat list the truncation lands at an arbitrary point once the KB grows and whatever fell below the line stops existing as far as Lisa is concerned. Ordering wiki pages by (1 + backlinks) x recency means the cut always eats the least-connected tail. Orphans and broken links are listed as the tending to-do list, and every truncation says how much it dropped instead of trailing off. Sources are now listed by TITLE ONLY. Layer 1 is raw captured material — after link ingest lands that means whole web pages — and index.md is always-on prompt text, so an excerpt there is a direct "any page on the internet -> Lisa's system prompt" path. This is a stronger form of the containment the plan assigned to K-I (it covers every source, not just origin: web), and it costs nothing: a title is all a map needs, and the body is one kb_read away. Wiki gists stay — Lisa wrote those herself. Also: kb/index.json (the machine-readable graph, for the web UI and tools), a kb_links tool (links to / linked from / shares tags with), and kb_read now takes a bare slug or a [[wikilink]] with the layer optional, and appends the pages that link to what you just read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(kb): handoff for the remaining knowledge-base v2 work (K-D … K-I) K-A/K-B/K-C are open as #278/#279/#280. This is the pickup document for whoever continues: the branch stack and how not to break it, the APIs the first three PRs landed, a file-by-file spec for each remaining PR, the repo conventions that are easy to trip over (LISA_HOME must be set before import, the tool-subset triple, the giant template literals in lisa-client.ts), and the eight decisions that are already settled so they don't get re-litigated. Two things it corrects against the plan: the index-side injection containment assigned to K-I already shipped in K-C, and in a stronger form (every source is title-only in index.md, not just origin: web), so K-I is down to the watchlist restriction and the kb_read fence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(kb): memory ⇄ KB links — inline titles, link-style memory, conservative autolink (K-D) Closes the "memory stores links, not content" loop from the v2.0 plan (user need ③). Four pieces: - prompt.ts inlines page titles for [[kb:slug]] pointers found in MEMORY.md / USER.md at assembly time ([[kb:oauth]] → [[kb:oauth]](OAuth 与 PKCE)), so a pointer is meaningful without a kb_read. Titles come from the already- generated kb/index.json behind a stat-fingerprint cache — the prompt builds every turn, so no full-KB read on this path. Unresolvable links are kept verbatim: a dangling pointer is a "write this page" signal, not noise. - memory tool description now steers knowledge OUT of memory (4KB cap) and into the KB with a [[kb:slug]] pointer. - The idle tending prompt closes the loop from the other side: after distilling a wiki page, back-link it from any related memory entry. - kb/autolink.ts: conservative title → [[link]] pass applied on kb_write. Whole-word title matches only, first occurrence, longest-first, never in code / existing links, short titles skipped — a wrong edge pollutes the (backlinks × recency) hub ranking the always-on index sorts by, so misses are preferred over false links. Emits [[slug|Title]] when the slug differs so CJK prose keeps reading as prose. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

…K-E) (#283) * docs: knowledge base v2.0 plan (ingest · daily brief · link graph) Design of record for three KB capabilities on top of v1.0 (#235-#242): - K1 paste any link (web / 公众号 / B站 / YouTube) -> Markdown source - K2 a watchlist -> daily fetched, classified, summarized brief - K3 [[slug]] parsed into a real link graph; index.md becomes a ranked map-of-content; memory stores [[kb:slug]] links instead of content Review of the shipped v1.0 code found five gaps this plan closes, one of them a confirmed bug: both TF-IDF tokenizers (kb/search.ts:45 and memory/vector.ts:212) split on whitespace, so a whole CJK run collapses into a single token and Chinese content only matches an exact whole-string query. Also: normalizeSlug strips non-ASCII so every Chinese title falls back to entry-<timestamp>; web_fetch is lossy plain text with no metadata and no store; [[slug]] is documented in SCHEMA.md but nothing parses it; and there is no KB scheduler at all. Includes the pro/con debate for each decision (self-built extractor vs deps vs remote reader, immutable sources vs re-ingest, autonomous ingest as a prompt-injection path, consent, ASCII vs CJK slugs, brief storage, and RSSHub for feeds without one) and a nine-PR ship order. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(kb): CJK-safe search + slugs, and provenance frontmatter Three foundation fixes the rest of the KB v2 work stands on (docs/PLAN_KNOWLEDGE_BASE_v2.0.md K-B). All three are things that would have corrupted or lost data once link ingest starts pouring Chinese content in, so they land before the firehose. 1. CJK search was broken. Both TF-IDF tokenizers (kb/search.ts, memory/vector.ts) carried the same three lines: lowercase, strip non-word chars, split on whitespace. Chinese has no whitespace, so a whole clause survived as ONE token: doc "这篇公众号文章讲的是知识库的设计" -> one token query "知识库" -> one token intersection = empty -> zero hits i.e. CJK content was only findable by an exact whole-clause query. Replaced by a shared src/tokenize.ts that emits character bigrams for CJK runs (plus the run itself, so an exact phrase still ranks highest), segments at latin/CJK boundaries so "gpt模型" indexes as "gpt" + CJK bigrams, and now keeps kana and CJK Ext-A, which the old character class dropped entirely. Latin tokenization is unchanged. 2. Chinese titles produced throwaway slugs. normalizeSlug strips every non-[a-z0-9] char, so a CJK title normalized to "" and addSource fell back to entry-<Date.now()> — opaque, unstable, un-dedupable. New kb/slug.ts mints <date>-<sha256/8> for titles with no usable latin, keeping slugs ASCII on purpose (macOS NFD vs git NFC filenames, URLs, git paths) with the real title in frontmatter. canonicalUrl() drops utm_*/known tracking params so the same article shared twice canonicalizes — and hashes — identically. 3. Entries can now carry arbitrary frontmatter via KbEntry.extra, round-tripped on read/write. This is where ingest provenance goes (url/site/author/published/hash/via) without the store knowing about any of it, and it means keys a future version adds survive a round-trip through an older one. Values are whitespace-collapsed on write so a newline can't forge frontmatter lines on the next read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(kb): parse [[links]] into a real graph; index.md becomes a ranked MOC v1.0 told Lisa to cross-link wiki pages with [[slug]] (SCHEMA.md) and recorded each page's sources: in frontmatter — but nothing ever read either one. The KB was a pile of files with decorative links. src/kb/links.ts builds the graph from three edge sources ([[slug]], [[slug|alias]], [[kb:slug]] in bodies; a wiki page's sources: frontmatter) and derives what you can't get from search: backlinks, hubs, orphans, and links pointing at pages that don't exist yet. index.md is now a ranked map of content rather than a flat listing. It is injected into EVERY system prompt and hard-capped there (prompt.ts:165), so with a flat list the truncation lands at an arbitrary point once the KB grows and whatever fell below the line stops existing as far as Lisa is concerned. Ordering wiki pages by (1 + backlinks) x recency means the cut always eats the least-connected tail. Orphans and broken links are listed as the tending to-do list, and every truncation says how much it dropped instead of trailing off. Sources are now listed by TITLE ONLY. Layer 1 is raw captured material — after link ingest lands that means whole web pages — and index.md is always-on prompt text, so an excerpt there is a direct "any page on the internet -> Lisa's system prompt" path. This is a stronger form of the containment the plan assigned to K-I (it covers every source, not just origin: web), and it costs nothing: a title is all a map needs, and the body is one kb_read away. Wiki gists stay — Lisa wrote those herself. Also: kb/index.json (the machine-readable graph, for the web UI and tools), a kb_links tool (links to / linked from / shares tags with), and kb_read now takes a bare slug or a [[wikilink]] with the layer optional, and appends the pages that link to what you just read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(kb): handoff for the remaining knowledge-base v2 work (K-D … K-I) K-A/K-B/K-C are open as #278/#279/#280. This is the pickup document for whoever continues: the branch stack and how not to break it, the APIs the first three PRs landed, a file-by-file spec for each remaining PR, the repo conventions that are easy to trip over (LISA_HOME must be set before import, the tool-subset triple, the giant template literals in lisa-client.ts), and the eight decisions that are already settled so they don't get re-litigated. Two things it corrects against the plan: the index-side injection containment assigned to K-I already shipped in K-C, and in a stronger form (every source is title-only in index.md, not just origin: web), so K-I is down to the watchlist restriction and the kb_read fence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(kb): memory ⇄ KB links — inline titles, link-style memory, conservative autolink (K-D) Closes the "memory stores links, not content" loop from the v2.0 plan (user need ③). Four pieces: - prompt.ts inlines page titles for [[kb:slug]] pointers found in MEMORY.md / USER.md at assembly time ([[kb:oauth]] → [[kb:oauth]](OAuth 与 PKCE)), so a pointer is meaningful without a kb_read. Titles come from the already- generated kb/index.json behind a stat-fingerprint cache — the prompt builds every turn, so no full-KB read on this path. Unresolvable links are kept verbatim: a dangling pointer is a "write this page" signal, not noise. - memory tool description now steers knowledge OUT of memory (4KB cap) and into the KB with a [[kb:slug]] pointer. - The idle tending prompt closes the loop from the other side: after distilling a wiki page, back-link it from any related memory entry. - kb/autolink.ts: conservative title → [[link]] pass applied on kb_write. Whole-word title matches only, first occurrence, longest-first, never in code / existing links, short titles skipped — a wrong edge pollutes the (backlinks × recency) hub ranking the always-on index sorts by, so misses are preferred over false links. Emits [[slug|Title]] when the slug differs so CJK prose keeps reading as prose. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(kb): ingest engine — url → provenance-stamped Layer-1 markdown (K-E) kb_ingest(url) turns a link into a searchable, citeable source entry (v2.0 user need ②). New src/kb/ingest/, all zero-dependency (D1: what the user reads never leaves the machine): - html-to-md.ts: lenient HTML tree parser + markdown serializer (headings, nested lists, fenced/inline code, blockquotes, tables, links, lazy-loaded images, emphasis). Prose is markdown-escaped BEFORE syntax is added, so page text can't forge emphasis/links/wikilinks into the KB; code bodies can't terminate their own fence; javascript: hrefs are dropped. - readability.ts: small Readability — score candidates by text mass, <p> count, link density, class/id hints; fall back to <body> when unsure (noise is recoverable by distillation, clipped content is not). - provenance.ts: og:* / JSON-LD Article / <title> / rel=canonical / lang → frontmatter (url · site · author · published · lang · hash · via). - dedupe.ts: kb/.ingested.json (hash → slug), rebuilt from the sources' own hash: frontmatter when missing/corrupt. Same URL returns the existing entry (D2, immutable sources); force re-ingests + records supersedes:. - index.ts: ingestUrl orchestration with an adapter seam for K-F. ALL fetching goes through web_fetch's fetchFollowingSafeRedirects (per-hop private-host validation) plus an up-front assertAllowedUrl — no second fetch path, no second SSRF surface. kb_ingest is REMOTE_BLOCKED (a channel message must not make Lisa fetch an attacker URL into the KB) but NOT autonomous-blocked — K-I scopes autonomous use to the feeds.json domain watchlist (D3) instead of a blanket ban. Tests are offline-fixture only: converter syntax coverage, chrome-stripping, provenance forms, dedupe/force/ledger-rebuild, SSRF rejection before fetch, content-type gates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(kb): ingest content-loss on self-closing drop tags + cap/reorder fetch - html-to-md: a self-closing drop tag (<svg/>, <iframe/>) has no matching close, so the parser jumped TAG_RE.lastIndex to html.length and silently discarded the rest of the document. Skip just the tag for self-closed forms; a genuinely unclosed drop tag still drops to EOF (safer than leaking raw script/style source). +regression test. - genericFetch: reject unsupported content-types from the header before buffering, reject an oversized Content-Length up front, and read the body through a bounded reader (readBodyCapped) so a chunked/undeclared response can't exhaust memory — matters once ingest runs unattended from the brief. (The untrusted-title→frontmatter injection flagged on this PR is fixed at its root in store.ts, already merged.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* docs: knowledge base v2.0 plan (ingest · daily brief · link graph) Design of record for three KB capabilities on top of v1.0 (#235-#242): - K1 paste any link (web / 公众号 / B站 / YouTube) -> Markdown source - K2 a watchlist -> daily fetched, classified, summarized brief - K3 [[slug]] parsed into a real link graph; index.md becomes a ranked map-of-content; memory stores [[kb:slug]] links instead of content Review of the shipped v1.0 code found five gaps this plan closes, one of them a confirmed bug: both TF-IDF tokenizers (kb/search.ts:45 and memory/vector.ts:212) split on whitespace, so a whole CJK run collapses into a single token and Chinese content only matches an exact whole-string query. Also: normalizeSlug strips non-ASCII so every Chinese title falls back to entry-<timestamp>; web_fetch is lossy plain text with no metadata and no store; [[slug]] is documented in SCHEMA.md but nothing parses it; and there is no KB scheduler at all. Includes the pro/con debate for each decision (self-built extractor vs deps vs remote reader, immutable sources vs re-ingest, autonomous ingest as a prompt-injection path, consent, ASCII vs CJK slugs, brief storage, and RSSHub for feeds without one) and a nine-PR ship order. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(kb): CJK-safe search + slugs, and provenance frontmatter Three foundation fixes the rest of the KB v2 work stands on (docs/PLAN_KNOWLEDGE_BASE_v2.0.md K-B). All three are things that would have corrupted or lost data once link ingest starts pouring Chinese content in, so they land before the firehose. 1. CJK search was broken. Both TF-IDF tokenizers (kb/search.ts, memory/vector.ts) carried the same three lines: lowercase, strip non-word chars, split on whitespace. Chinese has no whitespace, so a whole clause survived as ONE token: doc "这篇公众号文章讲的是知识库的设计" -> one token query "知识库" -> one token intersection = empty -> zero hits i.e. CJK content was only findable by an exact whole-clause query. Replaced by a shared src/tokenize.ts that emits character bigrams for CJK runs (plus the run itself, so an exact phrase still ranks highest), segments at latin/CJK boundaries so "gpt模型" indexes as "gpt" + CJK bigrams, and now keeps kana and CJK Ext-A, which the old character class dropped entirely. Latin tokenization is unchanged. 2. Chinese titles produced throwaway slugs. normalizeSlug strips every non-[a-z0-9] char, so a CJK title normalized to "" and addSource fell back to entry-<Date.now()> — opaque, unstable, un-dedupable. New kb/slug.ts mints <date>-<sha256/8> for titles with no usable latin, keeping slugs ASCII on purpose (macOS NFD vs git NFC filenames, URLs, git paths) with the real title in frontmatter. canonicalUrl() drops utm_*/known tracking params so the same article shared twice canonicalizes — and hashes — identically. 3. Entries can now carry arbitrary frontmatter via KbEntry.extra, round-tripped on read/write. This is where ingest provenance goes (url/site/author/published/hash/via) without the store knowing about any of it, and it means keys a future version adds survive a round-trip through an older one. Values are whitespace-collapsed on write so a newline can't forge frontmatter lines on the next read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(kb): parse [[links]] into a real graph; index.md becomes a ranked MOC v1.0 told Lisa to cross-link wiki pages with [[slug]] (SCHEMA.md) and recorded each page's sources: in frontmatter — but nothing ever read either one. The KB was a pile of files with decorative links. src/kb/links.ts builds the graph from three edge sources ([[slug]], [[slug|alias]], [[kb:slug]] in bodies; a wiki page's sources: frontmatter) and derives what you can't get from search: backlinks, hubs, orphans, and links pointing at pages that don't exist yet. index.md is now a ranked map of content rather than a flat listing. It is injected into EVERY system prompt and hard-capped there (prompt.ts:165), so with a flat list the truncation lands at an arbitrary point once the KB grows and whatever fell below the line stops existing as far as Lisa is concerned. Ordering wiki pages by (1 + backlinks) x recency means the cut always eats the least-connected tail. Orphans and broken links are listed as the tending to-do list, and every truncation says how much it dropped instead of trailing off. Sources are now listed by TITLE ONLY. Layer 1 is raw captured material — after link ingest lands that means whole web pages — and index.md is always-on prompt text, so an excerpt there is a direct "any page on the internet -> Lisa's system prompt" path. This is a stronger form of the containment the plan assigned to K-I (it covers every source, not just origin: web), and it costs nothing: a title is all a map needs, and the body is one kb_read away. Wiki gists stay — Lisa wrote those herself. Also: kb/index.json (the machine-readable graph, for the web UI and tools), a kb_links tool (links to / linked from / shares tags with), and kb_read now takes a bare slug or a [[wikilink]] with the layer optional, and appends the pages that link to what you just read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(kb): handoff for the remaining knowledge-base v2 work (K-D … K-I) K-A/K-B/K-C are open as #278/#279/#280. This is the pickup document for whoever continues: the branch stack and how not to break it, the APIs the first three PRs landed, a file-by-file spec for each remaining PR, the repo conventions that are easy to trip over (LISA_HOME must be set before import, the tool-subset triple, the giant template literals in lisa-client.ts), and the eight decisions that are already settled so they don't get re-litigated. Two things it corrects against the plan: the index-side injection containment assigned to K-I already shipped in K-C, and in a stronger form (every source is title-only in index.md, not just origin: web), so K-I is down to the watchlist restriction and the kb_read fence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(kb): memory ⇄ KB links — inline titles, link-style memory, conservative autolink (K-D) Closes the "memory stores links, not content" loop from the v2.0 plan (user need ③). Four pieces: - prompt.ts inlines page titles for [[kb:slug]] pointers found in MEMORY.md / USER.md at assembly time ([[kb:oauth]] → [[kb:oauth]](OAuth 与 PKCE)), so a pointer is meaningful without a kb_read. Titles come from the already- generated kb/index.json behind a stat-fingerprint cache — the prompt builds every turn, so no full-KB read on this path. Unresolvable links are kept verbatim: a dangling pointer is a "write this page" signal, not noise. - memory tool description now steers knowledge OUT of memory (4KB cap) and into the KB with a [[kb:slug]] pointer. - The idle tending prompt closes the loop from the other side: after distilling a wiki page, back-link it from any related memory entry. - kb/autolink.ts: conservative title → [[link]] pass applied on kb_write. Whole-word title matches only, first occurrence, longest-first, never in code / existing links, short titles skipped — a wrong edge pollutes the (backlinks × recency) hub ranking the always-on index sorts by, so misses are preferred over false links. Emits [[slug|Title]] when the slug differs so CJK prose keeps reading as prose. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(kb): ingest engine — url → provenance-stamped Layer-1 markdown (K-E) kb_ingest(url) turns a link into a searchable, citeable source entry (v2.0 user need ②). New src/kb/ingest/, all zero-dependency (D1: what the user reads never leaves the machine): - html-to-md.ts: lenient HTML tree parser + markdown serializer (headings, nested lists, fenced/inline code, blockquotes, tables, links, lazy-loaded images, emphasis). Prose is markdown-escaped BEFORE syntax is added, so page text can't forge emphasis/links/wikilinks into the KB; code bodies can't terminate their own fence; javascript: hrefs are dropped. - readability.ts: small Readability — score candidates by text mass, <p> count, link density, class/id hints; fall back to <body> when unsure (noise is recoverable by distillation, clipped content is not). - provenance.ts: og:* / JSON-LD Article / <title> / rel=canonical / lang → frontmatter (url · site · author · published · lang · hash · via). - dedupe.ts: kb/.ingested.json (hash → slug), rebuilt from the sources' own hash: frontmatter when missing/corrupt. Same URL returns the existing entry (D2, immutable sources); force re-ingests + records supersedes:. - index.ts: ingestUrl orchestration with an adapter seam for K-F. ALL fetching goes through web_fetch's fetchFollowingSafeRedirects (per-hop private-host validation) plus an up-front assertAllowedUrl — no second fetch path, no second SSRF surface. kb_ingest is REMOTE_BLOCKED (a channel message must not make Lisa fetch an attacker URL into the KB) but NOT autonomous-blocked — K-I scopes autonomous use to the feeds.json domain watchlist (D3) instead of a blanket ban. Tests are offline-fixture only: converter syntax coverage, chrome-stripping, provenance forms, dedupe/force/ledger-rebuild, SSRF rejection before fetch, content-type gates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(kb): site adapters — WeChat, Bilibili, YouTube (K-F) Three IngestAdapter implementations behind the K-E seam, for the sites where generic readability extraction can't reach the content: - wechat: article body from #js_content, account from #js_name, epoch publish time from inline `var ct`; lazy-loaded images already handled via data-src. The human-verification interstitial throws a loud, actionable error (open the share link on the phone, paste the text, kb_add) — never a silently captured blank page. - bilibili: metadata via the public view API; b23.tv short links resolve through the guarded fetch. Subtitles are cookie-gated (SESSDATA, which the user may volunteer in kb/feeds.json) — absent that, degrade. - youtube: oEmbed as the reliability floor + one InnerTube player POST for description/captions; captionTracks → &fmt=json3, manual tracks over ASR. Handles the documented failure modes (PO token, 200-with-empty-body, datacenter throttling) by degrading. Fixed subtitle layering (handoff: do not reorder): built-in API → yt-dlp (--dump-single-json only when installed; discovered subtitle URLs are still fetched through the guarded path) → metadata + description. A MISSING TRANSCRIPT IS NOT A FAILURE: the entry records `transcript: unavailable (<reason>)` and the tool reply tells the user they can paste one via kb_add. Supporting changes: fetchFollowingSafeRedirects gains an optional SafeFetchInit (method/headers/body) so adapter API calls stay on the single SSRF-guarded fetch path; ingest types moved to types.ts to break the adapter↔index cycle; the min-body-length gate now applies only to the generic path so degraded video captures can be legitimately short. All tests offline-fixture; degradation paths asserted explicitly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

… kb CLI (K-G) (#285) * docs: knowledge base v2.0 plan (ingest · daily brief · link graph) Design of record for three KB capabilities on top of v1.0 (#235-#242): - K1 paste any link (web / 公众号 / B站 / YouTube) -> Markdown source - K2 a watchlist -> daily fetched, classified, summarized brief - K3 [[slug]] parsed into a real link graph; index.md becomes a ranked map-of-content; memory stores [[kb:slug]] links instead of content Review of the shipped v1.0 code found five gaps this plan closes, one of them a confirmed bug: both TF-IDF tokenizers (kb/search.ts:45 and memory/vector.ts:212) split on whitespace, so a whole CJK run collapses into a single token and Chinese content only matches an exact whole-string query. Also: normalizeSlug strips non-ASCII so every Chinese title falls back to entry-<timestamp>; web_fetch is lossy plain text with no metadata and no store; [[slug]] is documented in SCHEMA.md but nothing parses it; and there is no KB scheduler at all. Includes the pro/con debate for each decision (self-built extractor vs deps vs remote reader, immutable sources vs re-ingest, autonomous ingest as a prompt-injection path, consent, ASCII vs CJK slugs, brief storage, and RSSHub for feeds without one) and a nine-PR ship order. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(kb): CJK-safe search + slugs, and provenance frontmatter Three foundation fixes the rest of the KB v2 work stands on (docs/PLAN_KNOWLEDGE_BASE_v2.0.md K-B). All three are things that would have corrupted or lost data once link ingest starts pouring Chinese content in, so they land before the firehose. 1. CJK search was broken. Both TF-IDF tokenizers (kb/search.ts, memory/vector.ts) carried the same three lines: lowercase, strip non-word chars, split on whitespace. Chinese has no whitespace, so a whole clause survived as ONE token: doc "这篇公众号文章讲的是知识库的设计" -> one token query "知识库" -> one token intersection = empty -> zero hits i.e. CJK content was only findable by an exact whole-clause query. Replaced by a shared src/tokenize.ts that emits character bigrams for CJK runs (plus the run itself, so an exact phrase still ranks highest), segments at latin/CJK boundaries so "gpt模型" indexes as "gpt" + CJK bigrams, and now keeps kana and CJK Ext-A, which the old character class dropped entirely. Latin tokenization is unchanged. 2. Chinese titles produced throwaway slugs. normalizeSlug strips every non-[a-z0-9] char, so a CJK title normalized to "" and addSource fell back to entry-<Date.now()> — opaque, unstable, un-dedupable. New kb/slug.ts mints <date>-<sha256/8> for titles with no usable latin, keeping slugs ASCII on purpose (macOS NFD vs git NFC filenames, URLs, git paths) with the real title in frontmatter. canonicalUrl() drops utm_*/known tracking params so the same article shared twice canonicalizes — and hashes — identically. 3. Entries can now carry arbitrary frontmatter via KbEntry.extra, round-tripped on read/write. This is where ingest provenance goes (url/site/author/published/hash/via) without the store knowing about any of it, and it means keys a future version adds survive a round-trip through an older one. Values are whitespace-collapsed on write so a newline can't forge frontmatter lines on the next read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(kb): parse [[links]] into a real graph; index.md becomes a ranked MOC v1.0 told Lisa to cross-link wiki pages with [[slug]] (SCHEMA.md) and recorded each page's sources: in frontmatter — but nothing ever read either one. The KB was a pile of files with decorative links. src/kb/links.ts builds the graph from three edge sources ([[slug]], [[slug|alias]], [[kb:slug]] in bodies; a wiki page's sources: frontmatter) and derives what you can't get from search: backlinks, hubs, orphans, and links pointing at pages that don't exist yet. index.md is now a ranked map of content rather than a flat listing. It is injected into EVERY system prompt and hard-capped there (prompt.ts:165), so with a flat list the truncation lands at an arbitrary point once the KB grows and whatever fell below the line stops existing as far as Lisa is concerned. Ordering wiki pages by (1 + backlinks) x recency means the cut always eats the least-connected tail. Orphans and broken links are listed as the tending to-do list, and every truncation says how much it dropped instead of trailing off. Sources are now listed by TITLE ONLY. Layer 1 is raw captured material — after link ingest lands that means whole web pages — and index.md is always-on prompt text, so an excerpt there is a direct "any page on the internet -> Lisa's system prompt" path. This is a stronger form of the containment the plan assigned to K-I (it covers every source, not just origin: web), and it costs nothing: a title is all a map needs, and the body is one kb_read away. Wiki gists stay — Lisa wrote those herself. Also: kb/index.json (the machine-readable graph, for the web UI and tools), a kb_links tool (links to / linked from / shares tags with), and kb_read now takes a bare slug or a [[wikilink]] with the layer optional, and appends the pages that link to what you just read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(kb): handoff for the remaining knowledge-base v2 work (K-D … K-I) K-A/K-B/K-C are open as #278/#279/#280. This is the pickup document for whoever continues: the branch stack and how not to break it, the APIs the first three PRs landed, a file-by-file spec for each remaining PR, the repo conventions that are easy to trip over (LISA_HOME must be set before import, the tool-subset triple, the giant template literals in lisa-client.ts), and the eight decisions that are already settled so they don't get re-litigated. Two things it corrects against the plan: the index-side injection containment assigned to K-I already shipped in K-C, and in a stronger form (every source is title-only in index.md, not just origin: web), so K-I is down to the watchlist restriction and the kb_read fence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(kb): memory ⇄ KB links — inline titles, link-style memory, conservative autolink (K-D) Closes the "memory stores links, not content" loop from the v2.0 plan (user need ③). Four pieces: - prompt.ts inlines page titles for [[kb:slug]] pointers found in MEMORY.md / USER.md at assembly time ([[kb:oauth]] → [[kb:oauth]](OAuth 与 PKCE)), so a pointer is meaningful without a kb_read. Titles come from the already- generated kb/index.json behind a stat-fingerprint cache — the prompt builds every turn, so no full-KB read on this path. Unresolvable links are kept verbatim: a dangling pointer is a "write this page" signal, not noise. - memory tool description now steers knowledge OUT of memory (4KB cap) and into the KB with a [[kb:slug]] pointer. - The idle tending prompt closes the loop from the other side: after distilling a wiki page, back-link it from any related memory entry. - kb/autolink.ts: conservative title → [[link]] pass applied on kb_write. Whole-word title matches only, first occurrence, longest-first, never in code / existing links, short titles skipped — a wrong edge pollutes the (backlinks × recency) hub ranking the always-on index sorts by, so misses are preferred over false links. Emits [[slug|Title]] when the slug differs so CJK prose keeps reading as prose. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(kb): ingest engine — url → provenance-stamped Layer-1 markdown (K-E) kb_ingest(url) turns a link into a searchable, citeable source entry (v2.0 user need ②). New src/kb/ingest/, all zero-dependency (D1: what the user reads never leaves the machine): - html-to-md.ts: lenient HTML tree parser + markdown serializer (headings, nested lists, fenced/inline code, blockquotes, tables, links, lazy-loaded images, emphasis). Prose is markdown-escaped BEFORE syntax is added, so page text can't forge emphasis/links/wikilinks into the KB; code bodies can't terminate their own fence; javascript: hrefs are dropped. - readability.ts: small Readability — score candidates by text mass, <p> count, link density, class/id hints; fall back to <body> when unsure (noise is recoverable by distillation, clipped content is not). - provenance.ts: og:* / JSON-LD Article / <title> / rel=canonical / lang → frontmatter (url · site · author · published · lang · hash · via). - dedupe.ts: kb/.ingested.json (hash → slug), rebuilt from the sources' own hash: frontmatter when missing/corrupt. Same URL returns the existing entry (D2, immutable sources); force re-ingests + records supersedes:. - index.ts: ingestUrl orchestration with an adapter seam for K-F. ALL fetching goes through web_fetch's fetchFollowingSafeRedirects (per-hop private-host validation) plus an up-front assertAllowedUrl — no second fetch path, no second SSRF surface. kb_ingest is REMOTE_BLOCKED (a channel message must not make Lisa fetch an attacker URL into the KB) but NOT autonomous-blocked — K-I scopes autonomous use to the feeds.json domain watchlist (D3) instead of a blanket ban. Tests are offline-fixture only: converter syntax coverage, chrome-stripping, provenance forms, dedupe/force/ledger-rebuild, SSRF rejection before fetch, content-type gates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(kb): site adapters — WeChat, Bilibili, YouTube (K-F) Three IngestAdapter implementations behind the K-E seam, for the sites where generic readability extraction can't reach the content: - wechat: article body from #js_content, account from #js_name, epoch publish time from inline `var ct`; lazy-loaded images already handled via data-src. The human-verification interstitial throws a loud, actionable error (open the share link on the phone, paste the text, kb_add) — never a silently captured blank page. - bilibili: metadata via the public view API; b23.tv short links resolve through the guarded fetch. Subtitles are cookie-gated (SESSDATA, which the user may volunteer in kb/feeds.json) — absent that, degrade. - youtube: oEmbed as the reliability floor + one InnerTube player POST for description/captions; captionTracks → &fmt=json3, manual tracks over ASR. Handles the documented failure modes (PO token, 200-with-empty-body, datacenter throttling) by degrading. Fixed subtitle layering (handoff: do not reorder): built-in API → yt-dlp (--dump-single-json only when installed; discovered subtitle URLs are still fetched through the guarded path) → metadata + description. A MISSING TRANSCRIPT IS NOT A FAILURE: the entry records `transcript: unavailable (<reason>)` and the tool reply tells the user they can paste one via kb_add. Supporting changes: fetchFollowingSafeRedirects gains an optional SafeFetchInit (method/headers/body) so adapter API calls stay on the single SSRF-guarded fetch path; ingest types moved to types.ts to break the adapter↔index cycle; the min-body-length gate now applies only to the generic path so degraded video captures can be legitimately short. All tests offline-fixture; degradation paths asserted explicitly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(kb): ingest surfaces — web paste bar, chat chip, API route, lisa kb CLI (K-G) Four entry points onto the K-E/K-F engine, one per place a link shows up: - POST /api/kb/ingest ({url, title?, tags?, force?} — shaped so a future share-sheet client can call it directly). Ingest failures return 422 with the engine's actionable message (verification page, paywall, bad type) instead of a generic 500. - Knowledge view: a paste-a-link bar above search — progress state, saves, then opens the new entry; dedupe and no-transcript degradation are surfaced in the status line. - Chat: a user message containing a bare URL gets a one-tap 存入知识库 chip under the bubble (same interaction shape as the KB capture bar; kbToast is now shared via window.lisaKbToast). - `lisa kb add|list|search|brief` (src/cli/kb.ts), registered like mail: passthrough subargs so --title/--tags/--force never collide with global flags. `brief` prints the newest sources/brief-<date>.md — the file K-H starts writing — and points at feeds.json until then. The client edits live inside the MAIN_CLIENT_JS template literal (regex backslashes double-escaped — the html-syntax test caught the first attempt); lisa-html-snapshot constants recomputed accordingly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

…l-write (K-H) (#286) * docs: knowledge base v2.0 plan (ingest · daily brief · link graph) Design of record for three KB capabilities on top of v1.0 (#235-#242): - K1 paste any link (web / 公众号 / B站 / YouTube) -> Markdown source - K2 a watchlist -> daily fetched, classified, summarized brief - K3 [[slug]] parsed into a real link graph; index.md becomes a ranked map-of-content; memory stores [[kb:slug]] links instead of content Review of the shipped v1.0 code found five gaps this plan closes, one of them a confirmed bug: both TF-IDF tokenizers (kb/search.ts:45 and memory/vector.ts:212) split on whitespace, so a whole CJK run collapses into a single token and Chinese content only matches an exact whole-string query. Also: normalizeSlug strips non-ASCII so every Chinese title falls back to entry-<timestamp>; web_fetch is lossy plain text with no metadata and no store; [[slug]] is documented in SCHEMA.md but nothing parses it; and there is no KB scheduler at all. Includes the pro/con debate for each decision (self-built extractor vs deps vs remote reader, immutable sources vs re-ingest, autonomous ingest as a prompt-injection path, consent, ASCII vs CJK slugs, brief storage, and RSSHub for feeds without one) and a nine-PR ship order. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(kb): CJK-safe search + slugs, and provenance frontmatter Three foundation fixes the rest of the KB v2 work stands on (docs/PLAN_KNOWLEDGE_BASE_v2.0.md K-B). All three are things that would have corrupted or lost data once link ingest starts pouring Chinese content in, so they land before the firehose. 1. CJK search was broken. Both TF-IDF tokenizers (kb/search.ts, memory/vector.ts) carried the same three lines: lowercase, strip non-word chars, split on whitespace. Chinese has no whitespace, so a whole clause survived as ONE token: doc "这篇公众号文章讲的是知识库的设计" -> one token query "知识库" -> one token intersection = empty -> zero hits i.e. CJK content was only findable by an exact whole-clause query. Replaced by a shared src/tokenize.ts that emits character bigrams for CJK runs (plus the run itself, so an exact phrase still ranks highest), segments at latin/CJK boundaries so "gpt模型" indexes as "gpt" + CJK bigrams, and now keeps kana and CJK Ext-A, which the old character class dropped entirely. Latin tokenization is unchanged. 2. Chinese titles produced throwaway slugs. normalizeSlug strips every non-[a-z0-9] char, so a CJK title normalized to "" and addSource fell back to entry-<Date.now()> — opaque, unstable, un-dedupable. New kb/slug.ts mints <date>-<sha256/8> for titles with no usable latin, keeping slugs ASCII on purpose (macOS NFD vs git NFC filenames, URLs, git paths) with the real title in frontmatter. canonicalUrl() drops utm_*/known tracking params so the same article shared twice canonicalizes — and hashes — identically. 3. Entries can now carry arbitrary frontmatter via KbEntry.extra, round-tripped on read/write. This is where ingest provenance goes (url/site/author/published/hash/via) without the store knowing about any of it, and it means keys a future version adds survive a round-trip through an older one. Values are whitespace-collapsed on write so a newline can't forge frontmatter lines on the next read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(kb): parse [[links]] into a real graph; index.md becomes a ranked MOC v1.0 told Lisa to cross-link wiki pages with [[slug]] (SCHEMA.md) and recorded each page's sources: in frontmatter — but nothing ever read either one. The KB was a pile of files with decorative links. src/kb/links.ts builds the graph from three edge sources ([[slug]], [[slug|alias]], [[kb:slug]] in bodies; a wiki page's sources: frontmatter) and derives what you can't get from search: backlinks, hubs, orphans, and links pointing at pages that don't exist yet. index.md is now a ranked map of content rather than a flat listing. It is injected into EVERY system prompt and hard-capped there (prompt.ts:165), so with a flat list the truncation lands at an arbitrary point once the KB grows and whatever fell below the line stops existing as far as Lisa is concerned. Ordering wiki pages by (1 + backlinks) x recency means the cut always eats the least-connected tail. Orphans and broken links are listed as the tending to-do list, and every truncation says how much it dropped instead of trailing off. Sources are now listed by TITLE ONLY. Layer 1 is raw captured material — after link ingest lands that means whole web pages — and index.md is always-on prompt text, so an excerpt there is a direct "any page on the internet -> Lisa's system prompt" path. This is a stronger form of the containment the plan assigned to K-I (it covers every source, not just origin: web), and it costs nothing: a title is all a map needs, and the body is one kb_read away. Wiki gists stay — Lisa wrote those herself. Also: kb/index.json (the machine-readable graph, for the web UI and tools), a kb_links tool (links to / linked from / shares tags with), and kb_read now takes a bare slug or a [[wikilink]] with the layer optional, and appends the pages that link to what you just read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(kb): handoff for the remaining knowledge-base v2 work (K-D … K-I) K-A/K-B/K-C are open as #278/#279/#280. This is the pickup document for whoever continues: the branch stack and how not to break it, the APIs the first three PRs landed, a file-by-file spec for each remaining PR, the repo conventions that are easy to trip over (LISA_HOME must be set before import, the tool-subset triple, the giant template literals in lisa-client.ts), and the eight decisions that are already settled so they don't get re-litigated. Two things it corrects against the plan: the index-side injection containment assigned to K-I already shipped in K-C, and in a stronger form (every source is title-only in index.md, not just origin: web), so K-I is down to the watchlist restriction and the kb_read fence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(kb): memory ⇄ KB links — inline titles, link-style memory, conservative autolink (K-D) Closes the "memory stores links, not content" loop from the v2.0 plan (user need ③). Four pieces: - prompt.ts inlines page titles for [[kb:slug]] pointers found in MEMORY.md / USER.md at assembly time ([[kb:oauth]] → [[kb:oauth]](OAuth 与 PKCE)), so a pointer is meaningful without a kb_read. Titles come from the already- generated kb/index.json behind a stat-fingerprint cache — the prompt builds every turn, so no full-KB read on this path. Unresolvable links are kept verbatim: a dangling pointer is a "write this page" signal, not noise. - memory tool description now steers knowledge OUT of memory (4KB cap) and into the KB with a [[kb:slug]] pointer. - The idle tending prompt closes the loop from the other side: after distilling a wiki page, back-link it from any related memory entry. - kb/autolink.ts: conservative title → [[link]] pass applied on kb_write. Whole-word title matches only, first occurrence, longest-first, never in code / existing links, short titles skipped — a wrong edge pollutes the (backlinks × recency) hub ranking the always-on index sorts by, so misses are preferred over false links. Emits [[slug|Title]] when the slug differs so CJK prose keeps reading as prose. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(kb): ingest engine — url → provenance-stamped Layer-1 markdown (K-E) kb_ingest(url) turns a link into a searchable, citeable source entry (v2.0 user need ②). New src/kb/ingest/, all zero-dependency (D1: what the user reads never leaves the machine): - html-to-md.ts: lenient HTML tree parser + markdown serializer (headings, nested lists, fenced/inline code, blockquotes, tables, links, lazy-loaded images, emphasis). Prose is markdown-escaped BEFORE syntax is added, so page text can't forge emphasis/links/wikilinks into the KB; code bodies can't terminate their own fence; javascript: hrefs are dropped. - readability.ts: small Readability — score candidates by text mass, <p> count, link density, class/id hints; fall back to <body> when unsure (noise is recoverable by distillation, clipped content is not). - provenance.ts: og:* / JSON-LD Article / <title> / rel=canonical / lang → frontmatter (url · site · author · published · lang · hash · via). - dedupe.ts: kb/.ingested.json (hash → slug), rebuilt from the sources' own hash: frontmatter when missing/corrupt. Same URL returns the existing entry (D2, immutable sources); force re-ingests + records supersedes:. - index.ts: ingestUrl orchestration with an adapter seam for K-F. ALL fetching goes through web_fetch's fetchFollowingSafeRedirects (per-hop private-host validation) plus an up-front assertAllowedUrl — no second fetch path, no second SSRF surface. kb_ingest is REMOTE_BLOCKED (a channel message must not make Lisa fetch an attacker URL into the KB) but NOT autonomous-blocked — K-I scopes autonomous use to the feeds.json domain watchlist (D3) instead of a blanket ban. Tests are offline-fixture only: converter syntax coverage, chrome-stripping, provenance forms, dedupe/force/ledger-rebuild, SSRF rejection before fetch, content-type gates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(kb): site adapters — WeChat, Bilibili, YouTube (K-F) Three IngestAdapter implementations behind the K-E seam, for the sites where generic readability extraction can't reach the content: - wechat: article body from #js_content, account from #js_name, epoch publish time from inline `var ct`; lazy-loaded images already handled via data-src. The human-verification interstitial throws a loud, actionable error (open the share link on the phone, paste the text, kb_add) — never a silently captured blank page. - bilibili: metadata via the public view API; b23.tv short links resolve through the guarded fetch. Subtitles are cookie-gated (SESSDATA, which the user may volunteer in kb/feeds.json) — absent that, degrade. - youtube: oEmbed as the reliability floor + one InnerTube player POST for description/captions; captionTracks → &fmt=json3, manual tracks over ASR. Handles the documented failure modes (PO token, 200-with-empty-body, datacenter throttling) by degrading. Fixed subtitle layering (handoff: do not reorder): built-in API → yt-dlp (--dump-single-json only when installed; discovered subtitle URLs are still fetched through the guarded path) → metadata + description. A MISSING TRANSCRIPT IS NOT A FAILURE: the entry records `transcript: unavailable (<reason>)` and the tool reply tells the user they can paste one via kb_add. Supporting changes: fetchFollowingSafeRedirects gains an optional SafeFetchInit (method/headers/body) so adapter API calls stay on the single SSRF-guarded fetch path; ingest types moved to types.ts to break the adapter↔index cycle; the min-body-length gate now applies only to the generic path so degraded video captures can be legitimately short. All tests offline-fixture; degradation paths asserted explicitly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(kb): ingest surfaces — web paste bar, chat chip, API route, lisa kb CLI (K-G) Four entry points onto the K-E/K-F engine, one per place a link shows up: - POST /api/kb/ingest ({url, title?, tags?, force?} — shaped so a future share-sheet client can call it directly). Ingest failures return 422 with the engine's actionable message (verification page, paywall, bad type) instead of a generic 500. - Knowledge view: a paste-a-link bar above search — progress state, saves, then opens the new entry; dedupe and no-transcript degradation are surfaced in the status line. - Chat: a user message containing a bare URL gets a one-tap 存入知识库 chip under the bubble (same interaction shape as the KB capture bar; kbToast is now shared via window.lisaKbToast). - `lisa kb add|list|search|brief` (src/cli/kb.ts), registered like mail: passthrough subargs so --title/--tags/--force never collide with global flags. `brief` prints the newest sources/brief-<date>.md — the file K-H starts writing — and points at feeds.json until then. The client edits live inside the MAIN_CLIENT_JS template literal (regex backslashes double-escaped — the html-syntax test caught the first attempt); lisa-html-snapshot constants recomputed accordingly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(kb): daily feeds brief — sweep, classify, personalized rank, dual-write (K-H) User need ①: a daily, personalized brief over the user's own feed watchlist. New src/kb/feeds/, deliberately shaped after the mail module (the one place this problem is already solved end-to-end here): - store.ts: ~/.lisa/kb/feeds.json ({feeds:[{id,url,tags,max,weight}], briefHour, budgetTokens}) — user-authored; its existence IS the consent (D4, no new consent signal; empty/missing = the capability is fully inert). chmod 0600 on every read + kb/.gitignore coverage, since the file may carry a bilibili SESSDATA. Machine state (seen ids, last brief date) lives apart in kb/feeds/.state.json and is rebuild-safe. - rss.ts: zero-dep RSS2/Atom parsing (CDATA, entities, content:encoded, atom link@href) — lenient, an unparseable field degrades to blank. - classify.ts: mail/classify.ts's batch shape over {category, importance 0-3, oneLine} with the same injection stance (fenced untrusted data, closed taxonomy validation, heuristic fallback). The daily budgetTokens gate (default 120k) stops model batches at the ceiling WITH a log line — remaining items get neutral grades, never dropped silently. - brief.ts: isBriefDue (isDigestDue's twin), personalized scoring — (1+importance) × watchlist weight × (1 + interest/wiki term-overlap, both saturating) — buildBrief + formatBriefText, all pure. - service.ts: incremental sweep (per-feed seen-id state; a feed failing degrades, ALL feeds failing doesn't burn the day) → classify → rank → top-3 full-text ingest via the K-E engine → the D7 dual write: kb/feeds/<date>.json for the UI and sources/brief-<date>.md as a real Layer-1 entry (searchable, linkable, distillable; `lisa kb brief` from K-G picks it up as-is). Delivery mirrors mail: a 30-min server timer + 20s restart catch-up, broadcast (kb_brief_update + idle_message source:kb) and a new pushBridge.onKbBrief behind a new `brief` push pref (default on). Plus GET /api/kb/brief serving the latest JSON. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* docs: knowledge base v2.0 plan (ingest · daily brief · link graph) Design of record for three KB capabilities on top of v1.0 (#235-#242): - K1 paste any link (web / 公众号 / B站 / YouTube) -> Markdown source - K2 a watchlist -> daily fetched, classified, summarized brief - K3 [[slug]] parsed into a real link graph; index.md becomes a ranked map-of-content; memory stores [[kb:slug]] links instead of content Review of the shipped v1.0 code found five gaps this plan closes, one of them a confirmed bug: both TF-IDF tokenizers (kb/search.ts:45 and memory/vector.ts:212) split on whitespace, so a whole CJK run collapses into a single token and Chinese content only matches an exact whole-string query. Also: normalizeSlug strips non-ASCII so every Chinese title falls back to entry-<timestamp>; web_fetch is lossy plain text with no metadata and no store; [[slug]] is documented in SCHEMA.md but nothing parses it; and there is no KB scheduler at all. Includes the pro/con debate for each decision (self-built extractor vs deps vs remote reader, immutable sources vs re-ingest, autonomous ingest as a prompt-injection path, consent, ASCII vs CJK slugs, brief storage, and RSSHub for feeds without one) and a nine-PR ship order. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(kb): CJK-safe search + slugs, and provenance frontmatter Three foundation fixes the rest of the KB v2 work stands on (docs/PLAN_KNOWLEDGE_BASE_v2.0.md K-B). All three are things that would have corrupted or lost data once link ingest starts pouring Chinese content in, so they land before the firehose. 1. CJK search was broken. Both TF-IDF tokenizers (kb/search.ts, memory/vector.ts) carried the same three lines: lowercase, strip non-word chars, split on whitespace. Chinese has no whitespace, so a whole clause survived as ONE token: doc "这篇公众号文章讲的是知识库的设计" -> one token query "知识库" -> one token intersection = empty -> zero hits i.e. CJK content was only findable by an exact whole-clause query. Replaced by a shared src/tokenize.ts that emits character bigrams for CJK runs (plus the run itself, so an exact phrase still ranks highest), segments at latin/CJK boundaries so "gpt模型" indexes as "gpt" + CJK bigrams, and now keeps kana and CJK Ext-A, which the old character class dropped entirely. Latin tokenization is unchanged. 2. Chinese titles produced throwaway slugs. normalizeSlug strips every non-[a-z0-9] char, so a CJK title normalized to "" and addSource fell back to entry-<Date.now()> — opaque, unstable, un-dedupable. New kb/slug.ts mints <date>-<sha256/8> for titles with no usable latin, keeping slugs ASCII on purpose (macOS NFD vs git NFC filenames, URLs, git paths) with the real title in frontmatter. canonicalUrl() drops utm_*/known tracking params so the same article shared twice canonicalizes — and hashes — identically. 3. Entries can now carry arbitrary frontmatter via KbEntry.extra, round-tripped on read/write. This is where ingest provenance goes (url/site/author/published/hash/via) without the store knowing about any of it, and it means keys a future version adds survive a round-trip through an older one. Values are whitespace-collapsed on write so a newline can't forge frontmatter lines on the next read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(kb): parse [[links]] into a real graph; index.md becomes a ranked MOC v1.0 told Lisa to cross-link wiki pages with [[slug]] (SCHEMA.md) and recorded each page's sources: in frontmatter — but nothing ever read either one. The KB was a pile of files with decorative links. src/kb/links.ts builds the graph from three edge sources ([[slug]], [[slug|alias]], [[kb:slug]] in bodies; a wiki page's sources: frontmatter) and derives what you can't get from search: backlinks, hubs, orphans, and links pointing at pages that don't exist yet. index.md is now a ranked map of content rather than a flat listing. It is injected into EVERY system prompt and hard-capped there (prompt.ts:165), so with a flat list the truncation lands at an arbitrary point once the KB grows and whatever fell below the line stops existing as far as Lisa is concerned. Ordering wiki pages by (1 + backlinks) x recency means the cut always eats the least-connected tail. Orphans and broken links are listed as the tending to-do list, and every truncation says how much it dropped instead of trailing off. Sources are now listed by TITLE ONLY. Layer 1 is raw captured material — after link ingest lands that means whole web pages — and index.md is always-on prompt text, so an excerpt there is a direct "any page on the internet -> Lisa's system prompt" path. This is a stronger form of the containment the plan assigned to K-I (it covers every source, not just origin: web), and it costs nothing: a title is all a map needs, and the body is one kb_read away. Wiki gists stay — Lisa wrote those herself. Also: kb/index.json (the machine-readable graph, for the web UI and tools), a kb_links tool (links to / linked from / shares tags with), and kb_read now takes a bare slug or a [[wikilink]] with the layer optional, and appends the pages that link to what you just read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(kb): handoff for the remaining knowledge-base v2 work (K-D … K-I) K-A/K-B/K-C are open as #278/#279/#280. This is the pickup document for whoever continues: the branch stack and how not to break it, the APIs the first three PRs landed, a file-by-file spec for each remaining PR, the repo conventions that are easy to trip over (LISA_HOME must be set before import, the tool-subset triple, the giant template literals in lisa-client.ts), and the eight decisions that are already settled so they don't get re-litigated. Two things it corrects against the plan: the index-side injection containment assigned to K-I already shipped in K-C, and in a stronger form (every source is title-only in index.md, not just origin: web), so K-I is down to the watchlist restriction and the kb_read fence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(kb): memory ⇄ KB links — inline titles, link-style memory, conservative autolink (K-D) Closes the "memory stores links, not content" loop from the v2.0 plan (user need ③). Four pieces: - prompt.ts inlines page titles for [[kb:slug]] pointers found in MEMORY.md / USER.md at assembly time ([[kb:oauth]] → [[kb:oauth]](OAuth 与 PKCE)), so a pointer is meaningful without a kb_read. Titles come from the already- generated kb/index.json behind a stat-fingerprint cache — the prompt builds every turn, so no full-KB read on this path. Unresolvable links are kept verbatim: a dangling pointer is a "write this page" signal, not noise. - memory tool description now steers knowledge OUT of memory (4KB cap) and into the KB with a [[kb:slug]] pointer. - The idle tending prompt closes the loop from the other side: after distilling a wiki page, back-link it from any related memory entry. - kb/autolink.ts: conservative title → [[link]] pass applied on kb_write. Whole-word title matches only, first occurrence, longest-first, never in code / existing links, short titles skipped — a wrong edge pollutes the (backlinks × recency) hub ranking the always-on index sorts by, so misses are preferred over false links. Emits [[slug|Title]] when the slug differs so CJK prose keeps reading as prose. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(kb): ingest engine — url → provenance-stamped Layer-1 markdown (K-E) kb_ingest(url) turns a link into a searchable, citeable source entry (v2.0 user need ②). New src/kb/ingest/, all zero-dependency (D1: what the user reads never leaves the machine): - html-to-md.ts: lenient HTML tree parser + markdown serializer (headings, nested lists, fenced/inline code, blockquotes, tables, links, lazy-loaded images, emphasis). Prose is markdown-escaped BEFORE syntax is added, so page text can't forge emphasis/links/wikilinks into the KB; code bodies can't terminate their own fence; javascript: hrefs are dropped. - readability.ts: small Readability — score candidates by text mass, <p> count, link density, class/id hints; fall back to <body> when unsure (noise is recoverable by distillation, clipped content is not). - provenance.ts: og:* / JSON-LD Article / <title> / rel=canonical / lang → frontmatter (url · site · author · published · lang · hash · via). - dedupe.ts: kb/.ingested.json (hash → slug), rebuilt from the sources' own hash: frontmatter when missing/corrupt. Same URL returns the existing entry (D2, immutable sources); force re-ingests + records supersedes:. - index.ts: ingestUrl orchestration with an adapter seam for K-F. ALL fetching goes through web_fetch's fetchFollowingSafeRedirects (per-hop private-host validation) plus an up-front assertAllowedUrl — no second fetch path, no second SSRF surface. kb_ingest is REMOTE_BLOCKED (a channel message must not make Lisa fetch an attacker URL into the KB) but NOT autonomous-blocked — K-I scopes autonomous use to the feeds.json domain watchlist (D3) instead of a blanket ban. Tests are offline-fixture only: converter syntax coverage, chrome-stripping, provenance forms, dedupe/force/ledger-rebuild, SSRF rejection before fetch, content-type gates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(kb): site adapters — WeChat, Bilibili, YouTube (K-F) Three IngestAdapter implementations behind the K-E seam, for the sites where generic readability extraction can't reach the content: - wechat: article body from #js_content, account from #js_name, epoch publish time from inline `var ct`; lazy-loaded images already handled via data-src. The human-verification interstitial throws a loud, actionable error (open the share link on the phone, paste the text, kb_add) — never a silently captured blank page. - bilibili: metadata via the public view API; b23.tv short links resolve through the guarded fetch. Subtitles are cookie-gated (SESSDATA, which the user may volunteer in kb/feeds.json) — absent that, degrade. - youtube: oEmbed as the reliability floor + one InnerTube player POST for description/captions; captionTracks → &fmt=json3, manual tracks over ASR. Handles the documented failure modes (PO token, 200-with-empty-body, datacenter throttling) by degrading. Fixed subtitle layering (handoff: do not reorder): built-in API → yt-dlp (--dump-single-json only when installed; discovered subtitle URLs are still fetched through the guarded path) → metadata + description. A MISSING TRANSCRIPT IS NOT A FAILURE: the entry records `transcript: unavailable (<reason>)` and the tool reply tells the user they can paste one via kb_add. Supporting changes: fetchFollowingSafeRedirects gains an optional SafeFetchInit (method/headers/body) so adapter API calls stay on the single SSRF-guarded fetch path; ingest types moved to types.ts to break the adapter↔index cycle; the min-body-length gate now applies only to the generic path so degraded video captures can be legitimately short. All tests offline-fixture; degradation paths asserted explicitly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(kb): ingest surfaces — web paste bar, chat chip, API route, lisa kb CLI (K-G) Four entry points onto the K-E/K-F engine, one per place a link shows up: - POST /api/kb/ingest ({url, title?, tags?, force?} — shaped so a future share-sheet client can call it directly). Ingest failures return 422 with the engine's actionable message (verification page, paywall, bad type) instead of a generic 500. - Knowledge view: a paste-a-link bar above search — progress state, saves, then opens the new entry; dedupe and no-transcript degradation are surfaced in the status line. - Chat: a user message containing a bare URL gets a one-tap 存入知识库 chip under the bubble (same interaction shape as the KB capture bar; kbToast is now shared via window.lisaKbToast). - `lisa kb add|list|search|brief` (src/cli/kb.ts), registered like mail: passthrough subargs so --title/--tags/--force never collide with global flags. `brief` prints the newest sources/brief-<date>.md — the file K-H starts writing — and points at feeds.json until then. The client edits live inside the MAIN_CLIENT_JS template literal (regex backslashes double-escaped — the html-syntax test caught the first attempt); lisa-html-snapshot constants recomputed accordingly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(kb): daily feeds brief — sweep, classify, personalized rank, dual-write (K-H) User need ①: a daily, personalized brief over the user's own feed watchlist. New src/kb/feeds/, deliberately shaped after the mail module (the one place this problem is already solved end-to-end here): - store.ts: ~/.lisa/kb/feeds.json ({feeds:[{id,url,tags,max,weight}], briefHour, budgetTokens}) — user-authored; its existence IS the consent (D4, no new consent signal; empty/missing = the capability is fully inert). chmod 0600 on every read + kb/.gitignore coverage, since the file may carry a bilibili SESSDATA. Machine state (seen ids, last brief date) lives apart in kb/feeds/.state.json and is rebuild-safe. - rss.ts: zero-dep RSS2/Atom parsing (CDATA, entities, content:encoded, atom link@href) — lenient, an unparseable field degrades to blank. - classify.ts: mail/classify.ts's batch shape over {category, importance 0-3, oneLine} with the same injection stance (fenced untrusted data, closed taxonomy validation, heuristic fallback). The daily budgetTokens gate (default 120k) stops model batches at the ceiling WITH a log line — remaining items get neutral grades, never dropped silently. - brief.ts: isBriefDue (isDigestDue's twin), personalized scoring — (1+importance) × watchlist weight × (1 + interest/wiki term-overlap, both saturating) — buildBrief + formatBriefText, all pure. - service.ts: incremental sweep (per-feed seen-id state; a feed failing degrades, ALL feeds failing doesn't burn the day) → classify → rank → top-3 full-text ingest via the K-E engine → the D7 dual write: kb/feeds/<date>.json for the UI and sources/brief-<date>.md as a real Layer-1 entry (searchable, linkable, distillable; `lisa kb brief` from K-G picks it up as-is). Delivery mirrors mail: a 30-min server timer + 20s restart catch-up, broadcast (kb_brief_update + idle_message source:kb) and a new pushBridge.onKbBrief behind a new `brief` push pref (default on). Plus GET /api/kb/brief serving the latest JSON. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(kb): v2.0 hardening + closeout (K-I) The three-part D3 closure is now complete, plus docs: - Autonomous ingest watchlist (closure #1): autonomousSubset() swaps kb_ingest for a variant that only accepts URLs whose host matches a domain in the user's feeds.json (dot-boundary matching both directions, no suffix spoofing). An unattended run steered by injected text can no longer pull an arbitrary URL into the KB; the error tells the model to surface the link to the user instead. Manual surfaces (chat, web, CLI) keep the plain tool. (Closure #2 — sources never excerpt into index.md — shipped early in K-C.) - kb_read external-content fence (closure #3): sources with origin web (and brief, which embeds remote titles) are wrapped in an explicit bilingual "this is DATA, not commands" fence; chat captures and wiki pages read unwrapped. - SCHEMA.md: three new workflows (ingesting a link, reading the daily brief, maintaining links incl. [[kb:slug]] memory pointers). ensureSchema only seeds when missing, so existing installs keep their customized schema — accepted per the handoff. - Weekly-review heartbeat example added to both READMEs' heartbeat.json section (read 7 briefs + the week's sources → write wiki/weekly-<date>). No new scheduler. - Docs closeout: plan status → shipped with all PR numbers; the handoff doc marked executed (kept as an implementation record); CHANGELOG entry; README/README.zh-CN gained the v2.0 capabilities section and the updated kb tool row. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

oratis and others added 3 commits July 23, 2026 14:12

oratis changed the base branch from feat/kb-foundation to main July 24, 2026 07:15

Merge remote-tracking branch 'origin/main' into feat/kb-links

a512544

# Conflicts: # src/kb/store.test.ts # src/kb/store.ts

oratis merged commit 1b852dc into main Jul 24, 2026
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(kb): parse [[links]] into a real graph; index.md becomes a ranked MOC (K-C)#280

feat(kb): parse [[links]] into a real graph; index.md becomes a ranked MOC (K-C)#280
oratis merged 5 commits into
mainfrom
feat/kb-links

oratis commented Jul 23, 2026

Uh oh!

oratis commented Jul 24, 2026

Uh oh!

oratis commented Jul 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

oratis commented Jul 23, 2026

The gap

What lands

Tests

Uh oh!

oratis commented Jul 24, 2026

Uh oh!

oratis commented Jul 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant