feat(kb): ingest engine — url → provenance-stamped Layer-1 markdown (K-E)#283
Merged
Conversation
Design of record for three KB capabilities on top of v1.0 (#235-#242): - K1 paste any link (web / 公众号 / B站 / YouTube) -> Markdown source - K2 a watchlist -> daily fetched, classified, summarized brief - K3 [[slug]] parsed into a real link graph; index.md becomes a ranked map-of-content; memory stores [[kb:slug]] links instead of content Review of the shipped v1.0 code found five gaps this plan closes, one of them a confirmed bug: both TF-IDF tokenizers (kb/search.ts:45 and memory/vector.ts:212) split on whitespace, so a whole CJK run collapses into a single token and Chinese content only matches an exact whole-string query. Also: normalizeSlug strips non-ASCII so every Chinese title falls back to entry-<timestamp>; web_fetch is lossy plain text with no metadata and no store; [[slug]] is documented in SCHEMA.md but nothing parses it; and there is no KB scheduler at all. Includes the pro/con debate for each decision (self-built extractor vs deps vs remote reader, immutable sources vs re-ingest, autonomous ingest as a prompt-injection path, consent, ASCII vs CJK slugs, brief storage, and RSSHub for feeds without one) and a nine-PR ship order. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three foundation fixes the rest of the KB v2 work stands on
(docs/PLAN_KNOWLEDGE_BASE_v2.0.md K-B). All three are things that would
have corrupted or lost data once link ingest starts pouring Chinese
content in, so they land before the firehose.
1. CJK search was broken. Both TF-IDF tokenizers (kb/search.ts,
memory/vector.ts) carried the same three lines: lowercase, strip
non-word chars, split on whitespace. Chinese has no whitespace, so a
whole clause survived as ONE token:
doc "这篇公众号文章讲的是知识库的设计" -> one token
query "知识库" -> one token
intersection = empty -> zero hits
i.e. CJK content was only findable by an exact whole-clause query.
Replaced by a shared src/tokenize.ts that emits character bigrams for
CJK runs (plus the run itself, so an exact phrase still ranks
highest), segments at latin/CJK boundaries so "gpt模型" indexes as
"gpt" + CJK bigrams, and now keeps kana and CJK Ext-A, which the old
character class dropped entirely. Latin tokenization is unchanged.
2. Chinese titles produced throwaway slugs. normalizeSlug strips every
non-[a-z0-9] char, so a CJK title normalized to "" and addSource fell
back to entry-<Date.now()> — opaque, unstable, un-dedupable. New
kb/slug.ts mints <date>-<sha256/8> for titles with no usable latin,
keeping slugs ASCII on purpose (macOS NFD vs git NFC filenames, URLs,
git paths) with the real title in frontmatter. canonicalUrl() drops
utm_*/known tracking params so the same article shared twice
canonicalizes — and hashes — identically.
3. Entries can now carry arbitrary frontmatter via KbEntry.extra,
round-tripped on read/write. This is where ingest provenance goes
(url/site/author/published/hash/via) without the store knowing about
any of it, and it means keys a future version adds survive a
round-trip through an older one. Values are whitespace-collapsed on
write so a newline can't forge frontmatter lines on the next read.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…d MOC v1.0 told Lisa to cross-link wiki pages with [[slug]] (SCHEMA.md) and recorded each page's sources: in frontmatter — but nothing ever read either one. The KB was a pile of files with decorative links. src/kb/links.ts builds the graph from three edge sources ([[slug]], [[slug|alias]], [[kb:slug]] in bodies; a wiki page's sources: frontmatter) and derives what you can't get from search: backlinks, hubs, orphans, and links pointing at pages that don't exist yet. index.md is now a ranked map of content rather than a flat listing. It is injected into EVERY system prompt and hard-capped there (prompt.ts:165), so with a flat list the truncation lands at an arbitrary point once the KB grows and whatever fell below the line stops existing as far as Lisa is concerned. Ordering wiki pages by (1 + backlinks) x recency means the cut always eats the least-connected tail. Orphans and broken links are listed as the tending to-do list, and every truncation says how much it dropped instead of trailing off. Sources are now listed by TITLE ONLY. Layer 1 is raw captured material — after link ingest lands that means whole web pages — and index.md is always-on prompt text, so an excerpt there is a direct "any page on the internet -> Lisa's system prompt" path. This is a stronger form of the containment the plan assigned to K-I (it covers every source, not just origin: web), and it costs nothing: a title is all a map needs, and the body is one kb_read away. Wiki gists stay — Lisa wrote those herself. Also: kb/index.json (the machine-readable graph, for the web UI and tools), a kb_links tool (links to / linked from / shares tags with), and kb_read now takes a bare slug or a [[wikilink]] with the layer optional, and appends the pages that link to what you just read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
K-A/K-B/K-C are open as #278/#279/#280. This is the pickup document for whoever continues: the branch stack and how not to break it, the APIs the first three PRs landed, a file-by-file spec for each remaining PR, the repo conventions that are easy to trip over (LISA_HOME must be set before import, the tool-subset triple, the giant template literals in lisa-client.ts), and the eight decisions that are already settled so they don't get re-litigated. Two things it corrects against the plan: the index-side injection containment assigned to K-I already shipped in K-C, and in a stronger form (every source is title-only in index.md, not just origin: web), so K-I is down to the watchlist restriction and the kb_read fence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rvative autolink (K-D) Closes the "memory stores links, not content" loop from the v2.0 plan (user need ③). Four pieces: - prompt.ts inlines page titles for [[kb:slug]] pointers found in MEMORY.md / USER.md at assembly time ([[kb:oauth]] → [[kb:oauth]](OAuth 与 PKCE)), so a pointer is meaningful without a kb_read. Titles come from the already- generated kb/index.json behind a stat-fingerprint cache — the prompt builds every turn, so no full-KB read on this path. Unresolvable links are kept verbatim: a dangling pointer is a "write this page" signal, not noise. - memory tool description now steers knowledge OUT of memory (4KB cap) and into the KB with a [[kb:slug]] pointer. - The idle tending prompt closes the loop from the other side: after distilling a wiki page, back-link it from any related memory entry. - kb/autolink.ts: conservative title → [[link]] pass applied on kb_write. Whole-word title matches only, first occurrence, longest-first, never in code / existing links, short titles skipped — a wrong edge pollutes the (backlinks × recency) hub ranking the always-on index sorts by, so misses are preferred over false links. Emits [[slug|Title]] when the slug differs so CJK prose keeps reading as prose. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…K-E) kb_ingest(url) turns a link into a searchable, citeable source entry (v2.0 user need ②). New src/kb/ingest/, all zero-dependency (D1: what the user reads never leaves the machine): - html-to-md.ts: lenient HTML tree parser + markdown serializer (headings, nested lists, fenced/inline code, blockquotes, tables, links, lazy-loaded images, emphasis). Prose is markdown-escaped BEFORE syntax is added, so page text can't forge emphasis/links/wikilinks into the KB; code bodies can't terminate their own fence; javascript: hrefs are dropped. - readability.ts: small Readability — score candidates by text mass, <p> count, link density, class/id hints; fall back to <body> when unsure (noise is recoverable by distillation, clipped content is not). - provenance.ts: og:* / JSON-LD Article / <title> / rel=canonical / lang → frontmatter (url · site · author · published · lang · hash · via). - dedupe.ts: kb/.ingested.json (hash → slug), rebuilt from the sources' own hash: frontmatter when missing/corrupt. Same URL returns the existing entry (D2, immutable sources); force re-ingests + records supersedes:. - index.ts: ingestUrl orchestration with an adapter seam for K-F. ALL fetching goes through web_fetch's fetchFollowingSafeRedirects (per-hop private-host validation) plus an up-front assertAllowedUrl — no second fetch path, no second SSRF surface. kb_ingest is REMOTE_BLOCKED (a channel message must not make Lisa fetch an attacker URL into the KB) but NOT autonomous-blocked — K-I scopes autonomous use to the feeds.json domain watchlist (D3) instead of a blanket ban. Tests are offline-fixture only: converter syntax coverage, chrome-stripping, provenance forms, dedupe/force/ledger-rebuild, SSRF rejection before fetch, content-type gates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 23, 2026
Owner
Author
|
Really nice engineering — zero-dep, single HTML parser shared with readability, escape-first serialization (wikilink/table/ Blocker — untrusted title injects into the always-on prompt. Content-loss bug. html-to-md.ts:68-73: a self-closing/unclosed drop tag ( Hardening (recommend before this goes autonomous in K-I):
Happy to re-review once the title collapse + drop-tag fix land. |
…fetch - html-to-md: a self-closing drop tag (<svg/>, <iframe/>) has no matching close, so the parser jumped TAG_RE.lastIndex to html.length and silently discarded the rest of the document. Skip just the tag for self-closed forms; a genuinely unclosed drop tag still drops to EOF (safer than leaking raw script/style source). +regression test. - genericFetch: reject unsupported content-types from the header before buffering, reject an oversized Content-Length up front, and read the body through a bounded reader (readBodyCapped) so a chunked/undeclared response can't exhaust memory — matters once ingest runs unattended from the brief. (The untrusted-title→frontmatter injection flagged on this PR is fixed at its root in store.ts, already merged.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts: # src/kb/links.test.ts # src/kb/links.ts # src/kb/store.test.ts # src/kb/store.ts # src/kb/tool.ts
Owner
Author
|
Reconciled onto main. Landed the two fixes from review: (1) self-closing/unclosed drop-tag content-loss in html-to-md (was jumping to EOF), (2) bounded/reordered fetch (content-type check before buffering + Content-Length precheck + capped streaming read) so unattended ingest can't OOM. +regression tests. The title→frontmatter injection is fixed at root in store.ts (#279). typecheck + 115 KB tests green. Merging. Deferred as follow-ups (noted): DNS-resolving SSRF guard, ledger file-lock, fetch timeout. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the knowledge-base v2.0 stack (plan: #278, handoff: #281). Stacked on #282 (K-D) — merge order: #278 → #279 → #280 → #281 → #282 → this.
差距
需求 ②"链接 → 知识"没有入口：贴个链接只能靠 web_fetch 读一次就丢，无出处、无去重、不进 KB。
做了什么
新目录
src/kb/ingest/（全部零依赖，D1）+ 新工具kb_ingest {url, title?, tags?, force?}：html-to-md.ts— 宽容树解析器 + MD 序列化：h1-6 / 段落 / 嵌套 ul-ol（含未闭合<li>与 HTML5 隐式闭合）/pre-code围栏（带语言、防围栏逃逸）/ 行内 code（防反引号逃逸）/ blockquote / table / a / img（含data-src懒加载）/ strong-em / hr。先转义正文再加语法：页面文本里的*、[[]]、|不可能伪造出 MD 结构或 wikilink；javascript:链接只留文字。readability.ts— 精简版正文抽取：文本量 /<p>数 / 链接密度 / class-id 加减分选最高分子树；分数或文本占比不够就回落整个<body>——噪声可以靠蒸馏消化，被裁掉的正文找不回来。provenance.ts— og:* / JSON-LD Article 系 /<title>/rel=canonical/<html lang>→{title, site, author, published, lang, canonical}。dedupe.ts—kb/.ingested.json（hash → slug），损坏/缺失时从 sources 的hash:frontmatter 全量重建。同 URL（canonicalUrl 去 utm 后）默认返回既有条目（D2）；force新写 +supersedes:。index.ts—ingestUrl编排 +IngestAdapter接口（K-F 的 wechat/bilibili/youtube 插这里）。所有请求走fetchFollowingSafeRedirects（逐跳私网校验），入口再加一道assertAllowedUrl——不另起 fetch，不开第二个 SSRF 面。为什么这么选
kb_ingest进REMOTE_BLOCKED_TOOL_NAMES（远程消息不能指挥 Lisa 抓任意 URL 入库），不进AUTONOMOUS_BLOCKED——按交接文档，自主限制在 K-I 用 watchlist 白名单实现（D3）。opts.fetchImpl，SSRF 校验在注入点之前，测试可断言"私网 URL 拒绝且 0 次网络调用"。测了什么
全部离线固件（不打网络）：转换器 13 例（含转义/围栏逃逸/隐式闭合）、正文抽取甩 nav-sidebar-footer、出处四来源、去重（utm 变体命中同条目）、force+supersedes、私网拒绝、content-type 门、坏 ledger 重建。
npm run typecheck干净；npm test1230 pass / 0 fail。🤖 Generated with Claude Code