fix(agent-memory): yaml-escape hint values in memory_observe and add max_hint_bytes config

[ralf003]

Problem

memory_observe writes the model-provided hint value into YAML frontmatter with minimal sanitization — only newlines are replaced with spaces. YAML special characters (# comment, : key-value separator, quotes, backslashes) can corrupt the frontmatter block. For example, hint: fix #123 and #456 is truncated by the YAML parser at fix because # starts a comment.

Additionally, there is no size limit on the hint field, unlike max_read_bytes/max_write_bytes/max_append_bytes for other MemoryConfig fields. A rogue or misconfigured model could inject arbitrarily long hints.

Changes (116 lines, 3 files)

src/tools/memory_observe.rs

• Add yaml_escape_hint() function (~20 lines) that double-quotes hint values and escapes ", \, \n/\r, and ASCII control characters for safe YAML inclusion
• Add max_hint_bytes enforcement (default 512) — hints exceeding the limit are rejected with InvalidArgument
• Add 9 unit tests covering: # comment characters, colons, quotes, backslashes, multiline, empty strings, control chars, and normal text

src/config.rs

• Add max_hint_bytes: u64 to MemoryConfig with default 512
• Add default_max_hint_bytes() function
• Add MEMORY_MAX_HINT_BYTES env var override in apply_env_overrides()

Service layer propagation

• src/service/mod.rs: pass &self.config.memory through to tools::memory_observe (no signature change at service level)

Verification

• All 147 unit tests pass on Linux (Alibaba Cloud ECS, x86_64, Rust 1.96)
• 9 new yaml_escape_hint tests pass
• Existing tests unchanged

[shiloong]

Review — fix(memory): yaml-escape hint values in memory_observe

先声明命名：本 PR 标题与 commit subject 的 scope 应为 memory，不是 agent-memory。.github/commitlint.config.json 的 scope-enum 只列了 memory；#1038(docs(memory))、#1043(test(memory)) 以及 src/agent-memory/ 全部历史 commit 均用 memory。agent-memory 既不在 enum 内也与其他 PR 不一致。建议 amend 把 fix(agent-memory) → fix(memory)（不新增 commit），分支名 fix/agent-memory-hint-sanitization → fix/memory/hint-sanitization 为可选清理。

结论按 block / suggest / clean 分级。

block — 修复基于误诊，且对反斜杠引入回退

src/tools/memory_observe.rs 写出的 frontmatter 不是用 YAML parser 读回的。实际读取方是：

• src/tools/user_profile.rs:345 parse_frontmatter_flat
• src/tools/memory_index.rs:225 extract_title_and_description
• src/tools/memory_sovereignty.rs:372 parse_frontmatter_flat

三者都是手写逐行解析：line.split_once(": ") 取值，再 value.trim().trim_matches('"')。Cargo.toml 也没有 serde_yaml 依赖。由此两个问题：

1. PR 描述的 # 截断在本代码库并不存在。手写解析器不把 # 当注释，hint: fix #123 and #456 经 split_once(": ") 得 value=fix #123 and #456，完整保留。problem statement 基于标准 YAML parser 的假设，属误诊。

2. YAML 双引号转义与读取方不对称，反斜杠回退。yaml_escape_hint 把 \ 写成 \\、" 写成 \"，但读取方只 trim_matches('"') 剥外层引号，不会反转义 \"/\\：

   • hint = C:\Users\admin：改前写入 hint: C:\Users\admin，读回 C:\Users\admin（正确）；改后写入 hint: "C:\\Users\\admin"，读回 C:\\Users\\admin（多出反斜杠，回退）。
   • hint = she said "hi"：改前读回 she said hi（丢引号，已坏）；改后读回 she said \"hi\"（残留字面反斜杠引号，换了一种坏法）。

正确修法二选一：① 写入与读取都改用真正的 YAML 序列化/解析（加 serde_yaml 或抽出统一 frontmatter 模块）；② 保持改前"非引号 + 仅替换换行"方案——该方案对手写解析器其实已足够（唯一真实风险是 \n---\n 早终止 frontmatter，改前 replace('\n', " ") 已覆盖）。

suggest — 缺少 round-trip 测试

新增 9 个 yaml_escape_hint 单测只验证转义输出格式，没有一条"写入 → 用真实读取方 parse_frontmatter_flat 读回 → 断言相等"的 round-trip 测试。这正是上面回退能溜过的原因。"147 单测全过"不能证明正确性，因为读取路径未被覆盖。

clean

• max_hint_bytes 用 h.len()（字节），字段名与文档均为 bytes，语义一致。但错误串 "hint length {} exceeds…" 用词模糊，建议改为 hint byte length。
• escape_hint_multiline 对 \r\n 产生两个空格，文档已注明，可接受；如想更干净可在 control 分支前统一 c.is_whitespace() 折叠。
• 默认 512 字节对 CJK hint ≈ 170 字符，偏紧，可考虑提到 2 KiB，非阻塞。

建议：命名 amend + 改用上述任一正确修法 + 补 round-trip 测试后重新提审。

[ralf003]

Hi @shiloong — gentle ping on this PR. It's been 10 days without review.

Rebased today onto latest main (v0.2.1 + CJK LIKE fallback + hermeticized tests). No conflicts.

Quick summary for reviewer:

• memory_observe writes model-provided hint values directly into YAML frontmatter, but YAML special characters (#, :, quotes, backslashes) are not escaped. A hint like "fix feat(agent-sec-core): add bundled loongshield build flow #123" is truncated at # because the YAML parser treats it as a comment.
• Added yaml_escape_hint() (~20 lines, double-quote + escape ", \, \n/\r, control chars)
• Added max_hint_bytes=512 (matching the pattern of existing max_read_bytes/max_write_bytes/max_append_bytes in MemoryConfig) to prevent unbounded model-generated hints
• 9 new unit tests cover edge cases; all 147 existing tests pass unchanged

Why 512 bytes: follows the existing MemoryConfig pattern — a hint is metadata, not content, so 512 bytes is generous enough for a useful one-sentence summary while preventing abuse. Configurable via MEMORY_MAX_HINT_BYTES env var if needed.

cc @Forrest-ly (for memory review — similar area to the CJK recall fix from 7/4)

Would appreciate a review when you have bandwidth. Thanks!

（If there's another memory maintainer I should tag instead, happy to redirect.）

[shiloong]

@ralf003 Re-ping after the rebase — checking the latest diff, the two block-level items from my earlier review (#1154 (comment)) are still unaddressed, and the commit subject is still fix(agent-memory). Restating concisely with concrete evidence so we can move forward:

1. The #-truncation premise is wrong for this codebase

memory_observe writes frontmatter, but nothing reads it back with a YAML parser. The hint field is read only by the hand-rolled line parsers:

• src/tools/user_profile.rs:345 parse_frontmatter_flat → line.split_once(": ") then value.trim().trim_matches('"')
• src/tools/memory_index.rs:225 extract_title_and_description → same pattern
• src/tools/memory_sovereignty.rs:372 parse_frontmatter_flat → same pattern

Cargo.toml has no serde_yaml / yaml-rust dependency. These parsers don't treat # as a comment, so hint: fix #123 and #456 round-trips intact as fix #123 and #456. The PR's stated problem ("truncated at #") doesn't occur on any read path in the repo.

2. yaml_escape_hint regresses backslash round-trip

Because the reader only does trim_matches('"') (strips outer quotes, no unescaping), the YAML-style \\ / \" escaping is asymmetric:

hint input	written line	read-back value
C:\Users\admin (was correct before this PR)	hint: "C:\\Users\\admin"	C:\\Users\\admin ← doubled backslashes, regression
she said "hi"	hint: "she said \"hi\""	she said \"hi\" ← literal backslash-quote remains

Before this PR, C:\Users\admin round-tripped correctly (single \ preserved); after, it comes back as C:\\Users\\admin. That's a real correctness regression on the actual read path.

Ask

Could you either:

• (a) point me to a YAML-parser read path I missed (in which case the escape is correct and I'll withdraw), or
• (b) adopt one of the two correct fixes:
  • add serde_yaml and use it for both write and read of frontmatter (shared module), or
  • revert to the pre-PR unquoted form (h.replace('\n', " ")) — which is already sufficient for the hand-rolled parser, since the only real risk is \n---\n early-terminating the block, and newline→space already covers it.

The max_hint_bytes cap and MEMORY_MAX_HINT_BYTES env override are fine independently — keep those regardless.

3. Missing round-trip test

The 9 new yaml_escape_hint unit tests only assert the escaped output form. Please add at least one test that writes via memory_observe and reads back via the real parse_frontmatter_flat, asserting the original hint survives — that's the test that would have caught the backslash regression above.

4. Naming (reminder)

Commit subject is still fix(agent-memory): …. Per .github/commitlint.config.json scope-enum and every existing src/agent-memory/ commit, the scope is memory (also matches #1038 docs(memory) / #1043 test(memory)). Please amend to fix(memory): … (no new commit).

Happy to be corrected on (a) if I missed a parser.

[ralf003]

@shiloong ready for re-review.

Changes since last review:

1. Replaced yaml_escape_hint() with sanitize_hint() — no more YAML escaping, only newline and control char replacement
2. Added max_hint_bytes config (default 512) for hint size control
3. Added 14 new tests (8 unit + 6 round-trip), all 178/178 passing
4. CI is all green

PTAL, thanks!

[Forrest-ly]

LGTM, THS!