alibaba · Forrest-ly · Jul 6, 2026 · Jul 2, 2026
@@ -24,6 +24,7 @@ Three integration paths are available:
 |---|---|---|
 | Schema compression | ~57% | Compresses OpenAI Function Calling tool schemas |
 | Response compression | ~26–78% | Compresses API / tool responses (varies by content type) |
+| Reversible compression (stash) | — | Dropped array items are stashed and retrievable via `<<tokenless:KEY>>` markers |
 | TOON context compression | 15–40% | Encodes JSON to TOON format for LLMs |
 | Command rewriting | 60–90% | Filters CLI output via RTK (70+ commands supported) |
 | Tool Ready | reduces retry waste | Pre-check env, auto-fix deps, failure attribution |
@@ -41,7 +42,7 @@ Three integration paths are available:
 Token-Less/
 ├── crates/tokenless-schema/   # Core library: SchemaCompressor + ResponseCompressor
 ├── crates/tokenless-ccr/      # Reversible compression stash (Compress-Cache-Retrieve)
-├── crates/tokenless-cli/      # CLI binary: `tokenless` command (env-check, compress, stats)
+├── crates/tokenless-cli/      # CLI binary: `tokenless` command (env-check, compress, retrieve, stats)
 ├── adapters/tokenless/        # FHS adapter bundle (manifest, common, openclaw, hermes, qoder, claude-code, codex)
 │   ├── manifest.json            # Adapter manifest (cosh + openclaw + hermes + qoder + claude-code + codex)
 │   ├── common/                  # Shared: hooks, spec, env-fix, commands, cosh-extension
@@ -106,6 +107,25 @@ tokenless compress-response -f response.json
 curl -s https://api.example.com/data | tokenless compress-response
 ```
 
+By default `compress-response` stashes dropped array items so they can be
+retrieved later (see [Reversible compression](docs/stash-reversible-compression.md)).
+Pass `--no-stash` for lossy truncation, or `--stash-db <path>` to override the
+stash database (default `~/.tokenless/stash.db`).
+
+### retrieve
+
+Recover a payload stashed during `compress-response`. Accepts a bare 24-hex
+hash or any text containing a `<<tokenless:HASH>>` marker:
+
+```bash
+# Bare hash
+tokenless retrieve c30ccf5ed1125e0ed871ba8e
+
+# Or paste the whole truncation line — the hash is extracted automatically.
+# (Use the FULL 24-hex hash from your output; the value below is shorthand.)
+tokenless retrieve "<... 195 items truncated, retrieve with <<tokenless:c30ccf5ed1125e0ed871ba8e>>"
+```
+
 ### compress-toon / decompress-toon
 
 Encode JSON to TOON format (or decode back to JSON):

@@ -556,10 +556,16 @@ mod tests {
         let compressor = ResponseCompressor::new().with_truncate_arrays_at(3);
         let arr: Vec<i32> = (1..=10).collect();
         let result = compressor.compress(&json!(arr));
-        let marker = result.as_array().unwrap().last().unwrap();
-        let s = marker.as_str().unwrap();
-        assert!(s.contains("more items truncated"));
-        assert!(!s.contains("tokenless:"));
+        let arr_result = result.as_array().unwrap();
+        // 3 kept items + 1 marker
+        assert_eq!(arr_result.len(), 4);
+        assert_eq!(arr_result[0], json!(1));
+        assert_eq!(arr_result[1], json!(2));
+        assert_eq!(arr_result[2], json!(3));
+        let marker = arr_result[3].as_str().unwrap();
+        assert!(marker.contains("more items truncated"));
+        assert!(marker.contains("7")); // 10 - 3 dropped
+        assert!(!marker.contains("tokenless:"));
     }
 
     #[test]
@@ -576,6 +582,10 @@ mod tests {
         let arr_result = result.as_array().unwrap();
         // 3 kept items + 1 marker
         assert_eq!(arr_result.len(), 4);
+        // Kept items are the first 3 (off-by-one in the slice would break this).
+        assert_eq!(arr_result[0], json!(1));
+        assert_eq!(arr_result[1], json!(2));
+        assert_eq!(arr_result[2], json!(3));
         let marker = arr_result[3].as_str().unwrap();
         assert!(marker.contains("retrieve with"));
         let hash = extract_hash(marker).expect("marker should embed a hash");
@@ -619,4 +629,81 @@ mod tests {
         assert!(s.contains("more items truncated"));
         assert!(!s.contains("tokenless:"));
     }
+
+    #[test]
+    fn test_stash_round_trip_with_cjk_items() {
+        // CJK payloads are multi-byte; the stashed JSON must round-trip
+        // byte-for-byte (review §12: char vs byte semantics).
+        use std::sync::Arc;
+        use tokenless_ccr::{InMemoryStore, StashStore, extract_hash};
+
+        let store = Arc::new(InMemoryStore::new());
+        let compressor = ResponseCompressor::new()
+            .with_truncate_arrays_at(2)
+            .with_stash_store(store.clone());
+        let arr = json!(["你好世界", "第二个条目", "第三个条目", "第四个条目"]);
+        let result = compressor.compress(&arr);
+        let arr_result = result.as_array().unwrap();
+        // Kept items are the first 2.
+        assert_eq!(arr_result[0], json!("你好世界"));
+        assert_eq!(arr_result[1], json!("第二个条目"));
+        let marker = arr_result.last().unwrap();
+        let hash = extract_hash(marker.as_str().unwrap()).unwrap();
+        let retrieved = store.retrieve(hash).unwrap().unwrap();
+        let recovered: Vec<String> = serde_json::from_str(&retrieved).unwrap();
+        assert_eq!(recovered, vec!["第三个条目", "第四个条目"]);
+    }
+
+    #[test]
+    fn test_stash_round_trip_with_object_array() {
+        // The "100 normal + 2 error" case: dropped object items must be
+        // recoverable verbatim, including fields the compressor would
+        // otherwise strip (debug/trace). The kept item carries a `debug`
+        // field too, so the test can prove kept items ARE compressed
+        // (debug stripped) while stashed items are raw (debug preserved).
+        use std::sync::Arc;
+        use tokenless_ccr::{InMemoryStore, StashStore, extract_hash};
+
+        let store = Arc::new(InMemoryStore::new());
+        let compressor = ResponseCompressor::new()
+            .with_truncate_arrays_at(1)
+            .with_stash_store(store.clone());
+        let arr = json!([
+            {"id": 1, "status": "ok", "debug": "should be stripped"},
+            {"id": 2, "status": "error", "debug": "trace data"},
+            {"id": 3, "status": "ok"}
+        ]);
+        let result = compressor.compress(&arr);
+        let arr_result = result.as_array().unwrap();
+        // Kept item is compressed: debug stripped.
+        assert_eq!(arr_result[0]["id"], json!(1));
+        assert!(
+            arr_result[0].get("debug").is_none(),
+            "kept items must be compressed (debug stripped)"
+        );
+        let marker = arr_result.last().unwrap();
+        let hash = extract_hash(marker.as_str().unwrap()).unwrap();
+        let retrieved = store.retrieve(hash).unwrap().unwrap();
+        let recovered: Vec<Value> = serde_json::from_str(&retrieved).unwrap();
+        // Stashed items are raw (pre-compression): debug survives.
+        assert_eq!(recovered.len(), 2);
+        assert_eq!(recovered[0]["debug"], json!("trace data"));
+    }
+
+    #[test]
+    fn test_stash_not_engaged_when_array_within_limit() {
+        // No truncation → no stash write → no marker. Stash stays empty.
+        use std::sync::Arc;
+        use tokenless_ccr::InMemoryStore;
+
+        let store = Arc::new(InMemoryStore::new());
+        let compressor = ResponseCompressor::new()
+            .with_truncate_arrays_at(10)
+            .with_stash_store(store.clone());
+        let arr: Vec<i32> = (1..=5).collect();
+        let result = compressor.compress(&json!(arr));
+        // No truncation marker at all.
+        assert!(result.as_array().unwrap().iter().all(|v| v.is_number()));
+        assert_eq!(store.len(), 0);
+    }
 }
@@ -0,0 +1,142 @@
+# Reversible Compression (Stash)
+
+Tokenless compression is *inline lossy, end-to-end lossless*: when a compressor
+truncates content, the dropped payload is stashed under a BLAKE3-derived key and
+a `<<tokenless:KEY>>` marker is embedded in the compressed output. The LLM can
+quote the marker back to retrieve the original payload on demand, so no
+information is permanently lost even though the inline representation is
+smaller.
+
+This mirrors Headroom's CCR (Compress-Cache-Retrieve); the mechanism here is
+called **stash** to avoid the proprietary abbreviation.
+
+## How it works
+
+1. **Compress**: `ResponseCompressor` truncates oversized arrays (default:
+   keep the first 32 items). The dropped tail is serialized to JSON and
+   `stash.stash(payload)` stores it, returning a 24-hex BLAKE3 key.
+2. **Mark**: the truncation marker becomes
+   `<... N items truncated, retrieve with <<tokenless:KEY>>`.
+3. **Retrieve**: the LLM emits the marker (or the bare key); the agent calls
+   `tokenless retrieve <KEY>` (or the future MCP `tokenless_retrieve` tool)
+   to fetch the original payload from the stash.
+
+When no stash store is attached (`Option<Arc<dyn StashStore>>` = `None`),
+truncation is lossy and non-retrievable — the original pre-stash behavior.
+This keeps the stash off the core compression path unless a caller explicitly
+enables it.
+
+## Marker format
+
+```
+<<tokenless:HASH>>
+```
+
+- `HASH` is the first 24 hex characters (12 bytes / 96 bits) of a BLAKE3 hash
+  of the stashed payload. 96 bits makes a collision astronomically unlikely
+  (2⁴⁸ birthday bound), so a key is treated as a unique handle.
+- The `tokenless:` namespace distinguishes these markers from Headroom's
+  `<<ccr:HASH>>` and from any user content.
+- `tokenless_ccr::parse_marker` accepts a string that is exactly a marker;
+  `tokenless_ccr::extract_hash` scans arbitrary text (e.g. a whole truncation
+  line) and returns the first embedded hash. Both reject malformed input
+  (wrong length, non-hex) by returning `None` rather than panicking, so
+  callers can pass untrusted LLM output directly.
+
+## Backends
+
+| Backend | Feature | Persistence | Use when |
+|---|---|---|---|
+| `InMemoryStore` | default | process memory | tests, single-process CLI runs |
+| `SqliteStore` | `sqlite` (on by default) | SQLite file (WAL) | **production hook path** |
+
+The tokenless hooks fork+exec a fresh process per call, so an in-memory store
+loses its contents between calls. `SqliteStore` is therefore the recommended
+production backend: it persists to `~/.tokenless/stash.db` so a `retrieve` in
+one process can read what a `compress` in another process wrote.
+
+Both backends enforce:
+
+- **TTL**: entries expire after a fixed lifetime (InMemory 5 min; SQLite 1 h).
+  An hour comfortably covers a typical agent session's compress→retrieve
+  round trip. Expiry is enforced **on read** — `retrieve()` filters out
+  expired rows (SQLite `WHERE expires_at >= now`) and `len()` counts only
+  live entries, so expired data is never returned. The rows themselves
+  remain on disk until either capacity-based FIFO eviction (triggered by
+  `stash()`) or an explicit `evict_expired()` call (available for bulk
+  cleanup but not called automatically), so the SQLite file can grow
+  beyond the capacity before a `stash()` triggers a trim.
+- **Capacity** (FIFO): once the live entry count exceeds the limit (InMemory
+  1000; SQLite 10 000), the oldest entries are evicted. This prevents
+  unbounded growth from runaway compression.
+
+## CLI
+
+```bash
+# Compress with stash on by default — dropped array items become retrievable.
+echo '[1,2,...,200]' | tokenless compress-response --truncate-arrays-at 5
+# -> [1,2,3,4,5,"<... 195 items truncated, retrieve with <<tokenless:c30c…>>"]
+
+# Retrieve the original dropped items (same stash db, separate process).
+tokenless retrieve c30ccf5ed1125e0ed871ba8e
+# -> [6,7,8,…,200]
+
+# Pass the whole truncation line; the hash is extracted automatically.
+tokenless retrieve "<... 195 items truncated, retrieve with <<tokenless:c30c…>>"
+
+# Opt out of stash (lossy truncation, the pre-stash behavior).
+echo '[...]' | tokenless compress-response --no-stash
+
+# Override the stash db path (must be under the trusted home directory).
+tokenless retrieve <hash> --stash-db ~/.tokenless/alt-stash.db
+```
+
+`TOKENLESS_STASH_DB` mirrors `TOKENLESS_STATS_DB` as an env override.
+
+## Security model
+
+The stash db path is resolved under the **trusted home directory** — derived
+from `getpwuid_r(getuid())`, never from `$HOME` (which a parent process can
+spoof to redirect state into attacker-writable paths). An override
+(`--stash-db` or `TOKENLESS_STASH_DB`) is validated by canonicalizing both the
+home anchor and the candidate and requiring the candidate to live under the
+home; a path outside home is rejected. This mirrors the stats DB trust model
+exactly, so an attacker cannot redirect the stash to a system-critical
+location.
+
+`retrieve` queries are parameterized SQL; a malformed hash simply yields "no
+payload" rather than an injection.
+
+## Fail-open policy
+
+- **Compress path**: if the stash cannot be opened (no trusted home, directory
+  cannot be created, db open fails) or `stash()` errors, compression proceeds
+  without stash and the marker degrades to the plain
+  `<... N more items truncated>` form. Compression never fails because of the
+  stash.
+- **Retrieve path**: retrieve is user-initiated, so failures surface as
+  errors (exit 1) rather than being swallowed.
+
+## What is not (yet) stashed
+
+- **String truncation**: long string values are truncated with a `… (truncated)`
+  marker but the tail is not stashed. The stash marker (~65 chars) against
+  small per-field limits would be proportionally large overhead; the
+  high-value case is array truncation, which is covered.
+- **Schema description truncation**: `SchemaCompressor::truncate_description`
+  remains lossy for the same marker-overhead reason.
+- **MCP `tokenless_retrieve`**: not yet implemented; retrieval is via the CLI
+  today. MCP integration is tracked separately.
+
+## Mapping to Headroom CCR
+
+| Headroom | Tokenless | Notes |
+|---|---|---|
+| CCR Store | stash store (`StashStore` trait) | InMemory / SQLite(WAL) / Redis* |
+| `<<ccr:HASH>>` | `<<tokenless:HASH>>` | 24-hex BLAKE3, same key length |
+| `headroom_retrieve` (MCP) | `tokenless retrieve` (CLI) | MCP tool pending |
+| DashMap `remove_if` TOCTOU fix | single-writer `Mutex<Connection>` | SQLite path |
+| default TTL 5 min / cap 1000 | InMemory 5 min / 1000; SQLite 1 h / 10 000 | tuned for hook process model |
+
+\* Redis backend is not yet implemented; it is tracked for the
+multi-worker case (no `cfg`-gated scaffolding exists yet).