# PR: Arrow 57 / DataFusion 51 / Lance 2 + BlasGraph Algebra + SPO Triple Store#151
Closed
AdaWorldAPI wants to merge 61 commits intolance-format:mainfrom
Closed
# PR: Arrow 57 / DataFusion 51 / Lance 2 + BlasGraph Algebra + SPO Triple Store#151AdaWorldAPI wants to merge 61 commits intolance-format:mainfrom
AdaWorldAPI wants to merge 61 commits intolance-format:mainfrom
Conversation
Align lance-graph's dependency matrix with ladybug-rs and rustynum: arrow 56.2 → 57 datafusion 50.3 → 51 lance 1.0 → 2.0 lance-* 1.0 → 2.0 All 491 tests pass with zero API breakages. The Python crate is excluded from the workspace resolver to avoid the pyarrow `links = "python"` conflict with pyo3. It continues to build separately via `maturin develop`. https://claude.ai/code/session_016SeGMg1pgf1MqK8YWkedvV
…g traversal + 7 ground truth tests Implements the full SPO (Subject-Predicate-Object) graph primitives stack: - graph/fingerprint.rs: label_fp() with 11% density guard, dn_hash(), hamming_distance() - graph/sparse.rs: Bitmap [u64;BITMAP_WORDS] (fixes old [u64;2] truncation), pack_axes() - graph/spo/truth.rs: TruthValue (NARS frequency/confidence), TruthGate (OPEN/WEAK/NORMAL/STRONG/CERTAIN) - graph/spo/builder.rs: SpoBuilder with forward/reverse/relation query vector construction - graph/spo/store.rs: SpoStore with 2^3 projection verbs (SxP2O, PxO2S, SxO2P), gated queries, semiring chain walk - graph/spo/semiring.rs: HammingMin semiring (min-plus over Hamming distance) - graph/spo/merkle.rs: MerkleRoot, ClamPath, BindSpace with verify_lineage (known gap documented) and verify_integrity - graph/mod.rs: ContainerGeometry enum with Spo=6 Ground truth integration tests (7/7 pass): 1. SPO hydration round-trip (insert + forward/reverse query) 2. 2^3 projection verbs consistency (all three agree on same triple) 3. TruthGate filtering (OPEN=2, STRONG=1, CERTAIN=0 for test data) 4. Belichtung prefilter rejection rate (<10 hits from 100 edges) 5. Semiring chain traversal (3 hops with increasing cumulative distance) 6. ClamPath+MerkleRoot integrity (documents verify_lineage no-op gap) 7. Cypher vs projection verb convergence (SPO side validated) 31 unit tests + 7 integration tests, all passing. Clippy clean. https://claude.ai/code/session_016SeGMg1pgf1MqK8YWkedvV
…ests BlasGraph module: GraphBLAS-style sparse matrix algebra over hyperdimensional 16384-bit binary vectors with 7 semiring types. Uses SplitMix64 PRNG. 10 SPO redisgraph parity integration tests. All 87 blasgraph + 10 parity tests pass under stable and miri. https://claude.ai/code/session_01Mcj8GxEtzmVba6RmuT7AjD
- Bump deltalake 0.29 → 0.30 (datafusion ^51.0 compatible) - Fix cargo fmt: sort mod/use declarations in blasgraph/mod.rs - Use workspace exclude for lance-graph-python (links=python conflict) - Auto-format spo/store.rs, spo/merkle.rs, spo_ground_truth.rs https://claude.ai/code/session_01Mcj8GxEtzmVba6RmuT7AjD
Claude/setup adaworld repos 4k pex
…0.26 The arrow 57 upgrade caused a `links = "python"` conflict because arrow-pyarrow 57 requires pyo3 0.26 while lance-graph-python used pyo3 0.25 — two different pyo3-ffi versions in the same workspace. Fix: bump pyo3 to 0.26 so all crates share one pyo3-ffi. Update deprecated pyo3 0.26 APIs (with_gil→attach, allow_threads→detach, PyObject→Py<PyAny>). Pyarrow zero-copy stays intact. - Cargo.toml: restore lance-graph-python to members (remove exclude) - lance-graph-python/Cargo.toml: pyo3 0.25 → 0.26 - executor.rs: update pyo3 0.26 API renames - graph.rs: PyObject → Py<PyAny> All crates: clippy clean, 423 tests pass. https://claude.ai/code/session_01Mcj8GxEtzmVba6RmuT7AjD
- style.yml: add clippy step for lance-graph-python - build.yml: add cargo check for lance-graph-python https://claude.ai/code/session_01Mcj8GxEtzmVba6RmuT7AjD
Covers all 3 major additions: version upgrades, BlasGraph semiring algebra, and SPO triple store — with architecture details, API signatures, test matrix, and module dependency graph. https://claude.ai/code/session_01Mcj8GxEtzmVba6RmuT7AjD
Arrow 57 / DataFusion 51 / Lance 2 + BlasGraph Algebra + SPO Triple Store
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c41fd718c2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+194
to
+196
| if let HdrScalar::Vector(v) = acc { | ||
| if !v.is_zero() { | ||
| result.set(i, v); |
There was a problem hiding this comment.
mxv only writes results when the accumulator is HdrScalar::Vector, so semirings that produce numeric values (e.g. HammingMin and SimilarityMax) are silently dropped at the end of each row. In practice this makes mxv/vxm return empty outputs for those semirings, which breaks the advertised shortest-path/similarity workflows that depend on scalar distances.
Useful? React with 👍 / 👎.
Author
There was a problem hiding this comment.
please re-review after resolve
Comment on lines
+232
to
+236
| // The semiring produces Float scalars for Hamming, but here | ||
| // we are working with BitVec elements; keep the "better" one | ||
| // (closer to zero). | ||
| if v.hamming_distance(&BitVec::zero()) | ||
| < existing.hamming_distance(&BitVec::zero()) |
There was a problem hiding this comment.
The SSSP update compares candidates by hamming_distance to the zero vector, which is just vector popcount and not the semiring path cost; this can choose a “denser” bit pattern over a genuinely shorter path. Since hdr_sssp is documented as Bellman-Ford-like shortest path over Hamming costs, this relaxation criterion yields incorrect rankings whenever popcount and path distance diverge.
Useful? React with 👍 / 👎.
Author
There was a problem hiding this comment.
please re-review after resolve
Comment on lines
+213
to
+214
| match &best_hit { | ||
| Some(existing) if d >= existing.distance => {} |
There was a problem hiding this comment.
When two candidate edges have the same distance (a common case when both subjects match exactly), the d >= existing.distance guard keeps whichever record was seen first in HashMap iteration. Because HashMap iteration order is randomized per process, identical data can produce different traversal paths across runs, which undermines reproducibility for experiments and downstream reasoning.
Useful? React with 👍 / 👎.
Author
There was a problem hiding this comment.
please re-review after resolve
…type. Delete 512-bit toy.
…/scalar), corrected cycle counts per platform
…terministic chain traversal Three correctness fixes flagged in PR review: 1. HammingMin/SimilarityMax semirings now produce Vector(XOR) instead of Float(distance). The distance is a separate u32 computed by the caller via popcount. This eliminates the mxv silent-drop bug — all semiring outputs are now Vector and flow through mxv/mxm naturally. 2. SSSP rewritten as proper Bellman-Ford with cumulative u32 path costs tracked alongside XOR-composed path vectors. Edge weight = popcount of edge BitVec. Costs stored in GrBVector scalar side-channel. The old code compared popcount-to-zero (bit density) which is not path cost. 3. Chain traversal tie-breaking in SpoStore::walk_chain_forward is now deterministic: when two candidates have equal Hamming distance, the smallest key wins (instead of depending on HashMap iteration order). Additional: GrBVector gains a scalar side-channel (set_scalar/get_scalar) for algorithms that need to annotate vector entries with numeric metadata. MonoidOp::MinPopcount added for min-Hamming-weight accumulation. All 430 tests pass. Clippy clean. https://claude.ai/code/session_01Mcj8GxEtzmVba6RmuT7AjD
Three benchmark tests that prove the core claims with numbers: 1. float_vs_hamming_sssp_equivalence — 100% pairwise ranking agreement between float Bellman-Ford and Hamming SSSP on a 1000-node random graph (490K comparisons). Prints speedup ratio. 2. belichtungsmesser_rejection_rate — 3-stage Hamming sampling cascade rejects 99.7% at stage 1 (1/16 sample), saves 93.5% compute vs full scan. 20 planted near-vectors all survive to stage 3. 3. float_cosine_vs_bf16_hamming_ranking — SimHash encoding preserves 8/10 top-k results vs float cosine similarity on 1000 128-dim vectors (16384-bit SimHash, well above the 7/10 threshold). These run in CI on every commit. The numbers do the selling. https://claude.ai/code/session_01Mcj8GxEtzmVba6RmuT7AjD
Self-calibrating integer-only Hamming distance cascade that eliminates 94%+ of candidates using sampled bit comparisons (1/16 → 1/4 → full). Key components: - isqrt: integer Newton's method, no float - Band classification: Foveal/Near/Good/Weak/Reject sigma bands - Cascade query: sampling-aware thresholds (μ-4σ for 1/16, μ-2σ for 1/4) - Welford's online shift detection with integer arithmetic - 7 passing tests with timing/ns measurements CI output (16384-bit vectors, 10K random candidates): Stage 1: 83% rejected, Stage 2: 94% combined rejection Brute force: 1784 ns/candidate, Cascade: 455 ns/candidate → 3.9x speedup Work savings: 83% fewer word-ops https://claude.ai/code/session_01Mcj8GxEtzmVba6RmuT7AjD
…sted) The cascade thresholds must match the confidence level the Stichprobe (sample size) can actually support: Stage 1 (1/16 sample, 1024 bits): bands[2] = μ-σ → 1σ confidence Stage 2 (1/4 sample, 4096 bits): bands[1] = μ-2σ → 2σ confidence Stage 3 (full, 16384 bits): exact classification into all bands The previous μ-4σ threshold with a 1/16 sample claimed a confidence level the sample size cannot deliver — 4σ requires a much larger Stichprobe. With only 16 words of data, the top-k survivors were random candidates that got lucky on sampling noise, not real matches. Removed cascade_s1/cascade_s2 fields. Cascade now uses bands[] directly, matching the design doc exactly. https://claude.ai/code/session_01Mcj8GxEtzmVba6RmuT7AjD
The cascade now precomputes thresholds at [1σ, 1.5σ, 2σ, 2.5σ, 3σ] from calibrated (warmup) σ. Stage 1 and stage 2 select from this table via stage1_level/stage2_level, allowing dynamic tightening as σ stabilises from observed data. cascade_at(quarter_sigmas) provides arbitrary quarter-sigma granularity (1.75σ, 2.25σ, 2.75σ) for fine-grained adjustment. The σ confidence must match what the Stichprobe supports: 1/16 sample → 1σ (stage1_level=0) 1/4 sample → 2σ (stage2_level=2) full → exact classification After warmup (calibrate), thresholds reflect observed σ. After shift detection (recalibrate), cascade table updates while stage level selections are preserved. 8 tests (added test_cascade_warmup_and_levels). https://claude.ai/code/session_01Mcj8GxEtzmVba6RmuT7AjD
Cascade table now has 8 entries at quarter-sigma intervals: [μ-1σ, μ-1.5σ, μ-1.75σ, μ-2σ, μ-2.25σ, μ-2.5σ, μ-2.75σ, μ-3σ] New test_warmup_2k_then_shift_10k: - Phase 1: Warmup with 2016 pairwise distances, sweep all 8 cascade levels showing rejection rate at each (59%→76% with theoretical σ) - Phase 2: Feed 10000 observations from shifted distribution (μ→7800), Welford detects shift, recalibrate, re-sweep showing the warmed-up cascade achieving 95.7%→100% rejection across levels The warmup is what makes the cascade work. Before calibration, theoretical σ produces mediocre rejection. After warmup, the confidence intervals are backed by observed data and the cascade eliminates 95%+ at 1σ alone. 9 tests, all passing. https://claude.ai/code/session_01Mcj8GxEtzmVba6RmuT7AjD
Print one-sided normal distribution expected rejection rates alongside actual rates at each cascade level. Makes the Stichprobe confidence gap visible: Pre-warmup (1/16 sample, σ=64): 59-76% actual vs 84-99.9% expected Post-shift (1/16 sample, σ=199): 95-100% actual vs 84-99.9% expected The post-shift over-rejection reveals the normal distribution assumption breaks when Welford's σ is inflated from mixing two distributions. https://claude.ai/code/session_01Mcj8GxEtzmVba6RmuT7AjD
feat: add Belichtungsmesser HDR popcount-stacking early-exit cascade
Structural changes: - Add ReservoirSample (Vitter's Algorithm R) for distribution-free quantile estimation. Works for any distribution shape. - Add empirical_bands/empirical_cascade computed from reservoir percentiles - Add auto-switch: when skewness/kurtosis indicate non-normality, band() and cascade_query() use empirical thresholds automatically - recalibrate() now resets Welford counters AND reservoir (fresh start) - Rename Belichtungsmesser → LightMeter, module → light_meter Confidence vs theory (12 tests, all pass): - Full-width rejection: Δ < 0.2% from normal distribution theory - 1/16 sample: matches predicted Z=k/4 variance inflation - 1/4 sample: matches predicted Z=k/2 variance inflation - Across 3 distribution shifts: average Δ = 0.17% - Bimodal detection: auto-switches to empirical (kurt=99 < 200 threshold) https://claude.ai/code/session_01Mcj8GxEtzmVba6RmuT7AjD
feat: integrate ReservoirSample + rename Belichtungsmesser → LightMeter
…de API, 6 algorithm synergies
- git mv light_meter.rs → hdr.rs - mod.rs: pub mod light_meter → pub mod hdr - LightMeter → Cascade (all 21 occurrences) - cascade_query() → query() - Add expose() and test_distance() thin wrappers - Update hdr_proof.rs references - Fix clippy: add is_empty(), use is_multiple_of() 435 tests pass, clippy clean. https://claude.ai/code/session_01Mcj8GxEtzmVba6RmuT7AjD
…kPEX rename LightMeter → hdr::Cascade (SESSION_B_HDR_RENAME)
- Add is_empty() to ReservoirSample (len_without_is_empty) - Use is_multiple_of() instead of % == 0 (manual_is_multiple_of) - Use iterators in f32_to_bitvec_simhash (needless_range_loop) - Remove unused dim variable - Allow needless_range_loop in test module (cascade sweeps index into multiple parallel arrays by design) 435 lib tests pass. clippy --tests -D warnings clean. https://claude.ai/code/session_01Mcj8GxEtzmVba6RmuT7AjD
…ascade=attention, hub saturation, fast seal, learned thresholds
…Ford as TD learning, ReLU=tropical=alpha, dictionary encoding, MATLAB toolbox
…DuckDB, PyTorch, LangChain — what we get for free
…ext + LanceSessionStorage
…pathfinding — the semantic revolution
…n8n+LangStudio into ndarray+lance-graph+rs-graph-llm
- Add Hamming variant to DistanceMetric, parser, and lance_vector_search - Add hamming_distance/similarity UDFs for FixedSizeBinary(2048) columns - Add binary vector extraction (FixedSizeBinaryArray) to vector_ops - Create ndarray_bridge.rs: BitVec↔Fingerprint zero-copy bridges with 4-tier SIMD dispatch (VPOPCNTDQ → AVX-512BW → AVX2 → scalar) - Create columnar.rs: Lance Arrow schemas for nodes (3 planes), edges (NARS truth), and stroke-packed fingerprints for cascade - Create cascade_ops.rs: CascadeScanConfig, hamming predicate → cascade translation, selectivity estimation - Wire semiring HammingMin/SimilarityMax to SIMD-dispatched popcount - Add BitVec::as_bytes/from_bytes for Arrow FixedSizeBinary interop - Produce .claude/FALKORDB_ANALYSIS.md: comprehensive FalkorDB architecture analysis (GraphBLAS pipeline, delta matrices, property storage) - All 720 tests pass, clippy clean https://claude.ai/code/session_01Dg6MsYU71FitYV2bB59bE3
…e pushdown - VersionedGraph: commit_encounter_round, at_version, diff, tag, graph_seal_check backed by three Lance datasets (nodes, edges, fingerprints) with ACID snapshots - GraphSealStatus: Wisdom (stable) vs Staunen (diverged) across versions - GraphDiff: new_nodes, modified_nodes, new_edges between version pairs - Storage backends: local/s3/azure/gcs via URI-based constructors - Cost estimation: band-based selectivity (Foveal 0.1% → Reject 100%) - Predicate pushdown: HammingPredicate detection, PushdownAnalysis with cascade vs full-scan strategy selection - ScanStrategy: automatic cascade selection when selectivity < 5% and rows > 1000 - Fix Python bindings for Hamming DistanceMetric variant 757 tests passing, clippy clean. https://claude.ai/code/session_01Dg6MsYU71FitYV2bB59bE3
…ntegration-TIkjC Claude/lance graph
5 documents mapping every Python LangGraph module, class, and function to Rust equivalents in rs-graph-llm/graph-flow: - LANGGRAPH_FULL_INVENTORY.md: 132 Python items mapped (46 done, 86 missing, 35% coverage) - LANGGRAPH_PARITY_CHECKLIST.md: prioritized gap analysis (P0-P3) - LANGGRAPH_CRATE_STRUCTURE.md: recommended crate layout with Python → Rust module mapping - LANGGRAPH_TRANSCODING_MAP.md: side-by-side code examples for every pattern - LANGGRAPH_OUR_ADDITIONS.md: 13 features we have that Python LangGraph doesn't https://claude.ai/code/session_01AKkBDoAf2Wrsir2o9vpVzn
…XRadt docs: add transcode inventory — Python LangGraph → Rust mapping
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…XRadt feat(graph): add cold-path MetadataStore for nodes/edges with Cypher query via DataFusion https://claude.ai/code/session_01AKkBDoAf2Wrsir2o9vpVzn
HOT_COLD_PATH_ARCHITECTURE.md documents: - 4-tier SIMD hamming dispatch (AVX-512 VPOPCNTDQ → scalar) - HDR Cascade 3-stage filter with sigma band classification - Cold path MetadataStore skeleton and calibration lifecycle DOCUMENTATION_DRIFT_AUDIT.md identifies documentation drift across ladybug-rs and lance-graph with severity ratings and git commit dates. https://claude.ai/code/session_016wjHu3AsaTCdHfGXEkEMvk
…ma-pVbYP docs: add hot/cold path architecture and documentation drift audit
… cascade Implement the primary neighborhood vector search system for lance-graph. ZeckF64 encodes SPO triple distances as progressive 8-byte values (94% precision from byte 0 alone). The 4-stage cascade explores ~200K nodes in 3 hops loading only ~1.2MB from disk. New modules: - zeckf64: 8-byte progressive edge encoding with lattice-legal scent byte - neighborhood: scope-based neighborhood vectors (10K nodes/scope) - heel_hip_twig_leaf: 4-stage search cascade (Heel→Hip→Twig→Leaf) - lance_neighborhood: Arrow schemas for Lance persistence - neighborhood_csr: CSR bridge for graph algorithms (secondary path) - clam_neighborhood: CLAM ball-tree for Pareto convergence conjecture test 38 new tests, 555 total tests passing, 0 warnings. https://claude.ai/code/session_01CdqyUTUfjKZuk8YGJzv6LB
Implements the primary search path for lance-graph using progressive 8-byte edge encodings and 3-hop neighborhood vector traversal. ZeckF64 encoding: byte 0 = 7 SPO band classifications (boolean lattice, 19 legal patterns, ~85% error detection), bytes 1-7 = distance quantiles. ScopeBuilder: O(N²) pairwise construction of [ZeckF64; N] vectors. SearchCascade: HEEL (1 vec) → HIP (50 vecs) → TWIG (50 vecs) → LEAF. 32 tests (22 unit + 10 integration), all passing. https://claude.ai/code/session_01NUMNX67KZrFiTQK7erFQuH
feat(blasgraph): add ZeckF64 neighborhood search — Heel/Hip/Twig/Leaf…
…ry-Op9kK feat(graph): add ZeckF64 neighborhood vector search (Heel/Hip/Twig/Leaf)
Migrate three unique modules from blasgraph/ to neighborhood/ (additive only): - clam.rs: CLAM ball-tree partitioning for Pareto convergence validation - storage.rs: Lance Arrow schemas + serialization for scopes/neighborhoods - sparse.rs: CSR bridge for graph algorithms (BFS, PageRank, spmv) Also adds zeckf64_scent_hamming_distance() variant from blasgraph implementation (popcount-based alternative to the L1 scent distance). All 54 tests pass (44 unit + 10 integration). No existing code modified. https://claude.ai/code/session_01NUMNX67KZrFiTQK7erFQuH
…ry-Op9kK feat(neighborhood): consolidate blasgraph modules into neighborhood/
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR: Arrow 57 / DataFusion 51 / Lance 2 + BlasGraph Algebra + SPO Triple Store
Title
feat: arrow 57, datafusion 51, lance 2 + BlasGraph semiring algebra + SPO triple storeBody
Follows up on #146 (closed — split was planned but the pieces are interdependent, so shipping as one clean PR rebased on current main).
Summary
Three additions in one PR because they share the dependency bump and build on each other:
All 146 new tests pass (423 total lib tests, up from 295). Clippy clean across all crates. No breaking changes to existing APIs.
1. Dependency Upgrades
API adaptations required:
Schema::from(df.schema())→df.schema().as_arrow().clone()with_gil→attach,allow_threads→detach,PyObject→Py<PyAny>All existing tests pass without modification after these changes.
2. BlasGraph — Semiring Algebra for Graph Computation
crates/lance-graph/src/graph/blasgraph/— 8 modules, 3,173 lines, 87 unit tests.GraphBLAS defines graph algorithms as sparse linear algebra. Instead of vertex-centric message passing (Pregel), graph operations become matrix multiplications parameterized by semirings. This enables expressing BFS, shortest path, PageRank, and similarity search as
mxm(A, A, semiring)calls.This implementation operates on 16,384-bit hyperdimensional binary vectors rather than scalar weights, making it suitable for lance-graph's fingerprint-based graph storage.
Type System (
types.rs)BitVec— 16,384-bit HD vector (256 × u64) with XOR bind, AND/OR/NOT, majority-vote bundle, cyclic permute, Hamming distance, and density operationsHdrScalar— tagged union wrappingBitVec,f32,bool, or empty for semiring generalityUnaryOp(5),BinaryOp(14),MonoidOp(10),SelectOp(5)Seven Semirings (
semiring.rs)XorBundleBindFirstHammingMinSimilarityMaxResonanceBooleanXorFieldSparse Storage (
sparse.rs)CooStorage— coordinate format for incremental constructionCsrStorage— compressed sparse row for efficient row-major iterationSparseVec— sorted sparse vector with O(log n) lookupCooStorage → CsrStorageviato_csr()Matrix Operations (
matrix.rs)GrBMatrix— CSR-backed sparse matrix parameterized by any semiringmxm(A, B, semiring)— matrix-matrix multiply (graph composition)mxv(A, v, semiring)— matrix-vector multiply (one-hop query)vxm(v, A, semiring)— vector-matrix multiply (reverse query)ewise_add,ewise_mult— element-wise union and intersectionextract,apply,reduce_rows,reduce_cols,transposeVector Operations (
vector.rs)GrBVector— sorted sparse vectorfind_nearest(query, k)— k-nearest by Hamming distancefind_within(query, radius)— range searchfind_most_similar(query)— single best matchDescriptors (
descriptor.rs)Descriptor— operation modifiers: transpose inputs, complement masks, replace semanticsdefault,t0,t1,t0t1,comp,replace,replace_comp,structureGraph Algorithms (
ops.rs)Three reference implementations demonstrating the semiring approach:
hdr_bfs(adj, source, max_depth)— level-synchronous BFS viaBindFirstsemiringhdr_sssp(adj, source, max_iters)— Bellman-Ford SSSP viaHammingMinsemiringhdr_pagerank(adj, max_iters, damping)— iterative PageRank viaXorBundlesemiringThese operate on bit vectors rather than floats, making them compatible with lance-graph's fingerprint-based storage without type conversion.
3. SPO Triple Store
crates/lance-graph/src/graph/spo/— 6 modules, 1,443 lines, 30 unit tests + 22 primitive tests + 7 integration tests.A content-addressable triple store that encodes Subject-Predicate-Object relationships as bitmap fingerprints for fast approximate nearest-neighbor lookup. Designed to sit beneath the Cypher query engine, providing direct graph operations without SQL round-trips.
Fingerprints (
fingerprint.rs)Fingerprint = [u64; 8](512 bits) with FNV-1a hashingBitmap Search (
sparse.rs)Bitmap = [u64; 8]matching fingerprint widthpack_axes(s, p, o)— OR-compose S+P+O for search vector constructionNARS Truth Values (
truth.rs)TruthValue { frequency, confidence }— evidence-weighted beliefrevision(other)— combines independent evidence sourcesTruthGate— 5 presets (OPEN/WEAK/NORMAL/STRONG/CERTAIN) for confidence-gated queriesStore (
store.rs)SpoStore— in-memory triple store with bitmap ANN searchquery_forward(s, p, radius)— S×P→O ("what does Alice love?")query_reverse(p, o, radius)— P×O→S ("who loves Bob?")query_relation(s, o, radius)— S×O→P ("how is Alice related to Bob?")query_forward_gated(s, p, radius, gate)— truth-gated variant, filters low-confidence results before distance computationwalk_chain_forward(start, radius, max_hops)— multi-hop traversal usingHammingMinsemiring with cumulative distance trackingMerkle Integrity (
merkle.rs)MerkleRoot— XOR-fold hash stamped at write timeClamPath— hierarchical path addressing with depth trackingverify_integrity()— full re-hash comparison detects corruptionverify_lineage()— structural check for path consistencyIntegration Tests (
spo_ground_truth.rs, 355 lines)spo_hydration_round_tripprojection_verbs_consistencytruth_gate_filters_low_confidencebelichtung_rejection_ratesemiring_walk_chainclam_merkle_integritycypher_vs_projection_convergenceWhat This Enables
With these three additions, lance-graph gains:
These compose naturally: the SPO store uses the
HammingMinsemiring from BlasGraph for chain traversal, and the bitmap fingerprints share the same bit-vector primitives as BlasGraph'sBitVectype.Migration Notes
graphmodule is additive — accessed vialance_graph::graph::{blasgraph, spo}lance-graph-pythoncrateStats