Migrate rustynum → ndarray HPC modules (Plateau 2, Phase 2A)#175
Migrate rustynum → ndarray HPC modules (Plateau 2, Phase 2A)#175AdaWorldAPI merged 6 commits intomainfrom
Conversation
Cross-repo reference document covering module map, BLAS parity, quantized GEMM verification, NaN guard audit, and migration roadmap. https://claude.ai/code/session_01CdqyUTUfjKZuk8YGJzv6LB
Document current state: 0/25 tasks completed, SPO/Merkle/16K done, awareness loop not started, server still on CogRedis. https://claude.ai/code/session_01CdqyUTUfjKZuk8YGJzv6LB
…tasks CLAUDE.md section 8 documents rustynum→ndarray type mappings and migration order. Surgery blackboard now references integration plan pre-conditions and dependencies. Created PROGRESS.md tracking Phases 2A/2B/2C against master plan. https://claude.ai/code/session_01CdqyUTUfjKZuk8YGJzv6LB
…cies
Complete migration of ladybug-rs from 6 rustynum crates to ndarray (single dep).
Cargo.toml: removed rustynum-rs, rustynum-core, rustynum-bnn, rustynum-arrow,
rustynum-holo, rustynum-clam. Added ndarray (AdaWorldAPI fork).
Import swaps (11 files):
- rustynum_core::simd::{popcount, hamming_distance, dot_i8}
→ ndarray::hpc::bitwise::{popcount_raw, hamming_distance_raw} + simd_avx2::dot_i8
- rustynum_core::{CollapseGate, SigmaGate, SignificanceLevel}
→ ndarray::hpc::bnn_cross_plane::CollapseGate + kernels::{SigmaGate, SignificanceLevel}
- rustynum_bnn::causal_trajectory::* → ndarray::hpc::bnn_causal_trajectory::*
- rustynum_bnn::{CrossPlaneVote, HaloType, GrowthPath, InferenceMode, MutationOp}
→ ndarray::hpc::bnn_cross_plane::*
- rustynum_rs::NumArrayU8::try_bundle_byte_slices
→ ndarray::Array::<u8, Ix1>::hdc_bundle_byte_slices
- rustynum_holo::* → ndarray::hpc::holo::*
- rustynum_clam::* → ndarray::hpc::clam::*
All 11 pre-existing warnings unchanged. Zero new errors.
https://claude.ai/code/session_01CdqyUTUfjKZuk8YGJzv6LB
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 261c23dc13
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| rustynum-arrow = { path = "../rustynum/rustynum-arrow", default-features = false, features = ["arrow"] } | ||
| rustynum-holo = { path = "../rustynum/rustynum-holo", features = ["avx512"] } | ||
| rustynum-clam = { path = "../rustynum/rustynum-clam", features = ["avx512"] } | ||
| ndarray = { path = "../ndarray" } |
There was a problem hiding this comment.
Switching Cargo to ndarray = { path = "../ndarray" } breaks every automated build path that still provisions only ../rustynum//rustynum. For example, .github/workflows/ci-master.yml:33-39, .github/workflows/proof.yml:60-66, and Dockerfile.release:57-68 all clone rustynum but never create the sibling path Cargo now resolves, so CI and release-image builds will fail before compilation with a missing path dependency.
Useful? React with 👍 / 👎.
| #[inline] | ||
| pub fn fingerprint_dot_i8(a: &Fingerprint, b: &Fingerprint) -> i64 { | ||
| rustynum_core::simd::dot_i8( | ||
| ndarray::simd_avx2::dot_i8( |
There was a problem hiding this comment.
dot_i8
Unlike rustynum_core::simd::dot_i8, ndarray::simd_avx2::dot_i8 is an AVX2-specific kernel, not a dispatcher—the upstream implementation executes _mm256_* intrinsics directly. Calling it here means fingerprint_dot_i8, container_dot_i8, and slice_dot_i8 will SIGILL on x86_64 hosts that lack AVX2, even though this module still advertises runtime dispatch for any x86_64 CPU.
Useful? React with 👍 / 👎.
| #[inline] | ||
| pub fn hamming_bytes(a: &[u8], b: &[u8]) -> u32 { | ||
| rustynum_core::simd::hamming_distance(a, b) as u32 | ||
| ndarray::hpc::bitwise::hamming_distance_raw(a, b) as u32 |
There was a problem hiding this comment.
This switches the UDF helper from rustynum_core::simd::hamming_distance, which asserted equal slice lengths, to ndarray::hpc::bitwise::hamming_distance_raw, whose upstream implementation only compares min(a.len(), b.len()). The BinaryArray/LargeBinaryArray paths below never validate lengths, so hamming() and similarity() will silently undercount whenever users compare different-sized binary values instead of failing fast.
Useful? React with 👍 / 👎.
rustynum is no longer a dependency — ndarray replaced it entirely. Updated ci-master.yml (4 jobs) and proof.yml (6 jobs) to clone AdaWorldAPI/ndarray instead of rustynum. Removed vendor/rustynum submodule init steps. https://claude.ai/code/session_01CdqyUTUfjKZuk8YGJzv6LB
- rust-toolchain.toml: 1.93.1 → 1.94.0 - Cargo.toml + ladybug-contract: rust-version = "1.94" - ci-master.yml: toolchain 1.94.0, removed Miri job (requires nightly) - proof.yml: toolchain 1.94.0 - All 5 Dockerfiles: rust:1.93-slim-bookworm → rust:1.94-slim-bookworm SIMD portable_simd stabilization requires 1.94+. https://claude.ai/code/session_01CdqyUTUfjKZuk8YGJzv6LB
2 participants
Summary
This PR completes Phase 2A of the master integration plan: replacing rustynum-rs with AdaWorldAPI/ndarray as the SIMD compute substrate. All source code imports have been updated to use ndarray's 55 HPC modules instead of the 6 rustynum crates.
Key Changes
Core SIMD Operations
rustynum_core::simd::hamming_distance→ndarray::hpc::bitwise::hamming_distance_rawrustynum_core::simd::popcount→ndarray::hpc::bitwise::popcount_rawrustynum_core::simd::dot_i8→ndarray::simd_avx2::dot_i8rustynum_rs::NumArrayU8::try_bundle_byte_slices→ndarray::hpc::hdc::HdcOps::hdc_bundle_byte_slicesType Migrations
rustynum_core::SigmaGate→ndarray::hpc::kernels::SigmaGaterustynum_core::SignificanceLevel→ndarray::hpc::kernels::SignificanceLevelrustynum_core::CollapseGate→ndarray::hpc::bnn_cross_plane::CollapseGaterustynum_bnn::causal_trajectory::NarsTruth→ndarray::hpc::bnn_causal_trajectory::NarsTruthrustynum_clam::tree::ClamTree→ndarray::hpc::clam::ClamTreeModule Reorganization
src/core/rustynum_accel.rsupdated to import from ndarray instead of rustynumsrc/spo/gestalt.rs: 25 type path updates across BundlingProposal, PlaneCalibration, EvidenceEventsrc/spo/spo_harvest.rs: Updated BNN cross-plane and causal trajectory importssrc/storage/bind_space.rs: Migrated HDC bundle and Hamming distance operationssrc/search/hdr_cascade.rs: Updated SigmaGate and SignificanceLevel referencessrc/graph/spo/store.rs: Migrated ClamTree and search primitivessrc/query/cognitive_udfs.rs: Updated SIMD hamming distance callsDocumentation & Planning
MIGRATION_INVENTORY.md: Comprehensive audit of 57K LOC rustynum → 36K LOC ndarray mapping, including BLAS parity gaps (15/23 routines), quantized GEMM verification, NaN guard audit, and priority recommendationsPROGRESS.md: Plateau-based progress tracker for integration planCLAUDE.md: Migration context and Plateau 2 sequencing.claude/SURGERY_BLACKBOARD.md: Integration plan dependencies for brain surgery tasksCargo.toml: Feature flagrustynum→ndarray-hpc, dependency comments updatedImplementation Details
ndarray::hpc::*) for clarity during transitionview_u64_as_bytes) preserved from rustynumBlockers & Next Steps
Unblocked by this PR:
https://claude.ai/code/session_01CdqyUTUfjKZuk8YGJzv6LB