Initial reimplementation of composefs-c#225
Draft
cgwalters wants to merge 14 commits intocomposefs:mainfrom
Draft
Initial reimplementation of composefs-c#225cgwalters wants to merge 14 commits intocomposefs:mainfrom
cgwalters wants to merge 14 commits intocomposefs:mainfrom
Conversation
Collaborator
Author
|
There's definitely some sub-tasks to this and pieces that we need to break out. One that I'm realizing is that the dumpfile format is hardcoded to sha256-12. I guess we can just auto-detect from length (like we're doing in other places) but the more I think about this the more I feel we need to formalize it (as is argued in #224 ) So how about a magic comment in the dumpfile like or so? |
cgwalters
force-pushed
the
composefs-c-compat
branch
2 times, most recently
from
4d43b61 to
1871128
Compare
Collaborator
Author
|
Let's make the format layout a choice to avoid breaking sealed UKIs as is today |
cgwalters
force-pushed
the
composefs-c-compat
branch
3 times, most recently
from
8a5c48d to
9cb1923
Compare
This was referenced Mar 11, 2026
cgwalters
force-pushed
the
composefs-c-compat
branch
4 times, most recently
from
6eda766 to
dc1fed7
Compare
This was referenced Mar 17, 2026
cgwalters
force-pushed
the
composefs-c-compat
branch
from
dc1fed7 to
9a845fa
Compare
cgwalters
force-pushed
the
composefs-c-compat
branch
from
9a845fa to
9823c67
Compare
cgwalters
force-pushed
the
composefs-c-compat
branch
4 times, most recently
from
a8d6802 to
25cbbb1
Compare
8 tasks
Assisted-by: OpenCode (Claude Sonnet 4.6) Signed-off-by: Colin Walters <walters@verbum.org>
Replace the secret-token-based publish workflow with OIDC trusted publishing (rust-lang/crates-io-auth-action), matching the pattern used across other bootc-dev Rust crates (ocidir-rs, containers-image-proxy-rs). Triggers on v* tag push instead of manual workflow_dispatch. Also adds composefs-ioctls to the publish list (published first since other crates depend on it) and removes the now-unnecessary 'crates.io' environment and CARGO_REGISTRY_TOKEN secret. Assisted-by: OpenCode (Claude Sonnet 4.6) Signed-off-by: Colin Walters <walters@verbum.org>
With cargo publish --workspace used in the publish workflow, any crate not intended for crates.io must be explicitly opt out. splitfdstream is internal-only; erofs-debug and integration-tests were already marked. Signed-off-by: Colin Walters <walters@verbum.org>
All workspace crates now share the composefs- namespace for consistency. The cfsctl binary name is unchanged (users still run `cfsctl`). Assisted-by: OpenCode (Claude Sonnet 4.6) Signed-off-by: Colin Walters <walters@verbum.org>
Add a FormatVersion enum (V1/V2) that controls the EROFS image format: V1 produces byte-identical output to C mkcomposefs. It sets composefs_version=0 in the superblock, uses compact inodes where possible, BFS inode ordering, C-compatible xattr sorting, and includes overlay whiteout character device entries in the root directory. The build_time is set to the minimum mtime across all inodes, matching the C implementation. V2 remains the default (composefs_version=2). It uses extended inodes, DFS ordering, and the composefs-rs native xattr layout. Key V1 writer differences from V2: - BFS (breadth-first) inode ordering vs DFS (depth-first) - Compact inodes when uid/gid fit in u16 and mtime == build_time - Xattr sorting by full key name for C compatibility - Overlay whiteout char devices (00-ff) added to root directory - trusted.overlay.opaque=y xattr on root directory Tests cover both format versions: insta snapshots, proptest round-trips, fsck validation, and byte-identical comparison against the C mkcomposefs tool. The fuzz corpus generator also produces both V1 and V2 seed images. Assisted-by: OpenCode (Claude Opus 4) Signed-off-by: Colin Walters <walters@verbum.org>
The repository fsck tests only exercised V2 (Rust-native) EROFS images. Add tests that create V1 (C-compatible) images via mkfs_erofs_versioned and verify fsck handles them correctly — both for healthy images and for detecting missing referenced objects. Also add a V1 digest stability test alongside the existing V2 one, pinning the fsverity digests so any accidental change to V1 output (which must match C mkcomposefs) is caught immediately. Assisted-by: OpenCode (Claude Opus 4) Signed-off-by: Colin Walters <walters@verbum.org>
Generate random filesystem trees via proptest, write them as V1 and V2 EROFS images, feed the images to C composefs-info dump, and compare the output against our Rust reader's interpretation. Both V1 and V2 tests pass with 64 cases each. Comparison uses Entry::canonicalize() to normalize spec-permitted differences (hardlink metadata fields, xattr ordering) before comparing parsed entries. Also fix erofs_to_filesystem to skip overlay whiteout entries (chardev with rdev 0), matching the C reader behavior. These are internal composefs overlay machinery, not user-visible filesystem content. Assisted-by: OpenCode (Claude Opus 4) Signed-off-by: Colin Walters <walters@verbum.org>
Previously we accepted any composefs_version value, which means a future format change could be silently misinterpreted. Reject unknown versions and only accept the two known ones: - V1 (composefs_version=0): original C format - V2 (composefs_version=2): Rust-native format Assisted-by: OpenCode (Claude Opus 4) Signed-off-by: Colin Walters <walters@verbum.org>
The Stat struct was missing nanosecond precision for mtime, while the C
implementation tracks a full struct timespec (sec + nsec). This had
three visible effects:
- Dumpfile output wrote a hardcoded ".0" suffix for the mtime field
regardless of the actual nsec value.
- calculate_min_mtime in the EROFS writer hardcoded nsec=0 when
tracking the minimum mtime across inodes.
- stat_fd (reading from a real filesystem) discarded the nsec from
rustix's fstat result.
Add st_mtim_nsec: u32 to Stat. Populate it from fstat()'s
st_mtime_nsec (safe cast: nsec is always 0..999_999_999). Thread it
through dumpfile write (now emits {sec}.{nsec} instead of {sec}.0) and
parse (entry.mtime.nsec -> st_mtim_nsec). Fix calculate_min_mtime to
track nsec as a tiebreaker when seconds are equal. Also update
copy_root_metadata_from_usr and canonicalize_run to propagate nsec.
The EROFS reader sets st_mtim_nsec=0 since compact inodes don't store
mtime at all and extended inodes only store seconds (mtime_nsec is not
in the on-disk format), so roundtripping through EROFS loses nsec
precision.
Test fixture Stats all have mtime=0 / nsec=0, so dumpfile output
remains ".0" and pinned EROFS digest tests are unaffected.
Assisted-by: OpenCode (Claude Sonnet 4.5)
Assisted-by: OpenCode (Claude Sonnet 4.6)
Signed-off-by: Colin Walters <walters@verbum.org>
Add set_write_concurrency() to Repository for overriding the default parallelism. Add read_filesystem_with_semaphore() as a public entry point that accepts an explicit Semaphore, and refactor the internal read_filesystem_impl() to centralize semaphore selection. Prep for wiring up --threads in mkcomposefs. Assisted-by: OpenCode (Claude Sonnet 4.6) Signed-off-by: Colin Walters <walters@verbum.org>
Introduce ObjectStore<ObjectID> as an abstraction over content-addressed storage so that read_filesystem can write file objects to different backends without duplicating the scanning logic. Implement ObjectStore for Repository (unchanged semantics) and add FlatDigestStore which writes objects to the C-compatible flat XX/DIGEST layout. Add read_filesystem_with_store() as the preferred entry point when a custom store is needed. Assisted-by: OpenCode (Claude Sonnet 4.6) Signed-off-by: Colin Walters <walters@verbum.org>
Document the current state of the C composefs reimplementation across the CLI tools, with specific TODO(compat) markers for each known gap. This makes it easy to grep for remaining work and understand what's implemented vs what's missing. Key gaps tracked: --use-epoch leaf mtimes, --threads, --digest-store path layout, --max-version auto-upgrade, mtime nanoseconds, and the C shared library (libcomposefs) which is the next major milestone. Also fixes an outdated comment that claimed compact inodes were not implemented (they are, and have been tested byte-for-byte against C mkcomposefs). Assisted-by: OpenCode (Claude Opus 4) Assisted-by: OpenCode (Claude Sonnet 4.6) Signed-off-by: Colin Walters <walters@verbum.org>
Add mkcomposefs and composefs-info modes to the cfsctl multi-call binary, providing C-compatible CLI interfaces: mkcomposefs SOURCE IMAGE — create a composefs EROFS image composefs-info dump IMAGE — dump image metadata mkcomposefs features: - --from-file: read from composefs dumpfile instead of directory - --min-version / --max-version: select EROFS format version with validation - --threads N: control tokio worker threads and verity concurrency - --digest-store PATH: store file objects in C-compatible flat XX/DIGEST layout - --print-digest / --print-digest-only: print fsverity digest - --skip-devices, --skip-xattrs, --user-xattrs, --use-epoch The --digest-store layout matches C mkcomposefs exactly (XX/DIGEST flat paths) so digest stores are interchangeable between the two tools. Integration tests verify byte-for-bit image compatibility and digest store layout. Assisted-by: OpenCode (Claude Sonnet 4.5) Assisted-by: OpenCode (Claude Sonnet 4.6) Signed-off-by: Colin Walters <walters@verbum.org>
The st_mtim_nsec field was wired up in 6abfcd6d4 but left hardcoded to zero. PAX extended headers carry mtime as a decimal string like "1234567890.123456789"; tar-core populates ParsedEntry::mtime with only the integer seconds part, but preserves the raw PAX bytes in ParsedEntry::pax. Add pax_mtime_nsec() to parse the fractional part and use it when constructing Stat in get_entry(). Handles up to 9 digits (nanosecond precision), padding or truncating as needed. Assisted-by: OpenCode (Claude Sonnet 4.6) Signed-off-by: Colin Walters <walters@verbum.org>
cgwalters
force-pushed
the
composefs-c-compat
branch
from
25cbbb1 to
895ccd1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Basically starting on composefs/composefs#423
3 key goals:
Assisted-by: OpenCode (Claude Sonnet 4)