Use S3 native CRC64NVME full-object checksums (v8.0.0) - #38
Merged
Conversation
….0.0) Uploads now use S3's native SHA-512 checksum (ChecksumAlgorithm="SHA512"), which S3 validates server-side against the received data before storing - the legacy awsimple-sha512 custom metadata was only a label that nothing ever verified. The checksum is read back on the same head_object call (ChecksumMode="ENABLED") that get_s3_object_metadata already makes. The legacy custom metadata hash is no longer written. It is still read (S3ObjectMetadata.legacy_sha512) so objects written by older awsimple versions can be detected - those are always re-uploaded so that every object gains the native checksum. Objects with no hash at all (written by other tools, or multipart uploads whose native checksum is composite rather than full-object) fall back to modification-time and file-size comparison. Uploads stay single-part up to S3's 5 GiB single-part limit (TransferConfig multipart_threshold) so the native checksum is a full-object checksum comparable with a locally computed file hash. Requires boto3>=1.43.0 (S3 SHA-512 checksum support, April 2026) - now pinned in setup.py and requirements-dev.txt. Major version bump to 8.0.0 for the behavior and dependency changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hold Per design review: CRC64NVME replaces the SHA-512/CRC64NVME split as the single checksum algorithm for all S3 writes. Rationale: - It's the only checksum family S3 computes as a full-object value for both single-part and multipart uploads, so change detection behaves identically at any file size (no behavior cliff at the threshold). - It's S3's own default when no checksum is provided, so objects written by other modern tools usually carry a comparable checksum too. - One code path instead of a size-dependent algorithm switch. - Not cryptographic, but random-collision odds (~2^-64) are more than adequate for change detection and cache keying. The multipart threshold returns to boto3's common 8 MiB default (the prior 5 GiB single-PUT ceiling risked unresumable transfers on flaky connections and long-lived request fragility). Upload change detection tiers: 1. native full-object CRC64NVME (compared via new get_file_crc64nvme) 2. native full-object SHA-512 (objects written by other tools) 3. legacy awsimple metadata hash -> always re-upload (migration) 4. modification time + file size (free and robust) get_bytes_crc64nvme/get_file_crc64nvme are exported. Dependency is now boto3[crt]>=1.43.0 (awscrt computes CRC64NVME client-side). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
hashy 0.14.0 added CRC64NVME support (get_bytes_crc64nvme, get_file_crc64nvme), so awsimple's own awscrt-based helpers are removed in favor of hashy's - verified byte-identical output against awscrt and S3's returned checksum values. hashy>=0.14.0 is now the minimum. boto3[crt] remains pinned since botocore itself requires awscrt to compute the CRC64NVME trailer it sends during uploads. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces awsimple's custom
awsimple-sha512S3 object metadata with S3's native checksums (April 2026 S3 checksum expansion), landing on CRC64NVME as the single algorithm for all writes.Why CRC64NVME
Behavior
Upload change detection tiers:
S3ObjectMetadata.legacy_sha512) to detect old objects for migration.S3ObjectMetadatagainscrc64nvmeandlegacy_sha512fields;get_sha512()(cache key) prefers native checksums and falls back to the substitute hash.get_bytes_crc64nvme/get_file_crc64nvme(verified byte-identical to awscrt and to S3's returned values, including a 3-part multipart upload).Dependencies
boto3[crt]>=1.43.0- verified 1.43.0 is the exact botocore floor with the new checksum enums; thecrtextra provides awscrt, which botocore requires to compute CRC64NVME during uploads.hashy>=0.14.0- CRC64NVME support.Versioning
Major bump to 8.0.0: checksum wire-format change, one-time re-upload migration of legacy objects, and new dependency floors.
Tests
7 new tests in
test_s3_native_checksum.pycovering the native checksum on writes, foreign SHA-512 objects, no-hash objects (substitute hash), legacy-object migration, mtime+size fallback, upload skip, and multipart full-object checksums (20 MB, 3 parts). Existing SHA-512 assertions updated to CRC64NVME. Full suite: 98 passed.🤖 Generated with Claude Code