Skip to content

Conversation

@a10y
Copy link
Contributor

@a10y a10y commented Jan 14, 2026

Fixes a bug where decoding a large FSST array results in an invalid VarBinViewArray.

When you have a large buffer that is FSST compressed, currently we generate a new VBV with the single buffer plus some views built against it. There will be trouble if the buffer is > 2GiB though.

This PR splits out a separate build_views function that takes a max_buffer_len parameter and as it generates views, it splits (zero-copy) the underlying buffer into segments of no more than max_buffer_len.

@codspeed-hq
Copy link

codspeed-hq bot commented Jan 14, 2026

Merging this PR will not alter performance

✅ 1254 untouched benchmarks
⏩ 1254 skipped benchmarks1


Comparing fsst-canonical (6a79200) with bufferhandles (e09bc36)

Open in CodSpeed

Footnotes

  1. 1254 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@a10y a10y force-pushed the fsst-canonical branch 4 times, most recently from 3f62639 to af7d307 Compare January 14, 2026 20:24
let len = len.as_();
assert!(len <= max_buffer_len, "values cannot exceed max_buffer_len");

if (offset + len) > max_buffer_len {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually very conservative, and doesn't account for the presence of inlined strings.

There isn't really a way to get around this without lots of data copying

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(FWIW this is already a problem on develop as well, except there everything lives in one giant buffer which sometimes is too large for VarBinView)

@a10y a10y added the fix label Jan 14, 2026
@a10y a10y requested a review from gatesn January 14, 2026 20:35
@codecov
Copy link

codecov bot commented Jan 14, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.39%. Comparing base (e09bc36) to head (6a79200).

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants