-
Notifications
You must be signed in to change notification settings - Fork 115
fix: FSST canonicalize when > 2GB of buffers #5961
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: bufferhandles
Are you sure you want to change the base?
Conversation
Merging this PR will not alter performance
Comparing Footnotes
|
3f62639 to
af7d307
Compare
Signed-off-by: Andrew Duffy <[email protected]>
| let len = len.as_(); | ||
| assert!(len <= max_buffer_len, "values cannot exceed max_buffer_len"); | ||
|
|
||
| if (offset + len) > max_buffer_len { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is actually very conservative, and doesn't account for the presence of inlined strings.
There isn't really a way to get around this without lots of data copying
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(FWIW this is already a problem on develop as well, except there everything lives in one giant buffer which sometimes is too large for VarBinView)
Codecov Report✅ All modified and coverable lines are covered by tests. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Fixes a bug where decoding a large FSST array results in an invalid VarBinViewArray.
When you have a large buffer that is FSST compressed, currently we generate a new VBV with the single buffer plus some views built against it. There will be trouble if the buffer is > 2GiB though.
This PR splits out a separate
build_viewsfunction that takes amax_buffer_lenparameter and as it generates views, it splits (zero-copy) the underlying buffer into segments of no more thanmax_buffer_len.