forked from cube-js/cube
-
Notifications
You must be signed in to change notification settings - Fork 0
Feature/ADBC Server #2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
borodark
wants to merge
105
commits into
master
Choose a base branch
from
feature/arrow-ipc-api
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PostgreSQL wire protocol (port 4444) was already working. This PR specifically introduces: - Arrow IPC native protocol (port 4445) - Optional query result cache
Port 4444 (PostgreSQL wire protocol) was already there. Port 4445 (Arrow IPC native) is what this PR introduces.
… MetaContext::new() Upstream added a second parameter `pre_aggregations: Vec<PreAggregationMeta>` to MetaContext::new() but the call in transport.rs wasn't updated. This fix: - Imports parse_pre_aggregations_from_cubes() function - Extracts pre-aggregations from cube metadata before creating MetaContext - Passes pre_aggregations as the 2nd parameter to MetaContext::new() Matches the implementation in cubesql's cubestore_transport.rs and service.rs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
…l env performance fixes
6f1c53f to
6fe781d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
backend:cli
client:core
client:ngx
client:playground
client:react
client:vue
cube store
cubejs-jdbc-driver
data source driver
docs
driver:athena
driver:bigquery
driver:clickhouse
driver:crate
driver:databricks
driver:dremio
driver:druid
driver:duckdb
driver:elasticsearch
driver:firebolt
driver:hive
driver:materialize
driver:mongodb
driver:mssql
driver:mysql
driver:mysql-aurora-serverless
driver:oracle
driver:pinot
driver:postgres
driver:prestodb
driver:questdb
driver:redshift
driver:snowflake
driver:sqlite
driver:trino
javascript
python
rust
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Check List
ADBC Access to Cube
The Problem: I need Cube to serve data over ADBC Client
Side effect: Reading pre-aggregates from CubeStore bypassing cache - pessimistic read
Some numbers
Speedup increases with result set size because columnar format amortizes overhead.
2. Type-Preserving Data Transfer
This isn't just aesthetic—columnar tools perform 2-5x faster with properly typed data.
Use Cases
Fact: The legal limit of Cube Result Set is 50000
Data Science Pipelines
Get query results directly into pandas/polars without serialization overhead:
Real-Time Dashboards
Reduce query-to-visualization latency for dashboards with large result sets.
Data Engineering
Integrate Cube semantic layer with Arrow-native tools:
Complete example with:
Breaking Changes
None. This is a pure addition. Default behavior unchanged.
Checklist
Future Work (Not in This PR)
Batch by 50K and stream, perhaps?
The Ask
This PR demonstrates measurable performance improvements (2-5x for typical analytics queries) with zero breaking changes and full backward compatibility. The implementation is clean, tested, and documented with working examples in three languages.
Would love to discuss:
The future of data transfer is columnar. Let's bring CubeSQL along for the ride. 🚀
Related Issues: [Reference any relevant issues]
Demo Video: [Optional - link to demo]
Live Example: See
examples/recipes/arrow-ipc/for complete working code