Hash table refactor and performance improvements - #26
Merged
Conversation
This refactor was executed by the Claude Opus 4.6 LLM under human direction. Tests pass, etc. but I wouldn't bet a kidney on it. The goal is to remove duplicated and nearly-duplicated hash probing and hash table construction logic. The chosen approach is to use callback functions and rely on the optimizer to inline these appropriately. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This change was executed by the Claude Opus 4.6 LLM under human direction. Tests pass, etc. but I wouldn't bet a spleen on it. This commit replaces the forward-chaining approach used in the accelerated hash join implementation with a two-pass approach for building a contiguous array of right-side row indices sorted by left-side row index. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
These commits refactor all hash-table related logic into shared construction and probing functions (with a callback interface) and replace forward chaining with a two-pass contiguous-array-based approach for hash join implementation. The net effect is that duplicate logic is removed and hash join performance is improved by up to ~2x for large tables in testing. This also means that the C-accelerated approach now beats the Python implementation for large-table joins; this wasn't always previously true.
Implementation was performed with Claude Opus 4.6; human prompting was extensive to drive the agent through the necessary profiling steps and away from various inadequate design choices. This is my first LLM-coauthored changeset for a public project, so while I have tried to review thoroughly, I may well have missed some things. All available tests remain passing, benchmarks meet or exceed previous performance, and
valgrinddoesn't find anything to complain about (at least intafracode).The refactor does rely on compiler inlining of callback functions for performance; the alternative would have been an ugly set of macros or a more obscure X-macro approach. Or C++, but that's a whole other
std::can<worm>!