Skip to content

Promql cpp functions#272

Open
cherep58 wants to merge 76 commits into
ppfrom
promql_cpp
Open

Promql cpp functions#272
cherep58 wants to merge 76 commits into
ppfrom
promql_cpp

Conversation

@cherep58
Copy link
Copy Markdown
Collaborator

@cherep58 cherep58 commented Mar 24, 2026

Summary

Implement C++ pushdown optimizations for PromQL functions at the storage layer. When the query engine provides SelectHints.Func, the storage can now reduce data before it reaches the Go PromQL engine — returning only the samples needed for the specific function instead of the full series.

New decorator iterators

  • Aggregation: MinOverTimeIterator, MaxOverTimeIterator, LastOverTimeIterator, SumOverTimeIterator (via OverTimeFuncIterator template with pluggable handlers)
  • Counter/rate family: RateIterator (rate, increase), IRateIterator (irate, idelta)
  • Transform: DeltaIterator, ChangesIterator, ResetsIterator
  • Downsampling: DownsamplingDecodeIterator — interval-based sample reduction

Refactored decode iterator infrastructure

  • CRTP-based DecodeIteratorTrait with seek(), seek_to(), invalidate(), set() methods
  • SeekResult enum for composable seek logic (kUpdateSample, kNext, kStop, kUpdateSampleNextAndStop)
  • UniversalDecodeIterator extended with seek(), seek_to(), invalidate(), set() that dispatch through std::visit
  • Outer DecodeIterator variant wrapping all iterator types for the Go binding layer

Function dispatch

  • gperf-generated perfect hash (FunctionNamesHash) for O(1) function name lookup
  • create_decode_iterator() dispatches on SelectHints.Func to construct the appropriate iterator

Go bridge changes

  • SelectHints passed through cppbridge via ABI-compatible GenericSelectHints<Go::String, Go::SliceView>
  • downsamplingMs parameter added to Query() and ChunkRecoder paths
  • DataStorageSerializedDataIterator control block simplified: direct Timestamp()/Value() accessors replacing raw field access
  • Namespace change: entrypoint::headentrypoint::series_data

Testing

  • Parametric C++ unit tests for each new iterator (boundary conditions, StaleNaN handling, time interval filtering, reset scenarios)
  • Go integration tests for all dispatched PromQL functions (downsampling, min/max/last/sum_over_time, rate, increase, changes, delta, irate, idelta, resets)

@cherep58 cherep58 marked this pull request as ready for review April 3, 2026 15:48
@cherep58 cherep58 marked this pull request as draft April 24, 2026 08:27
# Conflicts:
#	pp/entrypoint/go_constants.h
#	pp/entrypoint/head/serialization.h
#	pp/go/cppbridge/entrypoint.h
#	pp/go/cppbridge/head.go
#	pp/go/cppbridge/head_test.go
#	pp/head/chunk_recoder.h
#	pp/series_data/decoder.h
#	pp/series_data/decoder/decorator/interval_decode_iterator.h
#	pp/series_data/decoder/traits.h
#	pp/series_data/decoder/universal_decode_iterator.h
#	pp/series_data/serialization/serialized_data.h
@cherep58 cherep58 marked this pull request as ready for review April 29, 2026 14:15
const auto out = static_cast<Result*>(res);

RangeQuerierWithArgumentsWrapperV2 querier(*in->data_storage, in->query, out->serialized_data);
RangeQuerierWithArgumentsWrapperV2 querier(*in->data_storage, in->query, *in->hints, out->serialized_data, in->downsampling_ms);
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we always know that hints are valid? Go's SelectHints states that this parameter is optional. Even in our code (pp_api.go) : seriesSet := q.Select(ctx, false, nil, matchers...), where hint is nil

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hints is mandatory parameter for range query in cpp. I will research go code and fix nil hints, thanks)

Comment thread pp/go/storage/querier/series_bench_test.go
Comment thread pp/entrypoint/series_data/serialization.h
Comment thread pp/entrypoint/series_data_data_storage.h Outdated
Comment thread pp/prometheus/query.h
Comment thread pp/series_data/decoder/decorator/changes_iterator.h
Copy link
Copy Markdown
Collaborator

@vporoshok vporoshok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: WindowFunctionIterator sub-interval logic vs ADR-001

Analyzed the sub-interval splitting algorithm in WindowFunctionIterator against ADR-001 (range function iteration algorithm). The code works correctly for the most common case (s < w < 2s), but has critical issues in two other cases.

Summary of findings

# Severity Finding
1 Bug/Critical advance_interval() breaks for w > 2s — produces invalid interval (min > max), potential infinite loop
2 Bug/Critical Incorrect boundaries for w < s — aggregation captures samples from gaps between windows
3 Bug/Moderate Possible double-count for sum_over_time at interval boundaries (depends on seek_to inclusive/exclusive semantics)
4 Design count_over_time missing from dispatch (falls back to universal iterator — correct but no pushdown)
5 Design sum_over_time result timestamp is last raw sample's ts, not sub-interval end
6 Observation min/max sub-interval collapsing optimization from ADR not implemented (correct but sub-optimal)

Comment on lines +77 to +85
[[nodiscard]] PROMPP_ALWAYS_INLINE TimeInterval advance_interval() const noexcept {
auto interval = iterator_.interval();
if (interval.difference() == parameters_->step) {
interval.min = interval.max;

const auto diff = parameters_->range - parameters_->step;
interval.max += (diff == 0) ? parameters_->step : diff;
} else {
interval.min = std::exchange(interval.max, next_interval_boundary(interval.min));
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug (critical): advance_interval() produces invalid intervals when w > 2s

Trace for w = 9, s = 4, d = 1 (e.g. max_over_time(m[9m]) with 4m step):

1st interval:  [c, c+4]       len 4 = s         ✓
2nd:           diff == step → [c+4, c+9]  len 5 = w−s
3rd:           diff ≠ step → min = c+9,
               max = next_boundary(c+4) = c+8
               → [c+9, c+8]   min > max!  INVALID

When w − s > s, the second interval overshoots the next grid boundary. next_interval_boundary(old.min) returns a value less than the current max, producing an inverted interval. The advance() loop will either spin forever or produce UB on the unsigned difference.

Root cause: advance_interval alternates between adding step and w − s, but assumes w − s ≤ s. This breaks for w > 2s.

Per ADR-001, when w > s and d ≠ 0, the boundary grid should alternate at d and s − d intervals (not s and w − s). For w > 2s the grid still works: boundaries at a + n·s and a + n·s − d with sub-interval lengths always d and s − d.

Comment on lines +108 to +113
[[nodiscard]] PROMPP_ALWAYS_INLINE static Timestamp next_interval_boundary(Timestamp start, Timestamp step_ms, Timestamp range_ms) noexcept {
if (range_ms <= step_ms) [[likely]] {
return start + range_ms;
}

return start + range_ms - (range_ms - step_ms);
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug (critical): next_interval_boundary() produces wrong grid for w < s

When w ≤ s, this returns start + w. Trace for w = 2, s = 5:

Go windows: (c, c+2], (c+5, c+7], (c+10, c+12], ...  (gaps of 3)

Code produces:
[c, c+2]     ✓ (first window)
[c+2, c+2]   len 0 → skip
[c+2, c+4]   ← WRONG: not aligned to any Go window!
[c+4, c+4]   len 0 → skip
[c+4, c+6]   ← partially in gap (c+4,c+5] + partially in window (c+5,c+6]

The function creates a grid with spacing w regardless of s. But per ADR-001 case 3 (w < s), boundaries should be at a + n·s and a + n·s − w, giving alternating sub-intervals of length s − w (gap, skip) and w (actual window). The correct grid spacing is driven by s, not w.

Suggestion: next_interval_boundary should always produce boundaries from the grid {a + n·s, a + n·s − d} where d = w % s.

}

kahan_sum_inc(value, sum_.value, c_);
sum_.timestamp = timestamp;
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moderate: possible double-count at interval boundaries

Consecutive sub-intervals share a boundary: new interval's min = previous interval's max. OverTimeFuncIterator::find_element() calls seek_to(interval_.min) which positions at the first sample ≥ min.

If a sample has a timestamp exactly equal to the shared boundary, it would be included in both adjacent sub-intervals. For max/min this is idempotent, but for sum_over_time it causes a double-count.

Please verify: does seek_to position at > min (exclusive) or ≥ min (inclusive)?

Comment on lines +141 to +144
case kChanges:
return DecodeIterator(std::in_place_type<DecodeIterator::ChangesIterator>, select_hints.function_parameters);

default:
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Design: count_over_time not dispatched

ADR-001 lists count_over_time in the per-function output contract. It's not in the WindowFunction enum and falls through to the default branch here (universal iterator). This is functionally correct but misses the pushdown optimization.

Considering it only needs a count per sub-interval (similar to sum_over_time), it would be a small addition.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants