Promql cpp functions#272
Conversation
# Conflicts: # pp/entrypoint/go_constants.h # pp/entrypoint/head/serialization.h # pp/go/cppbridge/entrypoint.h # pp/go/cppbridge/head.go # pp/go/cppbridge/head_test.go # pp/head/chunk_recoder.h # pp/series_data/decoder.h # pp/series_data/decoder/decorator/interval_decode_iterator.h # pp/series_data/decoder/traits.h # pp/series_data/decoder/universal_decode_iterator.h # pp/series_data/serialization/serialized_data.h
# Conflicts: # pp/go/storage/querier/querier.go
| const auto out = static_cast<Result*>(res); | ||
|
|
||
| RangeQuerierWithArgumentsWrapperV2 querier(*in->data_storage, in->query, out->serialized_data); | ||
| RangeQuerierWithArgumentsWrapperV2 querier(*in->data_storage, in->query, *in->hints, out->serialized_data, in->downsampling_ms); |
There was a problem hiding this comment.
Do we always know that hints are valid? Go's SelectHints states that this parameter is optional. Even in our code (pp_api.go) : seriesSet := q.Select(ctx, false, nil, matchers...), where hint is nil
There was a problem hiding this comment.
Hints is mandatory parameter for range query in cpp. I will research go code and fix nil hints, thanks)
# Conflicts: # pp/go/storage/querier/merge_series_set_test.go
vporoshok
left a comment
There was a problem hiding this comment.
Review: WindowFunctionIterator sub-interval logic vs ADR-001
Analyzed the sub-interval splitting algorithm in WindowFunctionIterator against ADR-001 (range function iteration algorithm). The code works correctly for the most common case (s < w < 2s), but has critical issues in two other cases.
Summary of findings
| # | Severity | Finding |
|---|---|---|
| 1 | Bug/Critical | advance_interval() breaks for w > 2s — produces invalid interval (min > max), potential infinite loop |
| 2 | Bug/Critical | Incorrect boundaries for w < s — aggregation captures samples from gaps between windows |
| 3 | Bug/Moderate | Possible double-count for sum_over_time at interval boundaries (depends on seek_to inclusive/exclusive semantics) |
| 4 | Design | count_over_time missing from dispatch (falls back to universal iterator — correct but no pushdown) |
| 5 | Design | sum_over_time result timestamp is last raw sample's ts, not sub-interval end |
| 6 | Observation | min/max sub-interval collapsing optimization from ADR not implemented (correct but sub-optimal) |
| [[nodiscard]] PROMPP_ALWAYS_INLINE TimeInterval advance_interval() const noexcept { | ||
| auto interval = iterator_.interval(); | ||
| if (interval.difference() == parameters_->step) { | ||
| interval.min = interval.max; | ||
|
|
||
| const auto diff = parameters_->range - parameters_->step; | ||
| interval.max += (diff == 0) ? parameters_->step : diff; | ||
| } else { | ||
| interval.min = std::exchange(interval.max, next_interval_boundary(interval.min)); |
There was a problem hiding this comment.
Bug (critical): advance_interval() produces invalid intervals when w > 2s
Trace for w = 9, s = 4, d = 1 (e.g. max_over_time(m[9m]) with 4m step):
1st interval: [c, c+4] len 4 = s ✓
2nd: diff == step → [c+4, c+9] len 5 = w−s
3rd: diff ≠ step → min = c+9,
max = next_boundary(c+4) = c+8
→ [c+9, c+8] min > max! INVALID
When w − s > s, the second interval overshoots the next grid boundary. next_interval_boundary(old.min) returns a value less than the current max, producing an inverted interval. The advance() loop will either spin forever or produce UB on the unsigned difference.
Root cause: advance_interval alternates between adding step and w − s, but assumes w − s ≤ s. This breaks for w > 2s.
Per ADR-001, when w > s and d ≠ 0, the boundary grid should alternate at d and s − d intervals (not s and w − s). For w > 2s the grid still works: boundaries at a + n·s and a + n·s − d with sub-interval lengths always d and s − d.
| [[nodiscard]] PROMPP_ALWAYS_INLINE static Timestamp next_interval_boundary(Timestamp start, Timestamp step_ms, Timestamp range_ms) noexcept { | ||
| if (range_ms <= step_ms) [[likely]] { | ||
| return start + range_ms; | ||
| } | ||
|
|
||
| return start + range_ms - (range_ms - step_ms); |
There was a problem hiding this comment.
Bug (critical): next_interval_boundary() produces wrong grid for w < s
When w ≤ s, this returns start + w. Trace for w = 2, s = 5:
Go windows: (c, c+2], (c+5, c+7], (c+10, c+12], ... (gaps of 3)
Code produces:
[c, c+2] ✓ (first window)
[c+2, c+2] len 0 → skip
[c+2, c+4] ← WRONG: not aligned to any Go window!
[c+4, c+4] len 0 → skip
[c+4, c+6] ← partially in gap (c+4,c+5] + partially in window (c+5,c+6]
The function creates a grid with spacing w regardless of s. But per ADR-001 case 3 (w < s), boundaries should be at a + n·s and a + n·s − w, giving alternating sub-intervals of length s − w (gap, skip) and w (actual window). The correct grid spacing is driven by s, not w.
Suggestion: next_interval_boundary should always produce boundaries from the grid {a + n·s, a + n·s − d} where d = w % s.
| } | ||
|
|
||
| kahan_sum_inc(value, sum_.value, c_); | ||
| sum_.timestamp = timestamp; |
There was a problem hiding this comment.
Moderate: possible double-count at interval boundaries
Consecutive sub-intervals share a boundary: new interval's min = previous interval's max. OverTimeFuncIterator::find_element() calls seek_to(interval_.min) which positions at the first sample ≥ min.
If a sample has a timestamp exactly equal to the shared boundary, it would be included in both adjacent sub-intervals. For max/min this is idempotent, but for sum_over_time it causes a double-count.
Please verify: does seek_to position at > min (exclusive) or ≥ min (inclusive)?
| case kChanges: | ||
| return DecodeIterator(std::in_place_type<DecodeIterator::ChangesIterator>, select_hints.function_parameters); | ||
|
|
||
| default: |
There was a problem hiding this comment.
Design: count_over_time not dispatched
ADR-001 lists count_over_time in the per-function output contract. It's not in the WindowFunction enum and falls through to the default branch here (universal iterator). This is functionally correct but misses the pushdown optimization.
Considering it only needs a count per sub-interval (similar to sum_over_time), it would be a small addition.
Summary
Implement C++ pushdown optimizations for PromQL functions at the storage layer. When the query engine provides
SelectHints.Func, the storage can now reduce data before it reaches the Go PromQL engine — returning only the samples needed for the specific function instead of the full series.New decorator iterators
MinOverTimeIterator,MaxOverTimeIterator,LastOverTimeIterator,SumOverTimeIterator(viaOverTimeFuncIteratortemplate with pluggable handlers)RateIterator(rate, increase),IRateIterator(irate, idelta)DeltaIterator,ChangesIterator,ResetsIteratorDownsamplingDecodeIterator— interval-based sample reductionRefactored decode iterator infrastructure
DecodeIteratorTraitwithseek(),seek_to(),invalidate(),set()methodsSeekResultenum for composable seek logic (kUpdateSample,kNext,kStop,kUpdateSampleNextAndStop)UniversalDecodeIteratorextended withseek(),seek_to(),invalidate(),set()that dispatch throughstd::visitDecodeIteratorvariant wrapping all iterator types for the Go binding layerFunction dispatch
FunctionNamesHash) for O(1) function name lookupcreate_decode_iterator()dispatches onSelectHints.Functo construct the appropriate iteratorGo bridge changes
SelectHintspassed through cppbridge via ABI-compatibleGenericSelectHints<Go::String, Go::SliceView>downsamplingMsparameter added toQuery()andChunkRecoderpathsDataStorageSerializedDataIteratorcontrol block simplified: directTimestamp()/Value()accessors replacing raw field accessentrypoint::head→entrypoint::series_dataTesting