CalculateRelativeDifference only calculates change based on new upload

[Arjun-Parmani]

The new implementation of NodeService.calculateRelativeDifferenceValues calculates the domain values based on the new upload (root id) for matching fingerprint values.

[stalep]

Good work on the tests — they clearly demonstrate the expected behavior for both the recalculation and minPrevious scenarios.

A few things to consider:

Domain ordering in the range query

Setting domainNode and domainValue to null in the second findMatchingFingerprint call (line 417-418) removes the domain-based ordering for range values. The sliding window comparison (window + minPrevious values split into "previous" and "recent") depends on values being ordered by domain — without it, you might get values in an arbitrary order, and the window wouldn't represent the correct time-series segment.

Take a look at what findMatchingFingerprint does with domainNode and domainValue in ValueService — the domain node controls the ORDER BY, and the domain value controls the "preceding this point" cutoff. You want to keep that ordering but scope which domain points you iterate over.

Hint: The commented-out code block above your change (lines 388-414) was an earlier attempt at exactly this approach — it gets domain values scoped to the root, then for each one fetches range values with domain ordering preserved. That's the right structure. The reason it was commented out was the TODO about not handling values after previous detections, which is explicitly out of scope for this issue.

Test assertions

The value count assertions (Count: 11, Count: 17) verify the total number of values in the folder, which indirectly confirms change detection happened. A more direct assertion would check the actual change detection values — for example, verify the detection value contains the expected domain point or ratio. This would catch cases where the right number of values exist but the wrong changes were detected.

[Arjun-Parmani]

On Domain ordering in the range query

Initially the comparison was done based on domain-value (v.data≤domainValue) to fetch previous range values. Because data is uploaded out of sequence (non-sequentially), this approach created a "visibility" bug:

Upload x=3: The query finds no previous data. It assigns y=1.1. No change is detected due to insufficient data.

Upload x=2: The query searches for v.data<=2. Since x=3 is the only record in the DB, the query returns null. The system assigns y=2.1 but fails to detect a change because it cannot "see" the x=3 record (since 3>2).

Upload x=1: The query searches for v.data<=1. Again, it returns null, failing to detect a change against x=2 or x=3.

The bug occurred because the v.data <= domain filter assumes that "previous" data always has a smaller domain value, which is false when uploads are unordered.

Then When I passed null for both domain node and domain value, it checks if(domain value !=null || domain node!null)
and then sorted using created_at. This regardless of the sequence of upload fetches the correct rangeValues based on the fingerprint value for that upload.

[stalep]

Good catch — you're right that the v.data <= domainValue filter breaks for out-of-order uploads. When x=3 is uploaded first and then x=2 arrives, the query v.data <= 2 can't see the x=3 record because 3 > 2, even though it was uploaded earlier and should be part of the comparison window.

The fallback to created_at ordering when both domain parameters are null makes sense for this case — it gives you the window based on upload history, which is what matters for detecting changes introduced by a new upload.

One thing worth adding: a test that explicitly validates the out-of-order upload scenario you described. Your current tests upload in descending domain order (x=3, x=2, x=1) which does exercise this path, but a test that uploads in a scrambled order (e.g., x=2, x=4, x=1, x=3) and verifies the correct changes are detected would document this behavior more explicitly and protect against future regressions.

[stalep]

I think this looks good, @willr3 any objections?