[Documentation] Update GIFs and notebook tutorials for 0.835 flat detection threshold

[RussellSB]

Summary

The flat detection threshold in process_signals.py has been changed from 0.50x to 0.835x min_nonzero_std (PR #224). This is a minor sensitivity reduction that makes trend detection more robust to small dips, but it changes some segment boundaries in the series_synthetic gradual column.

Impact

The following assets need to be updated to reflect the new segment boundaries:

GIFs in docs/img/plots/

• Gradual trends GIF — segment boundaries shifted (Mar 15-17 Flat removed, Up extends to Mar 17)
• Noise examples GIF — some noise edge case boundaries shifted

Notebook tutorials

• Gradual trends notebook — uses series_synthetic gradual column, segment boundaries will differ
• Noise examples notebook — some noise edge case boundaries shifted

Static documentation pages

• README.md — contains copy-pasted print_summary() output and segment table with the old boundaries
• docs/index.md — docs home page with identical example output blocks

What changed

Before	After	Reason
9 segments (3 Up, 3 Down, 3 Flat)	8 segments (3 Up, 3 Down, 2 Flat)	Mar 15-17 Flat merged into Up
Up Feb 10 – Mar 14	Up Feb 10 – Mar 17	Extended by 3 days
Flat Mar 15-17 present	Flat Mar 15-17 absent	Threshold too low to catch this region

Action required

1. Regenerate GIFs in docs/img/plots/ for gradual and noise examples
2. Update notebook tutorials that display segment boundaries for the gradual column
3. Update copy-pasted example output in README.md and docs/index.md
4. Verify that documentation examples still make sense with the new boundaries

Reference

PR #224: fix: detect long gradual ramps truncated by false flat detection

[coderabbitai]

🔗 Related PRs

#170 - fix: avoid_noise=False produced unintended noise on flat zero baseline edgecase [merged]
#177 - fix: detect Up trends on zero-baseline market entry and preserve Down total_change [merged]
#185 - docs(whats-new): reword zero-baseline section title and prose [merged]
#224 - fix: detect long gradual ramps truncated by false flat detection [open]
#243 - Feat: add gradual padding for gradual trends [open]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[coderabbitai]

Coding Plan

Summary

• Nail down the code first: confirm/apply the 0.835x coefficient in process_signals.py, because every downstream artefact is derived from its output — regenerating anything beforehand bakes in the wrong boundaries.
• Realign the automated suite (visual baseline plus the hardcoded gradual expectations in test_core_cases.py, test_io_results.py, the noise tests, and the debug plot-count) so CI passes against the new 8-segment result.
• Refresh user docs (both notebooks with normalisation, plus the static README/docs-index tables) to match the extended Up trend and removed Mar 15–17 Flat.
• Stay clear of the automated release artefacts and the pipeline-less marketing GIFs; treat the PNG baseline as the "GIF" the ticket actually means.

Design Choices

Design Choice 1: Is the PR `#224` code change (the 0.835x multiplier) actually present in the working branch?

Options Considered:

1. Assume PR #224 is merged elsewhere and only touch docs/baselines — regenerate against whatever the branch produces.
2. Treat the code change as a verifiable prerequisite: confirm the 0.835x coefficient is in process_signals.py, and apply it if absent, before regenerating anything.

Chosen Option: 2

Rationale: The exploration searched the checkout end-to-end and found no multiplier at all on the flat threshold — the code currently uses the raw min_nonzero_std (effectively 1.0x), with neither the old 0.50x nor the new 0.835x present anywhere. Neither CHANGELOG.md nor docs/whats-new.md reference #224. This is either a shallow-clone artefact or the code change genuinely hasn't landed. Regenerating baselines/docs against a codebase that lacks the change would bake in the old boundaries and silently defeat the ticket, so the coefficient must be confirmed (or applied) first.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Establish the threshold change as ground truth

Everything downstream (baselines, test expectations, notebook outputs, README tables) derives from the same detect_trends() output. Before touching any asset, the flat-detection coefficient must be correct in code, otherwise every regenerated artefact will be wrong.

Confirm or apply the 0.835x flat-detection coefficient

Ensure the flat-detection comparison scales min_nonzero_std by 0.835 rather than using it raw.

• Inspect the flat-detection block in pytrendy/process_signals.py (the step computing rolling_std, min_nonzero_std, and setting flat_flag).
• If the 0.835 coefficient is absent (as the exploration indicates), apply it to the min_nonzero_std value used in the smoothed_std <= … comparison. Keep it a single, clearly-named scaling factor so it reads as an intentional sensitivity tuning, not a magic number buried inline.
• Do not alter the per-column, data-driven derivation of min_nonzero_std itself — only the multiplier applied to it.
• Sanity-check that detect_trends(df, value_col='gradual') now yields 8 segments with Up running Feb 10 → Mar 17 and no Flat at Mar 15–17, matching the ticket's "after" table.

🤖 Prompt for AI agents

Implement the flat-detection sensitivity fix in `pytrendy/process_signals.py` —
this is the single upstream change that every other phase depends on, so get it
right before anything else is touched.

- Inspect the flat-detection block that computes `rolling_std`,
`min_nonzero_std`, and sets `flat_flag`.
- Check whether the comparison already scales `min_nonzero_std` by 0.835. If it
does not (current exploration says it uses the raw value, i.e. an effective
1.0x), introduce the 0.835 multiplier applied to `min_nonzero_std` in the
`smoothed_std <= …` comparison. Name the scaling factor clearly so it reads as a
deliberate sensitivity tuning rather than a magic number.
- Do not touch the per-column, data-driven derivation of `min_nonzero_std`
itself — only the multiplier applied to it changes.
- Sanity-check the result: running `detect_trends(df, value_col='gradual')`
should now produce 8 segments, with the `Up` segment running Feb 10 → Mar 17 and
no `Flat` segment at Mar 15–17.

Phase 2: Realign the automated test suite

The same gradual-column computation feeds one visual baseline and multiple hardcoded numeric assertions. All of them will fail against the new boundaries and must be brought back into agreement, or CI (which runs --mpl via addopts and is split into -m core / -m "not core") will block the change.

Task 1: Regenerate the gradual-trends visual baseline

Refresh the pytest-mpl PNG so image comparison passes against the new segmentation.

• Regenerate tests/tests_plotting/core/test_plot_gradual_trends.png using the documented workflow: pytest tests/tests_plotting/core/test_plot_pytrendy_core.py --mpl-generate-path=tests/tests_plotting/core/.
• The environment must have matplotlib==3.10.8 (the hard pin); regenerating against any other version produces spurious rendering diffs.
• Leave the abrupt-column baselines (test_plot_abrupt_trends_no_padding.png, test_plot_abrupt_trends_with_padding.png) and all edgecase baselines untouched. Evidence is strong they're unaffected: those tests use value_col='abrupt', min_nonzero_std is computed independently per series, and the abrupt flats sit inside wide, non-borderline intervals far from the Mar 15–17 boundary. Verify by running the full plotting suite and confirming only the gradual baseline changed.

Task 2: Update hardcoded gradual-column segment expectations

Bring the numeric assertions in line with the 8-segment result.

• Update the expected segment list in tests/test_core_cases.py::test_gradual_trends: remove the Flat 2025-03-15 → 2025-03-17 entry and extend the preceding Up segment to end 2025-03-17.
• Update tests/test_io_results.py: the expected_flat list (drop the Mar 15–17 flat), the direction-count assertions (Flat count drops from 3 to 2), the total segment count (9 → 8), and any best-segment assertions if they shift.
• Re-verify tests/test_debug.py's plot-call count assertion (== 7); adjust only if the changed flat-processing path actually alters the count.

Task 3: Re-verify noise-suite gradual expectations

The noise tests inject spikes onto the gradual column and are equally sensitive to the flat threshold.

• Run and reconcile tests/tests_noise/test_noise_spikes_gradual.py (most exposed — hardcoded Noise/Flat/Up/Down dates, including explicit flat gap-fill cases), tests/tests_noise/test_noise_avoid_false.py (the one gradual-based test), and tests/tests_noise/test_noise_random.py (the gradual-noisy-20 noise-percentage range and relative-noise-level assertions).
• Update only the expectations that genuinely shift under the new threshold; the range/qualitative assertions may still hold and shouldn't be changed cosmetically.
• Leave abrupt-only noise tests (test_noise_spikes_abrupt.py) alone.

🤖 Prompt for AI agents

Bring the automated test suite back into agreement with the new 8-segment output
from Phase 1, covering both the visual baseline and every hardcoded numeric
expectation derived from the `gradual` column.

- Regenerate the visual baseline: run `pytest
tests/tests_plotting/core/test_plot_pytrendy_core.py
--mpl-generate-path=tests/tests_plotting/core/` to refresh
`tests/tests_plotting/core/test_plot_gradual_trends.png`. The environment must
use `matplotlib==3.10.8` (hard pin) — any other version will produce spurious
rendering diffs. Do not touch the abrupt-column baselines
(`test_plot_abrupt_trends_no_padding.png`,
`test_plot_abrupt_trends_with_padding.png`) or edgecase baselines, since those
use `value_col='abrupt'` with independently-computed `min_nonzero_std` values
far from the Mar 15–17 boundary — confirm this by running the full plotting
suite and checking only the gradual baseline changed.

- Update hardcoded gradual-column segment expectations:
- In `tests/test_core_cases.py::test_gradual_trends`, remove the `Flat
2025-03-15 → 2025-03-17` entry and extend the preceding `Up` segment to end
`2025-03-17`.
- In `tests/test_io_results.py`, update the `expected_flat` list (drop the Mar
15–17 flat), the direction-count assertions (`Flat` count drops from 3 to 2),
the total segment count (9 → 8), and any `best`-segment assertions that shift.
- Re-check `tests/test_debug.py`'s plot-call count assertion (currently `== 7`);
adjust only if the changed flat-processing path actually alters the count.

- Re-verify noise-suite gradual expectations:
- Run and reconcile `tests/tests_noise/test_noise_spikes_gradual.py` (hardcoded
Noise/Flat/Up/Down dates, including explicit flat gap-fill cases),
`tests/tests_noise/test_noise_avoid_false.py`, and
`tests/tests_noise/test_noise_random.py` (the `gradual-noisy-20`
noise-percentage range and relative-noise-level assertions).
- Update only the expectations that genuinely shift under the new threshold —
don't touch range/qualitative assertions that still hold just for the sake of
it.
- Leave `test_noise_spikes_abrupt.py` and other abrupt-only noise tests
untouched.

Phase 3: Refresh user-facing documentation

The prose and pre-baked outputs shown to users must match the new boundaries, or the tutorials will contradict the library's actual behaviour. Note: the ticket's "GIFs" wording is loose — see the follow-up notes.

Task 1: Regenerate the gradual and noise notebook outputs

Update the pre-saved cell outputs and any narrative prose that quotes specific dates/counts.

• Re-execute docs/examples/fundamentals/gradual.ipynb so print_summary(), df_summary, filter_segments(...), and best outputs reflect 8 segments with the extended Up trend.
• Re-execute docs/examples/fundamentals/noise.ipynb. Its "Before" baseline (gradual), the seeded gradual_noisy (std=15, seed=42) tables, the 9 noise-segment listing, the noise_std in [0,10,20,50] figures, and the gradual_spiky outputs all pass through the same flat gate and may shift. Update the hardcoded narrative markdown cells (e.g. "shortened from 39 days to 12 days", "9 segments were detected as noise", the downtrend-split sentence) to match whatever the re-executed numbers actually show.
• After regenerating outputs, run scripts/normalize_notebooks.sh so JupyterLite renders the saved outputs correctly (list-form text/data fields joined into single strings).

Task 2: Update the copy-pasted example output in README and docs index

These are static markdown blocks, not generated, so they need manual editing.

• Update the hardcoded print_summary() / segment table in README.md and the identical block in docs/index.md: remove the Flat 2025-03-15 – 2025-03-17 row, extend the Up segment to Mar 17, and adjust any counts or filter_segments top-N tables that shift.
• Verify surrounding prose still reads correctly against the revised numbers.

Task 3: Confirm scope boundaries on static assets and release artefacts

Prevent wasted effort and accidental breakage on things the ticket only loosely implicates.

• Do not hand-edit CHANGELOG.md or docs/whats-new.md — both are automation targets (semantic-release and the agentic whats-new workflow respectively). The changelog entry will be generated from the commit type post-merge.
• The plots/*.gif marketing assets (Gradual-Cropped.gif, etc.) have no generation pipeline in the repo and are hand-authored. The ticket's "GIFs" almost certainly means the pytest-mpl PNG baseline handled in Task 2.1. Flag to the reviewer that if the literal GIFs are meant to be refreshed, that's a manual re-recording task outside any automated path — do not block on it.
• Commit classification: if this PR carries the process_signals.py change itself, title it fix: (behaviour-correcting threshold tweak → patch bump); if it is purely the docs/baseline follow-up to an already-merged code change, title it docs:.

🤖 Prompt for AI agents

Bring user-facing documentation (notebooks and static markdown) into line with
the new 8-segment `gradual` output, while explicitly staying clear of automated
release artefacts and pipeline-less marketing assets.

- Re-execute `docs/examples/fundamentals/gradual.ipynb` so `print_summary()`,
`df_summary`, `filter_segments(...)`, and `best` outputs reflect 8 segments with
the extended Up trend.
- Re-execute `docs/examples/fundamentals/noise.ipynb`. Its "Before" baseline
(`gradual`), the seeded `gradual_noisy` (std=15, seed=42) tables, the 9
noise-segment listing, the `noise_std in [0,10,20,50]` figures, and the
`gradual_spiky` outputs all pass through the same flat gate and may shift —
update the hardcoded narrative markdown cells (e.g. "shortened from 39 days to
12 days", "9 segments were detected as noise", the downtrend-split sentence) to
match whatever the re-executed numbers actually show.
- After regenerating notebook outputs, run `scripts/normalize_notebooks.sh` so
JupyterLite renders the saved outputs correctly (list-form text/data fields
joined into single strings).

- Manually update the copy-pasted example output in `README.md` and
`docs/index.md`: remove the `Flat 2025-03-15 – 2025-03-17` row, extend the `Up`
segment to Mar 17, and adjust any counts or `filter_segments` top-N tables that
shift. Verify surrounding prose still reads correctly against the revised
numbers.

- Confirm scope boundaries: do not hand-edit `CHANGELOG.md` or
`docs/whats-new.md` (both are automation targets — semantic-release and the
agentic whats-new workflow respectively). Do not attempt to regenerate the
`plots/*.gif` marketing assets (e.g. `Gradual-Cropped.gif`) — they have no
generation pipeline in the repo and are hand-authored; the ticket's "GIFs"
reference almost certainly means the `pytest-mpl` PNG baseline from Phase 2, so
flag to the reviewer that literal GIF re-recording is a manual task outside this
plan's scope. For commit classification, title the PR `fix:` if it carries the
`process_signals.py` change itself (behaviour-correcting threshold tweak → patch
bump), or `docs:` if it is purely the docs/baseline follow-up to an
already-merged code change.

Research

pytrendy is a trend-detection library. The core signal pipeline lives in pytrendy/process_signals.py, called from detect_trends() in pytrendy/detect_trends.py. Flat detection flags rows where the smoothed rolling std sits at or below min_nonzero_std; flat-flagged rows are excluded from Up/Down classification, so the threshold directly governs segment boundaries.

Visual regression is enforced via pytest-mpl against PNG baselines under tests/tests_plotting/, with matplotlib hard-pinned to 3.10.8. Numeric segment expectations are hardcoded across several test modules.

User docs consist of Jupyter notebooks under docs/examples/fundamentals/ (rendered via JupyterLite), plus copy-pasted example output in README.md and docs/index.md. CHANGELOG.md and docs/whats-new.md are fully automated release artefacts and must not be hand-edited.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Implement the flat-detection sensitivity fix in `pytrendy/process_signals.py` —
this is the single upstream change that every other phase depends on, so get it
right before anything else is touched.

- Inspect the flat-detection block that computes `rolling_std`,
`min_nonzero_std`, and sets `flat_flag`.
- Check whether the comparison already scales `min_nonzero_std` by 0.835. If it
does not (current exploration says it uses the raw value, i.e. an effective
1.0x), introduce the 0.835 multiplier applied to `min_nonzero_std` in the
`smoothed_std <= …` comparison. Name the scaling factor clearly so it reads as a
deliberate sensitivity tuning rather than a magic number.
- Do not touch the per-column, data-driven derivation of `min_nonzero_std`
itself — only the multiplier applied to it changes.
- Sanity-check the result: running `detect_trends(df, value_col='gradual')`
should now produce 8 segments, with the `Up` segment running Feb 10 → Mar 17 and
no `Flat` segment at Mar 15–17.
===============================================================================

Task: 2

Bring the automated test suite back into agreement with the new 8-segment output
from Phase 1, covering both the visual baseline and every hardcoded numeric
expectation derived from the `gradual` column.

- Regenerate the visual baseline: run `pytest
tests/tests_plotting/core/test_plot_pytrendy_core.py
--mpl-generate-path=tests/tests_plotting/core/` to refresh
`tests/tests_plotting/core/test_plot_gradual_trends.png`. The environment must
use `matplotlib==3.10.8` (hard pin) — any other version will produce spurious
rendering diffs. Do not touch the abrupt-column baselines
(`test_plot_abrupt_trends_no_padding.png`,
`test_plot_abrupt_trends_with_padding.png`) or edgecase baselines, since those
use `value_col='abrupt'` with independently-computed `min_nonzero_std` values
far from the Mar 15–17 boundary — confirm this by running the full plotting
suite and checking only the gradual baseline changed.

- Update hardcoded gradual-column segment expectations:
- In `tests/test_core_cases.py::test_gradual_trends`, remove the `Flat
2025-03-15 → 2025-03-17` entry and extend the preceding `Up` segment to end
`2025-03-17`.
- In `tests/test_io_results.py`, update the `expected_flat` list (drop the Mar
15–17 flat), the direction-count assertions (`Flat` count drops from 3 to 2),
the total segment count (9 → 8), and any `best`-segment assertions that shift.
- Re-check `tests/test_debug.py`'s plot-call count assertion (currently `== 7`);
adjust only if the changed flat-processing path actually alters the count.

- Re-verify noise-suite gradual expectations:
- Run and reconcile `tests/tests_noise/test_noise_spikes_gradual.py` (hardcoded
Noise/Flat/Up/Down dates, including explicit flat gap-fill cases),
`tests/tests_noise/test_noise_avoid_false.py`, and
`tests/tests_noise/test_noise_random.py` (the `gradual-noisy-20`
noise-percentage range and relative-noise-level assertions).
- Update only the expectations that genuinely shift under the new threshold —
don't touch range/qualitative assertions that still hold just for the sake of
it.
- Leave `test_noise_spikes_abrupt.py` and other abrupt-only noise tests
untouched.
===============================================================================

Task: 3

Bring user-facing documentation (notebooks and static markdown) into line with
the new 8-segment `gradual` output, while explicitly staying clear of automated
release artefacts and pipeline-less marketing assets.

- Re-execute `docs/examples/fundamentals/gradual.ipynb` so `print_summary()`,
`df_summary`, `filter_segments(...)`, and `best` outputs reflect 8 segments with
the extended Up trend.
- Re-execute `docs/examples/fundamentals/noise.ipynb`. Its "Before" baseline
(`gradual`), the seeded `gradual_noisy` (std=15, seed=42) tables, the 9
noise-segment listing, the `noise_std in [0,10,20,50]` figures, and the
`gradual_spiky` outputs all pass through the same flat gate and may shift —
update the hardcoded narrative markdown cells (e.g. "shortened from 39 days to
12 days", "9 segments were detected as noise", the downtrend-split sentence) to
match whatever the re-executed numbers actually show.
- After regenerating notebook outputs, run `scripts/normalize_notebooks.sh` so
JupyterLite renders the saved outputs correctly (list-form text/data fields
joined into single strings).

- Manually update the copy-pasted example output in `README.md` and
`docs/index.md`: remove the `Flat 2025-03-15 – 2025-03-17` row, extend the `Up`
segment to Mar 17, and adjust any counts or `filter_segments` top-N tables that
shift. Verify surrounding prose still reads correctly against the revised
numbers.

- Confirm scope boundaries: do not hand-edit `CHANGELOG.md` or
`docs/whats-new.md` (both are automation targets — semantic-release and the
agentic whats-new workflow respectively). Do not attempt to regenerate the
`plots/*.gif` marketing assets (e.g. `Gradual-Cropped.gif`) — they have no
generation pipeline in the repo and are hand-authored; the ticket's "GIFs"
reference almost certainly means the `pytest-mpl` PNG baseline from Phase 2, so
flag to the reviewer that literal GIF re-recording is a manual task outside this
plan's scope. For commit classification, title the PR `fix:` if it carries the
`process_signals.py` change itself (behaviour-correcting threshold tweak → patch
bump), or `docs:` if it is purely the docs/baseline follow-up to an
already-merged code change.

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!

[RussellSB]

/oc also refine this issue description to mention also README.md and index.md docs home page would need updating too

[github-actions]

Done. I've updated issue #233 to add:

1. New subsection under Impact: "Static documentation pages" listing README.md and docs/index.md with their copy-pasted example output blocks
2. New action item: "Update copy-pasted example output in README.md and docs/index.md"

opencode session  |  github run

[RussellSB]

/oc can you try start this by basing doc branch off of the branch of PR #224 , and using it as target. I want you to try address what you can - updating PNGs, Example notebook stats, and md comments etc... Dont address GIFs, leave that to me to then contribute to on the draft PR as the current workflow for that animation/rendering process is manual at the moment.

Create a draft PR. See that you have permissions to do so before diving too deep,

[opencode-agent]

Done. Draft PR created: #246

What was done:

• Branch: docs/update-baselines-for-flat-threshold based off PR fix: detect long gradual ramps truncated by false flat detection #224's fix/gradual-up-truncated
• README.md & docs/index.md: Updated print_summary() output (9→8 segments, Flat Mar 15-17 removed, Up extended to Mar 17) and filter_segments table with corrected dates/metrics
• gradual.ipynb: Re-executed — all output cells now reflect 8-segment result
• noise.ipynb: Re-executed + fixed markdown ("9 segments" → "10 segments")
• abrupt.ipynb: Normalised via scripts/normalize_notebooks.sh (no content changes)
• Tests: All 101 pass (74 core + 27 non-core)

Left to you (as requested):

• GIFs in docs/img/plots/ — hand-authored marketing assets, no generation pipeline in repo

opencode session  |  github run