[Feature] Support weekly data (not just daily)

[RussellSB]

Best way would be to generalise to indices instead of timestamp, in all the pd timestamp related functions.

Would need to include higher level wrappers around this. Either detect the granularity automatically, or also allow an option for user to specify explicitly.

This will help people applying this for automated causal inference with synthetic control methods or MMMs.

Models will trigger/update for periods of significant change. This will allow them to get better reads on valid treatment periods of change

[RussellSB]

A later idea that could branch off this is candlestick charts for stock analysis.

E.g. can boxplot hourly, and then trend detect the daily. Or boxplot the daily, and trend detect the weekly... etc

[RussellSB]

Discussed making it abastract than allowing for final output data type (daily / weekly / numerical index) based on the date type of date_col.

In theory could keep date_col as date_col ... ? Up for debate whether worth refactoring. Will be a breaking change

[coderabbitai]

Coding Plan

Summary

• Introduce a single frequency-inference step that derives the series' sampling interval from the DatetimeIndex, defaulting to auto-detection with an optional explicit freq override added to the method_params allowlist.
• Thread the resolved step through the whole pipeline, replacing hard-coded pd.Timedelta(days=1)/.days/.floor('D') logic with step-scaled durations and positional row stepping for exact-label lookups, keeping row-count windows as index-based (frequency-agnostic).
• Update segment extraction, refinement constants, duration reporting, and plotting to scale with the inferred frequency, ensuring daily behavior is unchanged.
• Add weekly fixtures/tests, a freq parameter test, and documentation plus an example notebook following existing contribution conventions.

Design Choices

Design Choice 1: How should the sampling frequency be determined?

Options Considered:

1. Auto-detect only — infer step from the index on every call.
2. Explicit-only — require the user to pass a frequency.
3. Auto-detect by default with an optional explicit override via method_params.

Chosen Option: 3

Rationale: Auto-detect the sampling interval from the index by default (best zero-config UX for daily/weekly data) while allowing an explicit override for irregular or ambiguous inputs, matching the existing method_params extension pattern.

Design Choice 2: How should day-based timestamp arithmetic be generalized?

Options Considered:

1. Pure positional/index approach — convert all boundary logic to integer row positions (get_loc + iloc).
2. Frequency/step approach — infer one canonical sampling step and replace Timedelta(days=1) with that step, scaling day-based constants into number-of-periods.
3. Hybrid — infer a single sampling step threaded through the pipeline for scaling/duration, and use positional row stepping specifically where exact-label lookups would otherwise hit non-existent intermediate timestamps.

Chosen Option: 3

Rationale: A hybrid keeps disruption minimal against the existing string-serialized boundary pattern while robustly avoiding KeyError/silent no-ops on timestamps that don't exist for non-daily data.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Frequency Detection & Configuration Foundation

Establish a single, shared notion of the series' sampling interval that the rest of the pipeline can consume, and expose an optional user override. This phase introduces the mechanism without yet changing algorithmic behavior, so daily data continues to behave identically.

Task 1: Add a frequency-inference utility

Create a reusable helper that derives the sampling step (a pd.Timedelta and/or an offset) and a "periods-per-step" notion from a DatetimeIndex.

• Add a small utility module (e.g. pytrendy/frequency.py) that, given a sorted DatetimeIndex, returns the inferred step using pd.infer_freq when available and falling back to the median of consecutive Timedelta differences for irregular/short series.
• Expose the inferred step in both offset and elapsed-time form so downstream code can choose calendar-aware stepping vs. duration math.
• Include guardrails: raise a clear error (or emit a warning and fall back to daily) when the index is too short or too irregular to infer a step, keeping messaging consistent with existing docstring guidance.

Task 2: Thread frequency through the entry point and method_params

Wire detection and the optional override into detect_trends and pass the resulting step into the pipeline stages.

• Update pytrendy/detect_trends.py to invoke the inference utility after building the DatetimeIndex, and add a freq key to the method_params allowlist dict (default None = auto-detect), following the existing .get(key, default) convention used for abrupt_padding/avoid_noise.
• When freq is supplied, validate it against the index and use it as the authoritative step; otherwise use the inferred step.
• Pass the resolved step (and derived scaling factors) down into process_signals, get_segments, refine_segments, analyse_segments, and plot_pytrendy, so day-based logic in later phases can consume it rather than hard-coding days.
• Update the detect_trends Google-style docstring "Supported keys" list to document freq, and relax the "daily frequency recommended" note to describe supported regular frequencies.

🤖 Prompt for AI agents

Implement the foundation for frequency-aware processing: a reusable
frequency-inference utility and its wiring into the entry point.

1. Create `pytrendy/frequency.py` with a helper that, given a sorted
`DatetimeIndex`, infers the sampling step using `pd.infer_freq`, falling back to
the median of consecutive `Timedelta` differences when `pd.infer_freq` cannot
determine one (irregular/short series). Expose the inferred step in both offset
and elapsed-time (`Timedelta`) form so downstream code can pick calendar-aware
stepping or duration math as needed. Add guardrails that raise a clear error (or
emit a warning and fall back to daily) when the index is too short or too
irregular, keeping messaging consistent with existing docstring style.

2. Update `pytrendy/detect_trends.py`:
   - After building the `DatetimeIndex`, call the new inference utility.
- Add a `freq` key to the `method_params` allowlist dict with default `None`
(meaning auto-detect), following the existing `.get(key, default)` pattern used
for `abrupt_padding`/`avoid_noise`.
- If `freq` is explicitly supplied, validate it against the index and use it as
the authoritative step; otherwise use the auto-inferred step.
- Pass the resolved step (and any derived scaling factors) into
`process_signals`, `get_segments`, `refine_segments`, `analyse_segments`, and
`plot_pytrendy` so later phases can consume it instead of hard-coding days.
- Update the `detect_trends` Google-style docstring's "Supported keys" list to
document `freq`, and relax the "daily frequency recommended" note to instead
describe supported regular frequencies generally.

Keep behavior unchanged for daily data at this stage since this phase only
introduces the mechanism.

Phase 2: Generalize Core Signal Processing

Make the signal-processing stage operate on the inferred sampling interval and on row positions rather than on hard-coded calendar days, preserving current behavior for daily data.

Task 1: Replace day-based adjacency and duration logic in process_signals

Convert the noise-segment boundary and width logic to be frequency/index aware.

• Update pytrendy/process_signals.py so every pd.Timedelta(days=1) used to mean "one sample before/after" uses the resolved step (Phase 1); where exact-label .loc lookups occur, step via row position (index.get_loc + positional offset) to avoid landing on non-existent timestamps.
• Replace .days-based width/distance computations (used as point-count proxies and against GROUPING_DISTANCE) with number-of-periods (elapsed time divided by the step, or positional row counts), and re-express the narrow-window thresholds (<=4, <=2) in periods.
• Rework the center_start/center_end .floor('D') computation into a period/position-based interpolation so it lands on actual sampled timestamps regardless of frequency.

Task 2: Decide and document window-constant semantics

Ensure the rolling/smoothing windows behave consistently across frequencies.

• In pytrendy/process_signals.py, treat WINDOW_SMOOTH, WINDOW_FLAT, WINDOW_NOISE explicitly as row-count (index-based) windows, which is already frequency-agnostic in positional terms; add a comment clarifying they are point counts, not day spans.
• If preserving a consistent time-span is required for parity, derive these window sizes from a target span divided by the inferred step; otherwise keep them as fixed row counts. Keep this choice consistent with the periods-based thresholds introduced in Task 2.1.

🤖 Prompt for AI agents

Generalize the signal-processing stage (`pytrendy/process_signals.py`) so it
uses the resolved sampling step and row positions instead of hard-coded daily
arithmetic, while preserving current daily behavior.

1. Replace every use of `pd.Timedelta(days=1)` that means "one sample
before/after" with the resolved step from Phase 1. Where exact-label `.loc`
lookups are used, switch to positional stepping (`index.get_loc` + positional
offset) to avoid landing on non-existent timestamps for non-daily data.

2. Replace `.days`-based width/distance computations (used as point-count
proxies and compared against `GROUPING_DISTANCE`) with number-of-periods,
computed either as elapsed time divided by the step or as positional row counts.
Re-express the narrow-window thresholds (currently `<=4`, `<=2`) in periods
rather than days.

3. Rework the `center_start`/`center_end` computation that currently uses
`.floor('D')` into a period/position-based interpolation so the result lands on
an actual sampled timestamp regardless of frequency.

4. Decide on semantics for `WINDOW_SMOOTH`, `WINDOW_FLAT`, `WINDOW_NOISE`: treat
them explicitly as row-count (index-based) windows (already frequency-agnostic
positionally), and add a comment clarifying they represent point counts, not day
spans. If a consistent time-span across frequencies is instead required, derive
these window sizes from a target span divided by the inferred step. Whichever
approach is chosen, keep it consistent with the periods-based thresholds
introduced above.

Phase 3: Generalize Post-Processing & Refinement

Propagate frequency/index awareness through segment extraction, refinement, and metric analysis so boundaries, gaps, groupings, and durations are correct for non-daily data.

Task 1: Make segment extraction frequency-aware

Convert row-run counts to boundaries using the sampling step.

• Update pytrendy/post_processing/segments_get.py so the conversion from consecutive-row counts to start/end uses the step (one period back/forward) rather than Timedelta(days=segment_length_prev+1)/days=1; keep the '%Y-%m-%d' serialization.
• Re-express minimum segment length rules (e.g. Up/Down ≥ 3, Flat/Noise ≥ 1) in periods rather than days.

Task 2: Generalize refinement routines and shared constants

Update all boundary-shifting, grouping, and neighbour logic to scale with frequency.

• Update pytrendy/post_processing/segments_refine/* (abrupt_shaving.py, gradual_expand_contract.py, segment_grouping.py, artifact_cleanup.py, update_neighbours.py, trend_classify.py) to replace pd.Timedelta(days=N) offsets with step-based offsets, and to interpret GROUPING_DISTANCE, NEIGHBOUR_DISTANCE, DTW/z-score padding windows, and min-length checks in number-of-periods.
• Ensure abrupt_padding (currently day units) is applied in the resolved frequency's units so padding remains meaningful for weekly data; document the unit in the detect_trends docstring.
• Use positional stepping for exact-label lookups (df.loc[ts, col]) in these routines to avoid KeyError/silent skips on intermediate timestamps.

Task 3: Report duration in a frequency-correct way

Make the days metric and any duration reporting reflect the true sampling interval.

• Update pytrendy/post_processing/segments_analyse.py so segment duration is computed from number of periods (row span) and/or clearly-labeled elapsed time, rather than raw .days, so a 4-week segment isn't misreported.
• Update pytrendy/io/results_pytrendy.py summary columns accordingly (retain backward-compatible days where sensible, and/or add a periods/duration field), keeping the string date formatting for start/end.

🤖 Prompt for AI agents

Generalize segment extraction, refinement, and duration reporting so boundaries,
gaps, groupings, and durations are correct for non-daily data, using the
resolved sampling step from Phase 1.

1. In `pytrendy/post_processing/segments_get.py`, convert the
consecutive-row-count-to-boundary logic (`start`/`end`) to use the resolved step
(one period back/forward) instead of
`Timedelta(days=segment_length_prev+1)`/`Timedelta(days=1)`. Keep the existing
`'%Y-%m-%d'` string serialization of boundaries. Re-express minimum segment
length rules (Up/Down ≥ 3, Flat/Noise ≥ 1) in periods rather than days.

2. In `pytrendy/post_processing/segments_refine/` modules (`abrupt_shaving.py`,
`gradual_expand_contract.py`, `segment_grouping.py`, `artifact_cleanup.py`,
`update_neighbours.py`, `trend_classify.py`):
- Replace `pd.Timedelta(days=N)` offsets with step-based offsets derived from
the resolved frequency.
- Reinterpret `GROUPING_DISTANCE`, `NEIGHBOUR_DISTANCE`, DTW/z-score padding
windows, and minimum-length checks in number-of-periods instead of days.
- Apply `abrupt_padding` (currently expressed in day units) in the resolved
frequency's units so padding stays meaningful for weekly data, and document this
unit change in the `detect_trends` docstring.
- For exact-label lookups (`df.loc[ts, col]`), use positional stepping to avoid
`KeyError` or silent skips when timestamps don't exist for the given frequency.

3. In `pytrendy/post_processing/segments_analyse.py`, compute segment duration
from the number of periods (row span) and/or clearly-labeled elapsed time rather
than raw `.days`, so durations for coarser frequencies (e.g. a 4-week segment)
aren't misreported.

4. In `pytrendy/io/results_pytrendy.py`, update the summary columns accordingly:
retain the `days` column where it still makes sense for backward compatibility,
and/or add a periods/duration field, while keeping the existing string date
formatting for `start`/`end`.

Phase 4: Frequency-Aware Plotting

Ensure visualizations render correctly for non-daily series.

Generalize plot boundary and tick logic

Update pytrendy/io/plot_pytrendy.py.

• Replace pd.Timedelta(days=1) boundary/neighbour shading adjustments with the resolved step.
• Choose x-axis tick locators based on the inferred frequency (e.g. weekly/monthly locators for coarser data) instead of always using WeekdayLocator/DayLocator.

🤖 Prompt for AI agents

Generalize plotting so visualizations render correctly for non-daily series,
using the resolved sampling step from Phase 1.

1. In `pytrendy/io/plot_pytrendy.py`, replace uses of `pd.Timedelta(days=1)` for
boundary/neighbour shading adjustments with the resolved step.

2. Choose x-axis tick locators based on the inferred frequency (e.g. use
weekly/monthly locators for coarser data) instead of always defaulting to
`WeekdayLocator`/`DayLocator`.

Phase 5: Tests, Examples & Documentation

Add coverage and documentation for non-daily support following existing conventions.

Task 1: Add weekly test fixtures and pipeline tests

Prove the pipeline produces correct segments for weekly data.

• Add a weekly synthetic fixture (e.g. pd.date_range(..., freq='W')) in tests/ (a new conftest helper or a small bundled CSV mirroring the daily one), and add tests validating segment detection, boundaries, and durations for weekly spacing, reusing assert_segments_match/assert_segments_in_a_haystack.
• Add a regression test confirming daily data results are unchanged.

Task 2: Add freq method_params test

Follow the existing test_method_params.py pattern.

• Add a @pytest.mark.core test in tests/test_method_params.py exercising both auto-detection and an explicit freq override, asserting on resulting behavior (and on a clear error/warning for invalid/irregular input).

Task 3: Update documentation and example

Register and document the new capability.

• Update the detect_trends docstring, README.md, and docs/index.md to describe supported frequencies and the freq option (removing the daily-only framing).
• Add a weekly-data example notebook under docs/examples/fundamentals/ and register it in mkdocs.yml nav: alongside the existing fundamentals examples.
• Do not hand-edit CHANGELOG.md, docs/whats-new.md, or the pyproject.toml version (generated by automation); use a feat: PR title targeting develop.

🤖 Prompt for AI agents

Add tests, examples, and documentation covering non-daily support following
existing repository conventions.

1. Add a weekly synthetic fixture (e.g. via `pd.date_range(..., freq='W')`) in
`tests/`, either as a new conftest helper or a small bundled CSV mirroring the
existing daily fixture. Add tests validating segment detection, boundaries, and
durations for weekly spacing, reusing the existing
`assert_segments_match`/`assert_segments_in_a_haystack` helpers. Add a
regression test confirming daily data results remain unchanged.

2. In `tests/test_method_params.py`, add a `@pytest.mark.core` test exercising
both auto-detection and an explicit `freq` override in `method_params`,
asserting on the resulting behavior, and asserting a clear error/warning is
raised for invalid/irregular input.

3. Update documentation:
- Update the `detect_trends` docstring, `README.md`, and `docs/index.md` to
describe supported frequencies and the `freq` option, removing the daily-only
framing.
- Add a weekly-data example notebook under `docs/examples/fundamentals/` and
register it in the `mkdocs.yml` `nav:` section alongside the existing
fundamentals examples.
- Do not hand-edit `CHANGELOG.md`, `docs/whats-new.md`, or the `pyproject.toml`
version, since these are generated by automation; use a `feat:` PR title
targeting the `develop` branch.

Research

PyTrendy exposes a single orchestration entry point, detect_trends() (pytrendy/detect_trends.py), which parses a DataFrame's date column into a DatetimeIndex and runs a 5-stage pipeline: signal processing (process_signals.py), segment extraction (post_processing/segments_get.py), boundary refinement (post_processing/segments_refine/*), metric analysis (post_processing/segments_analyse.py), and plotting (io/plot_pytrendy.py). Configuration flows through a method_params allowlist dict rebuilt inline in detect_trends.py. The entire pipeline currently assumes daily spacing: adjacent-sample operations use pd.Timedelta(days=1), segment widths/distances use .days, boundaries round with .floor('D'), and constants (GROUPING_DISTANCE=7, NEIGHBOUR_DISTANCE=3, min segment lengths, abrupt_padding) are expressed in days. Segment boundaries are serialized between stages as '%Y-%m-%d' strings. DTW (simpledtw.py) is already date-agnostic (positional only). Tests rely on static daily CSV fixtures with no frequency coverage.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Implement the foundation for frequency-aware processing: a reusable
frequency-inference utility and its wiring into the entry point.

1. Create `pytrendy/frequency.py` with a helper that, given a sorted
`DatetimeIndex`, infers the sampling step using `pd.infer_freq`, falling back to
the median of consecutive `Timedelta` differences when `pd.infer_freq` cannot
determine one (irregular/short series). Expose the inferred step in both offset
and elapsed-time (`Timedelta`) form so downstream code can pick calendar-aware
stepping or duration math as needed. Add guardrails that raise a clear error (or
emit a warning and fall back to daily) when the index is too short or too
irregular, keeping messaging consistent with existing docstring style.

2. Update `pytrendy/detect_trends.py`:
   - After building the `DatetimeIndex`, call the new inference utility.
- Add a `freq` key to the `method_params` allowlist dict with default `None`
(meaning auto-detect), following the existing `.get(key, default)` pattern used
for `abrupt_padding`/`avoid_noise`.
- If `freq` is explicitly supplied, validate it against the index and use it as
the authoritative step; otherwise use the auto-inferred step.
- Pass the resolved step (and any derived scaling factors) into
`process_signals`, `get_segments`, `refine_segments`, `analyse_segments`, and
`plot_pytrendy` so later phases can consume it instead of hard-coding days.
- Update the `detect_trends` Google-style docstring's "Supported keys" list to
document `freq`, and relax the "daily frequency recommended" note to instead
describe supported regular frequencies generally.

Keep behavior unchanged for daily data at this stage since this phase only
introduces the mechanism.
===============================================================================

Task: 2

Generalize the signal-processing stage (`pytrendy/process_signals.py`) so it
uses the resolved sampling step and row positions instead of hard-coded daily
arithmetic, while preserving current daily behavior.

1. Replace every use of `pd.Timedelta(days=1)` that means "one sample
before/after" with the resolved step from Phase 1. Where exact-label `.loc`
lookups are used, switch to positional stepping (`index.get_loc` + positional
offset) to avoid landing on non-existent timestamps for non-daily data.

2. Replace `.days`-based width/distance computations (used as point-count
proxies and compared against `GROUPING_DISTANCE`) with number-of-periods,
computed either as elapsed time divided by the step or as positional row counts.
Re-express the narrow-window thresholds (currently `<=4`, `<=2`) in periods
rather than days.

3. Rework the `center_start`/`center_end` computation that currently uses
`.floor('D')` into a period/position-based interpolation so the result lands on
an actual sampled timestamp regardless of frequency.

4. Decide on semantics for `WINDOW_SMOOTH`, `WINDOW_FLAT`, `WINDOW_NOISE`: treat
them explicitly as row-count (index-based) windows (already frequency-agnostic
positionally), and add a comment clarifying they represent point counts, not day
spans. If a consistent time-span across frequencies is instead required, derive
these window sizes from a target span divided by the inferred step. Whichever
approach is chosen, keep it consistent with the periods-based thresholds
introduced above.
===============================================================================

Task: 3

Generalize segment extraction, refinement, and duration reporting so boundaries,
gaps, groupings, and durations are correct for non-daily data, using the
resolved sampling step from Phase 1.

1. In `pytrendy/post_processing/segments_get.py`, convert the
consecutive-row-count-to-boundary logic (`start`/`end`) to use the resolved step
(one period back/forward) instead of
`Timedelta(days=segment_length_prev+1)`/`Timedelta(days=1)`. Keep the existing
`'%Y-%m-%d'` string serialization of boundaries. Re-express minimum segment
length rules (Up/Down ≥ 3, Flat/Noise ≥ 1) in periods rather than days.

2. In `pytrendy/post_processing/segments_refine/` modules (`abrupt_shaving.py`,
`gradual_expand_contract.py`, `segment_grouping.py`, `artifact_cleanup.py`,
`update_neighbours.py`, `trend_classify.py`):
- Replace `pd.Timedelta(days=N)` offsets with step-based offsets derived from
the resolved frequency.
- Reinterpret `GROUPING_DISTANCE`, `NEIGHBOUR_DISTANCE`, DTW/z-score padding
windows, and minimum-length checks in number-of-periods instead of days.
- Apply `abrupt_padding` (currently expressed in day units) in the resolved
frequency's units so padding stays meaningful for weekly data, and document this
unit change in the `detect_trends` docstring.
- For exact-label lookups (`df.loc[ts, col]`), use positional stepping to avoid
`KeyError` or silent skips when timestamps don't exist for the given frequency.

3. In `pytrendy/post_processing/segments_analyse.py`, compute segment duration
from the number of periods (row span) and/or clearly-labeled elapsed time rather
than raw `.days`, so durations for coarser frequencies (e.g. a 4-week segment)
aren't misreported.

4. In `pytrendy/io/results_pytrendy.py`, update the summary columns accordingly:
retain the `days` column where it still makes sense for backward compatibility,
and/or add a periods/duration field, while keeping the existing string date
formatting for `start`/`end`.
===============================================================================

Task: 4

Generalize plotting so visualizations render correctly for non-daily series,
using the resolved sampling step from Phase 1.

1. In `pytrendy/io/plot_pytrendy.py`, replace uses of `pd.Timedelta(days=1)` for
boundary/neighbour shading adjustments with the resolved step.

2. Choose x-axis tick locators based on the inferred frequency (e.g. use
weekly/monthly locators for coarser data) instead of always defaulting to
`WeekdayLocator`/`DayLocator`.
===============================================================================

Task: 5

Add tests, examples, and documentation covering non-daily support following
existing repository conventions.

1. Add a weekly synthetic fixture (e.g. via `pd.date_range(..., freq='W')`) in
`tests/`, either as a new conftest helper or a small bundled CSV mirroring the
existing daily fixture. Add tests validating segment detection, boundaries, and
durations for weekly spacing, reusing the existing
`assert_segments_match`/`assert_segments_in_a_haystack` helpers. Add a
regression test confirming daily data results remain unchanged.

2. In `tests/test_method_params.py`, add a `@pytest.mark.core` test exercising
both auto-detection and an explicit `freq` override in `method_params`,
asserting on the resulting behavior, and asserting a clear error/warning is
raised for invalid/irregular input.

3. Update documentation:
- Update the `detect_trends` docstring, `README.md`, and `docs/index.md` to
describe supported frequencies and the `freq` option, removing the daily-only
framing.
- Add a weekly-data example notebook under `docs/examples/fundamentals/` and
register it in the `mkdocs.yml` `nav:` section alongside the existing
fundamentals examples.
- Do not hand-edit `CHANGELOG.md`, `docs/whats-new.md`, or the `pyproject.toml`
version, since these are generated by automation; use a `feat:` PR title
targeting the `develop` branch.

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!