Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions crates/crw-server/src/monitor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@
//! from the already-built [`AppState`], then spawns the background scheduler.
//! All monitor endpoints/scheduling live behind `#[cfg(feature = "monitor")]`
//! so the default open-core build never links the SQLite/HMAC/cron stack.
//!
//! ponytail: `boot()` is NOT called from server startup today — it is the
//! wiring stub for a deferred self-host feature (no HTTP/CLI surface, email is
//! an `EmailStub`). A `--features monitor` build links the library but does not
//! run a scheduler until something calls this. Wire it to a config toggle +
//! a minimal `/v1/monitor` route only when a self-hoster actually asks. See #142.

use crate::state::AppState;
use crw_monitor::config::MonitorConfig;
Expand Down
8 changes: 4 additions & 4 deletions docs/docs/monitoring.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<div class="page-intro">
<div class="page-kicker">More APIs</div>
<h1>Monitoring</h1>
<p class="page-subtitle">Schedule recurring scrapes or crawls, detect when a page actually changes, and notify your agent by signed webhook or email — with a structured diff and an optional LLM judge that filters out noise. A self-hostable, Firecrawl-compatible alternative to <code>/monitor</code>.</p>
<p class="page-subtitle">Schedule recurring scrapes or crawls, detect when a page actually changes, and notify your agent by signed webhook or email — with a structured diff and an optional LLM judge that filters out noise. A Firecrawl-compatible alternative to <code>/monitor</code>, with self-hostable change detection.</p>
<div class="page-capabilities">
<div class="page-capability"><strong>Best for:</strong> change detection on pages you rely on</div>
<div class="page-capability"><strong>Hosted:</strong> fastcrw.com (full scheduler + notifications)</div>
Expand All @@ -21,7 +21,7 @@ Use monitoring when you need to know the moment a page changes and only care abo
Reach for `monitoring` instead of polling `/v1/scrape` yourself when you want the schedule, snapshot storage, diffing, retries, and noise filtering handled for you.

:::note
**Self-hosted users**: the full scheduler + notification control plane is part of the hosted product. The open-core engine ships the **stateless `changeTracking` primitive** (diff one scrape against a snapshot you supply) plus an optional, feature-gated **`monitor` mode** (SQLite scheduler, default OFF). See [self-hosting monitoring](#monitoring) below.
**Self-hosted users**: the managed scheduler, notifications, and billing are part of the hosted product. The open-core engine ships the **stateless `changeTracking` primitive** (diff one scrape against a snapshot you supply) and `/v1/change-tracking/diff`, both default-on, to drive your own scheduling loop. An experimental, feature-gated **`monitor` Cargo feature** (default OFF) ships a SQLite scheduler library, but it is not yet wired to a runnable server surface — no monitor management API or CLI ships in the open-core build. See [self-hosting monitoring](#monitoring) below.
:::

:::info
Expand All @@ -31,7 +31,7 @@ Reach for `monitoring` instead of polling `/v1/scrape` yourself when you want th

**Engine endpoints** (scrape, crawl, map, search, change-tracking) are served at `https://api.fastcrw.com` on the hosted product and at your own origin when self-hosting.

**Self-hosted installs** have no monitor API. Use the stateless `changeTracking` primitive or build the engine with the optional `monitor` Cargo feature (SQLite scheduler) as described in [Self-hosting monitoring](#self-hosting-monitoring) below.
**Self-hosted installs have no monitor management API.** Use the stateless `changeTracking` primitive and `/v1/change-tracking/diff` with your own scheduler. (An experimental, feature-gated `monitor` SQLite scheduler library exists in-tree but is not yet wired to a server or CLI — see [Self-hosting monitoring](#self-hosting-monitoring) below.)
:::

## Endpoints
Expand Down Expand Up @@ -244,7 +244,7 @@ The open-core engine gives self-hosters the building blocks:

- **`changeTracking` scrape format** — add it to `/v1/scrape` `formats` with the diff `modes` and a `previous` snapshot you persist between checks. opencore is stateless: it returns the diff + the new snapshot for you to store.
- **`POST /v1/change-tracking/diff`** — diff a page (or a batch) against a supplied `previous` snapshot. The workhorse for crawl-based monitoring.
- **Optional `monitor` mode** — build the engine with the `monitor` Cargo feature (default OFF) for a SQLite-backed scheduler, set-level `new`/`removed`, an LLM judge, and signed local webhooks, with no external database.
- **Experimental `monitor` library (feature-gated, default OFF)** — the `monitor` Cargo feature links an in-tree SQLite scheduler with set-level `new`/`removed`, an LLM judge, and HMAC-signed local webhooks, with no external database. It currently has **no HTTP or CLI surface** and email delivery is a stub, so it is not a drop-in self-host monitor yet — the intended integration point is the library API (`Store`, `Scheduler`, `run_check`). For self-host change detection today, use `changeTracking` + `/v1/change-tracking/diff` with your own cron.

```bash
# diff the current scrape against your stored snapshot
Expand Down
Loading