Fix/sync cursor integrity

README.md

@@ -266,7 +266,7 @@ The application is stable and production-ready (see **`package.json`** → `vers
 - ✅ **Provider Pattern:** Multi-booru support via `IBooruProvider` interface (Rule34, Gelbooru)
 - ✅ **Rate Limiting:** Intelligent rate limiting with 1.5s delay between artists, 0.5s between pages
 - ✅ **Anti-Bot Measures:** Shared throttling/UA strategy applied across current providers.
-- ⚠️ **Open P0:** sync `lastPostId` cursor integrity — see [Roadmap](./docs/roadmap.md#open-p0-audit--remaining). Video-cache writes are atomic (tmp+rename) with size-capped eviction.
+- ✅ **Sync integrity:** `lastPostId` advances only after complete pagination; unfinished runs set `lastSyncIncomplete`. Video-cache writes are atomic (tmp+rename) with size-capped eviction — see [Roadmap](./docs/roadmap.md).
 
 ### UI/UX
 

docs/api-guide.md

@@ -471,6 +471,7 @@ type Artist = {
   apiEndpoint: string;
   lastPostId: number;
   newPostsCount: number;
+  lastSyncIncomplete: boolean;
   lastChecked: number | null;
   createdAt: number;
 };

docs/architecture.md

@@ -1170,9 +1170,7 @@ sequenceDiagram
      });
    ```
 
-   g. **Updates artist** - Updates artist's `lastPostId` and `newPostsCount`.
-
-   ⚠️ **Known integrity gap (open P0):** `lastPostId` may advance mid-pagination or after a partial error commit. An interrupted sync can leave a cursor ahead of fully persisted posts and skip gaps on the next run. Do not document “cursor = fully synced watermark” until the sync-cursor integrity fix lands.
+   g. **Updates artist** - Mid-batch and error paths update `newPostsCount` (and may set `lastSyncIncomplete`) without moving `lastPostId`. After natural pagination end (`postsData.length < PAGE_SIZE`), a single commit writes `lastPostId`, `lastChecked`, and clears `lastSyncIncomplete`.
 
    h. **Progress event** - Emits IPC event: `emit('sync:progress', 'Syncing artist_name...')`
 

docs/database.md

@@ -80,6 +80,9 @@ export const artists = sqliteTable(
     apiEndpoint: text("api_endpoint").notNull(),
     lastPostId: integer("last_post_id").default(0).notNull(),
     newPostsCount: integer("new_posts_count").default(0).notNull(),
+    lastSyncIncomplete: integer("last_sync_incomplete", { mode: "boolean" })
+      .notNull()
+      .default(false),
     lastChecked: integer("last_checked", { mode: "timestamp" }),
     createdAt: integer("created_at", { mode: "timestamp" })
       .notNull()

docs/glossary.md

@@ -154,9 +154,9 @@ Main-process `VideoProxyServer` serves local `http://127.0.0.1` URLs for `<video
 
 ### Sync cursor (`lastPostId`)
 
-Per-artist watermark used for incremental sync. **Open P0:** advancing the cursor on incomplete pagination/error paths can skip posts. Treat as “best effort” until the integrity fix ships.
+Per-artist watermark for incremental sync (`id:>lastPostId`). Advanced **only** after pagination completes naturally (`postsData.length < PAGE_SIZE`). Mid-batch and error partial commits persist posts and `newPostsCount` but never move the cursor. Incomplete runs set `lastSyncIncomplete` so the next sync can refill gaps.
 
-**Related:** [Architecture — Sync](./architecture.md), [Roadmap — Open P0](./roadmap.md#open-p0-audit--not-yet-shipped)
+**Related:** [Architecture — Sync](./architecture.md), [Roadmap](./roadmap.md)
 
 ---
 

docs/roadmap.md

@@ -148,11 +148,10 @@ The short version: the core product is shipped, now we focus on parity gaps and
 
 ### Open P0 (audit — remaining)
 
-| Item | Hazard | Target branch |
-|------|--------|---------------|
-| **Sync cursor integrity** | `lastPostId` may advance on incomplete sync → skipped posts | `fix/sync-cursor-integrity` |
+None — video-cache and sync-cursor integrity shipped.
 
 - ✅ **Video cache integrity** (`fix/video-cache-integrity`): atomic tmp+rename, abort cleanup, `VIDEO_CACHE_MAX_BYTES` eviction via maintenance tick + deferred start sweep.
+- ✅ **Sync cursor integrity** (`fix/sync-cursor-integrity`): `lastPostId` / `lastChecked` only after natural pagination end; partial post commits keep data; `lastSyncIncomplete` marks unfinished runs; axios network failures rethrow as `ProviderSearchError("network")`.
 
 - ⏳ **Tooling / hygiene:** keep `validate` green; remaining shared validation consolidation as needed. Dev-only audit noise (electron-builder transitive deps) is separate from production `npm audit`.
 
@@ -193,7 +192,7 @@ Items explicitly scheduled for product/engineering (beyond small bugs).
 
 | Area | What is still open |
 |------|--------------------|
-| **P0 integrity** | Sync `lastPostId` only after complete durable batches (video-cache atomic writes shipped). |
+| **P0 integrity** | Shipped: video-cache atomic writes + sync cursor only after complete pagination (`lastSyncIncomplete` for unfinished runs). |
 | **Filters** | Keep filter scope lean (`AI`, `Media`, `Source`) and avoid reintroducing removed panel controls without product decision (**scope is already implemented; this is a guardrail**). |
 | **Search** | Continue polish/regression coverage for chip-based syntax (`-tag`, OR groups, wildcard/fuzzy). |
 | **Navigation & layout** | **Optional** polish: tooltips, item order tuning, and small-window density improvements (see [Navigation, layout, shell](#d-navigation-layout-shell)). |

docs/user-guide.md

@@ -575,7 +575,7 @@ Open **Statistics** from the sidebar to see a quick health overview of your loca
 **Solutions:**
 
 1. Use **Repair** on the artist (resync from the beginning)
-2. Known limitation: `lastPostId` can advance on incomplete sync batches (open P0 — see [roadmap](./roadmap.md#open-p0-audit--not-yet-shipped))
+2. Incomplete runs set `lastSyncIncomplete` and leave `lastPostId` unchanged so the next sync can refill gaps. If gaps persist after a successful complete sync, use Repair.
 
 ### App is slow
 

drizzle/0030_add_artists_last_sync_incomplete.sql

@@ -0,0 +1 @@
+ALTER TABLE artists ADD COLUMN last_sync_incomplete integer DEFAULT 0 NOT NULL;

drizzle/meta/_journal.json

@@ -204,6 +204,13 @@
       "when": 1777405165438,
       "tag": "0029_sparkling_boomer",
       "breakpoints": true
+    },
+    {
+      "idx": 30,
+      "version": "6",
+      "when": 1777800000000,
+      "tag": "0030_add_artists_last_sync_incomplete",
+      "breakpoints": true
     }
   ]
 }

src/main/db/queries/artists.ts

@@ -40,6 +40,7 @@ export function getTrackedArtistsWithStats(db: AppDatabase): TrackedArtistWithSt
       `.as("newPostsCount"),
       syncStatus: artists.syncStatus,
       lastError: artists.lastError,
+      lastSyncIncomplete: artists.lastSyncIncomplete,
       lastChecked: artists.lastChecked,
       createdAt: artists.createdAt,
       postsCount: sql<number>`COALESCE(COUNT(${posts.id}), 0)`.as("postsCount"),

src/main/db/schema.ts

@@ -49,6 +49,10 @@ export const artists = sqliteTable(
       .notNull()
       .default("idle"),
     lastError: text("last_error"),
+    /** 1 when pagination did not finish; cursor must not advance until cleared. */
+    lastSyncIncomplete: integer("last_sync_incomplete", { mode: "boolean" })
+      .notNull()
+      .default(false),
     lastChecked: integer("last_checked", { mode: "timestamp" }),
     createdAt: integer("created_at", { mode: "timestamp" })
       .notNull()

src/main/services/sync-service.ts

@@ -8,7 +8,10 @@ import {
 } from "../utils/decrypted-credentials";
 import { eq, sql } from "drizzle-orm";
 import axios from "axios";
-import { isProviderSearchError } from "../providers/provider-search-errors";
+import {
+  isProviderSearchError,
+  ProviderSearchError,
+} from "../providers/provider-search-errors";
 import type { Artist, NewPost } from "../db/schema";
 import type { BetterSQLite3Database } from "drizzle-orm/better-sqlite3";
 import * as schema from "../db/schema";
@@ -445,6 +448,7 @@ export class SyncService {
     let hasMore = true;
     let newPostsCount = 0;
     let batchHighestPostId = isInitial ? 0 : currentLastPostId;
+    let paginationCompleted = false;
 
     // Batch posts for transaction - collect multiple pages before committing
     // This reduces transaction overhead (better-sqlite3 blocks DB on write)
@@ -488,8 +492,9 @@ export class SyncService {
             ? postsData 
             : postsData.filter((p) => p.id > currentLastPostId);
 
-          // Stop if no new posts found
+          // Stop if no new posts found — known territory reached (incremental complete)
           if (newPosts.length === 0) {
+            paginationCompleted = true;
             hasMore = false;
             break;
           }
@@ -532,7 +537,7 @@ export class SyncService {
 
           if (shouldCommitBatch && allPostsToSave.length > 0) {
             let insertedInBatch = 0;
-            // Single transaction for entire batch
+            // Mid-batch / in-loop commits never advance the sync cursor.
             db.transaction((tx) => {
               const postsCountBefore = countArtistPosts(tx, artist.id);
               bulkUpsertPosts(allPostsToSave, tx);
@@ -541,9 +546,7 @@ export class SyncService {
 
               tx.update(artists)
                 .set({
-                  lastPostId: batchHighestPostId,
                   newPostsCount: sql`${artists.newPostsCount} + ${insertedInBatch}`,
-                  lastChecked: new Date(),
                 })
                 .where(eq(artists.id, artist.id))
                 .run();
@@ -559,6 +562,7 @@ export class SyncService {
 
           // Continue pagination if we got a full page (PAGE_SIZE posts)
           if (postsData.length < PAGE_SIZE) {
+            paginationCompleted = true;
             hasMore = false;
             logger.debug(`SyncService: ${artist.name} - Page ${page} returned ${postsData.length} posts (< ${PAGE_SIZE}), stopping pagination`);
           } else {
@@ -569,48 +573,63 @@ export class SyncService {
           logger.error(`Sync error for ${artist.name}`, e);
           hasMore = false;
 
-          if (allPostsToSave.length > 0) {
-            try {
-              let partialSize = 0;
-              db.transaction((tx) => {
+          try {
+            let partialSize = 0;
+            db.transaction((tx) => {
+              if (allPostsToSave.length > 0) {
                 const postsCountBefore = countArtistPosts(tx, artist.id);
                 bulkUpsertPosts(allPostsToSave, tx);
                 const postsCountAfter = countArtistPosts(tx, artist.id);
                 partialSize = Math.max(0, postsCountAfter - postsCountBefore);
+              }
 
+              if (partialSize > 0) {
                 tx.update(artists)
                   .set({
-                    lastPostId: batchHighestPostId,
                     newPostsCount: sql`${artists.newPostsCount} + ${partialSize}`,
-                    lastChecked: new Date(),
+                    lastSyncIncomplete: true,
                   })
                   .where(eq(artists.id, artist.id))
                   .run();
-              });
+              } else {
+                tx.update(artists)
+                  .set({ lastSyncIncomplete: true })
+                  .where(eq(artists.id, artist.id))
+                  .run();
+              }
+            });
 
+            if (partialSize > 0) {
               newPostsCount += partialSize;
               logger.warn(
                 `SyncService: Partial commit of ${partialSize} posts after error for ${artist.name}`
               );
-            } catch (commitErr) {
-              logger.error(
-                `SyncService: Partial commit failed for ${artist.name}`,
-                commitErr
-              );
             }
+            allPostsToSave.length = 0;
+          } catch (commitErr) {
+            logger.error(
+              `SyncService: Partial commit failed for ${artist.name}`,
+              commitErr
+            );
           }
 
           if (isProviderSearchError(e)) {
-            if (e.kind === "auth" || e.kind === "rate_limit") {
+            if (
+              e.kind === "auth" ||
+              e.kind === "rate_limit" ||
+              e.kind === "network"
+            ) {
               throw e;
             }
-          } else if (!axios.isAxiosError(e)) {
+          } else if (axios.isAxiosError(e)) {
+            throw new ProviderSearchError("network");
+          } else {
             throw e;
           }
         }
       }
 
-      // Commit any remaining posts in batch
+      // Commit any remaining posts in batch (cursor still deferred)
       if (allPostsToSave.length > 0) {
         let insertedInFinalBatch = 0;
         db.transaction((tx) => {
@@ -620,26 +639,43 @@ export class SyncService {
           insertedInFinalBatch = Math.max(0, postsCountAfter - postsCountBefore);
 
           tx.update(artists)
-            .set({
-              lastPostId: batchHighestPostId,
-              newPostsCount: sql`${artists.newPostsCount} + ${insertedInFinalBatch}`,
-              lastChecked: new Date(),
-            })
+            .set(
+              paginationCompleted
+                ? {
+                    newPostsCount: sql`${artists.newPostsCount} + ${insertedInFinalBatch}`,
+                  }
+                : {
+                    newPostsCount: sql`${artists.newPostsCount} + ${insertedInFinalBatch}`,
+                    lastSyncIncomplete: true,
+                  }
+            )
             .where(eq(artists.id, artist.id))
             .run();
         });
 
         newPostsCount += insertedInFinalBatch;
+        allPostsToSave.length = 0;
         logger.debug(
           `SyncService: ${artist.name} - Committed final batch of ${insertedInFinalBatch} new posts`
         );
       }
 
-      // Final update of lastChecked even if no new posts were found
-      if (newPostsCount === 0) {
+      // Sole cursor write path: only after natural pagination end.
+      if (paginationCompleted) {
+        db.transaction((tx) => {
+          tx.update(artists)
+            .set({
+              lastPostId: batchHighestPostId,
+              lastChecked: new Date(),
+              lastSyncIncomplete: false,
+            })
+            .where(eq(artists.id, artist.id))
+            .run();
+        });
+      } else {
         db.transaction((tx) => {
           tx.update(artists)
-            .set({ lastChecked: new Date() })
+            .set({ lastSyncIncomplete: true })
             .where(eq(artists.id, artist.id))
             .run();
         });
@@ -648,7 +684,8 @@ export class SyncService {
       const previousLastPostId = isInitial ? artist.lastPostId : currentLastPostId;
       logger.info(
         `${syncType} sync finished for ${artist.name}. Added: ${newPostsCount} posts. ` +
-        `Final lastPostId: ${batchHighestPostId} (was: ${previousLastPostId})`
+        `Final lastPostId: ${paginationCompleted ? batchHighestPostId : previousLastPostId} ` +
+        `(was: ${previousLastPostId}, paginationCompleted: ${paginationCompleted})`
       );
     } finally {
       if (isInitial) {