Fix/tag resolve miss persistence

.ai/LESSONS.txt

@@ -114,6 +114,14 @@ Invariant Rule: Run `npm run validate` before finishing and remove all unused im
 **Root Cause:** Smart playlist queries were stored as raw JSON without schema versioning, so parser changes could break old records or misinterpret fields.
 **Invariant Rule:** Any JSON-in-column stored as user data MUST include a version column and be parsed only through a versioned resolver, never through raw JSON.parse.
 
+### Problem: Negative cache died with the process while rate-limit failures risked being cached as absent
+**Root Cause:** Tag resolve outcomes collapsed to found/miss; confirmed empty API responses lived only in an in-memory Map (TTL), and unresolved failures (429/network) could be confused with not_found if classification stayed binary.
+**Invariant Rule:** Cacheable outcomes of external lookups distinguish `not_found` (persist with TTL in the same table as hits) from `unresolved` (do not persist as absence). Never mix the two — mixing permanently corrupts categorization. One source of truth for negative state (SQLite), no split-brain RAM Map alongside it.
+
+### Problem: Raw-SQL maintenance deleted every fresh not_found row within seconds of startup
+**Root Cause:** Drizzle `mode: "timestamp"` stores seconds while maintenance DELETE compared `resolved_at` to a `Date.now()`-based millisecond cutoff — seconds are always below that cutoff, so the first tick wiped the entire negative cache.
+**Invariant Rule:** Timestamp columns touched by raw SQL use `mode: "timestamp_ms"` with an explicit unit comment in the schema. Any raw-SQL time comparison MUST have a test that writes via the ORM path and asserts the row survives/expires against the production DELETE helper — never a hand-copied SQL string alone.
+
 ## Security (credentials)
 
 ### Problem: API decrypt failure sends ciphertext to Booru APIs

docs/api-guide.md

@@ -1283,8 +1283,9 @@ const artistTags = await window.api.resolveTags(["tag1", "tag2", "tag3"]);
 - Missing tags are fetched one at a time via `name=` (not batched `names=`).
 - All tag lookups share the same `ProviderThrottle` instance as `fetchPosts`.
 - Concurrent IPC calls for the same tag share one in-flight promise (cross-call dedup).
-- HTTP 429 is retried with `Retry-After` or exponential backoff; rate-limited tags are **not** written to `tag_metadata`.
-- Confirmed empty API responses are negatively cached in memory (24h TTL) to avoid hammering nonexistent tags.
+- HTTP 429 is retried with `Retry-After` or exponential backoff; rate-limited / network failures are **unresolved** — not written to `tag_metadata` (must not be confused with `not_found`).
+- Confirmed empty API responses (`not_found`) are persisted in `tag_metadata` with `status='not_found'` and `resolved_at` in **milliseconds** (`mode: "timestamp_ms"`, TTL `TAG_RESOLVE_NOT_FOUND_TTL_MS`, 7 days). Expired rows are cache-misses; maintenance DELETEs them via `deleteExpiredNotFoundTagMetadata`.
+
 
 ---
 

docs/architecture.md

@@ -1614,7 +1614,7 @@ src/
 │   │   ├── maintenance-scheduler.ts # Daily checkpoint/optimize scheduler
 │   │   ├── MaintenanceService.ts   # User-triggered VACUUM status/run logic
 │   │   ├── updater-service.ts      # Auto-updater service
-│   │   ├── tag-resolve-coordinator.ts # Tag metadata resolve dedup / rate limit
+│   │   ├── tag-resolve-coordinator.ts # Tag metadata resolve: found/not_found persist, unresolved not cached
 │   │   └── video-proxy-server.ts   # Local video proxy + disk cache
 │   ├── workers/                   # Worker threads
 │   │   ├── downloadWorker.ts       # Batch download worker
@@ -1738,7 +1738,7 @@ Root:
 
 **Database & Schema:**
 
-- **Schema:** Core tables `artists`, `posts`, `settings`; also `tag_metadata`, `playlists`, `playlist_entries`, and FTS5 for post tags
+- **Schema:** Core tables `artists`, `posts`, `settings`; also `tag_metadata` (`status` found|not_found + `resolved_at` TTL for misses), `playlists`, `playlist_entries`, and FTS5 for post tags
 - **Migrations:** Fully functional migration system using `drizzle-kit` 0.30+ (`drizzle.config.ts`, `npm run db:generate` / `db:migrate`)
 - **Testing & CI:** Vitest (unit, integration, property), Playwright (E2E); CI runs `validate`, `npm test`, and production `npm audit`
 - **Indexes:** Optimized indexes on `artistId`, `isViewed`, `publishedAt`, `isFavorited`, `lastChecked`, `createdAt`

docs/database.md

@@ -271,6 +271,28 @@ export type Settings = typeof settings.$inferSelect;
 export type NewSettings = typeof settings.$inferInsert;
 ```
 
+### Table: `tag_metadata`
+
+Persistent cache for Rule34 tag type resolution (viewer TagsDrawer / resolve IPC). One row per tag name.
+
+| Column         | Type                              | Description                                                                 |
+| -------------- | --------------------------------- | --------------------------------------------------------------------------- |
+| `name`         | TEXT (PK, NOT NULL)               | Tag string (lowercase provider form)                                        |
+| `type`         | INTEGER (NOT NULL)                | Rule34 tag type for `found` rows; placeholder (`0`) when `status=not_found` |
+| `status`       | TEXT (NOT NULL, DEFAULT `found`)  | `found` or `not_found` — never store unresolved (429/network) as `not_found` |
+| `resolved_at`  | INTEGER (TIMESTAMP_MS, NOT NULL)  | When this outcome was written (**milliseconds**); TTL gate for `not_found`  |
+
+
+**Semantics:**
+
+- `found` — API returned the tag; used for categorization.
+- `not_found` — API answered successfully with empty/miss; TTL `TAG_RESOLVE_NOT_FOUND_TTL_MS` (7 days). Expired rows are treated as cache-miss (re-resolve); maintenance DELETEs them.
+- Unresolved failures are **not** written — they remain candidates for the next session.
+
+**Indexes:**
+
+- `tag_metadata_type_idx` — Index on `type` for filtering (e.g. all artists)
+
 ### Table: `playlists`
 
 Stores playlist/collection information for curated post collections.

drizzle/0031_tag_metadata_status.sql

@@ -0,0 +1,3 @@
+ALTER TABLE `tag_metadata` ADD `status` text DEFAULT 'found' NOT NULL;--> statement-breakpoint
+ALTER TABLE `tag_metadata` ADD `resolved_at` integer DEFAULT 0 NOT NULL;--> statement-breakpoint
+UPDATE `tag_metadata` SET `resolved_at` = CAST((strftime('%s', 'now') * 1000) AS INTEGER) WHERE `resolved_at` = 0;

drizzle/meta/_journal.json

@@ -211,6 +211,13 @@
       "when": 1777800000000,
       "tag": "0030_add_artists_last_sync_incomplete",
       "breakpoints": true
+    },
+    {
+      "idx": 31,
+      "version": "6",
+      "when": 1777900000000,
+      "tag": "0031_tag_metadata_status",
+      "breakpoints": true
     }
   ]
 }

src/main/config/tag-resolve-constants.ts

@@ -1,5 +1,8 @@
-/** In-memory TTL for tags confirmed absent from Rule34 tag DAPI (well-formed empty). */
-export const TAG_RESOLVE_NEGATIVE_CACHE_TTL_MS = 24 * 60 * 60 * 1000;
+/**
+ * SQLite TTL for confirmed not_found tag_metadata rows.
+ * Expired rows are treated as cache-misses (re-resolve); maintenance DELETEs them.
+ */
+export const TAG_RESOLVE_NOT_FOUND_TTL_MS = 7 * 24 * 60 * 60 * 1000;
 
 /** HTTP timeout for a single tag metadata lookup. */
 export const TAG_RESOLVE_REQUEST_TIMEOUT_MS = 10_000;

src/main/db/queries/tag-metadata.ts

@@ -0,0 +1,25 @@
+import type Database from "better-sqlite3";
+import { TAG_RESOLVE_NOT_FOUND_TTL_MS } from "../../config/tag-resolve-constants";
+
+type SqliteDatabase = InstanceType<typeof Database>;
+
+/**
+ * Delete expired not_found rows from tag_metadata.
+ * `resolved_at` is stored in milliseconds (schema mode timestamp_ms).
+ * Cutoff uses Date.now()-based ms — must stay aligned with Drizzle writes.
+ *
+ * @returns number of deleted rows
+ */
+export function deleteExpiredNotFoundTagMetadata(
+  sqlite: SqliteDatabase,
+  nowMs: number = Date.now()
+): number {
+  const cutoffMs = nowMs - TAG_RESOLVE_NOT_FOUND_TTL_MS;
+  const result = sqlite
+    .prepare(
+      `DELETE FROM tag_metadata
+       WHERE status = 'not_found' AND resolved_at < ?`
+    )
+    .run(cutoffMs);
+  return result.changes;
+}

src/main/db/schema.ts

@@ -28,6 +28,10 @@ export const TAG_TYPES = {
 
 export type TagType = (typeof TAG_TYPES)[keyof typeof TAG_TYPES];
 
+/** Cache outcome for tag_metadata rows (unresolved API failures are never stored). */
+export const TAG_METADATA_STATUSES = ["found", "not_found"] as const;
+export type TagMetadataStatus = (typeof TAG_METADATA_STATUSES)[number];
+
 // Settings ID constant for single profile design
 export const SETTINGS_ID = 1;
 
@@ -156,6 +160,13 @@ export const tagMetadata = sqliteTable(
   {
     name: text("name").primaryKey(),
     type: integer("type").notNull(), // Use TAG_TYPES constants: 0=General, 1=Artist, 3=Copyright, 4=Character, 5=Meta
+    status: text("status", { enum: TAG_METADATA_STATUSES })
+      .notNull()
+      .default("found"),
+    // Units: milliseconds since epoch. Raw SQL (maintenance DELETE) compares with Date.now()-based cutoffs.
+    resolvedAt: integer("resolved_at", { mode: "timestamp_ms" })
+      .notNull()
+      .$defaultFn(() => new Date()),
   },
   (t) => ({
     typeIdx: index("tag_metadata_type_idx").on(t.type), // Index for filtering by type (e.g., all artists)

src/main/ipc/controllers/SearchController.ts

@@ -3,7 +3,11 @@ import log from "electron-log";
 import { z } from "zod";
 import { BaseController } from "../../core/ipc/BaseController";
 import { container, DI_TOKENS } from "../../core/di/Container";
-import { settings, SETTINGS_ID, posts, tagMetadata, TAG_TYPES, type TagType } from "../../db/schema";
+import { settings, SETTINGS_ID, posts, TAG_TYPES, type TagType } from "../../db/schema";
+import {
+  loadTagMetadataCache,
+  resolveTagMetadataWave,
+} from "../../services/tag-resolve-coordinator";
 import { eq, inArray, and } from "drizzle-orm";
 import { getProvider } from "../../providers";
 import { IPC_CHANNELS } from "../channels";
@@ -26,7 +30,6 @@ import {
 } from "../../../shared/utils/provider-tag-sanitize";
 import { getAllBlacklistedTags } from "../../db/queries/blacklist";
 import { getDecryptedCredentialsFromRecord } from "../../utils/decrypted-credentials";
-import { resolveTagMetadataWave } from "../../services/tag-resolve-coordinator";
 import type { ProviderSettings } from "../../providers/types";
 import {
   isProviderSearchError,
@@ -171,19 +174,6 @@ export class SearchController extends BaseController {
     ];
   }
 
-  private loadCachedTagMap(
-    db: AppDatabase,
-    uniqueTags: string[]
-  ): Map<string, number> {
-    const cachedTags = db
-      .select()
-      .from(tagMetadata)
-      .where(inArray(tagMetadata.name, uniqueTags))
-      .all();
-
-    return new Map(cachedTags.map((t) => [t.name, t.type]));
-  }
-
   private toProviderSettings(
     settings: NonNullable<Awaited<ReturnType<SearchController["getDecryptedSettings"]>>>
   ): ProviderSettings {
@@ -209,25 +199,25 @@ export class SearchController extends BaseController {
       }
 
       const db = this.getDb();
-      const cachedMap = this.loadCachedTagMap(db, uniqueTags);
+      const cache = loadTagMetadataCache(db, uniqueTags);
 
       const settings = await this.getDecryptedSettings();
       if (!settings) {
         log.warn(
           `[SearchController] Cannot resolve tags (${context}): no settings available`
         );
-        return uniqueTags.filter((tag) => cachedMap.get(tag) === tagType);
+        return uniqueTags.filter((tag) => cache.foundTypes.get(tag) === tagType);
       }
 
       await resolveTagMetadataWave(
         db,
         uniqueTags,
-        cachedMap,
+        cache,
         this.toProviderSettings(settings),
         context
       );
 
-      return uniqueTags.filter((tag) => cachedMap.get(tag) === tagType);
+      return uniqueTags.filter((tag) => cache.foundTypes.get(tag) === tagType);
     } catch (error) {
       log.error(`[SearchController] Failed to resolve tags (${context}):`, error);
       return [];

src/main/services/maintenance-scheduler.ts

@@ -1,5 +1,6 @@
 import log from "electron-log";
 import { getSqliteInstance } from "../db/client";
+import { deleteExpiredNotFoundTagMetadata } from "../db/queries/tag-metadata";
 import type { VideoProxyServer } from "./video-proxy-server";
 
 const STARTUP_DELAY_MS = 10_000;
@@ -43,6 +44,14 @@ export class MaintenanceScheduler {
         // PRAGMA/VACUUM: no Drizzle equivalent, raw SQL required
         sqlite.exec("PRAGMA wal_checkpoint(PASSIVE);");
         sqlite.exec("PRAGMA optimize;");
+
+        const deletedExpiredNotFound = deleteExpiredNotFoundTagMetadata(sqlite);
+        if (deletedExpiredNotFound > 0) {
+          log.info(
+            `[MaintenanceScheduler] Deleted ${deletedExpiredNotFound} expired not_found tag_metadata rows`
+          );
+        }
+
         log.info(`[MaintenanceScheduler] Maintenance complete (trigger=${trigger})`);
       } catch (error) {
         log.error("[MaintenanceScheduler] Maintenance failed:", error);