Skip to content

Nothing ever writes the quadlets table, so four features silently no-op #281

Description

@karoltheguy

Overview

Nothing in production code has ever written a row to the quadlets table. Every
feature that reads it therefore operates on an empty set and silently does nothing.
The Monitor glance bar reading 0/0/0 is the most visible symptom, but it is one of
four.

The fix is not to route around the table. It is to give the server an inventory it
owns: a reconciler that scans each registered server's Quadlet directories and keeps
quadlets in sync, with every other component reading from the database rather than
issuing its own scan.

Details

What is broken, and where:

Reader Location Consequence of the empty table
Stats poller services/stats_engine.py:186 Glance bar Total/Running/Stopped read 0 on every server
External-modification watcher services/sync_engine.py:172 check_quadlets() iterates an empty list, so the file_changed SSE event has never fired
Save collision avoidance api/routes.py:669-674 The UPDATE quadlets SET last_known_mtime matches zero rows; the protection its comment describes does not exist
Delete cleanup api/routes.py:1733-1736 DELETE FROM quadlets removes nothing

The only INSERT INTO quadlets in the repository is a test fixture
(tests/test_stats_engine_unit_state.py:66). create_new_quadlet
(api/routes.py:815-860) writes the remote file and never records it. Confirmed
against the development database: 1 registered server, 0 rows in quadlets,
0 rows in unit_state.

Use Case / Steps:

  1. Register a server that has several .container Quadlet units, some active, some not.
  2. Open the Monitor tab and select that server.
  3. Observe the glance bar: Total, Running and Stopped all read 0.
  4. Separately, modify one of that server's Quadlet files directly on the host. No
    file_changed toast ever appears.

Expected vs Actual:

  • Expected: the server knows which Quadlet units exist on each host, keeps that
    knowledge current, and pushes it to clients. The glance bar counts real units, and
    external edits raise a notification.
  • Actual: the server knows nothing. All four features above no-op.

Chosen approach: server-owned inventory, pushed to clients.

The alternative considered was to have the stats poller run its own live find and
fold it into its batched SSH call, leaving the table empty. Rejected: it fixes one of
the four readers and creates a second, divergent source of truth for the same
question, which would disagree with the UI tree whenever one scan fails and the other
succeeds.

Instead:

  1. A reconciler reuses tree_scanner.fetch_all_quadlets
    (services/tree_scanner.py:40-63), diffs the result against the rows for that
    server, inserts new paths and deletes vanished ones, and runs in the sync loop at
    POLL_INTERVAL_SEC rather than the 5 second stats interval. Inventory changes on
    human timescales.
  2. Everything else keeps reading the database. _unit_names_for_scope
    (services/stats_engine.py:174-199) and _build_first_command (:278-289) do
    not change: their SQL, their .container-only filter and the sentinel batching
    are already correct.
  3. The quadlet tree endpoint stops issuing SSH on the client's request path and
    serves from the database, with changes pushed over SSE.
  4. App-initiated create and delete write their rows immediately, so the UI does not
    wait for a reconcile.

Constraints that the sub-issues carry:

  • The reconciler must seed last_known_mtime on first insert from the scan.
    check_quadlets() guards on q['last_known_mtime'] is not None
    (services/sync_engine.py:216), so a NULL there trades one dead feature for a
    quieter one.
  • The reconciler must never overwrite last_known_mtime or last_content_hash on
    rows that already exist, or it erases the save-time collision-avoidance write.
  • The quadlets table has no unique constraint on (server_id, file_path)
    (core/database.py:135-145), so repeated scans would duplicate rows and an upsert
    has nothing to conflict on.

Accepted trade-offs:

  • Externally added files appear after up to one poll interval. This is the same
    latency the external-modification watcher already assumes, and the SSE push covers
    it.
  • One extra find per scope per poll cycle, which replaces the per-tab-click SSH the
    UI does today.

Mechanics that are already correct and should not be touched:
build_unit_state_command (services/systemd_manager.py:23-43),
parse_systemctl_show (:46-80), the sentinel batching, the unit_state upsert
(services/stats_engine.py:350-384) and the transition events (:387-439).

Note on test coverage: existing tests cannot catch this. The E2E tests hand-feed
a populated units array into window.handleStatsUpdate
(tests/e2e/test_monitoring_ui.py:47-68, :356-380), and
tests/test_stats_engine_unit_state.py:48-100 INSERTs its own quadlets rows before
calling _unit_names_for_scope. A regression test needs to assert that a frame built
by the engine itself carries a non-empty units list.

Sub-issues

This issue is the umbrella. Work lands through the sub-issues, in order. #282 is
sequenced after the reconciler lands.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions