Overview
Nothing in production code has ever written a row to the quadlets table. Every
feature that reads it therefore operates on an empty set and silently does nothing.
The Monitor glance bar reading 0/0/0 is the most visible symptom, but it is one of
four.
The fix is not to route around the table. It is to give the server an inventory it
owns: a reconciler that scans each registered server's Quadlet directories and keeps
quadlets in sync, with every other component reading from the database rather than
issuing its own scan.
Details
What is broken, and where:
| Reader |
Location |
Consequence of the empty table |
| Stats poller |
services/stats_engine.py:186 |
Glance bar Total/Running/Stopped read 0 on every server |
| External-modification watcher |
services/sync_engine.py:172 |
check_quadlets() iterates an empty list, so the file_changed SSE event has never fired |
| Save collision avoidance |
api/routes.py:669-674 |
The UPDATE quadlets SET last_known_mtime matches zero rows; the protection its comment describes does not exist |
| Delete cleanup |
api/routes.py:1733-1736 |
DELETE FROM quadlets removes nothing |
The only INSERT INTO quadlets in the repository is a test fixture
(tests/test_stats_engine_unit_state.py:66). create_new_quadlet
(api/routes.py:815-860) writes the remote file and never records it. Confirmed
against the development database: 1 registered server, 0 rows in quadlets,
0 rows in unit_state.
Use Case / Steps:
- Register a server that has several
.container Quadlet units, some active, some not.
- Open the Monitor tab and select that server.
- Observe the glance bar: Total, Running and Stopped all read 0.
- Separately, modify one of that server's Quadlet files directly on the host. No
file_changed toast ever appears.
Expected vs Actual:
- Expected: the server knows which Quadlet units exist on each host, keeps that
knowledge current, and pushes it to clients. The glance bar counts real units, and
external edits raise a notification.
- Actual: the server knows nothing. All four features above no-op.
Chosen approach: server-owned inventory, pushed to clients.
The alternative considered was to have the stats poller run its own live find and
fold it into its batched SSH call, leaving the table empty. Rejected: it fixes one of
the four readers and creates a second, divergent source of truth for the same
question, which would disagree with the UI tree whenever one scan fails and the other
succeeds.
Instead:
- A reconciler reuses
tree_scanner.fetch_all_quadlets
(services/tree_scanner.py:40-63), diffs the result against the rows for that
server, inserts new paths and deletes vanished ones, and runs in the sync loop at
POLL_INTERVAL_SEC rather than the 5 second stats interval. Inventory changes on
human timescales.
- Everything else keeps reading the database.
_unit_names_for_scope
(services/stats_engine.py:174-199) and _build_first_command (:278-289) do
not change: their SQL, their .container-only filter and the sentinel batching
are already correct.
- The quadlet tree endpoint stops issuing SSH on the client's request path and
serves from the database, with changes pushed over SSE.
- App-initiated create and delete write their rows immediately, so the UI does not
wait for a reconcile.
Constraints that the sub-issues carry:
- The reconciler must seed
last_known_mtime on first insert from the scan.
check_quadlets() guards on q['last_known_mtime'] is not None
(services/sync_engine.py:216), so a NULL there trades one dead feature for a
quieter one.
- The reconciler must never overwrite
last_known_mtime or last_content_hash on
rows that already exist, or it erases the save-time collision-avoidance write.
- The
quadlets table has no unique constraint on (server_id, file_path)
(core/database.py:135-145), so repeated scans would duplicate rows and an upsert
has nothing to conflict on.
Accepted trade-offs:
- Externally added files appear after up to one poll interval. This is the same
latency the external-modification watcher already assumes, and the SSE push covers
it.
- One extra
find per scope per poll cycle, which replaces the per-tab-click SSH the
UI does today.
Mechanics that are already correct and should not be touched:
build_unit_state_command (services/systemd_manager.py:23-43),
parse_systemctl_show (:46-80), the sentinel batching, the unit_state upsert
(services/stats_engine.py:350-384) and the transition events (:387-439).
Note on test coverage: existing tests cannot catch this. The E2E tests hand-feed
a populated units array into window.handleStatsUpdate
(tests/e2e/test_monitoring_ui.py:47-68, :356-380), and
tests/test_stats_engine_unit_state.py:48-100 INSERTs its own quadlets rows before
calling _unit_names_for_scope. A regression test needs to assert that a frame built
by the engine itself carries a non-empty units list.
Sub-issues
This issue is the umbrella. Work lands through the sub-issues, in order. #282 is
sequenced after the reconciler lands.
Overview
Nothing in production code has ever written a row to the
quadletstable. Everyfeature that reads it therefore operates on an empty set and silently does nothing.
The Monitor glance bar reading 0/0/0 is the most visible symptom, but it is one of
four.
The fix is not to route around the table. It is to give the server an inventory it
owns: a reconciler that scans each registered server's Quadlet directories and keeps
quadletsin sync, with every other component reading from the database rather thanissuing its own scan.
Details
What is broken, and where:
services/stats_engine.py:186services/sync_engine.py:172check_quadlets()iterates an empty list, so thefile_changedSSE event has never firedapi/routes.py:669-674UPDATE quadlets SET last_known_mtimematches zero rows; the protection its comment describes does not existapi/routes.py:1733-1736DELETE FROM quadletsremoves nothingThe only
INSERT INTO quadletsin the repository is a test fixture(
tests/test_stats_engine_unit_state.py:66).create_new_quadlet(
api/routes.py:815-860) writes the remote file and never records it. Confirmedagainst the development database: 1 registered server, 0 rows in
quadlets,0 rows in
unit_state.Use Case / Steps:
.containerQuadlet units, some active, some not.file_changedtoast ever appears.Expected vs Actual:
knowledge current, and pushes it to clients. The glance bar counts real units, and
external edits raise a notification.
Chosen approach: server-owned inventory, pushed to clients.
The alternative considered was to have the stats poller run its own live
findandfold it into its batched SSH call, leaving the table empty. Rejected: it fixes one of
the four readers and creates a second, divergent source of truth for the same
question, which would disagree with the UI tree whenever one scan fails and the other
succeeds.
Instead:
tree_scanner.fetch_all_quadlets(
services/tree_scanner.py:40-63), diffs the result against the rows for thatserver, inserts new paths and deletes vanished ones, and runs in the sync loop at
POLL_INTERVAL_SECrather than the 5 second stats interval. Inventory changes onhuman timescales.
_unit_names_for_scope(
services/stats_engine.py:174-199) and_build_first_command(:278-289) donot change: their SQL, their
.container-only filter and the sentinel batchingare already correct.
serves from the database, with changes pushed over SSE.
wait for a reconcile.
Constraints that the sub-issues carry:
last_known_mtimeon first insert from the scan.check_quadlets()guards onq['last_known_mtime'] is not None(
services/sync_engine.py:216), so a NULL there trades one dead feature for aquieter one.
last_known_mtimeorlast_content_hashonrows that already exist, or it erases the save-time collision-avoidance write.
quadletstable has no unique constraint on(server_id, file_path)(
core/database.py:135-145), so repeated scans would duplicate rows and an upserthas nothing to conflict on.
Accepted trade-offs:
latency the external-modification watcher already assumes, and the SSE push covers
it.
findper scope per poll cycle, which replaces the per-tab-click SSH theUI does today.
Mechanics that are already correct and should not be touched:
build_unit_state_command(services/systemd_manager.py:23-43),parse_systemctl_show(:46-80), the sentinel batching, theunit_stateupsert(
services/stats_engine.py:350-384) and the transition events (:387-439).Note on test coverage: existing tests cannot catch this. The E2E tests hand-feed
a populated
unitsarray intowindow.handleStatsUpdate(
tests/e2e/test_monitoring_ui.py:47-68,:356-380), andtests/test_stats_engine_unit_state.py:48-100INSERTs its ownquadletsrows beforecalling
_unit_names_for_scope. A regression test needs to assert that a frame builtby the engine itself carries a non-empty
unitslist.Sub-issues
This issue is the umbrella. Work lands through the sub-issues, in order. #282 is
sequenced after the reconciler lands.