Skip to content

chore(infra,web): cap container logs and trim noisy server logs#358

Open
juandelperal wants to merge 1 commit into
peaberry-studio:mainfrom
juandelperal:chore/log-rotation-and-trim-noisy-logs
Open

chore(infra,web): cap container logs and trim noisy server logs#358
juandelperal wants to merge 1 commit into
peaberry-studio:mainfrom
juandelperal:chore/log-rotation-and-trim-noisy-logs

Conversation

@juandelperal

Copy link
Copy Markdown

Summary

Production disk was filling up because Podman keeps container logs forever by default (no size cap on either the json-file or k8s-file driver). A few server actions also write a handful of console.log lines per call on hot polling paths, which compounds the problem.

Container-side (the main fix)

Three places create long-lived containers and none of them passed log options:

  • infra/deploy/ansible/roles/app/templates/compose.yml.j2traefik, postgres, docker-socket-proxy, reaper, flows, cloudflared, and the local-dev web had no logging: block. Added an x-logging: &default-logging anchor (json-file, max-size: 10m, max-file: 3) and applied it to every service. Podman aliases json-file to k8s-file, so max-size is honored on Podman and max-file is honored by Docker.
  • infra/deploy/ansible/roles/app/tasks/main.yml — the production web container is started outside compose via podman run for blue-green deploys. Added --log-driver k8s-file --log-opt max-size=10m.
  • apps/web/src/lib/spawner/docker.ts — every user workspace container is created via dockerode with no HostConfig.LogConfig. With many users this was the largest source of unbounded logs. Added the same 10MB × 3 cap.

Worst-case per-container disk usage is now bounded to ~30MB.

Application-side (chatty loggers)

  • apps/web/src/actions/spawner.ts ensureInstanceRunningAction — was emitting 6 console.log lines per call. This action is polled every 2 seconds while a workspace is starting and on every workspace page load. Removed the trace logs; kept console.error for real failures.
  • apps/web/src/actions/opencode.ts sendMessageAction — was logging 5 lines per message (request/response excerpts). Removed.
  • apps/web/src/lib/opencode/transform.ts[transformParts] Unknown part type was dumping the full part object on every unknown part. Any new OpenCode part type could log multi-KB blobs on every streamed message. Throttled to one console.warn per kind per process.
  • apps/web/src/lib/spawner/core.ts — dropped two verbose info logs about direct-IP healthchecks that fired on every workspace start; kept the warning paths.

Out of scope

Did not touch user workspace (OpenCode) log verbosity — log rotation on those containers is enough to keep disk usage bounded.

Test plan

  • pnpm lint — 0 errors (pre-existing warnings only)
  • pnpm test --run — 4442 passed, 15 skipped, 0 failed
  • YAML anchor &default-logging and 7 *default-logging aliases verified
  • After deploy: podman inspect <container> | jq '.[0].HostConfig.LogConfig' shows { Type: 'json-file', Config: { 'max-size': '10m', 'max-file': '3' } }
  • du -sh /var/lib/containers/storage/overlay-containers/*/userdata/ctr.log stays bounded after a few days under load

In production the host disk was being filled by container logs because
Podman does not rotate logs by default and several server actions were
logging a handful of lines per call on hot polling paths.

- Add a default json-file logging block (10MB x 3 files) to every
  service rendered by compose.yml.j2 (traefik, postgres,
  docker-socket-proxy, reaper, flows, cloudflared, local-dev web).
- Add --log-driver k8s-file --log-opt max-size=10m to the production
  web container started via `podman run` in Ansible.
- Add HostConfig.LogConfig to every user workspace container created
  via dockerode in lib/spawner/docker.ts (this was the largest source
  on a busy host with many workspaces).
- Remove the per-call console.log spam in ensureInstanceRunningAction
  (polled every 2s during workspace start) and in sendMessageAction.
- Drop the verbose direct-IP healthcheck logs in spawner/core.ts.
- De-duplicate the `[transformParts] Unknown part type` log so a new
  unknown OpenCode part type can no longer flood logs once per
  streamed message.
@Inakitajes

Copy link
Copy Markdown
Contributor

Thanks for the contribution! A few small adjustments are needed and it has to be brought up to date with main, but I'll keep working on it and get it merged as soon as possible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants