chore(infra,web): cap container logs and trim noisy server logs#358
Open
juandelperal wants to merge 1 commit into
Open
chore(infra,web): cap container logs and trim noisy server logs#358juandelperal wants to merge 1 commit into
juandelperal wants to merge 1 commit into
Conversation
In production the host disk was being filled by container logs because Podman does not rotate logs by default and several server actions were logging a handful of lines per call on hot polling paths. - Add a default json-file logging block (10MB x 3 files) to every service rendered by compose.yml.j2 (traefik, postgres, docker-socket-proxy, reaper, flows, cloudflared, local-dev web). - Add --log-driver k8s-file --log-opt max-size=10m to the production web container started via `podman run` in Ansible. - Add HostConfig.LogConfig to every user workspace container created via dockerode in lib/spawner/docker.ts (this was the largest source on a busy host with many workspaces). - Remove the per-call console.log spam in ensureInstanceRunningAction (polled every 2s during workspace start) and in sendMessageAction. - Drop the verbose direct-IP healthcheck logs in spawner/core.ts. - De-duplicate the `[transformParts] Unknown part type` log so a new unknown OpenCode part type can no longer flood logs once per streamed message.
Contributor
|
Thanks for the contribution! A few small adjustments are needed and it has to be brought up to date with |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Production disk was filling up because Podman keeps container logs forever by default (no size cap on either the
json-fileork8s-filedriver). A few server actions also write a handful ofconsole.loglines per call on hot polling paths, which compounds the problem.Container-side (the main fix)
Three places create long-lived containers and none of them passed log options:
infra/deploy/ansible/roles/app/templates/compose.yml.j2—traefik,postgres,docker-socket-proxy,reaper,flows,cloudflared, and the local-devwebhad nologging:block. Added anx-logging: &default-logginganchor (json-file,max-size: 10m,max-file: 3) and applied it to every service. Podman aliasesjson-filetok8s-file, somax-sizeis honored on Podman andmax-fileis honored by Docker.infra/deploy/ansible/roles/app/tasks/main.yml— the production web container is started outside compose viapodman runfor blue-green deploys. Added--log-driver k8s-file --log-opt max-size=10m.apps/web/src/lib/spawner/docker.ts— every user workspace container is created via dockerode with noHostConfig.LogConfig. With many users this was the largest source of unbounded logs. Added the same 10MB × 3 cap.Worst-case per-container disk usage is now bounded to ~30MB.
Application-side (chatty loggers)
apps/web/src/actions/spawner.tsensureInstanceRunningAction— was emitting 6console.loglines per call. This action is polled every 2 seconds while a workspace is starting and on every workspace page load. Removed the trace logs; keptconsole.errorfor real failures.apps/web/src/actions/opencode.tssendMessageAction— was logging 5 lines per message (request/response excerpts). Removed.apps/web/src/lib/opencode/transform.ts—[transformParts] Unknown part typewas dumping the full part object on every unknown part. Any new OpenCode part type could log multi-KB blobs on every streamed message. Throttled to oneconsole.warnper kind per process.apps/web/src/lib/spawner/core.ts— dropped two verbose info logs about direct-IP healthchecks that fired on every workspace start; kept the warning paths.Out of scope
Did not touch user workspace (OpenCode) log verbosity — log rotation on those containers is enough to keep disk usage bounded.
Test plan
pnpm lint— 0 errors (pre-existing warnings only)pnpm test --run— 4442 passed, 15 skipped, 0 failed&default-loggingand 7*default-loggingaliases verifiedpodman inspect <container> | jq '.[0].HostConfig.LogConfig'shows{ Type: 'json-file', Config: { 'max-size': '10m', 'max-file': '3' } }du -sh /var/lib/containers/storage/overlay-containers/*/userdata/ctr.logstays bounded after a few days under load