perf(docker): replace inline SQLite compilation with pre-built image - #20824
Conversation
…mage
The SQLite from-source build was the slowest stage in our Docker builds.
Instead of compiling on every run, this PR publishes a versioned
`ghcr.io/prefecthq/prefect-sqlite:<version>` image once via a dedicated
workflow and pulls it in all subsequent builds.
- Add `Dockerfile.sqlite` — standalone Dockerfile for the builder image
- Add `.github/workflows/sqlite-builder.yaml` — builds and pushes the
pre-compiled image to GHCR; triggers on `Dockerfile`/`Dockerfile.sqlite`
changes to main or via `workflow_dispatch`; skips if the tag already exists
- Update `Dockerfile` and `client/Dockerfile` to use
`FROM ghcr.io/prefecthq/prefect-sqlite:${SQLITE_VERSION} AS sqlite-builder`
instead of the inline compile stage; remove the now-unused `SQLITE_YEAR`
and `SQLITE_FILE_VERSION` ARGs
The `COPY --from=sqlite-builder` lines in both final stages are unchanged.
To update SQLite: bump `SQLITE_VERSION` in `Dockerfile`/`client/Dockerfile`
and run the sqlite-builder workflow.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…qlite-builder - Add ca-certificates to Dockerfile.sqlite-builder so wget can verify SSL when downloading the SQLite tarball - Rename Dockerfile.sqlite to Dockerfile.sqlite-builder to avoid the .sqlite extension being misidentified as a SQLite database file - Update all references in workflow, Dockerfile, and client/Dockerfile Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…uilder image Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Publishes prefecthq/prefect-sqlite to DockerHub using PREFECT_SQLITE_DOCKERHUB_TOKEN instead of GHCR. Removes the GHCR login from docker-images.yaml since the DockerHub repo is public and no longer requires a separate registry auth step to pull. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Temporarily adds feat/sqlite-builder-cache-image to the push trigger so the image is published to DockerHub before the PR is merged. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…LE_VERSION These args only exist in Dockerfile.sqlite-builder, not Dockerfile, so they were resolving to empty strings and causing the wget to fail with a 404. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
desertaxle
added
the
development
desertaxle
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aabc9ea0eb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The sqlite-builder workflow publishes to DockerHub (prefecthq/prefect-sqlite), not GHCR, so the FROM and comment must reference DockerHub to stay consistent. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 003a63699c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…flow_dispatch Previously only sqlite_version was overridable, so a manual dispatch with a new version would still build from the old tarball coordinates. All three args now follow the same pattern: use the input if provided, otherwise read from Dockerfile.sqlite-builder. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5d8d725acb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ersion sync Adds client/Dockerfile to the push paths so a SQLite version bump there triggers the publish workflow. Also fails the build if SQLITE_VERSION differs between Dockerfile and client/Dockerfile, enforcing a single source of truth. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3e197c6cc0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Without this, a PR bumping SQLITE_VERSION references a DockerHub tag that doesn't exist until after merge, deadlocking CI. Adding the pull_request trigger with the same paths filter publishes the image during the PR so subsequent Docker build jobs can pull it successfully. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 26d8f75ec4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…f tag exists The existence check now only short-circuits on pull_request events so that a push to main (after merge) always rebuilds and retags, ensuring the published image reflects the final merged source rather than an earlier PR commit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 32c6d32259
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Fork pull_requests don't have access to repository secrets, so the DockerHub login would fail for any new version tag. Detect this case early and skip the build — the push-to-main event after merge will do the actual publish. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bf93afeafb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Fork PRs can't publish to DockerHub, so a new SQLITE_VERSION tag doesn't exist until after merge. This adds a sqlite-from-source build stage to both Dockerfiles (selected via --build-arg SQLITE_SOURCE=from-source) that compiles SQLite inline, bypassing the DockerHub dependency. python-tests.yaml now detects fork PRs and passes SQLITE_SOURCE=from-source automatically, so their CI Docker builds pass without requiring a pre-published image. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: abfb8625e8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… PRs The benchmark workflow builds Dockerfile on pull_request but had no fallback for fork PRs bumping SQLITE_VERSION to an unpublished tag. Add the same fork detection logic used in python-tests.yaml. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8bfc96759c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…atch The previous existence check meant subsequent commits on a same-repo PR would skip rebuilding, leaving CI validating against a stale image. Now only fork PRs (which lack secrets) skip the build; all other triggers always rebuild to ensure the published tag reflects the current branch. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d4af171eac
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…mage The DockerHub image is public so no credentials are needed to pull it. Fork PRs can pull the image just fine; they just can't push new tags. Remove the SQLITE_SOURCE build arg and from-source fallback stages from Dockerfile and client/Dockerfile, revert python-tests.yaml and time-docker-build.yaml to their original build-args, and clean up the duplicate comment in sqlite-builder.yaml. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1e85f48aa2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Removes push and pull_request triggers. The image is published manually when the SQLite version needs to be bumped. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 133ddead9a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Prevents publishing a mislabeled image by checking that SQLITE_VERSION matches across Dockerfile, client/Dockerfile, and Dockerfile.sqlite-builder before the build runs. A bump that updates the tag but not the build args (SQLITE_YEAR/SQLITE_FILE_VERSION) will be caught here. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 06de3b964a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
zzstoatzz
left a comment
zzstoatzz
left a comment
There was a problem hiding this comment.
wow, is all the comments just codex getting invoked on commit?
Yeah, I might've marked this as ready to review too soon. |
Add 7 newly merged PRs to the release notes: - #20869: ProcessPoolTaskRunner subprocess message processors - #20876: Fix silently lost subprocess events - #20824: Pre-built SQLite Docker image - #20870-#20873: UI bug fixes (breadcrumbs, null indicators, locale formatting) Co-authored-by: bot_apk <apk@cognition.ai>
…refectHQ#20824) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
3 participants
This PR replaces the slowest stage in our Docker builds — compiling SQLite from source — with a pre-built image pulled from DockerHub.
How it works
The SQLite compilation only needs to happen once per version, not on every
docker build. We now publish a versionedprefecthq/prefect-sqlite:<version>image to DockerHub that bothDockerfileandclient/Dockerfilepull as a named stage. TheCOPY --from=sqlite-builderlines in the final stages are completely unchanged.To update SQLite in the future:
SQLITE_VERSIONinDockerfile,client/Dockerfile, andDockerfile.sqlite-buildersqlite-builderworkflow manually viaworkflow_dispatchto publish the new imageChanges
Dockerfile.sqlite-builder— new standalone Dockerfile that builds SQLite from source and installs it to/usr/local; published asprefecthq/prefect-sqlite:<version>on DockerHub forlinux/amd64andlinux/arm64.github/workflows/sqlite-builder.yaml— new workflow that builds and pushesprefecthq/prefect-sqlite:<version>; triggered manually viaworkflow_dispatchwith optional version overrides; skips if the tag already existsDockerfile— removes inline SQLite compile stage, replaces withFROM prefecthq/prefect-sqlite:${SQLITE_VERSION} AS sqlite-builder; removes now-unusedSQLITE_YEARandSQLITE_FILE_VERSIONARGsclient/Dockerfile— same changes as above