feat(sql): load remote SQLite and DuckDB databases

docs/changelog.md

@@ -2,6 +2,9 @@
 
 ## in progress
 
+- Database: Load SQLite and DuckDB databases that live on a remote filesystem,
+  addressed by their plain object URI, for example
+  `s3://analytics/snapshots/events.duckdb`. Thanks, @hampsterx.
 - Filesystem: Treat XLSX and ODS workbooks as multi-table sources by default,
   while preserving one-table loads through explicit worksheet selectors.
   Thanks, @hampsterx.

docs/decisions/ADR-001-stage-remote-file-databases.md

@@ -0,0 +1,85 @@
+# ADR-001: Stage remote file databases before SQL ingestion
+
+**Status**: Proposed
+**Date**: 2026-08-03
+
+## Context
+
+SQLite and DuckDB require random access to a database file through their
+existing SQLAlchemy integrations. Object-storage streams do not provide a
+shared database interface that both engines can open directly.
+
+Remote database access needs the same URI and lifecycle semantics across
+supported storage transports. Storage credentials must remain separate from
+logged object locations, and the database file must remain available while dlt
+constructs and consumes its lazy SQL resource.
+
+## Decision
+
+We materialize remote SQLite and DuckDB source files into a run-scoped local
+temporary directory before invoking the existing SQL source path.
+
+A remote database is addressed by its plain object URI, the same carrier the
+filesystem sources read files from, with storage options as its query
+parameters. A filesystem source URI whose object name carries a database
+extension routes to the SQL source rather than to a format reader, so
+`--source-table` keeps selecting a table inside the database. Transport support
+covers the storage schemes that share the repository's existing filesystem
+credential parsing and emulator coverage: `s3://`, `r2://`, `gs://`, `az://`,
+`adls://`, and `abfss://`.
+
+The engine is resolved from the staged file's header, with an unambiguous
+extension as the fallback, because `.db` names both engines and an object's name
+is not evidence of its contents.
+
+The staging operation checks the remote object's reported size and available
+disk space, streams the object while counting bytes, and keeps the local copy
+alive through the complete ingestion run. It removes the copy on normal
+completion and exception exits. The staged database is source-only, and local
+changes are never written back to object storage.
+
+## Alternatives considered
+
+- **Carry the object URI in a `location` query parameter of the SQL URI**
+  (`duckdb:///?location=<encoded-object-uri>&<storage-options>`): rejected
+  because it makes the caller encode a URI inside a URI for a location the
+  storage scheme already names unambiguously.
+- **Stack storage schemes inside the SQL URI**: rejected because nested URI
+  authorities, query strings, and credential ownership are ambiguous to
+  standard URL parsers.
+- **Decode database files through a filesystem reader**: rejected because the
+  SQL source already owns reflection, chunking, type mapping, and table
+  selection. A reader would reimplement them one file format at a time.
+- **Use DuckDB `httpfs` or `ATTACH` as the primary path**: rejected because it
+  does not provide the same mechanism for SQLite. Engine-specific fast paths
+  can be added later without changing the public URI grammar.
+- **Expose object storage through an engine-specific VFS**: rejected because
+  there is no shared, maintained random-access interface for both engines and
+  all three launch transports.
+- **Keep a persistent local cache**: rejected because cache invalidation,
+  credential boundaries, and stale-object behavior require a separate policy.
+  Run-scoped staging has explicit ownership and cleanup.
+- **Write staged database changes back remotely**: rejected because concurrent
+  writers and atomic replacement require consistency guarantees that a source
+  ingestion does not need.
+
+## Consequences
+
+- Remote file databases reuse the established SQL reflection, extraction, and
+  loading path.
+- Each run downloads the whole object and requires enough local disk for the
+  database plus any engine sidecar files.
+- Normal success and failure paths remove staged data. A process terminated
+  without cleanup may leave a temporary directory when a persistent staging
+  parent is configured.
+- Credentials remain storage options and are excluded from safe object
+  locations and object representations.
+- A database extension on a storage URI is reserved: an object named `.db`,
+  `.ddb`, `.duckdb`, `.sqlite`, or `.sqlite3` is never offered to a format
+  reader, and one that holds neither engine reports that instead of a format
+  error.
+- Additional transports need a credential adapter and integration coverage. A
+  transport that builds its filesystem client inside its own source class needs
+  that construction lifted out before it can stage objects.
+- A future engine-specific fast path must preserve the same source-only behavior
+  and public URI contract.

docs/index.md

@@ -16,13 +16,14 @@ inherited by [dlt]: `append`, `merge`, and `delete+insert`.
 
 We recommend using [uv](https://github.com/astral-sh/uv) to run `omniload`.
 
-```
+```bash
 pip install uv
 uvx omniload
 ```
 
 Alternatively, if you'd like to install it globally:
-```
+
+```bash
 uv pip install --system omniload
 ```
 
@@ -35,15 +36,15 @@ Check out the {ref}`Quickstart` guide to get started with omniload.
 ### License
 
 The project is licensed under the MIT License, see the [LICENSE] file for details.
-Some components are licensed under the Apache 2.0 license, see the [NOTICE] file for details.
+Some components are licensed under the Apache 2.0 license, see the [NOTICE]
+file for details.
 
 ### Acknowledgements
 
 This project would not have been possible without the amazing work by the
 authors and contributors to [SQLAlchemy], [dlt], and [ingestr], turtles all
 the way down. Kudos.
 
-
 ```{toctree}
 :caption: Commands and adapters
 :maxdepth: 1
@@ -79,13 +80,13 @@ tutorials/*
 :maxdepth: 1
 :hidden:
 :glob:
+decisions/*
 sandbox
 changelog
 contributors
 backlog
 ```
 
-
 [dlt]: https://github.com/dlt-hub/dlt
 [ingestr]: https://bruin-data.github.io/ingestr/
 [LICENSE]: https://github.com/panodata/omniload/blob/main/LICENSE

docs/supported-sources/duckdb.md

@@ -1,16 +1,43 @@
 # DuckDB
+
 DuckDB is an in-memory database designed to be fast and reliable.
 
 omniload supports DuckDB as both a source and destination.
 
 ## URI format
+
 The URI format for DuckDB is as follows:
 
 ```text
 duckdb:///<database-file>
 ```
 
 URI parameters:
+
 - `database-file`: the path to the DuckDB database file
 
-The same URI structure can be used both for sources and destinations. You can read more about SQLAlchemy's DuckDB dialect [here](https://github.com/Mause/duckdb_engine).
+The same URI structure can be used both for sources and destinations. See the
+[DuckDB SQLAlchemy dialect][duckdb-sqlalchemy] for details.
+
+## Remote source files
+
+A DuckDB source file that lives in [Amazon S3](s3.md), [Cloudflare R2](r2.md),
+[Azure Blob Storage](azure-storage.md), or
+[Google Cloud Storage](google-cloud-storage.md) is addressed by its object URI,
+with the storage credentials as query parameters:
+
+```bash
+omniload ingest \
+  --source-uri 's3://analytics/snapshots/events.duckdb?access_key_id=ACCESS&secret_access_key=SECRET' \
+  --source-table 'main.events' \
+  --dest-uri 'duckdb:///local.duckdb' \
+  --dest-table 'raw.events'
+```
+
+Percent-encode every query value that contains reserved characters such as `+`,
+`/`, `=`, `&`, or `?`.
+
+The object is staged locally for the duration of the run, so remote databases
+are sources only. See {ref}`database-files` for the whole contract.
+
+[duckdb-sqlalchemy]: https://github.com/Mause/duckdb_engine

docs/supported-sources/filesystem.md

@@ -215,6 +215,52 @@ cloud blob destinations currently address one table path. They reject a plural
 workbook load. Select one worksheet or use a dataset-capable destination.
 :::
 
+(database-files)=
+
+## Database files
+
+An object whose name carries a database extension is a database, not a file to
+decode. omniload hands it to the SQL source instead of a reader, so
+`--source-table` selects a table inside the database and the usual SQL
+reflection, chunking, and type mapping apply:
+
+```sh
+omniload ingest \
+    --source-uri 's3://analytics/snapshots/events.duckdb?access_key_id=ACCESS&secret_access_key=SECRET' \
+    --source-table 'main.events' \
+    --dest-uri 'duckdb:///local.duckdb' \
+    --dest-table 'raw.events'
+```
+
+| Extension              | Engine                             |
+|:-----------------------|:-----------------------------------|
+| .db                    | SQLite or DuckDB, read from header |
+| .ddb, .duckdb          | DuckDB                             |
+| .sqlite, .sqlite3      | SQLite                             |
+
+The engine is read from the file header, so a mislabeled object still loads. An
+empty file falls back to its extension, and a `.db` file that is neither engine
+reports both candidates.
+
+Databases are read from `s3://`, `r2://`, `gs://`, `az://`, `adls://`, and
+`abfss://`. Credentials and storage options are the same query parameters the
+matching {ref}`filesystem source <filesystem-types>` takes. The URI names one
+object: globs, `#` fragments, and credentials in the authority are rejected
+rather than silently reinterpreted. The object path also has to ride
+`--source-uri`, because `--source-table` names a table inside the database
+rather than the object, unlike the split form the file sources accept.
+
+Local databases need no filesystem source. Address them with
+[DuckDB](duckdb.md) or [SQLite](sqlite.md) directly, as
+`duckdb:///path/to/events.duckdb`.
+
+:::{note}
+The object is downloaded whole into a run-scoped temporary directory before it
+is opened, so the database must fit on local disk. The copy is removed after
+success and after failure, and changes to it are never written back. Remote
+databases are sources only.
+:::
+
 (file-format-routing)=
 
 ## File format routing

docs/supported-sources/sqlite.md

@@ -1,16 +1,44 @@
 # SQLite
-SQLite is a C-language library that implements a small, fast, self-contained, high-reliability, full-featured, SQL database engine.
+
+SQLite is a C-language library that implements a small, fast, self-contained,
+high-reliability, full-featured SQL database engine.
 
 omniload supports SQLite as a source and a destination.
 
 ## URI format
+
 The URI format for SQLite is as follows:
 
 ```text
 sqlite:///<database-file>
 ```
 
 URI parameters:
+
 - `database-file`: the path to the SQLite database file.
 
-The same URI structure can be used both for sources and destinations. You can read more about SQLAlchemy's SQLite dialect [here](https://docs.sqlalchemy.org/en/20/core/engines.html#sqlite).
+The same URI structure can be used both for sources and destinations. See the
+[SQLite SQLAlchemy dialect][sqlite-sqlalchemy] for details.
+
+## Remote source files
+
+An SQLite source file that lives in [Amazon S3](s3.md), [Cloudflare R2](r2.md),
+[Azure Blob Storage](azure-storage.md), or
+[Google Cloud Storage](google-cloud-storage.md) is addressed by its object URI,
+with the storage credentials as query parameters:
+
+```bash
+omniload ingest \
+  --source-uri 's3://analytics/snapshots/events.sqlite?access_key_id=ACCESS&secret_access_key=SECRET' \
+  --source-table 'main.events' \
+  --dest-uri 'duckdb:///local.duckdb' \
+  --dest-table 'raw.events'
+```
+
+Percent-encode every query value that contains reserved characters such as `+`,
+`/`, `=`, `&`, or `?`.
+
+The object is staged locally for the duration of the run, so remote databases
+are sources only. See {ref}`database-files` for the whole contract.
+
+[sqlite-sqlalchemy]: https://docs.sqlalchemy.org/en/20/dialects/sqlite.html

pyproject.toml

@@ -98,7 +98,7 @@ dependencies = [
   "google-cloud-bigquery-storage<3",
   "google-cloud-spanner<4",
   "influxdb-client<2",
-  "mq-bridge-py>=0.3.2,<0.4",
+  "mq-bridge-py>=0.3.2,<0.4,!=0.3.9",
   "mysql-connector-python<27",
   "omniload[filesystem]",
   "oracledb<5",