feat: add endpoint to download all dataset URLs as a JSON Lines file

[candleindark]

Toward #407. Adds a single-file bulk download of all dataset URLs so a consumer can grab one file instead of paging GET /api/v2/dataset-urls hundreds of times.

New endpoint: GET /api/v2/dataset-urls/all

Returns every dataset URL as a JSON Lines (.jsonl) file, one JSON object per line. Each line has the same shape as an element of the dataset_urls array of the paginated endpoint (a DatasetURLRespModel), so no internal database columns are exposed. A binary return_metadata flag (default false), the counterpart of the tri-state return_metadata query parameter of the paginated endpoint, controls whether each line includes the metadata field populated by content.

Response details and behavior

• Content-Type: application/x-ndjson, served as a download via Content-Disposition: attachment; filename=dataset-urls.jsonl.
• Without return_metadata (default), each line omits the metadata field.
• With return_metadata=true, each line includes metadata as a list of metadata objects by content, the same content form as return_metadata=content on the paginated endpoint.
• The response is streamed with yield_per so the full set of records is not serialized into memory at once, and metadata is eagerly loaded via selectinload when requested to avoid an N+1 query.
• The path /all does not collide with the existing /<int:id> route, since static rules take precedence over converter rules in Werkzeug.

Scope: what this intentionally leaves out

• No filter/search parameters. Unlike the paginated endpoint, the dump takes only return_metadata; it is always a full snapshot of all dataset URLs.
• No caching or scheduled publishing. The thread floats caching the pre-serialized gzipped bytes and publishing a dump on a schedule. Deferred as a potential follow-up: the plan is to test this endpoint in operation first, and pursue caching/publishing only if the current setup's performance is not acceptable.

Test plan

• black, isort, flake8, mypy clean on the changed files.
• Full test_dataset_urls.py module passes (148 tests, 9 new in TestAllDatasetURLs).
• OpenAPI spec documents the endpoint (the application/x-ndjson 200 response and the return_metadata parameter both appear at /openapi/openapi.json).
• Exercise the endpoint against a production-scale dataset to confirm performance is acceptable before considering the caching/publishing follow-up.

New test coverage (TestAllDatasetURLs)

• Empty database returns an empty body with the application/x-ndjson content type.
• Content-Disposition offers the response as dataset-urls.jsonl.
• Without return_metadata: all dataset URLs are returned and no line has a metadata field.
• With return_metadata=true: each line includes metadata by content, with the expected per-URL counts.
• A non-boolean return_metadata value is rejected with 422.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.84%. Comparing base (a69c005) to head (ff44a2e).

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #412      +/-   ##
==========================================
+ Coverage   98.82%   98.84%   +0.02%     
==========================================
  Files          55       55              
  Lines        2630     2690      +60     
==========================================
+ Hits         2599     2659      +60     
  Misses         31       31              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[yarikoptic]

binary?

[candleindark]

format: binary is correct as long as we are on OpenAPI 3.0. flask-openapi3 hardcodes openapi_version = "3.0.3", and in
3.0 {"type": "string", "format": "binary"} is the idiomatic way to declare an opaque file payload, which is what makes Swagger UI offer a download rather than trying to rende
r a schema. It would need to become contentMediaType if we ever move to 3.1. I will add a comment saying so.

Nevertheless, this will most likely be replaced by the new design/implementation.

[yarikoptic]

Let's implement celery task and thus caching right away, because if this is implemented and deployed, could put too much stress on the server. When caching implemented, just make API call return a redirect to target file, and ideally collect and fill out relevant to cached data information such as ETag (md5sum IIRC would suffice) and potentially other fields (like expiration date and when was generated etc).

[yarikoptic]

might be worth right away to compress it but better be done along with caching so we do not overburden the server. And then see what mime type etc to provide back for that .jsonl.gz.

For when caching implemented, make sure for atomic rename of the (re)generated file into the place where served from.

[candleindark]

Let's implement celery task and thus caching right away, because if this is implemented and deployed, could put too much stress on the server. When caching implemented, just make API call return a redirect to target file, and ideally collect and fill out relevant to cached data information such as ETag (md5sum IIRC would suffice) and potentially other fields (like expiration date and when was generated etc).

Agreed on doing the Celery task and caching up front, and on having the API return a redirect. Implementing that raises an infrastructure question since it depends on what the servers can host.

The problem: the worker that would generate the dump runs on the primary, but the public-facing instance is the read-only replica. The two share no filesystem, so there is currently nowhere for the worker to write a file that the public instance can serve.

What I would like: a location the primary can upload/write the generated dump to, and that is served publicly.

[candleindark]

Note:

The follow query shows that, before gzip, jsonl file is about 9.6 MB without metadata and 70 MB with metadata.

dlreg=# WITH slim AS (
  SELECT r.id, jsonb_build_object(
           'url', r.url, 'id', r.id, 'ds_id', r.ds_id,
           'head_describe', r.head_describe,
           'annex_key_count', r.annex_key_count,
           'annexed_files_in_wt_count', r.annexed_files_in_wt_count,
           'annexed_files_in_wt_size', r.annexed_files_in_wt_size,
           'last_update_dt', r.last_update_dt,
           'git_objects_kb', r.git_objects_kb,
           'processed', r.processed, 'last_chk_dt', r.last_chk_dt) AS obj
  FROM repo_url r WHERE r.processed
), meta AS (
  SELECT m.url_id, jsonb_agg(jsonb_build_object(
           'extractor_name', m.extractor_name,
           'dataset_describe', m.dataset_describe,
           'dataset_version', m.dataset_version,
           'extractor_version', m.extractor_version,
           'extraction_parameter', m.extraction_parameter,
           'extracted_metadata', m.extracted_metadata)) AS arr
  FROM url_metadata m GROUP BY m.url_id
)
SELECT count(*) AS n_lines,
       pg_size_pretty(sum(octet_length(s.obj::text) + 1)) AS slim_jsonl,
       pg_size_pretty(sum(octet_length(
         (s.obj || jsonb_build_object('metadata', coalesce(mt.arr, '[]'::jsonb)))::text) + 1
       )) AS full_jsonl
FROM slim s LEFT JOIN meta mt ON mt.url_id = s.id;
 n_lines | slim_jsonl | full_jsonl
---------+------------+------------
   24792 | 9809 kB    | 70 MB
(1 row)