Improve performance of initiate and complete directory multipart upload

[GianlucaFicarelli]

Description

Improve performance of initiate and complete directory multipart upload.
The multipart upload for directories works fine for directories with small number of files.

However, there are directory assets with hundreds of files, that cause the initiate and complete endpoints to be slow.

Since most of the time is spent in calling S3, a possible improvement can be:

1. to make requests in parallel using threads, limiting the number of threads in a pool
2. to make requests in parallel using asyncio -> doesn't fit well with the current approach of sync endpoints, because it would require to:
   • create an async loop inside the fastapi/starlette thread
   • or rewrite the endpoint to be fully async, requiring the installation of async libraries
3. to create a background job, and allow the client to check when it's finished, but it would add more complexity. It can be considered if the number of files is too high to be handled by parallel requests

Example in staging for a directory with 1527 files (maximum number currently registered):

{
  "time": "2026-07-05T08:55:59.254862+00:00",
  "level": "INFO",
  "name": "app.middleware",
  "message": "request_completed",
  "extra": {
    "method": "POST",
    "url": "http://staging.cell-a.openbraininstitute.org/api/entitycore/em-cell-mesh/40f6e39b-c940-4bd4-942f-b564fc5b5890/assets/directory/multipart-upload/initiate",
    "route_template": "/{entity_route}/{entity_id}/assets/directory/multipart-upload/initiate",
    "status_code": 200,
    "status_class": 2,
    "process_time_ms": 58992,
    "response_size": 3436907,
    "client": "10.0.2.153",
    "forwarded_for": "213.55.247.216",
    "user_agent": "python-httpx/0.28.1",
    "request_id": "5d30d1fd-f344-4a45-8efa-b36e1c9f814f",
    "user_id": "3d4e4581-1a32-4356-beb6-74099d65de96"
  },
  "exception": null
}

{
  "time": "2026-07-05T08:59:58.094867+00:00",
  "level": "INFO",
  "name": "app.middleware",
  "message": "request_completed",
  "extra": {
    "method": "POST",
    "url": "http://staging.cell-a.openbraininstitute.org/api/entitycore/em-cell-mesh/40f6e39b-c940-4bd4-942f-b564fc5b5890/assets/10ae3e97-68b7-4ebe-a93b-b74d3037d72e/directory/multipart-upload/complete",
    "route_template": "/{entity_route}/{entity_id}/assets/{asset_id}/directory/multipart-upload/complete",
    "status_code": 200,
    "status_class": 2,
    "process_time_ms": 215249,
    "response_size": 426,
    "client": "10.0.2.153",
    "forwarded_for": "213.55.247.216",
    "user_agent": "python-httpx/0.28.1",
    "request_id": "2eeb4e1f-a15f-4a10-9907-4e22426b7d61",
    "user_id": "3d4e4581-1a32-4356-beb6-74099d65de96"
  },
  "exception": null
}

Acceptance criteria

• Include a checkbox list
• Include a checkbox list
• Include a checkbox list

[GianlucaFicarelli]

Parallelization of initiation and creation are deployed to staging.
The PR #676 is needed to parallelize also deletion.

[GianlucaFicarelli]

Tested the performance in staging with a directory asset of 1527 files (same as in the ticket description) with:

• entitysdk v0.19.6
• entitycore 2026.7.12, configured with S3_MAX_WORKERS=32 (used for initialization, completion, deletion of assets)

Process times on the server side, including the creation, update, or deletion of records in the db, that is still serial:

• POST directory/multipart-upload/initiate: from 59 to 10.0 seconds ~ 6x faster
• POST directory/multipart-upload/complete: from 215 to 11.7 seconds ~ 18x faster
• DELETE /em-cell-mesh/{id_}: from 44 to 7.8 seconds ~ 5.5x faster

On the client side, including the time needed for entitysdk to upload the 1527 to s3 (around 20 MB in total), using the default max_concurrency=10 for uploading files:

• registration of assets with client.upload_directory(initiate+upload+complete): 47.8 s
• deletion of entity and assets with client.delete_entity: 8.34 s

Looking at the detailed logs in entitycore:

• deletion of the first file completed at 2026-07-21T09:41:14.009395+00:00
• deletion of the last file completed at 2026-07-21T09:41:18.751782+00:00
• completion of the request at 2026-07-21T09:41:18.760528+00:00
• difference, only for deleting 1527 files in s3 from entitycore: 4.75 seconds (excluding the time for the first request to complete)

New request logs:

{"time":"2026-07-21T09:28:25.748410+00:00","level":"INFO","name":"app.middleware","message":"request_completed","extra":{"method":"POST","url":"http://staging.cell-a.openbraininstitute.org/api/entitycore/em-cell-mesh/35fa4a2f-3604-45a3-90ca-1e5951a9ab88/assets/directory/multipart-upload/initiate","route_template":"/{entity_route}/{entity_id}/assets/directory/multipart-upload/initiate","status_code":200,"status_class":2,"process_time_ms":10025,"response_size":3562259,"client":"10.0.2.171","forwarded_for":"188.61.112.208","user_agent":"python-httpx/0.28.1","request_id":"a248c380-d2d2-4252-8826-a90a700fec08","user_id":"dcb8ee63-c8fd-4284-8ec3-a3b6ff3fab88"},"exception":null}
{"time":"2026-07-21T09:29:02.776081+00:00","level":"INFO","name":"app.middleware","message":"request_completed","extra":{"method":"POST","url":"http://staging.cell-a.openbraininstitute.org/api/entitycore/em-cell-mesh/35fa4a2f-3604-45a3-90ca-1e5951a9ab88/assets/0255fcf2-2819-4296-ac4f-e4b401652051/directory/multipart-upload/complete","route_template":"/{entity_route}/{entity_id}/assets/{asset_id}/directory/multipart-upload/complete","status_code":200,"status_class":2,"process_time_ms":11749,"response_size":426,"client":"10.0.2.171","forwarded_for":"188.61.112.208","user_agent":"python-httpx/0.28.1","request_id":"fc4a2c01-52bd-4c19-8922-54206e1012ec","user_id":"dcb8ee63-c8fd-4284-8ec3-a3b6ff3fab88"},"exception":null}
{"time":"2026-07-21T09:41:18.760528+00:00","level":"INFO","name":"app.middleware","message":"request_completed","extra":{"method":"DELETE","url":"http://staging.cell-a.openbraininstitute.org/api/entitycore/em-cell-mesh/35fa4a2f-3604-45a3-90ca-1e5951a9ab88","route_template":"/em-cell-mesh/{id_}","status_code":200,"status_class":2,"process_time_ms":7861,"response_size":45,"client":"10.0.2.151","forwarded_for":"188.61.112.208","user_agent":"python-httpx/0.28.1","request_id":"a5723407-6374-47bc-9923-8e8c76aa04f6","user_id":"dcb8ee63-c8fd-4284-8ec3-a3b6ff3fab88"},"exception":null}