Dedicated crypto ExecutionContext: eliminate CPU queueing between ECIES and DB

[elliottdehn]

Summary

Add a dedicated thread pool for CPU-bound asymmetric crypto (ECIES/ECDH), separate from Canton's main ExecutionContext. Prevents crypto work from blocking DB callbacks, protocol orchestration, and gRPC handlers.

The Problem

Canton runs everything on one shared ExecutionContext (*-env-ec). JFR profiling shows ECIES/ECDH consumes 38-43% of CPU. When all cores are saturated with elliptic curve math, DB callback threads can't be scheduled — they sit in the CPU run queue behind ECIES operations.

The DB operations themselves are fast (IO-wait, ~1.2% of CPU). But they can't start until a core is available. A 2ms DB write becomes 2ms + queueing delay. This queueing delay is invisible in CPU profiling but dominates end-to-end latency under load.

Our bipartite PoC benchmark showed p50 latency dropping from 260ms to 38ms — primarily from eliminating this queueing contention, not from having more total cores.

The Fix

Two separate thread pools:

Before (shared):
  env-ec thread 1: [ECIES][ECIES][ECIES][DB callback finally scheduled]
  env-ec thread 2: [ECIES][ECIES][ECIES][ECIES]...

After (dedicated):
  crypto-ec thread 1: [ECIES][ECIES][ECIES][ECIES]...
  crypto-ec thread 2: [ECIES][ECIES][ECIES][ECIES]...
  env-ec thread 1:    [DB callback→done][protocol msg][DB callback→done]
  env-ec thread 2:    [gRPC handler][DB callback→done][protocol msg]

DB callbacks run on the main EC, which is no longer saturated. Crypto runs on its own pool. No contention.

Changes

File	Change
Environment.scala	New cryptoExecutionContext pool alongside executionContext
SyncCryptoApiParticipantProvider.scala	SynchronizerCryptoClient and SynchronizerSnapshotSyncCryptoApi gain optional cryptoEc parameter. encryptFor dispatches to crypto pool via parTraverse + Future(...)(cryptoPool) when set.

All existing call sites unaffected (cryptoEc defaults to None). To enable: pass cryptoExecutionContext when constructing SynchronizerCryptoClient.

Relation to other PRs

• Captures the latency win from the bipartite architecture ([main] Update 2024-09-17.07 #224) without separate machines. The PoC benchmark's 260ms → 38ms p50 improvement was primarily driven by eliminating CPU queueing — this PR achieves the same effect within a single JVM.
• Stacks with Bridge session key encryption cache to decryption cache (#495) #487 (session key cache bridge for decrypt side). Independent code paths.
• Subsumes Parallelize per-member ECIES encryption and commitment computation (#494) #486's encrypt parallelism — this PR dispatches to the crypto pool via parTraverse, which implicitly parallelizes across crypto-pool threads.
• Complements Background ECDH precomputation (#497) #489 (background precompute) — Background ECDH precomputation (#497) #489 moves ECDH off the critical path entirely; this PR ensures that even when ECDH runs on the critical path (cold cache), it doesn't block DB callbacks.

Test plan

• Existing EncryptedViewMessageFactory tests pass
• SimplestPingIntegrationTest passes
• When cryptoEc is None, behavior is identical to before (sequential on main EC)
• When cryptoEc is set, encryption runs on the crypto pool (verify via thread names in logs)

🤖 Generated with Claude Code

[elliottdehn]

I have hereby read the Digital Asset CLA and agree to its terms

[elliottdehn]

Benchmark Results: Thread pool separation doesn't help in single JVM

Benchmarked shared pool vs split pool (same total threads) with 10 recipients, 16 concurrent transactions, real Postgres:

Config	Shared tx/s	Split tx/s	Shared p50	Split p50
2 threads	158	198	12.5ms	80.6ms
4 threads	307	221	12.8ms	40.4ms
8 threads	316	266	13.9ms	23.9ms

Split is consistently worse on latency and mostly worse on throughput.

The problem: the transaction pipeline is sequential (encrypt → DB → decrypt → DB). When the main-pool thread dispatches crypto to the crypto pool and calls .get(), it blocks — wasting its thread slot while waiting. This is strictly worse than just doing the crypto inline.

The bipartite architecture's latency win comes from a different mechanism: B handles many concurrent transactions, so while transaction A waits for its crypto response from node A, transaction B's DB callback runs on B's now-free core. This interleaving only works across separate machines where B never does crypto at all.

Closing this PR — the dedicated crypto EC doesn't provide the benefit we hypothesized. The queueing contention requires truly separate machines (the full bipartite architecture) or restructuring Canton's per-transaction pipeline to be non-blocking (which would be a much larger change than thread pool separation).

The PRs that DO help:

• Bridge session key encryption cache to decryption cache (#495) #487 (session key cache bridge): eliminates redundant decrypt, no thread pool needed
• Background ECDH precomputation (#497) #489 (background precompute): moves ECDH to background thread that runs BETWEEN transactions, not blocking any transaction's pipeline
• Per-participant ECDH channel keys (#498) #490 (channel keys): eliminates ECDH entirely for warm channels

[elliottdehn]

Closing — benchmark shows thread pool separation doesn't help when the per-transaction pipeline is sequential. See benchmark results in comments.