This document captures current performance characteristics and a repeatable profiling plan.
Analyzed areas:
- generation latency
- adaptive retry overhead
- queue throughput behavior
- storage and retrieval path
- Single active generation job per process due to GPU lock.
- Up to 10 attempts per job in worst-case quality scenarios.
- Additional retry attempts increase end-to-end latency linearly.
- Prompt pipeline grammar stage may add startup/first-use overhead.
- Artifact persistence adds I/O cost per attempt.
- Diffusion inference time in ModelManager.generate/img2img.
- CLIP scoring overhead in QualityEvaluator when enabled.
- First-use model warmup and weight download latency.
- Queue wait time under concurrent user submissions.
- FIFO + mutex improves stability and OOM safety but caps throughput.
- Quality retries improve output quality but can extend response time.
- In-memory mode removes DB overhead but sacrifices durability.
- Enable info-level logs.
- Capture per-attempt generation_time from metadata.
- Record job completion time versus attempt count.
- Compare prompt sets at fixed threshold values.
- Submit N concurrent jobs.
- Track queue wait and run durations per job.
- Evaluate fairness and tail latency.
- Measure /jobs/{id}/image response latency for hot and cold DB cache states.
Track these metrics over time:
- avg_attempts_per_job
- p50/p95 job completion time
- generation failure rate
- OOM retry count
- queue depth over time
- DB write latency for artifacts and metadata
- Add benchmark script for repeatable latency baselines.
- Add frontend and API pagination for large histories.
- Add optional distributed queue and worker scaling path.
- Introduce configurable retry policy per request class.