DD-42807 - Manual by-case ingestion endpoint (POST /ingestions/start-by-case)

[rkakumanuhmcts]

JIRA link (if applicable)

DD-42807

Change description

What

Adds POST /ingestions/start-by-case — a synchronous ingestion entry point triggered by the "Process IDPC" button on the AI Search page, which accepts a single caseId and returns the real outcome (STARTED / NOT_REQUIRED / FAILED) in the response.

Why

The existing POST /ingestions/start endpoint is designed for the scheduled/multi-case flow: it looks up cases via GET_CASES_FOR_HEARING and returns 202 ACCEPTED immediately without knowing the outcome. The AI Search UI needs a single-case, synchronous variant that waits for the actual IDPC-availability result before responding, so the user gets an immediate, accurate answer rather than a fire-and-forget acknowledgement.

Changes

• New endpoint: IngestionController.startIngestionProcessByCase, delegating to a new IngestionProcessorByCase interface / IngestionProcessorByCaseService implementation — mirrors the existing IngestionProcessor / JobManagerService pattern.
• Shared, plain business rule: IdpcAvailabilityService (no JobManager/task-framework types) is the single owner of the IDPC-availability rule and is called directly, in-process, by both IngestionProcessorByCaseService (this endpoint, synchronous) and CheckIdpcAvailabilityAllDefendantsTask (the existing scheduled JobManager task). There is no separate case-eligibility step/service/task — an empty document list already covers "case doesn't exist", "no defendants", and "already ingested" alike, so the earlier CaseEligibilityService/CheckCaseEligibilityTask round-trip was redundant and has been removed; GetCasesForHearingTask now dispatches CHECK_IDPC_AVAILABILITY_ALL_DEFENDANTS directly.
• Shared job-data building: RetrieveMaterialAndUploadJobDataService builds the per-document RETRIEVE_MATERIAL_AND_UPLOAD job data once, reused by both the task and the service, instead of duplicating that logic in two places. Each caller still owns its own final ExecutionInfo/executionService.executeWith(...) dispatch shape.
• New JobPriority constants (HIGH/DEFAULT): every JobManager task submitted from this endpoint runs at HIGH priority, so manual clicks are picked up ahead of scheduled/nightly ingestion. The existing scheduled endpoint is unaffected and stays at default priority.
• ACL: added a Drools rule for the new casedocumentknowledge-service.ingestion-process-by-case action (cdks-rules.drl) — without it the endpoint 403s. Also removed a pre-existing dead duplicate rule and fixed a rule-name typo while in the file.
• Removed now-dead progression client code: with the eligibility step gone, ProgressionClient.getProsecutionCaseEligibilityInfo (and its ProsecutionCase/ProsecutionCaseResponse/Defendant/ProsecutionCaseEligibilityInfo DTOs, and config) had no remaining caller and have been deleted.
• Docs: updated .claude/context/csdk-context.md (API surface table + endpoint behaviour note, kept in sync with the current flow) and .claude/context/tech-stack.md (API contract version).
• Tests: unit tests for IngestionProcessorByCaseService, IdpcAvailabilityService, CheckIdpcAvailabilityAllDefendantsTask, and IngestionController covering all three response phases; a compose-backed integration test (IngestionProcessByCaseHttpLiveTest) covering STARTED, NOT_REQUIRED, and FAILED end-to-end (FAILED now exercises a downstream court-document-search failure, since that's the only remaining downstream call in the synchronous path).

Does this PR introduce a breaking change?

[ ] Yes
[x] No

[rkakumanuhmcts]

Code Review — DD-42807 POST /ingestions/start-by-case (Claude, pipeline stage 6)

Reviewed diff of dev/DD-42807 against develop (merge-base 161f554, so scope = this PR only). 16 files, +953/-114.

Verdict: Approve with minor changes (non-blocking). The four deliberate architectural decisions all hold up in the code (see below). No correctness, security, PII, or Flyway issues. Three MEDIUM quality items and a few LOW notes are recommended before merge — human reviewer makes the final call.

---

Design decisions — verified against the code (not just intent)

#	Decision	Verdict
1	Synchronous in-process invocation of the two tasks (not executeWith)	PASS — IngestionProcessorByCaseService calls checkCaseEligibilityTask.execute(...) then checkIdpcAvailabilityAllDefendantsTask.registerNewDocumentsAndDispatch(...) directly; HTTP response reflects real STARTED/NOT_REQUIRED/FAILED before returning.
2	CTX_SYNCHRONOUS_INVOCATION_KEY prevents double-dispatch	PASS with a latent trap (MEDIUM-2). Flag is read in exactly one place (CheckCaseEligibilityTask.execute L92); when set + eligible it returns the follow-on ExecutionInfo without executeWith. CHECK_IDPC_AVAILABILITY_ALL_DEFENDANTS is therefore never enqueued in the sync path, and CHECK_CASE_ELIGIBILITY itself is never enqueued (only execute() is called directly). No double-dispatch path exists today.
3	JobPriority.HIGH propagates via ExecutionInfo.Builder#from(...) down to the queued RETRIEVE_MATERIAL_AND_UPLOAD job	PASS in code, but partially untested (MEDIUM-3). Leg 1 (eligibility→IDPC) copies priority via .from(executionInfo) and is asserted by CheckCaseEligibilityTaskTest (result.getPriority()==HIGH). Leg 2 (IDPC→RETRIEVE_MATERIAL_AND_UPLOAD) also uses .from(executionInfo), so the chain is intact — but no test asserts the dispatched retrieval job carries HIGH, and that is the only job that actually enters the queue where ORDER BY priority ASC takes effect.
4	New Drools rule for casedocumentknowledge-service.ingestion-process-by-case, dead duplicate + typo cleaned up	PASS. Confirmed against full file both sides: the original top ingestion-process rule is retained (existing POST /ingestions/start still authorised — no 403 regression); the pre-existing duplicate ingestion-process rule at the bottom was repurposed into the new ingestion-process-by-case rule; the "ingestion sattus"→"ingestion status" name typo was fixed (action ...ingestion unchanged). Action name matches the controller media type application/vnd.casedocumentknowledge-service.ingestion-process-by-case+json. Nothing else in the file changed.
5	resolveEligibleCase/withDefendantContext package-private	PASS. Only callers are CheckCaseEligibilityTask itself and its test in the same package (jobmanager.caseflow). Access level sufficient; nothing broken.
6	API contract bump 0.0.10 → 0.0.11	PASS. Bumped consistently in gradle.properties (version.cdk) and documented in tech-stack.md; supplies IngestionProcessByCaseRequest, IngestionApi.startIngestionProcessByCase, and IngestionProcessPhase.NOT_REQUIRED. (Not independently verifiable without resolving the artifact, but code compiles/tests against it per the PR.)

---

Findings (ranked)

MEDIUM-1 — Misleading NOT_REQUIRED message for the "not eligible" branch (IngestionProcessorByCaseService, MSG_NOT_REQUIRED)
NOT_REQUIRED is returned for two semantically different situations: (a) eligible but no newer IDPC version, and (b) case not eligible / not found / no defendants. Both return the same message: "...no newer IDPC version is available and an Answers version already exists." For branch (b) neither clause is verified — the code never checks whether an Answers version exists, and the case may simply not exist. IngestionProcessorByCaseServiceTest.returnsNotRequired_whenNotEligible even asserts this misleading text for the not-eligible case. Recommend distinguishing the two branches (or softening the message to not assert an unverified "Answers version already exists").

MEDIUM-2 — synchronousInvocation flag leaks into the async continuation (CheckIdpcAvailabilityAllDefendantsTask.registerNewDocumentsAndDispatch)
The dispatched RETRIEVE_MATERIAL_AND_UPLOAD job data is built with createObjectBuilder(jobData), which copies synchronousInvocation=true forward. That flag then rides through the async chain (RetrieveMaterialAndUpload and anything after it). Harmless today because only CheckCaseEligibilityTask.execute reads it — but it is a latent foot-gun: if any downstream task is later taught to honour the flag, it would wrongly suppress its own dispatch in the async continuation. Recommend stripping the flag when crossing the sync→async boundary (i.e. before dispatching the retrieval job).

MEDIUM-3 — No test guards HIGH-priority propagation to the queued job (design point 3)
CheckIdpcAvailabilityAllDefendantsTaskTest captures executeWith(...) but never asserts getPriority(); the service test mocks registerNewDocumentsAndDispatch. So the one leg where priority actually affects JobExecutor ordering is unverified. Add a captor assertion that the dispatched RETRIEVE_MATERIAL_AND_UPLOAD ExecutionInfo carries JobPriority.HIGH.

LOW-1 — FAILED detection couples the sync service to async retry encoding. IngestionProcessorByCaseService treats ExecutionStatus.INPROGRESS as "eligibility failed" (that is what CheckCaseEligibilityTask's catch block returns, alongside shouldRetry). It works and is tested, but the sync caller reading a retry-signalling status as a terminal failure is an abstraction leak; a dedicated failure signal would be more robust.

LOW-2 — HTTP 200 for FAILED. An internal failure returns 200 OK with phase=FAILED in the body (per the generated contract). Clients/monitoring cannot detect failures by status code — they must parse the body. Confirm this matches the API-contract intent (it appears to).

LOW-3 — requestId is freshly generated, not derived from an inbound correlation header. UUID.randomUUID() in the service means a manual click is only traceable within this service; there is no MDC correlationId/requestId set for the request (consistent with the existing code, not a regression). Worth confirming against logging-standards.md MDC expectations.

INFO — Logging of caseId. New log lines log caseId (UUID). Consistent with the existing codebase convention; flagged only so the human reviewer confirms a case UUID is not treated as court-reference data under the no-PII-in-logs rule. No document content / PII is logged.

---

Checklist summary

• Correctness / double-dispatch / priority chain / ACL wiring: PASS (see table).
• Security — no secrets, no connection strings/SAS/keys, Managed-Identity-only: PASS / N/A (no Azure or config changes).
• PII — synthetic UUIDs only in tests/stubs; ffffffff-... failing-case id; no real court references: PASS.
• Logging — @Slf4j JSON, no System.out, no document bodies logged: PASS (see LOW-3, INFO).
• Flyway append-only: N/A (no migrations; IT seeds via JDBC in test only).
• Tests — behaviour-focused, all three phases covered at unit + controller + compose IT level; sync-invocation branch and package-private helpers unit-tested: PASS (gap noted in MEDIUM-3).
• Docs — csdk-context.md / tech-stack.md updated to match: PASS.
• Accessibility: N/A (backend endpoint, no UI).

---

Stage 6 is a human gate. This is an automated pre-review — a human engineer must approve before CI/ci-orchestrator proceeds. None of the findings above are blocking; MEDIUM-1..3 are recommended before merge.

🤖 Generated with Claude Code

[natrajmolala]

could the task be still retrying in the background when though response to the client is FAILED?

[maheshsubramanian]

this is userId

[maheshsubramanian]

this is not fixed..

[maheshsubramanian]

static import

[maheshsubramanian]

method signature should be free of job manager library POJOs.. pass things like caseId, userId explicitly as required by then service

[maheshsubramanian]

rename perhaps to clearly call out what the method does..

[maheshsubramanian]

make this class complete free of job manager classes

[maheshsubramanian]

delete this and figure out where this is actually needed

[maheshsubramanian]

this should not be public.. and hence will not be needed here

[maheshsubramanian]

this is not fixed..