feat(external-call): confirmation-response router#555
Conversation
b7d9e4d to
65e975f
Compare
|
Start reviewing... |
| case _ => None | ||
| } | ||
|
|
||
| def abstainsForView( |
There was a problem hiding this comment.
Please provide a brief ScalaDoc, as for rejectsForView.
The contract probably gets simpler if hostedConfirmingParties and rejectedParties get combined into a single argument.
There was a problem hiding this comment.
Done in bbd4540 — ScalaDoc added, and the two sets are merged into a single nonRejectingConfirmingParties parameter (the predicate only ever consumed hosted && !rejected). The factory call site later in the series passes the difference.
| private lazy val hasAnyVisibleExternalCallResults: Boolean = | ||
| viewValidationResults.valuesIterator.exists { viewValidationResult => | ||
| val viewParticipantData = viewValidationResult.view.viewParticipantData | ||
| viewParticipantData.supportsExternalCallResults && |
There was a problem hiding this comment.
I think this line is redundant. Drop it?
There was a problem hiding this comment.
Agreed, dropped in 1b22c76d83. supportsExternalCallResults is redundant with externalCallResults.nonEmpty — ViewParticipantData.validated guarantees an unsupported view holds no external-call results, so the conjunct can never flip the result.
| .toSet | ||
|
|
||
| lazy val recordedConsistencyResult: ExternalCallConsistencyChecker.Result = | ||
| if (hasAnyVisibleExternalCallResults) { |
There was a problem hiding this comment.
What's the reason for this if?
I kindly ask you to proactively remove all short-circuiting ifs, unless there is a good reason to have them. This will reduce the time needed to review the code.
There was a problem hiding this comment.
This one's a deliberate fast-path, not a redundant guard. recordedConsistencyResult is evaluated for every transaction, and when no view has any external-call results (the common case) hasAnyVisibleExternalCallResults lets us skip building allHostedConfirmingParties (a flatMap over every view) and the check call entirely — so it avoids extra passes over all views on the overwhelmingly common no-external-call path. Happy to drop it for uniformity if you'd prefer, but I kept it per your "unless there's a good reason" note.
| viewParticipantData.externalCallResults.nonEmpty | ||
| } | ||
|
|
||
| private lazy val recordedReplayDisagreementInconsistencies = |
There was a problem hiding this comment.
Please add type annotations for all top level declarations.
There was a problem hiding this comment.
Also would it make sense to rename Rename -> ExternalCall for consistency with other names?
There was a problem hiding this comment.
Done in 6a15c45 — renamed to recordedExternalCallDisagreementInconsistencies, matching the companion method it is assigned from.
There was a problem hiding this comment.
Not sure if you have added type annotations for all top level declarations. Please double-check this point.
There was a problem hiding this comment.
Good catch — a sweep of the slice found seven unannotated top-level declarations (the routing context's two lazy vals, the router test's factory/key fixtures, and the checker test's three party fixtures), annotated in 48318ba. A second, adversarial pass over the slice caught one more that the first sweep missed — the recording validator's queue field in the shared test util — annotated in 7238315. That accounts for every template-level declaration in the slice's four files.
| val viewParticipantData = | ||
| viewWithHostedParties.validationResult.view.viewParticipantData | ||
| if ( | ||
| !viewParticipantData.supportsExternalCallResults || |
There was a problem hiding this comment.
Redundant condition? If so, please drop it.
There was a problem hiding this comment.
Agreed, dropped the !supportsExternalCallResults || disjunct in 7b44694179. Same invariant: an unsupported view has no external-call results, so !supportsExternalCallResults || externalCallResults.isEmpty is just externalCallResults.isEmpty.
| recordedConsistencyResult: ExternalCallConsistencyChecker.Result, | ||
| )(implicit traceContext: TraceContext): Unit = | ||
| recordedConsistencyResult.visibleInconsistencies.foreach { inconsistency => | ||
| val details = inconsistency.description.limit(maxExternalCallDisagreementDetailsLength) |
There was a problem hiding this comment.
PrettyPrinting already has a built-in limit (c.f. object Pretty), so no need to limit again.
There was a problem hiding this comment.
Dropped in e6b998f, together with the now-unused constant. Verified: description renders via Pretty's default pprinter (bounded), and the Inconsistency rendering only prints payload sizes, never payloads.
65e975f to
161bdd7
Compare
|
I've concluded my first review pass. I struggled a bit with this slice, as it’s difficult to see exactly what this component is trying to achieve in isolation. Because of this, I will have to defer the detailed review of this routing logic until we reach the context of #555. To help speed up the review process when we get there, please add meaningful specification details as ScalaDocs to the public methods of ExternalCallResponseRouter and ExternalCallRoutingContext. Specifically, it would be highly beneficial to document the intended data flow - how data moves from validationOccurrences down into the final reject/abstain maps - as well as how these errors are intended to interface with the downstream components. Moreover:
|
7b44694 to
648a12d
Compare
Requested in the first review pass of digital-asset#555: specification-level ScalaDocs for the public methods, the intended data flow from occurrence selection to the reject/abstain routes, and the downstream interface.
|
Thanks for the pointer — done in 1254683: a data-flow narrative on Inline comments from the pass are addressed (replies on the threads), and the rebase will follow as soon as #554 merges. |
Requested in the first review pass of digital-asset#555: specification-level ScalaDocs for the public methods, the intended data flow from occurrence selection to the reject/abstain routes, and the downstream interface. (cherry picked from commit 1254683)
…r and router Adaptations owed to the review fixes cherry-picked from digital-asset#554/digital-asset#555: - checker Result.inconsistencies -> hostedInconsistencies at the remaining router and router-test call sites, - factory abstainsForView call adapted to the merged nonRejectingConfirmingParties parameter, - factory disagreement-details truncation dropped alongside the removed length constant (Pretty already bounds the output), - shared test fixtures that digital-asset#554 slimmed away restored for the router and factory tests, with view-position values matching their names (as promised in digital-asset#554 review reply 3521058115); relative ordering of the positions is unchanged, so no order-sensitive assertion is affected.
…ication Call-stack verification of the restructured docs (the digital-asset#555 method) refuted four claims; the code is unchanged, only the wording: - validateExternalCalls no longer claims a key with disagreeing recorded outputs anywhere in the request is never re-validated: the single-output aggregation is scoped to the selected occurrences, so if one side of a disagreement has no hosted, non-rejecting checker, the remaining side is still re-validated for the parties that check it (the dropped side stays covered by the consistency check's rejections and alarms). - The model-conformance suppression comment no longer claims every routed error surfaces in exactly one logged rejection: affected views log one rejection each when emitting it, and a view rejected as malformed for another reason subsumes the external-call rejection; the visible-disagreement alarm covers the error in every case. - The router class doc names the checks the suite actually runs concurrently with (model conformance, internal consistency) instead of 'the other validation suites', and describes the remaining parties as responding with the view's own verdict rather than 'approval otherwise'. - Re-validation timing is stated as 'kicked off without awaiting the model-conformance result' (the result may already be available), and the abstain sentence carries its otherwise-approved-views qualifier.
Requested in the first review pass of digital-asset#555: specification-level ScalaDocs for the public methods, the intended data flow from occurrence selection to the reject/abstain routes, and the downstream interface.
8921f2a to
46e33e9
Compare
…r and router Adaptations owed to the review fixes cherry-picked from digital-asset#554/digital-asset#555: - checker Result.inconsistencies -> hostedInconsistencies at the remaining router and router-test call sites, - factory abstainsForView call adapted to the merged nonRejectingConfirmingParties parameter, - factory disagreement-details truncation dropped alongside the removed length constant (Pretty already bounds the output), - shared test fixtures that digital-asset#554 slimmed away restored for the router and factory tests, with view-position values matching their names (as promised in digital-asset#554 review reply 3521058115); relative ordering of the positions is unchanged, so no order-sensitive assertion is affected.
…ication Call-stack verification of the restructured docs (the digital-asset#555 method) refuted four claims; the code is unchanged, only the wording: - validateExternalCalls no longer claims a key with disagreeing recorded outputs anywhere in the request is never re-validated: the single-output aggregation is scoped to the selected occurrences, so if one side of a disagreement has no hosted, non-rejecting checker, the remaining side is still re-validated for the parties that check it (the dropped side stays covered by the consistency check's rejections and alarms). - The model-conformance suppression comment no longer claims every routed error surfaces in exactly one logged rejection: affected views log one rejection each when emitting it, and a view rejected as malformed for another reason subsumes the external-call rejection; the visible-disagreement alarm covers the error in every case. - The router class doc names the checks the suite actually runs concurrently with (model conformance, internal consistency) instead of 'the other validation suites', and describes the remaining parties as responding with the view's own verdict rather than 'approval otherwise'. - Re-validation timing is stated as 'kicked off without awaiting the model-conformance result' (the result may already be available), and the abstain sentence carries its otherwise-approved-views qualifier.
|
Rebased onto main HEAD (e3c82a3, post-#552); the twelve content commits replayed byte-identically. The new tip commit 46e33e9 adapts this slice to the #554 checker changes that now reach the branch through the stack: the renamed checker-result field at its router and test call sites, and the shared test fixtures that #554 slimmed away restored for the router test — with view-position values matching their names, as promised in #554 (comment). I'll rebase again once #554 merges, per your note. |
Requested in the first review pass of digital-asset#555: specification-level ScalaDocs for the public methods, the intended data flow from occurrence selection to the reject/abstain routes, and the downstream interface.
…reement routing On empty disagreements the general path already flat-maps to nothing and groupMap yields Map.empty, so the isEmpty short-circuit is redundant (the only skipped work is a cheap, pure view sort).
…ByParty The supportsExternalCallResults guard is dead by the checkExternalCallResults invariant (unsupported => results empty), so filter/flatMap over the results already yields an empty Seq.
…leExternalCallResults supportsExternalCallResults is redundant with externalCallResults.nonEmpty by the checkExternalCallResults invariant (unsupported => results empty), so the conjunct never changes the predicate. Verified behavior-preserving by adversarial review.
…ccurrences !supportsExternalCallResults is redundant with externalCallResults.isEmpty by the checkExternalCallResults invariant (unsupported => results empty), so the disjunct never changes which branch the if takes. Verified behavior-preserving by adversarial review.
… parameters The two party sets were only ever consumed as hosted-and-not-rejecting, so pass that single set instead.
PrettyPrinting already bounds the rendered description (Pretty's default pprinter), and the Inconsistency rendering only prints payload sizes.
Requested in the first review pass of digital-asset#555: specification-level ScalaDocs for the public methods, the intended data flow from occurrence selection to the reject/abstain routes, and the downstream interface.
…flow The replay-disagreement source spans a reinterpreted view together with its subviews, not a single view; and the rejections that replace a suppressed model-conformance rejection cover fewer parties than the blanket rejection did.
The routed rejections cover at most as many parties as the suppressed blanket rejection (equality is possible), and the routing-context class doc now uses the same suppress-and-partially-replace framing as the method doc.
…ker changes Post-rebase adaptations for the digital-asset#554 review fixes now in this branch's base: - checker Result.inconsistencies -> hostedInconsistencies at the router's inconsistenciesForView and the four router-test assertion sites, - the shared test fixtures that digital-asset#554 slimmed away are restored for the router test (validator mocks, view positions, withConfirmers, requestId, alarm-assertion helper), with view-position values matching their names as promised in digital-asset#554 review reply 3521058115; relative ordering of the positions is preserved, so no order-sensitive assertion is affected.
…allResult Addresses the post-merge MINOR from the digital-asset#554 approval (discussion r3529116181): the fixture is an ExternalCallResult, not an output; renamed at the declaration and all usage sites.
cfd179a to
b96374f
Compare
…r and router Adaptations owed to the review fixes cherry-picked from digital-asset#554/digital-asset#555: - checker Result.inconsistencies -> hostedInconsistencies at the remaining router and router-test call sites, - factory abstainsForView call adapted to the merged nonRejectingConfirmingParties parameter, - factory disagreement-details truncation dropped alongside the removed length constant (Pretty already bounds the output), - shared test fixtures that digital-asset#554 slimmed away restored for the router and factory tests, with view-position values matching their names (as promised in digital-asset#554 review reply 3521058115); relative ordering of the positions is unchanged, so no order-sensitive assertion is affected.
…ication Call-stack verification of the restructured docs (the digital-asset#555 method) refuted four claims; the code is unchanged, only the wording: - validateExternalCalls no longer claims a key with disagreeing recorded outputs anywhere in the request is never re-validated: the single-output aggregation is scoped to the selected occurrences, so if one side of a disagreement has no hosted, non-rejecting checker, the remaining side is still re-validated for the parties that check it (the dropped side stays covered by the consistency check's rejections and alarms). - The model-conformance suppression comment no longer claims every routed error surfaces in exactly one logged rejection: affected views log one rejection each when emitting it, and a view rejected as malformed for another reason subsumes the external-call rejection; the visible-disagreement alarm covers the error in every case. - The router class doc names the checks the suite actually runs concurrently with (model conformance, internal consistency) instead of 'the other validation suites', and describes the remaining parties as responding with the view's own verdict rather than 'approval otherwise'. - Re-validation timing is stated as 'kicked off without awaiting the model-conformance result' (the result may already be available), and the abstain sentence carries its otherwise-approved-views qualifier.
Post-rebase adaptation: digital-asset#555 renamed the shared fixture per the digital-asset#554 approval; this branch's test copies still used the old name.
Sweep of the PR slice per review request: the routing context's two lazy vals, the router test's factory and key fixtures, and the checker test's party fixtures.
…r and router Adaptations owed to the review fixes cherry-picked from digital-asset#554/digital-asset#555: - checker Result.inconsistencies -> hostedInconsistencies at the remaining router and router-test call sites, - factory abstainsForView call adapted to the merged nonRejectingConfirmingParties parameter, - factory disagreement-details truncation dropped alongside the removed length constant (Pretty already bounds the output), - shared test fixtures that digital-asset#554 slimmed away restored for the router and factory tests, with view-position values matching their names (as promised in digital-asset#554 review reply 3521058115); relative ordering of the positions is unchanged, so no order-sensitive assertion is affected.
…ication Call-stack verification of the restructured docs (the digital-asset#555 method) refuted four claims; the code is unchanged, only the wording: - validateExternalCalls no longer claims a key with disagreeing recorded outputs anywhere in the request is never re-validated: the single-output aggregation is scoped to the selected occurrences, so if one side of a disagreement has no hosted, non-rejecting checker, the remaining side is still re-validated for the parties that check it (the dropped side stays covered by the consistency check's rejections and alarms). - The model-conformance suppression comment no longer claims every routed error surfaces in exactly one logged rejection: affected views log one rejection each when emitting it, and a view rejected as malformed for another reason subsumes the external-call rejection; the visible-disagreement alarm covers the error in every case. - The router class doc names the checks the suite actually runs concurrently with (model conformance, internal consistency) instead of 'the other validation suites', and describes the remaining parties as responding with the view's own verdict rather than 'approval otherwise'. - Re-validation timing is stated as 'kicked off without awaiting the model-conformance result' (the result may already be available), and the abstain sentence carries its otherwise-approved-views qualifier.
Post-rebase adaptation: digital-asset#555 renamed the shared fixture per the digital-asset#554 approval; this branch's test copies still used the old name.
Follow-up to the annotation sweep: the nested RecordingExternalCallValidator's observedKeys val was missed.
|
Suspending review to focus on #567. |
## Summary This PR completes the minimal protocol integration of external calls (PR "6b" of the series tracked in #513, as agreed with @matthiasS-da): recorded external-call results are validated during transaction confirmation, and any disagreement rejects the request on behalf of all hosted confirming parties. The external-call stack (#513): #506 added the transaction-side representation for recording external-call results; #514 added the LF `EXTERNAL_CALL` builtin surface; #518 wired it into Speedy and the LF engine; #522 added transaction protobuf encoding/decoding; #526 added Canton protocol serialization for recorded results; #537 added the participant-side extension-service client; #549–#554 added hashing (V4), the prepared-transaction codec, errors and the handler SPI, engine execution and the validation SPI, extension-service wiring, and the consistency checker. This PR adds the missing runtime consumer. ## Scope - **`ExternalCallCheck`**: a validation suite invoked from `doParallelChecks` in `TransactionProcessingSteps`, following the `ModelConformanceChecker.check` pattern (kicked off there, awaited when confirmation responses are created, network I/O concurrent with the other suites). It checks consistency of the recorded results across the request (`ExternalCallConsistencyChecker`; every visible disagreement is alarmed) and re-validates undisputed recorded outputs against the configured extension service, once per distinct call. - **Verdicts** (via the response factory's existing per-view verdict chain — its structure is unchanged): a disagreement from either part rejects each view on behalf of all its hosted confirming parties (`LOCAL_VERDICT_EXTERNAL_CALL_RESULT_DISAGREEMENT`); a recorded result that cannot be re-validated (e.g. no extension service configured) downgrades approvals to `CANNOT_PERFORM_ALL_VALIDATIONS` abstentions. One consequence of keeping the factory's first-match chain: the new non-malformed rejection ranks with the other consistency rejections, so — like the pre-existing time and contract-consistency rejections — it is reported in preference to a co-occurring malformed rejection from the later suites (authentication, model conformance, internal consistency). - The consistency checker now takes the participant views directly (view validation results do not exist yet at `doParallelChecks` time), and the `ExternalCallValidator` binding is threaded through the participant wiring. - Basic test coverage for the check outcomes and their response translation; completing the coverage is tracked in #564. Requests without recorded external-call results short-circuit before any topology or network access, so the feature remains inert on protocol versions without external-call support and unless an extension service is configured. ## Out of scope (deliberate, per the agreed short-cut) - Per-party verdict routing (distinguishing the affected checking parties per view) — continues in #555/#556 as further improvements, in the context of the fail-fast refactor tracked in #565. - Rejection deduplication. - Full test coverage — tracked in #564. Replay disagreements within a single reinterpretation already surface as model-conformance errors through the merged #552 machinery, independently of this check. ## Coordination With this PR the feature is functional end-to-end on `ProtocolVersion.dev`. The Daml-side PR (digital-asset/daml#23099) can proceed once a Canton snapshot/artifact containing this PR is published, per the coordination steps in #513. Refs #513.
|
Unsubscribing for now as this has been superseded by #567. Happy to resume if you request it. |
Summary
This is PR 7 of 8 in the external-call runtime-integration series tracked in #513, stacked on PR 6.
The external-call stack (#513): #506 added the transaction-side representation for recording external-call results; #514 added the LF
EXTERNAL_CALLbuiltin surface; #518 wired it into Speedy and the LF engine; #522 added transaction protobuf encoding/decoding; #526 added Canton protocol serialization for recorded results; #537 added the participant-side extension-service client. #541 implemented the remaining runtime integration as one PR; this series splits #541 into 8 independently reviewable PRs (each < 1000 lines) and supersedes it.This PR adds the router that turns recorded-result consistency outcomes into confirmation-response routing decisions, directing disagreements to the relevant checking parties.
Scope
This PR adds:
ExternalCallResponseRouter, which maps consistency-check outcomes to confirmation-response routing (including routing disagreements to the relevant checking parties), plus the relocatedViewWithHostedPartiesrouter inputExternalCallResponseRouterTestcovering the router's scenarios directly against its APIOut of scope
Assembly of confirmation responses and activation of validation in transaction processing (PR 8).
Stacking & review
This PR is stacked on PR 6, so its diff against
mainis cumulative. The incremental change for this PR alone:zenith-network:angelol/external-call-06-split-pr6-checker...zenith-network:angelol/external-call-06-split-pr7-router
Refs #513.