Investigate Meeting Summary and post-processing quality regressions

[woosublee]

Summary

Meeting Summary and post-processing currently have quality problems significant enough that both features are being kept off during normal use.

This issue is a place to collect evidence and investigate the full text-processing pipeline. It does not assume that one model or one code path is already proven to be the root cause. Future concrete failures can be added as comments and compared against the known processing flow and recent model-routing changes.

This is separate from the Native Whisper timestamp and Auto Detect fixes. The raw transcription problem is being handled independently.

Current workaround

• Keep post-processing disabled when transcript details must be preserved.
• Keep Meeting Summary disabled until its source quality and active model routing are verified.
• Use the raw transcript as the reference when reporting a failure.

How the pipeline works

The important flow is:

raw transcript
  → post-processing
  → postProcessedTranscript
  → Meeting Summary

Meeting Summary normally uses postProcessedTranscript. It uses the raw transcript only when the processed transcript is empty.

This means a problem introduced by post-processing can look like a Meeting Summary problem later. Investigation must therefore compare all three outputs:

1. Raw transcript
2. Post-processed transcript
3. Final Meeting Summary

Confirmed current behavior

The following sections describe code behavior that has already been verified. They do not, by themselves, prove which behavior caused a particular bad summary.

1. Meeting Summary inherits the initial post-processing choice

When Meeting Summary settings are initialized for the first time, the app copies the current post-processing:

• backend
• model
• fallback model
• output language

If the inherited choice is ready, Meeting Summary can become enabled automatically.

Relevant code:

• Sources/AppState.swift:2393-2419
• Sources/AppState.swift:2949-2969

Why this matters: a user may expect Summary to use a separately chosen cloud model, while it can initially inherit an already-selected Local AI model.

2. Local processing has no cloud fallback

When post-processing or Meeting Summary uses a Local AI backend, the cloud fallback model is intentionally set to nil.

Relevant code:

• Sources/AppState.swift:3950-3978
• Sources/MeetingSummaryService.swift:270-279

Why this matters: a successful but low-quality local result is used as-is, and a local request does not retry with the remembered cloud fallback model.

3. Meeting Summary prefers processed text over raw text

The Summary source uses postProcessedTranscript when it is non-empty. It falls back to rawTranscript only when processed text is empty.

Relevant code:

• Sources/AppState.swift:5663-5690

Why this matters: details removed, rewritten, expanded, or misunderstood during cleanup become the source for the final Summary.

4. Preserve Exact Wording was removed

Before commit 33f528d1, Preserve Exact Wording could bypass normal cleanup and return the raw transcript unchanged when no output language was requested.

After that removal, enabled post-processing always uses the normal cleanup path.

Relevant code:

• Previous behavior: Sources/AppState.swift@33f528d1^:7507-7520
• Current behavior: Sources/AppState.swift:8438-8453

Why this matters: the cleanup prompt can remove filler and duplicate starts, but a meeting may also contain meaningful repetition, corrections, tentative opinions, or wording that should remain available for later summarization.

5. Long summaries use non-overlapping character chunks

Transcripts longer than 12,000 characters are split using a character limit without overlap.

Each chunk becomes partial JSON. The final merge receives those partial summaries rather than the complete original transcript.

Relevant code:

• Sources/MeetingSummaryTextChunker.swift:10-55
• Sources/MeetingSummaryService.swift:226-260

Why this matters: a person, decision, question, or action item introduced in one chunk may need information from another chunk that is no longer present during extraction or merge.

Related change timeline

• eead1b40, 9c15866f, cd8ae1e1, and PR Add Local AI backends for Post-processing and Context #232: introduced independent Cloud/Local processing choices and Local execution without cloud fallback.
• 33f528d1: removed Preserve Exact Wording.
• 17ea8676 and PR Add in-app meeting summary (#187) #237: added in-app Meeting Summary, inherited first-run settings, processed-first source selection, and long-text extract/merge.
• b857d65f, 064d79f0, and PR Separate prompt controls from model settings #238: moved prompt controls in Settings. Current investigation found no evidence that this UI move changed the stored prompt text or runtime prompt meaning.

What is confirmed vs. what still needs evidence

Confirmed

• The routing and source-selection behavior described above exists in the current code.
• Local processing does not use a cloud fallback.
• Meeting Summary can receive text already changed by post-processing.
• Long transcripts use an extract-then-merge path with no chunk overlap.
• Preserve Exact Wording no longer bypasses cleanup.

Not yet confirmed

• Whether Local Qwen, a cloud model, the cleanup prompt, context quality, processed-first source selection, or summary chunking is the primary quality problem.
• Whether one issue explains all observed failures.
• Whether failures are limited to long meetings or also affect short transcripts.
• Whether the UI always makes the actually active backend and model clear enough.

These questions require concrete failure examples and same-input A/B comparisons.

Information to add for each future failure

Please add a comment containing as much of the following as possible:

• Quill or Quill Dev
• App version
• Whether post-processing was enabled
• Post-processing backend and model
• Whether Meeting Summary was enabled
• Meeting Summary backend and model
• Whether the raw transcript was already wrong
• Whether the processed transcript first introduced the error
• Whether only the final Summary was wrong
• Approximate transcript length
• Whether the transcript exceeded 12,000 characters
• Expected result
• Actual result
• Reproduction steps
• Whether the failure is consistent when retried
• Relevant meetingSummaryJSON metadata when available:
  • modelID
  • backendKind
  • sourceFingerprint
  • promptVersion

Do not include API keys, credentials, private transcript text, or other sensitive information. A short anonymized example is enough when wording is necessary to understand the failure.

Suggested investigation matrix

Post-processing

• Same raw transcript with post-processing OFF vs. ON
• Same raw transcript with Cloud vs. Local post-processing
• Same raw transcript with default prompt vs. custom prompt
• Compare raw and processed text for missing details, unsupported additions, and excessive rewriting

Meeting Summary

• Same Summary source with Cloud vs. Local Meeting Summary
• Raw source vs. processed source using the same Summary model
• Transcript below 12,000 characters vs. above 12,000 characters
• Short meeting vs. long meeting
• Compare single-pass output with extract/merge output
• Check decision, owner, action item, and open-question relationships across chunk boundaries

Routing and metadata

• Verify that the backend/model shown in the UI matches generation metadata
• Verify the first-run Summary inheritance behavior
• Verify behavior when a Local model is selected but unavailable or returns a poor result
• Confirm whether the lack of Local-to-Cloud fallback is intended for each feature

Acceptance criteria

• Identify whether quality first degrades in raw transcription, post-processing, or Meeting Summary.
• Reproduce at least one failure using the same input across two controlled configurations.
• Record the active backend and model from generation metadata.
• Determine whether the fix belongs to model routing, cleanup behavior, source selection, fallback policy, or Summary chunking.
• Decide whether meeting notes need a raw/exact-wording source option independent of general dictation cleanup.
• Add a semantic or golden regression fixture for each confirmed failure mode.
• Re-enable the features only after the relevant failure modes have repeatable regression coverage.