Add opt-out for sending screenshots to the context model - #208
Conversation
Adds a "Use screen context for smarter dictation" toggle in Settings → Prompts → Context Prompt, plus a Skip button on the Setup wizard's Screen Recording step. Both default to on, so existing users see zero behavior change. Motivation: the context inference is genuinely useful but requires Screen Recording permission, which feels invasive to some users. Every dictation captures a JPEG of the active window and ships it to whatever inference endpoint they have configured. Today there is no way to disable just the screenshot without revoking the OS permission (which also kills the grant, so flipping it back on means re-granting). How it works: AppContextService.captureActiveWindowScreenshot gates on the new use_screenshot_context UserDefaults flag (default true). When off it returns the same (nil, nil, error) tuple shape as the permission-denied path — so the existing screenshotDataURL == nil fallback handles everything downstream. The error string is user-facing copy that points the user back at the toggle, and it surfaces in the Run Log's Capture Context step via a new no-image caption added in this PR. Test button in Settings refuses to run when the toggle is off, with a clear message pointing back at the toggle. Observability: adds an os_log subsystem com.zachlatta.freeflow category Context that emits one line per capture outcome (skipped / blocked / captured-active-window / captured-focused-title / captured-fullscreen / failed). Useful for verifying gate behavior from outside the app via log show. 79 insertions, 0 deletions. Purely additive — no upstream logic modified. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughA new user-configurable ChangesScreenshot Context Opt-In
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
does not build Sources/SettingsView.swift:1696:13: error: cannot find 'SettingsSubcard' in scope |
2 participants
Summary
Adds a "Use screen context for smarter dictation" toggle in Settings → Prompts → Context Prompt, plus a Skip button on the Setup wizard's Screen Recording step. Both default to on, so existing users see zero behavior change.
Why
The context inference is genuinely useful, but it requires the Screen Recording permission — and every dictation captures a JPEG of the active window and ships it to whatever inference endpoint the user has configured. There is currently no way to disable just the screenshot without revoking the OS permission entirely (which also kills the grant, so flipping it back on means re-granting through System Settings).
This PR adds a clean opt-out:
How it works
AppContextService.captureActiveWindowScreenshotgates on a newuse_screenshot_contextUserDefaults flag (defaulttrue). When off, it returns the same(nil, nil, error)tuple shape as the existing permission-denied path — so the existingscreenshotDataURL == nilfallback throughout the codebase keeps working without modification. No new code paths, just a clean early return with a user-facing error string that points back at the toggle.The Run Log's "Capture Context" step (Step 1) gets a small no-image caption when
contextScreenshotDataURLis nil butcontextScreenshotStatusis non-empty — so the user can see at a glance why a given dictation has no screenshot attached.Test button in the Context Prompt section refuses to run when the toggle is off, with a clear message pointing back at the toggle, instead of silently running a degraded test.
Observability
Adds an
os_logsubsystemcom.zachlatta.freeflowcategoryContextthat emits one line per capture outcome:screenshot skipped — disabled in settings (Use screen context for smarter dictation = off)screenshot blocked — Screen Recording permission not grantedscreenshot captured via active-window match (N bytes)screenshot captured via focused-title match (N bytes)screenshot captured via full-screen fallback (N bytes)screenshot failed — CGWindowListCreateImage returned nilscreenshot failed — image could not be encoded within size limitsUseful for verifying gate behavior from outside the app:
What changed
Sources/AppState.swift— newuseScreenshotContext@PublishedBool, persisted at UserDefaults keyuse_screenshot_context, defaulttrue.Sources/AppContextService.swift—import os.log, OSLog declaration, gate incaptureActiveWindowScreenshot, and oneos_logline on every outcome path.Sources/SettingsView.swift— adds the toggle subcard at the top ofcontextPromptSection, the test-button refusal, and the no-image caption in the Run Log's Capture Context step.Sources/SetupView.swift— adds a Skip button to the Screen Recording step that setsuseScreenshotContext = falseand advances without granting the permission.+79 / -0. Purely additive, no upstream logic modified.
Testing
Verified locally on a downstream build:
log show --predicate 'subsystem == "com.zachlatta.freeflow" AND category == "Context"'showsscreenshot captured via active-window match (N bytes)per dictation, same asmaintoday.screenshot skipped — disabled in settingsinstead, dictation still completes using the existing fallback context (app + window + selection), and the Run Log's Capture Context step displays the new "Screen context disabled in settings..." caption in place of the missing image.Screen context is disabled. Toggle "Use screen context for smarter dictation" on above to run this test.instead of running a degraded test.Screenshots
The Settings toggle in Prompts → Context Prompt:
The Run Log's Capture Context step when the toggle is off (new no-image caption in this PR):
The Test Context Prompt button refusal when the toggle is off:
Notes
Default-on means zero regression for existing users. Defaults can flip later in a separate PR if that becomes desired.
Summary by CodeRabbit
Release Notes
New Features
Improvements