feat(runtime): evidence-forced objective detection with 3-state and scoring floor - #2
Open
watanabefam wants to merge 48 commits into
Open
feat(runtime): evidence-forced objective detection with 3-state and scoring floor#2watanabefam wants to merge 48 commits into
watanabefam wants to merge 48 commits into
Conversation
The chat auto-scrolled to the bottom on every message render, making it impossible to scroll up and read earlier messages while the conversation was ongoing. Now only auto-scrolls if the user is within 60px of the bottom. If they've scrolled up to read history, their position is respected.
…or reliable scroll justify-content: flex-end causes inconsistent scrollTop behavior across browsers when combined with overflow-y: auto. Replaced it with margin-top: auto on the inner container, which pushes messages to the bottom naturally while keeping scroll position math consistent. Also added overflow-x: hidden, overscroll-behavior: contain, and scroll-behavior: smooth for a more reliable scroll experience.
…ring floor Detection prompt now requires evidence quotes and 3-state values (0=not met, 1=partial, 2=complete) instead of a flat list of IDs. UI shows yellow count badges for partial progress (e.g. '1/2'). Scoring uses detector state as a floor: 75% of weight for complete, 25% for partial - prevents 0/10 when objectives were clearly met.
ObjectiveState is an enum (needs runtime value), but was imported with in multiple files, causing it to be undefined at runtime. Changed to regular imports for the enum while keeping type-only imports for interfaces and type aliases.
…red topics The detector was evaluating the full conversation transcript, crediting objectives when topics came up naturally through the AI persona's responses. Changed the prompt to explicitly only count the LEARNER's active contributions — if a topic only appeared because the PERSONA brought it up unprompted, it does not count toward the objective.
Detection now returns state 1 (partial) vs 2 (complete) using 'id: state' format. Sidebar shows SVG circular progress rings: grey empty ring = not met, yellow arc with count = partial, green filled with checkmark = complete.
…oxy maxTokens The snapshot builder expected Set<string> but completedObjectives is now a Map. Also increased proxy maxTokens default from 512 to 2048 so longer responses (scoring, detection) don't get truncated.
…ntributions Detection now returns state 1 (vague/barely) or state 2 (clear, specific, on-topic) instead of binary found/not-found. Merge logic respects these states — a high-quality question turns green immediately on first detection, while vague contributions stay yellow until improved. Replaced code-based yellow→green promotion with LLM-based quality assessment.
LLM was only assessing quality of the best contribution, ignoring quantity requirements like 'ask at least two questions'. Updated prompt to require meeting both thresholds for state 2.
Circle now splits into segments when count is available (e.g. '1/2'). Each segment represents one required item — filled if quality met, empty if not. Gives a clear visual of both quantity and quality.
…jectives Debrief now animates: score counts up from 0 with ease-out, stars fill one by one as the number climbs. Summary fades in, per-objective scores slide up with staggered timing, close button appears last.
Rings now fill silently — no center text, just proportional arc fill based on score (single arc) or per-item scores (segmented arcs).
Replaced CSS transition + layout flush approach with Web Animations API which reliably animates from the current stroke-dashoffset to the target value regardless of DOM context or update frequency.
Detection and scoring now pass { jsonMode: true } so the proxy forwards
response_format: { type: 'json_object' } to DeepSeek. This reduces format
deviations like 'STATE=2' from ~15% to ~2%.
When LLM returns state 1 with 0 quality scores, the ring fraction was 0 — visually identical to not-met grey. Now floors at 5% for partial state so any yellow state is visibly distinguishable.
…om greetings Detection now uses jsonMode for reliable JSON output. Prompt explicitly excludes greetings and pleasantries. Returns empty array for social chit-chat, correctly detects substantive questions.
jsonMode caused schema drift — DeepSeek invented different JSON shapes each call. Line format is more predictable. Kept stricter prompt that excludes greetings and pleasantries. Detection returns state 2 for all results; code-based promotion handles yellow→green progression.
Added isSubstantive check before detection calls — messages under 16 chars or matching greeting patterns (hi, hello, hey, thanks, etc.) skip detection entirely. This is model-agnostic and 100% reliable unlike prompt-level instructions which DeepSeek ignores ~20% of the time.
Messages under 16 chars matching greeting patterns (hi, hello, hey, thanks, ok, yes, no, etc.) skip detection entirely. Longer messages with substantive content still get evaluated by the LLM.
Detection prompt now includes rubric's full credit description as a comparative reference — LLM compares student text against the standard. Code-level greeting filter tightened: catches 'Hi. Yes wine' style messages, skips short messages that start with greeting words unless they contain a real question.
Added negative criteria extracted from rubric partial/zero descriptions. Prompt now includes both what counts and what does NOT count per objective. Detection output supports [evidence quote] format for transparency. Parser extracts evidence brackets while remaining backward compatible.
Detection returns binary found/not-found. Merge logic promotes first detection to yellow, re-detection to green. LLM quality assessment moved entirely to the final scorer which is more reliable.
Persona responses were biasing the detector — LLM scored Lydia's answer instead of the student's question. Now only learner messages are sent to the detection call.
Typing 'Debug: what did you detect?' sends the query directly to the LLM with a simple system prompt — no character persona, no turn tracking, no detection. Response shown as a system note.
Updated composition objectives with concrete historical anchors (Tyrian purple, Roman colony, Gangites River) and explicit negative examples (what does NOT count). Updated blank.html template with guidance for writing specific, detectable objectives.
…ements Objectives now use 'for example' language to keep role-play natural while still giving the detector concrete anchors. Counter-examples show what does NOT count without being prescriptive.
… attribute Added detect attribute to edu-objective elements. Human text between tags is shown in sidebar. AI-facing detect attribute is sent to the detection LLM with specific examples, counter-examples, and DOES NOT COUNT rules — keeping the student view natural while improving accuracy.
… counter-examples detect attributes now include exhaustive COUNTS and DOES NOT COUNT lists covering every false positive we've observed: greetings, pleasantries, pirate jokes, water/food requests, self-disclosures, anachronisms, and vague references. Designed for DeepSeek and other instruction-following models.
Updated agent guidance to always include detect attribute with COUNTS and DOES NOT COUNT rules. Documented the attribute in composition format spec. Updated example in SKILL.md to show the pattern.
…tate command Short messages under 30 chars with pirate/food/drink keywords are now filtered. Debug: state shows current detection state with colors and evidence. Debug: detect '...' uses AI-facing objective text.
Debug: state shows real detection state. Debug: filter 'text' tests the code-level filter. No more LLM-generated meta-commentary.
…n on test input Debug: detect 'text' shows code filter result, then calls the real detection function and shows what the LLM returns. Gives full transparency into the detection pipeline.
Debug: on shows filter results and LLM detection output automatically on every turn. Debug: off silences it. The ⚙ notes show what the code filter decided and what the LLM returned, so no manual debug commands needed.
…d merge diffs Debug mode now shows: input text → filter decision (with complete reasons) → LLM response (with AI-facing objective text) → merge result (state changes per objective).
…talk UNLESS it also contains a legitimate topic word (Paul, Lydia, baptism, Roman, etc.)
…s filter analysis on it Instead of showing help, Debug: why did 'Arrrgh...'? now extracts the quoted text and shows whether it was blocked by filter or sent to LLM. Without quotes, shows current detection state.
Any debug query now: (1) shows code filter analysis for quoted text, (2) calls LLM with full system state — AI-facing objective texts, rubric, detection state, filter result, learner messages, conversation history. LLM answers specifically using objective IDs and filter rules.
…ic nonsense detector Filter now uses general heuristics: no question mark, no question word, under 50 chars, no topic word → nonsense. No more enumerating specific pirate or food/drink vocabulary. LLM handles nuance via detect attribute.
Animation played but never updated the SVG attribute, so the next detection read the stale initial value. onfinish now sets the attribute to the final animated value.
…rial politics; add 'examples not exhaustive' instruction The detect text was too literal — DeepSeek treated the COUNTS list as exhaustive and missed valid topics like emperor worship. Now marks examples as non-exhaustive and explicitly instructs to count any on-topic historical question regardless of specific keywords.
…OUNTS approach COUNTS examples should be marked as non-exhaustive so agents write flexible detect texts that don't require specific keywords. Added 'Count any on-topic question — do not require specific keywords' instruction to all examples.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The objective detector returned a flat list of "done" IDs with no evidence requirement, causing false positives (greetings triggering completion). There was no partial progress state, so an objective requiring "ask 2 questions" was either all or nothing. The final AI scorer could return 0/10 even when all objectives were clearly met.
Changes
Testing