feat: Additional quill evals-- not answer giving and appropriate emotional pitch#139
Open
aychi1 wants to merge 14 commits into
Open
feat: Additional quill evals-- not answer giving and appropriate emotional pitch#139aychi1 wants to merge 14 commits into
aychi1 wants to merge 14 commits into
Conversation
Removed unnecessary metadata from a code cell in the notebook.
Removed unnecessary metadata and outputs from a code cell.
…esponse Removed unnecessary metadata from a code cell in the notebook.
Removed unnecessary metadata and outputs from a code cell in the Jupyter notebook.
Removed unnecessary metadata from a code cell in the notebook.
There was a problem hiding this comment.
Pull request overview
This PR adds a new set of “productive coaching writing feedback” evaluators under evals/feedback/productive-coaching-writing-feedback/, each packaged with prompt assets (system/user), JSON input/output schemas, fixture sets, and a demonstration notebook.
Changes:
- Added multiple new evaluator bundles (configs + prompts + schemas + fixtures) for productive-coaching feedback dimensions.
- Added example notebooks demonstrating how to load assets, verify prompt SHA256, and run the evaluator + fixtures.
- Added fixture datasets (including “expert sniff test” derived cases for some dimensions).
Reviewed changes
Copilot reviewed 49 out of 49 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| evals/feedback/productive-coaching-writing-feedback/not-answer-giving/user.txt | User prompt template for the “not answer giving” dimension. |
| evals/feedback/productive-coaching-writing-feedback/not-answer-giving/system.txt | Rubric + instructions for “not answer giving” scoring. |
| evals/feedback/productive-coaching-writing-feedback/not-answer-giving/output_schema.json | Output JSON schema for “not answer giving”. |
| evals/feedback/productive-coaching-writing-feedback/not-answer-giving/input_schema.json | Input JSON schema for “not answer giving”. |
| evals/feedback/productive-coaching-writing-feedback/not-answer-giving/fixtures.json | Fixtures for “not answer giving”. |
| evals/feedback/productive-coaching-writing-feedback/not-answer-giving/example_notebook.ipynb | Demo notebook for “not answer giving”. |
| evals/feedback/productive-coaching-writing-feedback/not-answer-giving/config.json | Evaluator config for “not answer giving”. |
| evals/feedback/productive-coaching-writing-feedback/manageable/user.txt | User prompt template for the “manageable” dimension. |
| evals/feedback/productive-coaching-writing-feedback/manageable/system.txt | Rubric + instructions for “manageable” scoring. |
| evals/feedback/productive-coaching-writing-feedback/manageable/output_schema.json | Output JSON schema for “manageable”. |
| evals/feedback/productive-coaching-writing-feedback/manageable/input_schema.json | Input JSON schema for “manageable”. |
| evals/feedback/productive-coaching-writing-feedback/manageable/fixtures.json | Fixtures for “manageable”. |
| evals/feedback/productive-coaching-writing-feedback/manageable/example_notebook.ipynb | Demo notebook for “manageable”. |
| evals/feedback/productive-coaching-writing-feedback/manageable/config.json | Evaluator config for “manageable”. |
| evals/feedback/productive-coaching-writing-feedback/appropriate-feedback/user.txt | User prompt template for “appropriate feedback decision”. |
| evals/feedback/productive-coaching-writing-feedback/appropriate-feedback/system.txt | Rubric + instructions for “appropriate feedback decision”. |
| evals/feedback/productive-coaching-writing-feedback/appropriate-feedback/output_schema.json | Output JSON schema for “appropriate feedback decision”. |
| evals/feedback/productive-coaching-writing-feedback/appropriate-feedback/input_schema.json | Input JSON schema for “appropriate feedback decision”. |
| evals/feedback/productive-coaching-writing-feedback/appropriate-feedback/fixtures.json | Fixtures for “appropriate feedback decision”. |
| evals/feedback/productive-coaching-writing-feedback/appropriate-feedback/example_notebook.ipynb | Demo notebook for “appropriate feedback decision”. |
| evals/feedback/productive-coaching-writing-feedback/appropriate-feedback/config.json | Evaluator config for “appropriate feedback decision”. |
| evals/feedback/productive-coaching-writing-feedback/appropriate-emotional-pitch/user.txt | User prompt template for “appropriate emotional pitch”. |
| evals/feedback/productive-coaching-writing-feedback/appropriate-emotional-pitch/system.txt | Rubric + instructions for “appropriate emotional pitch”. |
| evals/feedback/productive-coaching-writing-feedback/appropriate-emotional-pitch/output_schema.json | Output JSON schema for “appropriate emotional pitch”. |
| evals/feedback/productive-coaching-writing-feedback/appropriate-emotional-pitch/input_schema.json | Input JSON schema for “appropriate emotional pitch”. |
| evals/feedback/productive-coaching-writing-feedback/appropriate-emotional-pitch/fixtures.json | Fixtures for “appropriate emotional pitch”. |
| evals/feedback/productive-coaching-writing-feedback/appropriate-emotional-pitch/example_notebook.ipynb | Demo notebook for “appropriate emotional pitch”. |
| evals/feedback/productive-coaching-writing-feedback/appropriate-emotional-pitch/config.json | Evaluator config for “appropriate emotional pitch”. |
| evals/feedback/productive-coaching-writing-feedback/anchored-in-student-response/user.txt | User prompt template for “anchored in student response”. |
| evals/feedback/productive-coaching-writing-feedback/anchored-in-student-response/system.txt | Rubric + instructions for “anchored in student response”. |
| evals/feedback/productive-coaching-writing-feedback/anchored-in-student-response/output_schema.json | Output JSON schema for “anchored in student response”. |
| evals/feedback/productive-coaching-writing-feedback/anchored-in-student-response/input_schema.json | Input JSON schema for “anchored in student response”. |
| evals/feedback/productive-coaching-writing-feedback/anchored-in-student-response/fixtures.json | Fixtures for “anchored in student response”. |
| evals/feedback/productive-coaching-writing-feedback/anchored-in-student-response/example_notebook.ipynb | Demo notebook for “anchored in student response”. |
| evals/feedback/productive-coaching-writing-feedback/anchored-in-student-response/config.json | Evaluator config for “anchored in student response”. |
| evals/feedback/productive-coaching-writing-feedback/actionable-revision/user.txt | User prompt template for “actionable revision”. |
| evals/feedback/productive-coaching-writing-feedback/actionable-revision/system.txt | Rubric + instructions for “actionable revision”. |
| evals/feedback/productive-coaching-writing-feedback/actionable-revision/output_schema.json | Output JSON schema for “actionable revision”. |
| evals/feedback/productive-coaching-writing-feedback/actionable-revision/input_schema.json | Input JSON schema for “actionable revision”. |
| evals/feedback/productive-coaching-writing-feedback/actionable-revision/fixtures.json | Fixtures for “actionable revision”. |
| evals/feedback/productive-coaching-writing-feedback/actionable-revision/example_notebook.ipynb | Demo notebook for “actionable revision”. |
| evals/feedback/productive-coaching-writing-feedback/actionable-revision/config.json | Evaluator config for “actionable revision”. |
| evals/feedback/productive-coaching-writing-feedback/acknowledges-strength/user.txt | User prompt template for “acknowledges strength”. |
| evals/feedback/productive-coaching-writing-feedback/acknowledges-strength/system.txt | Rubric + instructions for “acknowledges strength”. |
| evals/feedback/productive-coaching-writing-feedback/acknowledges-strength/output_schema.json | Output JSON schema for “acknowledges strength”. |
| evals/feedback/productive-coaching-writing-feedback/acknowledges-strength/input_schema.json | Input JSON schema for “acknowledges strength”. |
| evals/feedback/productive-coaching-writing-feedback/acknowledges-strength/fixtures.json | Fixtures for “acknowledges strength”. |
| evals/feedback/productive-coaching-writing-feedback/acknowledges-strength/example_notebook.ipynb | Demo notebook for “acknowledges strength”. |
| evals/feedback/productive-coaching-writing-feedback/acknowledges-strength/config.json | Evaluator config for “acknowledges strength”. |
Comments suppressed due to low confidence (3)
evals/feedback/productive-coaching-writing-feedback/anchored-in-student-response/example_notebook.ipynb:163
- Typo in docstring: “wheter” → “whether”.
" Evaluate wheter feedback is anchored in student response using the canonical\n",
evals/feedback/productive-coaching-writing-feedback/actionable-revision/example_notebook.ipynb:176
- Typo in docstring: “wheter” → “whether”.
" Evaluate wheter feedback gives a clear next step when revision is needed using the canonical\n",
evals/feedback/productive-coaching-writing-feedback/acknowledges-strength/example_notebook.ipynb:163
- Typo in docstring: “wheter” → “whether”.
" Evaluate wheter feedback acknowledges strengths in the student's text using the canonical\n",
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "%pip install -qU langchain-openai langchain pydantic textstat typing_extensions" |
czi-fsisenda
approved these changes
Jul 24, 2026
czi-fsisenda
left a comment
czi-fsisenda
left a comment
Contributor
There was a problem hiding this comment.
LGTM! 🚀
Check with Adnan about when we should merge this.
adnanrhussain
approved these changes
Jul 24, 2026
adnanrhussain
left a comment
adnanrhussain
left a comment
Contributor
There was a problem hiding this comment.
lgtm. A couple of references need to be updated
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import pprint as pp\n", |
Contributor
There was a problem hiding this comment.
nit: duplicate import
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import pprint as pp\n", |
| "\n", | ||
| " n = len(results)\n", | ||
| " n_exact = sum(1 for r in results if r[\"status\"] == \"exact\")\n", | ||
| " n_adj = sum(1 for r in results if r[\"status\"] == \"adjacent\")\n", |
Contributor
There was a problem hiding this comment.
this is not binary like AEP?
| "def is_not_answer_giving(student_text: str, feedback_text: str):\n", | ||
| " \"\"\"\n", | ||
| " Evaluate whether the feedback avoids giving away the answer using the canonical\n", | ||
| " config in evals/prompts/purpose/config.json + system.txt + user.txt.\n", |
Contributor
There was a problem hiding this comment.
P0 -> pls update ref
| "# Load source-of-truth assets: config.json + every prompt file declared\n", | ||
| "# in config.steps[*].prompt.messages\n", | ||
| "# -------------------------------------------------------------------------\n", | ||
| "# The canonical evaluator definition lives in evals/prompts/purpose.\n", |
Contributor
There was a problem hiding this comment.
P0 - pls update reference
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two new evals: not answer giving and appropriate emotional pitch.