Skip to content

evaluation_protocol.md

Latest commit

 

History

History
139 lines (104 loc) · 3.41 KB

evaluation_protocol.md

File metadata and controls

139 lines (104 loc) · 3.41 KB
Raw

Evaluation Protocol

This document describes how to evaluate artifacts produced by an external agent or sandbox.

The protocol is intentionally independent of the repository's optional sandbox/ wrapper. Any agent runner can be used as long as it writes the run directory described below.

Inputs

The construction pipeline exports a task file:

eval_tasks.jsonl

Each task row should contain:

  • task_id
  • agent_prompt
  • evaluation.rule_based
  • semantic_rubric
  • optional metadata such as big_class, subclass_id, input_fixture_paths

The evaluator does not run the sandbox. The user should run each task with their own agent and save outputs using this layout:

runs/<run_name>/<task_id>/
  response.txt
  artifacts/
    <agent output files>
  task.json       # optional copy of the task row
  history.jsonl   # optional
  metadata.json   # optional model/harness/token/time metadata

response.txt and artifacts/ are required. history.jsonl and metadata.json are audit-only.

For reliable auditing, also write task.json and prompt.txt when possible. task.json should be the exact task row from eval_tasks.jsonl; prompt.txt should be the exact prompt sent to the agent, including any fixture-path instructions you added at runtime.

Judge Routing

Main score routing is strict:

  • Text-only cases use the text judge.
  • If a case has any visual artifact suffix, it must use the visual judge.
  • If visual evidence preparation or visual judge fails, main_score is left empty and main_score_error records the reason.
  • Visual cases never fall back to text judge as the main score.

Visual suffixes:

.html .htm .pdf .pptx .ppt .docx .doc .xlsx .xls .png .jpg .jpeg .webp .svg

Text suffixes:

.txt .md .markdown .json .jsonl .py .js .ts .tsx .jsx .sh .bash .zsh .css .scss .sql .yaml .yml .toml .ini .csv .tsv

Runtime/intermediate files are excluded before judging:

ocr/
verify_temp/
__pycache__/
venv/
.venv/
site-packages/
crop*
gray_*
autoc_*
*_tsv.tsv

Multimodal Evidence

The visual judge receives images only:

  • HTML/SVG: Playwright full-page screenshot.
  • XLSX/XLS: sheet HTML rendered with Playwright.
  • PDF: page screenshots via pdftoppm.
  • DOC/DOCX/PPT/PPTX: LibreOffice converts to PDF, then pdftoppm screenshots.
  • PNG/JPG/JPEG/WEBP: uploaded directly.

Default budgets:

--max-output-images 32
--max-output-pages 6
--max-output-text-chars 12000

If Playwright or LibreOffice is unavailable, the evaluator records warnings in:

<task_id>/<visual_judge_name>/prepared_output_evidence.json

and the visual main score is not silently replaced by text.

Commands

Validate a run:

cd /path/to/EnterpriseClawBench/evaluation && \
python -m enterprise_clawbench_evaluation.cli validate-run \
  --tasks /path/to/eval_tasks.jsonl \
  --run-dir /path/to/runs/demo_run

Judge a run:

cd /path/to/EnterpriseClawBench/evaluation && \
export DMX_API_KEY='<your api key>' && \
python -m enterprise_clawbench_evaluation.cli judge-run \
  --tasks /path/to/eval_tasks.jsonl \
  --run-dir /path/to/runs/demo_run \
  --out-dir /path/to/outputs/judge_demo \
  --base-url https://www.dmxapi.cn/v1 \
  --text-judge-model claude-sonnet-4-6 \
  --visual-judge-model claude-sonnet-4-6

Outputs:

judge_results.jsonl
judge_summary.json
REPORT_EVALUATION.md
<task_id>/semantic_sonnet_text_judge/
<task_id>/semantic_sonnet_visual_judge/