Skip to content

feat(checkup lint): implement pdd checkup lint prompt linter#1154

Draft
DianaTao wants to merge 8 commits into
promptdriven:mainfrom
DianaTao:codex/pr-1122-prompt-lint
Draft

feat(checkup lint): implement pdd checkup lint prompt linter#1154
DianaTao wants to merge 8 commits into
promptdriven:mainfrom
DianaTao:codex/pr-1122-prompt-lint

Conversation

@DianaTao
Copy link
Copy Markdown

Closes #829

Overview

This PR implements pdd checkup lint, a comprehensive, prompt-native static analysis engine that enforces strict prompt engineering guidelines and checks prompts and user stories for ambiguity before executing downstream agentic actions.

The linter is designed with two distinct, complementary phases:

  1. Deterministic Static Linter: High-performance, fast, and offline checks using pre-defined regex structures and tokenization rules.
  2. Interactive/Cloud-Backed LLM Linter: Advisory semantic linter powered by PDD Cloud that checks prompts for complex natural-language ambiguity.

Technical Details & Architecture

1. Deterministic Linter Engine (pdd/prompt_lint.py)

  • Extracts structured contract tags (<contract_rules>, <requirements>, <acceptance_tests>) and comment blocks (% prose lines).
  • Identifies known vague terms (valid, successful, authorized, etc.) within rules.
  • Glossary/Vocabulary Waivers: Allows developers to explicitly waive ambiguity warnings by defining the term under a <vocabulary>, ## Glossary, or ## Definitions block in the prompt/story.
  • Strict Mode (--strict): Elevates all warnings to errors (exit code 2) and expands the vocabulary check to strict english terms (proper, correct, normal).
  • Observable-Outcome Verification: Validates that all contract rules and acceptance criteria include concrete action verbs (e.g. returns, raises, writes, emits) to ensure formal testability.

2. PDD Cloud LLM Ambiguity Review (pdd/prompts/prompt_lint_LLM.prompt)

  • Enabled via pdd checkup lint --ambiguity.
  • Leverages prompt_lint_LLM.prompt to prompt the LLM for potential semantic misinterpretations.
  • Returns structured advisory reports detailing distinct readings and suggested vocabulary definitions.
  • Operates under a non-blocking mode: failures due to API key exhaustion or network timeouts are degraded gracefully so CI suites are never blocked by cloud outages.

3. Subprocess venv Parity & Unit Test Fixes

To prevent execution discrepancy inside local environments and CI pipelines:

  • Refactored E2E test runs in tests/test_operation_logging_e2e.py to use sys.executable -m pdd instead of "pdd". This prevents path priority issues where a stale global installation of pdd takes precedence over the active workspace wheel in the virtual environment.
  • Patched the unfinished_prompt method in test_generate_test.py's test_generate_test_uses_example_template to ensure unit tests execute completely offline and complete under <0.2s without trying to connect to Google Vertex AI.

Files Added/Modified

  • [NEW] pdd/prompt_lint.py — Linter and glossary parsing core engine.
  • [NEW] pdd/commands/prompt.py — Prompt CLI subcommand for lint.
  • [NEW] pdd/prompts/prompt_lint_LLM.prompt — LLM prompt instructions for ambiguity review.
  • [NEW] tests/commands/test_checkup_prompt_lint.py — CLI integration unit tests.
  • [MODIFY] pdd/commands/checkup.py — Integrated pdd checkup lint subcommand routing.
  • [MODIFY] tests/test_operation_logging_e2e.py — Subprocess command execution venv parity.
  • [MODIFY] tests/test_generate_test.py — Offline unit test execution fixes.

Verification & Staging Results

A. Deterministic Lint Unit Tests (pytest tests/commands/test_checkup_prompt_lint.py)

tests/commands/test_checkup_prompt_lint.py::test_checkup_lint_clean_prompt_json PASSED [ 33%]
tests/commands/test_checkup_prompt_lint.py::test_checkup_lint_reports_warning PASSED [ 66%]
tests/commands/test_checkup_prompt_lint.py::test_checkup_lint_strict_promotes_warning_to_error PASSED [100%]
========================= 3 passed in 0.19s =========================

B. Prompt Generation Offline Unit Tests (pytest tests/test_generate_test.py)

tests/test_generate_test.py::test_generate_test_uses_example_template PASSED [ 80%]
...
======================== 21 passed in 0.18s ========================

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tooling: Add pdd prompt lint --ambiguity to flag vague or undefined terms

1 participant