Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
b3bb15e
chore(deps): update actions/upload-pages-artifact action to v5
renovate[bot] Apr 29, 2026
1a4573f
chore(deps): update astral-sh/setup-uv action to v8
renovate[bot] Apr 29, 2026
ca9121e
chore(deps): update react monorepo to v19.2.6
renovate[bot] May 6, 2026
8aad832
fix(deps): update dependency respx to >=0.23.1,<0.24.0
renovate[bot] May 18, 2026
65e8a59
fix(deps): update dependency importlib-resources to v7
renovate[bot] May 18, 2026
610ebfc
fix(deps): update dependency openai to >=2.38.0,<2.39.0
renovate[bot] May 21, 2026
fa15c9a
chore(deps): update dependency typescript to v6.0.3
renovate[bot] Jun 1, 2026
3ea80b9
fix(deps): update dependency pytest-asyncio to >=1.4.0,<1.5.0
renovate[bot] Jun 1, 2026
8593200
fix(deps): update dependency rich to v15
renovate[bot] Jun 1, 2026
7d1a823
docs: update README
colbytimm Jun 1, 2026
7689016
feat(adapters): add Azure OpenAI adapter
colbytimm Jun 1, 2026
f090d83
Merge remote-tracking branch 'origin/renovate/openai-2.x' into feat/a…
colbytimm Jun 1, 2026
4b3d934
Merge remote-tracking branch 'origin/renovate/importlib-resources-7.x…
colbytimm Jun 1, 2026
a04d186
Merge remote-tracking branch 'origin/renovate/respx-0.x' into feat/az…
colbytimm Jun 1, 2026
8cd5d5b
Merge remote-tracking branch 'origin/renovate/pytest-asyncio-1.x' int…
colbytimm Jun 1, 2026
8284b2b
Merge remote-tracking branch 'origin/renovate/rich-15.x' into feat/az…
colbytimm Jun 1, 2026
1d35eac
Merge remote-tracking branch 'origin/renovate/astral-sh-setup-uv-8.x'…
colbytimm Jun 1, 2026
e80e01d
Merge remote-tracking branch 'origin/renovate/actions-upload-pages-ar…
colbytimm Jun 1, 2026
2636cd1
Merge remote-tracking branch 'origin/renovate/typescript-6.x-lockfile…
colbytimm Jun 1, 2026
ed76afc
Merge remote-tracking branch 'origin/renovate/react-monorepo' into fe…
colbytimm Jun 1, 2026
c181c6c
fix(adapters): wire azure_openai through CLI/Runner and trim unused m…
colbytimm Jun 1, 2026
d190156
docs: restore status emojis in README tables
colbytimm Jun 1, 2026
6435011
fix(adapters): require Azure OpenAI endpoint with a clear error
colbytimm Jun 1, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/deploy-docs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ jobs:
- name: Build website
run: cd docs && npm run build
- name: Upload artifact
uses: actions/upload-pages-artifact@v4
uses: actions/upload-pages-artifact@v5
with:
path: docs/build

Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/shared-build-and-test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ jobs:
python-version: ${{ inputs.python-version }}

- name: 🚀 Install uv with caching
uses: astral-sh/setup-uv@v7
uses: astral-sh/setup-uv@v8.1.0
with:
enable-cache: true

Expand Down Expand Up @@ -111,7 +111,7 @@ jobs:
python-version: ${{ inputs.python-version }}

- name: 🚀 Install uv with caching
uses: astral-sh/setup-uv@v7
uses: astral-sh/setup-uv@v8.1.0
with:
enable-cache: true

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/shared-publish-package.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ jobs:
python-version: ${{ inputs.python-version }}

- name: 🚀 Install uv
uses: astral-sh/setup-uv@v7
uses: astral-sh/setup-uv@v8.1.0

- name: ⤵️ Download dist artifacts
uses: actions/download-artifact@v8
Expand Down
50 changes: 25 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,17 +15,17 @@
</p>

<p align="center">
<a href="#-quick-start">🏃 Quick-Start</a> - <a href="#-why-promptdrifter">Why PromptDrifter?</a> - <a href="#-demo">🎬 Demo</a> - <a href="#-docs">📚 Docs</a>
<a href="#quick-start">Quick-Start</a> - <a href="#why-promptdrifter">Why PromptDrifter?</a> - <a href="#demo">Demo</a> - <a href="#docs">Docs</a>
<br />
<a href="https://github.com/Code-and-Sorts/PromptDrifter/issues/new?assignees=&template=bug_report.md">🐛 Bug Report</a> - <a href="https://github.com/Code-and-Sorts/PromptDrifter/issues/new?assignees=&template=feature_request.md">Feature Request</a>
<a href="https://github.com/Code-and-Sorts/PromptDrifter/issues/new?assignees=&template=bug_report.md">Bug Report</a> - <a href="https://github.com/Code-and-Sorts/PromptDrifter/issues/new?assignees=&template=feature_request.md">Feature Request</a>
</p>

### PromptDrifter is a one-command CI guardrail, open source platform for catching prompt drift and fails if your LLM answers change.

> [!IMPORTANT]
> **Development Notice**: This project is under active development. Breaking changes may occur between versions. Please check the changelog and release notes before updating.

## 🏃 Quick-Start
## Quick-Start

### Basic Installation
```bash
Expand Down Expand Up @@ -58,13 +58,13 @@ This will create a `promptdrifter.yaml` file in your current directory.
promptdrifter run promptdrifter.yaml
```

#### 🏃💨 Sample Run
#### Sample Run

<p align="center">
<img src="./docs/static/img/promptdrifter-demo.gif" alt="PromptDrifter Demo" width="700"/>
</p>

## Why PromptDrifter?
## Why PromptDrifter?

The landscape of Large Language Models (LLMs) is one of rapid evolution. While exciting, this constant change introduces a critical challenge for applications relying on them: **prompt drift**.

Expand All @@ -74,62 +74,62 @@ Over time, updates to LLM versions, or even subtle shifts in their training data

### Undetected prompt drift can lead to:
<details>
<summary>🚨 Unexpected Failures</summary>
<summary>Unexpected Failures</summary>
Applications or CI/CD pipelines may break silently or with cryptic errors when LLM outputs deviate from expected formats or content.
</details>

<details>
<summary>📉 Degraded User Experience</summary>
<summary>Degraded User Experience</summary>
Features relying on consistent LLM responses can malfunction, leading to user frustration.
</details>

<details>
<summary>⏱️ Increased Maintenance</summary>
<summary>Increased Maintenance</summary>
Engineers spend valuable time diagnosing issues, tracing them back to changed LLM behavior rather addressing features and bugs in code.
</details>

<details>
<summary>🚧 Blocked Deployments</summary>
<summary>Blocked Deployments</summary>
Uncertainty about LLM stability can slow down development cycles and deployment frequency.
</details>

---

### PromptDrifter tackles these challenges head-on by providing:
<details>
<summary>🛡️ Automated Guardrails</summary>
<summary>Automated Guardrails</summary>
A simple, command-line driven tool to integrate LLM response validation directly into your development and CI/CD workflows.
</details>

<details>
<summary>🔍 Early Drift Detection</summary>
<summary>Early Drift Detection</summary>
By comparing LLM outputs against version-controlled expected responses or predefined patterns (like regex), **PromptDrifter** catches deviations as soon as they occur.
</details>

<details>
<summary>⚙️ Consistent and Reliable Applications</summary>
<summary>Consistent and Reliable Applications</summary>
Ensures that your LLM-powered features behave predictably by failing builds when significant response changes are detected, *before* they impact users or production systems.
</details>

<details>
<summary>🔌 Model Agnostic Design</summary>
<summary>Model Agnostic Design</summary>
Through a flexible adapter system, PromptDrifter can interact with various LLM providers and models (e.g., OpenAI, Ollama, and more to come).
</details>

<details>
<summary>📝 Declarative Test Suites</summary>
<summary>Declarative Test Suites</summary>
Define your prompt tests in easy-to-understand YAML files, making them simple to create, manage, and version alongside your codebase.
</details>

<details>
<summary>😌 Developer Peace of Mind</summary>
<summary>Developer Peace of Mind</summary>
Build with greater confidence, knowing you have a safety net that monitors the stability of your critical prompt interactions.
</details>

> [!NOTE]
> By making prompt-response testing a straightforward and automated part of your workflow, **PromptDrifter** helps you harness the power of LLMs while mitigating the risks associated with their dynamic nature.

## 🎬 Demo
## Demo

### Running with cache

Expand All @@ -149,27 +149,27 @@ Build with greater confidence, knowing you have a safety net that monitors the s
<img src="./docs/static/img/promptdrifter-demo-failure.gif" alt="PromptDrifter Demo" width="700"/>
</p>

## 🤖 Supported LLM Adapters
## Supported LLM Adapters

PromptDrifter is designed to be extensible to various Large Language Models through its adapter system. Here's a current list of supported and planned adapters:

| Provider / Model Family | Adapter Status | Details / Model Examples | Linked Issue |
| :---------------------- | :------------- | :----------------------------------------------------------------------- | :----------- |
| **GPT (OpenAI)** | ✅ Available | `gpt-3.5-turbo`, `gpt-4`, `gpt-4o`, etc. | N/A |
| **Azure OpenAI** | ✅ Available | `gpt-3.5-turbo`, `gpt-4`, `gpt-4o`, etc. | N/A |
| **Ollama** | ✅ Available | `llama3`, `mistral`, `gemma`, etc. | N/A |
Comment thread
colbytimm marked this conversation as resolved.
| **Claude (Anthropic)** | ✅ Available | `claude-3-7-sonnet`, `claude-3-5-sonnet`, `claude-3-opus` | N/A |
| **Gemini (Google)** | ✅ Available | `gemini-2.5-pro`, `gemini-2.5-flash`, `gemini-2.0-flash-thinking` | N/A |
| **Qwen (Alibaba)** | ✅ Available | `qwen3-30b-a3b`, `qwq-32b` | N/A |
| **Grok (xAI)** | ✅ Available | `grok-3`, `grok-2`, etc. | N/A |
| **DeepSeek** | ✅ Available | `deepseek-r1`, `deepseek-v3-0324`, etc. | N/A |
| **Mistral** | ✅ Available | `mistral-small-24b-instruct-2501`, `mistral-small-3.1-24b-instruct-2503` | N/A |
| **Azure OpenAI** | 📋 To Do | `gpt-3.5-turbo`, `gpt-4`, `gpt-4o`, etc. | [#10](https://github.com/Code-and-Sorts/PromptDrifter/issues/10) |
| **Llama (Meta)** | 📋 To Do | `llama-4-maverick`, `llama-4-scout`, etc. | [#11](https://github.com/Code-and-Sorts/PromptDrifter/issues/11) |


If there's a model or provider you'd like to see supported, please [open a feature request](https://github.com/Code-and-Sorts/PromptDrifter/issues/new?assignees=&template=feature_request.md) or consider contributing an adapter!

## 🧪 Supported Drift Tests
## Supported Drift Tests

| Name | Config key | Description | Installation |
| :--- | :--------- | :---------- | :----------- |
Expand All @@ -179,13 +179,13 @@ If there's a model or provider you'd like to see supported, please [open a featu
| **Substring Case Insensitive** | `expect_substring_case_insensitive` | Case insensitive substring match | ✅ Core |
| **Text Similarity** | `text_similarity` | Semantic similarity using sentence transformers | `pip install 'promptdrifter[similarity]'` |

## ⚙️ GitHub Action
## GitHub Action

Automate your prompt drift detection by integrating PromptDrifter directly into your GitHub workflows!

We provide a reusable GitHub Action that makes it easy to run your PromptDrifter tests on every push or pull request.

➡️ **Find the PromptDrifter GitHub Action here: [PromptDrifter on GitHub Marketplace](https://github.com/marketplace/actions/promptdrifter)**
**Find the PromptDrifter GitHub Action here: [PromptDrifter on GitHub Marketplace](https://github.com/marketplace/actions/promptdrifter)**

### Basic Usage

Expand Down Expand Up @@ -224,7 +224,7 @@ This action allows you to:

By using the action, you can ensure that any changes to your LLM's responses that violate your defined tests will automatically flag your CI builds, preventing unexpected issues from reaching production.

## 📚 Docs
## Docs

Our documentation is built with Docusaurus and deployed to GitHub Pages.

Expand All @@ -238,14 +238,14 @@ The documentation includes:

If you want to contribute to the documentation, see the [docs/README.md](./docs/README.md) file for instructions.

## 🧑‍💻 Contributing
## Contributing

Follow the [contributing guide](./.github/CONTRIBUTING.md).

## 🔖 Code of Conduct
## Code of Conduct

Please make sure you read the [Code of Conduct guide](./.github/CODE-OF-CONDUCT.md).

## 📝 Changelog
## Changelog

- [0.0.x](./changelogs/0.0.x.md)
20 changes: 10 additions & 10 deletions docs/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 5 additions & 5 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,11 @@ readme = "scripts/pypi-readme.md"
requires-python = ">=3.11"
dependencies = [
"typer (>=0.26.4,<0.27.0)",
"rich (>=14.0.0,<15.0.0)",
"openai (>=2.31.0,<2.32.0)",
"rich (>=15.0.0,<15.1.0)",
"openai (>=2.38.0,<2.39.0)",
"httpx (>=0.28.1,<0.29.0)",
"pyyaml (>=6.0.2,<7.0.0)",
"importlib-resources (>=6.5.2,<7.0.0)",
"importlib-resources (>=7.1.0,<7.2.0)",
"coverage (>=7.8.0,<8.0.0)",
"jsonschema (>=4.23.0,<5.0.0)",
"rapidfuzz (>=3.0.0,<4.0.0)",
Expand Down Expand Up @@ -54,9 +54,9 @@ dev = [
"pytest-cov>=6.1.1",
"pytest-mock>=3.14.0",
"pytest (>=9.0.2,<9.1.0)",
"pytest-asyncio (>=1.3.0,<1.4.0)",
"pytest-asyncio (>=1.4.0,<1.5.0)",
"ruff>=0.11.8",
"respx (>=0.22.0,<0.23.0)",
"respx (>=0.23.1,<0.24.0)",
]
similarity = [
"sentence-transformers>=2.9.0",
Expand Down
2 changes: 2 additions & 0 deletions src/promptdrifter/adapter_manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
from dataclasses import dataclass
from typing import Any, Dict, Optional, Type

from .adapters.azure_openai import AzureOpenAIAdapter
from .adapters.base import Adapter
from .adapters.claude import ClaudeAdapter
from .adapters.deepseek import DeepSeekAdapter
Expand Down Expand Up @@ -82,6 +83,7 @@ def __init__(self):
def _initialize_registry(self):
"""Initialize adapter registry with metadata."""
adapter_classes = {
"azure_openai": AzureOpenAIAdapter,
"claude": ClaudeAdapter,
"deepseek": DeepSeekAdapter,
"gemini": GeminiAdapter,
Expand Down
2 changes: 2 additions & 0 deletions src/promptdrifter/adapters/__init__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
from .azure_openai import AzureOpenAIAdapter
from .base import Adapter
from .claude import ClaudeAdapter
from .deepseek import DeepSeekAdapter
Expand All @@ -12,6 +13,7 @@
__all__ = [
"Adapter",
"OpenAIAdapter",
"AzureOpenAIAdapter",
"OllamaAdapter",
"GeminiAdapter",
"QwenAdapter",
Expand Down
Loading
Loading