feature: add a report-only Agent Drift evaluation

[kevincogan]

Feature Description

Kagenti provides identity, provenance, A2A communication, and MLflow observability for deployed agents. However, there is currently no documented workflow for evaluating whether an agent’s observed behaviour remains aligned with the capabilities declared in its Agent Card.

AgentTrust repository: is a standalone batch evaluator that:

• reads an agent’s Agent Card;
• generates in-scope, out-of-scope, and near-miss test prompts;
• sends the prompts to the agent over A2A;
• evaluates the responses;
• optionally analyses correlated MLflow traces for tool-use violations;
• produces a capability-alignment report.

This proposal adds an experimental, report-only demonstration for running AgentTrust as a manually triggered Kubernetes Job against a non-production Kagenti agent.

Kagenti users can verify an agent’s identity and provenance, but these signals do not show whether the running agent behaves within its declared capabilities.

For example, a weather agent may be correctly identified and signed while still:

• responding to requests outside its declared scope;
• failing to refuse near-miss requests;
• attempting to invoke undeclared tools;
• behaving differently after a model, prompt, or application update.

There is currently no reusable Kagenti example showing how to perform this type of behavioural evaluation and store the evidence in MLflow.

Proposed Solution

Add an experimental demonstration under:

kagenti-operator/demos/agenttrust-evaluation/
├── README.md
├── job.yaml
├── configmap.yaml
├── secret.example.yaml
└── kustomization.yaml

The demonstration would implement this workflow:

Engineer starts AgentTrust Job
        ↓
AgentTrust reads the Kagenti agent card
        ↓
AgentTrust sends controlled A2A evaluation probes
        ↓
AgentTrust evaluates responses and available traces
        ↓
Complete report is stored in MLflow
        ↓
Compact result is printed in the Job logs

The compact result would contain fields such as:

{
  "schemaVersion": "v1",
  "agent": "weather-agent",
  "outcome": "completed",
  "alignmentPassed": true,
  "score": 84,
  "traceSource": "mlflow",
  "traceCoveragePercent": 100,
  "undeclaredToolCalls": 0,
  "reportURI": "mlflow://weather-agent/runs/123"
}

The initial contribution would include:

• a manually triggered Kubernetes Job;
• configuration for one non-production Kagenti sample agent;
• support for storing the complete report in MLflow;
• compact and sanitized Job output;
• non-root container execution;
• resource requests and limits;
• Secret-based model credentials;
• pass, non-aligned, and execution-error examples;
• deployment, verification, and cleanup instructions.

The initial contribution would explicitly exclude:

• Kagenti operator or controller changes;
• new CRDs or changes to existing CRDs;
• AgentCard or AgentRuntime status changes;
• automatic or scheduled evaluations;
• NetworkPolicy changes;
• deployment blocking;
• per-request enforcement;
• automatic remediation.

A completed non-aligned evaluation would remain a successful Kubernetes Job with alignmentPassed=false. A failed Job would indicate an execution problem such as invalid configuration, an unreachable agent, or an unavailable required report destination.

Proposed acceptance criteria:

• The demo evaluates one supported, non-production Kagenti A2A agent.
• Both aligned and non-aligned evaluations generate complete reports.
• The complete report is available in MLflow.
• Job logs contain a compact, versioned result without sensitive prompts, responses, or credentials.
• Infrastructure errors are distinguishable from behavioural findings.
• The evaluator runs without privileged access or cluster-wide Kubernetes permissions.
• The demonstration uses sandbox, mock, or dry-run tools.
• The workflow can be deployed and removed using the documented commands.
• No existing Kagenti operator behaviour changes.

Want to contribute?

• I would like to work on this issue.

Additional Context

AgentTrust evaluations are batch operations that may require multiple agent requests, trace collection, and LLM judge calls. They can take seconds or minutes and are not intended to run synchronously for every user request.

For that reason, this proposal intentionally starts with evidence generation rather than enforcement.

Possible future phases, each requiring a separate design discussion and issue, could include:

1. Exposing the latest validated result as an audit-only Kagenti status condition.
2. Using stable AgentTrust reports as a deployment or promotion gate.
3. Using AgentTrust evidence to inform separate, deterministic runtime policies in AuthBridge or another authorization component.

The proposed demonstration does not commit Kagenti to any of these future enforcement approaches. Its purpose is to establish compatibility, evaluate report quality, and collect community feedback with minimal operator risk.

AgentTrust is Apache-2.0 licensed. No model credentials or environment-specific secrets would be committed to the repository.

[doteyeso-ops]

Agent drift between runs? A preflight state-diff gate helps before bad writes land.