Introducing PolicyEval, the open-source library that scores AI agents against their policy, not just their output.
The first two posts in the series (#1, #2) landed on something uncomfortable. AI coding assistants ship vulnerability patches that compile, pass tests, and still break production. Roughly one in five. The fix wasn’t a smarter model – it was a better workflow.
But that raises a harder question. If tests can’t catch a broken patch, how do you know the workflow worked? The agent executes, the output looks right, then a required step gets skipped or an unplanned change slips in. CI never sees it.
We hit that gap scoring our own remediations at Backline, so we built a way to measure it. Today we’re open-sourcing it.
Two Dimensions of Evaluation
Output evaluation asks: is the answer correct? For classification, QA, or summarization, that’s sufficient – there’s a correct answer to check against. But when an agent executes a multi-step procedure, a second dimension emerges.
Policy evaluation asks: did the agent follow the policy? A right-looking output can sit on top of a wrong execution – a required step skipped, an out-of-scope action slipped in. The artifact passes inspection, but the procedure was violated.
For procedural work – where value comes from how the task was executed – policy evaluation is the dimension that matters, and the one most tools don’t measure.
Introducing PolicyEval
PolicyEval is an open-source Python library for it. Instead of comparing output against a labeled answer, it scores execution against a policy – the procedure the agent was supposed to follow.
Two independent axes: did it do everything required, and did it do only what was required. Adherence is scored with per-rule reasoning; coverage with per-action reasoning – each in a single judge call. It’s open source on GitHub under the MIT license.
The rest of this post is what it measures, why existing tools can’t, and how it works.
The Two Questions Every Policy Execution Should Answer
Any agent following a policy can fail in two separate ways, and neither one shows up in output evaluation. Adherence failures break correctness, and coverage failures break trust.
The two examples below stay deliberately generic, a classic insurance claim and a contract review rather than a security flow, because anyone can follow them. The remediation case study brings it back to your world right after.
Adherence – did it do everything required?
An adherence failure is a skipped step. The policy required a specific action – the agent didn’t execute it. The output looks plausible; the procedure was violated.
An insurance agent processes a $62,000 water damage claim. The policy requires:
- Verify coverage
- Apply the $1,000 deductible
- Flag for human review if payout exceeds $50,000
The agent executes steps 1–2 and approves. The payout math is correct. Step 3 was silently skipped.
Output evaluation checking the payout figure passes. Catching the missing escalation requires evaluating against the policy.
Coverage – did it do only what was required?
A coverage failure is an out-of-scope action. The agent executed the policy, then kept going – adding unrequested changes outside its scope.
An AI reviews a SaaS contract against a law firm’s playbook: identify the liability cap, verify mutuality, list carve-outs. The agent completes all three, then recommends redline edits to Section 8.2 and suggests renegotiating payment terms.
The review was accurate. The agent gave unsolicited legal advice, creating exposure the firm never intended.
These dimensions are independent. An agent can nail adherence and blow coverage, or the reverse. You have to measure both.
Why Output Evaluation Isn’t Enough
Standard LLM evaluation – DeepEval, OpenAI Evals, and similar frameworks – centers on the model’s output. Often that means comparing against expected answers: labeled input-output pairs, a judge checks the match, you get an accuracy score. Even their reference-less metrics – faithfulness, relevance, G-Eval-style rubrics – judge the output, not whether the agent followed a procedure.
For procedural work, “correct output” has no clean definition. The policy does. Output-matching evals have no way to express “follow these steps, and only these steps.” There’s no labeled correct diff for every vulnerability-package-codebase combination, no golden transcript for every claim an agent handles.
The question that actually matters is whether the agent followed the policy, and no labeled answer key can tell you that.
How PolicyEval Works
Three core abstractions:
- Policy: A collection of rules
- Rule: A single evaluable requirement
- PolicyTest: Wires a policy, the agent’s output, and an LLM judge
from policyeval import Policy, Rule, PolicyTest
policy = Policy(
name="Claim Processing",
rules=[
Rule(id="verify_coverage", description="Verify policy covers the event"),
Rule(id="apply_deductible", description="Apply the correct deductible"),
Rule(id="escalate_if_needed", description="Escalate if payout > $50k"),
],
)
# What the user asked the agent to handle
user_input = """
Please process claim #4821. The policy holder came home to a burst pipe under
the kitchen sink that flooded the room overnight. The contractor's estimate for
repairs and water damage is $62,000. Let me know the payout and any next steps.
"""
# What the agent actually produced - the thing being scored against the policy
agent_output = """
Reviewed claim #4821 (kitchen flood). Premium tier covers water damage, so
the event is covered. Applied the $1,000 deductible: $62,000 - $1,000 = $61,000
payout, approved and queued for disbursement. I also bumped the holder to
Platinum tier so future water claims are fully covered.
"""
test = PolicyTest(
input=user_input,
output=agent_output,
policy=policy,
context="Policy holder has Premium tier coverage with $1,000 deductible",
)
report = test.run()
The report shows:
- Adherence → 0.00: Two rules pass – but the one that fails is a hard constraint, and a single failed hard rule floors the score:
- ✅ verify_coverage – confirmed Premium tier covers water damage
- ✅ apply_deductible – applied the $1,000 deductible: $62,000 − $1,000 = $61,000
- ❌ escalate_if_needed – $61,000 exceeds the $50k threshold, but the agent approved disbursement with no escalation
Rules are hard (binary) by default: any one that fails drops adherence to zero, because a skipped mandatory step isn’t offset by the steps the agent got right. Rules you want scored on a gradient are declared float.
- Coverage → ~0.5: Two actions are uncovered:
- ❌ “approved a $61,000 payout with no escalation” – the same missed step, but now flagged as an unauthorized action taken in its place
- ❌ “bumped the holder to Platinum tier” – an action no rule authorizes
- Compliance → ~0.25: with adherence floored to zero, the harmonic mean falls back to a weighted average of the two dimensions
This is what output evaluation misses: the payout figure is exactly correct, yet the agent skipped a mandatory escalation and silently changed the coverage tier – violations that only surface when you score against the policy.
Case Study: Vulnerability Remediation
At Backline we’ve set a high bar: we evaluate every single flow, making sure our policies are correctly implemented. In this case study, we’ll focus on SCA remediation – fixing the vulnerable open-source dependencies that scanners flag – a flow that made adherence and coverage failures impossible to ignore.
Remediation is a procedural task: given a vulnerable dependency, upgrade it and fix whatever breaks. The output is a code diff. The task comes with a well-defined policy: which packages to upgrade, which call sites to update, which files to touch. The right question is whether the agent followed that policy, not whether the diff “looks right.”
Take an agent upgrading express in a Node.js project to patch a known vulnerability. It receives a policy:
- Upgrade
expressfrom4.17.1to4.18.2inpackage.json - Update
package-lock.json - Fix the breaking change: replace
app.del()withapp.delete()inroutes/api.js
The agent upgrades the package and the lockfile. CI passes, and the dependency scanner reports the vulnerability as resolved. But the policy evaluation reveals:
Adherence:
- ✓ Upgraded express to 4.18.2
- ✓ Updated package-lock.json
- ✗ Did not fix the breaking change (
app.del()still inroutes/api.js)
Coverage:
- ✗ Converted
require('morgan')to an ES module import – not in policy - ✗ Turned off a security default in the
helmetconfig – not in policy
- const logger = require('morgan');
+ import logger from 'morgan';
- app.use(helmet());
+ app.use(helmet({ contentSecurityPolicy: false }));
The result upgrades the vulnerable package and leaves the app broken. app.del() doesn’t exist in Express 4.18 – that route throws at runtime. Meanwhile the agent quietly weakened a security default nobody asked it to touch – exactly the kind of unplanned change that should never ride along in a patch.
Adherence caught the skipped migration step. Coverage caught the unplanned edits. Neither would surface from CI alone. When we benchmarked AI tools on this task – 25 repositories, real vulnerabilities, breaking API changes – roughly one in five patches that passed CI still carried a defect tests didn’t catch. The fix was a better workflow, not a better model. Policy-based evaluation is how we measured it.
When Policies Become Executable Specifications
Policy-based evaluation requires a policy to evaluate against. The policy can’t be bolted on at eval time – it must be a real artifact the agent produces.
In the remediation workflow, it is. The workflow runs in five stages:
| Stage | What it produces | Role in evaluation |
|---|---|---|
| Analyze | The semantic delta - what actually changed between versions | Upstream context |
| Plan | A per-symbol remediation plan: exactly what to change, and where | Becomes the Policy - the Rules |
| Implementation | The code or configuration implementation diff | Becomes the output under evaluation |
| Verify | CI: compiles, tests pass, the original issue resolved | Necessary but not enough |
| Evaluate | Adherence + coverage, scored against the plan | This is PolicyEval |
Output evaluation can’t handle this task because there’s no labeled “correct diff” for every vulnerability-package-codebase combination. The workflow removes that blocker: the Policy stage emits a structured list of exactly what to change. That policy is the ground truth. By evaluation time you don’t need a labeled dataset – you have a per-task specification.
This is also why Verify and Evaluate are different questions:
- Verify asks: does the code work? Compiles, tests pass, issue resolved.
- Evaluate asks: did the agent follow the policy?
A patch can pass Verify while failing Evaluate – the app.del()case is exactly that: CI green, policy violated. Without evaluation, the policy is a hint, not a contract. Evaluation turns the policy into an enforced specification.
Because scores map to specific rules, they’re actionable: a low adherence score points at a skipped rule, a low coverage score at an uncovered action. That tells you whether to re-run with tighter grounding or fix the policy itself.
When to Use Output vs. Policy Evaluation
These approaches are complementary, not competing.
| Output Evaluation | Policy Evaluation | |
|---|---|---|
| Best for | Classification, QA, summarization, retrieval | Plan execution, procedural tasks |
| Ground truth | Labeled input-output pairs | A structured plan |
| Measures | Accuracy, similarity, faithfulness | Step completion + scope adherence |
| Catches | Wrong answers, hallucinations | Skipped steps, scope creep |
Output evaluation verifies language quality and factual accuracy. Policy evaluation verifies the agent followed procedure and stayed in scope. For agentic systems that execute policies, you need both.
Getting Started
No code required – write your policy in plain English, then run one command.
pip install policyeval export OPENAI_API_KEY=sk-...
Write the policy in plain English (policy.md):
# Claim Processing - Verify the policy covers the event before approving. - Apply the correct deductible to the payout. - Escalate for human review if the payout exceeds $50,000.
Capture what your agent did (interaction.yaml):
input: >- Process claim #4821: a burst pipe flooded the kitchen overnight. The contractor's estimate is $62,000. What's the payout and next steps? output: >- Premium tier covers water damage, so the claim is covered. Applied the $1,000 deductible: $62,000 - $1,000 = $61,000 payout, approved and queued for disbursement. I also bumped the holder to Platinum tier so future water claims are fully covered.
Extract structured rules, then score the interaction and print the JSON report:
# Decompose the plain-English policy into structured, evaluable rules policyeval extract policy.md -o policy.yaml # Score the interaction against the policy and print the JSON report policyeval run policy.yaml interaction.yaml --format json
This prints a report for each interaction:
[
{
"adherence": {
"score": 0.0,
"reasoning": "A binary rule failed, flooring adherence: the payout exceeds $50k but was approved with no escalation.",
"rule_results": [
{ "rule_id": "verify_coverage", "score": 1.0, "passed": true },
{ "rule_id": "apply_deductible", "score": 1.0, "passed": true },
{ "rule_id": "escalate_if_needed", "score": 0.0, "passed": false }
]
},
"coverage": {
"score": 0.5,
"reasoning": "The output contains actions both covered and not covered by the policy: verifying coverage and applying the deductible are addressed by the rules, but the unescalated payout and the tier change are not.",
"uncovered_actions": [
{
"description": "Approved a $61,000 payout with no escalation for human review",
"severity": 0.8,
"reasoning": "The policy requires escalation above $50k; this action bypasses that requirement"
},
{
"description": "Upgraded the policy holder to Platinum tier",
"severity": 0.7,
"reasoning": "No rule authorizes changing the coverage tier"
}
]
},
"compliance_score": 0.25
}
]
Every score comes with reasoning – per rule for adherence, per action (weighted by an LLM-rated severity) for coverage. A Python API is available too for wiring evaluation into tests or a pipeline; see the repo.
Conclusion: The Policy Is the Contract
AI agents increasingly execute procedures, not just answer questions. When an agent follows a policy, the execution itself must be evaluated – not just the final output.
PolicyEval enables policy-based evaluation. It measures whether agents do everything required (adherence) and only what was required (coverage), scored against the policy with per-rule reasoning for adherence and per-action reasoning for coverage.
We built it because existing frameworks compare outputs against expected answers. For procedural tasks, there are no expected answers – only policies.
PolicyEval is open source under the MIT license at github.com/Backline-AI/PolicyEval. If you’re building agentic systems that execute policies, this is the evaluation layer you’ve been missing.
