Druid sits with hundreds of customers running AI agents in production, and we hear the same question over and over: how do we actually tell if our agent is effective? There was no clean answer out there, so we built one: an AI agent reference model and a measurement framework we use to help customers evaluate their agents.
We presented both at AI Engineer World's Fair 2026 in San Francisco through a simple analogy: which of two interns deserves the return offer? The dashboard said one thing; a full inspection said the opposite. This post is a summary of that presentation: 6 pillars and 26 components that turn a full inspection into one defensible score. Would your AI agent get the job?
Picture a banking customer service team that hired two interns for the summer. Same bank, same job description. By August, Intern A contains 80% of conversations and escalates 20% to senior bankers. The dashboard flags it. Intern B contains 96% and escalates almost nothing. The dashboard is green.
Figure 1. The setup: two interns, one dashboard verdict each.
One return offer. Most rooms pick B. B is the wrong offer.
Here's what the dashboard missed. A customer authenticates on the mobile app at 7 PM and asks what it would take to pay off their mortgage early. Intern A answers fully: pulls the balance, runs the servicing math. Contained by every definition. But that question is a signal. The customer is rethinking their mortgage, and that's a refinance conversation with real revenue attached. So a bank-written rule fires: answer the customer, then book a banker and route the transcript, identity, and context. That designed handoff is part of A's 20%. Containment called it a miss.
Those numbers aren't hypothetical. The 80/20 containment split comes from Druid's 2026 AI Adoption Benchmark, drawn from 15 months of production usage, and it's the published financial services figure. The same benchmark shows why no single containment number can be the goal: higher education runs at 99.5%, HR & IT at 93%, healthcare at 87%. Banking designs in more handoffs on purpose.
A single runtime metric can't tell you whether an agent deserves the job. A full inspection can.
Which numbers matter, and which artifacts should you inspect? The reference model answers both. It separates what an agent is configured with from what it executes and produces at runtime. Configured is what the agent is made of before it runs: Agent Mandate, Platform Capabilities, Skills, and Tools. Runtime is what it does: Tasks and Outputs. Governance spans all six as a cross-cutting layer of controls.
Across the pillars sit 26 components, and every component has three parts: a definition, an artifact you can open, and a measurement that shows whether it's present, working, or improving. The discipline in one sentence: every component is an artifact that must be measured, and a result you can't measure is a claim you can't make.
Figure 2. The AI agent reference model: six pillars in two groups, with governance as a cross-cutting layer.
Governance defines the controls, evidence, and review mechanisms that keep agent behavior compliant, auditable, and within acceptable risk. It carries more scoring weight than any other pillar, and deliberately so.
| Component | Definition | Artifact | Measurement |
|---|---|---|---|
| Error tolerance | Measurable failure thresholds for the agent or task | Threshold config file | Actual error rate vs threshold |
| Risk | Downside exposure and required controls | Risk register | % of required controls in place |
| Compliance | Regulatory, privacy, and security requirements, mapped to standards such as HIPAA, SOC 2, and GDPR | Compliance matrix | % passing; last audit date |
| Observability | Traces, logs, monitoring | Trace/log store, dashboards | % of runs logged; retention days |
| Drift & deviation | Change in agent behavior or quality over time | Drift monitor | Drift score vs baseline |
| Accuracy & hallucination | Checks factual accuracy, source support, and hallucination risk | Accuracy / hallucination eval results | Hallucination rate |
| Tone & sentiment | Governs response tone, emotional register, and on-brand communication | LLM-judge eval output | Judge score (1–5) |
Agent Mandate defines why the agent exists, what domain it serves, and what boundaries it must respect. Five components map to paperwork nobody would skip for a human hire: the agent definition is the role, business impact is the targets, policies are the handbook, access is the badge, and cost is the budget.
| Component | Definition | Artifact | Measurement |
|---|---|---|---|
| Agent definition | Standing instruction that defines the agent's role, purpose, domain, and operating boundaries | Versioned system prompt file | Versioned (Y/N) |
| Business impact | Business outcomes the agent is expected to create, e.g. cases resolved, hours saved, cost reduced, revenue assisted, and CSAT improved | KPI scorecard | Actual vs target |
| Policies | Business rules and operating policies the agent must follow | Versioned policy rules file | Violations per N tasks |
| Access | Systems, records, actions, and tools the agent is allowed to use | Permission config (RBAC) | Out-of-scope access attempts |
| Cost | Expected cost and budget limits for agent operation | Cost dashboard | Budget vs actual |
The mortgage handoff from the opening started here, as a policy the bank wrote in advance. That's the point of the mandate: the agent's most valuable behavior was configured before the conversation ever happened.
These are the reusable platform abilities every agent draws on: memory that persists across sessions and channels, an orchestrator that routes work across workflows or delegates to specialized agents, and human handoff for escalation, approval, and exception handling.
| Component | Definition | Artifact | Measurement |
|---|---|---|---|
| Memory | Persisted store of session state, history, and prior results the agent reads within and across tasks | Memory store | Recall accuracy |
| Orchestrator | Routing across workflows or delegation to specialized agents or sub-agents | Routing definition | Routing success rate |
| Human handoff | Escalation, approval, exception handling, and human review | Escalation rule | Successful handoff rate |
Handoff is where the measurement philosophy matters most. What earns credit is context-rich escalation to the right team, with account context, intent, authentication status, and the reason for escalating carried across. Raw escalation avoidance earns nothing.
Skills are reusable packages of operational know-how: instruction files, templates, and scripts that help the agent perform a class of work consistently. Tools are the controlled interface between probabilistic frontier models and deterministic systems: retrieval over approved knowledge, and the APIs the agent can act in.
| Component | Definition | Artifact | Measurement |
|---|---|---|---|
| Instructions | Procedures the agent should follow | Instruction file (SKILL.md) | Versioned (Y/N) |
| Templates | Reusable structure or format | Template files | Present (Y/N) |
| Scripts | Executable code used by the skill to perform a deterministic procedure via self-contained logic or tool/API calls | Script code files | Execution success rate |
| Retrieval (RAG) | Knowledge base search, vector search, database query | Retrieval data store | Grounded answer rate |
| Tool interfaces | SoR APIs (CRM, EHR, ticketing), MCP-exposed tools | Tool registry | Tools registered (count) |
Both are necessary and both are easy to verify, which is why they carry the lowest weights in the framework. A comprehensive list of skills and integrations is now table stakes.
Tasks are the runtime unit of work: the input that came in, the context required to do it right, and success criteria that define done, plus the terminal state. Outputs are what task execution produces: the response, the system actions taken, and the success rate against those criteria. Outputs answer one question: did the work get done, and did it land?
| Component | Definition | Artifact | Measurement |
|---|---|---|---|
| Input | User request, messages, event, ticket, alert, etc. | Input record | Present (Y/N) |
| Required context | Information needed to complete the task correctly | Context bundle | Context completeness % |
| Success criteria | Conditions that define correct completion of a single task instance, plus the terminal end state (complete, blocked, escalated, timed out, max attempts reached) | Acceptance check (eval) | Per-task pass (Y/N) |
| Response | Answer, summary | Response object | Produced (Y/N) |
| System actions | Ticket updated, refund issued, appointment booked, CRM field updated, email sent | Action audit log | Action success rate |
| Success rate | Share of task instances that met their success criteria | Task-outcome log | Success rate |
A model tells you what to inspect. The measurement framework tells you how to grade what you find. Every component earns one of three grades: Missing (0), nobody can inspect it; Measured (1), the artifact exists and has been checked, even by hand; Managed (2), the measurement is tracked or versioned over time.
Each component's grade is multiplied by a weight, and the weights are deliberately uneven. They sum to 100, so every agent gets one comparable score: Governance carries 39 points, Agent Mandate 26, Platform Capabilities 12, Outputs 9, Tools 6, Tasks 5, and Skills 3.
Here's the method applied to the two interns, now scored as agents. The comparison is a worked example we use with customers; the values illustrate the scoring method against banking-grade expectations rather than reporting a specific production deployment. Agent A earns the offer, 83 of 100 against 45, and the pillar-by-pillar breakdown below shows why: B's gaps are Missing artifacts on the highest-weighted controls.
Figure 3. The worked comparison, pillar by pillar. Agent B wins Skills alone; the heavy pillars decide the offer.
Keep two numbers straight: 80% was Agent A's containment rate, 83 is its scorecard. And the score has a shelf life, so re-run the loop weekly on live traffic: instrument, evaluate, improve, re-test. An 83 in March can be 60 by June behind a flat green dashboard. A person gets an annual review; your agent gets a weekly one.
Ask a CFO to measure an AI agent today and there's usually one column on the sheet: which human does it replace. That's the fear behind every agent business case, and it persists because most teams have no defined way to measure anything else.
The reference model and measurement framework give you that defined way. Every part of the agent is defined, inspectable, and measured, so the question shifts from headcount to a measurable one: what did the agent take off your experts' plates, and what did that freed time buy? In the story that opened this post, the agent handled the balance math at 7 PM so the banker could spend the next morning with the customer rethinking their mortgage. The agent made the banker better at banking.
That's the ROI these frameworks let you measure, methodically and component by component: did the agent help your humans do more valuable work. Agents are hired to make employees better.
The framework comes down to four moves. Define the mandate from the work as it's actually done. Contract the boundaries, the weights, and the error budget. Score the whole agent, every pillar, and resist grading the headline metric alone. Then run the loop weekly against the human baseline.
End of summer's coming. Would yours get the offer?
Download the AI Engineer World's Fair presentation, the full reference model, and the effectiveness scorecard example here.