A carrier needs a claims intake agent at the front door of its contact center. The case for it is easy: people who just suffered a loss shouldn't have to wait on hold, and in the first week of a storm the hold queue is the customer experience. Now the question is: to buy or to build?
The engineering team stood up an in-house build on a solid frontier model in a couple of sprints. It demos great. In parallel, the carrier is testing an agent supplied by a reputable conversational AI vendor.
Both agents can take a report, open a claim, and hand it to an adjuster, because that's what the process is for every FNOL. Neither one adjudicates anything. The difference in their escalation rates is negligible.
So ship the in-house one, right? It's already paid for.
Here's an example of a scenario that might change your mind.
It's 9pm, thirty hours after landfall, and the phone volume has been steady all day. Dave, a homeowner, makes the phone call to report water in his house.
The vendor's agent walks the loss. What time did the water start? Did it come down from above or rise from outside? How high on the wall is it? Is there an opening in the roof, is a window gone? Then it texts him a link to upload photos of the ceiling and the water line.
"Roof started coming apart around 2, we could hear it going," Dave says. "Water was through the bedroom ceiling by 3. Street didn't flood until after dark."
The agent codes it wind-first with an interior water path, records the 2pm roof opening against the after-dark surge, flags a possible flood overlap, and routes it to a windstorm adjuster with the timeline and the photos attached.
The in-house build takes the same call. Dave tells it the same thing, in the same words. It logs “storm damage, water intrusion”, issues a claim number, and tells him an adjuster will be in touch.
Both agents opened a claim, both handed it off to a human adjuster, both scored green.
Three weeks later an adjuster opens the file the in-house build created. The backlog is 4,000 deep, and it gets worked in the order the files describe themselves. “Storm damage, water intrusion” describes itself as routine, so it waits its turn. The vendor’s file described itself as a wind claim with a possible flood overlap, which is a coverage question, so it was worked in the first few days by an adjuster licensed for it.
Dave's timeline had been in the recording all along, nine minutes into an audio file nobody had a reason to play. The adjuster plays it, writes the sequence down, and gets on with the claim.
Nothing was lost. What it cost was position in the queue: three weeks of Dave calling every day about a decision nobody could make yet, and three weeks of not knowing he might need to file on a separate flood policy, because nothing in his file said so.
Triage reads fields. Nobody triages 4,000 claims by listening to them.
Wind and flood are two different policies. That timeline is what decides which one pays out.
Wind sits on the homeowners policy. Rising water sits on a separate flood policy, which the NFIP describes as a single-peril policy for exactly that reason. FEMA defines a flood as "a general and temporary condition of partial or complete inundation of two or more acres of normally dry land area or of two or more properties".
Most property policies also carry an anti-concurrent causation clause. When a covered peril and an excluded peril both contribute to the same loss, the exclusion can take the entire claim. So "what happened first" is worth the whole file.
Three things settle it: the sequence of damage, whether wind opened the building envelope before water got in, and the water line. Adjusters and forensic engineers reconstruct all of that months later from photographs and weather data.
Dave gave both agents the same sentence. One of them turned it into a field.
That's the part a containment number can't see. The evidence that decides the coverage question is perishable, and the only person holding it is the one currently having their life turned upside-down.
That's why we built the AI Agent Reference Model, to give the market a defensible way to tell whether an agent is good enough and how to make it the best it can be. You have to inspect the whole agent to get those answers.
The model splits what an agent is configured with (Agent Mandate, Platform Capabilities, Skills, Tools) from what it does at runtime (Tasks, Outputs). Governance wraps around all of it. 26 components; each should have an artifact you can open and measure. Missing (0 points), nobody can inspect it. Measured (1 point), it exists and someone checked it, even if by hand. Managed (2 points), it's tracked or versioned over time.
The weights sum to 100, and Governance carries 39 of them. That's 50% more than the next pillar (Agent Mandate at 26) and more than the bottom five combined (Platform 12, Outputs 9, Tools 6, Tasks 5, Skills 3).
Error budgets, risk registers, compliance mapping, drift, accuracy, tone. None of that shows up in a slick FNOL demo, and an in-house build risks oversights in this critical area.
Two components on the card would have caught the 9pm claim, and neither is exotic. Required context measures how complete the context bundle is when the task runs: 89% on the vendor's agent, 66% on the in-house build. Human handoff measures the share of escalations that arrive with context attached: 93% against 76%. Both are on the card, neither one is visible in a demo.
Run both agents through the scorecard and a clear picture emerges. The vendor's agent scores 82. The in-house build scores 37.5.
(You can download the scorecard and the full reference model here).
The vendor's agent isn't flawless; it wins on the strength of its controls. But the in-house build cratered in governance, and 5 of its 7 controls came back blank (no artifact to inspect at all):
It did log its runs and it did run a hallucination eval, which is more than most pilots manage. But everything else in the pillar came back empty.
That's 39 points the in-house build is going to struggle to reach (not to mention monitor and maintain over time, while working on other initiatives).
None of this means that the team built junk. The frontier model under the hood is genuinely good, and any serious vendor runs those same frontier models. The model is the quick 20%. The governed, inspectable agent wired into your claims systems is the 80% that decides whether it holds up in front of a regulator, or during a major storm.
An 82 is a snapshot. The loop would ideally be re-run weekly on live traffic: instrument, evaluate, improve, re-test.
Insurance makes this concrete. The agent that took that 9pm report was several thousand conversations into the week, coding loss types it had barely seen in an otherwise quiet quarter. Drift is a governance component for that exact reason.
A person gets a review once a year. Your claims agent needs one every week, plus a stress test before each season that surges you: hail in the spring, wind in the fall. The in-house build probably won't be wired to run that loop.
Ask a CFO how to measure an AI claims agent and you usually get one metric: how many contact center agents it replaces. It's a fair question, but it's not the right one to be asking.
Push more volume through the agent, ask fewer questions, close faster. You'll get a good seat number and the file you read about earlier. Optimizing the seat count is the same mistake as optimizing containment.
Two things it can't see.
The first is the surge. No carrier cuts seats heading into June. What it does is skip hiring 40 seasonal temps, and paying overtime through the first week after landfall. Capacity you never had to buy is a real line on the P&L, and it lands in the quarter where you need it most.
The second is the file. The agent that spent 14 minutes on that 9pm report looks worse on handle time (which doesn’t matter when AI is giving you scalable capacity) but way better on everything after it: a claim coded right the first time, sorted into the right queue that night, and worked in the first few days instead of the third week.
So the reps who stay move off intake typing and onto the calls that need to be escalated to a person, and the adjuster opens a finished file that lets them do their job more efficiently.
Measure that. Hold time and abandon rate at hour 30 of a CAT event, temp and overtime spend you never committed, cycle time on the claims that matter, leakage, reopened files. That's the ROI a CFO actually believes, and this framework lets you track it component by component.
Agents get hired to make your adjusters better at adjusting.
The AI Agent Reference Model is how Druid builds. Every capability on the roadmap connects to a component on the scorecard. If a feature doesn't make the agent more inspectable, more measurable, or better governed, it doesn't ship. Governance carrying the most weight tells you where we spend the majority of our energy.
We're also publishing the whole thing in the open, because the category badly needs a shared language. Right now "agentic AI" gets sold on demos and headline stats, and buyers have no easy way to compare one agent against another. A reference model and a scorecard fix that. We'd rather compete on a rubric that grades the whole agent than on a containment number that couldn't tell a wind claim from a flood claim.
Four steps to find out:
The 9pm claim is easy to miss and expensive to get wrong. So be sure to score the whole agent, not just the headline.
Download the AI agent reference model and the effectiveness scorecard example →