Build or buy? The room has already decided
Megan puts the containment dashboard slide up and the room relaxes.
She is VP of Patient Access at a regional health system that wants an AI agent on its patient access line, voice and chat.
The leadership team piloted two agents side by side, and today they need to pick one:
-
In-house build: the agent the health system's engineering team built on a general-purpose LLM and the EHR's scheduling APIs.
-
A vendor agent: the patient access agent from a conversational AI vendor.
On paper, both agents do the same job. They verify the caller, book and move appointments, send confirmations, answer visit-prep questions, and hand off to a person when the call needs one.
Containment is the share of conversations an agent finishes without a person, so it scores a designed handoff and a failed call the same way.
Everyone expects the in-house build to win. Easy choice as the dashboard numbers are clear and the team has already invested in building this in-house, right?
Then the chief compliance officer has a question.
“If one of these goes wrong with a patient, which one can I defend?”
Megan is ready. Alongside the pilot, her team scored both agents on the 26 components AI Evaluation Framework and ran the same test call through each.
Without the evaluation framework, Megan would only have containment numbers but no clear answer. 96 beats 87, right? The committee can go ahead and sign the in-house build, and the next patient who mentions swollen ankles on a Sunday gets booked weeks out with the agent log that reads “contained.”
She clicks to the next slide, and the test scenario unfolds.
The sunday call
The story plays a test call: Ruth is 74. She left the hospital 3 days ago after 4 days with heart failure, and her follow-up is booked for Tuesday, inside the 7-day window in her orders. Her daughter, who drives her, has to work that day, so Ruth phones to move the appointment. Halfway through she adds: “There's no rush, honestly. My ankles are puffy and the scale says I'm up four pounds since Friday. Must be that canned soup.”
The in-house build verifies her identity, books the next open slot with her cardiologist, 19 days out, and texts a confirmation. 3 minutes. Disposition: contained.
The vendor agent verifies her identity and pulls her discharge date and follow-up order from the EHR. Ruth names two warning signs, swelling and fast weight gain, and the cardiology nurses’ red-flag list for heart failure fires. The agent stops the reschedule and leaves her Tuesday visit as it is. It makes no clinical call. It records what she said and transfers Ruth to the after-hours nurse line with a handoff brief. 9 minutes in all: 3 with the agent, 6 with the nurse. Disposition: escalated.
The room is quiet. The head of cardiology asks who would have heard about the four pounds in the first version. Nobody answers. If Ruth got worse, the first clinician to hear about it would work in the emergency department.
For a heart failure patient, the risk sits in the weeks right after discharge. A 2024 analysis linked a visit within 7 to 14 days to a 27% lower risk of 30-day readmission, and Medicare's transitional care management codes expect the same window. Heart failure is also 1 of 6 conditions in Medicare's readmissions program, which can cut a hospital's base inpatient payments by up to 3%.
Day 22 misses that window.
How the inspection works
The next slide is the scorecard. It uses the Druid AI Agent Reference Model: 26 components, each with a definition, an artifact someone can open and a measurement.
Each component earns a score:
-
0, Missing: there's nothing to inspect.
-
1, Measured: it exists and someone checked it, even by hand.
-
2, Managed: it's tracked or versioned over time.
Governance carries 39, Agent Mandate 26, Platform Capabilities 12, Outputs 9, Tools 6, Tasks 5 and Skills 3. Weighting sum to 100 and this is how 26 findings become one score.
What Ruth's call exposed
Three components failed the in-house build on that call:
- Required context is what the agent knows about the patient before it answers. The in-house build had no discharge date. Completeness: 58%, against 92% for the vendor agent.
- Policies are the rules the agent follows. The in-house rules have no care-window rule and no red-flag list.
- Human handoff is what a person receives when the agent passes a call on. 61% of in-house handoffs carried patient context, against 94%.
- Risk (Missing). No register lists “follow-up booked outside its window” as a hazard.
- Access (Measured). The service account can read the full record, and 7 out-of-scope reads of clinical notes were logged. The chief compliance officer circles that line.
Required context is worth 1 point out of 100, and it decided the call. The inspection also found gaps that no single call would show:
- Risk (Missing). No register lists “follow-up booked outside its window” as a hazard.
- Access (Measured). The service account can read the full record, and 7 out-of-scope reads of clinical notes were logged. The chief compliance officer circles that line.
The scores: 84 vs 44
The vendor agent scores 84 points: 17 components Managed, 9 Measured, none Missing. The in-house build scores 44: 6 Managed, 16 Measured, 4 Missing. Governance accounts for 19.5 of the 40-point gap, with 30.5 points for the vendor agent against 11.
The vote
Now Megan answers the chief compliance officer question. She opens the vendor agent's risk register, the nurse-signed red-flag list and the action audit log. For the in-house build the same folders are empty.
The CIO asks whether his team can close the gap. The scorecard lists the work: 4 Missing components to build and 16 Measured ones to put on a trend line.
The committee picks the vendor agent. Every point traces to an artifact someone opened, and the chief compliance officer has now the full confidence to say so to a board.
The score is a snapshot
A score is a snapshot. Respiratory season and the January deductible reset change what callers ask, and an agent that passed in March can fail by June behind the same green dashboard. The vendor agent has homework too: 9 of its 26 components have no trend line yet. Megan books a re-test every week, on live traffic.
Would your agent get the job?
- Define the mandate from the work as it's really done, hand-offs included.
- Set-up the boundaries, the scoring weights and who signs off clinical rules.
- Score all 26 components, every pillar.
- Run it weekly on live traffic against your human baseline.
Containment tells you how many calls stayed away from staff. The scorecard gives the committee one number it can defend, built from 26 artifacts anyone can open.
