Imagine an executive at a justice-technology program sitting in front of two final vendor proposals. The product is a GPS monitoring system for parolees released under supervision. The demos went well. Both vendors are credible. Both reference customers said positive things. The pricing is within a few percentage points of each other. On a normal vendor scorecard, this is a coin flip. The executive looks at the two stacks of paper and asks the question that the scorecard does not have a column for: what happens at 2am when the GPS tracking on a single high-risk subject drops out for forty-five minutes, and the subject was last seen heading toward a geo-fenced exclusion zone? Which of these two vendors do I trust to be on the bridge when that call comes in?
That question is the entire post. Standard vendor evaluation, the kind I have written about in the general framework, works well when the cost of being wrong is bounded. You pay too much, the integration takes longer than projected, the renewal conversation gets uncomfortable, you switch in three years and lose some sunk cost. Annoying, expensive, recoverable. Regulated and justice and healthcare contexts break that frame. The cost of being wrong is not bounded. A single failure mode, in a single moment, can produce harm that no amount of contract penalty or refund or apology can undo. The vendor selection process has to be reweighted to match.
flowchart TD
Start[Vendor evaluation begins]
Start --> Std{Is the downside<br/>recoverable?}
Std -->|Yes, money fixes it| Norm[Standard evaluation]
Std -->|No, harm cannot<br/>be unwound| Amp[Amplified evaluation]
Norm --> N1[Demo, references, roadmap]
N1 --> N2[Contract terms, SLAs]
N2 --> Sign1[Sign with normal diligence]
Amp --> A1[Named accountable engineer]
A1 --> A2[Written incident playbook]
A2 --> A3[Audit trail your regulator accepts]
A3 --> A4[Peer review from real operators]
A4 --> A5[Exit and transition clauses]
A5 --> Sign2[Sign with eyes wide open]
class Sign1 good
class Sign2 accent
class Amp warn
classDef good fill:#163a26,stroke:#44cc77,color:#d7ffe6;
classDef bad fill:#3a1620,stroke:#ff5555,color:#ffd9d9;
classDef warn fill:#3a2e16,stroke:#ffaa33,color:#ffe9c7;
classDef accent fill:#15233b,stroke:#4488ff,color:#dce9ff;
The asymmetry of irreversible failure is what changes the rules
In a typical enterprise software purchase, the buyer and the vendor are roughly aligned on what failure looks like. The vendor wants you to renew. You want the system to work. If the system underperforms, you negotiate, you escalate, you eventually switch. The relationship has a natural disciplining force: the vendor knows you can leave, and the cost of you leaving disciplines their behavior. Failure is a negotiating event.
That logic breaks in domains where the failure event produces consequences that cannot be negotiated away. A parolee in a place they were not supposed to be is not a negotiating event. A pharmacy system that routes a medication to the wrong patient is not a negotiating event. A claims processing system that exposes protected health information for thirty thousand patients is not a negotiating event. These are events that produce regulatory findings, civil liability, license consequences, and in some cases criminal exposure for the people running the program. The vendor whose system caused the failure can be fired, sued, and replaced, and the harm that already happened is still done. The cost of being wrong is not symmetric between the vendor and the buyer.
This asymmetry is what reshapes vendor selection. The standard process is built around the assumption that you have time and optionality to course-correct after the contract is signed. The irreversible-downside process has to be built around the assumption that the first major failure is the one you will be judged on, and the evaluation has to surface, before signing, the evidence that the vendor can be trusted in that specific moment.
What the G4S Justice Services context actually required
Years ago I worked as solutions architect on a GPS satellite-tracking system for monitoring parole offenders. The system tracked subjects against geo-fenced coordinate areas: inclusion zones the subject had to remain inside, exclusion zones the subject had to stay out of, schedule windows, escalation thresholds. The buyers were county and state justice agencies. The end users were parole officers, monitoring center staff, and the wider justice apparatus that depended on the data the system produced. The architecture sat on an SOA built in C# with MSMQ message queuing and SQL Server as the system of record, designed for the reality that messages would arrive late, out of order, and sometimes not at all from devices that lost signal in basements or rural dead zones.
The thing the experience clarified for me was what a buyer in that domain was actually evaluating when they evaluated a vendor. They were not evaluating the slickness of the UI. They were not evaluating the elegance of the data model. They were evaluating a chain of questions that started with: when a device drops signal at 2am for a high-risk subject, what happens? How fast does the monitoring center see the alert? Who calls who? What is the escalation path? What is on the screen of the parole officer who is going to be making the next decision? What does the post-incident audit trail look like, and is it going to hold up when the local newspaper FOIA requests the records?
That evaluation chain is not unique to justice technology. It is the same chain that a pharmacy director runs when evaluating a medication dispensing system, that a hospital CIO runs when evaluating a clinical decision support system, that a payments executive runs when evaluating a fraud detection vendor. The shape is consistent: identify the failure mode that produces irreversible harm, then evaluate the vendor against their performance in that specific moment.
Five questions that do not appear on a normal vendor scorecard
Standard vendor scorecards ask about features, references, pricing, roadmap, integration, support. Those questions are necessary and they are not sufficient. Five additional questions belong on the scorecard whenever the downside is irreversible, and the absence of any one of them should produce a pause before signing.
One. Who is the named accountable engineer at the vendor, and can I speak to them before I sign? Not the account executive, not the customer success manager, not the sales engineer. The engineer whose name is on the on-call rotation, whose phone rings when a P1 incident is declared, who has the authority to stop a release if your environment is impacted. If the vendor will not name that person, or cannot, the on-call function is structurally diffuse and you are not going to get the response you need when you need it. If the vendor will name that person and put them on a call with you during evaluation, that single conversation will tell you more about what the relationship is going to feel like than any reference call.
Two. What does the real-time incident response capability look like, in writing, for the specific failure type that produces irreversible harm in your domain? Generic SLAs do not answer this. A four-hour median response time is meaningless if the failure type that matters is the kind that produces harm in forty-five minutes. Ask the vendor to write down, in their own words, what happens at minute zero, minute five, minute fifteen, minute thirty, and minute sixty of an incident of the specific kind you are worried about. The artifact they produce, or fail to produce, is the answer.
Three. How deep and how immutable is the audit trail, and will it hold up to the regulator who is going to look at it? This is a question about your post-incident defensibility, not just your operational logging. Audit trails get scrutinized after a bad day, not before, and the depth and integrity of the trail determines whether the post-incident narrative is “we investigated promptly with full information” or “we cannot reconstruct what happened.” Ask whether the audit trail is append-only or mutable. Ask how long records are retained. Ask whether the vendor can produce, on demand, a complete record of every state change for a given subject across a given time window. If the answer is vague, the audit trail is vague.
Four. Where is the documented failure-mode playbook, and has it been exercised? Vendors who have operated in irreversible-downside domains for any meaningful period of time have a written playbook for the specific failure modes that matter. They have run tabletop exercises against those playbooks. They have updated the playbooks based on what the exercises revealed. Ask for the playbook. Ask when it was last exercised. Ask what changed as a result. A vendor that has not done this work is going to discover what their playbook should have been in the middle of the first incident.
Five. What does the contract say about transition and exit when a failure has happened? Standard contract review focuses on the orderly exit at the end of the term. The exit you need to read for is the involuntary exit triggered by a vendor failure serious enough that you cannot continue. Can you exit immediately on a defined trigger? Does the vendor commit to continuity of service during the transition? What does data extraction look like under emergency conditions rather than under a planned migration timeline? The contract is the only document that controls what happens in that moment, and a vendor whose contract does not contemplate the moment is a vendor whose engineering culture does not contemplate it either.
When this lens applies
The irreversible-downside lens is not a universal vendor evaluation framework. Most vendor decisions are bounded enough that the standard process is correct. The lens applies when a single failure can produce harm that money cannot reverse. That includes justice technology with monitoring, supervision, or detention components. Healthcare technology touching patient safety, medication routing, clinical decision support, or device control. Financial systems handling settlement, anti-fraud, or transaction integrity at scale. Identity and access systems serving regulated populations where a breach has cascading consequences. Public-safety infrastructure including dispatch, communications, and situational awareness. Any system whose failure produces a regulatory inquiry rather than a customer complaint.
The test I run with executives is the unwind question. If the vendor fails for one hour, one day, one week, can the harm be unwound by money? If the answer is yes, run the standard evaluation. If the answer is no, meaning the harm includes a person, a license, a regulatory finding, or a public-trust event, the standard evaluation is necessary but not sufficient, and the five questions above need to be on the scorecard with as much weight as anything else.
The general vendor evaluation framework still applies underneath. What changes is the threshold of evidence required for each question. A vendor that gives you confident, written, specific answers to the standard questions and to the five additional questions is a vendor worth signing with. A vendor that gives you confident answers only on the standard questions is a vendor whose strengths are aligned with normal buyers and whose weaknesses are going to show up the first time the failure mode you should have asked about actually happens.
If you are evaluating a vendor in a domain where the downside cannot be unwound by money, the fractional CTO engagement covers vendor evaluation as part of the broader technology decision portfolio, with particular attention to the questions a normal sales process is not designed to surface.
