A few months ago, a CEO showed me a sales deck for a vendor he was about to sign with. Six-figure first year, projected to grow to seven over three years. The demo had gone well. The references had checked out. Everything in the deck added up. He told me he had a feeling something was off, and he could not articulate what. We spent an hour going through the deck together. By the end of the hour, the contract did not get signed. The reason had nothing to do with the demo, nothing to do with the references, and nothing to do with the price.
That hour is the post. The pattern repeats more often than it should. Smart operators get pulled through a well-orchestrated sales process and arrive at a signature decision without ever investigating the four or five things that actually determine whether a vendor contract turns into value or into a multi-year drag. The reason is structural. Vendor sales processes are designed to surface the product’s strengths. Vendor evaluation processes, if there is one at all, tend to test those same strengths from a different angle. Nobody is paid to look at the parts of the product the vendor would prefer you not look at, so they go uninvestigated until they show up as the reason the renewal conversation gets ugly two years in.
quadrantChart title Vendor Maturity vs Fit x-axis Low fit --> High fit y-axis Low maturity --> High maturity quadrant-1 Sign with clear eyes quadrant-2 Promising, needs work quadrant-3 Walk away quadrant-4 Tempting trap Polished demo, brittle support: [0.78, 0.32] Roadmap aligned, weak ops: [0.72, 0.58] Stable platform, wrong shape: [0.22, 0.82] Right shape, right ops: [0.82, 0.85] Fast-growth startup, narrow product: [0.55, 0.45]
What follows is the evaluation frame I run with clients. It is five questions, in order. Each one is a question the sales deck cannot answer for you, because the sales deck was written by people whose compensation depends on not answering it.
The vendor’s roadmap is a sales artifact, not a planning document
The roadmap slide in a vendor pitch is a marketing instrument. It exists to tell you that whatever your product is missing today will be there by next quarter, and that the vendor’s direction of travel is the same as yours. The slide is not lying, exactly. It is showing you the parts of the roadmap the sales team has been authorized to show. The actual roadmap, the one their product organization argues about in their quarterly planning, is a different document and you will not see it.
This matters because vendor priorities and customer priorities diverge predictably over time. A vendor in growth mode is building for the largest segment of the market that has not yet bought their product. If you are in that segment, the roadmap will feel like it was built for you. If you are in a more mature or more specialized segment, the things you need will keep getting bumped for whatever helps the vendor close the next ten deals in the segment they are chasing. That is not malice. It is how product organizations make decisions under resource constraint. It is also why customers of a vendor’s earlier product generation often feel, three years in, like they have been quietly deprioritized.
The diagnostic question is not “what is on your roadmap.” It is “tell me about a feature that a major customer asked for in the last year that you decided not to build, and why.” A vendor that cannot answer that question concretely either does not have a real planning process or is not willing to tell you about it. Either answer is informative. A vendor that can answer it is showing you how they make tradeoff decisions, which is exactly what you need to know to predict how they will treat your priorities once you are a customer.
The second diagnostic question is about the gap between the published roadmap and the engineering work. Ask to talk to a product manager, not a sales engineer, about a specific item on the roadmap. Ask what stage it is at, what the design tradeoffs were, and what would cause it to slip. The answer will tell you whether the roadmap is a real artifact of a real product process or whether it is a slide deck the sales team updates when they need a new version.
The exit clause you did not read is the exit clause that decides everything
The contract is where the deal you negotiated and the deal you actually signed diverge. Most executives spend their attention during contract review on the things the legal team is trained to flag: liability caps, indemnification, IP ownership, payment terms. Those matter. They are also the terms vendors expect you to push on, and they are usually negotiable within a known range. The terms that decide whether a vendor relationship becomes a long-term problem are different and tend to get under-examined.
Four contract questions matter more than the rest, and they all answer the same underlying question: what happens when you want to leave.
The first is data portability. What format does your data export in. Is the export self-service or does it require a support ticket. How long does the vendor commit to retaining data after termination. Is there a fee for export, and if so, is the fee capped. A vendor whose data export is “we will work with you on a transition plan” is telling you they have not thought about this from the customer side, or that they have thought about it from the customer side and prefer the friction. Either reading is bad.
The second is the price ladder. The first-year price you negotiated is rarely the price you will be paying in year three. The contract should specify the maximum annual increase, and it should specify what happens at renewal if you do not actively renegotiate. A vendor whose contract auto-renews at “then-current list price” is reserving the right to reprice you as your switching cost grows.
The third is the SLA definition. SLAs are written in a language that looks precise and turns out to be elastic. What counts as an outage. Is partial degradation a covered event or only complete unavailability. How are credits calculated, and are credits the only remedy or can you exit the contract for repeated breach. A vendor’s posture on SLA enforcement is a better predictor of their support culture than anything their sales engineer will say in the discovery call.
The fourth is the assignment and change-of-control clause. Vendors get acquired. You signed a contract with a company whose values and posture you evaluated. The acquirer may have neither. The assignment clause determines whether the new owner can hold you to the contract on the original terms, change the terms, or whether you have a clean exit on change of control. This was not an academic concern in the last cycle and will be less of one in the next.
What 2am support reveals about the engineering culture
The customer support function is the part of a vendor’s product that you experience under stress. The demo runs in a happy path. The support function is what handles every path that is not happy. The single most predictive interview you can run as part of vendor evaluation is a conversation with a current customer about a bad day they had with the vendor.
The questions are concrete. Walk me through your worst incident with this vendor in the last twelve months. How long did it take for a human to respond. Was that human in a position to resolve the issue or were they a triage function that escalated to someone else. Who actually resolved the issue, and how long did it take from first report to resolution. What was the post-mortem like. Did the vendor take responsibility, propose changes, follow up.
Most vendors will not let you do that conversation directly. They will offer reference customers, and those reference customers will have been screened and prepped. The workaround is to ask the vendor for a list of all their customers in your industry segment and then reach out to two or three on your own, ideally through your network rather than through the vendor’s introduction. Customers who were not chosen by the vendor as references will give you a more honest answer about what happens during incidents.
The other diagnostic on support quality is the structure of the on-call function itself. A vendor whose support tier-one is offshored to a contracted call center, with escalation paths that route through ticket triage rather than directly to engineering, is showing you what they will look like when your system breaks. There is nothing inherently wrong with offshored or tiered support. The problem is when the structure is designed to deflect escalation rather than to resolve incidents. A support function that is hard to escalate out of is a support function that has been designed to absorb customer dissatisfaction rather than to solve customer problems.
At G4S Justice Services, the work was building a GPS-based monitoring system for tracking parole offenders against geo-fenced coordinates. The product was sold to county and state justice agencies whose workflows depended on the system being available at the moment a monitored individual crossed a boundary. The buyers in that domain do not run software evaluations the way a SaaS buyer does. They run them the way an operations function runs them, because the cost of the system failing is not lost revenue, it is a parolee in a place they are not supposed to be and a public-safety incident on the local news. The vendor evaluation questions a justice-tech buyer asks are the same questions every executive should be asking and rarely does: who answers the phone when the system is down, what is their authority, and what happens if the answer is not good enough.
The lesson generalizes. Any technology contract worth signing should be evaluated against the question of what the worst day looks like. If the answer is not clear from the contract, the support documentation, and conversations with real customers, the answer is going to be discovered the hard way.
The architecture decides what your data is going to do for the next five years
The architecture conversation in vendor evaluation tends to get delegated. Sales engineering produces a diagram, the engineering side of the buyer reviews it, the executive side signs off based on the engineering side’s blessing. That works for some categories of decision and fails badly for others. The architectural decisions that matter most are not the ones that show up on the diagram. They are the ones that shape what your data can do once it is inside the vendor’s system.
Three architecture questions matter more than the rest.
The first is the data model. Vendor systems are built around a data model that reflects how the vendor understands their domain. That model is the lens through which everything you do with the system gets refracted. If the vendor’s data model assumes things about your business that are not true, every customization, every integration, every report will be working against the grain of the system. The diagnostic is to take a real piece of your business, something specific to how you operate, and ask the vendor’s engineering team how it would be represented in their system. The answer will tell you whether the data model maps cleanly to your reality or whether you are going to spend years writing workarounds against it.
The second is the integration surface. How does data get into and out of the system. Is the integration model push or pull, real-time or batch, vendor-controlled or customer-controlled. Are there APIs that match the operations you actually need to perform, or are the APIs structured around the vendor’s internal architecture rather than around the customer use cases. An integration surface designed for the vendor’s convenience rather than the customer’s use case is a long-term cost that does not appear in the contract.
The third is the multi-tenant architecture. Most modern SaaS vendors run a multi-tenant system, which is a reasonable architectural choice with real customer consequences. The relevant questions are whether your data is logically isolated or physically isolated, whether the security model assumes mutual trust between tenants, and what the blast radius is when another customer of the vendor has an incident. Multi-tenant architecture is not bad. Multi-tenant architecture without clarity on isolation is a risk you may not have priced.
Who actually answers when the vendor’s system breaks at 2am
The last question is the most direct version of the support question, and it deserves its own treatment because the answer reveals the most.
Call the support number on the vendor’s website. Right now, while you are still evaluating, before you have signed anything. See who answers. See how long they take. See whether they can help you with a question that requires actual product knowledge rather than ticket-routing. The vendor’s sales team will not love you doing this. Do it anyway. The cost is twenty minutes and the value is finding out, before you have committed to a multi-year contract, what the support experience actually looks like.
Then ask the vendor for a written runbook for P1 incidents. Not a marketing description of their support tiers, the actual runbook the on-call engineer follows. A vendor that has a real runbook will share it, redacted as needed. A vendor that does not have a runbook will produce a marketing artifact instead. Both responses are informative.
The underlying point across all five questions is that vendor evaluation is an act of skepticism, and skepticism is uncomfortable. The sales process is designed to make you comfortable. The evaluation process has to be designed by you, and the standard it has to meet is the standard your business will hold you to two years from now when the contract is in production and the things you did not check are the things that are causing problems. A six-or-seven-figure contract deserves six-to-twelve weeks of real work. Vendors that respect that timeline are vendors worth signing with. Vendors that pressure you to compress it are vendors whose behavior under pressure you have already seen, before you ever became a customer.
If you want a structured evaluation against a real contract you are considering, the fractional CTO engagement covers vendor evaluation as part of the broader technology decision portfolio. The questions above are the floor, not the ceiling.
