The State of the Stack: How to Run Your Software Evaluations Like Diligence

I heard the term “demo fatigue” for the first time the other day when, coincidentally, I was leading a prospect through a demo. I asked the investor how many demos he had sat through this year. He counted out loud and landed on 11. Nine of them, he said, had shown him roughly the same thing.

He was not complaining. He was describing a real procurement problem that people both inside and outside PE are experiencing. The market has become very good at looking identical for the first twenty minutes. I spent years on the buy side before moving to the vendor side, and this article is the guide I would want if I were still sitting where he is.

Why all software looks and sounds the same

Over the last 18 months, nearly every platform selling into private markets or fintech added the same three capabilities: natural language search across firm data, automated summarization of CIMs and diligence materials, and some sort of MCP. Some of those builds are genuinely workflow-native. Some are a thin wrapper over a model API. In a demo, though, they are indistinguishable, because the demo is the one environment a vendor fully controls.

That means the things that determine whether a platform or a piece of software works at your firm are the things nobody puts on a slide: How long implementation really takes; whether the data model bends to your process or your process bends to it; or what the system does when it is wrong. Buyers have figured this out, which is why Gartner’s sales research found that 61% of B2B buyers would prefer a rep-free buying experience, and 69% report inconsistencies between what a vendor’s website says and what its sellers say. 

Like it or not, the age of autonomous enterprise buying is still a ways away. 

5 Questions that separate built from bolted-on

You do not need a technical background to be able to distinguish features that are natively built in from those that were tacked on. Ask these same five questions of every vendor and compare the answers side by side.

1. “What exactly does the model see?” 

The vendor should be able to name the objects and the permissions. A vague answer, usually some version of “it is trained on your data,” tends to mean either almost nothing is retrieved, or everything is. The second one is a confidentiality problem.

2. “Can you show me where that number came from?” 

Click the output and see if you can get back to the source record. If a summary cannot be traced, nobody senior will put it in an IC memo, and they are right not to. Provenance is the difference between a tool people use and a tool people abandon.

3. “What happens in month seven when we redefine our workflow?” 

You need to know what you’re able to configure yourself, and what will require a professional services ticket with a queue. That answer predicts the next three years more reliably than any feature comparison.

4. “Who did you build this with, and can I call them?” 

A feature built with design partners who have read a CIM behaves differently from one built to win a competitive deal. Ask for a reference at your size and strategy, then ask that reference what broke.

5. “What does this do badly?” 

A vendor who can name a real limitation has hit that limitation in production. A vendor with no failures has not shipped enough features to find the problems. Which means you will find the bugs.

Four tells that a platform is not there yet

These four tells will alert you that a platform isn't ready for prime time...or your firm. None of these is a disqualifier alone, but if you spot two or more, you are looking at a roadmap wearing a product’s clothes:

1. The demo is entirely recorded. If a vendor cannot stand up a live demo and answer questions in real time, it’s a red flag.
2. Present-tense questions get roadmap answers. You asked what it does and were told what it will do. That conflict recurs in implementation and creates an expensive complication and a serious case of buyer’s remorse.
3. The AI lives in a separate tab from the work. Output that has to be copied somewhere to be useful is a second system to maintain, not a faster workflow.
4. Time to value is quoted as “typically.” Don’t ignore this vague answer. Ask for a named customer at your size and their real date from contract to first use.

Run the process like a deal

With all that said, my advice is actually simple, and something you’re used to doing already: Apply the same discipline to software evaluations that you apply to deal diligence. 

Gartner surveyed 1,120 respondents at manager level or above involved in enterprise technology evaluations. Fifty-six percent reported high regret over their largest tech-related purchase in the prior two years, and high-regret organizations took 7-10 months longer to complete the implementation. Regret peaked among buyers who had not yet started implementation, meaning the frustration was with the buying process itself.

Write your buying thesis before the first demo. List 3-5 workflows that must work on day 30, in your own words, agreed upon by the people who will use them. If you don’t bring your own criteria, the vendor supplies them. (And the vendor’s criteria will be the ones they win on.)

Cap the field at three. More options do not produce a better decision, only a longer one. Narrow the field on short calls and public material, not in ninety-minute demos.

Name an owner and set a kill date. One person at the firm should own the decision, and the process should have an expiration. Evaluations without an end date do not end. They trickle on.

Make one reverse reference call. Ask a reference who else they evaluated and why they passed. Fifteen minutes there beats the eleventh demo call.

Demo fatigue is real, and it is not a sign that you are bad at evaluation. It is a sign that vendors have stopped differentiating. Decide what you need before anyone shows you anything, ask the same hard questions of every vendor, and give yourself permission to stop looking.

‍

‍