How to Evaluate AI Vendors When You Have No Digital Strategy Only a Broken Process

Test vendors by having them rebuild your broken process on your actual operational data.

· 9 min read
Cover illustration for “How to Evaluate AI Vendors When You Have No Digital Strategy Only a Broken Process”
AI Vendor Eval · October 28, 2026 · 9 min read · 1,946 words

AI operations transformation in asset-heavy industries can begin with one broken process, without a platform strategy or an enterprise software commitment. The process itself supplies the evaluation framework, including its failure mode, data, decisions, and operating constraints. Evaluate vendors on whether they can rebuild that workflow and prove the result on the buyer’s operational data before commitment.

Why starting with a broken process is more useful than starting with a strategy

An operations leader recognizes the failure before having a framework for fixing it. A handoff keeps breaking, exceptions need cleanup by hand, or a schedule does not match operating conditions. People closest to the work already absorb the cost. The absence of a digital roadmap does not prevent that leader from identifying what goes wrong, where it happens, and what a working process would need to produce.

Those observations are the beginnings of an evaluation brief. A broken process has a measurable failure mode, identifiable decision points, known data sources, and people who understand its daily consequences. A strategy document may not provide that specificity. Conventional advice to establish a strategy first assumes a clarity of direction that organizations under operational pressure may not achieve quickly enough to act.

Strategy work risks replacing a concrete operational question with a platform selection exercise. Once the discussion turns to what a platform can do, vendors can seem like a good fit without fixing the failure that led to the search. Starting with the process keeps what is being evaluated small enough to look at closely and specific enough to test.

How the conventional vendor evaluation process produces the wrong answer

In standard procurement, vendors are asked to explain how they work, show their products, and provide reference cases. These materials describe capability under selected conditions. They do not establish whether a system will work with the buyer’s taxonomy, legacy naming conventions, inconsistent documents, and approval bottlenecks. The evidence answers a different question from the one operations needs resolved.

tracebloc’s 2026 vendor evaluation analysis identifies RFPs, methodology documents, and loosely comparable reference cases as proxies for testing on proprietary data. The substitution has a structural cause. Buyers cannot simply hand sensitive operational records to every vendor they might hire, but a useful evaluation needs to show how a system performs on data beyond what the vendor chose for its demonstration. Months of legal review, NDA negotiation, and security assessment can leave only one vendor at the table before testing begins.

The same analysis compares vendors for fraud-detection to show how the choice can go wrong. Vendor C looked attractive on paper, with the highest recall and lowest cost, but failed two production hard constraints. Vendor B looked overpriced in proposals yet delivered the only deployable model.

A polished demo cannot resolve that difference. Its clean inputs and narrow happy path hide the operating model needed to deal with messy records, exceptions, and approvals from people. When those conditions arrive in production, promised automation can shrink to partial acceleration of one subtask when the inputs happen to be clean.

What automating a broken process actually produces

Human friction often slows a process, but it can also catch errors or contain their effects. Adding an agent without redesigning the surrounding decisions removes some of that friction while leaving the causes of failure intact. Errors then propagate faster through the same faulty sequence.

Fiddler.ai’s April 29, 2026 report estimates that 88% of enterprise agents that work in controlled demos fail when deployed to real workflows. It identifies compounding errors, tool breakdowns, and hallucinations as primary mechanisms, with wasted compute, manual cleanup costs, and eroded organizational trust among the consequences. Faster reports or summaries do not establish that the workflow’s decisions and handoffs have improved.

Gartner’s May 2026 analysis points to another way things go wrong: binary agent governance. Treating agents as either locked down or fully trusted over-restricts simple agents while under-restricting more autonomous ones. The former slows delivery and drives shadow development. The latter increases operational, security, and compliance risk. Governance therefore needs to reflect the decisions an agent can make, rather than a single permission setting for all automation.

Most platform SLAs measure uptime of the wrapper. A model can degrade, hallucinate, or become unavailable while the platform remains technically up. The SLA can hold while the workflow fails.

How to map the broken process before any vendor enters the room

The practical first step is a process audit. Start where the failure becomes visible and record what breaks, at which step, under which conditions, and who absorbs the cost. Do not begin with desired AI features. A vendor intake questionnaire cannot do the work of this audit because the brief must describe what is going wrong for the buyer before a vendor proposes a way to fix it.

Walk through every decision in the workflow as it works today. Find where a person uses judgment, where work passes between systems or teams, and where exceptions take it off the normal path. Record who handles those exceptions today. These points show where an agent could improve coordination and where it could remove a safeguard that the process still needs.

Then follow the data behind those decisions. Find out where records live, whether teams or legacy systems use different naming conventions, and whether gaps in past records would affect calibration. Identify who owns access and how permissions work. A process can look well documented while its usable data remains fragmented across inconsistent records and separate access arrangements.

Define success in the terms the process already generates. Record its current output, how that output is measured, and what would count as a working replacement. Dan Cumberland Labs’ 2026 checklist says firms often skip this question of what success would look like for their business and calls solving the wrong problem the most expensive mistake when choosing a vendor. Select one process to rebuild first, so the proposed improvement remains falsifiable. A tightly defined workflow gives both sides a result they can prove or fail to prove.

Where specific workflows break in construction, manufacturing, and logistics — and what that means for evaluation

Construction scheduling exposes the gap between average performance and expensive exceptions. A model can confidently optimize a schedule while missing the site condition known to a veteran superintendent. Physical availability introduces another limit on recovery. Zacua Ventures’ 2026 industry report says the US has too few power transformers and distribution transformers, shortages that schedule optimization cannot fix. Evaluation must therefore test whether a vendor recognizes unavailable recovery options and preserves the field judgment needed for site-specific exceptions.

The construction data stack adds a prerequisite. AI tools consume what the system of record produces, so inconsistent daily logs weaken the layers above them. Computer vision or schedule optimization may detect a problem without a workflow capable of resolving it. The evaluation must cover both the record feeding the tool and the operational response to its output.

Manufacturing demands fit to the particular facility. IDC’s 2026 manufacturing forecast projects that over 40% of manufacturers with a production scheduling system in place will upgrade it with AI-driven capabilities to start enabling autonomous processes. A manufacturing module must be tested against a plant’s equipment, workforce, and operating constraints. A generic platform and a configurable system for one facility are not equivalent simply because their feature lists overlap.

Logistics presents a different diagnostic question. Uber Freight CTO Val Marchevsky identifies process redesign, rather than layering AI onto broken processes, as the larger opportunity to eliminate waste and streamline execution. To show it can do that work, a vendor needs to look at how routing, dispatch, or carrier management works today. Evaluation should distinguish experience redesigning those workflows from experience automating their existing steps.

The company redesigned planning positions around decisions requiring human intervention, with less emphasis on data manipulation. For evaluation, the relevant evidence is whether a vendor can identify which decisions need human judgment and shape the engagement around that division of work.

The five questions that turn a broken process into a vendor evaluation brief

The first question asks whether the vendor can show performance on the buyer’s data. A vendor can test securely inside the buyer’s infrastructure, use synthetic data made from real operational records, or use federated evaluation without raw data transfer. Benchmarks and other clients’ results do not answer the question. Inability to propose a secure mechanism is a disqualifying condition.

The second question asks what the rebuilt process looks like and which decisions remain with people. The answer must describe the workflow, showing which decisions are automated, which need human judgment, and who is responsible for each. A vendor that cannot specify the human role has not demonstrated sufficient understanding to propose full automation.

The vendor must specify where data preparation lands and who performs it. tracebloc’s analysis identifies understated preparation requirements as a structural budgeting failure, while real projects spend the majority of effort there. The proposal must specify what preparation is required before calibration, who does it, how long it takes, and what budget assumption supports it. Responsibility must be explicit from the beginning. Vague answers in week one signal how an unresolved preparation gap may surface in week ten.

The vendor must explain how the system behaves on edge cases and who is accountable when it is wrong. The vendor’s answer should cover input types the system has not seen before and the escalation path. A good answer names known failure modes and the human review required for each. Claims of no known limitations avoid the rare, high-stakes exceptions that a production evaluation needs to expose.

The vendor must establish whether the first rebuilt process can be live in weeks and what would prevent that. The condition is a single well-defined workflow with accessible data. A credible vendor names the dependencies that could extend delivery and assigns ownership to each. The obstacles should be specific and examinable. A long roadmap without those conditions indicates a platform implementation.

How to structure a proof of concept that tests the rebuild, not the demo

Convert the vendor’s answers into a bounded operational test. Scope the proof of concept to the single most painful step in the mapped process. A broad pilot creates ambiguous evidence and allows underperformance to be attributed to scope complexity.

Set the success metric before work begins. Cycle time, error rate, exception volume, and handoff failures are useful where they describe the actual failure under examination. Compare the result with the existing baseline. If the metric is chosen after the pilot starts, the evaluation can drift toward whatever the vendor finds easiest to demonstrate.

The timeline should be bounded as well. The checklist puts a thorough AI vendor evaluation at 6-8 weeks when done properly. For a single scoped workflow, a request for more time needs a concrete explanation tied to the dependencies already identified. Duration alone is not evidence of rigor.

Require the test to use operational data within the buyer’s security boundary. In the fraud-detection case, vendors fine-tuned on synthesised transaction data inside the buyer’s infrastructure without accessing raw records. The pilot should likewise test the buyer’s actual failure mode rather than go back to sample inputs that are easy to use.

Establish who maintains the system when it drifts, how wrong answers escalate, and what the handoff to the buyer’s team includes. Before the pilot ends, require demonstrations of audit trails, explainability for a specific incorrect output, and the human-in-the-loop path for identified edge cases.

The pilot should leave a written record of how it performed against the buyer’s failure mode, with clear measurements, operating constraints, and who is responsible for what. That evidence either justifies commitment or ends the conversation cleanly.

Sources

  1. An early pipeline framework for assessing vendor AI solutions to support return on investment - PMC
Filed underAI Vendor Eval

More in AI Vendor Eval