What Platform Vendors Omit From the Demo When Your Real Problem Is One Broken Shift Handover

Vendors demo the happy path where handover data exists; real failures happen when it doesn't.

· 8 min read
Cover illustration for “What Platform Vendors Omit From the Demo When Your Real Problem Is One Broken Shift Handover”
AI Vendor Eval · November 5, 2026 · 8 min read · 1,859 words

AI operations transformation for asset-heavy industries without a platform strategy should begin with an audit of one broken process. For shift handover, the test is whether the incoming crew receives an accurate picture of system state. A tool should earn deployment by resolving the gaps that audit exposes, including missing context, conflicting accounts, and unresolved safety conditions.

Why shift handover is where operations actually break — and why the demo never shows it

A crew walks onto the plant floor knowing that a machine is down, but not what the previous shift actually left behind. The fault might be electrical, mechanical, or still unexplained. A repair may be underway without a clear account of what was attempted. The incoming crew inherits the problem before it understands it. The failure mechanism is inaccurate or absent system-state transfer at the moment responsibility changes hands.

TeamSense identifies the same gap in production handovers, where a crew inherits a batch in progress without knowing the actual run rate, scrap trend, or distance behind target. Those omissions change the operational picture even when the batch record exists. Relay distinguishes machine visibility from the context of the response. Manufacturers can see when a machine stopped and how long it remained down, yet lack an account of who responded, what was communicated, and which risks remain unresolved.

A conventional demonstration assumes that account is already accurate. It tests whether software captures and presents the outgoing team's report. HSE identifies failures in handover communication in the 1983 Sellafield Beach Incident and among the contributing factors to Piper Alpha.

What the outgoing team actually knows that no system record captures

An asset can be running while an operator considers it unstable, and a suspected root cause can guide the next response without appearing in telemetry. That knowledge has a different structure from a machine log. It includes intent, attempted interventions, and judgments about what deserves attention. A timestamp cannot substitute for that context.

Bishop et al. (2026) at University of Colorado Boulder describe system records and human reports as complementary but partial sources. Records can be precise and timestamped while observing only part of the task. Human accounts contain intent and task knowledge absent from logs, but remain vulnerable to omission and memory error. Neither source is sufficient on its own. In a controlled spatial multitask environment, their reconciliation approach kept more of the estimated task-state utility than human reports or telemetry did on their own.

The study also found that human reports held a good deal of strategic knowledge that state-focused metrics missed. Relay identifies practical examples, including suspected root cause, troubleshooting already attempted, unresolved work, and conditions the next crew needs to monitor. Much of this depends on manual recall under time pressure. The gap is therefore structural. Retirement compounds it when experienced operators leave with informal knowledge of machines, processes, and shortcuts that has no structured route into the handover record.

How platform demos are structured to avoid the conditions that actually cause failures

The clean demonstration begins with a complete, structured report from the outgoing operator. The incoming crew opens the system, sees an accurate status, and confirms the handover. Each step can work exactly as presented. The demonstration starts with the accurate operational account whose absence causes the real failure.

Relay describes operators managing disruptions while recollecting a full shift and trying to preserve continuity for the next team. A partially completed repair may consequently appear as “in progress” without explaining what was tried, what failed, or what the next crew must not do first. The entry exists, but its meaning is incomplete. A polished display of that entry does not recover the missing repair context.

A developing fault may not yet set off an alarm and may be visible only during a structured equipment walk-down. Telemetry can also conflict with the outgoing operator's account, the condition Bishop et al. designed their reconciliation pipeline to address. These are different tests from entering and retrieving information. A demo that excludes them can validate the interface and documentation workflow, but cannot establish whether the transferred state is accurate enough for action.

This is a limitation of the demonstration's design. It does not require an adversarial explanation of the vendor's intentions. The workflow looks as though it works because it starts with the accurate account it most needs, so the buyer has no evidence of how it works when that account is wrong or missing.

Where the information gap is most dangerous — commissioning, energization, and construction-to-operations transition

Construction-to-operations transition intensifies the same problem through compressed timelines, overlapping workstreams, and divided ownership. Assets may never have run at nameplate capacity, so completion records alone cannot show what condition they are in. Aon's claims data shows design or specification issues that appear only under load, commissioning that is rushed or left incomplete, and maintenance responsibilities changing hands before the new team has all the capability it needs. Project delays can also put equipment warranties at risk of expiring before testing begins.

The transition is an extended phase. Phased handovers, sectional completion, and early operations allow construction, testing, commissioning, and operation to coexist. Different parties can therefore have different views of the same asset. Treating mechanical completion as operational readiness makes that disagreement consequential. The contractor may see an asset as handed over while the operations team still cannot run it safely.

Energization makes the distinction especially concrete and consequential. High-voltage interconnections, on-site substations, redundant distribution paths, and standby generation require precisely sequenced state changes across multiple parties. A missed annotation such as “substation not yet cleared” is a safety risk. Aon describes transition disputes as “rarely about what happened, but which policy is meant to respond”. Determining which policy responds depends on a clearly established asset state and transfer sequence, as does assigning operational responsibility.

Why AI-assisted handover tools inherit the same blind spot — and add a new one

An agent working from an incomplete record inherits its omissions, even if it produces a clear and orderly summary. Clean demonstration inputs conceal that weakness just as they do in conventional software. The workflow must reconcile the information needed to resume work.

Bishop et al. show how direct generation differs from structured reconciliation. Their end-to-end LLM, given the same inputs, maintained comparable estimated utility but incurred substantially more misinformation than their structured approach. The finding concerns a controlled task environment. It nevertheless identifies a precise evaluation risk. A plausible handover can retain useful information while adding errors that a comparison based only on usefulness would miss.

The architecture determines which evidence the agent can use. A workflow that marks handover complete when a form is submitted establishes only submission. An agent with access to only one record stream cannot find knowledge about the work that never went into it. Reconciling telemetry and human reports addresses that separation, while provenance tracking makes their contributions traceable. Without it, reconstructing what the agent saw, concluded, and passed forward becomes difficult.

Escalation remains a separate dependency within the operational workflow. Word of a developing fault must reach the right maintenance authority before the incoming crew starts work on the asset. Summarizing a log does not repair an escalation trigger that was never captured. AI therefore adds the risks of a poor summary and a structured output whose errors and omissions are difficult to trace.

What a process audit of a real handover reveals that no demo ever would

A process audit checks what crews actually pass on to each other. Its central question is whether the incoming team receives an accurate and sufficient state picture to act safely and effectively. Evidence of a completed form answers a different question. The audit therefore follows unresolved work and safety conditions across the shift boundary. This moves the evaluation from workflow completion to operational continuity.

TeamSense identifies the formal transfer of open work orders, their current status, safety status, and next required action as part of that continuity. An audit distinguishes this from work that is verbally mentioned and then lost. It also examines whether structured pre-handover walk-downs capture developing faults before alarms appear. Relay says active lockout/tagout records, permit-to-work status, and energy isolation points must be formally passed on. Near-misses and other safety incidents need enough context for the next crew to understand the inherited risk.

Provenance adds another dimension to this examination of handover quality. Bishop et al.'s framework makes the source of information traceable so disagreements between telemetry and human accounts can be detected and reconciled. An audit can therefore examine not only whether a fact appears, but whether its origin remains visible when another source contradicts it. At the construction-to-operations boundary, Aon identifies defined testing plans, responsibility matrices, commissioning criteria, and scenario-tested handover processes. Contractual milestones and policy triggers must also align with the actual delivery sequence from the outset.

For operations leaders, the practical step is to audit the broken scenario before evaluating a platform. That means looking at the disputed repair status, missing isolation context, or developing fault that the current process fails to pass on. The vendor must then show how the tool works under the conditions the audit found. A clean-input demo supplies no evidence of that capability, and a vendor unable to reproduce the audited conditions has not established whether the tool can resolve the gaps the audit found.

How do you evaluate an AI shift handover tool without buying a whole platform?

The evaluation should begin with the audited failure. Give the demonstration team an outgoing shift record that has gaps and some wrong information and was written under time pressure. Ask how the system helps the incoming crew find and close the gaps before they act. The scope is one operational process. This gives a way to judge whether AI improves manufacturing without first signing up to an enterprise software strategy.

State confirmation is the first criterion, because form submission alone cannot establish an accurate handover. Conflict handling is the next, including whether the system exposes disagreement between telemetry and the operator's account or silently favors one source. A developing fault needs a different test. The evaluation must show how observations from a structured walk-down enter the workflow when there is no alarm to retrieve. These conditions distinguish evidence about operational performance from evidence about interface quality.

Escalation and provenance then test what happens when uncertainty remains. If an incoming supervisor cannot confirm a safety isolation, the demonstration should establish who owns reconciliation and what happens if nobody acts. Where an AI agent participates, an auditor should be able to reconstruct what it received, concluded, and passed forward. Deployment should also fit the work itself. The tool should capture state during the shift rather than depend on end-of-shift reconstruction, with explicit mechanisms for detecting and escalating conflicts before the incoming crew acts.

Aon's transition principle is “don't hand the asset over just because you can”. The equivalent for technology is not to deploy because the demonstration succeeded. A handover tool placed on a broken process can accelerate the breakdown, so selection must follow the audit. The operations leader's first question is not what the tool does, but what the handover actually breaks on.

Sources

  1. From Construction to Operations: Where Transition Risk Really Bites
  2. Structured State Reconciliation for Human-AI Task Handover
  3. Bridging the Gap: Solutions for Managing Construction-to-Operation Transition Risk
Filed underAI Vendor Eval

More in AI Vendor Eval