Why AI Agents Can Complete a Task and Still Be Hard to Trust

Imagine asking an AI agent to rearrange a trip. It needs to find another flight, check the fare conditions, keep the hotel dates aligned, and avoid charging the traveller twice.

The agent completes the request. The itinerary looks right. A confirmation arrives.

Now imagine giving it the same assignment again, starting from the same conditions. Would it reach the same acceptable outcome? What if the airline returns an unexpected message? What if the traveller changes one requirement halfway through? Would the agent notice that a payment succeeded even though the confirmation failed to appear?

These questions concern different properties of the system. A demonstration establishes that something can happen. Delegation requires confidence about what will happen across a range of ordinary and inconvenient circumstances.

As AI systems become capable of taking actions, this distinction becomes central. An incorrect answer can misinform a decision. An agent can turn that incorrect answer into a change in someone’s account, schedule, files, or software.

The research challenge is to understand what happens between an agent possessing the necessary skills and reliably using them.

What changes when a model starts acting

An agent is more than the language model at its centre. Its behaviour also depends on the tools it can call, the information it receives, the way it stores progress, and the conditions under which it continues or stops.

Anthropic’s engineering guidance distinguishes between workflows, where software defines the path through a task, and agents, where the model has more control over the sequence of actions. It also describes a practical trade-off: additional autonomy can introduce greater cost, latency, and opportunities for compounding errors. This is guidance from a model developer, rather than an independent comparison of all possible architectures. [Anthropic, Building effective agents](https://www.anthropic.com/research/building-effective-agents).

Consider the difference between extracting a date from a document and deciding what to do about it.

Extraction has a relatively contained objective. Acting on the date might require identifying the relevant person, checking another system, resolving a conflict, and deciding whether an existing arrangement should change.

Each decision alters the circumstances of the next one. If the agent chooses the wrong record early on, later actions can be internally coherent while serving the wrong objective.

A stronger model may reduce some errors. It does not remove the need to understand the surrounding process.

The second attempt is a different test

A useful early contribution came from τ-bench, introduced by Sierra researchers in 2024.

The benchmark placed agents in simulated service interactions involving users, tools, and policy requirements. Evaluation considered whether the resulting state of the system matched the intended outcome.

It also introduced a reliability measure, pass^k, concerned with success across repeated trials. That asks a different question from whether at least one of several attempts works.

The original experiments found substantial inconsistency in the systems tested at the time. Those results should be read as historical findings, not as scores for today’s models. The enduring contribution is the evaluation question: can the system perform the task repeatedly? [Yao and colleagues, τ-bench](https://arxiv.org/abs/2406.12045).

The difference matters because repeated attempts are not always interchangeable.

A researcher can generate several draft answers and select one. A service that changes a reservation cannot necessarily execute several versions and keep the best. Each attempt may leave consequences.

Even where retries are possible, someone must establish whether the previous attempt failed, partially succeeded, or completed without returning a clear confirmation.

Repeating an action is safe only when the system understands what has already happened.

Reliability has several dimensions

Suppose two hypothetical agents complete the same percentage of assignments.

The first fails by stopping early and reporting uncertainty. The second sometimes completes the wrong transaction and announces success.

Their average success rates are identical. The experience of depending on them would be very different.

A 2026 study, Towards a Science of AI Agent Reliability, addresses precisely this limitation of aggregate scores. The researchers organise reliability into four dimensions: consistency, robustness, predictability, and safety.

In the revised paper, they evaluate 15 models across two benchmarks and report that capability improvements produced only small gains in reliability within those evaluations. The finding does not establish a universal law that more capable models cannot become dependable. It shows that higher task scores and better operational behaviour need to be measured separately. [Rabanser and colleagues, Towards a Science of AI Agent Reliability](https://arxiv.org/abs/2602.16666).

These distinctions expose questions that a single leaderboard position cannot answer.

Does an agent withstand a small change in wording? Does it recognise the situations in which it is likely to fail? When it makes a mistake, how far can that mistake propagate?

A system may improve along one dimension while remaining weak along another.

What does an “eight-hour task” actually mean?

Research into longer tasks provides another important view of progress.

METR measures AI performance against tasks characterised by how long they take humans to complete. Its 50% task-completion time horizon estimates the human task duration at which an agent is predicted to succeed half the time.

That is an interpretable way to describe capability. It is also easy to misread.

A model with a particular time horizon has not necessarily worked independently for that length of elapsed time. The duration refers to the human baseline. The agent might finish much faster.

METR also cautions that results depend on the task domain and evaluation distribution. Software and research tasks do not represent every activity performed through a computer. Its January 2026 methodology update expanded the evaluation while acknowledging limited directly measured human baselines for some longer tasks. [METR’s methodology update](https://metr.org/blog/2026-01-29-time-horizon-1-1/), [research note on interpretation and limitations](https://metr.org/notes/2026-01-22-time-horizon-limitations/).

The distinction between a 50% threshold and dependable completion is equally important.

For research, solving half of a demanding task set can mark substantial progress. For someone delegating a particular consequential task, a roughly even chance of success leaves a different practical question.

Time horizons help describe which tasks are coming within reach. They do not, on their own, establish the level of supervision a user can safely remove.

An agent has to preserve its understanding of the job

As work continues, an agent accumulates information: instructions, search results, tool responses, partial solutions, and unsuccessful approaches.

More information can help. It can also make the relevant details harder to isolate.

In a long task, the system needs to distinguish what the user requested from what it inferred, what it verified from what it merely attempted, and what remains unresolved.

Anthropic’s work on context engineering describes techniques including summarising accumulated context and maintaining structured notes. It also identifies a difficulty with compression: a detail discarded as unimportant may matter later. These are engineering approaches to maintaining continuity, rather than evidence that long-term coherence has been solved. [Anthropic, Effective context engineering for AI agents](https://www.anthropic.com/research/effective-context-engineering-for-ai-agents).

Return to the hypothetical travel agent. It might accurately remember that a flight needed changing while losing a qualification: the traveller wanted options presented before any purchase.

The system still has a plausible objective. Its representation of the user’s authority has changed.

This is why the quality of an agent’s memory involves more than retaining facts. The relationships between facts, instructions, permissions, and completed actions also matter.

Verification becomes part of the work

A long sequence of fluent explanations can make an agent appear methodical. But describing a check does not establish that the relevant state was inspected.

In the travel example, three claims require different evidence:

The agent attempted to change the flight.

The booking system accepted a change.

The final booking matches the traveller’s requirements.

An activity log might establish the first. A tool response could support the second. The third requires comparing the resulting itinerary with the original constraints.

This is an analytical distinction, rather than a result from a particular benchmark. It illustrates why evaluating only the final message leaves important behaviour unexamined.

Verification can also change the economics of delegation. Someone who must reconstruct every step may save little time, even if the agent performs the actions quickly.

The meaningful unit of progress is therefore the amount of useful work completed at an acceptable total cost, including checking and recovery.

That cost depends on the task. Some outcomes are straightforward to inspect. Others require the expertise or effort that the agent was supposed to save.

A dependable agent may sometimes do less

Autonomy is often presented as a ladder: better systems require progressively less human involvement.

There is another possibility. Better systems may become more selective about where independence is appropriate.

An agent could continue confidently through a routine, reversible step, then stop when a missing fact or ambiguous instruction would materially change the outcome. That interruption is useful if it occurs at the right moment. Constant requests for confirmation would simply transfer the work back to the user.

The difficult capability is knowing the difference.

Research on repeatability, task horizons, context, and failure behaviour suggests that this judgement deserves evaluation in its own right. A system’s ability to finish increasingly difficult tasks is real progress. So is its ability to recognise when the conditions for finishing correctly are absent.

In the imagined travel assignment, the most revealing moment may arrive when the airline’s response is unclear. Does the agent retry the charge, declare success, abandon the request, or inspect the booking before deciding?

The answer determines whether an impressive demonstration can become something a person is willing to depend on.