Do Enterprises Really Need AI Agents? When to Let Models Decide the Next Step

TL;DR

Adopting an AI assistant isn’t about how smart it is—it’s about whether you dare to give it the authority to decide what to do next based on changing conditions. To deploy safely, don’t just prompt it to “be careful”; you must put a tight harness on it and enforce strict, unbreakable action boundaries:

• Lock Down Permissions: Never grant AI superuser privileges. It can only do what the employee is allowed to do and what the current task explicitly permits—halt immediately if any approval is missing. Connecting to a system does not mean permission to modify data directly.
• Prevent Disruption: Always re-verify live data before executing to avoid acting blindly on stale information. Ensure write operations never create duplicate orders during network retries, and log every reasoning trace for auditing.
• Count True Costs: Measure value only by verifiable net gain—how much more it earned or how much loss it prevented compared to human baselines—while absorbing all failed retries and escalation overhead. Never confuse gross transaction volume with revenue, and don’t expect instant headcount cuts.
• Delegate in Small Steps: First let it observe silently, then recommend, then draft, and finally execute under strict constraints—relaxing only one rule at a time. The more powerful the model, the tighter the guardrails it needs.

At nine in the morning, an employee sits down to work and discovers that a system they used yesterday now says “Access denied.”

The same problem can affect an office project platform, a store’s back-office system, a warehouse scanner, or a mobile app used by field staff. The employee usually knows only that access has suddenly stopped. They take a screenshot and submit an IT service ticket titled “Access problem.”

The service desk still has to establish which system is affected, what device the employee is using, where they are signing in from, whether access worked yesterday, and whether their role or line manager has recently changed. A technician may then inspect the identity system, group membership, software licences, device-compliance status, and application logs. The cause might be an expired session, a failed group synchronisation, a device that no longer meets security requirements, or a request for new access that has not yet been approved.

Large organisations handle these requests repeatedly. The final fix is not always the expensive part. Most of the effort goes into gathering information, classifying the problem, checking several systems, and routing the ticket to the right team.

A retrieval-augmented generation (RAG) assistant can retrieve relevant documentation, and a fixed workflow can collect necessary details to open a ticket. Neither, however, dynamically reassesses its investigation path after each intermediate finding.

An AI agent can investigate further in response to what each step reveals. It might first check whether the service is down for many users. If the service is healthy, it can verify the employee’s identity, device status, and existing access. It can guide the employee through reauthentication when a session has expired, or run an approved diagnostic when a device is non-compliant. If current access no longer matches an earlier approval, it can assemble the evidence and open a request to restore it. If the employee is asking for new access, the agent should stop and refer the request to the line manager or data owner.

The agent must not add the employee to a new group, disable a security control, or copy another person’s permissions merely because the employee says, “It worked yesterday.” Enterprise systems must still verify identity, prior approvals, and the authority granted for the current task. Every access change must record who requested it, who approved it, what changed, and when the permission expires.

The earlier article “Is Enterprise RAG Worth Implementing: From Scenario Selection to Ongoing Governance” discussed how enterprises can let models use internal knowledge. In the scenario considered here, RAG mainly determines what the system can know; a fixed workflow follows a predefined route; and an agent uses changing state to decide what to inspect next, which tool to call, and when to stop and hand the task to a person. The employee may see only a faster IT assistant. The enterprise, however, has delegated authority to choose the next action.

The central question is therefore not whether a model can invoke tools. It is whether a task benefits enough from feedback-driven path selection to justify delegating that choice to a model—and, if it does, where the boundaries of its execution space must lie.

1. Agents Change Who Chooses the Next Step #

Software has long been able to coordinate several systems automatically. After an ecommerce order is placed, a program can check inventory, create a picking task, notify the warehouse, and update the delivery status. A monitoring platform can restart a process or add computing capacity when it detects a service failure. These systems may execute many steps without being agents in the sense used here.

Anthropic distinguishes workflows, in which models and tools follow predefined code paths, from agents, in which a model dynamically directs the process and tool use [1]. OpenAI similarly describes an agent as using a model to manage execution, select tools according to the current state, and decide when the task is complete [2].

Both definitions point to an easily overlooked change: decisions that engineers once encoded in advance may now be made by a model at runtime.

A fixed workflow can be represented as:

Input → Select branch based on predefined conditions → Execute step → Output

An agent is closer to a feedback loop:

Observe current state → Select next step → Invoke tool to query or change state
        ↑                         ↓
        └────── Read result and re-evaluate ──────┘

The access-failure example shows how an agent can adapt its investigation to each result. Once an agent moves beyond retrieving information and begins replenishing stock, issuing refunds, or changing accounts, the risk increases: each operation changes the business state that the next iteration will read.

A replenishment agent that detects a sudden rise in sales might check the promotion calendar. If no promotion explains the change, it can look for local events, then verify supplier lead times and, if necessary, investigate alternatives. Engineers do not have to enumerate every possible combination because the model selects the next step from the evidence collected so far.

That flexibility creates both the value and the risk. A fixed program can fail because of its rules, code, or input data. An agent can also reinterpret the goal, select different evidence, and choose a new action on every iteration. Its judgment does not remain confined to an answer: it can alter the state on which subsequent decisions depend.

Suppose the system mistakes a one-off group purchase for sustained consumer demand and places an additional order. By the next iteration, inventory and in-transit quantities have changed. If the agent misreads those changes, it may alter replenishment plans for other stores as well. Each step can appear reasonable even as the sequence drifts away from the original objective.

Tool use alone does not make a task suitable for an agent architecture. The real test is whether feedback requires the path to be reconsidered at runtime, and whether the enterprise is willing to let a model make that choice.

Tools and Protocols Provide Connectivity, Not Authority #

A tool is an interface for a query, calculation, or business operation, such as reading inventory, calculating safety stock, or creating a draft order. The agent may decide whether to call it, but the tool neither understands the overall objective nor chooses the next step in the task.

Current connectivity technologies address distinct dimensions of integration:

• Model Context Protocol (MCP): Standardises how AI applications discover, read, and invoke local or remote resources and tools [3];
• Handoff Mechanisms: Enable a primary agent to transfer task execution and conversation state dynamically to a domain-specialist agent upon intent classification [4];
• Agent2Agent (A2A) Protocols: Standardise dynamic capability discovery, message routing, and collaboration across heterogeneous multi-agent networks [5].

Key Takeaway: Connectivity protocols determine what can be called, not what is authorised to execute, and they cannot substitute for rigorous architectural justification.

2. Which Tasks Justify Model-Directed Path Selection? #

Agents are not suitable for every complex task. They fit a narrower class of work: the objective is reasonably clear, the required steps cannot be exhaustively specified in advance, and the system can observe the outcome of each step and adapt.

Complexity alone does not justify an agent. Tax calculations may involve many rules, but a rules engine is usually more reliable when those conditions can be encoded. An approval process may cross several departments, but a workflow platform can handle it when the sequence, permissions, and exceptions are known. Adding an LLM to fixed steps produces an LLM-assisted workflow, not a reason to let the model control an adaptive loop.

Four conditions help determine whether model-directed path selection is worthwhile.

Is the Task Path Genuinely Difficult to Predetermine? #

Agents are better suited to tasks with varied inputs, dispersed information, and many combinations of exceptions. When diagnosing a software failure, for example, an engineer uses the logs to decide whether to inspect configuration, networking, a dependent service, or a recent code change. Each finding shapes the next step, making a complete decision tree difficult to write in advance.

If the task has only a few stable branches, an agent merely replaces logic that could be coded and thoroughly tested with a probabilistic decision.

Does Each Step Yield Actionable Feedback? #

An agent needs to observe what happened after it acted. A replenishment system can read updated inventory, orders, and delivery status. A coding agent can run tests. A service agent can confirm that a ticket was created.

Without timely feedback, the system cannot correct itself reliably. It may continue from stale state, apparently completing a multistep task without knowing whether the first operation succeeded.

Can the Impact of an Error Be Reliably Contained? #

Early deployments will usually encounter cases that testing did not cover. An error that produces only a draft is relatively cheap to detect and correct. An error that orders stock for every store, changes a customer’s entitlements, or initiates an external payment leaves little room for experimentation.

Enterprises should therefore begin where errors have limited impact, or where underlying systems can reverse an operation or compensate for its effects. The task with the largest theoretical value is not necessarily the right first deployment if it also has the most severe failure modes.

Can the Expected Benefit Justify the Added Cost? #

Agents add model calls, state management, evaluation, monitoring, and exception handling. A complex autonomous system is unlikely to pay off when a person can confirm a recommendation in two minutes. The case is stronger when employees repeatedly gather information across systems, decide what to check next, and wait for feedback. Even when the first three conditions hold, the enterprise still has to show that the benefits of adaptive path selection can cover these additional costs. This is only an initial screen for scenarios with no plausible room for benefit; it cannot replace an estimate of the project’s actual costs and business outcomes.

Combining path variability, feedback quality, error impact, and net value produces a simple framework for choosing an approach:

Anthropic recommends starting with the simplest approach that solves the problem. Agents trade higher cost and latency for flexibility; if retrieval, one model call, or a fixed workflow is enough, the system does not need greater autonomy [1].

A first-person interview with a Chinese forward-deployed engineer (FDE) describes a similar case. An education group wanted to use AI across the full workflow for school research and report generation. A professional team spent three months building four agents, but the system required school leaders to change how they recorded, entered, and uploaded information. Incomplete inputs led to inconsistent output, and few people used the system. A school principal who had attended the training instead kept the existing manual process and used a simpler tool only to generate report templates; other principals found that approach easier to adopt [6].

The account is an anecdote, not a controlled study, and it does not show that simple systems are always better than agents. It does show that a “full-workflow” design may merely transfer operating cost to users if it first requires frontline staff to rebuild their working habits. Enterprises should identify the smallest effective point of AI intervention before deciding whether a model needs to control the whole task path.

Passing this initial screen establishes only that an agent fits the task technically and operationally. Whether the project merits investment still requires a separate comparison between its additional costs and measurable business outcomes.

3. Can the Benefits of an Agent Project Justify Its Additional Cost? #

A task can be technically suitable for an agent without the project being economically worthwhile. An enterprise should first estimate model-call spending, then include tools, runtime, and human review in the total cost of each successful task.

How Should Enterprises Estimate Token Costs? #

A simple formula can establish a lower-bound monthly token budget for one model and billing tier. The easiest variable to underestimate is \(A\): one business task submitted by a user does not imply only one model invocation. Let \(N\) be the number of user-submitted tasks per month and \(M\) the number of billable model invocations per month:

where:

• \(U\) is the number of users.
• \(D\) is the number of active days per month.
• \(C\) is the number of business tasks each user submits per day, not the number of calls made inside the agent.
• \(A\) is the average number of billable model calls triggered by each business task, including routing, planning, execution, review, retries, and subagents, but excluding database queries or ordinary business API calls that do not use a model.
• \(M\) is the total number of billable model invocations per month.
• \(T_{\mathrm{in}}\) and \(T_{\mathrm{out}}\) are the average input and output tokens per model invocation (the fundamental units of text and data processed by the model).
• \(P_{\mathrm{in}}\) and \(P_{\mathrm{out}}\) are the prices per million tokens.
• \(K_{\mathrm{token}}\) is the monthly token cost for the selected model and billing tier.

None of these variables is a fixed constant; each must be estimated for the specific business case. \(U\), \(D\), and \(C\) should come from the target scenario’s user population and task volume. \(A\), \(T_{\mathrm{in}}\), and \(T_{\mathrm{out}}\) should be estimated through prototype tests or execution traces. \(P_{\mathrm{in}}\) and \(P_{\mathrm{out}}\) depend on the model, billing tier, caching conditions, and region actually used. \(M\) and \(K_{\mathrm{token}}\) are calculated from those inputs. Until observed data is available, enterprises should plan capacity with multiple scenarios rather than treat one set of assumptions as an industry norm.

At \(A=1\), each business task triggers only one billable model invocation on average, with no additional calls for routing, planning, review, retries, or subagents. This article uses \(A=1\) only to show the theoretical lower bound of the formula. For an agent that repeatedly observes results and chooses the next step, it is usually an unrealistically optimistic operating assumption.

In production, teams can calculate \(A\) from observed usage:

The numerator should include billable calls made inside a hosted agent service, not only requests sent directly by the client. Separately priced database queries, MCP tools, and business APIs need their own cost lines.

The following is a capacity-planning example, not an estimate of typical enterprise usage. Consider 500 users, 20 active days per month, and 10 business tasks per user per day. The monthly task volume is:

To show how prices enter the formula, start with \(A=1\) as a theoretical lower bound. As of August 28, 2026, the paid Standard tier for Gemini 3.7 Flash Standard charged $0.75 per million input tokens and $3.75 per million output tokens, including thinking tokens [7]. Assume that each invocation averages 8,000 input tokens and 2,000 billable output tokens. This is a medium-complexity capacity-planning scenario, not an industry average: short routing calls may use far fewer tokens, while calls carrying substantial business material, tool results, or prior state may use substantially more.

If each business task triggers five billable model calls on average, \(A=5\), the same workload produces 500,000 model invocations and the monthly token cost rises to \(\$1{,}350\times5=\$6{,}750\). Five calls per task is also a scenario assumption, not an industry norm; enterprises ultimately need to measure the real value of \(A\) from execution traces.

As of the same date, the four representative models produce the following estimates under the same task-volume and token assumptions:

These models differ in capability, tokenisation, and intended workload. The figures show only the order of magnitude of the budget; they are not a performance or value ranking.

Public sources do not provide an average token count that applies to every enterprise agent, but engineering disclosures from major model providers explain why short-context assumptions severely underestimate costs. In an iterative feedback loop, token consumption is compounded by three multiplier mechanisms:

• Sub-Agent Exploration Overhead: A specialized subagent exploring a problem space in a sandbox can consume tens of thousands of tokens or more, only to return a compressed summary of roughly 1,000–2,000 tokens to the lead agent [11]; Anthropic also reported that in its multi-agent research system, multi-agent architectures consumed approximately fifteen times as many tokens as standard chat interactions [12].
• Context Accumulation: Google’s Gemini documentation confirms that system instructions and tool definitions count towards input tokens and that multi-turn usage includes prior context [13]; OpenAI’s Prompt Caching documentation similarly illustrates that multi-turn requests carrying tool schemas, message histories, and tool results readily reach 12,000 to 15,000 input tokens or more [14].
• Tiered Model Invocations: Production systems often use lightweight/lower-cost models for routing, frontier models for deep reasoning and planning, and specialized models for compliance checks—compounding billable invocations and prompt caching misses on a single business task.

Production systems also incur costs for paid tools, hosting, orchestration, vector databases, monitoring, human review, and incident response. The capacity formula therefore establishes only a theoretical lower bound for token spending.

Business Value Lies in Task Outcomes, Not Agent Counts or Headcount Cuts #

The calculation above shows only where the model-call budget begins; it does not determine whether the project is worthwhile. Once the system architecture and the pattern of agent calls and collaboration are known, teams should replace the assumptions with observed call and token counts.

To distinguish monetary values from task-volume symbols, this article uses \(K\) for monetary cost and \(S\) for the number of successfully completed business tasks. The earlier \(K_{\mathrm{token}}\) includes only the monthly token cost for one model and billing tier; \(K_{\mathrm{total}}\) represents the total cost of models, tools, runtime, infrastructure allocation, and human review over the same measurement period. The total cost per successful task, \(K_{\mathrm{success}}\), is:

The denominator uses the number of successfully completed tasks, \(S\), rather than total initiated runs. This ensures that the costs of aborted runs, ineffective retries, and manual escalations are fully absorbed by successful deliveries, accurately reflecting the true unit cost per completed task.

Return on investment (ROI) is then calculated against the economic benefit attributable to the agent relative to baseline, \(V_{\mathrm{total}}\), over the same measurement period:

Understanding \(V_{\mathrm{total}}\) requires strictly distinguishing gross transaction volume (book value) from attributable incremental gain. For instance, if an agent facilitates a procurement or sales task with a nominal transaction value of USD 100, that entire USD 100 must never be counted as AI revenue. After excluding underlying product costs, physical fulfillment, and baseline labor, only the incremental margin or loss avoided relative to the manual baseline—such as reduced stockouts, avoided discount leakage, or recovered engineering hours—can be legitimately attributed to the agent.

In enterprise deployments, the autonomous resolution rate is not a static constant achieved on day one; it follows an evolutionary learning curve across the project lifecycle. Enterprises must evaluate project viability across two distinct phases.

1. Early Pilot Phase: High Unit Delivery Cost and Negative ROI #

During early cold-start and shadow pilot stages, strict action boundaries, uncalibrated edge cases, and frequent safety trips keep autonomous resolution low.

Assuming a monthly volume of 100,000 submitted tasks, the early system achieves only a 15% autonomous pass rate (\(S_{\mathrm{early}} = 15{,}000\)), with the remaining 85,000 tasks escalating to human operators. Total monthly expenditure of USD 10,000 (covering all 100,000 attempted runs) is absorbed entirely by these 15,000 deliveries:

If attributable gain per task is only USD 0.50 in early deployment, 15,000 successful runs produce USD 7,500 in total attributable value, falling short of operating expenditure:

Factoring in upfront development and process-engineering amortization makes early-stage losses even more pronounced. This explains why many enterprise pilots face skepticism during their initial months.

2. Mature Operational Phase: Scale Dilution and Positive ROI Expansion #

After multiple iterations of trace analysis, tool interface tuning, prompt calibration, and regression testing, the system converges within its action envelope, significantly lifting autonomous resolution.

Assuming the mature system achieves an 85% autonomous pass rate (\(S_{\mathrm{mature}} = 85{,}000\)), with 15,000 tasks triggering safe human escalation, unit delivery cost is diluted dramatically:

As integration deepens, attributable gain per task climbs to USD 1.00 (roughly 1%–2% of transaction value). 85,000 successful tasks generate USD 85,000 in attributable gain, unlocking strong steady-state returns:

3. Business Implications of the Two-Phase Comparison #

Comparing the two phases side by side establishes clear financial guardrails for executive decision-making:

Under this definition, \(\mathrm{ROI}=0\%\) means that the benefit exactly covers operating costs; only a negative ROI indicates that the project has failed to recover its run-rate expense. This represents an operational return estimate during steady-state (run-rate); when assessing full project viability, enterprises must also incorporate upfront system development and process-engineering costs, calculating the payback period accordingly. Here, \(V_{\mathrm{total}}\) should include only incremental, verifiable cost savings, avoided losses, or margin contribution relative to the baseline. It must not count gross revenue routed through the agent, nor should theoretical hours saved be treated automatically as realized cash returns.

In high-stakes scenarios—such as enterprise sales lead qualification, insurance fraud detection, or critical supply chain routing—the attributable gain per successful task may reach USD 10–20 or more, though monthly volume in such specialized domains is typically lower. Conversely, for trivial micro-lookups, unit benefit might be only USD 0.25. Across all scenarios, the net incremental contribution relative to baseline remains the sole valid metric for economic justification.

Industry research consistently reveals that the vast majority of generative AI and agent pilots struggle to transition from proof of concept (PoC) to scaled production. The primary bottleneck is rarely raw model reasoning, but rather the absence of clear business baselines, runaway unit costs per successful delivery, and the inability to prove sustained incremental margin. When evaluating business value, enterprises must avoid confusing automation with headcount displacement:

• Automated Throughput Does Not Equate to Redundant Headcount (The CBA Voice Bot Case). After introducing an AI voice bot, Commonwealth Bank of Australia announced that it would remove 45 customer-service roles. Following a union challenge, the bank acknowledged that the roles were not redundant, reversed the decision, and apologised. The union said call volumes and the workload handled by people had increased after the bot was introduced [15]. The system was not an LLM agent under the definition used here, but the case still shows that automated handling volume cannot be translated directly into displaced jobs. Repeat contacts, transfers to people, complaints, resolution time for difficult cases, and employee workload all matter.
• Partial Task Improvement Cannot Directly Drive Scale Downsizing (The Home-Improvement Enterprise Case). The same interview also describes a project at a privately owned home-improvement company. Management wanted AI to reduce a workforce of more than 6,000 people to 3,000 and treated the number of positions an agent could remove as its measure of value. The three-month project delivered seven agents, but according to the interviewee, most improved the quality of particular tasks and only one sales agent might generate additional revenue; they did not map directly to the original headcount target [6]. This is one implementer’s account of a specific project, not evidence about every enterprise. It nevertheless exposes the mismatch created when a headcount target substitutes for a defined business task and acceptance criteria.
• Decompose Work into Tasks as the Fundamental Unit of Accounting. Enterprises are best served by evaluating agents at the task level rather than treating an entire job as the unit of automation. A job usually combines information processing, coordination, exception handling, negotiation, and accountability. Automating one component does not make the remaining responsibilities disappear, nor does it translate directly into positions that can be eliminated. An enterprise should first identify which specific task the system replaces or accelerates, then measure cycle time, quality, escalation rates, human correction volume, and residual human workload.

Showing that a project is worth the investment still does not justify unconstrained action. The enterprise must next turn the permitted tools, data, parameters, and state changes into an explicit boundary.

4. Enterprises Need to Design the Agent’s Action Envelope #

Traditional access control asks who may use which system. An agent needs a more specific constraint: for this objective, state, and period, which actions may it take, and within what parameter ranges?

I refer to this set of constraints as the action envelope. The enterprise does not have to prescribe every step, but it must define the space within which the agent may act.

Deterministic Systems Must Enforce the Action Envelope #

A replenishment agent’s action envelope might include:

An action envelope is not a prompt that says “act carefully.” A prompt still asks the model to interpret the restriction. The envelope must be enforced by tool interfaces, identity systems, rules engines, and business state machines.

The model and deterministic controls have different responsibilities:

Key Takeaway: A model can propose what to do next, but deterministic code must decide whether that action is permitted to take effect now. An action envelope is not a cautious prompt; it is a deterministic boundary enforced externally by systems and rules.

A model might recommend increasing a product’s replenishment quantity by 30%, while the order interface permits drafts only within 15% of the baseline. It might infer that a supplier will be late, but it cannot redirect the order to a supplier that has not passed onboarding checks. If the business needs to cross the existing boundary, the system must explain why, present the evidence, and refer the decision to an authorised person.

Effective Authority Is the Intersection of User, Agent, Task, and Tool Constraints #

A common failure mode in enterprise deployments is provisioning an agent with a single, highly privileged service account while exposing a uniform interface to all users. This simplifies integration but completely bypasses source-system row-, column-, department-, and tenant-level access controls. An employee permitted to view inventory for one region might query national data through the agent; someone allowed to read supplier records might trigger an order operation they could not perform directly.

Each task should instead record the initiating user, the agent performing the task, and the authority granted for that task. Effective permission is the intersection of several boundaries:

Effective authority = User permissions ∩ Agent permissions ∩ Task delegation ∩ Tool policy

User permissions establish what the initiator is authorized to view; agent permissions cap the baseline capabilities of the system; task delegation binds the session to a specific purpose and lifespan; and tool policies validate parameters, safety thresholds, and state transition rules at execution time. Failure at any layer must immediately halt execution—it cannot be bypassed by rephrasing a prompt, switching tools, or delegating the task.

The system needs to implement the following types of controls separately:

The data source, MCP server, or business API must enforce permission checks. A prompt is not an authorisation mechanism, and a model cannot establish identity from a user’s own description. MCP authorisation requires a protected server to validate the access token, its intended resource, and its scopes [16]. In an enterprise deployment, the design should also verify the user, the acting agent, and the delegation between them, then apply least privilege at the resource and tool layers. Hiding a tool in the client reduces its visibility to the model; it does not create a security boundary.

Data minimisation still applies. A task that needs aggregate store inventory should not receive customer records, employee data, or entire supplier contracts. Nor should tracing retain all raw inputs by default merely because the system needs an audit trail.

When a task is handed to a specialist agent, it should receive only the permissions required to complete that task—not the managing agent’s full identity, context, or credentials.

Which Actions Require Step-Up Authorisation or Human Approval? #

Data access is only one part of the envelope. Once an agent can call write tools, the enterprise must consider whether an operation has lasting effects, whether it can be reversed, and how far the consequences can spread.

A NIST summary of an expert workshop on agent tools identifies dimensions including read or write access, lasting effects, reversibility, observability, and autonomy [17]. A tool name therefore says little about risk. An “update order” operation restricted to one draft is a very different capability from one that can alter confirmed orders in bulk.

The boundary must be tied to the current task, not only to the agent’s identity. A system that can create purchase drafts in one workflow should not use that capability in every conversation. Before each action, it should recheck the task objective, initiator, applicable data, permitted tools, expiry time, and proposed parameters against the original authority.

This determines where human approval belongs. Low-risk reads need not be approved one by one, but a system can require confirmation before crossing an action boundary. The user should approve a concrete change—“increase store A’s order for product X from 100 to 112 units”—not a vague instruction to “continue optimising inventory.”

Anthropic has described the trade-off between approving every operation and reviewing a plan before execution. A user can approve a bounded plan while retaining the ability to intervene, avoiding a stream of context-poor confirmation prompts [18]. Crucially, enterprises must guard against the risk of human oversight devolving into mere “rubber-stamping.” When a system bombards frontline staff with low-stakes confirmation dialogues for routine queries or minor adjustments, operational pressure inevitably triggers severe supervision fatigue. Users instinctively treat prompts as software terms of service, clicking “Approve” without reading and completely disabling the safety boundary. Effective approval workflows must be triggered by exception and variance: standard operations within established baselines should execute deterministically and silently, while human intervention is reserved for material deviations, sensitive counterparties, or boundary crossings—accompanied by structured diffs that make critical trade-offs immediately legible.

Authority to act on a user’s behalf depends on more than the user’s consent. Perplexity’s Comet agent can sign in to user accounts, compare products, and place orders. In 2025, Amazon sued Perplexity, alleging that the agent concealed its automated identity, accessed private customer accounts, and created risks to data and the shopping experience. Perplexity denied the allegations and argued that Amazon was restricting user choice. In March 2026, a district court issued a preliminary injunction; in August, the Ninth Circuit vacated the injunction and remanded the case, holding that Amazon was unlikely, on the existing record, to show that Perplexity had “accessed” Amazon’s computers within the meaning of the relevant law [19].

The case remains unresolved on the merits, but it already shows that delegated authority is not one-sided. A user’s permission for an agent to shop does not mean the retailer has authorised that agent to act for the user. Allowing a user to sign in does not necessarily authorise a third-party system to access the account and place orders automatically. Enterprises need to verify the user, the agent, and the delegation between them, and establish who accepts the transaction terms and who bears responsibility for mistaken purchases or data access.

Audit Trails Must Reconstruct Authority, Actions, and State Transitions #

A chat transcript is not an agent audit trail. It may show what the user asked, but not which records the system retrieved, whose authority a tool call used, who assumed responsibility after a handoff, or how business state changed.

An enterprise-grade, reviewable audit chain must capture three causal dimensions:

• Identity and Delegation Provenance: Identify who initiated the task, under whose authority the system acted, and which credentials were stripped when delegating to sub-agents;
• Execution and Approval Traces: Record precisely what data ranges the agent accessed, which tools and parameters it invoked, and which human operator approved the state change;
• State Transitions and Exception Logs: Reconstruct business data changes before and after execution, intermediate failures and retries, and rollback or compensation logs triggered upon anomaly.

High-risk writes also require tamper-resistant, immutable audit logs. Auditors should be able to trace a macro business state change back to an originating task and reconstruct every micro state transition within that execution.

Audit logs may themselves contain prompts, customer data, access tokens, or commercially sensitive information. Enterprises should redact sensitive fields, control access, prevent tampering, and set retention periods. Audit and security teams need the evidence required for review, not unrestricted access to every raw business record. The purpose is to establish accountability, not to create a second, higher-risk data warehouse.

An action envelope limits what a model can do; it does not make every permitted step correct. A different class of risk emerges when individually compliant actions interact inside a loop.

5. Why Agent Errors Propagate Beyond a Single Output #

When no tool call or business write follows, an error in a question-answering system usually does not directly change external system state. An agent’s output, by contrast, can become a tool input, and the tool can change business state. That state then becomes evidence for the next decision, allowing the original error to propagate.

A replenishment agent that only recommends an order produces one bad suggestion when it misreads demand. An agent that can modify orders changes in-transit inventory, budgets, and supplier commitments. On the next iteration, it must distinguish its own changes from market changes and from operations that have not yet completed.

Several mechanisms make this amplification more likely.

State Changes While the Plan Remains Stale (TOCTOU) #

After an agent reads inventory, an employee may change the order or a supplier may update its price before execution. A condition that was true when checked may no longer be true when used. This time-of-check to time-of-use (TOCTOU) problem is not unique to agents, but longer tasks with more steps create more opportunities for it.

Permissions and business state cannot be validated only at the beginning. Before an important write, the system must reread critical fields and confirm that they still match the plan.

Retries Without Idempotency Multiply Operations #

If a tool call times out, the agent may not know whether the order was created. Retrying simply because no success response arrived can create a duplicate.

Write operations therefore need idempotency: repeated delivery of the same business request must take effect only once. Without an idempotency key and explicit state, an automatic retry converts a communication failure into a business failure.

Local Optimization Drifts Away from the Global Objective #

If an enterprise asks only for a lower stockout rate, the agent may keep increasing safety stock. Availability improves while obsolescence, waste, and working capital rise. The model may violate no individual rule while repeatedly optimising an incomplete objective.

Adding “also consider cost” to a prompt does not solve the problem. Competing metrics, hard limits, and stop conditions need explicit enforcement, and the enterprise must watch for ways the agent improves the visible metric at the expense of the real objective.

Downstream Systems Treat Probabilistic Output as Ground Truth #

Later agents may read an earlier agent’s summaries, tags, forecasts, or decision records as facts. Unless the system distinguishes source data, human-confirmed results, and model-generated content, an early inference can gradually become accepted evidence. Generated content needs provenance and confirmation status; an unverified inference should not be written into a field that downstream systems treat as authoritative.

Multi-Agent Workflows Introduce Coordination Failures #

A single action loop can amplify an error; multiple agents also create shared-state and ownership problems. Two subagents may recommend conflicting actions from inventory snapshots taken at different times, or both may create the same draft order. One agent may write a hypothesis into a summary that another treats as verified. Without an explicit owner, a task can circulate between agents, be investigated twice, or be handed off without anyone completing it. Production multi-agent systems also pay coordination costs for decomposition, parallel execution, and result integration [12].

Recording who called whom is not enough. Every delegation needs a task ID, state version, current owner, and completion condition. Before merging parallel results, the system must compare timestamps, provenance, and write conflicts. State-changing operations should have one clear executor, or use idempotency and transactions to prevent duplicate effects.

Evaluation Must Inspect Action Traces, Not Just Final Answers #

The OWASP Top 10 for Agentic Applications 2026 includes memory and context poisoning, tool misuse, insecure inter-agent communication, and cascading failures. Its mitigations emphasise least privilege, action-level validation, resource limits, and continuous monitoring [20]. Because agent errors compound across iterative tool invocations and state changes, evaluation cannot rely on whether the final text response appears coherent; it must audit the complete execution trace [21].

Industrial-grade agent trace evaluation should be built on four core pillars:

• Environment and Version Provenance: Capture exact versions of the underlying foundation model, system prompts, tool schemas, permission policies, and evaluation test suites, alongside precise counts of model calls, token consumption (input, output, cached), and billable API invocations to ensure regressions and cost anomalies remain reproducible and diagnosable.
• Causal Decision Auditing: Validate whether the model selected optimal tools, avoided infinite loops or redundant retries, resolved conflicting outputs, and refrained from reasoning over stale state snapshots.
• Deterministic Rule Enforcement: Implement automated programmatic gates over execution traces to detect privilege escalation, boundary violations, duplicate writes, hazardous parameters, and anomalous tool sequences.
• End-to-End Business and Economic Attribution: Connect raw technical completion rates to tangible business impact—measuring human correction volume, residual human workload, mean time to recovery (MTTR), dispute rates, and inventory carrying costs to guard against cosmetic automation that inflates operational overhead.

These metrics become evidence for expanding authority only when they feed continuous evaluation. Teams can build test sets from real tasks, past failures, and edge cases; record a baseline for the manual or existing automated process; and test the agent where it cannot change production state. Deterministic checks suit permissions, parameters, formats, duplicate writes, and state transitions. Tool choice and task path require trace review. People, or evaluation models calibrated against people, can assess open-ended recommendations. High-risk outcomes still require professional review.

After each round, the team should attribute failures to the model, data, tools, or process before changing prompts, adding data, tightening permissions, or revising the workflow. New production failures should enter the regression set.

Once errors can propagate through a loop, autonomy is not a launch switch. It is a deployment variable that must be increased only as evidence accumulates.

6. How Should Enterprises Gradually Delegate Action Authority? #

Autonomy is often described by how long a model can work without intervention. Enterprises, however, deploy systems made up of a model, tools, permissions, state, and supervision—not a model in isolation.

Anthropic’s study of deployed agent use likewise finds that autonomy is shaped by deployment rather than fixed by the model. The same model can exhibit different autonomy and risk under different tool permissions and oversight [22]. The evidence comes from one model provider and is weighted towards software-engineering work, so it cannot represent every industry. It nevertheless shows why autonomy must be discussed in the context of a particular system.

Shadow Observation, Decision Recommendation, and Controlled Execution #

Enterprises should delegate action authority through four progressive, ascending stages:

• Stage 1: Shadow Mode. The system generates analytical and action traces over real or de-identified data while write tools remain strictly disabled. The team compares its proposed reasoning with existing workflows to identify the exact operational conditions under which the model misjudges state, lacks required context, or chooses suboptimal tools.
• Stage 2: Decision Support. The agent produces diagnostic findings, candidate actions, and evidence rationale, leaving execution entirely to human operators. The enterprise measures acceptance rates, modification frequency, and review latency; if verifying a recommendation takes longer than manual execution, the deployment has not achieved genuine operational leverage.
• Stage 3: Drafting Mode. The agent automatically compiles structured purchase drafts, refund tickets, or operational plans, presenting side-by-side diffs for explicit human sign-off; final submission and execution remain 100% human-triggered, validating the accuracy of structured business mutations.
• Stage 4: Controlled Execution. Within tightly bounded action envelopes, the system autonomously executes low-risk, easily observable, and reversible operations. Runaway execution is mitigated through strict daily budget caps, invocation rate limits, and automated circuit-breakers that halt execution immediately upon state inconsistency.

Expand Scope by Relaxing One Constraint at a Time #

The enterprise may increase scale, widen parameter ranges, extend runtime, or reduce human confirmation, but it should not change all of them at once. Otherwise, a change in results cannot be attributed to the model, tools, data, or expanded authority.

Expansion is justified only when task quality, boundary violations, human corrections, cost, and recovery time meet predefined thresholds. If key metrics deteriorate, the system should return to the previous model, policy, or permission scope.

Before each expansion, the team should answer at least these questions:

• Which business outcome improved relative to the existing process?
• Where do errors occur most often, and what is their blast radius?
• Can the system detect and stop errors before they escalate?
• Can an operation be reversed, and if not, is there a compensating process?
• Is human intervention resolving genuine exceptions or repeatedly repairing system defects?
• Does the value cover model, engineering, monitoring, and incident-response costs?

Different stages of one task can carry different levels of autonomy. A replenishment agent might read data, investigate causes, and prepare an adjustment automatically; pause when the change exceeds a threshold; and write the approved change only within fixed parameters. The enterprise is not surrendering the whole job. It is delegating specific, bounded permissions that can be withdrawn.

The NIST AI Risk Management Framework places governance, context mapping, measurement, and ongoing management across the AI lifecycle, with explicit roles for human-AI oversight [23]. Business owners cannot merely request automation, and technical teams cannot decide alone how much operational risk the organisation will accept. Formal operating procedures must identify who sets the boundary, approves expansion, monitors anomalies, and can stop the system.

Who Owns the Business Objective and Acceptance Criteria? #

The same interview also directly exposed the problem of organisational ownership. Corporate leadership purchased a multi-agent project without defining specific business scenarios or designating a business owner accountable for the operational outcome. The delivery team could not obtain frontline data, oriented the work entirely around demonstrations and slide decks, and never established what would count as acceptance [6]. The primary obstacle was not model reasoning, but the absence of an accountable business owner, empirical baselines, and rigorous acceptance criteria.

An agent can choose the next step within a defined task. It cannot decide for the enterprise why the task exists, who owns the business outcome, or which state counts as completion. Nor can a technical team decide alone which processes are worth changing, which frontline practices may be disrupted, and who bears the cost of failure.

Enterprises must establish three explicit organisational responsibilities:

• Business Owners: Establish Baselines and Single-Point Accountability. Appoint an operational leader accountable for the final business outcome—not merely an AI department or IT liaison. Replace vague commitments to “deliver a number of agents” with specific tasks, baseline comparisons, and verifiable acceptance metrics, while proactively granting the engineering team access to frontline staff, workflows, data, and APIs. Adoption rates, human correction volumes, and residual workload must be integral to acceptance.
• Frontline Operators: Maintain Psychological Contracts and Incentive Alignment. An agent’s adaptive reasoning depends entirely on receiving authentic, comprehensive domain context. If leadership frames deployment around headcount reduction, frontline workers naturally adopt a defensive posture—withholding tacit operational knowledge, providing perfunctory entries, or actively bypassing the system. The agent starves of high-quality data and quickly fails. Enterprises must explicitly position agents as collaborative tools designed to eliminate repetitive administrative drudgery and empower complex judgment, while positively incentivizing staff who actively identify boundary violations and workflow edge cases.
• Forward-Deployed Engineers (FDEs): Enforce Task Decomposition and Phased Validation. Implementation teams must first decompose jobs into discrete tasks, assessing whether rule-based code, deterministic workflows, RAG, or agents provide the lightest, most effective solution. Before writing code, they need confirmed data sources, access permissions, a business owner, and acceptance criteria. Work should begin in shadow mode or bounded pilots, using real-world tasks to measure completion rates, human correction, adoption, and costs, rather than substituting demos, agent counts, or API invocation volumes for business value.

Who Enforces the Action Envelope and Provides Meaningful Oversight? #

Action boundaries and oversight mechanisms must ultimately map to specific organizational roles. Enterprises should delineate four distinct areas of responsibility:

• Business Owners (Define Boundaries and Risk Tolerances): Determine the verifiable business value sought, define which business states the system may modify, and establish organizational risk thresholds; approve the expansion of action authority when empirical metrics meet thresholds, and swiftly contract permissions if anomalies emerge.
• Engineering & Development Teams (Deterministic Controls and Failure Fallbacks): Implement action envelopes within enterprise identity providers, tool interfaces, and state machines; rigorously test system behavior under stale data, API timeouts, duplicate requests, partial completion, prompt injection, privilege escalation, and concurrent writes, ensuring robust controls for pause, privilege reduction, transaction rollback, and disaster recovery.
• Frontline Operators (Context-Rich Exception Approval): Understand precisely on whose behalf the system is acting, which permissions it uses, and what business records it intends to alter. Before executing high-risk writes, the interface must present affected objects, key parameters, state diffs, and reversibility; human operators exercise authentic approval, amendment, or rejection, preventing oversight from degenerating into a mindless “rubber stamp.”
• Domain Audit Experts (Asynchronous Sampling and Ongoing Calibration): Avoid noisy, flow-blocking confirmation dialogues in favor of asynchronous sampling audits, periodically reviewing executed decision traces for high-value or high-risk tasks to evaluate operational quality and recalibrate boundary parameters.

User review does not transfer system responsibility to the user. Approval is meaningful only when the person has enough information, time, authority, and expertise to judge the change. Business and development owners remain responsible for sensible defaults, automated stop controls, and preventing one mistaken click from causing unbounded harm.

Conclusion #

Key Takeaway: Enterprises should not pursue autonomy as an abstract metric. The operational challenge is deciding which decisions are genuinely worth delegating and what evidence justifies expanding that authority. Advancing model capability does not eliminate the action envelope; it makes continuous, evidence-based governance the sole defensible basis for granting, pausing, or revoking the authority to act.

An agent should advance from answering questions to altering business state only when adaptive path selection measurably outperforms deterministic workflows, every action can be observed and audited, and the underlying business can reliably recover from failure.

Even as future models handle longer horizons, invoke broader toolsets, and collaborate across multi-agent meshes, systems will appear increasingly autonomous. Yet whether they belong in procurement, financial ledgers, customer entitlements, or critical infrastructure depends less on how long a model can run unassisted than on whether identity federation, access control, state management, trace evaluation, and rollback mechanisms keep pace. Advancing model capability does not eliminate the action envelope; it makes continuous, evidence-based governance the sole defensible basis for granting, pausing, or revoking the authority to act.

References #

[1] Anthropic. Building Effective Agents. 2024. https://www.anthropic.com/engineering/building-effective-agents

[2] OpenAI. A Practical Guide to Building Agents. 2025. https://openai.com/business/guides-and-resources/a-practical-guide-to-building-ai-agents/

[3] Model Context Protocol. What Is the Model Context Protocol (MCP)? 2026-07-28. https://modelcontextprotocol.io/docs/2026-07-28/getting-started/intro

[4] OpenAI. Orchestration and Handoffs. Accessed 2026-08-28. https://developers.openai.com/api/docs/guides/agents/orchestration

[5] Google. Announcing the Agent2Agent Protocol (A2A). 2025. https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/

[6] AI Nao. “硅谷最火职位在中国：好苦，好痛，正在救火路上” [Silicon Valley’s Hottest Role Comes to China: Painful, Exhausting, and Constantly Fighting Fires]. Accessed 2026-08-29. https://mp.weixin.qq.com/s/aM3AqRksDV-w-1RYwZUeNA

[7] Google. Gemini Developer API Pricing. Accessed 2026-08-28. https://ai.google.dev/gemini-api/docs/pricing

[8] OpenAI. Models. Accessed 2026-08-28. https://developers.openai.com/api/docs/models

[9] Anthropic. Pricing. Accessed 2026-08-28. https://platform.claude.com/docs/en/about-claude/pricing

[10] DeepSeek. Models & Pricing. Accessed 2026-08-28. https://api-docs.deepseek.com/quick_start/pricing/

[11] Anthropic. Effective Context Engineering for AI Agents. 2025-09-29. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

[12] Anthropic. How We Built Our Multi-Agent Research System. 2025-06-13. https://www.anthropic.com/engineering/multi-agent-research-system

[13] Google. Understand and Count Tokens. Accessed 2026-08-29. https://ai.google.dev/gemini-api/docs/tokens

[14] OpenAI. Prompt Caching. Accessed 2026-08-29. https://developers.openai.com/api/docs/guides/prompt-caching

[15] Australian Broadcasting Corporation. Commonwealth Bank Backtracks on AI Job Cuts, Apologises for ‘Error’ as Call Volumes Rise. 2025. https://www.abc.net.au/news/2025-08-21/cba-backtracks-on-ai-job-cuts-as-chatbot-lifts-call-volumes/105679492

[16] Model Context Protocol. Understanding Authorization in MCP. 2026-07-28. https://modelcontextprotocol.io/docs/2026-07-28/tutorials/security/authorization

[17] NIST. Lessons Learned from the Consortium: Tool Use in Agent Systems. 2025. https://www.nist.gov/news-events/news/2025/08/lessons-learned-consortium-tool-use-agent-systems

[18] Anthropic. Trustworthy Agents in Practice. 2026. https://www.anthropic.com/research/trustworthy-agents

[19] U.S. Court of Appeals for the Ninth Circuit. Amazon.com Services, LLC v. Perplexity AI, Inc. 2026. https://cdn.ca9.uscourts.gov/datastore/opinions/2026/08/04/26-1444.pdf

[20] OWASP GenAI Security Project. OWASP Top 10 for Agentic Applications for 2026. 2025. https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/

[21] OpenAI. Evaluate Agent Workflows. Accessed 2026-08-28. https://developers.openai.com/api/docs/guides/agent-evals

[22] Anthropic. Measuring AI Agent Autonomy in Practice. 2026. https://www.anthropic.com/research/measuring-agent-autonomy

[23] NIST. Artificial Intelligence Risk Management Framework (AI RMF 1.0). 2023. https://doi.org/10.6028/NIST.AI.100-1