Do Enterprises Really Need AI Agents? When to Let Models Decide the Next Step

TL;DR

Most enterprise tasks do not need an agent. Delegating the choice of what to do next is worthwhile only when the path cannot be predetermined, each step produces useful feedback, errors can be contained, and the benefit can cover the added cost. Even then, safe deployment does not come from telling the system to “be careful”; it comes from hard rules that put it in a tight harness and define a clear action envelope:

• Keep Permissions in Check: Never give an agent superuser privileges. It may do only what the employee is allowed to do, the current task permits, and the tool interface authorises. Being connected to a system does not mean it may modify data.
• Prevent Disruption: Re-check the latest state before executing a critical operation so the agent does not act on stale information. It must not retry blindly after a network timeout. Every query, decision, and action should leave an auditable record.
• Count True Costs: Measure how much additional value the agent creates or how much loss it prevents compared with the existing process, and include the costs of failures, retries, and human review. Do not count transaction volume as a return or translate a handful of completed tasks directly into headcount cuts.
• Delegate in Small Steps: First let the agent observe without acting, then make recommendations and prepare drafts, and only later allow it to execute within a clear action envelope. Relax one constraint at a time, then use actual results to decide whether to expand, pause, or withdraw its authority.

At nine in the morning, an employee sits down to work and discovers that a system they used yesterday now says “Access denied.”

The same problem can affect an office project platform, a store’s back-office system, a warehouse scanner, or a mobile app used by field staff. The employee usually knows only that access has suddenly stopped. They take a screenshot and submit an IT service ticket titled “Access problem.”

The service desk still has to establish which system is affected, what device the employee is using, where they are signing in from, whether access worked yesterday, and whether their role or line manager has recently changed. A technician may then inspect the identity system, group membership, software licences, device-compliance status, and application logs. The cause might be an expired session, a failed group synchronisation, a device that no longer meets security requirements, or a request for new access that has not yet been approved.

Large organisations handle these requests repeatedly. The final fix is not always the expensive part. Most of the effort goes into gathering information, classifying the problem, checking several systems, and routing the ticket to the right team.

A retrieval-augmented generation (RAG) assistant can retrieve relevant documentation, and a fixed workflow can collect information and open a ticket. An AI agent can instead continue the investigation from each finding and choose the next check dynamically. It might inspect service health, then verify identity, device state, and existing access before guiding reauthentication, running an approved diagnostic, or preparing a restoration request. If the employee wants new access, the system should stop and refer the request to the line manager or data owner.

The agent must not add the employee to a new group, disable a security control, or copy another person’s permissions merely because the employee says, “It worked yesterday.” Enterprise systems must still verify identity, prior approvals, and the authority granted for the current task.

The earlier article “Is Enterprise RAG Worth Implementing: From Scenario Selection to Ongoing Governance” discussed how enterprises can let models use internal knowledge: RAG mainly determines what a system can know, a fixed workflow follows a predefined route, and an agent chooses the next step from changing state. The employee may see only a faster IT assistant. The enterprise, however, has delegated authority to choose the next action.

The central question is therefore not whether a model can invoke tools. It is whether a task benefits enough from feedback-driven path selection to justify delegating that choice to a model—and, if it does, where the boundaries of its execution space must lie.

1. Agents Change Who Chooses the Next Step #

Software has long been able to execute multistep operations across systems, from processing orders and updating delivery status to restarting a failed service. Such systems are not necessarily agents in the sense used here.

Anthropic distinguishes workflows, in which models and tools follow predefined code paths, from agents, in which a model dynamically directs the process and tool use [1]. OpenAI similarly describes an agent as using a model to manage execution, select tools according to the current state, and decide when the task is complete [2].

Both definitions point to an easily overlooked change: decisions that engineers once encoded in advance may now be made by a model at runtime.

A fixed workflow can be represented as:

Input → Select branch based on predefined conditions → Execute step → Output

An agent is closer to a feedback loop:

Observe current state → Select next step → Invoke tool to query or change state
        ↑                         ↓
        └────── Read result and re-evaluate ──────┘

The access failure will serve as the running scenario: an agent chooses each next check from service, identity, device, licensing, and approval findings. It may run approved diagnostics or prepare a restoration request, but deterministic systems and authorised people must still decide whether to grant new access.

That flexibility creates both the value and the risk. A fixed program can fail because of its rules, code, or input data. An agent can also reinterpret the goal, select different evidence, and choose a new action on every iteration. Its judgment does not remain confined to an answer: it can alter the state on which subsequent decisions depend.

Replenishment provides a higher-risk contrast. If a system mistakes a one-off group purchase for sustained consumer demand and places an additional order, the inventory and in-transit quantities read in the next iteration have already changed. Each step can appear reasonable even as the sequence drifts away from the original objective.

Tool use alone does not make a task suitable for an agent architecture. The real test is whether feedback requires the path to be reconsidered at runtime, and whether the enterprise is willing to let a model make that choice.

Tools and Protocols Provide Connectivity, Not Authority #

A tool is an interface for a query, calculation, or business operation, such as checking service health, reading a device-compliance result, or creating an access-restoration request. The agent may decide whether to call it, but the tool neither understands the overall objective nor chooses the next step in the task.

Current connectivity technologies address distinct dimensions of integration:

• Model Context Protocol (MCP): Standardises how AI applications discover, read, and invoke local or remote resources and tools [3];
• Handoff Mechanisms: Enable a primary agent to transfer task execution and conversation state dynamically to a domain-specialist agent upon intent classification [4];
• Agent2Agent (A2A) Protocols: Standardise dynamic capability discovery, message routing, and collaboration across heterogeneous multi-agent networks [5].

Key Takeaway: Connectivity protocols determine what can be called, not what is authorised to execute, and they cannot substitute for rigorous architectural justification.

2. Which Tasks Justify Model-Directed Path Selection? #

Agents fit a narrower class of work: the objective is reasonably clear, the steps cannot be exhaustively specified in advance, and the system can adapt from timely feedback. Complexity alone does not justify one. A rules engine is usually more reliable for tax calculations whose conditions can be encoded, while a workflow platform can handle cross-department approvals when the sequence, permissions, and exceptions are known. Adding an LLM to fixed steps still produces an LLM-assisted workflow, not a reason to let the model control an adaptive loop.

Four conditions help determine whether model-directed path selection is worthwhile.

Is the Task Path Genuinely Difficult to Predetermine? #

Agents are better suited to tasks with varied inputs, dispersed information, and many combinations of exceptions. The same “access denied” report might be caused by an outage, a change in employment status, a non-compliant device, an expired licence, or failed group synchronisation. Each finding shapes the next step, making a complete decision tree difficult to write in advance.

If the task has only a few stable branches—such as resetting a password through a fixed process after identity has been verified—an agent merely replaces logic that could be coded and thoroughly tested with a probabilistic decision.

Does Each Step Yield Actionable Feedback? #

An agent needs to observe what happened after it acted. An access-support agent can reread identity, device, licence, and group state. A coding agent can run tests. A service agent can confirm that a ticket was created.

Without timely feedback, the system cannot correct itself reliably. It may continue from stale state, apparently completing a multistep task without knowing whether the first operation succeeded.

Can the Impact of an Error Be Reliably Contained? #

Early deployments will usually encounter cases that testing did not cover. An error that produces only an investigation report or draft access-restoration request is relatively cheap to detect and correct. An error that grants a new permission, orders stock for every store, or initiates an external payment leaves little room for experimentation.

Enterprises should therefore begin where errors have limited impact, or where underlying systems can reverse an operation or compensate for its effects. The task with the largest theoretical value is not necessarily the right first deployment if it also has the most severe failure modes.

Can the Expected Benefit Justify the Added Cost? #

Agents add model calls, state management, evaluation, monitoring, and exception handling. A complex autonomous system is unlikely to pay off when a person can confirm a recommendation in two minutes. The case is stronger when employees repeatedly gather information across systems, decide what to check next, and wait for feedback. Even when the first three conditions hold, the enterprise still has to show that the benefits of adaptive path selection can cover these additional costs. This is only an initial screen for scenarios with no plausible room for benefit; it cannot replace an estimate of the project’s actual costs and business outcomes.

Combining path variability, feedback quality, error impact, and net value produces a simple framework for choosing an approach:

Anthropic recommends starting with the simplest approach that solves the problem: agents trade higher cost and latency for flexibility, so greater autonomy is unnecessary when retrieval, one model call, or a fixed workflow already works [1].

A first-person interview with a Chinese forward-deployed engineer (FDE) describes a similar case. An education group wanted to use AI across the full workflow for school research and report generation. A professional team spent three months building four agents, but the system required school leaders to change how they recorded, entered, and uploaded information. Incomplete inputs led to inconsistent output, and few people used the system. A school principal who had attended the training instead kept the existing manual process and used a simpler tool only to generate report templates; other principals found that approach easier to adopt [6].

The account is an anecdote, not a controlled study, and it does not show that simple systems are always better than agents. It does show that a “full-workflow” design may merely transfer operating cost to users if it first requires frontline staff to rebuild their working habits. Enterprises should identify the smallest effective point of AI intervention before deciding whether a model needs to control the whole task path.

3. Can the Benefits of an Agent Project Justify Its Additional Cost? #

A task can be technically suitable for an agent without the project being economically worthwhile. Without execution traces, an enterprise can begin with a single-model budget and token-amplification stress test. Once traces are available, it should calculate model cost by call category and include tools, runtime, and human review in the total cost of each successful task.

Token Cost Is Only a Budget Starting Point #

Start With a Reference Budget for One Model #

A simple formula can establish a reference monthly token budget for one model and billing tier. The easiest variable to underestimate is \(A\): one business task submitted by a user does not imply only one model invocation. Let \(N\) be the number of user-submitted tasks per month and \(M\) the number of billable model invocations per month:

The symbols are defined as follows:

\(U\), \(D\), and \(Q\) should come from actual task volume. \(A\) and token counts should be estimated from prototypes or execution traces; prices depend on the model, billing tier, caching, and region. Until observed data is available, enterprises should use multiple scenarios.

Teams can calculate \(A\) from observed usage:

The numerator should include calls made inside a hosted agent service. Separately priced database queries, MCP tools, and business APIs need their own cost lines. Under this definition, average \(A\) may be below 1 if some submitted tasks are rejected, combined, or completed by deterministic systems before a model is called. This article therefore uses \(A=1\) as a single-call reference scenario, not a universal theoretical lower bound. It is a lower bound only when the denominator is restricted to tasks that trigger at least one model call.

The following is a capacity-planning example, not typical enterprise usage. Assume 500 users, 20 active days per month, and 10 business tasks per user per day:

To show how prices enter the formula, start with \(A=1\) as a single-call reference scenario. As of August 30, 2026, the paid Standard tier for Gemini 3.7 Flash Standard charged $0.75 per million input tokens and $3.75 per million output tokens, including thinking tokens [7]. Assume 8,000 input tokens and 2,000 billable output tokens per call. These figures are for capacity planning, not an industry average.

Holding task volume constant, assuming that every call uses 8,000 input tokens and 2,000 output tokens, and varying only \(A\) and model list prices produces the following price-sensitivity comparison. As of August 30, 2026, the results are:

The DeepSeek row spans its published off-peak and peak rates; the other rows use published standard or base prices. The table excludes caching, batch processing, search, paid tools, and storage. Tokenisers, reasoning-token treatment, caching, and task quality differ across models, so this is a list-price comparison under common token assumptions, not a performance or value ranking.

Calculate Multi-Agent Systems by Call Category #

A single-model average is sufficient for an initial estimate. If the lead agent, subagents, router, and reviewer use different models or billing tiers, costs should instead be separated by call category. A subagent may explore independently and return only a compressed result to the lead agent, changing both call count and context length [11]. Let \(j\) denote a call role or model category. The detailed formula is:

Here, \(A_j\) is the average number of calls to category \(j\) per business task; the other variables carrying a \(j\) subscript use that category’s average token counts and prices. No additional multi-agent multiplier is needed when all lead-agent, subagent, routing, review, and retry calls are included in the relevant \(A_j\). An agent-to-agent message is not necessarily a model charge: it enters \(A_j\) when it triggers another billable model call, or \(T_{\mathrm{in},j}\) when its content becomes context for a later call.

Use Stress Tests Only When Execution Traces Are Unavailable #

Before execution traces are available, an enterprise can separately track a dimensionless overall token amplification factor \(F_{\mathrm{token}}\) against a comparable chat task for stress testing. Let \(A_0\), \(T_{\mathrm{in},0}\), and \(T_{\mathrm{out},0}\) be the baseline chat task’s average call count and average input and output tokens per call. Under a common token-accounting basis:

If the baseline chat task makes one call, \(A_0=1\), then \(F_{\mathrm{token}}\) depends on both the agent system’s call count and its tokens per call. Only when every agent call category averages the same number of tokens per call as the baseline does \(F_{\mathrm{token}}\) approximate the total average call count per task, \(\sum_j A_j\); with one call category, this further reduces to \(A\). More generally, under that equal-token condition, \(F_{\mathrm{token}}\approx\sum_j A_j/A_0\). Raw token counts should not be added or compared across incompatible tokenisers or billing-token definitions.

In Anthropic’s Research system, single agents typically consumed about four times as many tokens as chat interactions, while multi-agent systems used about fifteen times as many [12]. This is a whole-task token ratio, not the number of model calls per task \(A\); system instructions, tool definitions, history, and tool results also enter context [13][14]. The fourfold and fifteenfold observations are therefore reference stress scenarios for one system, not values to substitute directly as \(A=4\) or \(A=15\).

Cost estimation therefore has two methods that must not be stacked:

• Bottom-up estimate: Once prototypes or execution traces exist, use the detailed \(A_j\), \(T_{\mathrm{in},j}\), \(T_{\mathrm{out},j}\), \(P_{\mathrm{in},j}\), and \(P_{\mathrm{out},j}\) formula to calculate each call category separately.
• Top-down stress test: Before traces exist, and only when the agent and baseline use the same model, have a similar input-output mix, and perform comparable tasks under the same acceptance criteria, approximate:

Here, \(C_{\mathrm{baseline}}\) is the comparable chat baseline’s token cost and \(C_{\mathrm{scenario}}\) is the stress-scenario token cost. Using the Gemini single-call result of USD 1,350 as the baseline, \(F_{\mathrm{token}}=4\) and \(15\) produce approximately USD 5,400 and USD 20,250. These are budget stress tests, not cost forecasts for agent systems in general. Once the detailed formula is used, multiplying by \(F_{\mathrm{token}}\) would double-count the amplification.

The formula above is only a token-budget starting point; full operating cost also includes paid tools, hosting, orchestration, vector databases, monitoring, human review, and incident response.

Use Successful Tasks and the Break-Even Point to Judge Value #

Define Successful Tasks and Attributable Value First #

Once the system design is known, teams should replace the budget assumptions with operating data. The remaining symbols use the same measurement period; in the scenario below, that period is one month:

\(C_{\mathrm{total}}\) includes models, tools, runtime, allocated infrastructure, and human review; \(C_{\mathrm{token}}\) is only one component. If the agent investigates before a person independently completes the task, the result does not count in \(S\), but the model, tool, and escalation costs remain in \(C_{\mathrm{total}}\). Cost per successful task is:

In the access-failure scenario, closing a ticket does not make it successful. The employee must regain valid prior access, or a new-access request must reach the correct approver with the right evidence, without reopening because it was handled incorrectly. Using \(S\) as the denominator makes abandoned runs, ineffective retries, and pre-escalation investigation part of the calculation.

The total attributable value over the same measurement period is:

Return on investment (ROI) is:

Understanding \(V_{\mathrm{total}}\) requires distinguishing gross transaction volume from attributable incremental value. An agent handling a USD 100 transaction did not create USD 100 of value; only baseline-relative avoided loss, realised labour savings, or additional margin is attributable. For access support, this might include shorter employee downtime, less IT investigation, and fewer repeat tickets. Theoretical time savings count only when they release capacity, reduce expenditure, or avoid measurable loss.

Test Return Assumptions With the Break-Even Point #

Assume that an enterprise submits 100,000 tasks per month, incurs USD 10,000 in total monthly agent-system cost, and receives USD 0.50 in attributable incremental value from each successful task. To isolate the effect of successful-task volume, the example holds task volume, total cost \(C_{\mathrm{total}}\), and value per successful task \(v\) constant. These figures are a calculation scenario, not an industry average. The number of successful tasks required to break even is:

Under these assumptions, at least 20,000 of the 100,000 monthly submissions must meet the acceptance criteria and generate incremental value to break even:

\(\mathrm{ROI}=0\%\) means that value exactly covers cost; a negative value means the expense was not recovered. The table is not a maturity forecast: more successful tasks may raise tool, monitoring, review, and incident-response costs, while a changing task mix may alter \(v\). Enterprises must keep the variables updated with operating data.

This is only operating return over the measurement period. A full-project assessment also needs upfront development and process-change costs and a payback period.

Automated Throughput Does Not Equal Job Displacement #

Automated throughput cannot be translated directly into displaced jobs. After Commonwealth Bank of Australia introduced an AI voice bot and announced the removal of 45 customer-service roles, a union challenge led the bank to acknowledge that the roles were not redundant, reverse the decision, and apologise. The union said human workload had increased [15]. The system was not an LLM agent under this article’s definition, but it shows why repeat contacts, human transfers, difficult-case resolution time, and residual workload still matter.

In the same FDE interview, a home-improvement company sought to reduce a workforce of more than 6,000 to 3,000, and a team delivered seven agents in three months. The interviewee said most improved only parts of tasks and only one sales agent might increase revenue, so they did not map directly to the headcount target [6]. This is one implementer’s account, but it makes the same point: jobs combine information processing, communication, exception handling, and accountability. Enterprises should measure defined tasks rather than infer that completing one component removes the whole job.

The enterprise therefore needs to show not how many models an agent called or jobs it might replace, but whether it can deliver more accepted outcomes at an acceptable total cost. Even then, tools, data, parameters, and state changes still need an explicit boundary.

4. Enterprises Need to Design the Agent’s Action Envelope #

Traditional access control asks who may use which system. An agent also needs an action envelope: the enterprise need not prescribe every step, but it must define which actions are permitted for the current objective, state, and period, and within what parameter ranges.

Deterministic Systems Must Enforce the Action Envelope #

Actions in the access scenario fall into three classes:

• Diagnostic actions: Query service, identity, device, licence, and approval state without changing access.
• Restoration actions: Return the system to a state supported by an existing approval, such as revalidating a session.
• New authorisation actions: Add group membership, privileged access, or cross-department access through a formal approval process.

Across these classes, an access-support agent’s action envelope might include:

An action envelope is not a prompt that says “act carefully.” A prompt still asks the model to interpret the restriction. The envelope must be enforced by tool interfaces, identity systems, rules engines, and business state machines.

The model and deterministic controls have different responsibilities:

Key Takeaway: A model can propose what to do next, but deterministic code must decide whether that action is permitted to take effect now.

A model may select a diagnostic path and propose restoration; restoration still requires checks against identity, prior approval, and task scope, while new authorisation may only be documented and routed for approval.

Effective Authority Is the Intersection of User, Agent, Task, and Tool Constraints #

Enterprises do not generally intend to expose a shared superuser account to every employee. The more realistic risk appears when a validation environment moves towards production: a team temporarily uses an overprivileged account for a proof of concept (PoC) or integration test, but incomplete environment separation, a release checklist that omits privilege reduction, or missing ownership, expiry, and handover requirements allow the test configuration to survive into production. As discussed in Between Relationships and Rules: How to Get Things Done in Imperfect Organizations, temporary arrangements need to become transferable organisational capability. In an agent project, account purpose, ownership, expiry, environment separation, privilege reduction, and removal should therefore become verifiable parts of release and handover.

Once such an account reaches production, it can bypass source-system row-, column-, department-, and tenant-level access controls. An employee permitted to view inventory for one region might query national data through the agent; someone allowed to read supplier records might trigger an order operation they could not perform directly.

Each task should instead record the initiating user, the agent performing the task, and the authority granted for that task. Effective permission is the intersection of several boundaries:

Effective authority = User permissions ∩ Agent permissions ∩ Task delegation ∩ Tool policy

User permissions establish what the initiator is authorized to view; agent permissions cap the baseline capabilities of the system; task delegation binds the session to a specific purpose and lifespan; and tool policies validate parameters, safety thresholds, and state transition rules at execution time. Failure at any layer must immediately halt execution—it cannot be bypassed by rephrasing a prompt, switching tools, or delegating the task.

The system needs to implement the following types of controls separately:

The data source, MCP server, or business API must enforce permission checks, and a model cannot establish identity from a user’s own description. MCP authorisation requires a protected server to validate the access token, its intended resource, and its scopes [16]. Enterprise deployments should also verify the user, the acting agent, and the delegation between them, then apply least privilege at the resource and tool layers. Hiding a tool in the client reduces its visibility to the model; it does not create a security boundary.

Data minimisation still applies. Investigating one employee’s access to a named system should not expose other employees’ permissions, full personnel files, or unrelated business data.

Which Actions Require Step-Up Authorisation or Human Approval? #

Human Approval Should Be Triggered Before a Boundary Is Crossed #

Data access is only one part of the envelope. Once an agent can call write tools, the enterprise must consider whether an operation has lasting effects, whether it can be reversed, and how far the consequences can spread.

A NIST summary of an expert workshop on agent tools identifies dimensions including read or write access, lasting effects, reversibility, observability, and autonomy [17]. A tool name therefore says little about risk. An “update order” operation restricted to one draft is a very different capability from one that can alter confirmed orders in bulk.

The boundary must also bind the current task. Even if a system can create access-restoration requests, it may use that capability only within this task’s objective, initiator, applicable data, permitted tools, and expiry; every action must remain within the original authority.

This determines where human approval belongs. Low-risk reads need not be approved one by one, but a system can require confirmation before crossing an action boundary. An approver should authorise a concrete change—“restore employee A to group X under approval record Y until this date”—not a vague instruction to “fix the access.”

Anthropic has described the trade-off between approving every operation and reviewing a plan before execution. A user can approve a bounded plan while retaining the ability to intervene, avoiding a stream of context-poor confirmation prompts [18].

Human approval should not cover every step. If routine queries and minor adjustments repeatedly trigger confirmation prompts, frontline staff may develop supervision fatigue and stop evaluating them carefully. Enterprises can distinguish actions according to whether they cross an established boundary:

This design preserves human attention for changes that require genuine judgement and prevents approval from becoming a rubber stamp.

User Consent Does Not Equal Platform Authorisation #

Authority to act on a user’s behalf depends on more than the user’s consent. Perplexity’s Comet agent can sign in to user accounts, compare products, and place orders. In 2025, Amazon sued Perplexity, alleging that the agent concealed its automated identity, accessed private customer accounts, and created risks to data and the shopping experience. Perplexity denied the allegations and argued that Amazon was restricting user choice. In March 2026, a district court issued a preliminary injunction; in August, the Ninth Circuit vacated the injunction and remanded the case, holding that Amazon was unlikely, on the existing record, to show that Perplexity had “accessed” Amazon’s computers within the meaning of the relevant law [19].

The case remains unresolved on the merits, but it already shows that delegated authority is not one-sided. A user’s permission for an agent to shop does not mean the retailer has authorised that agent to act for the user. Allowing a user to sign in does not necessarily authorise a third-party system to access the account and place orders automatically. Enterprises need to verify the user, the agent, and the delegation between them, and establish who accepts the transaction terms and who bears responsibility for mistaken purchases or data access.

Identity Recovery Must Rely on Deterministic Verification #

Real incidents at IT service desks also show why restoring access is more than answering a question correctly. In 2025, Clorox alleged in a lawsuit that Cognizant’s service desk reset credentials without performing the agreed caller verification, allowing attackers to enter its network; Reuters reported that Clorox sought approximately USD 380 million. Cognizant disputed responsibility and said its scope was limited to help-desk services rather than cybersecurity [20][21].

This was not an AI-agent incident, and allegations in a complaint are not adjudicated facts. The relevant boundary lesson is narrower: credential recovery must rely on identity verification from a trusted, deterministic source, not on the caller’s own account or a model’s judgment that someone “looks legitimate.”

Audit Trails Must Reconstruct Authority, Actions, and State Transitions #

A chat transcript is not an agent audit trail. It may show what the user asked, but not which records the system retrieved, whose authority a tool call used, who assumed responsibility after a handoff, or how business state changed.

An enterprise-grade, reviewable audit chain must capture three causal dimensions:

• Identity and Delegation Provenance: Identify who initiated the task, under whose authority the system acted, and which credentials were stripped when delegating to sub-agents;
• Execution and Approval Traces: Record precisely what data ranges the agent accessed, which tools and parameters it invoked, and which human operator approved the state change;
• State Transitions and Exception Logs: Reconstruct business data changes before and after execution, intermediate failures and retries, and rollback or compensation logs triggered upon anomaly.

High-risk writes also require tamper-resistant, immutable audit logs. Auditors should be able to trace a macro business state change back to an originating task and reconstruct every micro state transition within that execution.

Audit logs may themselves contain prompts, customer data, access tokens, or commercially sensitive information. Enterprises should redact sensitive fields, control access, prevent tampering, and set retention periods. Audit and security teams need the evidence required for review, not unrestricted access to every raw business record. The purpose is to establish accountability, not to create a second, higher-risk data warehouse.

An action envelope limits what a model can do; it does not make every permitted step correct. A different class of risk emerges when individually compliant actions interact inside a loop.

5. Why Agent Errors Propagate Beyond a Single Output #

When no tool call or business write follows, an error in a question-answering system usually does not directly change external system state. An agent’s output, by contrast, can become a tool input, and the tool can change business state. That state then becomes evidence for the next decision, allowing the original error to propagate.

A diagnostic error may affect only a recommendation, while an executed restoration changes identity and access state. The next iteration must distinguish changes caused by the agent, administrators, and source systems, as well as operations still in progress. Duplicate replenishment orders create more immediate financial harm, but the amplification mechanism is the same.

Several mechanisms make this amplification more likely.

Why State and Execution Fall Out of Sync #

The earlier check is no longer current. After an agent reads identity and approval records, an employee’s role, device-compliance state, or access approval may change before execution. A condition that was true when checked may no longer be true when used. This time-of-check to time-of-use (TOCTOU) problem is not unique to agents, but longer tasks with more steps create more opportunities for it.

Permissions and business state cannot be validated only at the beginning. Before an important write, the system must reread critical fields and confirm that they still match the plan.

A timeout retry repeats the operation. If a tool call times out, the agent may not know whether an access-restoration request or session reset took effect. Retrying simply because no success response arrived can create duplicate requests or even invalidate credentials that were just issued.

Write operations therefore need idempotency: repeated delivery of the same business request must take effect only once. Without an idempotency key and explicit state, an automatic retry converts a communication failure into a business failure.

How Goals and Information Drift Inside the Loop #

A local metric replaces the overall objective. If an enterprise asks an access agent only to close tickets faster, the system may declare success too early or restore overly broad permissions to reduce handling time. Closure speed improves while repeat tickets and unauthorised-access risk rise. The model may violate no individual rule while repeatedly optimising an incomplete objective.

Adding “also consider cost” to a prompt does not solve the problem. Competing metrics, hard limits, and stop conditions need explicit enforcement, and the enterprise must watch for ways the agent improves the visible metric at the expense of the real objective.

Downstream systems treat model inferences as facts. Later agents may read an earlier agent’s statement that “identity was verified” or “the manager approved” as fact. Unless the system distinguishes source identity records, human-confirmed results, and model-generated content, an early inference can gradually become accepted evidence.

Generated content therefore needs provenance and confirmation status. An unverified inference should not be written into a field that downstream systems treat as authoritative.

Why Multi-Agent Workflows Add Coordination Failures #

A single action loop can amplify an error; multiple agents also create shared-state and ownership problems. Identity, device, licensing, and application subagents may reach conflicting conclusions from snapshots taken at different times, or create duplicate restoration requests. One agent may write a hypothesis into a summary that another treats as verified. Without an explicit owner, a task can circulate between agents, be investigated twice, or be handed off without anyone completing it.

Recording who called whom is not enough. Every delegation needs a task ID, state version, current owner, and completion condition, with transactions and ordered execution to resolve parallel-write conflicts.

Evaluation Must Inspect Action Traces #

The OWASP Top 10 for Agentic Applications 2026 includes memory and context poisoning, tool misuse, insecure inter-agent communication, and cascading failures. Its mitigations emphasise least privilege, action-level validation, resource limits, and continuous monitoring [22]. Because agent errors compound across iterative tool invocations and state changes, evaluation cannot rely on whether the final text response appears coherent; it must audit the complete execution trace [23].

Drawing on these risks and the earlier discussion of cost and action envelopes, enterprises can organise trace evaluation into four groups of checks:

• Environment and versions: Capture the foundation model, system prompts, tool schemas, permission policies, and evaluation dataset versions, alongside model-call counts, token consumption (input, output, and cached), and billable API usage so that quality regressions and cost anomalies remain reproducible.
• Decision and tool traces: Check whether the model selected an appropriate tool, entered a loop, retried without effect, produced conflicting results, or reasoned over stale state.
• Deterministic boundaries: Use code to detect privilege escalation, boundary violations, duplicate writes, hazardous parameters, and anomalous tool combinations in the execution trace.
• Business outcomes and cost: Connect technical completion rates to human corrections, residual workload, recovery time, repeat-ticket rates, unauthorised access, and employee downtime so that a higher completion rate does not conceal higher operating cost.

These metrics become evidence for expanding authority only when they feed continuous evaluation. Teams can build test sets from real tasks, past failures, and edge cases; record a baseline for the manual or existing automated process; and test the agent where it cannot change production state.

Deterministic checks suit permissions, parameters, formats, duplicate writes, and state transitions. Tool choice and task path require trace review. People, or evaluation models calibrated against people, can assess open-ended recommendations. High-risk outcomes still require professional review.

After each round, the team should attribute failures to the model, data, tools, or process before changing prompts, adding data, tightening permissions, or revising the workflow. New production failures should enter the regression set.

6. How Should Enterprises Develop and Deploy Agent Applications in Stages? #

Autonomy is often described by how long a model can work without intervention. Enterprises, however, deploy systems made up of a model, tools, permissions, state, and supervision—not a model in isolation. Here, “develop and deploy” means more than completing code and launching it: the system moves from tests that cannot change business state towards controlled execution, with deployment paused or rolled back when evidence is insufficient.

Anthropic’s study of deployed agent use likewise finds that autonomy is shaped by deployment rather than fixed by the model. The same model can exhibit different autonomy and risk under different tool permissions and oversight [24]. The evidence comes from one model provider and is weighted towards software-engineering work, so it cannot represent every industry. It nevertheless shows why autonomy must be discussed in the context of a particular system.

From Shadow Mode to Controlled Execution #

Enterprises should not launch an agent application in one step. They should progress through four stages:

• Stage 1: Shadow Mode. The system generates traces over historical or current de-identified access tickets with all write tools disabled; the team compares them with the existing process.
• Stage 2: Decision Support. The agent performs diagnosis and presents evidence; IT staff decide and execute. If review takes longer than manual investigation, the system has not produced genuine leverage.
• Stage 3: Drafting Mode. The agent prepares a structured request and change diff for restoration or new authorisation; submission and approval remain human-triggered.
• Stage 4: Controlled Execution. The system may execute only low-risk, reversible restoration actions supported by valid approval. New authorisation actions still go to a person.

Microsoft’s enterprise IT agent pattern lists password resets, access provisioning, and device troubleshooting as common service-desk tasks. It also recommends human approval for sensitive actions such as granting access, together with escalation and human-handoff paths [25]. This preserves the distinction at the centre of the staged model: an agent may investigate dynamically, but finding a likely cause does not confer authority to change a permission.

A positive case is most useful here when people still retain the decision. In a customer story published by ServiceNow, EY says its service desk handles more than one million tickets a year and uses generative AI to draft resolution notes for human approval. The case reports about five minutes saved per ticket and 103,000 notes generated in one month, 70% of which were accepted without edits [26]. These are vendor-published, customer-reported results from one organisation, not an industry average. They nevertheless show that an enterprise can first use generative AI to prepare reviewable drafts and test measurable value without delegating final action authority, then decide whether dynamic agent investigation or execution is warranted.

What Must Be Checked Before Expanding Authority? #

The enterprise may increase scale, widen parameter ranges, extend runtime, or reduce human confirmation, but it should manage these as separate deployment variables. Expansion is justified only when task quality, boundary violations, human corrections, cost, and recovery time meet predefined thresholds; if key metrics deteriorate, the system should return to the previous model, policy, or permission scope.

The enterprise can consolidate these checks into a decision table for expanding, maintaining, or withdrawing authority:

Each enterprise should set thresholds and observation periods according to its business risk; this article does not prescribe universal numbers. Expanding only one variable at a time remains necessary to attribute changes to the model, tools, data, or authority adjustment.

Diagnostic, restoration, and new authorisation actions within the same task can carry different levels of autonomy. The enterprise is not surrendering the whole job; it is delegating specific, bounded permissions that can be withdrawn.

The NIST AI Risk Management Framework places governance, context mapping, measurement, and ongoing management across the AI lifecycle, with explicit roles for human-AI oversight [27]. Business owners cannot merely request automation, and technical teams cannot decide alone how much operational risk the organisation will accept; formal procedures must identify who sets boundaries, approves expansion, monitors anomalies, and can stop the system.

Who Defines, Executes, Oversees, and Owns the Outcome? #

The same interview also directly exposed the problem of organisational ownership. Corporate leadership purchased a multi-agent project without defining specific business scenarios or designating a business owner accountable for the operational outcome. The delivery team could not obtain frontline data, oriented the work entirely around demonstrations and slide decks, and never established what would count as acceptance [6]. The primary obstacle was not model reasoning, but the absence of an accountable business owner, empirical baselines, and rigorous acceptance criteria.

An agent can choose the next step within a defined task. It cannot decide why the task exists, which state counts as completion, or who owns the outcome. Nor can a technical team decide alone which processes are worth changing or which frontline practices may be disrupted. Action boundaries and oversight must map to specific roles rather than a vague statement that “business and technology share responsibility.”

Responsibility also needs to align with project incentives. If management presents an agent primarily as a headcount-reduction programme, frontline workers may withhold unstructured context, keep informal knowledge to themselves, or bypass the system because they fear replacement. The agent then loses the high-quality input on which adaptive reasoning depends. Enterprises are better served by presenting it as a collaborative tool that removes repetitive investigation and supports complex judgement, while rewarding people who identify boundary defects and anomalous traces.

User review does not transfer system responsibility. Approval is meaningful only when the person has enough information, time, authority, and expertise; business and development owners remain responsible for sensible defaults, automated stop controls, and preventing one mistaken click from causing systemic harm.

Conclusion #

Key Takeaway: Enterprises should not pursue autonomy as an abstract metric. They must decide which action authority is worth delegating and what evidence justifies expanding, pausing, or withdrawing it.

Bringing LLMs and agents into an enterprise is not equivalent to adding a chat interface to existing software or simply deploying another technology. It is a system-level upgrade. Business processes, identity and access controls, data flows, state management, evaluation and recovery mechanisms, and the division of responsibility among business, technology, and management must all change with it.

Amid the market rush, many companies are racing to release agent products for enterprises. A demonstration that calls tools and executes a sequence of tasks may appear ready for deployment. Once it enters production, however, the enterprise encounters system integration, permission boundaries, exception recovery, and organisational coordination—challenges that are often underestimated.

What determines whether an agent can be deployed, therefore, is not how much capability it can demonstrate. It is whether adaptive path selection creates more value than a deterministic workflow, whether the enterprise can observe and verify each action, and whether business state can be restored after failure. Only under those conditions should an agent move from answering questions to altering business state.

Future models will handle longer-running tasks, invoke more tools, and collaborate more frequently with other agents, making the surrounding systems appear increasingly autonomous. Whether those systems belong in procurement, finance, customer entitlements, or critical infrastructure does not depend on how long a model can run unassisted. It depends on whether identity verification, access control, state management, evaluation, and recovery mechanisms advance with them. The stronger the model, the more an enterprise needs external constraints that turn its capabilities into business actions that are controlled, accountable, and recoverable.

References #

[1] Anthropic. Building Effective Agents. 2024. https://www.anthropic.com/engineering/building-effective-agents

[2] OpenAI. A Practical Guide to Building Agents. 2025. https://openai.com/business/guides-and-resources/a-practical-guide-to-building-ai-agents/

[3] Model Context Protocol. What Is the Model Context Protocol (MCP)? 2026-07-28. https://modelcontextprotocol.io/docs/2026-07-28/getting-started/intro

[4] OpenAI. Orchestration and Handoffs. Accessed 2026-08-28. https://developers.openai.com/api/docs/guides/agents/orchestration

[5] Google. Announcing the Agent2Agent Protocol (A2A). 2025. https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/

[6] AI Nao. “硅谷最火职位在中国：好苦，好痛，正在救火路上” [Silicon Valley’s Hottest Role Comes to China: Painful, Exhausting, and Constantly Fighting Fires]. Accessed 2026-08-29. https://mp.weixin.qq.com/s/aM3AqRksDV-w-1RYwZUeNA

[7] Google. Gemini Developer API Pricing. Accessed 2026-08-30. https://ai.google.dev/gemini-api/docs/pricing

[8] OpenAI. GPT-5.6 Terra Model. Accessed 2026-08-30. https://developers.openai.com/api/docs/models/gpt-5.6-terra

[9] Anthropic. Pricing. Accessed 2026-08-30. https://platform.claude.com/docs/en/about-claude/pricing

[10] DeepSeek. Models & Pricing. Accessed 2026-08-30. https://api-docs.deepseek.com/quick_start/pricing/

[11] Anthropic. Effective Context Engineering for AI Agents. 2025-09-29. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

[12] Anthropic. How We Built Our Multi-Agent Research System. 2025-06-13. https://www.anthropic.com/engineering/multi-agent-research-system

[13] Google. Understand and Count Tokens. Accessed 2026-08-29. https://ai.google.dev/gemini-api/docs/tokens

[14] OpenAI. Prompt Caching. Accessed 2026-08-29. https://developers.openai.com/api/docs/guides/prompt-caching

[15] Australian Broadcasting Corporation. Commonwealth Bank Backtracks on AI Job Cuts, Apologises for ‘Error’ as Call Volumes Rise. 2025. https://www.abc.net.au/news/2025-08-21/cba-backtracks-on-ai-job-cuts-as-chatbot-lifts-call-volumes/105679492

[16] Model Context Protocol. Understanding Authorization in MCP. 2026-07-28. https://modelcontextprotocol.io/docs/2026-07-28/tutorials/security/authorization

[17] NIST. Lessons Learned from the Consortium: Tool Use in Agent Systems. 2025. https://www.nist.gov/news-events/news/2025/08/lessons-learned-consortium-tool-use-agent-systems

[18] Anthropic. Trustworthy Agents in Practice. 2026. https://www.anthropic.com/research/trustworthy-agents

[19] U.S. Court of Appeals for the Ninth Circuit. Amazon.com Services, LLC v. Perplexity AI, Inc. 2026. https://cdn.ca9.uscourts.gov/datastore/opinions/2026/08/04/26-1444.pdf

[20] Reuters. Clorox Accuses IT Provider Cognizant of Enabling Devastating 2023 Cyberattack. 2025. https://www.investing.com/news/stock-market-news/lawsuit-says-clorox-hackers-got-passwords-simply-by-asking-4146778

[21] The Clorox Company and Clorox Services Company v. Cognizant TriZetto Software Group, Inc. Complaint. 2025. https://storage.ghost.io/c/6b/16/6b16ac9c-cd67-432f-b0f3-bbec941084ff/content/files/2025/07/07-22-redacted-clorox-complaint.pdf

[22] OWASP GenAI Security Project. OWASP Top 10 for Agentic Applications for 2026. 2025. https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/

[23] OpenAI. Evaluate Agent Workflows. Accessed 2026-08-28. https://developers.openai.com/api/docs/guides/agent-evals

[24] Anthropic. Measuring AI Agent Autonomy in Practice. 2026. https://www.anthropic.com/research/measuring-agent-autonomy

[25] Microsoft. Workplace and IT Services Pattern. Accessed 2026-08-30. https://learn.microsoft.com/en-us/agents/adoption-patterns/pattern-workplace-it-services

[26] ServiceNow. EY Uses AI Agents to Transform Employee Support. Accessed 2026-08-30. https://www.servicenow.com/customers/ey-ai.html

[27] NIST. Artificial Intelligence Risk Management Framework (AI RMF 1.0). 2023. https://doi.org/10.6028/NIST.AI.100-1