Why building an agent and governing it are two different projects
An AI agent is a program that uses an LLM to plan a sequence of tool calls in response to a goal. It reads documents, queries databases, writes outputs, sends notifications, and may trigger other agents. The LLM is not what creates the governance problem. The tool calls are. When an agent reads a file, that action is reversible: nothing changed. When it writes to a customer record, sends an outbound email, or approves a transaction in a connected system, the action may be irreversible, may have an external audience, and may be legally attributable to the organization that deployed the agent.
Most agents in enterprise deployment today inherit the credentials of the user who triggered them. If a sales manager runs an agent that processes customer contracts, the agent's API calls appear in every connected system's audit log as actions taken by the sales manager. The agent and the human are indistinguishable in the record. That is not an edge case or a configuration oversight. It is the default behavior of agents built on user-delegated OAuth tokens, which is how most enterprise integrations work. The compliance implication is that the organization cannot demonstrate after the fact which actions were taken by a human exercising judgment and which were taken by an agent operating autonomously.
The governance decisions below address the five points where enterprise agent deployments most commonly fail compliance and audit requirements. They are ordered by where the failure causes the most damage: attribution and reversibility first, then oversight quality, then escalation handling, and finally the deployment gate that keeps ungoverned agents from reaching production.
“You can build an AI agent in a day. Making it auditable, reversible, and compliant takes longer. Most enterprises start the second project only after the first incident.”
The 5 governance decisions and what an audit or incident reveals when each is missing
These decisions apply to any agent that takes actions with external effects: writing to production systems, sending communications, calling APIs that trigger downstream processes, or modifying records that other systems or users depend on. Agents that only read and summarize face a narrower set of these concerns, but every agent that writes should be evaluated against all five.
| Governance decision | What most teams do | The compliance gap it creates | How it surfaces | Risk level |
|---|---|---|---|---|
| Build an action inventory: classify every tool the agent can call as reversible or irreversible before deployment | The agent is given access to all tools needed to complete its workflow. No distinction is made between reading a document, writing to a CRM record, and sending an outbound email. All actions are treated the same in terms of access control and logging | No differential controls exist. The agent applies the same level of autonomy to a read operation as to an irreversible write. When an incident involves an irreversible action, the organization has no documented basis for why the agent was permitted to take that action without additional authorization, because the question of reversibility was never asked at design time | An agent managing a customer onboarding workflow sends 400 account confirmation emails to customers in a data quality review queue before the queue was validated. The emails cannot be recalled. The team stops the agent but cannot demonstrate that a human reviewed and approved the email-sending step. Audit question: why was an agent permitted to take irreversible customer-facing actions without a documented authorization checkpoint? | Critical |
| Give agents their own identity: agent actions must appear under agent credentials in every system log, not under human credentials | Agents run on the OAuth token or API key of the user who triggered them. All agent actions appear as that user's actions in downstream system logs. The agent and the human are indistinguishable in the audit record | Attribution failure. The organization cannot demonstrate after the fact which actions were taken by a human exercising judgment and which were taken autonomously by the agent. For regulated industries, this collapses the distinction between supervised human action and autonomous AI action, which is precisely the distinction that human oversight requirements exist to preserve. When an agent causes a compliance incident, the logs show it as a human action, which prevents both root cause analysis and regulatory defense | A compliance audit of a financial document processing workflow asks for a log of all actions taken on customer files. The log shows all edits attributed to the analyst who triggered the agent. The auditor asks which actions the analyst reviewed and authorized versus which the agent took autonomously. The organization cannot answer the question from the existing log. Finding: inadequate audit trail for automated processing of regulated documents | Critical |
| Define human-in-the-loop as decision authority, not approval presence: the human must have context, recourse, and the ability to override | Human-in-the-loop means a human sees a notification or approval prompt before the agent takes a specified action. The notification typically shows the action name and a description. The human approves or times out. No context is provided about the agent's reasoning, the data it reviewed, or the alternatives it considered | Oversight without authority. The human cannot make an informed decision from the information provided, so approval becomes a habit rather than a control. The EU AI Act's human oversight requirement for high-risk AI explicitly requires that the responsible person be able to understand the system's output, detect and correct failures, and override the system's decisions. A notification that shows “action: update customer record” with an approve/decline button does not satisfy that requirement. If a regulator asks whether a human reviewed and authorized a specific agent action, the answer depends on whether the approval prompt gave the human enough information to actually make that determination | An EU AI Act audit reviews a healthcare documentation agent. The human oversight record shows that a clinician approved 847 document updates in a single shift, averaging 12 seconds per approval. The auditor determines that the approval process did not provide sufficient information for a clinician to review each update meaningfully. Finding: human oversight does not meet the EU AI Act's standard of enabling detection and correction of failures | High |
| Define an escalation path for every stuck state: when the agent cannot proceed, there must be a named person who receives the case and a defined resolution process | When an agent encounters an error or a decision it cannot resolve, it logs the error and stops. No notification is sent to a human. The workflow sits incomplete. The user who triggered the agent may not know it stopped until they check manually | Silent workflow failure. Long-running agent workflows can halt midway through a multi-step process, leaving downstream systems in an inconsistent state: some records updated, others not. Because no escalation fires, nobody investigates. The inconsistency propagates through connected systems until something downstream fails or a human reconciles the records manually. For time-sensitive workflows, the halt may cause a deadline miss or a compliance gap that neither the triggering user nor the compliance team is aware of | An agent automating a contract renewal workflow stops at step 4 of 7 after encountering an unexpected data format in a vendor record. No notification fires. The contract renewal deadline passes. The compliance team discovers the lapse during a quarterly review. The agent's log shows it stopped cleanly, but no one was notified. Finding: no defined escalation path for agent workflow failures in time-sensitive compliance processes | High |
| Build rollback into the deployment design: define before go-live how completed agent actions can be reversed for every irreversible action class | Rollback is not addressed at design time. If the agent needs to be stopped, the team kills the agent process. Actions already completed by the agent remain in place. Manual remediation is performed case by case if needed | No recovery capability. When an agent takes a wrong or unauthorized action, the organization can stop further actions but cannot reverse what has already occurred. Rubrik Zero Labs found 88% of organizations are in this position. For regulated workflows, this creates a situation where the organization must demonstrate that an erroneous agent action has been remediated, but has no documented process for how that remediation works because rollback was never designed. Manual remediation on a case-by-case basis is both slower and harder to audit than a defined rollback process | An agent updating financial period-end records applies an incorrect calculation to 1,200 records before an error is detected and the agent is stopped. The records must be corrected manually. The process takes three days and requires coordination across two departments. Audit question: what is the organization's documented procedure for reversing autonomous AI actions on financial records? Answer: manual correction, case by case, no documented process. Finding: inadequate change management controls for autonomous AI processing of financial data | Moderate |
The two Critical-level gaps share a structure: the agent was designed to accomplish a task, and the team focused entirely on whether it could do the task. Whether it could be held accountable for doing it, and whether the organization could demonstrate that human judgment was involved in the right places, was left for later. Later usually means after the first incident forces the question.
Not sure where your AI agent governance gaps are?
10decoders builds agent governance frameworks for enterprise workflows: action inventories, agent-scoped credential architecture, meaningful HITL checkpoint design, escalation routing, and rollback planning. We review your current agent deployments and produce a governance gap report covering what is auditable, what is not, and what changes would satisfy regulatory review.
Book a Free AI Assessment →Why the token reuse problem is harder to fix than it looks
Giving agents their own identity sounds straightforward. In practice, most enterprise AI frameworks are built on top of OAuth integrations that were designed for human users. When an agent needs to write to Salesforce, send a message in Teams, or update a Jira ticket, the most direct path is to use the access token of the user who triggered the agent. The integration already works. The agent can take the action. The fact that the action appears as a human action in the downstream system's log is not visible at the point of building the agent, because nobody is looking at the audit log yet.
The technical fix is to provision a service account for each agent, grant it the minimum permissions needed to complete its workflow, and configure every tool call to authenticate with that service account rather than the user's token. This requires the agent framework to support multi-credential management, and it requires someone to decide what permissions the agent service account should have, which forces the action inventory decision earlier in the process. Organizations that implement agent-scoped credentials almost always also build an action inventory, because the two decisions are connected: you cannot define the agent's permissions without first knowing what actions it will take.
Beyond cleaner attribution, scoped credentials also limit the blast radius. An agent that authenticates as itself, with its own permissions, can only do what that service account is authorized to do. If the agent is compromised, misconfigured, or makes an unexpected decision, the damage is bounded by the service account's permission set. An agent running on a user's token can do anything that user is authorized to do in every connected system, which is typically far more than any agent should be able to do autonomously.
No governance baseline
Agents run on user credentials, take actions with no classification, have no escalation path, and cannot be rolled back. All agent actions are attributed to the triggering human. The first audit or incident starts from scratch with no documentation. Governance must be reconstructed retroactively, action by action.
Attribution and inventory in place
Agents have their own service account credentials, and an action inventory classifies reversible and irreversible tool calls. Basic HITL checkpoints exist for irreversible actions, though without adequate context in the approval prompt. The audit trail attributes agent actions correctly. Escalation routing and rollback procedures are still missing, and oversight design needs work.
Auditable, reversible, governed
All five decisions in place before go-live. Each agent authenticates with its own scoped service account. HITL checkpoints provide context, reasoning, and an override option for every irreversible action class. Every stuck state routes to a named owner with a tested notification path. Rollback procedures are documented and exercised in staging before the agent ships. Governance sign-off sits alongside functional testing as a go-live requirement.
The enterprise AI agent governance checklist
This checklist covers the governance documentation and design decisions an organization should have in place before any agent that takes external-facing or system-modifying actions reaches production. These are the records most likely to be requested in an audit and most likely to be absent when the organization built the agent to accomplish a task and treated governance as a later concern.
“88% of organizations cannot roll back agent actions without disrupting their systems. That is not a capability problem. It is a design problem. Rollback was never on the requirements list.”
What to do this week
Inventory every agent in production and classify its tool calls
Start with a list of every agent or agentic workflow your organization currently runs in production. For each one, list every tool it can invoke and classify each tool call as read-only, reversible write, or irreversible write. This exercise will almost certainly surface tool calls that the team did not think of as irreversible when they built the agent. The most common surprises are email sends, messaging platform notifications, CRM record updates, and API calls to vendor systems. This exercise exists to find out which of your current agents are taking actions that cannot be undone, with no documented procedure for what happens if something goes wrong.
Check the audit logs of your connected systems to see whose identity agent actions are logged under
Pick the two or three connected systems your most active agent integrates with and pull the audit logs for the past week. Find actions you know were taken by the agent and check what identity they are attributed to. If they appear under a human user's name, your attribution is broken and those logs will not support a compliance defense that separates human judgment from autonomous agent action. This check takes less than an hour and answers a question most teams do not know the answer to until an audit forces it.
Review your most recent HITL approval prompts and ask whether they provide enough context for a human to actually make a decision
For each approval checkpoint in your agents, review what information is shown to the human approver. Ask: if a regulator reviewed this approval log, would they conclude that the human had enough context to exercise meaningful oversight? If the prompt shows only an action name and a yes/no button, the answer is almost certainly no. Adding the agent's reasoning, the data it reviewed, and an explicit override option to the prompt design is usually a few hours of engineering work. That change is typically a few hours of engineering work, and the difference between an approval log that demonstrates oversight and one that demonstrates only that a human was present.
Document the rollback procedure for your highest-risk agent action before this week is out
Pick the one agent tool call that would cause the most damage if it executed incorrectly at scale: the email send, the payment trigger, the record update that feeds downstream systems. Write down, in plain language, what happens if that action fires incorrectly on 100 records: who is notified, what manual steps are taken in which systems, who has authority to execute those steps, and what gets communicated to affected customers or counterparties. This is not a legal document and does not need to be comprehensive to be useful. The goal is to have an answer ready before the incident forces an improvised one, because improvised remediation after an agent incident is always slower, more expensive, and harder to defend in an audit than a procedure that was written and tested before the agent went live.
Let 10decoders govern your AI agent workflows before the first incident
We produce action inventories, agent credential architecture, HITL checkpoint designs, escalation routing maps, and rollback procedures for enterprise agentic workflows. Our AI engineering practice has built governed agent deployments for healthcare and FinTech organizations where compliance review is part of the production gate, not an afterthought. We deliver governance documentation that holds up to audit review, not just internal review.
