The risks are real. The challenge is turning fear into governance before autonomy moves faster than our ability to control it.

The fear is reasonable. Panic isn't the answer. Neither is blind acceleration.

In September 2026, Anthropic CEO Dario Amodei published an essay titled We Must Pace the Frontier.

One line particularly stood out to me:

“We have tried to prioritize caution over speed and prudence over profit.”

Amodei's argument isn't that AI development should stop. He argues that AI capabilities may advance faster than our ability to understand and control their risks, and that safety needs time to keep pace. His proposed response includes embedded independent safety evaluation, coordination among frontier AI companies, and broader international coordination.

That is a frontier-AI question.

But enterprises now face their own version of it:

How fast should we increase AI autonomy when our ability to govern that autonomy is still catching up?

The answer begins by reconsidering what a rogue enterprise agent might actually look like.

The Most Dangerous Agent May Be the One You Authorized

When people hear “rogue AI agent,” the image is usually dramatic.

An autonomous system breaks free of human control.

It ignores its instructions.

It penetrates systems.

It begins acting against its creator.

Those scenarios deserve serious attention. But inside an enterprise, the more immediate risk may be far less cinematic.

Imagine this:

The agent authenticates successfully.

The API works exactly as designed.

The permission check passes.

The transaction succeeds.

And the outcome is still wrong.

Nothing was hacked.

No security boundary was necessarily breached.

The agent may have followed its technical instructions perfectly.

The problem is that the agent exercised a capability that should never have been available in that context.

Consider a hypothetical client-service agent asked to prepare a summary before a customer meeting.

The agent has legitimate credentials. It queries the CRM, follows relationships to connected accounts, and retrieves information its identity is technically permitted to access.

But an old sharing rule exposes records that were never intended for this particular business context.

The agent hasn't bypassed authentication.

It hasn't violated an API policy.

Every technical control may have said yes.

A human user might never have navigated through all those relationships or assembled the information at that scale. The agent can do it systematically, repeatedly, and at machine speed.

The problem isn't that the agent escaped the controls.

The problem is that the controls were designed for an operating model that no longer exists.

Most enterprise systems were built around an assumption we rarely needed to state explicitly:

A human is operating them.

Humans read screens.

They understand organizational context.

They hesitate when something looks unusual.

They ask colleagues questions.

They recognize that just because they technically can do something doesn't necessarily mean they should.

And humans operate at human speed.

Agents change those assumptions.

An AI agent can navigate connected systems, invoke APIs, retrieve information, combine context, execute permitted actions, and repeat those activities at a scale and speed fundamentally different from the human users for whom many existing permissions were originally designed.

That changes the risk equation.

The most dangerous enterprise agent may not be the one that breaks the rules. It may be the one that follows outdated rules perfectly, at machine speed.

Authentication succeeded. Authorization succeeded. The action was allowed. The outcome can still be wrong.

Suddenly, the rogue-agent problem looks less like science fiction.

It looks like enterprise architecture.

The Fears Are Different. The Pattern Is Familiar.

The fear surrounding AI can feel unprecedented.

In some ways, it is. We have never deployed software with this combination of reasoning, access, autonomy, and ability to act across systems at machine speed.

But the pattern surrounding technological change is much older.

New technologies create new possibilities. Those possibilities introduce new risks. Fear follows.

The important point is that not every fear is the same.

Railways introduced new safety and operational risks as transportation expanded at an unfamiliar speed and scale. The appropriate response was increasingly safer engineering, operating practices, standards, and regulatory oversight.

ATMs raised a different concern: the effect of automation on employment. Economist James Bessen argued that technology can automate particular tasks while shifting workers toward different activities requiring different skills. That doesn't prove AI won't eliminate jobs. It demonstrates why workforce disruption needs its own response, including adaptation, reskilling, and redesign of work.

Quantum computing presents another kind of risk. In May 2025, Google Quantum AI researcher Craig Gidney estimated that RSA-2048 could theoretically be factored in less than a week using fewer than one million noisy physical qubits, significantly reducing the resource estimate from earlier work. Google also emphasized that relevant contemporary machines remained far below that threshold.

The response to that future threat is already underway. NIST finalized its first three principal post-quantum cryptography standards in August 2024 and advises organizations to begin migrating toward quantum-resistant cryptography. NIST says its transition work will ultimately remove quantum-vulnerable algorithms from its standards by 2035, with high-risk systems moving earlier.

These examples aren't equivalent, and they shouldn't be treated as if they are.

Railways illustrate the need to make powerful infrastructure safer.

Automation illustrates the need to adapt work as technology changes tasks.

Quantum computing illustrates the importance of preparing architecture for a future threat before that threat fully materializes.

Different technology. Different risk. Different response.

Fear becomes useful when it is translated into a specific problem and an appropriate response.

History doesn't tell us that technological fears are foolish.

It doesn't guarantee that everything eventually works itself out.

It tells us something more useful:

Fear becomes valuable when we translate it into a specific problem we know how to manage.

That distinction matters because AI brings several concerns together at once.

There is a workforce question: What happens as AI performs more tasks previously performed by people?

There is a security question: What happens when AI gains access to sensitive data and enterprise systems?

And increasingly, there is an autonomy question:

What happens when we give AI not just information, but authority to act?

That final question is the focus of this article.

It is where rogue agents become an enterprise architecture problem.

AI May Expose Governance Debt We Already Have

This connects directly to something I have previously described as latent governance debt.

Enterprise platforms accumulate it slowly.

Old permissions.

Excessive access.

Stale sharing rules.

Unused APIs.

Legacy integrations.

Weak data classifications.

Controls that depend more on what a user sees in an interface than on what the underlying system actually enforces.

These conditions can remain almost invisible for years.

Partly because humans themselves provide an informal layer of governance.

People develop institutional knowledge. They understand exceptions. They recognize which technically possible actions aren't normally appropriate.

They hesitate.

Agents don't automatically inherit that institutional intuition simply because we give them credentials.

I see this challenge particularly clearly in platforms such as Salesforce, where years of accumulated profiles, permission sets, sharing rules, integrations, APIs, and business logic were largely designed around human users and established application flows.

As enterprises introduce autonomous agents into these environments, cleaning up that foundation is no longer simply platform hygiene.

It becomes part of AI readiness.

When autonomous systems begin navigating environments designed around human behavior, yesterday's governance shortcuts can become tomorrow's agent risk.

That leads to an important architectural principle:

AI governance cannot begin with the agent. It has to begin with the environment the agent is entering.

Before asking what an agent should do, organizations need to understand what their systems already allow it to do.

In many situations, AI may not create the underlying governance weakness.

It may expose it.

What Pacing Should Mean Inside the Enterprise

This is where I think Amodei's idea of pacing has a useful enterprise interpretation:

Adopt AI at the speed you can govern it.

That doesn't mean slowing every AI initiative.

It means matching autonomy with controls.

An AI assistant that summarizes information creates one level of risk.

An agent that retrieves enterprise data creates another.

An agent that modifies data increases the risk again.

An agent capable of initiating transactions, communicating externally, invoking multiple tools, or orchestrating other agents requires a substantially stronger control environment.

Perhaps enterprises shouldn't measure AI maturity primarily by how many agents they deploy.

A more useful question might be:

How much autonomy are we prepared to govern?

Think of autonomy as a progression:

Assist → Recommend → Act with approval → Act within defined boundaries → Operate autonomously

Moving from one level to the next shouldn't simply be a product decision.

It should be a governance decision.

Every increase in autonomy should be accompanied by a corresponding increase in permissions discipline, observability, testing, containment, and accountability.

As autonomy increases, governance must increase with it.

That is what enterprise pacing should mean.

Machine-Speed Risk Requires Machine-Speed Defense

There is another uncomfortable reality.

Agents operate at machine speed.

If autonomous systems produce enormous volumes of actions across enterprise applications, APIs, and data, relying exclusively on people to inspect every activity creates an obvious scaling problem.

We are already seeing a related challenge emerge in cybersecurity.

In April 2026, Anthropic launched Project Glasswing, an initiative bringing together AWS, Apple, Broadcom, Cisco, CrowdStrike, Google, JPMorganChase, the Linux Foundation, Microsoft, NVIDIA, Palo Alto Networks, and Anthropic to apply frontier AI capabilities to defensive cybersecurity. Anthropic also says it extended access to more than 40 additional organizations that build or maintain critical software infrastructure.

At the center of the initiative is Claude Mythos Preview. Anthropic describes the model as having strong agentic coding, reasoning, search, and computer-use capabilities. Anthropic says the model identified thousands of previously unknown vulnerabilities and that, for a published subset, it identified nearly all of the vulnerabilities and developed many related exploits autonomously, without human steering.

Glasswing isn't an architecture for governing rogue AI agents.

It illustrates a broader principle:

As AI increases the speed and scale of technological capability, AI may also become part of the defensive layer that helps humans keep pace.

Enterprise agent governance may eventually face a similar scaling challenge.

Supervisory AI could help organizations identify anomalous agent behavior, unusual access patterns, unexpected tool usage, or activity across volumes of operations that people cannot continuously inspect manually.

But this introduces an important boundary.

A governing AI system is still a probabilistic system.

AI can help us observe and detect at scale.

Watching, however, is not the same as controlling.

The foundation of control still needs to be deterministic.

Permissions must be enforced.

Boundaries must be explicit.

High-impact actions need defined approval policies.

Organizations must be able to stop agent activity whether another AI recognizes the problem or not.

That leads to what I think of as the Enterprise Agent Governance Stack.

The Enterprise Agent Governance Stack

1. Deterministic Boundaries

Start with what the agent is technically capable of doing.

Permissions, scopes, data boundaries, tool access, refusal rules, and approval gates should constrain the potential blast radius before reasoning begins.

Don't rely on the agent simply deciding not to perform an inappropriate action.

Remove capabilities it doesn't need.

2. Complete Observability

Every significant agent action should be attributable and reviewable.

Organizations should be able to answer:

• What identity acted?
• What information was accessed?
• What tool was invoked?
• What action was taken?
• What was the outcome?

Autonomy without traceability eventually becomes unmanageable.

3. Supervisory Intelligence

Once deterministic controls and observability exist, AI can assist with monitoring activity at scale.

Supervisory systems could complement traditional security and operational monitoring by identifying patterns worthy of attention.

But this layer should augment enforcement.

It should never replace it.

4. Human Escalation

Not every decision belongs to an autonomous system.

Organizations need clear thresholds where control returns to accountable people, particularly for actions with significant financial, regulatory, security, customer, or operational consequences.

Good agent architecture isn't about removing humans from every process.

It is about understanding:

Where does human judgment still matter most?

5. Tested Containment

Finally, organizations need a reliable way to stop autonomous activity.

A kill switch shouldn't exist only in an architecture document.

Containment mechanisms need to be understood and exercised before they're needed.

Because the first time you discover whether you can stop an agent shouldn't be when you're desperately trying to stop one.

Probabilistic oversight on top of deterministic enforcement. Never instead of it.

Fear Is a Signal, Not a Strategy

Fear has a useful function.

It tells us where to look.

But fear itself cannot tell us what to build.

That requires understanding the specific risk.

Workforce disruption requires thoughtful redesign and reskilling.

Cybersecurity threats require architecture, standards, and technical controls.

Autonomous systems require permissions, boundaries, observability, accountability, and containment.

The mistake is treating all AI fear as either irrational pessimism or proof that we should stop.

It is equally dangerous to move in the opposite direction and accelerate simply because everyone else seems to be accelerating.

History doesn't guarantee that everything will turn out fine.

That isn't the lesson.

The lesson is that societies and organizations have repeatedly learned to turn technological risk into something they can understand, govern, and adapt around.

Railways required safer infrastructure and operating standards.

Automation forced work to evolve.

Quantum computing is forcing cryptography to prepare before the threat fully materializes.

AI-powered cybersecurity is beginning to demonstrate why defenses may also need to operate at technological speed.

Agentic AI will demand its own response.

And perhaps this time the challenge is harder because the technology isn't merely giving us another tool.

We are beginning to give the tool authority to act.

That changes the responsibility of enterprise architecture.

The question is no longer simply:

What can AI do?

It becomes:

What should AI be allowed to do, under what conditions, with what visibility, and how quickly can we stop it when something goes wrong?

The answer to rogue agents isn't to stop building agents.

And it isn't to deploy autonomous agents everywhere while assuming another AI will eventually keep them under control.

It is to build environments where autonomy can increase without control decreasing.

The organizations that succeed in the agentic era may not be the ones that move fastest.

Nor will they be the ones waiting for every uncertainty to disappear.

They will be the ones capable of increasing autonomy without losing control.

The fear is reasonable.

Panic isn't the answer.

Neither is blind acceleration.

Caution over speed doesn't mean standing still.

It means moving at the speed we can govern.

Sources

• Dario Amodei, “We Must Pace the Frontier”, September 2026.
• https://www.imf.org/external/pubs/ft/fandd/2015/03/bessen.htm, IMF Finance & Development, 2015.
• https://arxiv.org/abs/2505.15917, May 2025.
• https://blog.google/security/tracking-cost-of-quantum-factori/, May 2025.
• NIST Post-Quantum Cryptography.
• Anthropic, “Project Glasswing: Securing critical software for the AI era”, April 2026.