Your First AI Agent Will Go Fine. Your Fiftieth Is Where Things Get Interesting.
The first agent is almost always a success story. A well-scoped task, a small team, clear ownership, everyone watching it closely. It works. Leadership gets excited. More agents get approved.
And somewhere between agent five and agent fifty, the wheels come off.
This is not a warning about AI being dangerous or not ready. Agents are ready. The question is whether the infrastructure around them is. Karpathy put it clearly in early 2026 when he described the shift happening in real-time: humans are no longer the ones writing the code or running the tasks. We are directing, supervising, and orchestrating agents. The new unit is teams of agents. Which means the new failure mode is a team you cannot see, cannot account for, and cannot stop.
Here is what actually goes wrong when enterprises scale agents. Not theory, but pattern.
You stop knowing what you have running
Agents get built in pockets across teams and tools. A customer support team spins one up for ticket triage. Engineering builds one for code review. Finance deploys one for reconciliation. None of these are bad decisions in isolation. The problem is what accumulates in aggregate when there is no shared registry, no clear ownership model, no governed endpoint per agent. The teams building them are moving fast, and the review processes that exist were designed for a different era of software. The gap between "we built this" and "we have formal accountability for this" opens quietly.
A 2025 survey by Gravitee of over 900 executives and technical practitioners found that only 14.4% of agents go live with full security approval. It's less a story about negligence and more a story about pace. Most teams reach a point where they genuinely cannot say with confidence how many agents are running, who owns each one, or what each one is authorised to do.
This matters because you cannot govern what you cannot see, and you cannot shut down, audit, or revoke access to something you have lost track of. The inventory gap is what all the other problems tend to build on.
What agents are allowed to do versus what they end up doing
There is a failure mode that does not show up in demos. An agent gets provisioned with credentials for its primary task, say, read access to the CRM to personalise responses. But the credential it was given also has write access, because scoping it more precisely would have taken time and a separate request. Nobody was careless. It just happened, the way it always happens when tooling moves faster than process.
The same Gravitee research found that 45.6% of organisations rely on shared API keys for agent-to-agent authentication, and 27.2% use custom hardcoded authorisation logic. These approaches are practical in development. In production, when you have many agents operating continuously, they become the gaps through which agents drift outside their intended scope, not through any malicious act, but through the natural behaviour of systems that were never given firm limits.
OWASP's AI Agent Security guidelines describe this as excessive agency: agents given more access than their specific job actually requires. The fix is not complicated in principle. You scope permissions to the task, not to what is convenient to provision. In practice, this requires a governance layer that most teams only think to build after something has already drifted.
The security exposure that looks like a feature
Simon Willison, who coined the term "prompt injection" described something in a June 2025 piece that is worth understanding in full.
He identified three capabilities that, when combined in a single agent, create what he calls the lethal trifecta: access to private data, exposure to untrusted content, and the ability to communicate externally.
The uncomfortable part is that all three are completely normal properties of a useful agent. An agent handling customer support will access your customer records, read inbound messages from people you do not know, and send replies. The usefulness and the vulnerability are the same feature. What makes this hard to see is that the attack surface is not in your codebase. There is no port to firewall, no traditional vulnerability to patch. The exposure lives in what the agent reads and decides to act on. An attacker can send your agent an email with instructions embedded in the message body, and your agent, which is very good at following instructions, might just follow them.
Although, that is not a reason to stop building. It is a reason to understand exactly what each agent can access, what it can ingest, and what it can communicate outward, and to make those properties visible and governed.
When chains fail and you cannot trace why
A single agent failing is a recoverable event. A chain of agents, where each one's output becomes the next one's input, is a different kind of problem.
A small hallucination in step two does not stay small. It gets reasoned upon at step three, acted on at step four, and by step six the downstream consequences have nothing obvious connecting them to the original error.
Because chains can move quickly, the blast radius often expands before anyone has had a chance to look. And without complete traces across every run, including every tool call and model decision, debugging becomes archaeology. You are trying to reconstruct what might have happened rather than reading a record of what did. Gravitee found that 60% of organisations cannot even terminate a misbehaving agent once it has started operating. Observability and a working kill switch are not optional infrastructure. They are the baseline for anything running with real consequences.
The governance gap is the real bottleneck
When you look at all of the above together, the pattern that emerges is not really a security story or a cost story or even an engineering story. It is an operational maturity story. Most teams build agents before they have the production layer to run them safely. That is not a failure of planning, it is just the order in which things get built. You build for what works. You govern for what scales.
A 2026 analysis from Solutions Review captured what is becoming a shared realisation across engineering and executive leadership:
CTOs and CIOs will realise that their biggest bottleneck isn't model performance. It's governance. They'll discover that traditional IAM and RBAC tools can't keep pace with short-lived, dynamic agents acting across hundreds of services.
The teams navigating this well are not waiting for the bottleneck to reveal itself. They are treating agent governance the way mature engineering organisations treat DevOps: not as a compliance layer bolted on at the end, but as the infrastructure the whole system runs on. A registry where every agent has a home. Governed endpoints. Policy enforcement. Real-time cost and usage metrics. The ability to revoke access or stop an agent without touching its code.
The teams that are not doing this are building fast and accumulating operational debt that becomes very expensive to address once something has gone wrong.
Why we built Agent Gateway
Everything above reflects what we have watched happen, repeatedly, across the thousands of teams building production AI on Portkey. These are not hypothetical risks. They are the exact walls teams hit when they scale past their first few agents. Every team that ships agents at scale hits the same questions fast: who can call this, what does it cost, what did it do, and who is responsible when something goes wrong? Most teams patch together answers themselves, differently, incompletely, and usually after something has already surfaced.
That is what Agent Gateway is built for. Register any agent, get a governed endpoint, and everything flows through a single stack from that point on:
- Full traces across every run, including every MCP call, so you always know what happened
- A registry so every agent has an owner, an endpoint, and defined access controls
- Automatic fallbacks so provider outages never reach your agents
- 50+ guardrails out of the box, PII detection, PHI filtering, content moderation
- One consistent auth model across every agent, regardless of how it was built
We are hosting a webinar this Friday, April 24th, where we will walk through the architecture, access control, and policy enforcement in depth.
If you are building agents in production, or getting ready to, this is the hour that saves you several painful weeks later. See you there 👋