How to scale GenAI apps built on Azure AI services

Teams everywhere are jumping into generative AI, and many are choosing Azure AI services. Azure OpenAI, alone, has seen a 50% increase in customers over the past year, with more than half of the Fortune 500 companies using it.

Azure now offers direct access to powerful models through Azure OpenAI, alongside the new Azure AI Foundry and Azure AI content studio, giving developers a solid base for creating, testing, and growing AI applications with proper security and compliance.

What makes the Azure AI ecosystem attractive? It combines top-tier models like GPT-4 and DALL·E with features businesses actually need - data privacy, regional deployment options, and safety controls. Azure AI Foundry expands these options further, providing access to over 1,800 models, including open-source options from Meta and Mistral, plus Microsoft's own Phi models. Azure AI content safety provides the guardrails to enhance the safety of these apps.

This setup creates real opportunities for AI teams working on various projects—whether you're building copilots, automating business processes, or creating search and summarization tools. Azure provides the necessary infrastructure, model access, and enterprise integration.

Scaling with Azure AI services

As enterprises double down on generative AI, Azure OpenAI becomes the natural starting point. It gives them access to state-of-the-art models like GPT-4, coupled with the security, compliance, and identity controls they already rely on. Teams across the organization start spinning up AI-powered copilots, knowledge assistants, document processors, and more.

What begins as one or two experiments quickly grows. Within months, there are dozens of AI initiatives in flight — each with different teams, environments, use cases, and production readiness levels. Naturally, questions arise: How much is each app costing us? Are these experiments worth continuing? Who’s responsible for which workloads?

To answer these, many companies adopt a now-common workaround: they create a separate Azure subscription for each app or team. This lets them isolate budgets, monitor usage, and impose basic access controls without stepping on each other. On paper, this works — you get clear billing lines and cleaner separation. But it comes at a cost.

With every new subscription, you also need to re-provision infrastructure: key vaults, virtual networks, deployment configurations, access policies, quota requests. Multiply this by every environment - dev, staging, prod - and soon you're duplicating setup processes and slowing your teams down. Spinning up a new use case becomes an exercise in coordination, not innovation.

In response, some teams try to consolidate, returning to a shared Azure OpenAI subscription to reduce overhead and speed up app development. This approach does streamline provisioning, but it introduces a new class of problems.

With shared infrastructure, it becomes much harder to track spending at a granular level. One app might be quietly consuming thousands of dollars' worth of tokens in the background, while another team is blocked by quota exhaustion. There's no easy way to tell who’s spending what, or to enforce limits per team or user.

Governance becomes blurry, too. Without application-level controls, it’s difficult to apply safety filters, usage policies, or rate limits tailored to individual use cases. Everyone ends up with the same level of access, regardless of their risk profile or needs. And while Azure AI Content Safety helps at the platform level, there's no out-of-the-box way to configure different guardrails for different apps or environments.

As more teams join the shared infrastructure, reliability also becomes an issue. If one app hits a model limit or fails, it affects others. There’s no built-in system to handle fallbacks, retries, or intelligent routing across models, especially now that teams are starting to experiment with models from Azure AI Foundry like Meta’s Llama or Microsoft’s Phi.

Finally, there’s the challenge of experimentation. Teams want to iterate on prompts, test model performance, and ship improvements fast — but in a shared setup, there's no common layer to manage prompt versions, test scenarios, or A/B deployments across different tools. Everyone builds their own band-aid solutions, and best practices rarely scale across teams.

So while Azure gives enterprises the foundation to build powerful AI systems, the challenges emerge in the scaling phase, when governance, cost control, and developer velocity all clash.

How Portkey solves these problems for AI teams

Portkey is built to sit between your AI applications and your model providers, giving you a single control layer to manage observability, control costs, and ensure safe, reliable behavior across every AI call. Now, with full support for Azure AI services, Portkey helps teams experience Azure's full potential without introducing operational overhead.

Portkey is deeply integrated into the Azure ecosystem:

• You can deploy Portkey directly from the Azure Marketplace with minimal setup
• Microsoft Entra SSO & SCIM support allows you to control access, manage users, and stay compliant with org-level policies
• You can connect 1800+ models via Azure OpenAI and Azure AI Foundry and manage them through a single Portkey interface
• Azure AI Content Safety guardrails can be enforced on every Portkey request, helping ensure responsible AI behavior by default
• Developers can keep using familiar tooling — Portkey is compatible with the official OpenAI C# SDK, as well as Microsoft’s Autogen and Semantic Kernel frameworks

Instead of spinning up new Azure subscriptions for every team or app just to track usage or enforce quotas, Portkey introduces a better abstraction: virtual keys. Each virtual key can map to an individual app, team, or use case. You can add rate limits and budget limits to the virtual keys. This way, you centrally manage all consumption on a single subscription, while maintaining isolation and guardrails for each consumer.

Portkey also supports role-based access control (RBAC) and hierarchical org management. That means you can mirror your internal org structure directly in Portkey, assign teams to business units, manage who can deploy prompts, and enforce policy boundaries without manual overhead.

You also gain real-time observability - tracking latency, usage, and cost for every request across every model. Whether you’re using GPT-4 via Azure OpenAI, or exploring Meta’s Llama or Microsoft’s Phi via Foundry, Portkey gives you full visibility and analytics at every layer.

Need to ensure uptime and consistency? Portkey supports fallbacks, retries, and intelligent routing across models and providers, so your end users aren't impacted if one fails or slows down. You can also test, version, and deploy prompts without code changes, making it easy to iterate and ship improvements quickly.

Build faster. Scale smarter. Stay in control.

As enterprises lean into Azure to power their most critical AI workloads, the ability to maintain agility without sacrificing control becomes non-negotiable. Portkey fills the operational gap, giving AI teams the tools to monitor, govern, and optimize their Azure OpenAI and Azure AI Foundry usage without friction.

No need to re-architect. No need to manage dozens of isolated subscriptions. With Portkey, you get a centralized, scalable way to manage all your AI applications, with full compatibility across Azure services, Microsoft frameworks, and your existing developer tools.

If you're building on Azure, Portkey is the ideal control layer. Get started today.