Why We Chose TypeScript Over Python for the World's Fastest AI Gateway

As developers pushing the boundaries of AI tooling, we at Portkey faced a critical decision when building our open-source AI Gateway: choosing the right technology stack. Today, I'll dive deep into why we chose TypeScript over Python and Rust, and how this decision led to creating the fastest, most performant AI Gateway in the world.

Since we started Portkey, we've made numerous crucial technical decisions. Perhaps none was more pivotal than choosing the right technology stack for our open-source AI Gateway. Today, I want to share the story of why we chose TypeScript over Python and Rust, and how this decision led to creating the fastest, most performant AI Gateway in the world.

The Challenge: Building a Lightning-Fast AI Gateway

In the AI realm, latency is the enemy. For real-time applications, the difference between a 1000ms and a 2000ms response time can make or break user experience. Our mission was clear: build a gateway with the lowest possible latency while handling complex tasks like request routing, API management, load balancing, and security.

Embracing Edge Computing

As we delved deeper into solving the latency challenge, we realized that traditional cloud computing couldn't meet our ultra-low latency requirements. We turned to edge computing, distributing our gateway globally to run closer to end-users. This approach promised to dramatically reduce latency.

After evaluating various options, we settled on Cloudflare Workers for our distributed edge computing needs. Cloudflare's vast network of global data centers aligned perfectly with our vision of a low-latency, globally distributed AI Gateway.

Now, The Language Dilemma: 🐍 Python, 🦀 Rust, or TypeScript?

Here's how we initially thought about the 3 languages:

Python

• Pros: Familiar in AI, rich ecosystem
• Cons: Not supported on edge computing platforms, relatively slower execution

Rust

• Pros: Blazing fast, memory-efficient
• Cons: Steep learning curve, smaller talent pool

TypeScript

• Pros: Strong typing, excellent async support, wide adoption, compiles to optimized JavaScript
• Cons: Slightly less performant than Rust in theory

Choosing Cloudflare Workers narrowed down our options to two: Rust or TypeScript. Python, while familiar in the AI world, wasn't a viable option due to our edge computing requirements. Rust was tempting. It's fast, memory-efficient, and would have been the performance king. A part of us, the performance-obsessed engineers, was rooting for Rust.

But then reality kicked in. We looked at our team - a group of brilliant engineers, most of whom were very proficient in JavaScript and TypeScript, but had limited experience with Rust. We thought about the project timeline, the learning curve, and the potential challenges in hiring Rust developers down the line.

That's when it hit us - sometimes, the theoretically "best" option isn't the right choice for your specific situation.

Case in point: here's some recent user comments on a popular agent library, with consistent demands for a TS SDK:

So, TypeScript it is.

Why TypeScript Emerged Victorious

1. Edge Deployment Optimization: TypeScript compiles to highly optimized JavaScript, perfectly suited for Cloudflare Workers' V8 engine. This compilation process allows for aggressive optimizations, resulting in near-native performance.
2. Asynchronous Operations Mastery: TypeScript's async/await syntax and JavaScript's event-driven nature align perfectly with the concurrent request handling required in an AI Gateway. This allows for efficient management of multiple AI model calls and API interactions without blocking the event loop.
3. Type Safety at Scale: As our codebase grew, TypeScript's static typing became invaluable. It caught errors at compile-time that might have slipped through in a dynamically typed language like Python, crucial for a system handling millions of requests.
4. Performance Tuning: While not as low-level as Rust, TypeScript allowed us to implement performance optimizations like object pooling, efficient memory management, and leveraging V8's JIT compiler optimizations.
5. WebAssembly Integration: For performance-critical sections, TypeScript allowed seamless integration with WebAssembly modules, giving us the best of both worlds - the development speed of TypeScript and the raw performance of low-level languages when needed.
6. Rich Ecosystem: We leveraged battle-tested Node.js libraries for everything from routing to monitoring.
7. Open-Source Friendly: TypeScript's popularity and excellent documentation made our project more accessible to contributors and users. It allows contributors and users to easily understand, deploy, and extend the gateway, fostering a more active and engaged community around our project.

The Results: Unmatched Performance

A Success Story

Our TypeScript-powered AI Gateway, running on Cloudflare's edge network, delivers exceptional performance:

• Single-digit millisecond latencies globally: Consistently fast responses worldwide.
• 2B+ requests processed: Battle-tested at scale.
• 99.994% API uptime: Rock-solid reliability.
• 5.8k+ GitHub stars: Strong community validation.

Moreover, our open-source approach has fostered a growing community of contributors and users, further validating our technology choice.

Recent Developments: Guardrails and Beyond

The flexibility of TypeScript has allowed us to rapidly iterate. We've recently added Guardrails directly on the Gateway, enhancing security and control without sacrificing performance. This is just the beginning - we're accelerating  the Gateway's development even further.

Conclusion

While we could have gone with Python (the familiar choice in AI) or Rust (the performance king), choosing TypeScript struck the perfect balance between development speed, performance, and maintainability. It has enabled us to create the fastest, most performant AI Gateway in the world. And importantly, create an open-source project that's accessible to a wide range of developers.

As we continue to evolve our AI Gateway, we're excited about the foundation we've laid with TypeScript. It's a decision that has set us up for long-term success in the dynamic and demanding world of AI infrastructure.

If you're interested in exploring our AI Gateway or contributing to the project, check out our GitHub repository.

We welcome feedback, contributions, and discussions on how we can further improve this critical piece of AI infrastructure!