GPT-4 is Getting Faster 🐇

Over the past few months, we've been keenly observing latencies for both GPT 3.5 & 4. The emerging patterns have been intriguing.

The standout observation? GPT-4 is catching up in speed, closing the latency gap with GPT 3.5.

Our findings reveal a consistent decline in GPT-4 latency. While your mileage might vary based on the specific prompts, the general trend is clear, as illustrated in the following plot showcasing median requests over a three-month period.

But what factors contribute to latency? Let's break it down:

• Round trip time: Typically around 100-200ms
• Queuing time: Ranges between 100-500ms
• Processing time: This can vary significantly based on the complexity and length of your prompt

It's worth noting that a high token count doesn’t always translate to a slower response. For instance, a prompt with 204 tokens, despite being simple, can receive a response in a brisk 4.5 seconds. On the other hand, a 33-token prompt, if complex, might take a whole 32 seconds to process.

So, we observed Latency/Total Tokens ratios for median and 99 percentile requests.

Median request latencies remain consistent across both models, staying under 1 ms per token.

However, the real surprise comes with the 99th percentile, where latencies have more than halved in just three months!

GPT-4 is costlier still, but it’s not slower anymore for majority of your requests. 🐢

We're also exploring another intriguing aspect: Does your latency increase as you get near your rate limits? i.e. does OpenAI deliberately slow you down?

Stay tuned for our next deep dive!