Benchmarks
Over the past few months, we've been keenly observing latencies for both GPT 3.5 & 4. The emerging patterns have been intriguing. The standout observation? GPT-4 is catching up in speed, closing the latency gap with GPT 3.5. Our findings reveal a consistent decline in GPT-4
paper summaries
This paper presents a method for compressing prompts in large language models (LLMs) to accelerate model inference and reduce cost. The method involves a budget controller, a token-level iterative compression algorithm, and an instruction tuning based method for distribution alignment. Experimental
LLaMA2
Llama 2 is an open-source large language model (LLM) developed by Meta. See Llama 2's capabilities, comparisons, and how to run LLAMA 2 locally using Python.
notes
As developers and founders, you might find yourself asking how does your startup differentiate from ChatGPT. More so, how do you convince a customer to try your product over a generic Chat client.
Fundraise
Portkey is building a full-stack LLMOps platform that empowers AI builders to productionize their Gen AI apps reliably and securely.
paper summaries
This paper introduces the Skeleton-of-Thought (SoT) method to decrease the generation latency of large language models (LLMs). SoT guides LLMs to first generate the skeleton of the answer and then conducts parallel API calls or batched decoding to complete the contents of each skeleton point. The m
ChatGPT
As I reflect upon my last 1 year coding journey, I am struck by how deeply Artificial Intelligence (AI) has woven itself into my development practices. From the likes of GitHub Copilot to the genius behind ChatGPT, my approach to writing code has undergone a transformative shift. Allow me to
features
Portkey's analytics 2.0 give our users complete visibility into their LLM calls across requests, users, errors, cache and feedback.
Production Guides
In this blog post, we explore a roadmap for building reliable large language model applications. Let’s get started!
Production Guides
Implementing semantic cache from scratch for production use cases.
Conversations
Rohit from Portkey is joined by Weaviate's Research Scientist Connor where they go on a deep dive about the differences between MLOps and LLMOps, building RAG systems, and what lies ahead for building production-grade LLM-based apps. This and much more in this podcast! Rohit Agarwal on Portkey -
Conversations
Portkey CEO Rohit Agarwal shares practical tips from his own experience on crafting production-grade & reliable LLM systems. Read more LLM reliability tips here.