Portkey can help you easily create fallbacks from one LLM to another, making your application more reliable. While Fallback ensures reliability, it also means that you’ll be running a prompt optimized for one LLM on another, which can often lead to significant differences in the final output.
portkey
instance to trigger a prompt completions API.
GPT
4 was with the following prompt with default model parameters (based on satisfactory response in the playground):
System:You are a productivity expert. When given a task, you can smartly suggest possible subtasks. You list the subtasks in less than 10 items, keeping each short and actionable.User:The following is the goal I want to achieve:I want to become fit in 6 months | GPT4:1. Visit a doctor for a health check-up.2. Set specific fitness goals (like weight loss, strength, etc)…9. Stay hydrated and make adjustments as required. | Claude:Here are some suggested subtasks to help you achieve six-pack abs in six months:1. Develop a balanced and nutritious meal plan that focuses on lean proteins, vegetables, and healthy fats while limiting processed foods and sugary drinks.2. Create a sustainable calorie deficit by tracking your daily food intake and ensuring you burn more calories than you consume…9. Stay motivated by setting short-term goals, rewarding yourself for progress, and seeking support from friends, family, or a fitness coach. |
---|
claude-3-opus-20240229
model to instruct it to generate sub-tasks for an user’s goal. You can declare an variable using moustache syntax to substitute an value when prompt is triggered. For example, {{goal}}
is substituted with “I want to earn six packs in six months” in the playground.
Now, create another prompt template that can act as a fallback.
You can create the same prompt this time but use a different model, such as gpt-4
. You have created two prompt templates by now. You must have noticed each prompt has a slightly different system
message based on the model. After experimenting with each model, the above prompt was best suited for suggesting actionable steps to reach the goal.
The models on this page require you to save OpenAI and Anthropic API keys to the Portkey Vault. For more information about Portkey Vault, read more on Virtual Keys.
For further exploration, Try learning about OpenAI SDK to work with Prompt Templates.
targets
is an array of objects ordered by preference in favor of Anthropic and then on to OpenAI.
Pass these config
s at instance creation from Portkey
portkey
will have the context of above gateway configs for every request sent through portkey.
Read more about different ways to work with Gateway Configs.
promptID
invokes the prompt template you want to trigger on a prompt completions API. Since we already pass the gateway configs as an argument to the configs
parameter during client instance creation, the value against the promptID
is ignored, and task_to_subtasks_anthropic
will be treated as the first target where requests will routed to, then fallback to task_to_subtasks_openai
as defined in the targets
.
Notice how variables
hold the information to be substituted in the prompt templates at runtime. Also, even when the promptID
is valid, the gateway configs will be respected in precedence.
See the reference to learn more.
weight
s. As an outcome, you have fewer chances of hitting the rate limits and not overwhelming the models.
Here is how you can update the gateway configs:
See the full code: