Get the LLM to adhere to your JSON schema using Anyscale & Together AI's newly introduced JSON modes
LLMs excel at generating creative text, but production applications demand structured outputs for seamless integration. Instructing LLMs to only generate the output in a specified syntax can help make their behaviour a bit more predictable. JSON is the format of choice here - it is versatile enough and is widely used as a standard data exchange format.
Several LLM providers offer features that help enforce JSON outputs:
OpenAI has a feature called JSON mode that ensures that the output is a valid JSON object.
While this is great, it doesn't guarantee adherence to your custom JSON schemas, but only that the output IS a JSON.
Anyscale and Together AI go further - they not only enforce that the output is in JSON but also ensure that the output follows any given JSON schema.
Using Portkey, you can easily experiment with models from Anyscale & Together AI and explore the power of their JSON modes:
import Portkey from'portkey-ai';constportkey=newPortkey({ apiKey:"PORTKEY_API_KEY",// OR "TOGETHER_VIRTUAL_KEY"})asyn functionmain(){constjson_response=awaitportkey.chat.completions.create({ messages: [{role:"user",content:`Give me a recipe for making Ramen, in JSON format`}], model:"mistralai/Mistral-7B-Instruct-v0.1", response_format: { type:"json_object", schema: { type:"object", properties: { title: { type:"string" }, description: { type:"string" }, steps: { type:"array" } } } } });}console.log(json_response.choices[0].message.content);main()
from portkey_ai import Portkeyfrom pydantic import BaseModel, Fieldportkey =Portkey( api_key="PORTKEY_API_KEY", virtual_key="ANYSCALE_VIRTUAL_KEY"# OR #TOGETHER_VIRTUAL_KEY)classRecipe(BaseModel): title:str description:str steps: List[str]json_response = portkey.chat.completions.create( messages = [{ "role": 'user', "content": 'Give me a recipe for making Ramen, in JSON format' }], model ='mistralai/Mistral-7B-Instruct-v0.1', response_format = {"type":"json_object","schema": Recipe.schema_json() })print(json_response.choices[0].message.content)
Output JSON:
{"title":"Fried Rice with Chicken and Vegetables", "description": "A delicious recipe for fried rice that includes chicken and a mix of colorful vegetables. Perfect for a healthy and satisfying meal. yum yum yum yum yum",
"steps": ["1. Cook rice and set aside","2. In a large skillet or wok, sauté sliced chicken in oil until browned","3. ..." ]}
As you can see - it's pretty simple. Just define the JSON schema, and pass it at the time of making your request using the response_format param. The response_format's type is json_object and the schema contains all keys and their expected type.
Supporting Models
Model/Provider
Ensure JSON
Ensure Schema
mistralai/Mistral-7B-Instruct-v0.1
Anyscale
✅
✅
mistralai/Mixtral-8x7B-Instruct-v0.1
Anyscale
✅
✅
mistralai/Mixtral-8x7B-Instruct-v0.1
Together AI
✅
✅
mistralai/Mistral-7B-Instruct-v0.1
Together AI
✅
✅
togethercomputer/CodeLlama-34b-Instruct
Together AI
✅
✅
gpt-4 and previous releases
OpenAI / Azure OpenAI
✅
❌
gpt-3.5-turbo and previous releases
OpenAI / Azure OpenAI
✅
❌
Ollama models
✅
❌
Creating Nested JSON Object Schema
Here's an example showing how you can also create nested JSON schema and get the LLM to enforce it:
classIngredient(BaseModel): name:str quantity:strclassRecipe(BaseModel): title:str description:str ingredients: List[Ingredient] steps: List[str]json_response = portkey.chat.completions.create( messages = [{ "role": 'user', "content": 'Give me a recipe for making Ramen, in JSON format' }], model="mistralai/Mistral-7B-Instruct-v0.1", response_format={"type": "json_object", "schema": Recipe.schema_json() })
Add your Anyscale or Together AI virtual keys to Portkey vault, and get started!