together-ai
#10151
input_validation_error
Input validation error: The sum of 'inputs' tokens and 'max_new_tokens' must not exceed 4097. Please adjust your inputs accordingly.
This error has been identified and solved.
Reason
The 400 status error you are encountering in the TogetherAI API, specifically the "Input validation error: inputs tokens + max_new_tokens must be <= 4097," is due to the following reasons:
The total number of tokens in your input and the maximum number of new tokens requested exceeds the API's limit. In your case, the sum of
inputstokens (5013) andmax_new_tokens(128) is 5141, which is greater than the allowed limit of 4097 tokens.This error indicates a violation of the API's defined schema and token limits, which are designed to manage and restrict the size of the input and output to prevent excessive resource usage.
Solution
To fix the "Input validation error: inputs tokens + max_new_tokens must be <= 4097" in the TogetherAI API, you need to adjust the input and the maximum new tokens to stay within the limit. Here are some steps to resolve this issue:
Summarize or truncate the input: Reduce the length of the input text to lower the number of tokens.
Chunk the input: Break down large inputs into smaller chunks and process them separately.
Optimize prompts: Make your prompts more concise to reduce the token count.
Adjust
max_new_tokens: Lower the value ofmax_new_tokensif possible, to stay within the total token limit.
By implementing these strategies, you can ensure that the total tokens do not exceed the API's limit of 4097.
Suggested Links
https://portkey.ai/collection/input-error-10148
https://huggingface.co/spaces/huggingchat/chat-ui/discussions/430
https://appwrite.io/docs/products/ai/integrations/togetherai
https://community.openai.com/t/error-code-400-max-token-length/716391
https://github.com/evo-design/evo/issues/95
https://community.openai.com/t/fixed-prompt-token-limit-exceeded-error-for-long-conversation-gpt-3-and-gpt-4/294077
https://cheatsheet.md/chatgpt-cheatsheet/openai-api-token-limit
https://github.com/huggingface/transformers/issues/28523
https://github.com/langgenius/dify/issues/2383
