Open
Description
Description
When making requests to the Anthropic API, responses can sometimes take too long, especially when the model is overloaded. This results in wasted time before a retry can occur.
To improve efficiency, we should set a timeout so that if a response is not received within a specific threshold (e.g., 15 seconds), the request is canceled, allowing for a faster fallback to another model.
Proposed Changes
- Use the native timeout parameter in the Anthropic SDK’s built-in timeout (via httpx) to enforce a time limit on responses.
- For example, setting a 5-second connection timeout and a 15-second read timeout ensures that if any token or chunk takes longer than 15 seconds, the request will time out.
import httpx
from anthropic import Anthropic
custom_timeout = httpx.Timeout(connect=5.0, read=15.0)
client = Anthropic(timeout=custom_timeout)
This approach helps prevent long waits and ensures more efficient retries.
Metadata
Assignees
Labels
No labels
Activity