Route requests to the right LLM for cost, latency & quality
The overview
Around the internet
Public experiences, questions, and discussions.
Cost is definitely the one that keeps me up. We tried running a bunch of different models in parallel to compare outputs and the bill basically tripled overnight. The caching part sounds interesting though, if it actually works well for multimodal stuff that could be a real difference maker.
Thats a really good idea routing and failover are usually what decide whether these setups hold up in production
Hi, I'm using a key from https://fastrouter.ai/ [https://models.dev/?search=fastrouter] as I got some free credits but I'm getting this error now. First time it worked and did some coding. I checked I have ~$4.6 credits still available. Any idea what could be the issue? Are there any restrictions on using custom keys in cursor?
thank you, its working now. I had accidentally set 0.01 limit on key
Inside the toolkit
The OpenAI-compatible API selects a model based on query complexity, domain, and cost efficiency.
Choose a provider by price, latency, or throughput, with prioritized routing and automatic failover.
Group providers and models behind an alias and route requests using a configured selection strategy.
The site lists guardrails, role-based access controls, and evaluations for routed AI workloads.
The gateway lists observability among its capabilities for AI applications.
Where it fits
Keep exploring
