LLM gateway
TemplatesArchitecture
One gateway in front of every model call: policy and PII filtering, rate limits and budgets, a semantic cache, and routing with fallbacks across providers, with token-cost traces.
What is inside
- Routing and fallbacks
- Budgets and semantic cache
- Provider-neutral
Use this template
Opens in the playground, no account needed. Sign up to save it.
Show the Inkline text
diagram architecture "LLM gateway"
direction right
zone apps "Applications" {
service support "Support bot"
service search "Search answers"
fn batch "Batch summaries"
}
zone gateway "LLM gateway" {
service router "Router + fallbacks"
guardrail policy "Policy + PII filter"
cache semantic "Semantic cache"
service limits "Rate limits + budgets"
}
llm primary "Primary provider" use=anthropic/claude
llm backup "Fallback provider"
llm local "Self-hosted model"
tracer usage "Traces + token cost"
support -> policy
search -> policy
batch -> policy
policy -> limits
limits -> semantic
semantic -> router : miss
router -> primary
router -> backup : on error
router -> local : cheap tasks
router ~> usage