Inkline
Menu

LLM gateway

Templates
LLM gateway 11 items and 10 connections. Applications LLM gateway miss on error cheap tasks Support bot Search answers Batch summaries Router + fallbacks Policy + PII filter Semantic cache Rate limits + budgets Primary provider Fallback provider Self-hosted model Traces + token cost
LLM gateway — preview
Architecture

One gateway in front of every model call: policy and PII filtering, rate limits and budgets, a semantic cache, and routing with fallbacks across providers, with token-cost traces.

What is inside

  • Routing and fallbacks
  • Budgets and semantic cache
  • Provider-neutral
Use this template

Opens in the playground, no account needed. Sign up to save it.

Show the Inkline text
diagram architecture "LLM gateway"
  direction right

  zone apps "Applications" {
    service support "Support bot"
    service search  "Search answers"
    fn      batch   "Batch summaries"
  }

  zone gateway "LLM gateway" {
    service   router  "Router + fallbacks"
    guardrail policy  "Policy + PII filter"
    cache     semantic "Semantic cache"
    service   limits  "Rate limits + budgets"
  }

  llm primary  "Primary provider" use=anthropic/claude
  llm backup   "Fallback provider"
  llm local    "Self-hosted model"
  tracer usage "Traces + token cost"

  support  -> policy
  search   -> policy
  batch    -> policy
  policy   -> limits
  limits   -> semantic
  semantic -> router   : miss
  router   -> primary
  router   -> backup   : on error
  router   -> local    : cheap tasks
  router   ~> usage