Every model call,
priced, routed and traced.
Sentinel sits in front of your model providers and does the operational work: classify the prompt, route it to the cheapest capable model, cache what is safe to cache, retry and fail over when an upstream breaks, then account for every token and trace every hop.
spend avoided
—
versus the premium model
requests routed
—
last 7 days
ttft p95
—
time to first token
gateway overhead
—
excludes upstream time
cache hit rate
—
exact-match cache
connecting to the gateway
the problem
Calling a model is easy. Operating the call is not.
One HTTP request to a provider is an afternoon's work. Knowing what it cost, why that model was chosen, where the three seconds went, and what happens when the provider returns 503 for four minutes — that is the part teams keep rebuilding badly.
Cost is invisible until the invoice
Spend is aggregated per month, per provider, with no line back to the request or the team that made it. Nobody can answer 'what would this have cost on the cheaper model?'
Latency is a single number
Without time-to-first-token separated from generation time and gateway overhead, a slow endpoint has no diagnosis — only a complaint.
Failure is all-or-nothing
No retry budget, no breaker, no failover. One upstream incident becomes your incident, and the retry storm makes it worse.
the pipeline
Seven stages, every one traced.
A single code path handles every call. Each stage emits a span, so the waterfall for any request shows exactly where its milliseconds went and which decision produced its cost.
- 01
Guard
Size caps, output ceilings, and redaction of emails, phone numbers, card-like digits and API-key patterns before anything leaves the process.
- 02
Admit
Token-bucket rate limits per key, a monthly budget check, and a concurrency semaphore that records how long a call waited for a slot.
- 03
Route
A rule-based classifier scores the prompt, then the policy picks the cheapest — or fastest — model at or above the required tier. The reason is stored with the request.
- 04
Cache
Exact-match lookup on a canonical hash of messages and parameters. Only deterministic calls qualify unless the caller opts in. A hit costs nothing and is credited as avoided spend.
- 05
Call
Bounded retries with jittered backoff, ordered failover across providers, and a circuit breaker that stops hammering an upstream that is already down.
- 06
Account
Tokens from the provider when reported, estimated when not. Cost from an editable price book, plus the counterfactual cost of the premium model.
- 07
Observe
W3C spans for every hop, Prometheus counters and histograms, and a live event stream — all emitted whether or not a collector is attached.
The measurable claims: share of spend avoided versus routing everything to the premium model, sustained requests/second before time-to-first-token breaches its objective, and the added gateway overhead of recording spans. All three are computed by the gateway from its own traffic, not estimated in a slide.
architecture
One container, or a fleet.
No broker, no Redis, no sidecar required. It starts on SQLite with a deterministic local engine and grows into Postgres, real upstreams and an OTLP collector without touching application code.
Degrades honestly
No provider keys? The local engine serves four priced tiers with configurable time-to-first-token, throughput and failure rate — which is what makes the load test and CI reproducible. Add a key and that provider joins the routing pool immediately.
data plane · /v1
pipeline
control plane · /api
state & egress
The span buffer and the rate-limit buckets are per process. That is a deliberate trade for a single-container deployment and a documented limitation for a fleet: a shared bus and a Redis counter are the fix, and neither is pretended away.
capabilities
What is actually built.
Routing you can interrogate
Cheapest-capable, latency-first, weighted A/B, ordered failover and shadow traffic. Every decision keeps its candidate set, estimated costs and the sentence explaining the pick.
Cost accounting per request
Tokens and dollars against a price book you edit in the console, including prompt-cache-hit rates. The premium-baseline counterfactual turns 'we optimised spend' into a percentage.
Tracing with a measured price tag
Spans persisted locally for the built-in waterfall, mirrored to OTLP when configured. The load test reports what recording them actually costs instead of asserting it is free.
Resilience that is visible
Per-provider circuit breakers, retry budgets, upstream timeouts and failover chains — with the attempt sequence stored on the request so a failure is explainable after the fact.
Multi-tenant metering
Hashed keys with their own RPM/TPM limits and monthly budgets. Spend is attributed at request time, so a key that exhausts its budget stops rather than surprises you.
OpenAI-compatible surface
Drop-in /v1/chat/completions with streaming. Existing SDKs point at it unchanged; the extra routing fields and the response's sentinel block are purely additive.
Policy at runtime
Cache, tracing, retries, breaker thresholds, limits and service objectives are rows the hot path reads — changed from the console, effective on the next request.
Load testing built in
Ramp concurrency against a deterministic local engine and get sustained throughput, TTFT percentiles and the queueing point that an autoscaling threshold should be set from.
stack
Boring dependencies, deliberately.
- Gateway
- FastAPI · async SQLAlchemy · httpx · Pydantic v2
- Store
- Postgres or SQLite · no broker, no Redis required
- Upstreams
- DeepSeek · any OpenAI-compatible endpoint · deterministic local engine
- Console
- Next.js 16 App Router · TypeScript · Tailwind v4 · Recharts
- Observability
- Prometheus · W3C spans · optional OTLP/HTTP export · SSE
- Delivery
- Docker · compose · Kustomize + KEDA · GitHub Actions
Send a prompt and watch the decision.
The playground calls the same public endpoint your services would. Every answer carries the model that served it, why it was chosen, what it cost against the premium baseline, and a trace id you can open into a waterfall.