Model Gateway, Routing & Data Residency

TL;DR: Configured clients send AI traffic through one provider-neutral gateway, on /v1/messages (Anthropic Messages), /v1/chat/completions (OpenAI Chat) and /v1/responses (OpenAI Responses). Each model is declared once and served by an ordered list of deployments, which is both its routing and its failover. Claude is served only from Anthropic's own API, on the customer's key, and every request records which endpoint actually answered. A model that is neither declared nor a route alias is refused. Every route is open to every signed-in role. The residency and no-retain requirements are a working mechanism, but no route declares one.

One gateway, any provider

The gateway accepts three client wire formats and sends to each provider in that provider's own format:

Endpoint	Client format	Typical client
POST /v1/messages	Anthropic Messages	Claude Code, the Anthropic SDKs
POST /v1/chat/completions	OpenAI Chat Completions	OpenCode, OpenAI-compatible tools
POST /v1/responses	OpenAI Responses	Responses-API clients

The AI catalog is five files under services/ai/, each with one job:

File	Declares
credentials.yaml	who authenticates and who pays: platform (a profile secret) or personal (each member stores their own key)
providers.yaml	the vendor account (headers, user agent, governance) and its endpoints:, each one wire (anthropic, openai-chat, gemini) plus one URL
models.yaml	each model once: capabilities, limits, base price, upstream_model, and the clients: families that list it (clients: [] is never advertised)
deployments.yaml	model to ordered endpoints (provider/endpoint); order is failover order
gateway.yaml	the one default_model, and routes that are policy only: alias, pin or narrow, each with a declared id

Dispatch resolves model, then deployment, then endpoint, then provider, then credential, with no pattern matching. Request and wire details are in the Gateway API.

What this instance serves

The catalog ships inside the image, so every environment boots the same set.

Model	Deployments, in order	Credential
claude-opus-5-5, claude-sonnet-5-5, claude-haiku-5-5	anthropic/api	anthropic (platform)
deepseek-v4-pro	deepseek/api, then opencode-go/chat	deepseek (platform), then opencode-go (personal)
deepseek-flash	deepseek/api	deepseek (platform)
go-*	opencode-go/chat or opencode-go/messages	opencode-go (personal)

Claude is served only by the customer's Anthropic API key. Claude hosts list the claude-* models; OpenCode lists DeepSeek and the Go models. The one route, go-deepseek-v4-pro, is an alias for deepseek-v4-pro pinned to opencode-go/chat, so a member can ask for DeepSeek V4 Pro on their own Go key. Failover never crosses from a platform key to a member's key.

The Claude models, at Anthropic's rates:

Model	Input / output per million	Context	Max output
claude-opus-5-5	$4.00 / $20.00	1M	128k
claude-sonnet-5-5	$2.00 / $10.00	1M	128k
claude-haiku-5-5	$0.10 / $0.50 (≤100K prompt; $0.50 / $2.50 beyond)	1M	128k

gateway.default_model is claude-sonnet-5-5. Claude Code's [1m] 1M-context marker is never written in config: the bridge derives it from the model's limits.context_window and seeds it into Claude Code, and the gateway strips it before routing, so the request reaches Anthropic as claude-sonnet-5-5. Without the marker, Claude Code budgets 200k for every gateway model. Only the latest Claude models are declared, and a request for a retired id is refused rather than forwarded.

Provider failover

A model's entry in deployments.yaml is its failover chain: the first endpoint is tried first, and the rest follow in order. Each entry is either provider/endpoint or an object with endpoint: plus an optional upstream_model and pricing override for that deployment. The shipped Claude models have one deployment each: Anthropic's API is their only upstream. deepseek-v4-pro has two:

deployments:
  deepseek-v4-pro:
  - deepseek/api
  - endpoint: opencode-go/chat
    pricing:
      cache_read_per_million: 0.132

Per-project chains. A route with a when: scope narrows the deployments for requests attributed to that scope, for example a project's own endpoint. The following is an illustration, not the shipped configuration:

gateway:
  routes:
  - id: project-odoo-delivery
    when: { scope: { project: odoo-delivery } }
    models: claude-*
    deployments: [anthropic/api]

The project is the one the gateway attributed the request to: the x-systemprompt-scope-project header, the API key's bound project, or the person's primary project. The audit row's route match records scope:project=<id>.

When it triggers. The primary keeps its bounded retry budget for transient answers (429, 503). Failover happens when that budget is spent, when the primary answers any other 5xx, or when the connection fails (DNS, TLS, timeout). The same governed request is then re-sent to the next deployment in the chain, under that deployment's own retry budget, and so on down the list. A 4xx other than 429 never fails over, because the fault is in the request, not the provider. A per-provider circuit breaker skips a primary that keeps failing and tries the fallback first; when every deployment is tripped, all are tried anyway, so a request is never left unsent.

What is re-sent. The request that governance and the safety scanners already judged is rebuilt in the fallback's wire format. Neither plane runs again, because the content has not changed.

What the audit shows. The ai_requests row keeps provider as the route's primary and records the provider that answered in served_provider. The request is repriced at that provider's rate, and the route-match descriptor gains a failover:<primary>-><fallback> segment. systemprompt infra logs audit <request-id> prints both.

route_match contains	Meaning
failover:deepseek->opencode-go	the primary failed and the next deployment answered (a->b->c for a longer chain)
scope:project=<id>	the request was attributed to that project and served by a route scoped to it
The Prometheus
counter gateway_upstream_failovers_total{from,to,reason} (reason is
status_<code>, transport or circuit_open) is on /metrics.

What boot checks. Every deployment must name a declared endpoint, appear once in its model's list, and satisfy any requires: block on a route that serves the model. If a fallback cannot be used at request time, the client gets the primary's own error, as it would with no fallback declared.

Selection strategy

A route can set strategy to choose which deployment a request tries first. Failover after that first attempt works as described above.

strategy	First attempt
ordered (default)	The order in deployments.yaml, or the route's own deployments: list when it pins one.
weighted	Drawn by each deployment's weight (default 1) among deployments whose circuit breaker is closed. The rest follow in descending weight.
least_busy	The healthy deployment with the fewest requests in flight on this server process.

weight sits on each entry of the route's deployments: list ({endpoint: provider/endpoint, weight: 3}). weight: 0 is refused at load. A weight under ordered is ignored. A non-ordered strategy adds strategy:<name> to the request's route_match, and gateway_deployment_selected_total{route,provider,strategy} counts every selection. This instance keeps the default ordered.

Context-window check

Before any upstream call, the gateway estimates the request's input tokens from its text and compares the estimate with the selected model's limits.context_window in models.yaml. Images and tool inputs are not counted, so the estimate errs low and never refuses a request that fits.

• It fits. The request is sent as normal. Failover deployments whose declared window is too small are skipped.
• It does not fit, and the route lists context_fallbacks. The request goes to the first context fallback whose window fits. route_match records context_window:a->b.
• It does not fit, and no fallback fits. The request is refused with a 400 in the caller's API format, with error_key: context_window_exceeded naming the estimate, the limit and the model. Nothing is sent upstream, so nothing is billed.

A model with no declared window is never refused. An oversized max_tokens is clamped to the model's output ceiling rather than refused.

Model access control

Model access is an access-control entity, gateway_route/*, declared in services/access-control/rules.yaml and administered at /admin/access-control. The mechanism grants or denies routes per role, group or project, and a request for a route the caller is not entitled to gets HTTP 403 (authz denied [<policy>]: <reason>), and the denial is audited.

As configured, every route is open to every signed-in role. Usage is shaped by governance and quota windows, not by entitlement. To restrict a model, declare a rule for its route with an allow band and a reason. A route's id is declared in gateway.yaml (for example go-deepseek-v4-pro), and the console shows its label:.

/admin/gateway has four tabs: Overview (dispatch order and provider health), Providers (each provider with its endpoints, models and deployments), Routes (the ordered dispatch table) and Settings. Policies is a fifth tab that opens its own page. The route editor there writes back to gateway.yaml, and a saved route is dispatched after the next restart.

Shadow-AI posture

The gateway forwards only declared models and route aliases, so a model that is not configured cannot be called by anyone. Traffic that never reaches the gateway, such as someone calling a provider directly from a laptop, is a network-control matter for corporate IT. The usual control is an egress allow-list that lets only the gateway reach provider APIs.

Latency SLOs

The analytics Overview tab reports p50 and p95 latency and the share of requests over an SLO threshold, per model. The threshold is a view setting (?slo_ms=, 5 s by default, clamped to 500 ms–60 s) and is not stored anywhere. There is no SLO alerting. Per-use-case SLO targets and error-budget alerts are on the roadmap.

Quota windows

Quota windows are warning thresholds on this instance (quota_mode: warn). They record breaches and never refuse. See Cost Management and Governance Modes.

Private and self-hosted endpoints

A private endpoint that speaks the OpenAI or Anthropic format is a configuration entry, not a code change. A provider declared with the model declared with clients: [] is served without appearing in any model list, and every deployment is validated at boot. None is configured on this instance.

Data classification & residency

Each provider, and optionally each model, carries a governance: block with two booleans: european (data stays in the EU) and no_retain (the provider contractually does not retain prompts or completions). A route states what it requires with a requires: block. The following is an illustration, not the shipped configuration:

gateway:
  routes:
  - id: claude-no-retain
    models: claude-*
    requires:
      no_retain: true

Enforcement happens twice:

• At boot: the server refuses to start a route whose provider or model does not satisfy its requires: block.
• At dispatch: a request that would reach a provider that does not satisfy the block is denied and audited, with the failed condition named in the route-match descriptor.

What is declared today:

• Every provider is no_retain: true and european: false.
• No route declares requires:, so neither flag is enforced at dispatch, and there is no EU route.

To enforce no-retain, add requires: {no_retain: true} to a route. An EU guarantee needs an EU-located provider entry first. The flags are declarations, so keep them in step with the provider agreements; they do not prove physical residency on their own. Classifying the data itself, rather than the route, is on the roadmap.

Check it yourself

• /admin/gateway?tab=routes: the dispatch order, with each route's provider and fallback.
• /admin/gateway?tab=providers: each provider's endpoint, wire and models.
• /admin/analytics: the Overview tab's latency and SLO breach share, and the Models tab.
• /admin/requests: open a request to see provider and served_provider.
• systemprompt infra logs request list --since 1h