Architecture Overview

This document provides a high-level overview of the architecture of the Inference Gateway. The gateway is designed to be modular and extensible, so new providers and routing strategies drop in without changing the request surface clients see.

General Overview

A unified OpenAI-compatible request enters the gateway and passes through a middleware pipeline - optional OIDC authentication, optional guardrails, and the MCP tool-call loop - before model routing resolves the target model and the provider proxy dispatches it to whichever upstream provider serves it. A2A agents are clients too: the CLI delegates tasks to them, and they call the same OpenAI-compatible API themselves - the gateway does not proxy the A2A protocol.

Clients

💻Your appOpenAI SDK, curl, or the infer CLI

🤖A2A agentsDelegated by the CLI, and they call the gateway themselvesA2A_SubmitTask -> calendar-agent

POST /v1/chat/completions

🛡️ Inference Gateway

1OIDC authoptional2Guardrailsoptional3MCP tool looptools/call, executed server-side4Model routingalias resolved to an upstream deployment

5Provider proxydispatch and stream back

provider request

🧠LLM providersOpenAI, Anthropic, Groq, Ollama, and more🔌MCP serversCalled from the tool loop, not the proxytools/call filesystem.read

The pipeline order matches the binary: cmd/gateway/main.go registers the auth, guardrails, and MCP middlewares in exactly this sequence before the /v1/* handlers. The gateway is stateless - replicas scale horizontally behind any load balancer, and per-request state (tool-call iteration, MCP context) lives in the request lifecycle, not the process. See Supported Providers for the full provider matrix: OpenAI, DeepSeek, Anthropic, Cohere, Groq, Cloudflare, Ollama, Ollama Cloud, Google, Mistral, MiniMax, Moonshot, and Nvidia.

Kubernetes Setup

The Inference Gateway is built to run on Kubernetes. Traffic enters through the Kubernetes Gateway API - an Envoy Gateway data plane fronting a Service - and reaches a pool of stateless gateway pods, each fronting the same provider proxy. The operator provisions these Gateway API resources from a Gateway CR's gatewayAPI spec (the successor to Ingress); see the Kubernetes Operator guide. Telemetry is scraped on a dedicated metrics port via a ServiceMonitor, and providers stay external.

Clients

💻External clientsReach the cluster through the data plane

💻In-cluster clientsReach the Service directly, skipping the data plane

HTTPS

☸️ Cluster edge

1Gateway APIEnvoy Gateway data plane

2Gateway ServiceClusterIP fronting every pod

load balanced

🛡️ Gateway pods

Gateway Podstateless, HPA-scaled

Gateway Podstateless, HPA-scaled

Gateway Podstateless, HPA-scaled

📊Monitoring stackServiceMonitor + Prometheus + Grafana scrape each podscrape :9464 /metrics

provider API

🧠External LLM providersOutside the cluster, reached over the network

Pods are interchangeable. Add capacity with an HPA, remove pods with rolling updates. The Monitoring Stack here represents the ServiceMonitor + Prometheus + Grafana pipeline kube-prometheus-stack deploys around the gateway - see Observability for the full setup, and the Kubernetes Operator for managing this topology declaratively as Custom Resources.