Kubernetes Operator

The Inference Gateway Operator is a Kubernetes controller that manages Inference Gateway and related resources declaratively through Custom Resources (CRs). It is the recommended way to run Inference Gateway on Kubernetes - manage gateways, A2A agents, MCP servers, and chat-channel orchestrators as first-class CRs in the same cluster API your other workloads use.

The operator publishes five CRDs under core.inference-gateway.com/v1alpha1:

• Gateway - the gateway proxy itself, with providers, auth, MCP, model routing, Gateway API routing, and HPA.
• Agent - an A2A worker that an Orchestrator (or any A2A client) can dispatch tasks to.
• MCP - a Model Context Protocol server.
• Orchestrator - runs a chat bot backed by the gateway. It listens on a messaging channel (Telegram today; more channels planned), drives the conversation with an LLM, and can delegate work to Agents and MCP tools.
• GPU - leases an externally hosted, GPU-backed inference runtime from a cloud provider.

The API is v1alpha1 and breaking changes can land between releases.

Installation

The operator is distributed as pre-rendered manifests on each GitHub release. The install.yaml artifact bundles the namespace, CRDs, RBAC, and controller deployment.

The operator also requires the Kubernetes Gateway API standard-channel CRDs - even when Gateway API routing is never used. It watches gateway.networking.k8s.io/v1 Gateway and HTTPRoute resources, so without those CRDs the manager fails its cache sync and the pod CrashLoopBackOffs. install.yaml does not bundle them; install separately:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.1/standard-install.yaml

kubectl apply -f https://github.com/inference-gateway/operator/releases/latest/download/install.yaml

For production, pin to a release rather than latest (the current release is v0.27.0):

kubectl apply -f https://github.com/inference-gateway/operator/releases/download/v0.27.0/install.yaml

GitOps

For GitOps (ArgoCD, Flux), point your source at the operator repository's manifests/ directory at a tagged ref - it contains the same install.yaml and a CRD-only crds.yaml for split installs.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: inference-gateway-operator
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/inference-gateway/operator
    targetRevision: v0.27.0
    path: manifests
  destination:
    server: https://kubernetes.default.svc
    namespace: inference-gateway-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

The equivalent Flux source:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: inference-gateway-operator
  namespace: flux-system
spec:
  interval: 1h
  url: https://github.com/inference-gateway/operator
  ref:
    tag: v0.27.0
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: inference-gateway-operator
  namespace: flux-system
spec:
  interval: 10m
  prune: true
  sourceRef:
    kind: GitRepository
    name: inference-gateway-operator
  path: ./manifests
  targetNamespace: inference-gateway-system

At a release tag, manifests/install.yaml references that release's operator image, so pinning targetRevision (ArgoCD) or ref.tag (Flux) pins the operator version as well - no need to fall back to the release asset for a pinned install. On main the manifest tracks the most recent release.

Verification

kubectl get pods -n inference-gateway-system
kubectl get crd | grep inference-gateway.com

# View operator logs
kubectl logs -n inference-gateway-system deployment/operator-inference-gateway -f

Expected output - the operator pod Running with a single operator container (1/1) and all five CRDs:

NAME                                          READY   STATUS    RESTARTS   AGE
operator-inference-gateway-74c9c5f5b-x4d2k    1/1     Running   0          2m

agents.core.inference-gateway.com          2025-06-21T17:30:00Z
gateways.core.inference-gateway.com        2025-06-21T17:30:00Z
gpus.core.inference-gateway.com            2025-06-21T17:30:00Z
mcps.core.inference-gateway.com            2025-06-21T17:30:00Z
orchestrators.core.inference-gateway.com   2025-06-21T17:30:00Z

Namespace scoping

The shipped operator Deployment sets WATCH_NAMESPACE_SELECTOR=inference-gateway.com/managed=true, so Gateway, Agent, MCP and GPU resources are only reconciled in namespaces carrying that label. In an unlabeled namespace the operator just logs skipping gateway, namespace does not match WATCH_NAMESPACE_SELECTOR and creates nothing. Orchestrator resources are not namespace-filtered.

Label every namespace you deploy resources into:

kubectl label namespace <namespace> inference-gateway.com/managed=true

To change the scope, edit WATCH_NAMESPACE_SELECTOR on the operator Deployment - any valid label selector works, and an empty value makes the operator watch all namespaces:

kubectl set env -n inference-gateway-system deployment/operator-inference-gateway WATCH_NAMESPACE_SELECTOR=

The Gateway, Agent, MCP and GPU examples on this page assume their namespace is labeled accordingly.

Custom Resources

Gateway

Deploys the gateway proxy. Source: api/v1alpha1/gateway_types.go.

Field	Description
replicas	Pod count (1-100, default 1).
image	Container image, default ghcr.io/inference-gateway/inference-gateway:latest.
environment	One of development, staging, production (default production).
server.port / server.host / server.timeouts / server.tls	HTTP server settings. See Gateway Server (spec.server).
auth.enabled / auth.provider / auth.oidc	Authentication. provider is oidc, jwt, or basic. See Authentication (OIDC).
providers[]	Each item: name, enabled, and an env list of corev1.EnvVar. Provider keys are passed through unchanged.
telemetry.enabled / telemetry.metrics.{enabled,port}	OpenTelemetry metrics. metrics.port configures the gateway process - see Gateway Server (spec.server).
telemetry.traces.exporter.otlp.endpoint	Tracing. With telemetry.enabled: true, it sets TELEMETRY_TRACING_ENABLED=true and TELEMETRY_TRACING_OTLP_ENDPOINT on the gateway container.
mcp.enabled / mcp.expose / mcp.resourceUrl / mcp.toolMode / mcp.timeouts	MCP client configuration. See MCP Servers (spec.mcp).
mcp.servers[] / mcp.serviceDiscovery	Static MCP servers (name, url, healthCheck) and discovery of MCP CRs by label selector. Both feed MCP_SERVERS.
a2a.enabled / a2a.resourceUrl / a2a.timeouts / a2a.cardRefreshInterval	Turn the gateway into an A2A server. See A2A Agents (spec.a2a).
a2a.agents[] / a2a.serviceDiscovery	Static agents (name, url) and discovery of Agent CRs by label selector. Both feed A2A_AGENTS.
service.{type,port,annotations}	Kubernetes Service for the gateway. See Gateway Service (spec.service).
routing.{enabled,config,configMapRef}	Gateway-native round-robin model routing (ROUTING_ENABLED / ROUTING_CONFIG_PATH). Distinct from gatewayAPI. See Model Routing.
gatewayAPI.{enabled,gateway,httpRoute}	North-south traffic via the Kubernetes Gateway API (gateway.networking.k8s.io). Successor to the removed ingress field. See Routing (Gateway API).
hpa.{enabled,config}	Wraps a HorizontalPodAutoscalerSpec. See the Kubernetes HPA docs for the metrics[] and behavior shape.
guardrails.{enabled,failMode,externalUrl,externalTimeout,configMapRef}	OPA/Rego policy enforcement. See Guardrails (spec.guardrails).
serviceAccount.{create,name}	Pod service account.
resources.requests / resources.limits	CPU and memory.

Provider env vars referenced via Secrets follow the standard valueFrom.secretKeyRef pattern - see Configuration for the full list of variables each provider accepts.

Gateway Server (spec.server)

spec.server.host, spec.server.port and spec.telemetry.metrics.port configure the gateway process itself, not just the Deployment ports, probes and Service. The controller emits them as SERVER_HOST, SERVER_PORT and TELEMETRY_METRICS_PORT on the gateway container, so any value in the CRD's 1024-65535 range works - not only the defaults:

apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
  name: my-gateway
  namespace: inference-gateway
spec:
  server:
    host: 0.0.0.0
    port: 9000
  telemetry:
    enabled: true
    metrics:
      enabled: true
      port: 9465

Field	Env var	Default	Notes
server.host	SERVER_HOST	0.0.0.0	Bind address. Keep the default - a loopback address makes kubelet probes fail.
server.port	SERVER_PORT	8080	Also used for the http container port, the liveness and readiness probes, and the Service.
telemetry.metrics.port	TELEMETRY_METRICS_PORT	9464	Emitted only when both telemetry.enabled and telemetry.metrics.enabled are true.

Operators older than the release that wired these variables up only shaped the Deployment and Service, so a non-default port left the gateway listening on 8080/9464 while the probes targeted the new port and the pod never became Ready.

TLS (spec.server.tls)

With spec.server.tls.enabled: true the controller mounts the certificate material into the pod and points SERVER_TLS_CERT_PATH and SERVER_TLS_KEY_PATH at the mounted files. The gateway then also exposes the https container port 8443.

certificateRef and keyRef are both SecretKeySelectors (name plus key). Each is mounted from its own Secret, under its own directory:

spec:
  server:
    tls:
      enabled: true
      certificateRef:
        name: my-certs
        key: server.pem
      keyRef:
        name: my-keys
        key: server-key.pem

SERVER_TLS_CERT_PATH=/app/tls/cert/server.pem
SERVER_TLS_KEY_PATH=/app/tls/key/server-key.pem

When a reference is unset, the controller falls back to the conventional inference-gateway-tls Secret mounted at /app/tls with the standard tls.crt / tls.key keys - the shape cert-manager produces:

spec:
  server:
    tls:
      enabled: true

SERVER_TLS_CERT_PATH=/app/tls/tls.crt
SERVER_TLS_KEY_PATH=/app/tls/tls.key

The fallback Secret must exist in the Gateway's namespace, otherwise the pod stays pending on the missing volume. Setting only one of the two references mounts both that reference and the fallback Secret, so the unset side keeps its /app/tls path.

Gateway Service (spec.service)

The operator creates a Service for each Gateway. type defaults to ClusterIP; use LoadBalancer (or NodePort) to expose the gateway outside the cluster, passing cloud load balancer options via annotations:

apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
  name: my-gateway
  namespace: inference-gateway
spec:
  service:
    type: LoadBalancer
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
      service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing

These fields require operator v0.27.0 or later - older operators silently ignore type and annotations and create the Service as the API server default (ClusterIP, no annotations).

annotations are authoritative on the operator-owned Service: the reconcile loop matches the Service's annotations to spec.service.annotations exactly, so removing a key from the spec removes it from the Service.

Agent

Deploys an A2A worker. Agents are dispatched to by an Orchestrator, by a Gateway with spec.a2a enabled, or by any other A2A client. The agent typically calls back into the gateway for its own LLM completions via agent.llm.baseURL. Source: api/v1alpha1/agent_types.go. See A2A Integration for protocol background.

Field	Description
image	Required agent container image.
port / host / readTimeout / writeTimeout / idleTimeout	HTTP server settings (defaults: 8080, 0.0.0.0, 30s, 30s, 60s).
logging.{level,format}	Defaults: info, json.
tls.{enabled,secretRef}	TLS for the agent's HTTP server.
agent.maxConversationHistory / agent.maxChatCompletionIterations / agent.maxRetries	LLM loop limits (defaults 10, 5, 3).
agent.llm.baseURL	LLM endpoint (typically the gateway URL).
agent.llm.model	provider/model format. The prefix is split out as A2A_AGENT_CLIENT_PROVIDER, the rest as A2A_AGENT_CLIENT_MODEL.
agent.llm.maxTokens / agent.llm.temperature / agent.llm.systemPrompt / agent.llm.customHeaders[]	LLM tuning.
agent.llm.apiKeySecretRef	corev1.SecretKeySelector for the LLM API key.
queue.{enabled,maxSize,cleanupInterval}	Optional task queue.
env[]	Additional pod env vars.

The operator publishes the agent's discovered capabilities into status.card, which the orchestrator and Gateway.spec.a2a.serviceDiscovery consume through service discovery.

metadata.name doubles as the agent's alias when a Gateway picks the CR up through spec.a2a.serviceDiscovery, so keep it short and matching ^[a-z0-9_-]+$.

MCP

Deploys a Model Context Protocol server. Source: api/v1alpha1/mcp_types.go. See MCP Integration for protocol background.

Field	Description
replicas	Pod count (default 1).
image	Container image, default node:lts.
server.port	Listen port (default 8080).
server.command / server.args	Override the container command.
server.timeout	Request timeout (default 30s).
server.tls.{enabled,secretName}	TLS cert from a Secret. secretName is required when TLS is enabled.
hpa.{enabled,config}	Same shape as Gateway.hpa.

metadata.name doubles as the server's alias when a Gateway picks the CR up through spec.mcp.serviceDiscovery, so keep it short and matching ^[a-z0-9_-]+$ - an MCP named time exposes its tools as mcp_time_<tool>.

Orchestrator

Deploys the Inference Gateway CLI daemon - a chat bot that bridges a messaging channel and the gateway. It receives incoming messages, runs them through an LLM, optionally delegates work to Agents and MCP tools, and posts the reply back to the channel. Source: api/v1alpha1/orchestrator_types.go.

The Deployment is forced to a singleton (replicas: 1, strategy: Recreate) because Telegram allows only one active getUpdates consumer per bot token - running two replicas would 409. For HA today, run multiple Orchestrator resources with different tokens and disjoint allowed-user lists.

Field	Description
image	Required CLI image, e.g. ghcr.io/inference-gateway/cli:latest.
channels.maxWorkers / channels.imageRetention / channels.requireApproval	Top-level channel runtime.
channels.telegram.enabled	Toggle the Telegram channel.
channels.telegram.tokenSecretRef	SecretKeySelector for the bot token (required).
channels.telegram.allowedUsersSecretRef	SecretKeySelector for a comma-separated allow-list.
channels.telegram.pollTimeout	metav1.Duration for getUpdates long-polling.
gateway.url	Required URL of the gateway this orchestrator talks to.
gateway.apiKeySecretRef	Optional API key for the gateway.
agent.model	Required provider/model for the orchestrating LLM.
agent.systemPrompt	Optional system prompt.
tools.enabled / tools.schedule	Built-in CLI tools (incl. scheduling).
a2a.enabled	Toggle the CLI daemon's own A2A client, so the orchestrating LLM can delegate to agents. Separate from Gateway.spec.a2a, which makes the gateway an A2A server.
a2a.agents[]	Static agent URLs - plain strings here, unlike Gateway.spec.a2a.agents[], which takes {name, url} objects.
a2a.serviceDiscovery.{enabled,namespace,selector}	Discover Agent CRs by label selector. The pod is rolled when the discovered set changes.
mcp.enabled	Toggle MCP integration (OrchestratorMCPSpec).
mcp.servers[]	Static MCP server URLs, for externally hosted servers with no MCP CR in the cluster.
mcp.serviceDiscovery.{enabled,namespace,selector}	Discover MCP CRs by label selector. Static and discovered servers are written into the pod's ~/.infer/mcp.yaml; the pod is rolled when the set changes.
telemetry.enabled / telemetry.traces / telemetry.metrics	OpenTelemetry telemetry. The daemon consumes only a master switch and a single shared OTLP endpoint. See Orchestrator Telemetry.
resources / env[]	Standard pod knobs.

GPU

Leases an externally hosted, GPU-backed inference runtime through a pluggable provider interface (RunPod first). The cluster acts as control plane only - GPU hardware, drivers, and inference execution stay with the external provider. Source: api/v1alpha1/gpu_types.go.

Field	Description
provider	Required provider name, e.g. runpod.
credentialsRef	Required corev1.SecretKeySelector for the provider management API key. Never exposed in the generated connection Secret.
image / command	Required runtime container image and optional command override.
gpuTypes[]	Required GPU type identifiers, e.g. NVIDIA GeForce RTX 4090.
endpoint.{port,readinessPath}	Defaults 8080 and /health. The Ready condition stays False until the readiness path responds.
maxRuntime	Required hard runtime cap, e.g. 2h. After startedAt + maxRuntime the resource transitions to Expired - no automatic reprovision; create a new GPU resource instead.

The phase progresses Pending → Provisioning → Starting → Ready → Terminating (with Expired and Failed as terminal detours). When Ready, the operator generates a Secret named <gpu-name>-connection containing url and apiKey - the apiKey is a per-allocation token injected into the runtime as API_KEY, not the provider management key. Deleting the GPU releases the provider allocation via finalizer, and the connection Secret is garbage-collected by owner reference.

A Gateway consumes the connection Secret through provider env:

apiVersion: core.inference-gateway.com/v1alpha1
kind: GPU
metadata:
  name: my-gpu
spec:
  provider: runpod
  credentialsRef:
    name: runpod-credentials
    key: RUNPOD_API_KEY
  image: ghcr.io/inference-gateway/inference-gateway:latest
  gpuTypes:
    - 'NVIDIA GeForce RTX 4090'
  maxRuntime: 2h
---
apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
  name: my-gateway
spec:
  providers:
    - name: Custom
      enabled: true
      env:
        - name: CUSTOM_API_URL
          valueFrom:
            secretKeyRef:
              name: my-gpu-connection
              key: url
        - name: CUSTOM_API_KEY
          valueFrom:
            secretKeyRef:
              name: my-gpu-connection
              key: apiKey

For a complete runnable example, see examples/gpu/ in the operator repository.

Authentication (OIDC)

spec.auth secures the gateway proxy with OpenID Connect. When enabled is true, the Gateway controller translates the CRD fields into the gateway's AUTH_* environment variables, and the gateway validates every request's Bearer token against the issuer's JWKS. For the token flow, Keycloak setup, and how to obtain access tokens, see the Authentication guide - this section covers only how the same OIDC contract is expressed on the Gateway CR.

spec.auth

Field	Description
enabled	Toggle authentication (default false). Always emitted as AUTH_ENABLED.
provider	Provider type: oidc (default), jwt, or basic. oidc is the path wired end to end and documented here.
oidc	OIDC configuration (see below).

spec.auth.oidc

Field	Emitted env var	Description
issuerUrl	AUTH_OIDC_ISSUER	OIDC issuer URL (default http://keycloak:8080/realms/inference-gateway-realm).
clientId	AUTH_OIDC_CLIENT_ID	OAuth2 client id, used as the expected aud (default inference-gateway-client).
audiences	AUTH_OIDC_AUDIENCE	[]string of token aud values the gateway accepts, for example a separate API identifier. Emitted comma-separated when non-empty; when empty, the gateway expects clientId. See Audience validation.
clientSecretRef	AUTH_OIDC_CLIENT_SECRET	SecretKeySelector (name + key) for the client secret. The operator wires it through as a valueFrom.secretKeyRef, so the secret value never lands in the CR.
caCertRef	SSL_CERT_FILE	ConfigMapKeySelector (name + key) holding a PEM-encoded CA certificate. The operator mounts it into the gateway pod at /usr/local/share/ca-certificates/oidc-ca.crt and points SSL_CERT_FILE at it, so the Go runtime trusts a self-signed issuer (for example a Keycloak with its own CA) during OIDC discovery.

The OIDC variables are emitted only when enabled: true and an oidc block is present. AUTH_OIDC_AUDIENCE is added only when audiences is non-empty, AUTH_OIDC_CLIENT_SECRET only when clientSecretRef is set, and SSL_CERT_FILE only when caCertRef is set. The variables themselves are documented on the gateway side in Configuration.

One gap against the current gateway: clientSecretRef still emits AUTH_OIDC_CLIENT_SECRET, which the gateway no longer reads (it only verifies tokens, never requests them).

Example: Gateway with OIDC

This mirrors the Kubernetes authentication example expressed as a Gateway CR. Create a Secret holding the client secret and (for a self-signed issuer) a ConfigMap holding the issuer's CA certificate, then reference both from spec.auth.oidc:

apiVersion: v1
kind: Secret
metadata:
  name: gateway-oidc
  namespace: inference-gateway
type: Opaque
stringData:
  client-secret: <your-client-secret>
---
# Only needed for a self-signed issuer (for example Keycloak with its own CA).
apiVersion: v1
kind: ConfigMap
metadata:
  name: keycloak-ca
  namespace: inference-gateway
data:
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    ...your issuer's PEM CA...
    -----END CERTIFICATE-----
---
apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
  name: my-gateway
  namespace: inference-gateway
spec:
  replicas: 1
  auth:
    enabled: true
    provider: oidc
    oidc:
      issuerUrl: https://keycloak.example.com/realms/inference-gateway-realm
      clientId: inference-gateway-client
      clientSecretRef:
        name: gateway-oidc
        key: client-secret
      caCertRef:
        name: keycloak-ca
        key: ca.crt
  providers:
    - name: OpenAI
      enabled: true
      env:
        - name: OPENAI_API_KEY
          valueFrom:
            secretKeyRef:
              name: openai-secret
              key: OPENAI_API_KEY

Drop the caCertRef block and its ConfigMap when the issuer presents a publicly trusted certificate - it is only required to trust a self-signed or private-CA issuer.

Quick Start: Minimal Gateway

Create a Secret with your provider API key, then a Gateway. The provider shape is name + enabled + env[] - env vars are passed through to the gateway pod.

apiVersion: v1
kind: Secret
metadata:
  name: openai-secret
  namespace: inference-gateway
type: Opaque
stringData:
  OPENAI_API_KEY: sk-...
---
apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
  name: my-gateway
  namespace: inference-gateway
spec:
  replicas: 1
  environment: development
  telemetry:
    enabled: true
    metrics:
      enabled: true
      port: 9464
  providers:
    - name: OpenAI
      enabled: true
      env:
        - name: OPENAI_API_KEY
          valueFrom:
            secretKeyRef:
              name: openai-secret
              key: OPENAI_API_KEY

Apply it and port-forward to test:

kubectl apply -f gateway.yaml
kubectl get gateway -n inference-gateway -w
kubectl port-forward -n inference-gateway svc/my-gateway 8080:8080

For a full end-to-end example with an Orchestrator, two Agents, and Redis state, see examples/orchestrator/ in the operator repository.

Keyless Providers (Self-Hosted)

Self-hosted, OpenAI-compatible providers - llamacpp (llama.cpp's llama-server) and ollama - use auth_type: none, so their Gateway provider block needs no Secret and no *_API_KEY. Point the provider's *_API_URL at your in-cluster server and enable it:

apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
  name: my-gateway
  namespace: inference-gateway
spec:
  replicas: 1
  providers:
    - name: Llamacpp
      enabled: true
      env:
        - name: LLAMACPP_API_URL
          value: http://llama-server:8080/v1

name is matched case-insensitively against the schema Provider enum, so Llamacpp resolves to the llamacpp provider. LLAMACPP_API_URL defaults to http://llamacpp:8080/v1; override it (as above) to reach your own llama-server Service. Address models with the llamacpp/ prefix, for example llamacpp/llama-3.2-3b-instruct. Ollama follows the same keyless shape with OLLAMA_API_URL.

The llamacpp provider requires operator >= v0.19.0 - earlier releases do not recognize it, so pin at least that version when installing (see Installation). The gateway's hybrid example (examples/kubernetes/hybrid) pins v0.25.1 and ships the Llamacpp block already enabled, so task deploy-llamacpp brings up the local llama.cpp path with no manual edit to gateway.yaml.

MCP Servers (spec.mcp)

spec.mcp wires the gateway's MCP client. The controller renders every configured server into the gateway pod's MCP_SERVERS env var as a comma-separated list of name=url entries - the name is the server's alias, and the gateway namespaces that server's tools as mcp_<alias>_<tool>. See Server Aliases and Tool Namespacing for how aliases reach the model.

Field	Description
enabled	Toggle the gateway's MCP client (MCP_ENABLED, default false).
expose	Expose the gateway itself as an MCP server on root-level POST /mcp, MCP 2026-07-28 (MCP_EXPOSE, default false).
resourceUrl	Canonical public URL of POST /mcp (MCP_RESOURCE_URL). Defaults to the first gatewayAPI.httpRoute hostname.
toolMode	selector (default) or direct (MCP_TOOL_MODE). See Tool Exposure Mode.
servers[].name	Required. Becomes the server's alias, and therefore its tool namespace.
servers[].url	Required server URL.
servers[].healthCheck.{enabled,path,interval}	Per-server health check (defaults true, /health, 30s).
serviceDiscovery.{enabled,namespace,selector}	Discover MCP CRs by label selector. namespace defaults to the Gateway's own namespace; an empty selector matches all.
timeouts.{client,dial,tlsHandshake,responseHeader,expectContinue,request}	MCP client timeouts, emitted as the matching MCP_*_TIMEOUT variables.

With expose: true, route POST /mcp at the root of the gateway Service - it deliberately does not sit under /v1. The endpoint is stateless, so it needs no session affinity and load-balances freely across replicas, but the HTTPRoute (and any proxy in front of it) must pass the MCP-Protocol-Version, Mcp-Method and Mcp-Name headers through unchanged; stripping or rewriting one answers 400 with -32020. Do not aim a probe or gateway health check at /mcp - it is POST only and answers 405 to GET; use /health. See Deploying Behind a Proxy.

Protected resource URL (mcp.resourceUrl)

mcp.resourceUrl pins the canonical public URL of POST /mcp and reaches the pod as MCP_RESOURCE_URL. The gateway publishes it as the resource of the RFC 9728 Protected Resource Metadata document served at GET /.well-known/oauth-protected-resource/mcp, and in the resource_metadata parameter of every 401 challenge on /mcp - so it is what clients use as their RFC 8707 resource indicator when requesting a token. See Protected Resource Metadata.

Left unset, the operator derives it as <scheme>😕/<first gatewayAPI.httpRoute hostname>/mcp, where the scheme is https when gatewayAPI.gateway.tls.enabled is true and http otherwise:

spec:
  mcp:
    enabled: true
    expose: true
    resourceUrl: https://api.inference-gateway.local/mcp # optional - derived from the hostname below
  gatewayAPI:
    enabled: true
    gateway:
      tls:
        enabled: true
    httpRoute:
      hostnames:
        - api.inference-gateway.local

Nothing is emitted when spec.gatewayAPI is disabled or the first hostname is a wildcard (*.example.com) - there is no single public URL to derive. The gateway then falls back to building the URL from the request scheme (honouring X-Forwarded-Proto) and Host, which is wrong behind a proxy that rewrites either. Set resourceUrl explicitly in that case, or clients discover a URL they cannot reach.

The operator-managed HTTPRoute matches on the / prefix, so /.well-known/oauth-protected-resource/mcp is already routed alongside /mcp - no extra route or path match is needed for discovery to work.

Statically declared servers and discovered MCP CRs are unioned, deduplicated on URL, and sorted for determinism. Discovered servers use the MCP CR's metadata.name as their alias, so an MCP named time yields mcp_time_get_current_time rather than a long host-derived name like mcp_time-service_inference-gateway_svc_cluster_local_get_current_time - which matters in direct tool mode, where names over 64 characters are rejected by providers.

Alias rules

A name (static) or metadata.name (discovered) is used as the alias only when it:

• matches ^[a-z0-9_-]+$,
• is unique across static and discovered servers, and
• is not the reserved alias tools (which belongs to the gateway's selector meta-tools).

A name failing any of these is rendered as a bare URL instead, and the gateway derives the alias from the URL host.

Example: static servers plus discovery

apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
  name: my-gateway
  namespace: inference-gateway
spec:
  mcp:
    enabled: true
    expose: true
    toolMode: selector
    servers:
      - name: deepwiki
        url: https://mcp.deepwiki.com/mcp
      - name: search
        url: http://mcp-search-server:8082/mcp
        healthCheck:
          enabled: true
          path: /health
          interval: 30s
    serviceDiscovery:
      enabled: true
      selector:
        matchLabels:
          app.kubernetes.io/part-of: inference-gateway
    timeouts:
      client: 10s
      request: 10s

With an MCP CR named time in the same namespace matching the selector, the gateway pod receives:

MCP_SERVERS=deepwiki=https://mcp.deepwiki.com/mcp,search=http://mcp-search-server:8082/mcp,time=http://time.inference-gateway.svc.cluster.local:8080/mcp

The Gateway status mirrors that value in status.mcpServers, so kubectl shows the same name=url entries:

kubectl get gateway my-gateway -n inference-gateway -o jsonpath='{.status.mcpServers}'

[
  "deepwiki=https://mcp.deepwiki.com/mcp",
  "search=http://mcp-search-server:8082/mcp",
  "time=http://time.inference-gateway.svc.cluster.local:8080/mcp"
]

A2A Agents (spec.a2a)

spec.a2a turns the gateway itself into an A2A server: it publishes one merged agent card at GET /.well-known/agent-card.json and relays every POST /a2a call to the agent the request names. The controller renders every configured agent into the gateway pod's A2A_AGENTS env var as a comma-separated list of name=url entries, where name is the agent's alias.

This block requires gateway v0.58.0 or newer - spec.a2a is ignored by older gateway versions.

Field	Description
enabled	Toggle the gateway's A2A server (A2A_ENABLED, default false). Nothing below is emitted while this is false.
agents[].name	Required. Becomes the agent's alias, and therefore its skill-id and task-id prefix.
agents[].url	Required agent base URL.
serviceDiscovery.{enabled,namespace,selector}	Discover Agent CRs by label selector. namespace defaults to the Gateway's own namespace; an empty selector matches all.
resourceUrl	Canonical public URL of POST /a2a (A2A_RESOURCE_URL). Defaults to the first gatewayAPI.httpRoute hostname.
timeouts.client	Timeout for one non-streaming call to an agent, including the card fetch (A2A_CLIENT_TIMEOUT, default 30s).
timeouts.streamIdle	Idle cutoff for a relayed stream (A2A_STREAM_IDLE_TIMEOUT, default 5m). 0 disables it.
cardRefreshInterval	How often agent cards are re-fetched in the background (A2A_CARD_REFRESH_INTERVAL). 0 disables the background refresh.

Background card refresh is what retries an agent that was unreachable at startup and picks up skill changes, so leave it on unless the agent set is frozen.

Discovering Agent CRs

With serviceDiscovery.enabled: true the operator lists Agent CRs in serviceDiscovery.namespace (the Gateway's own namespace when unset), filtered by serviceDiscovery.selector (matchLabels and matchExpressions both work; a nil or empty selector matches every Agent in the namespace). The operator watches Agent resources, so creating, deleting or relabeling one re-renders A2A_AGENTS without a Gateway edit.

Each discovered Agent contributes <metadata.name>=<url>, where the URL is:

1. status.card.url - the card URL the Agent controller reports, preferred once it is populated; otherwise
2. the in-cluster Service URL http://<name>.<namespace>.svc.cluster.local:<spec.port> (or spec.card.url when set), used while the card URL has not been reported yet.

Static agents[] entries and discovered Agent CRs are unioned, deduplicated on URL, and sorted for determinism. A discovery error is logged and ignored, so a transient API failure leaves the static list in place rather than blanking it.

Alias rules

A name (static) or metadata.name (discovered) is used as the alias only when it matches ^[a-z0-9_-]+$ and is unique across static and discovered agents. A name failing either check is rendered as a bare URL instead, and the gateway derives the alias from the URL host - the entry is never dropped.

Per-agent credentials go into the URL as basic auth (https://user:token@agent.example.com), because the gateway does not forward the caller's bearer token to agents. Keep such URLs out of plain Gateway manifests you commit.

Protected resource URL (a2a.resourceUrl)

a2a.resourceUrl pins the canonical public URL of POST /a2a, which the gateway publishes on its agent card and as the resource of the RFC 9728 document at GET /.well-known/oauth-protected-resource/a2a. Left unset, the operator derives <scheme>😕/<first gatewayAPI.httpRoute hostname>/a2a, with https when gatewayAPI.gateway.tls.enabled is true.

Nothing is emitted when spec.gatewayAPI is disabled or the first hostname is a wildcard, and the gateway then builds the URL from the request scheme and Host - wrong behind a proxy that rewrites either. Set resourceUrl explicitly in that case, exactly as for mcp.resourceUrl.

Example: static agents plus discovery

apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
  name: my-gateway
  namespace: inference-gateway
spec:
  a2a:
    enabled: true
    agents:
      - name: calendar
        url: https://calendar-agent.example.com
    serviceDiscovery:
      enabled: true
      selector:
        matchLabels:
          app.kubernetes.io/part-of: inference-gateway
    timeouts:
      client: 30s
      streamIdle: 5m
    cardRefreshInterval: 5m
  gatewayAPI:
    enabled: true
    gateway:
      tls:
        enabled: true
    httpRoute:
      hostnames:
        - api.inference-gateway.local

With an Agent CR named documentation in the same namespace matching the selector, the gateway pod receives:

A2A_ENABLED=true
A2A_AGENTS=calendar=https://calendar-agent.example.com,documentation=http://documentation.inference-gateway.svc.cluster.local:8080
A2A_RESOURCE_URL=https://api.inference-gateway.local/a2a

Model Routing

spec.routing turns on the gateway's opt-in round-robin model routing: a client requests a stable logical alias (e.g. {"model": "fast-chat"}) and the gateway rotates round-robin through that alias's pool of upstream provider/model deployments, forwarding to the one it resolves. Operators can swap the upstreams behind an alias without touching client code.

This is distinct from spec.gatewayAPI, which exposes the gateway to north-south Kubernetes Gateway API traffic. spec.routing configures the gateway's internal model-alias -> provider/model selection; the two are unrelated and can be used together or independently.

The operator maps this block onto the gateway's two routing environment variables:

Field	Maps to	Description
enabled	ROUTING_ENABLED	Toggle model routing (default false). When true, the operator sets ROUTING_ENABLED=true on the gateway container.
config	ROUTING_CONFIG_PATH	Inline routing YAML (a models: map of alias -> ordered deployment pool). The operator renders it into a ConfigMap named <gateway>-routing and mounts it at /etc/inference-gateway/routing/routing.yaml. Mutually exclusive with configMapRef.
configMapRef	ROUTING_CONFIG_PATH	Reference (name, key) to an existing ConfigMap holding the routing YAML, mounted at /etc/inference-gateway/routing/<key> instead of rendering config. Mutually exclusive with config.

See Model Routing for the routing-file schema, selection semantics, and the X-Selected-Provider / X-Selected-Model response headers.

Example: Gateway with a routed alias

The fast-chat alias below fans out round-robin across a two-deployment pool (Groq and OpenAI). Every provider a deployment references must also be configured under providers - routing selects among configured providers, it does not configure them.

apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
  name: my-gateway
  namespace: inference-gateway
spec:
  routing:
    enabled: true
    config: |
      models:
        fast-chat:
          strategy: round_robin # only strategy in Phase 1; omit to default to round_robin
          deployments:
            - provider: groq
              model: llama-3.3-70b-versatile
            - provider: openai
              model: gpt-4o-mini
  providers:
    - name: Groq
      enabled: true
      env:
        - name: GROQ_API_KEY
          valueFrom:
            secretKeyRef:
              name: provider-secrets
              key: GROQ_API_KEY
    - name: OpenAI
      enabled: true
      env:
        - name: OPENAI_API_KEY
          valueFrom:
            secretKeyRef:
              name: provider-secrets
              key: OPENAI_API_KEY

To reuse an existing ConfigMap instead of inline config, swap the config block for a configMapRef:

spec:
  routing:
    enabled: true
    configMapRef:
      name: my-routing-config
      key: routing.yaml

For a runnable manifest, see gateway-with-model-routing in the operator repository.

Guardrails (spec.guardrails)

spec.guardrails turns on the gateway's OPA/Rego policy enforcement. The controller maps the block onto the gateway's GUARDRAILS_* environment variables, which are documented on the gateway side in Configuration.

Field	Maps to	Description
enabled	GUARDRAILS_ENABLED	Toggle policy enforcement (default false). When true, the operator sets GUARDRAILS_ENABLED=true and GUARDRAILS_POLICY_DIR=/etc/inference-gateway/guardrails on the gateway container.
failMode	GUARDRAILS_FAIL_MODE	deny (default) or allow, for what happens when a policy evaluation fails. The operator translates deny to closed and allow to open, the values the gateway reads.
externalUrl	GUARDRAILS_EXTERNAL_URL	URL of an external OPA policy evaluation endpoint, instead of evaluating local policies.
externalTimeout	GUARDRAILS_EXTERNAL_TIMEOUT	Timeout for the external endpoint (default 5s). Emitted only alongside externalUrl.
configMapRef	-	LocalObjectReference (name) to a ConfigMap holding Rego policy files. The operator mounts it at /etc/inference-gateway/guardrails, the policy directory.

apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
  name: my-gateway
  namespace: inference-gateway
spec:
  guardrails:
    enabled: true
    failMode: deny
    configMapRef:
      name: guardrails-policies

Routing (Gateway API)

spec.gatewayAPI exposes the gateway to north-south traffic through the Kubernetes Gateway API (gateway.networking.k8s.io). It is the v1alpha1 successor to the removed ingress field: instead of an Ingress fronted by ingress-nginx, the operator provisions a Gateway API Gateway and HTTPRoute for the gateway Service, served by Envoy Gateway (the envoy GatewayClass). It is unrelated to spec.routing (gateway-native model routing) above - the field was renamed from spec.routing to spec.gatewayAPI to free that name for model routing.

The ingress: field and the NGINX Ingress path are gone - Kubernetes deployments now front the gateway with the Gateway API. This mirrors the migration in the gateway's examples/kubernetes.

Install the Gateway API and Envoy Gateway

Routing needs the Gateway API CRDs plus an implementation that registers the envoy GatewayClass. Install them once per cluster, before applying a Gateway with gatewayAPI.enabled: true:

# 1. Gateway API standard CRDs (GatewayClass, Gateway, HTTPRoute).
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.1/standard-install.yaml

# 2. Envoy Gateway (the data plane).
helm upgrade --install envoy-gateway \
  oci://docker.io/envoyproxy/gateway-helm \
  --version v1.2.0 \
  --namespace envoy-gateway-system \
  --create-namespace \
  --skip-crds \
  --wait

# 3. Register the "envoy" GatewayClass.
kubectl apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: envoy
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
EOF

The gateway's examples/kubernetes wrap these steps as task deploy-infrastructure (cluster + Gateway API + Envoy Gateway + operator).

spec.gatewayAPI

Field	Description
enabled	Toggle Gateway API routing (default false). When true, the operator creates the Gateway API resources for the gateway Service.
gateway.gatewayClassName	GatewayClass that implements the listener (default envoy). Ignored when gateway.parentRefs is set.
gateway.parentRefs[]	Advanced mode: attach the HTTPRoute to an existing, platform-managed Gateway instead of creating one.
gateway.tls.{enabled,issuer,secretName}	Terminate TLS at the Gateway listener. issuer names a cert-manager ClusterIssuer; secretName defaults to the gateway name suffixed with -tls.
httpRoute.hostnames[]	Hostnames the HTTPRoute matches. In default mode they also become the managed Gateway listener's hostnames.

In default mode the operator owns both the Gateway and the HTTPRoute. Setting gateway.parentRefs switches to advanced mode, where a platform team owns a shared Gateway and the operator only creates the HTTPRoute attached to it.

Example: Gateway with routing

apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
  name: my-gateway
  namespace: inference-gateway
spec:
  replicas: 1
  providers:
    - name: OpenAI
      enabled: true
      env:
        - name: OPENAI_API_KEY
          valueFrom:
            secretKeyRef:
              name: openai-secret
              key: OPENAI_API_KEY
  gatewayAPI:
    enabled: true
    gateway:
      gatewayClassName: envoy
    httpRoute:
      hostnames:
        - api.inference-gateway.local

Reach the gateway

Routing provisions a Gateway API Gateway and HTTPRoute in the gateway's namespace, while Envoy Gateway runs the data-plane Service in envoy-gateway-system. Confirm the resources are ready, then port-forward the Envoy Service and send the request with the routed hostname as a Host header (no /etc/hosts edit needed):

kubectl get gateway.gateway.networking.k8s.io -n inference-gateway
kubectl get httproute -n inference-gateway

ENVOY_SVC=$(kubectl get svc -n envoy-gateway-system \
  -l gateway.envoyproxy.io/owning-gateway-name=my-gateway \
  -o jsonpath='{.items[0].metadata.name}')
kubectl -n envoy-gateway-system port-forward "svc/${ENVOY_SVC}" 8080:80

curl -H 'Host: api.inference-gateway.local' http://localhost:8080/v1/models

For runnable manifests, see gateway-with-routing-simple and gateway-with-routing-advanced in the operator repository.

Orchestrator Telemetry

spec.telemetry on an Orchestrator configures OpenTelemetry traces and metrics. It reuses the shared TelemetrySpec that Gateway and Agent use, so the block accepts the same enabled, traces, metrics, and exporter fields. The orchestrator runs the CLI daemon, which consumes a narrower slice of that type: the controller maps the whole block onto just two environment variables on the orchestrator pod. When telemetry is enabled, the daemon pushes operational metrics (messages processed, message duration, active channels) over OTLP alongside traces and logs.

Emitted env var	Sourced from	Description
INFER_TELEMETRY_ENABLED	telemetry.enabled	Master switch (default false). Always emitted - it is set to false when spec.telemetry is omitted or disabled.
INFER_TELEMETRY_OTLP_ENDPOINT	telemetry.traces.exporter.otlp.endpoint, falling back to telemetry.metrics.exporter.otlp.endpoint	Single OTLP/HTTP collector endpoint shared by all signals. Emitted only when telemetry is enabled and an OTLP endpoint is set.

Single shared OTLP endpoint

Unlike Gateway and Agent, the daemon CLI exposes one OTLP endpoint for both traces and metrics - there is no per-signal OTLP field. When both telemetry.traces.exporter.otlp.endpoint and telemetry.metrics.exporter.otlp.endpoint are set, the traces endpoint wins and the metrics endpoint is ignored. Point both signals at one collector that ingests them together, or export metrics over Prometheus pull (which needs no OTLP endpoint) to sidestep the conflict.

The remaining TelemetrySpec fields - the telemetry.metrics.* Prometheus settings and the per-signal protocol - are accepted for schema parity with Gateway/Agent but are inert for the orchestrator: it is a forced singleton daemon with no scrape Service, so only the master switch and the shared OTLP endpoint reach the CLI. Configure Prometheus scraping and per-signal protocols on a Gateway or Agent instead. See CLI Telemetry for how the daemon exports to the OTLP endpoint, and Daemon for the daemon-specific metrics reference.

Example: Orchestrator with OTLP traces and Prometheus metrics

apiVersion: core.inference-gateway.com/v1alpha1
kind: Orchestrator
metadata:
  name: my-orchestrator
  namespace: inference-gateway
spec:
  image: ghcr.io/inference-gateway/cli:latest
  channels:
    telegram:
      enabled: true
      tokenSecretRef:
        name: telegram-bot
        key: token
  gateway:
    url: http://my-gateway:8080
  agent:
    model: openai/gpt-4o
  telemetry:
    enabled: true
    traces:
      exporter:
        otlp:
          endpoint: http://otel-collector:4318
    metrics:
      enabled: true
      exporter:
        prometheus:
          port: 9464

The orchestrator pod receives exactly two telemetry variables from this block:

INFER_TELEMETRY_ENABLED=true
INFER_TELEMETRY_OTLP_ENDPOINT=http://otel-collector:4318

Traces and daemon-specific operational metrics are pushed to the collector over OTLP/HTTP. The metrics.exporter.prometheus block is schema-valid but does not translate to any orchestrator env var today (no scrape endpoint is exposed); it is shown here to illustrate the full shared TelemetrySpec shape. Because Prometheus metrics do not claim an OTLP endpoint, pairing them with OTLP traces also keeps the single shared endpoint unambiguous.

The daemon emits the following metrics when telemetry is enabled:

Metric	Type	Description
infer.daemon.messages_processed	Counter	Inbound messages processed
infer.daemon.message.duration	Histogram	Per-message processing duration
infer.daemon.active_channels	UpDown Counter	Number of active channels

All daemon metrics carry the resource attribute infer.execution.mode=daemon. See Daemon for example PromQL queries.

Status and Monitoring

kubectl get gateway -A
kubectl describe gateway my-gateway -n inference-gateway

The Gateway status surfaces:

• phase - Pending, Running, Failed, or Unknown.
• readyReplicas / availableReplicas.
• url - the resolved access URL (the routing hostname when spec.gatewayAPI is enabled, otherwise the cluster service URL).
• providerSummary - comma-separated list of enabled providers.
• mcpServers[] / mcpServerCount - the sorted name=url entries (static plus discovered) the pod is configured with, mirroring MCP_SERVERS, and their count (0 when MCP is disabled). See MCP Servers (spec.mcp).
• a2aAgents[] / a2aAgentCount - the sorted name=url entries (static plus discovered) the pod is configured with, mirroring A2A_AGENTS, and their count (0 when A2A is disabled). See A2A Agents (spec.a2a).
• conditions[] - standard Available / Progressing / ReplicaFailure conditions.

kubectl get gateways prints the two counts as the MCPS and A2A columns, so a wired-up gateway is visible without describe:

kubectl get gateways -n inference-gateway

NAME         URL                                   PORT   PROVIDERS   MCPS   A2A   AGE
my-gateway   https://api.inference-gateway.local   8080   openai      3      2     5m

Read the full agent list the same way as the MCP one:

kubectl get gateway my-gateway -n inference-gateway -o jsonpath='{.status.a2aAgents}'

Agent, MCP, and Orchestrator expose the standard metav1.Condition slice plus a boolean ready. Orchestrator additionally exposes discoveredAgents[] and discoveredAgentCount when service discovery is enabled. See Observability for end-to-end metrics and tracing setup.

Cleanup

Delete custom resources before uninstalling the operator so finalizers can run:

kubectl delete gateway,agent,mcp,orchestrator,gpu --all -A
kubectl delete -f https://github.com/inference-gateway/operator/releases/latest/download/install.yaml

Examples

The operator repository ships runnable examples for each CRD:

• gateway-minimal - single provider, no routing.
• gateway-complete - HPA, multiple providers, TLS via cert-manager.
• gateway-with-model-routing - round-robin model routing over a provider pool via spec.routing.
• gateway-with-routing-simple and gateway-with-routing-advanced - Gateway API routing patterns (operator-managed Gateway, or an HTTPRoute attached to a shared Gateway).
• agent-server - minimal A2A agent.
• mcp-server - MCP server.
• orchestrator - gateway + two agents + orchestrator with service discovery.