Kubernetes Operator

The Inference Gateway Operator is a Kubernetes controller that manages Inference Gateway and related resources declaratively through Custom Resources (CRs). Pick the operator over the Helm chart when you want to manage gateways, A2A agents, MCP servers, and chat-channel orchestrators as first-class CRs in the same cluster API your other workloads use.

The operator publishes four CRDs under core.inference-gateway.com/v1alpha1:

• Gateway - the gateway proxy itself, with providers, auth, MCP, ingress, and HPA.
• Agent - an A2A worker that an Orchestrator (or any A2A client) can dispatch tasks to.
• MCP - a Model Context Protocol server.
• Orchestrator - runs a chat bot backed by the gateway. It listens on a messaging channel (Telegram today; more channels planned), drives the conversation with an LLM, and can delegate work to Agents and MCP tools.

The API is v1alpha1 and breaking changes can land between releases.

Installation

The operator is distributed as pre-rendered manifests on each GitHub release. The install.yaml artifact bundles the namespace, CRDs, RBAC, and controller deployment.

kubectl apply -f https://github.com/inference-gateway/operator/releases/latest/download/install.yaml

For production, pin to a release rather than latest:

kubectl apply -f https://github.com/inference-gateway/operator/releases/download/v<VERSION>/install.yaml

For GitOps (ArgoCD, Flux), point your source at the operator repository's manifests/ directory at a tagged ref - it contains the same install.yaml and a CRD-only crds.yaml for split installs.

Verification

kubectl get pods -n inference-gateway-system
kubectl get crd | grep inference-gateway.com

You should see four CRDs: gateways, agents, mcps, and orchestrators.

Custom Resources

Gateway

Deploys the gateway proxy. Source: api/v1alpha1/gateway_types.go.

Provider env vars referenced via Secrets follow the standard valueFrom.secretKeyRef pattern - see Configuration for the full list of variables each provider accepts.

Agent

Deploys an A2A worker. Agents are dispatched to by an Orchestrator (or any A2A client) - the gateway itself does not call agents; it only proxies inference. The agent typically calls back into the gateway for its own LLM completions via agent.llm.baseURL. Source: api/v1alpha1/agent_types.go. See A2A Integration for protocol background.

The operator publishes the agent's discovered capabilities into status.card, which the orchestrator consumes through service discovery.

MCP

Deploys a Model Context Protocol server. Source: api/v1alpha1/mcp_types.go. See MCP Integration for protocol background.

Orchestrator

Deploys the Inference Gateway CLI's channels-manager daemon - a chat bot that bridges a messaging channel and the gateway. It receives incoming messages, runs them through an LLM, optionally delegates work to Agents and MCP tools, and posts the reply back to the channel. Source: api/v1alpha1/orchestrator_types.go.

The Deployment is forced to a singleton (replicas: 1, strategy: Recreate) because Telegram allows only one active getUpdates consumer per bot token - running two replicas would 409. For HA today, run multiple Orchestrator resources with different tokens and disjoint allowed-user lists.

Quick Start: Minimal Gateway

Create a Secret with your provider API key, then a Gateway. The provider shape is name + enabled + env[] - env vars are passed through to the gateway pod.

apiVersion: v1
kind: Secret
metadata:
  name: openai-secret
  namespace: inference-gateway
type: Opaque
stringData:
  OPENAI_API_KEY: sk-...
---
apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
  name: my-gateway
  namespace: inference-gateway
spec:
  replicas: 1
  environment: development
  telemetry:
    enabled: true
    metrics:
      enabled: true
      port: 9464
  providers:
    - name: OpenAI
      enabled: true
      env:
        - name: OPENAI_API_KEY
          valueFrom:
            secretKeyRef:
              name: openai-secret
              key: OPENAI_API_KEY

Apply it and port-forward to test:

kubectl apply -f gateway.yaml
kubectl get gateway -n inference-gateway -w
kubectl port-forward -n inference-gateway svc/my-gateway 8080:8080

For a full end-to-end example with an Orchestrator, two Agents, and Redis state, see examples/orchestrator/ in the operator repository.

Status and Monitoring

kubectl get gateway -A
kubectl describe gateway my-gateway -n inference-gateway

The Gateway status surfaces:

• phase - Pending, Running, Failed, or Unknown.
• readyReplicas / availableReplicas.
• url - the resolved access URL (ingress host when ingress is enabled, otherwise the cluster service URL).
• providerSummary - comma-separated list of enabled providers.
• conditions[] - standard Available / Progressing / ReplicaFailure conditions.

Agent, MCP, and Orchestrator expose the standard metav1.Condition slice plus a boolean ready. Orchestrator additionally exposes discoveredAgents[] and discoveredAgentCount when service discovery is enabled. See Observability for end-to-end metrics and tracing setup.

Cleanup

Delete custom resources before uninstalling the operator so finalizers can run:

kubectl delete gateway,agent,mcp,orchestrator --all -A
kubectl delete -f https://github.com/inference-gateway/operator/releases/latest/download/install.yaml

Examples

The operator repository ships runnable examples for each CRD:

• gateway-minimal - single provider, no ingress.
• gateway-complete - HPA, multiple providers, ingress with cert-manager.
• gateway-with-ingress-simple and gateway-with-ingress-advanced - ingress patterns.
• agent-server - minimal A2A agent.
• mcp-server - MCP server.
• orchestrator - gateway + two agents + orchestrator with service discovery.