Inference Gateway

Appearance

Configuration ​

Breaking change (schemas v0.15.3): All _ENABLE env-var suffixes have been renamed to _ENABLED. Existing deployments must update their configuration: TELEMETRY_ENABLE -> TELEMETRY_ENABLED, TELEMETRY_METRICS_PUSH_ENABLE -> TELEMETRY_METRICS_PUSH_ENABLED, TELEMETRY_TRACING_ENABLE -> TELEMETRY_TRACING_ENABLED, MCP_ENABLE -> MCP_ENABLED, MCP_POLLING_ENABLE -> MCP_POLLING_ENABLED, AUTH_ENABLE -> AUTH_ENABLED. (MCP_ENABLE_RECONNECT is unchanged.)

Breaking change (schemas v0.27.4): ENABLE_VISION has been renamed to VISION_ENABLED to match the other feature flags (IMAGES_ENABLED, AUDIO_ENABLED, MCP_ENABLED, A2A_ENABLED). ENABLE_VISION is deprecated and no longer takes effect - update any existing configuration to VISION_ENABLED.

Inference Gateway provides flexible configuration options to adapt to your specific needs. As a proxy server designed to facilitate access to various language model APIs, proper configuration is essential for optimal performance and security.

Configuration Methods ​

Inference Gateway supports multiple configuration methods to suit different deployment scenarios:

  1. Environment Variables - Recommended for most deployments
  2. Kubernetes ConfigMaps and Secrets - For Kubernetes-based deployments
  3. Configuration Files - For local development and testing

Environment Variables ​

Environment variables are the primary method for configuring Inference Gateway. These variables control everything from basic server settings to provider-specific API configurations.

General Settings ​

VariableDescriptionDefault
ENVIRONMENTDeployment environmentproduction
ALLOWED_MODELSComma-separated list of models to allow. If empty, all models will be available""
DISALLOWED_MODELSComma-separated list of models to disallow. If empty, no models will be blocked. Takes lower precedence than ALLOWED_MODELS""
VISION_ENABLEDEnable vision/multimodal handling for all providers. When enabled, image content is stripped from requests to models known to accept only non-image input; unknown models are passed through. When disabled, image content is forwarded to the provider untouchedfalse
IMAGES_ENABLEDEnable the Images API (POST /v1/images/generations and /v1/images/edits). When disabled, the endpoints return a 404false
AUDIO_ENABLEDEnable the Audio API (POST /v1/audio/speech, /v1/audio/sfx and /v1/audio/music). When disabled, the endpoints return a 404. Speech is served by the openai and elevenlabs providers and by the local llama-tts engine (local/qwen3-tts); sound effects and music are served by elevenlabs only; llamacpp speech is a work in progress and not supported yetfalse
VIDEOS_ENABLEDEnable the Videos API (POST /v1/videos, GET /v1/videos/{video_id} and GET /v1/videos/{video_id}/content). When disabled, the endpoints return a 404false
AUDIO_LOCAL_AUTO_DOWNLOADAllow fetching the llama-tts binary and GGUF models, in the background at startup when the Audio API is enabled and on demand otherwise. A cached llama-tts is replaced when its sha256 no longer matches the latest release checksums.txt - a binary found on PATH and already present GGUF weights are never re-downloaded, whether written by an earlier run, the CLI, a mounted volume, or a pre-baked image layer. When false, the gateway serves only from the existing cache or PATH and returns an actionable error when assets are missingtrue
AUDIO_LOCAL_MAX_CONCURRENCYMaximum concurrent local syntheses; requests beyond the limit queue rather than fail2
AUDIO_LOCAL_TIMEOUTTimeout in seconds for a single local synthesis, surfaced as a 504300
DEBUG_CONTENT_TRUNCATE_WORDSNumber of words to truncate per content section in debug logs (development mode only)10
DEBUG_MAX_MESSAGESMaximum number of messages to show in debug logs (development mode only)100
AUTH_ENABLEDEnable OIDC authenticationfalse

VISION_ENABLED controls whether the gateway inspects image content in chat completion requests. When disabled (the default), the gateway does not look at image content at all - image parts are forwarded to the provider untouched. When set to true, the gateway checks each request's model against the community modalities table: image parts are stripped only for models the table lists as accepting non-image input only (the request continues with the text parts only). Models the table does not cover are treated as image-capable and passed through. Either way, the gateway never rejects a request because it contains an image.

IMAGES_ENABLED, AUDIO_ENABLED and VIDEOS_ENABLED are separate opt-ins for the Images API, the Audio API and the Videos API. While they are false, those endpoints return 404. Only providers that implement the corresponding API can serve them; other providers return 400. Audio is served by openai (speech) and elevenlabs (speech, sound effects and music), plus the built-in local/qwen3-tts engine for speech. The llamacpp provider has a speech endpoint wired in the gateway registry, but that path is a work in progress and is not supported yet.

The AUDIO_LOCAL_* variables tune the built-in local speech engine that serves the reserved model id local/qwen3-tts without any provider. They are ignored when the request routes to a provider.

Telemetry ​

These settings control telemetry and metrics exposure:

VariableDescriptionDefault
TELEMETRY_ENABLEDEnable OpenTelemetry metrics and tracingfalse
TELEMETRY_METRICS_PUSH_ENABLEDEnable the OTLP metrics push endpoint (POST /v1/metrics)false
TELEMETRY_METRICS_PORTPort for telemetry metrics server9464
TELEMETRY_TRACING_ENABLEDEmit OpenTelemetry tracing spans (requires TELEMETRY_ENABLED)false
TELEMETRY_TRACING_OTLP_ENDPOINTOTLP/HTTP endpoint for trace exporthttp://localhost:4318

When TELEMETRY_ENABLED is set to true, the gateway always installs the Prometheus exporter and serves pull metrics at /metrics on TELEMETRY_METRICS_PORT. There is no exporter selection to make: the OTEL_METRICS_EXPORTER and OTEL_EXPORTER_PROMETHEUS_* variables are read by the Go and TypeScript ADKs, not by the gateway. Distributed tracing is a separate opt-in: set TELEMETRY_TRACING_ENABLED=true (alongside TELEMETRY_ENABLED=true) to emit spans, and point TELEMETRY_TRACING_OTLP_ENDPOINT at your OTLP/HTTP collector (default http://localhost:4318). Sampling and exporter tuning use the standard OTEL_TRACES_SAMPLER / OTEL_EXPORTER_OTLP_* variables. See the Distributed Tracing section for span coverage and context propagation.

When TELEMETRY_METRICS_PUSH_ENABLED is also set to true (alongside TELEMETRY_ENABLED=true), the gateway exposes an OTLP/HTTP metrics push endpoint at POST /v1/metrics. This allows subscription clients that bypass the gateway's inference path to push their usage metrics. See the Observability page for details.

OpenID Connect ​

If authentication is enabled (AUTH_ENABLED=true), configure the following OIDC settings:

VariableDescriptionDefault
AUTH_OIDC_ISSUEROIDC issuer URL. Discovery runs once at startup against {issuer}/.well-known/openid-configuration. Required when AUTH_ENABLED=true""
AUTH_OIDC_CLIENT_IDOIDC client ID. Used as the expected token audience when AUTH_OIDC_AUDIENCE is empty. Required when AUTH_ENABLED=true""
AUTH_OIDC_AUDIENCEComma-separated list of accepted aud values, for example an API identifier. Empty means AUTH_OIDC_CLIENT_ID""

AUTH_OIDC_ISSUER and AUTH_OIDC_CLIENT_ID have no defaults - with AUTH_ENABLED=true and either one unset, the gateway fails at startup. The gateway only verifies tokens against the issuer's public keys and never requests one, so it needs no client secret; AUTH_OIDC_CLIENT_SECRET was removed.

When authentication is enabled, all API requests except /health must include a valid JWT token in the Authorization header:

http
Authorization: Bearer YOUR_JWT_TOKEN

The gateway checks the token's signature, issuer, expiry and audience. AUTH_OIDC_AUDIENCE lists the accepted aud values (comma-separated for providers that need more than one, such as Microsoft Entra ID); leaving it empty expects AUTH_OIDC_CLIENT_ID. A token carrying no aud claim at all is accepted when its client_id claim matches one of the configured values, which is how Amazon Cognito machine-to-machine tokens work.

Rejected requests return 401 with an RFC 6750 WWW-Authenticate: Bearer realm="inference-gateway" challenge, gaining error="invalid_token" when a token was presented but failed verification. See the Authentication guide for per-provider issuer and audience values and runnable examples.

Guardrails ​

Guardrails evaluate every request against OPA/Rego policies compiled at startup, and optionally against an external HTTP service:

VariableDescriptionDefault
GUARDRAILS_ENABLEDEnable gateway guardrails (OPA/Rego policy enforcement)false
GUARDRAILS_POLICY_DIRDirectory of .rego files compiled at startup. All files are compiled into one query: data.guardrails.main""
GUARDRAILS_FAIL_MODEclosed or open: behavior when a policy evaluation, the external service, or a timeout errorsclosed
GUARDRAILS_EXTERNAL_URLOptional external HTTP guardrail service, sent the same input document and answering with the same decision shape""
GUARDRAILS_EXTERNAL_TIMEOUTTimeout for the external guardrail service5s

A policy returns a decision object - {"action": "allow" | "block" | "redact" | "warn", "message": "..."} - from the rule data.guardrails.main. A missing or undefined decision allows the request. GUARDRAILS_FAIL_MODE governs evaluation errors rather than decisions: closed (the default) blocks, open allows and logs a warning.

Phases and policy input ​

A policy runs at four points in the request lifecycle and sees which one it is in input.phase:

PhaseWhen it runsinput.methodinput.pathinput.request.body
pre_callbefore the request is forwardedthe HTTP verbthe request paththe request body
post_callon the response bodythe HTTP verbthe request paththe response body
tool_argsbefore an MCP tool runsTOOL_CALLmcp_<alias>_<tool>the tool arguments
tool_outputafter an MCP tool returnsTOOL_CALLmcp_<alias>_<tool>the tool output

Every phase also carries input.request.model (the requested model, empty on the tool phases) and input.identity - the verified OIDC claims when authentication is enabled - so the same identity checks apply to tool arguments and tool outputs.

The tool phases cover both surfaces that run MCP tools: the agent loop behind /v1/chat/completions and a tools/call on POST /mcp, so one policy applies to both. On /mcp, a block at any phase answers HTTP 403 with a JSON-RPC error envelope carrying code -32001.

rego
package guardrails

import rego.v1

default main := {"action": "allow"}

main := {"action": "block", "message": "that tool is off limits"} if {
	input.phase == "tool_args"
	input.path == "mcp_filesystem_write_file"
}

Server Settings ​

These settings control the core HTTP server behavior:

VariableDescriptionDefault
SERVER_HOSTServer bind address. Loopback by default; containers must use 0.0.0.0, which the published images set for you unless an env_file or ConfigMap overrides it127.0.0.1
SERVER_PORTServer port8080
SERVER_MAX_REQUEST_BODY_SIZEMaximum request body size in bytes (10 MiB)10485760
SERVER_READ_TIMEOUTRead timeout30s
SERVER_WRITE_TIMEOUTWrite timeout30s
SERVER_IDLE_TIMEOUTIdle timeout120s
SERVER_TLS_CERT_PATHTLS certificate path""
SERVER_TLS_KEY_PATHTLS key path""

SERVER_HOST defaults to 127.0.0.1, so a gateway started from a release binary listens on loopback only. Set SERVER_HOST=0.0.0.0 to accept connections from other hosts or from inside a container network. The published container images already set SERVER_HOST=0.0.0.0 for you.

Containerized deployments - Docker, Docker Compose, Kubernetes - must end up with SERVER_HOST=0.0.0.0. Bound to loopback, the gateway answers only inside the container's own network namespace, so a published port or a Service in front of it refuses the connection. The image default already covers this, but any explicitly supplied value wins over it: a Compose env_file or environment: entry, a Kubernetes ConfigMap key, or a Helm value carrying SERVER_HOST=127.0.0.1 puts the gateway back on loopback. Either set 0.0.0.0 there too or leave SERVER_HOST out of the container's environment entirely. The process default stays 127.0.0.1 - it only applies when you run the binary directly.

For production deployments, it's strongly recommended to configure TLS:

bash
SERVER_TLS_CERT_PATH=/path/to/certificate.pem
SERVER_TLS_KEY_PATH=/path/to/private-key.pem

Client Settings ​

These settings control how Inference Gateway connects to third-party APIs:

VariableDescriptionDefault
CLIENT_TIMEOUTClient timeout30s
CLIENT_MAX_IDLE_CONNSMaximum idle connections20
CLIENT_MAX_IDLE_CONNS_PER_HOSTMaximum idle connections per host20
CLIENT_IDLE_CONN_TIMEOUTIdle connection timeout30s
CLIENT_TLS_MIN_VERSIONMinimum TLS versionTLS12
CLIENT_DISABLE_COMPRESSIONDisable compression for faster streamingtrue
CLIENT_RESPONSE_HEADER_TIMEOUTResponse header timeout10s
CLIENT_EXPECT_CONTINUE_TIMEOUTExpect continue timeout1s

For high-throughput deployments, consider increasing the connection pool settings:

bash
CLIENT_MAX_IDLE_CONNS=100
CLIENT_MAX_IDLE_CONNS_PER_HOST=50

Provider Settings ​

Configure access to various LLM providers. At minimum, you should configure the providers you plan to use.

ElevenLabs is the one exception to "LLM provider": it has no chat-completions API and serves only the Audio API, so ELEVENLABS_API_KEY is worth setting only when you synthesize speech.

OpenAI ​

VariableDescriptionDefault
OPENAI_API_URLOpenAI API URLhttps://api.openai.com/v1
OPENAI_API_KEYOpenAI API Key""

DeepSeek ​

VariableDescriptionDefault
DEEPSEEK_API_URLDeepSeek API URLhttps://api.deepseek.com
DEEPSEEK_API_KEYDeepSeek API Key""

Anthropic ​

VariableDescriptionDefault
ANTHROPIC_API_URLAnthropic API URLhttps://api.anthropic.com/v1
ANTHROPIC_API_KEYAnthropic API Key""

Cohere ​

VariableDescriptionDefault
COHERE_API_URLCohere API URLhttps://api.cohere.ai
COHERE_API_KEYCohere API Key""

Groq ​

VariableDescriptionDefault
GROQ_API_URLGroq API URLhttps://api.groq.com/openai/v1
GROQ_API_KEYGroq API Key""

Cloudflare ​

VariableDescriptionDefault
CLOUDFLARE_API_URLCloudflare API URLhttps://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/ai
CLOUDFLARE_API_KEYCloudflare API Key""

Ollama Cloud ​

VariableDescriptionDefault
OLLAMA_CLOUD_API_URLOllama Cloud API URLhttps://ollama.com/v1
OLLAMA_CLOUD_API_KEYOllama Cloud API Key""

Ollama ​

VariableDescriptionDefault
OLLAMA_API_URLOllama API URLhttp://ollama:8080/v1

llama.cpp ​

VariableDescriptionDefault
LLAMACPP_API_URLllama.cpp API URLhttp://llamacpp:8080/v1
LLAMACPP_API_KEYllama.cpp API Key""

Google ​

VariableDescriptionDefault
GOOGLE_API_URLGoogle AI API URLhttps://generativelanguage.googleapis.com/v1beta/openai
GOOGLE_API_KEYGoogle AI API Key""

Mistral ​

VariableDescriptionDefault
MISTRAL_API_URLMistral AI API URLhttps://api.mistral.ai/v1
MISTRAL_API_KEYMistral AI API Key""

MiniMax ​

VariableDescriptionDefault
MINIMAX_API_URLMiniMax API URLhttps://api.minimax.io/v1
MINIMAX_API_KEYMiniMax API Key""

Moonshot ​

VariableDescriptionDefault
MOONSHOT_API_URLMoonshot AI API URLhttps://api.moonshot.ai/v1
MOONSHOT_API_KEYMoonshot AI API Key""

NVIDIA ​

VariableDescriptionDefault
NVIDIA_API_URLNVIDIA NIM API URLhttps://integrate.api.nvidia.com/v1
NVIDIA_API_KEYNVIDIA API Key""

Z-AI ​

VariableDescriptionDefault
ZAI_API_URLZ-AI API URLhttps://api.z.ai/api/coding/paas/v4
ZAI_API_KEYZ-AI API Key""

ElevenLabs ​

VariableDescriptionDefault
ELEVENLABS_API_URLElevenLabs API URLhttps://api.elevenlabs.io/v1
ELEVENLABS_API_KEYElevenLabs API Key""

Routing ​

These settings control gateway-native model routing - mapping a logical model alias to a pool of upstream provider deployments:

VariableDescriptionDefault
ROUTING_ENABLEDEnable gateway-native model routing: logical model aliases backed by a pool of upstream provider deployments, selected round-robin per replica. Opt-in; when disabled, direct provider/model routing is unchangedfalse
ROUTING_CONFIG_PATHPath to a YAML file mapping logical model aliases to their upstream deployment pools. Required when ROUTING_ENABLED is true""

Routing is opt-in. When ROUTING_ENABLED=false (the default), the gateway routes only by explicit provider/model prefix or ?provider= query parameter. When enabled, set ROUTING_CONFIG_PATH to a YAML file that maps each logical alias to its deployment pool. Selection is round-robin per replica, and the chosen upstream is returned in the X-Selected-Provider / X-Selected-Model response headers. See the Model Routing guide for the file format, semantics, and a migration example.

Model Context Protocol (MCP) Settings ​

These settings control MCP integration for external tool access:

VariableDescriptionDefault
MCP_ENABLEDEnable MCP middlewarefalse
MCP_EXPOSEExpose the gateway as an MCP server on POST /mcp (requires MCP_ENABLED)false
MCP_RESOURCE_URLCanonical public URL of POST /mcp, e.g. https://gateway.example.com/mcp. Published as the resource of the OAuth 2.0 Protected Resource Metadata (RFC 9728) document served at /.well-known/oauth-protected-resource/mcp, and pointed at by the resource_metadata of every 401 challenge on /mcp. Defaults to the request scheme (honouring X-Forwarded-Proto) and Host with /mcp appended; set it behind an ingress that rewrites either""
MCP_SERVERSComma-separated list of MCP servers as alias=url. Without alias= the alias is derived from the URL host. Aliases must match ^[a-z0-9_-]+$ and namespace the tools as mcp_<alias>_<tool>""
MCP_INCLUDE_TOOLSComma-separated allowlist of MCP tool names to inject. If empty, all tools are injected. Takes precedence over MCP_EXCLUDE_TOOLS""
MCP_EXCLUDE_TOOLSComma-separated denylist of MCP tool names to skip injecting. If empty, no tools are excluded. Takes lower precedence than MCP_INCLUDE_TOOLS""
MCP_TOOL_MODEMCP tool exposure mode: selector (default) injects two meta-tools (mcp_tools_get, mcp_tools_execute); direct injects every tool schema into every requestselector
MCP_CLIENT_TIMEOUTMCP client HTTP timeout5s
MCP_DIAL_TIMEOUTMCP client dial timeout3s
MCP_TLS_HANDSHAKE_TIMEOUTMCP client TLS handshake timeout3s
MCP_RESPONSE_HEADER_TIMEOUTMCP client response header timeout3s
MCP_EXPECT_CONTINUE_TIMEOUTMCP client expect continue timeout1s
MCP_REQUEST_TIMEOUTTimeout for MCP server startup initialization and each server tools/list discovery call. Tool calls are not bounded by it; they use MCP_CLIENT_TIMEOUT and the request context5s
MCP_MAX_RETRIESMaximum number of connection retry attempts3
MCP_RETRY_INTERVALInterval between connection retry attempts5s
MCP_INITIAL_BACKOFFInitial backoff duration for exponential backoff retry1s
MCP_ENABLE_RECONNECTEnable automatic reconnection for failed serverstrue
MCP_RECONNECT_INTERVALInterval between reconnection attempts30s
MCP_POLLING_ENABLEDEnable health check pollingtrue
MCP_POLLING_INTERVALInterval between health check polling requests30s
MCP_POLLING_TIMEOUTTimeout for individual health check requests5s
MCP_DISABLE_HEALTHCHECK_LOGSDisable health check log messages to reduce noisetrue

Each entry in MCP_SERVERS may be written as alias=url. The alias namespaces that server's tools as mcp_<alias>_<tool name>, both for the tools injected into chat completions and for those listed over the /mcp endpoint. When alias= is omitted the alias is derived from the URL host; aliases must match ^[a-z0-9_-]+$, and an invalid, duplicate, or reserved (tools) alias fails startup.

bash
MCP_SERVERS="deepwiki=https://mcp.deepwiki.com/mcp,http://mcp-time-server:8081/mcp"

On Kubernetes the Operator renders this variable for you as name=url entries - spec.mcp.servers[].name for static servers, the discovered MCP CR's metadata.name for spec.mcp.serviceDiscovery - and mirrors the result in status.mcpServers.

MCP_EXPOSE=true (together with MCP_ENABLED=true) serves the gateway's own JSON-RPC MCP endpoint at root-level POST /mcp, which aggregates every configured server behind one URL. The endpoint speaks MCP 2026-07-28 only and is stateless - no initialize handshake, no session - so every request must carry the MCP-Protocol-Version and Mcp-Method headers (Mcp-Name for tools/call); a proxy that strips them breaks the endpoint. It is covered by the gateway's AUTH_* settings like any other route, so with AUTH_ENABLED=true clients must send a bearer token. Left at the default false, POST /mcp answers 403. Because there is no session, /mcp needs no session affinity; and since it is POST only (GET and DELETE answer 405), never point a probe at it - use /health. See the MCP guide.

MCP_RESOURCE_URL is the canonical public URL of POST /mcp (https://gateway.example.com/mcp). It is published as the resource of the OAuth 2.0 Protected Resource Metadata document at GET /.well-known/oauth-protected-resource/mcp and referenced by the resource_metadata parameter of every 401 challenge on /mcp. Left empty it defaults to the request scheme (honouring X-Forwarded-Proto) and Host with /mcp appended - set it explicitly behind an ingress that rewrites either, otherwise clients discover a URL they cannot reach.

Use MCP_INCLUDE_TOOLS and MCP_EXCLUDE_TOOLS to control exactly which discovered tools are injected into LLM requests. Both accept a comma-separated list of tool names and default to empty:

  • MCP_INCLUDE_TOOLS is an allowlist. When empty (the default), all discovered tools are injected. When set, only the listed tools are injected.
  • MCP_EXCLUDE_TOOLS is a denylist. When empty (the default), no tools are excluded. When set, the listed tools are skipped.

MCP_INCLUDE_TOOLS takes precedence over MCP_EXCLUDE_TOOLS: when both are set, the allowlist is applied and the denylist is ignored. Both match either the bare tool name or the namespaced <alias>_<tool name> form, so use the namespaced form to single out one server when several expose the same tool name.

Agent2Agent (A2A) Settings ​

These settings turn the gateway into an A2A server that relays to downstream agents:

VariableDescriptionDefault
A2A_ENABLEDExpose the gateway as an A2A server that delegates every call to the agents in A2A_AGENTS: GET /.well-known/agent-card.json, POST /a2a and GET /a2a/agents. When disabled the routes return 404 and nothing A2A-related runsfalse
A2A_AGENTSComma-separated list of A2A agents as alias=url, e.g. research=http://research-agent:8080. Without alias= the alias is derived from the URL host. Aliases must match ^[a-z0-9_-]+$ and be unique; they prefix the skill ids on the gateway card, name the tenant of the per-agent interface, and prefix every task id the gateway hands out as <alias>:<task id>. Per-agent credentials go in the url as basic auth""
A2A_RESOURCE_URLCanonical public URL of POST /a2a, e.g. https://gateway.example.com/a2a. Published as the url of the gateway agent card interfaces and as the resource of the OAuth 2.0 Protected Resource Metadata (RFC 9728) document served at /.well-known/oauth-protected-resource/a2a. Defaults to the request scheme (honouring X-Forwarded-Proto) and Host with /a2a appended; set it behind an ingress that rewrites either""
A2A_CLIENT_TIMEOUTTimeout for one non-streaming call to an agent, including the agent card fetch30s
A2A_STREAM_IDLE_TIMEOUTIdle cutoff for a relayed SendStreamingMessage or SubscribeToTask stream: the relay closes when the agent sends nothing for this long. 0 disables the cutoff5m
A2A_CARD_REFRESH_INTERVALInterval at which agent cards are re-fetched in the background so unreachable agents are retried and skill changes picked up. Cards are always fetched once at startup; 0 disables the background refresh5m

With A2A_ENABLED=true the gateway registers GET /.well-known/agent-card.json, POST /a2a, GET /a2a/agents and GET /.well-known/oauth-protected-resource/a2a. Left at the default false the routes are not registered at all and answer 404. Every A2A JSON-RPC call on POST /a2a is relayed to the agent the request names, resolved by alias from A2A_AGENTS.

bash
A2A_ENABLED=true
A2A_AGENTS="research=http://research-agent:8080,http://mock-agent:8080"

On Kubernetes the Operator renders A2A_AGENTS for you from spec.a2a.agents[] and spec.a2a.serviceDiscovery, and mirrors the result in status.a2aAgents.

Logging and Debugging ​

These settings control logging and debugging behavior:

VariableDescriptionDefault
ENVIRONMENTLog verbosity is derived from the deployment environment: development selects zap's development config (debug level and human-readable output), any other value selects the JSON production config (info level)production

Environment Variable File (.env) ​

For local development, you can use a .env file. Create a file named .env in your project root:

bash
# .env file example
ENVIRONMENT=development
TELEMETRY_ENABLED=false
OPENAI_API_KEY=your-openai-key
ANTHROPIC_API_KEY=your-anthropic-key

When the gateway is started by the CLI, provider keys missing from both the system environment and the project .env fall back to ~/.infer/auth.yaml (system environment > project .env > ~/.infer/auth.yaml, first hit wins; the legacy ~/.infer/auth.json is still read when auth.yaml is absent). See Provider API keys.

Kubernetes ConfigMaps and Secrets ​

When deploying in Kubernetes, use ConfigMaps for non-sensitive configuration and Secrets for API keys and other sensitive information.

Example ConfigMap ​

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: inference-gateway-config
data:
  ENVIRONMENT: 'production'
  TELEMETRY_ENABLED: 'true'
  SERVER_HOST: '0.0.0.0'
  SERVER_PORT: '8080'
  SERVER_READ_TIMEOUT: '30s'
  SERVER_WRITE_TIMEOUT: '30s'
  SERVER_IDLE_TIMEOUT: '120s'

Example Secret ​

yaml
apiVersion: v1
kind: Secret
metadata:
  name: inference-gateway-secrets
type: Opaque
data:
  ANTHROPIC_API_KEY: '<base64-encoded-key>'
  COHERE_API_KEY: '<base64-encoded-key>'
  OPENAI_API_KEY: '<base64-encoded-key>'

The AUTH_OIDC_* settings are public identifiers, not secrets, so they belong in the ConfigMap above.

Complete Configuration Example ​

Here's a comprehensive example for configuring Inference Gateway in a production environment:

bash

# General settings
ENVIRONMENT=production
ALLOWED_MODELS=
VISION_ENABLED=false
IMAGES_ENABLED=false
AUDIO_ENABLED=false
VIDEOS_ENABLED=false
DEBUG_CONTENT_TRUNCATE_WORDS=10
DEBUG_MAX_MESSAGES=100
# Telemetry
TELEMETRY_ENABLED=false
TELEMETRY_METRICS_PUSH_ENABLED=false
TELEMETRY_METRICS_PORT=9464
TELEMETRY_TRACING_ENABLED=false
TELEMETRY_TRACING_OTLP_ENDPOINT=http://localhost:4318
# Model Context Protocol (MCP)
MCP_ENABLED=false
MCP_EXPOSE=false
MCP_RESOURCE_URL=
MCP_SERVERS=
MCP_INCLUDE_TOOLS=
MCP_EXCLUDE_TOOLS=
MCP_TOOL_MODE=selector
MCP_CLIENT_TIMEOUT=5s
MCP_DIAL_TIMEOUT=3s
MCP_TLS_HANDSHAKE_TIMEOUT=3s
MCP_RESPONSE_HEADER_TIMEOUT=3s
MCP_EXPECT_CONTINUE_TIMEOUT=1s
MCP_REQUEST_TIMEOUT=5s
MCP_MAX_RETRIES=3
MCP_RETRY_INTERVAL=5s
MCP_INITIAL_BACKOFF=1s
MCP_ENABLE_RECONNECT=true
MCP_RECONNECT_INTERVAL=30s
MCP_POLLING_ENABLED=true
MCP_POLLING_INTERVAL=30s
MCP_POLLING_TIMEOUT=5s
MCP_DISABLE_HEALTHCHECK_LOGS=true
# Agent2Agent (A2A)
A2A_ENABLED=false
A2A_AGENTS=
A2A_RESOURCE_URL=
A2A_CLIENT_TIMEOUT=30s
A2A_STREAM_IDLE_TIMEOUT=5m
A2A_CARD_REFRESH_INTERVAL=5m
# Authentication
AUTH_ENABLED=false
AUTH_OIDC_ISSUER=
AUTH_OIDC_CLIENT_ID=
AUTH_OIDC_AUDIENCE=
# Server settings
# 127.0.0.1 is the gateway default; containers need 0.0.0.0, and a value passed
# in through an env_file or ConfigMap overrides the image default
SERVER_HOST=0.0.0.0
SERVER_PORT=8080
SERVER_MAX_REQUEST_BODY_SIZE=10485760
SERVER_READ_TIMEOUT=30s
SERVER_WRITE_TIMEOUT=30s
SERVER_IDLE_TIMEOUT=120s
SERVER_TLS_CERT_PATH=
SERVER_TLS_KEY_PATH=
# Client settings
CLIENT_TIMEOUT=30s
CLIENT_MAX_IDLE_CONNS=20
CLIENT_MAX_IDLE_CONNS_PER_HOST=20
CLIENT_IDLE_CONN_TIMEOUT=30s
CLIENT_TLS_MIN_VERSION=TLS12
CLIENT_DISABLE_COMPRESSION=true
CLIENT_RESPONSE_HEADER_TIMEOUT=10s
CLIENT_EXPECT_CONTINUE_TIMEOUT=1s
# Routing
ROUTING_ENABLED=false
ROUTING_CONFIG_PATH=
# Providers
OPENAI_API_URL=https://api.openai.com/v1
OPENAI_API_KEY=
DEEPSEEK_API_URL=https://api.deepseek.com
DEEPSEEK_API_KEY=
ANTHROPIC_API_URL=https://api.anthropic.com/v1
ANTHROPIC_API_KEY=
COHERE_API_URL=https://api.cohere.ai
COHERE_API_KEY=
GROQ_API_URL=https://api.groq.com/openai/v1
GROQ_API_KEY=
CLOUDFLARE_API_URL=https://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/ai
CLOUDFLARE_API_KEY=
OLLAMA_API_URL=http://ollama:8080/v1
OLLAMA_CLOUD_API_URL=https://ollama.com/v1
OLLAMA_CLOUD_API_KEY=
LLAMACPP_API_URL=http://llamacpp:8080/v1
LLAMACPP_API_KEY=
GOOGLE_API_URL=https://generativelanguage.googleapis.com/v1beta/openai
GOOGLE_API_KEY=
MISTRAL_API_URL=https://api.mistral.ai/v1
MISTRAL_API_KEY=
MINIMAX_API_URL=https://api.minimax.io/v1
MINIMAX_API_KEY=
MOONSHOT_API_URL=https://api.moonshot.ai/v1
MOONSHOT_API_KEY=
NVIDIA_API_URL=https://integrate.api.nvidia.com/v1
NVIDIA_API_KEY=
ZAI_API_URL=https://api.z.ai/api/coding/paas/v4
ZAI_API_KEY=
ELEVENLABS_API_URL=https://api.elevenlabs.io/v1
ELEVENLABS_API_KEY=

Configuration Best Practices ​

  1. API Key Security: Never commit API keys to version control. Use environment variables or secrets management.
  2. TLS in Production: Always use TLS in production environments to secure data in transit.
  3. Authentication: Enable authentication in production environments to control access.
  4. Timeouts: Adjust timeouts based on your expected workloads and response times from LLM providers.
  5. Monitoring: Enable telemetry in production for observability and performance tracking.

Next Steps ​

Once you've configured Inference Gateway, you might want to: