Inference Gateway

Appearance

Configuration ​

Breaking change (schemas v0.15.3): All _ENABLE env-var suffixes have been renamed to _ENABLED. Existing deployments must update their configuration: TELEMETRY_ENABLE -> TELEMETRY_ENABLED, TELEMETRY_METRICS_PUSH_ENABLE -> TELEMETRY_METRICS_PUSH_ENABLED, TELEMETRY_TRACING_ENABLE -> TELEMETRY_TRACING_ENABLED, MCP_ENABLE -> MCP_ENABLED, MCP_POLLING_ENABLE -> MCP_POLLING_ENABLED, AUTH_ENABLE -> AUTH_ENABLED. (MCP_ENABLE_RECONNECT is unchanged.)

Inference Gateway provides flexible configuration options to adapt to your specific needs. As a proxy server designed to facilitate access to various language model APIs, proper configuration is essential for optimal performance and security.

Configuration Methods ​

Inference Gateway supports multiple configuration methods to suit different deployment scenarios:

  1. Environment Variables - Recommended for most deployments
  2. Kubernetes ConfigMaps and Secrets - For Kubernetes-based deployments
  3. Configuration Files - For local development and testing

Environment Variables ​

Environment variables are the primary method for configuring Inference Gateway. These variables control everything from basic server settings to provider-specific API configurations.

General Settings ​

VariableDescriptionDefault
ENVIRONMENTDeployment environmentproduction
ALLOWED_MODELSComma-separated list of models to allow. If empty, all models will be available""
DISALLOWED_MODELSComma-separated list of models to disallow. If empty, no models will be blocked. Takes lower precedence than ALLOWED_MODELS""
ENABLE_VISIONEnable vision/multimodal support for all providersfalse
DEBUG_CONTENT_TRUNCATE_WORDSNumber of words to truncate per content section in debug logs (development mode only)10
DEBUG_MAX_MESSAGESMaximum number of messages to show in debug logs (development mode only)100
AUTH_ENABLEDEnable OIDC authenticationfalse

When ENABLE_VISION is set to true, Inference Gateway enables vision/multimodal capabilities, allowing you to send images alongside text in chat completion requests. When disabled (default), requests with image content will be rejected even if the provider and model support vision. This is disabled by default for performance and security reasons.

Telemetry ​

These settings control telemetry and metrics exposure:

VariableDescriptionDefault
TELEMETRY_ENABLEDEnable OpenTelemetry metrics and tracingfalse
TELEMETRY_METRICS_PUSH_ENABLEDEnable the OTLP metrics push endpoint (POST /v1/metrics)false
TELEMETRY_METRICS_PORTPort for telemetry metrics server9464
TELEMETRY_TRACING_ENABLEDEmit OpenTelemetry tracing spans (requires TELEMETRY_ENABLED)false
TELEMETRY_TRACING_OTLP_ENDPOINTOTLP/HTTP endpoint for trace exporthttp://localhost:4318
OTEL_METRICS_EXPORTERMetrics exporter mode: otlp (push), prometheus (pull), or noneotlp
OTEL_EXPORTER_PROMETHEUS_HOSTPrometheus pull endpoint bind host0.0.0.0
OTEL_EXPORTER_PROMETHEUS_PORTPrometheus pull endpoint port9464

When TELEMETRY_ENABLED is set to true, Inference Gateway exposes a /metrics endpoint for Prometheus scraping. Distributed tracing is a separate opt-in: set TELEMETRY_TRACING_ENABLED=true (alongside TELEMETRY_ENABLED=true) to emit spans, and point TELEMETRY_TRACING_OTLP_ENDPOINT at your OTLP/HTTP collector (default http://localhost:4318). Sampling and exporter tuning use the standard OTEL_TRACES_SAMPLER / OTEL_EXPORTER_OTLP_* variables. See the Distributed Tracing section for span coverage and context propagation.

When TELEMETRY_METRICS_PUSH_ENABLED is also set to true (alongside TELEMETRY_ENABLED=true), the gateway exposes an OTLP/HTTP metrics push endpoint at POST /v1/metrics. This allows subscription clients that bypass the gateway's inference path to push their usage metrics. See the Observability page for details.

OpenID Connect ​

If authentication is enabled (AUTH_ENABLED=true), configure the following OIDC settings:

VariableDescriptionDefault
AUTH_OIDC_ISSUEROIDC issuer URLhttp://keycloak:8080/realms/inference-gateway-realm
AUTH_OIDC_CLIENT_IDOIDC client IDinference-gateway-client
AUTH_OIDC_CLIENT_SECRETOIDC client secret""

When authentication is enabled, all API requests must include a valid JWT token in the Authorization header:

http
Authorization: Bearer YOUR_JWT_TOKEN

Server Settings ​

These settings control the core HTTP server behavior:

VariableDescriptionDefault
SERVER_HOSTServer host0.0.0.0
SERVER_PORTServer port8080
SERVER_READ_TIMEOUTRead timeout30s
SERVER_WRITE_TIMEOUTWrite timeout30s
SERVER_IDLE_TIMEOUTIdle timeout120s
SERVER_TLS_CERT_PATHTLS certificate path""
SERVER_TLS_KEY_PATHTLS key path""

For production deployments, it's strongly recommended to configure TLS:

bash
SERVER_TLS_CERT_PATH=/path/to/certificate.pem
SERVER_TLS_KEY_PATH=/path/to/private-key.pem

Client Settings ​

These settings control how Inference Gateway connects to third-party APIs:

VariableDescriptionDefault
CLIENT_TIMEOUTClient timeout30s
CLIENT_MAX_IDLE_CONNSMaximum idle connections20
CLIENT_MAX_IDLE_CONNS_PER_HOSTMaximum idle connections per host20
CLIENT_IDLE_CONN_TIMEOUTIdle connection timeout30s
CLIENT_TLS_MIN_VERSIONMinimum TLS versionTLS12
CLIENT_DISABLE_COMPRESSIONDisable compression for faster streamingtrue
CLIENT_RESPONSE_HEADER_TIMEOUTResponse header timeout10s
CLIENT_EXPECT_CONTINUE_TIMEOUTExpect continue timeout1s

For high-throughput deployments, consider increasing the connection pool settings:

bash
CLIENT_MAX_IDLE_CONNS=100
CLIENT_MAX_IDLE_CONNS_PER_HOST=50

Provider Settings ​

Configure access to various LLM providers. At minimum, you should configure the providers you plan to use.

OpenAI ​

VariableDescriptionDefault
OPENAI_API_URLOpenAI API URLhttps://api.openai.com/v1
OPENAI_API_KEYOpenAI API Key""

DeepSeek ​

VariableDescriptionDefault
DEEPSEEK_API_URLDeepSeek API URLhttps://api.deepseek.com
DEEPSEEK_API_KEYDeepSeek API Key""

Anthropic ​

VariableDescriptionDefault
ANTHROPIC_API_URLAnthropic API URLhttps://api.anthropic.com/v1
ANTHROPIC_API_KEYAnthropic API Key""

Cohere ​

VariableDescriptionDefault
COHERE_API_URLCohere API URLhttps://api.cohere.ai
COHERE_API_KEYCohere API Key""

Groq ​

VariableDescriptionDefault
GROQ_API_URLGroq API URLhttps://api.groq.com/openai/v1
GROQ_API_KEYGroq API Key""

Cloudflare ​

VariableDescriptionDefault
CLOUDFLARE_API_URLCloudflare API URLhttps://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/ai
CLOUDFLARE_API_KEYCloudflare API Key""

Ollama Cloud ​

VariableDescriptionDefault
OLLAMA_CLOUD_API_URLOllama Cloud API URLhttps://ollama.com/v1
OLLAMA_CLOUD_API_KEYOllama Cloud API Key""

Ollama ​

VariableDescriptionDefault
OLLAMA_API_URLOllama API URLhttp://ollama:8080/v1
OLLAMA_API_KEYOllama API Key""

llama.cpp ​

VariableDescriptionDefault
LLAMACPP_API_URLllama.cpp API URLhttp://llamacpp:8080/v1
LLAMACPP_API_KEYllama.cpp API Key""

Google ​

VariableDescriptionDefault
GOOGLE_API_URLGoogle AI API URLhttps://generativelanguage.googleapis.com/v1beta/openai
GOOGLE_API_KEYGoogle AI API Key""

Mistral ​

VariableDescriptionDefault
MISTRAL_API_URLMistral AI API URLhttps://api.mistral.ai/v1
MISTRAL_API_KEYMistral AI API Key""

MiniMax ​

VariableDescriptionDefault
MINIMAX_API_URLMiniMax API URLhttps://api.minimax.io/v1
MINIMAX_API_KEYMiniMax API Key""

Moonshot ​

VariableDescriptionDefault
MOONSHOT_API_URLMoonshot AI API URLhttps://api.moonshot.ai/v1
MOONSHOT_API_KEYMoonshot AI API Key""

NVIDIA ​

VariableDescriptionDefault
NVIDIA_API_URLNVIDIA NIM API URLhttps://integrate.api.nvidia.com/v1
NVIDIA_API_KEYNVIDIA API Key""

Z-AI ​

VariableDescriptionDefault
ZAI_API_URLZ-AI API URLhttps://api.z.ai/api/paas/v4
ZAI_API_KEYZ-AI API Key""

Routing ​

These settings control gateway-native model routing - mapping a logical model alias to a pool of upstream provider deployments:

VariableDescriptionDefault
ROUTING_ENABLEDEnable gateway-native model routing: logical model aliases backed by a pool of upstream provider deployments, selected round-robin per replica. Opt-in; when disabled, direct provider/model routing is unchangedfalse
ROUTING_CONFIG_PATHPath to a YAML file mapping logical model aliases to their upstream deployment pools. Required when ROUTING_ENABLED is true""

Routing is opt-in. When ROUTING_ENABLED=false (the default), the gateway routes only by explicit provider/model prefix or ?provider= query parameter. When enabled, set ROUTING_CONFIG_PATH to a YAML file that maps each logical alias to its deployment pool. Selection is round-robin per replica, and the chosen upstream is returned in the X-Selected-Provider / X-Selected-Model response headers. See the Model Routing guide for the file format, semantics, and a migration example.

Model Context Protocol (MCP) Settings ​

These settings control MCP integration for external tool access:

VariableDescriptionDefault
MCP_ENABLEDEnable MCP middlewarefalse
MCP_EXPOSEExpose MCP endpoints for debuggingfalse
MCP_SERVERSComma-separated list of MCP server URLs""
MCP_INCLUDE_TOOLSComma-separated allowlist of MCP tool names to inject. If empty, all tools are injected. Takes precedence over MCP_EXCLUDE_TOOLS""
MCP_EXCLUDE_TOOLSComma-separated denylist of MCP tool names to skip injecting. If empty, no tools are excluded. Takes lower precedence than MCP_INCLUDE_TOOLS""
MCP_TOOL_MODEMCP tool exposure mode: selector (default) injects two meta-tools (mcp_tools_get, mcp_tools_execute); direct injects every tool schema into every requestselector
MCP_CLIENT_TIMEOUTMCP client HTTP timeout5s
MCP_DIAL_TIMEOUTMCP client dial timeout3s
MCP_TLS_HANDSHAKE_TIMEOUTMCP client TLS handshake timeout3s
MCP_RESPONSE_HEADER_TIMEOUTMCP client response header timeout3s
MCP_EXPECT_CONTINUE_TIMEOUTMCP client expect continue timeout1s
MCP_REQUEST_TIMEOUTMCP client request timeout for initialize and tool calls5s
MCP_MAX_RETRIESMaximum number of connection retry attempts3
MCP_RETRY_INTERVALInterval between connection retry attempts5s
MCP_INITIAL_BACKOFFInitial backoff duration for exponential backoff retry1s
MCP_ENABLE_RECONNECTEnable automatic reconnection for failed serverstrue
MCP_RECONNECT_INTERVALInterval between reconnection attempts30s
MCP_POLLING_ENABLEDEnable health check pollingtrue
MCP_POLLING_INTERVALInterval between health check polling requests30s
MCP_POLLING_TIMEOUTTimeout for individual health check requests5s
MCP_DISABLE_HEALTHCHECK_LOGSDisable health check log messages to reduce noisetrue

Use MCP_INCLUDE_TOOLS and MCP_EXCLUDE_TOOLS to control exactly which discovered tools are injected into LLM requests. Both accept a comma-separated list of tool names and default to empty:

  • MCP_INCLUDE_TOOLS is an allowlist. When empty (the default), all discovered tools are injected. When set, only the listed tools are injected.
  • MCP_EXCLUDE_TOOLS is a denylist. When empty (the default), no tools are excluded. When set, the listed tools are skipped.

MCP_INCLUDE_TOOLS takes precedence over MCP_EXCLUDE_TOOLS: when both are set, the allowlist is applied and the denylist is ignored.

Logging and Debugging ​

These settings control logging and debugging behavior:

VariableDescriptionDefault
LOG_LEVELSet logging level (debug, info, warn, error)info

Environment Variable File (.env) ​

For local development, you can use a .env file. Create a file named .env in your project root:

bash
# .env file example
ENVIRONMENT=development
TELEMETRY_ENABLED=false
OPENAI_API_KEY=your-openai-key
ANTHROPIC_API_KEY=your-anthropic-key

Kubernetes ConfigMaps and Secrets ​

When deploying in Kubernetes, use ConfigMaps for non-sensitive configuration and Secrets for API keys and other sensitive information.

Example ConfigMap ​

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: inference-gateway-config
data:
  ENVIRONMENT: 'production'
  TELEMETRY_ENABLED: 'true'
  SERVER_HOST: '0.0.0.0'
  SERVER_PORT: '8080'
  SERVER_READ_TIMEOUT: '30s'
  SERVER_WRITE_TIMEOUT: '30s'
  SERVER_IDLE_TIMEOUT: '120s'

Example Secret ​

yaml
apiVersion: v1
kind: Secret
metadata:
  name: inference-gateway-secrets
type: Opaque
data:
  ANTHROPIC_API_KEY: '<base64-encoded-key>'
  COHERE_API_KEY: '<base64-encoded-key>'
  OPENAI_API_KEY: '<base64-encoded-key>'
  AUTH_OIDC_CLIENT_SECRET: '<base64-encoded-key>'

Complete Configuration Example ​

Here's a comprehensive example for configuring Inference Gateway in a production environment:

bash

# General settings
ENVIRONMENT=production
ALLOWED_MODELS=
ENABLE_VISION=false
DEBUG_CONTENT_TRUNCATE_WORDS=10
DEBUG_MAX_MESSAGES=100
# Telemetry
TELEMETRY_ENABLED=false
TELEMETRY_METRICS_PUSH_ENABLED=false
TELEMETRY_METRICS_PORT=9464
TELEMETRY_TRACING_ENABLED=false
TELEMETRY_TRACING_OTLP_ENDPOINT=http://localhost:4318
OTEL_METRICS_EXPORTER=otlp
OTEL_EXPORTER_PROMETHEUS_HOST=0.0.0.0
OTEL_EXPORTER_PROMETHEUS_PORT=9464
# Model Context Protocol (MCP)
MCP_ENABLED=false
MCP_EXPOSE=false
MCP_SERVERS=
MCP_INCLUDE_TOOLS=
MCP_EXCLUDE_TOOLS=
MCP_TOOL_MODE=selector
MCP_CLIENT_TIMEOUT=5s
MCP_DIAL_TIMEOUT=3s
MCP_TLS_HANDSHAKE_TIMEOUT=3s
MCP_RESPONSE_HEADER_TIMEOUT=3s
MCP_EXPECT_CONTINUE_TIMEOUT=1s
MCP_REQUEST_TIMEOUT=5s
MCP_MAX_RETRIES=3
MCP_RETRY_INTERVAL=5s
MCP_INITIAL_BACKOFF=1s
MCP_ENABLE_RECONNECT=true
MCP_RECONNECT_INTERVAL=30s
MCP_POLLING_ENABLED=true
MCP_POLLING_INTERVAL=30s
MCP_POLLING_TIMEOUT=5s
MCP_DISABLE_HEALTHCHECK_LOGS=true
# Authentication
AUTH_ENABLED=false
AUTH_OIDC_ISSUER=http://keycloak:8080/realms/inference-gateway-realm
AUTH_OIDC_CLIENT_ID=inference-gateway-client
AUTH_OIDC_CLIENT_SECRET=
# Server settings
SERVER_HOST=0.0.0.0
SERVER_PORT=8080
SERVER_READ_TIMEOUT=30s
SERVER_WRITE_TIMEOUT=30s
SERVER_IDLE_TIMEOUT=120s
SERVER_TLS_CERT_PATH=
SERVER_TLS_KEY_PATH=
# Client settings
CLIENT_TIMEOUT=30s
CLIENT_MAX_IDLE_CONNS=20
CLIENT_MAX_IDLE_CONNS_PER_HOST=20
CLIENT_IDLE_CONN_TIMEOUT=30s
CLIENT_TLS_MIN_VERSION=TLS12
CLIENT_DISABLE_COMPRESSION=true
CLIENT_RESPONSE_HEADER_TIMEOUT=10s
CLIENT_EXPECT_CONTINUE_TIMEOUT=1s
# Routing
ROUTING_ENABLED=false
ROUTING_CONFIG_PATH=
# Providers
OPENAI_API_URL=https://api.openai.com/v1
OPENAI_API_KEY=
DEEPSEEK_API_URL=https://api.deepseek.com
DEEPSEEK_API_KEY=
ANTHROPIC_API_URL=https://api.anthropic.com/v1
ANTHROPIC_API_KEY=
COHERE_API_URL=https://api.cohere.ai
COHERE_API_KEY=
GROQ_API_URL=https://api.groq.com/openai/v1
GROQ_API_KEY=
CLOUDFLARE_API_URL=https://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/ai
CLOUDFLARE_API_KEY=
OLLAMA_API_URL=http://ollama:8080/v1
OLLAMA_API_KEY=
OLLAMA_CLOUD_API_URL=https://ollama.com/v1
OLLAMA_CLOUD_API_KEY=
GOOGLE_API_URL=https://generativelanguage.googleapis.com/v1beta/openai
GOOGLE_API_KEY=
MISTRAL_API_URL=https://api.mistral.ai/v1
MISTRAL_API_KEY=
MINIMAX_API_URL=https://api.minimax.io/v1
MINIMAX_API_KEY=
MOONSHOT_API_URL=https://api.moonshot.ai/v1
MOONSHOT_API_KEY=
NVIDIA_API_URL=https://integrate.api.nvidia.com/v1
NVIDIA_API_KEY=
ZAI_API_URL=https://api.z.ai/api/paas/v4
ZAI_API_KEY=

Configuration Best Practices ​

  1. API Key Security: Never commit API keys to version control. Use environment variables or secrets management.
  2. TLS in Production: Always use TLS in production environments to secure data in transit.
  3. Authentication: Enable authentication in production environments to control access.
  4. Timeouts: Adjust timeouts based on your expected workloads and response times from LLM providers.
  5. Monitoring: Enable telemetry in production for observability and performance tracking.

Next Steps ​

Once you've configured Inference Gateway, you might want to: