Configuration

Inference Gateway provides flexible configuration options to adapt to your specific needs. As a proxy server designed to facilitate access to various language model APIs, proper configuration is essential for optimal performance and security.

Configuration Methods

Inference Gateway supports multiple configuration methods to suit different deployment scenarios:

1. Environment Variables - Recommended for most deployments
2. Kubernetes ConfigMaps and Secrets - For Kubernetes-based deployments
3. Configuration Files - For local development and testing

Environment Variables

Environment variables are the primary method for configuring Inference Gateway. These variables control everything from basic server settings to provider-specific API configurations.

General Settings

When ENABLE_TELEMETRY is set to true, Inference Gateway exposes a /metrics endpoint for Prometheus scraping and generates distributed traces that can be collected by OpenTelemetry collectors.

OpenID Connect

If authentication is enabled (ENABLE_AUTH=true), configure the following OIDC settings:

When authentication is enabled, all API requests must include a valid JWT token in the Authorization header:

Authorization: Bearer YOUR_JWT_TOKEN

Server Settings

These settings control the core HTTP server behavior:

For production deployments, it's strongly recommended to configure TLS:

SERVER_TLS_CERT_PATH=/path/to/certificate.pem
SERVER_TLS_KEY_PATH=/path/to/private-key.pem

Client Settings

These settings control how Inference Gateway connects to third-party APIs:

For high-throughput deployments, consider increasing the connection pool settings:

CLIENT_MAX_IDLE_CONNS=100
CLIENT_MAX_IDLE_CONNS_PER_HOST=50

Provider Settings

Configure access to various LLM providers. At minimum, you should configure the providers you plan to use.

OpenAI

Anthropic

Cohere

Groq

Ollama

Cloudflare

DeepSeek

Environment Variable File (.env)

For local development, you can use a .env file. Create a file named .env in your project root:

# .env file example
ENVIRONMENT=development
ENABLE_TELEMETRY=false
OPENAI_API_KEY=your-openai-key
ANTHROPIC_API_KEY=your-anthropic-key

Kubernetes ConfigMaps and Secrets

When deploying in Kubernetes, use ConfigMaps for non-sensitive configuration and Secrets for API keys and other sensitive information.

Example ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: inference-gateway-config
data:
  ENVIRONMENT: 'production'
  ENABLE_TELEMETRY: 'true'
  SERVER_HOST: '0.0.0.0'
  SERVER_PORT: '8080'
  SERVER_READ_TIMEOUT: '30s'
  SERVER_WRITE_TIMEOUT: '30s'
  SERVER_IDLE_TIMEOUT: '120s'

Example Secret

apiVersion: v1
kind: Secret
metadata:
  name: inference-gateway-secrets
type: Opaque
data:
  ANTHROPIC_API_KEY: '<base64-encoded-key>'
  COHERE_API_KEY: '<base64-encoded-key>'
  OPENAI_API_KEY: '<base64-encoded-key>'
  OIDC_CLIENT_SECRET: '<base64-encoded-key>'

Complete Configuration Example

Here's a comprehensive example for configuring Inference Gateway in a production environment:

# General settings
ENVIRONMENT=production
ENABLE_TELEMETRY=true
ENABLE_AUTH=true

# Authentication
OIDC_ISSUER_URL=https://auth.example.com/realms/inference-gateway
OIDC_CLIENT_ID=inference-gateway
OIDC_CLIENT_SECRET=your-client-secret

# Server settings
SERVER_HOST=0.0.0.0
SERVER_PORT=8080
SERVER_READ_TIMEOUT=30s
SERVER_WRITE_TIMEOUT=30s
SERVER_IDLE_TIMEOUT=120s
SERVER_TLS_CERT_PATH=/certs/tls.crt
SERVER_TLS_KEY_PATH=/certs/tls.key

# Client settings
CLIENT_TIMEOUT=45s
CLIENT_MAX_IDLE_CONNS=100
CLIENT_MAX_IDLE_CONNS_PER_HOST=50
CLIENT_IDLE_CONN_TIMEOUT=60s
CLIENT_TLS_MIN_VERSION=TLS12

# Provider settings
OPENAI_API_KEY=your-openai-api-key
ANTHROPIC_API_KEY=your-anthropic-api-key
GROQ_API_KEY=your-groq-api-key

Configuration Best Practices

1. API Key Security: Never commit API keys to version control. Use environment variables or secrets management.
2. TLS in Production: Always use TLS in production environments to secure data in transit.
3. Authentication: Enable authentication in production environments to control access.
4. Timeouts: Adjust timeouts based on your expected workloads and response times from LLM providers.
5. Monitoring: Enable telemetry in production for observability and performance tracking.

Next Steps

Once you've configured Inference Gateway, you might want to:

• Check out the API Reference for details on available endpoints
• Explore SDK options for integrating with your application
• Review Observability options for monitoring and logging