Inference GatewayOne API for every LLM

Why Inference Gateway? ​

Building against multiple LLM providers means juggling SDKs, API quirks, auth schemes, and streaming protocols that drift constantly. Inference Gateway sits in front of every provider and exposes a single, stable, OpenAI-compatible surface so your application code never has to care which model is on the other end.

• Switch providers with one config change, no application redeploys.
• Centralise API keys, rate limiting, and audit logging at the gateway.
• Add MCP tools or A2A agents once, get them for every model that supports tool calls.
• Run the same binary in Docker, Kubernetes, or as a Helm chart - or let the Kubernetes Operator manage gateways, agents, MCP servers, and orchestrators as Custom Resources.

How it works ​

Inference Gateway acts as an intermediary between your applications and various LLM providers. By standardising the API interactions, it lets you:

• Access multiple LLM providers through a single integration.
• Switch between providers without changing application code.
• Implement sophisticated routing and fallback mechanisms.
• Centralise API key management and security policies.

Model Context Protocol (MCP) ​

Native support for the Model Context Protocol lets LLMs automatically access external tools and data sources. With MCP integration, you can:

• Automatically discover tools from connected MCP servers.
• Execute tool calls seamlessly without client-side management.
• Connect multiple data sources like filesystems, databases, and APIs.
• Extend LLM capabilities with custom tools and integrations.

# Enable MCP with multiple servers
export MCP_ENABLE=true
export MCP_SERVERS="http://filesystem-server:8081/mcp,http://search-server:8082/mcp"

# LLMs automatically get access to all available tools
curl -X POST http://localhost:8080/v1/chat/completions \
  -d '{"model": "openai/gpt-4o", "messages": [{"role": "user", "content": "List files and search for recent AI news"}]}'

Learn more about MCP Integration and explore the examples.

Agent-to-Agent (A2A) ​

Agent-to-Agent support lets LLMs coordinate with multiple specialised agents in a single conversation. Agents can:

• Coordinate multiple agents in a single conversation.
• Access specialised services like calendars, calculators, and weather APIs.
• Discover agent capabilities automatically.
• Scale agent ecosystems with distributed architecture.

The best way to use A2A is through the Inference Gateway CLI, which provides seamless integration with A2A agents:

# Install the CLI
curl -fsSL https://raw.githubusercontent.com/inference-gateway/cli/main/install.sh | bash

# Initialize and start chatting
infer init
infer chat

# Delegate tasks to A2A agents
> "Schedule a team meeting for tomorrow at 2 PM"
> "Check my calendar for conflicts this week"

Learn more about A2A Integration and see how to build your own agents.

Community ​

Inference Gateway is an open-source project maintained by a growing community. Contributions are welcome on GitHub.