Text-to-Speech (Qwen3-TTS)

The Inference Gateway CLI can turn text into spoken audio through the TextToSpeech agent tool, in two modes:

• Text to speech - text in, spoken .wav out, using a stock voice.
• Voice to voice (voice cloning) - text plus a short reference recording of the target speaker (~10-30s of clean .wav) in, spoken WAV out in that voice. This is the interesting mode for dubbing and video-editing workflows.

Everything runs locally: the tool shells out to llama.cpp's llama-tts binary running Qwen3-TTS GGUF models, the same GGUF ecosystem as the whisper.cpp speech-to-text feature. It is disabled by default: while text_to_speech.enabled is false, the TextToSpeech tool definition is not sent to the LLM at all, so it costs zero prompt tokens.

Note: Text-to-speech shells out to llama-tts and ffmpeg - no CGO is added to the infer binary. When a required tool is missing, the CLI reports an actionable error naming what to install; it never fails silently.

Prerequisites

Tool	Used for	Install
llama-tts	Synthesis	Build the llama-tts target from llama.cpp, or set text_to_speech.binary_path to your build
ffmpeg	Normalizing the voice sample for cloning (16 kHz mono, max 30s)	macOS: brew install ffmpeg - Debian/Ubuntu: apt install ffmpeg

ffmpeg is only needed for voice cloning; stock-voice synthesis passes text straight to llama-tts. If ffmpeg is missing and auto_download is on, a prebuilt binary is downloaded into ~/.infer/bin as a last resort, mirroring speech-to-text.

Building llama-tts is one cmake invocation:

cmake -B build -DGGML_NATIVE=ON && cmake --build build --target llama-tts

Enabling

Add a text_to_speech section to .infer/config.yaml (project) or ~/.infer/config.yaml (user):

text_to_speech:
  enabled: true # feature flag (default: false) - tool absent from the LLM payload when false
  engine: qwen3-tts # synthesis engine (only engine for now)
  model: '' # "" = base preset; q8 | bf16 | or explicit "<backbone>[,<mmproj>].gguf" filenames
  auto_download: true # download models (and ffmpeg) on first use if missing
  output_dir: '' # where generated WAVs go; empty = ~/.infer/tts
  # Optional overrides:
  binary_path: '' # explicit llama-tts path; empty = resolve on PATH
  models_dir: '' # model cache; empty = ~/.infer/models/tts
  timeout: 300 # synthesis timeout (seconds)
  ffmpeg_path: '' # explicit ffmpeg path; empty = resolve on PATH

Configuration reference

All options live under text_to_speech in .infer/config.yaml. Every key also has an INFER_TEXT_TO_SPEECH_-prefixed environment variable that takes precedence over the config file.

Key	Environment variable	Type	Default	Description
text_to_speech.enabled	INFER_TEXT_TO_SPEECH_ENABLED	bool	false	Feature flag - must be true for the TextToSpeech tool to reach the LLM
text_to_speech.engine	INFER_TEXT_TO_SPEECH_ENGINE	string	qwen3-tts	Synthesis engine; only qwen3-tts for now
text_to_speech.binary_path	INFER_TEXT_TO_SPEECH_BINARY_PATH	string	""	Explicit llama-tts path; empty resolves the binary on PATH
text_to_speech.model	INFER_TEXT_TO_SPEECH_MODEL	string	""	Preset (""/base, q8, bf16) or explicit <backbone>[,<mmproj>].gguf filenames
text_to_speech.models_dir	INFER_TEXT_TO_SPEECH_MODELS_DIR	string	""	Where models are cached; empty defaults to ~/.infer/models/tts
text_to_speech.output_dir	INFER_TEXT_TO_SPEECH_OUTPUT_DIR	string	""	Where generated WAVs are written; empty defaults to ~/.infer/tts
text_to_speech.auto_download	INFER_TEXT_TO_SPEECH_AUTO_DOWNLOAD	bool	true	Download models (and ffmpeg) on first use if missing
text_to_speech.timeout	INFER_TEXT_TO_SPEECH_TIMEOUT	int	300	Synthesis timeout in seconds
text_to_speech.ffmpeg_path	INFER_TEXT_TO_SPEECH_FFMPEG_PATH	string	""	Explicit ffmpeg path; empty resolves ffmpeg on PATH

For example:

export INFER_TEXT_TO_SPEECH_ENABLED=true
export INFER_TEXT_TO_SPEECH_MODEL=q8

Choosing a model

The backbone and mmproj (audio adapter) GGUF files are downloaded on first use from huggingface.co/ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF and cached under ~/.infer/models/tts/:

Model	Backbone	Notes
"" / base (default)	Qwen3-TTS-12Hz-1.7B-Base-Q4_K_M.gguf	~1 GB, good balance
q8	Qwen3-TTS-12Hz-1.7B-Base-Q8_0.gguf	~1.9 GB, slightly better quality
bf16	Qwen3-TTS-12Hz-1.7B-Base-bf16.gguf	~3.4 GB, best fidelity

Each preset downloads the matching mmproj-* file automatically. You can also pass explicit filenames as model: '<backbone>.gguf,<mmproj>.gguf' (or just the backbone, in which case the mmproj-<name>-Q8_0.gguf pair is derived), or place both files in models_dir manually and set auto_download: false.

The llama-tts binary is resolved from binary_path, then from PATH. No prebuilt llama-tts asset is published today, so build it once from llama.cpp; if an asset is added later it is downloaded into ~/.infer/bin automatically on first use, like ffmpeg.

Using the tool

With text_to_speech.enabled set, the agent gains a TextToSpeech tool:

• text (required) - the text to speak.
• voice_sample (optional) - path to a WAV of the target speaker. The sample is normalized with ffmpeg (16 kHz mono, capped at 30s) and passed to the engine's --tts-speaker-file for zero-shot cloning.
• output_path (optional) - destination WAV; otherwise a timestamped file is written to output_dir (default ~/.infer/tts/). The tool result reports the path and audio duration.

In chat, just ask: "say this out loud and write it to say.wav" for a stock voice, or "read this in the voice from ~/samples/narrator.wav" to clone.

Voice cloning quality depends entirely on the reference sample: one speaker, minimal background noise, no music, roughly 10-30 seconds.

Troubleshooting

Symptom	What to check
llama-tts binary not found	Build the llama-tts target from llama.cpp, or set text_to_speech.binary_path
ffmpeg not found	Install ffmpeg, or set text_to_speech.ffmpeg_path
tts model ... not found ... auto_download disabled	Enable auto_download, or place the backbone and mmproj GGUFs in models_dir
Slow first call	Models download once (~1 GB by default); subsequent runs use the cache under ~/.infer/models/tts/
Clone sounds wrong	Use a cleaner or longer reference sample (10-30s, single speaker), and try the q8 or bf16 preset
Timeouts on long text	Raise timeout - synthesis takes multiple seconds of compute per second of audio on most hardware
The model never calls TextToSpeech	Set text_to_speech.enabled: true (or INFER_TEXT_TO_SPEECH_ENABLED=true) - the tool is hidden when disabled

Related

• CLI - overview of the infer command-line tool, chat mode, and the full tool reference
• Speech-to-Text - the reverse direction, local transcription with whisper.cpp
• Configuration - full configuration system across the gateway and CLI
• llama.cpp - the local synthesis engine