Text-to-Video and Avatars

The Inference Gateway CLI can render short video clips through the TextToVideo agent tool, in two modes:

• Prompt render - a text prompt in, an MP4 out. Optionally pass a portrait as the first frame.
• Avatar render (lip-sync) - a portrait plus a .wav or .mp3 clip in, an MP4 of that face speaking the audio out. Pair it with Text-to-Speech to go from a script to a talking-head clip without leaving the chat.

Rendering goes through the gateway's Videos API (POST /v1/videos, then polling GET /v1/videos/{id} until the job finishes and downloading GET /v1/videos/{id}/content) - the CLI holds no provider key, the gateway does, so renders show up in gateway logs, traces and pricing.

Disabled by default. While text_to_video.enabled is false, the TextToVideo tool definition is not sent to the LLM at all, so it costs zero prompt tokens. Avatar renders upload a picture of your face and a recording of your voice to a third-party provider - that is why this feature is opt-in rather than on by default.

Enabling

Add a top-level text_to_video section to .infer/config.yaml (project) or ~/.infer/config.yaml (user):

text_to_video:
  enabled: true # feature flag (default: false) - tool absent from the LLM payload when false
  # model: elevenlabs/veo-3.1-fast-generate-001 # prompt renders
  # avatar_model: elevenlabs/creatify-aurora # lip-synced avatar renders
  # size: '' # "widthxheight", passed through to the provider; empty = provider default
  # output_dir: ~/.infer/tmp/video
  # timeout: 900 # seconds to wait for a render job
  # poll_interval: 5 # seconds between job status polls
  require_approval: false # optional; unset = no approval, like the image tools

Gateway requirements: the Videos API must be enabled on the gateway (VIDEOS_ENABLED=true, gateway v0.54.0 or newer) and the gateway must hold credentials for the provider behind model / avatar_model (an ElevenLabs API key for the defaults). A gateway the CLI starts itself gets VIDEOS_ENABLED=true automatically while text_to_video.enabled is on, and an already-running instance without the Videos API is restarted. Point the CLI at an externally managed gateway and you set VIDEOS_ENABLED=true and the provider key there yourself.

Configuration reference

All options live under text_to_video in .infer/config.yaml. Every key also has an INFER_TEXT_TO_VIDEO_-prefixed environment variable that takes precedence over the config file.

Config key	Environment variable	Type	Default	Notes
text_to_video.enabled	INFER_TEXT_TO_VIDEO_ENABLED	bool	false	Feature flag - must be true for the TextToVideo tool to reach the LLM
text_to_video.model	INFER_TEXT_TO_VIDEO_MODEL	string	elevenlabs/veo-3.1-fast-generate-001	Gateway provider/model id used for prompt renders
text_to_video.avatar_model	INFER_TEXT_TO_VIDEO_AVATAR_MODEL	string	elevenlabs/creatify-aurora	Gateway provider/model id used for lip-synced avatar renders
text_to_video.size	INFER_TEXT_TO_VIDEO_SIZE	string	""	widthxheight passed through to the provider; empty leaves the provider default
text_to_video.output_dir	INFER_TEXT_TO_VIDEO_OUTPUT_DIR	string	~/.infer/tmp/video	Where rendered MP4s are written
text_to_video.timeout	INFER_TEXT_TO_VIDEO_TIMEOUT	int	900	Seconds to wait for a render job before giving up
text_to_video.poll_interval	INFER_TEXT_TO_VIDEO_POLL_INTERVAL	int	5	Seconds between job status polls
text_to_video.require_approval	INFER_TEXT_TO_VIDEO_REQUIRE_APPROVAL	bool	unset (no approval)	Tri-state: unset keeps the tool's own default, an explicit value pins the policy either way

For example:

export INFER_TEXT_TO_VIDEO_ENABLED=true
export INFER_TEXT_TO_VIDEO_AVATAR_MODEL=elevenlabs/creatify-aurora

Using the tool

With text_to_video.enabled set, the agent gains a TextToVideo tool - see TextToVideo in the tools reference for the full parameter list. In chat, just ask:

• "render a 5 second clip of rain on a window" - prompt render with model.
• "make my alice avatar say this" (with a generated or recorded audio clip) - lip-synced render with avatar_model.

Limits: the portrait plus the audio clip travel in one request, so together they must stay under the gateway's 10 MiB request body limit - trim the clip or downscale the portrait if a render is rejected. creatify-aurora renders at 480p or 720p and keeps the portrait's aspect ratio, so a vertical portrait yields a vertical video.

The avatar library

Avatars live under ~/.infer/avatars/, one folder per avatar holding one or more portrait images (.png, .jpg, .jpeg, .webp) - for example a few angles of the same face:

~/.infer/avatars/
  alice/
    alice.png          # primary image, first in sort order
    alice-left.png
    alice-right.png
  bob/
    bob.jpg

Lip-sync models take a single image, so the CLI uses the first image in sort order within the folder. The library is userspace-wide (shared by every project) and is preserved by /reset along with the rest of your configuration.

Managing avatars

# List avatars and their images
infer avatars list
infer avatars list --format json

# Create an avatar from a photo, generating extra views
infer avatars create alice --from ~/Pictures/alice.png

# Delete an avatar folder and its images
infer avatars delete alice

infer avatars create <name> --from <photo> copies the photo in as the primary image and then generates additional views of the same face - by default both three-quarter angles - through the gateway's image edit API using the tools.image_edit.model model.

Flag	Default	Description
--from	required	Path to the source photo
--angles	both angles	Which extra views to generate
--quality	high	Image quality passed to the image edit API
--size	1024x1536	Size of the generated views

Privacy: generating views uploads the photo to the image-edit provider (OpenAI by default), and an avatar render uploads the portrait and the audio clip to the video provider. Drop images into ~/.infer/avatars/<name>/ by hand to build a library without the image-edit round trip.

Troubleshooting

Symptom	What to check
The model never calls TextToVideo	Set text_to_video.enabled: true (or INFER_TEXT_TO_VIDEO_ENABLED=true) - the tool is hidden when disabled
404 or "videos API not enabled"	Run the gateway with VIDEOS_ENABLED=true (gateway v0.54.0+); an externally managed gateway is not reconfigured for you
Request rejected as too large	Portrait plus audio must fit in the gateway's 10 MiB request body - shorten the clip or downscale the portrait
The render times out	Raise text_to_video.timeout - renders routinely take minutes
avatar ... not found	Check infer avatars list; the folder name is the avatar name and it must contain at least one supported image
The wrong face angle is used for a lip-sync	Lip-sync uses the first image in sort order - rename the preferred portrait so it sorts first

Related

• CLI - the tool reference entry, parameters, and the rest of the infer command-line tool
• Text-to-Speech - generate the voice track an avatar render lip-syncs to
• TextToMusic and TextToSFX - the audio siblings, with the same output_path rules
• Configuration - full configuration system across the gateway and CLI