Multimodal embeddings in Bifrost: Embed images, audio, video and PDFs through a single API

Bifrost's /v1/embeddings API can now embed images, audio, video and PDFs, and Bifrost translates each request into the format of the model serving it.

Engineering teams use Bifrost as a single Gateway for their model traffic. Their text embedding calls already go through one /v1/embeddings API, with the same keys and governance irrespective of the provider being called.

But what happens when the content you need to search isn't text, such as product photos or recorded support calls?

Gemini Embedding 2 embeds audio, video and PDFs. Cohere Embed v4 embeds interleaved text and images. Vertex's multimodalembedding@001 returns a separate vector per modality and slices video into timed segments. Amazon Nova embeds audio and video straight from S3. Each of those APIs has its own request shape, its own set of knobs and its own response envelope.

Previously, Bifrost's /v1/embeddings API carried text only. Embedding anything else meant writing each provider's own request format and reading its own response, and rewriting both whenever you switched models.

As an engineer building with these capabilities and routing traffic through Bifrost, you would want to know:

• Can one request mix text with an image, audio, video or a PDF, and get one vector back?
• What does each model accept, and what does it send back?
• Will my existing text embedding calls keep working as they are?

Now, Bifrost's /v1/embeddings can carry text, image, audio, video and file parts, and Bifrost translates them for Gemini, Vertex AI, Cohere and Bedrock (Titan Multimodal  and Amazon Nova 2). Parasail now serves embeddings too. This is available in both open source and Bifrost Enterprise.

Send text, images, audio, video and PDFs in one request

Before, input was a string, an array of strings, or token arrays. Those still work and still go out on the wire exactly as they did. What is new is that each entry in input can be an array of typed parts, and all the parts in one entry produce one vector:

The part types are text, image, audio, video, file and tokens. Media parts take either inline data (a base64 data URI) or a url, plus an optional mime_type and filename. Validation runs before the request reaches any plugin or provider: each part must set exactly one modality, media parts must set exactly one of data or url, and an empty entry is a 400, not a silent no-op.

The point of the entry-is-a-list design is that it maps cleanly onto every provider. OpenAI sees a list of strings. Cohere v4 sees an inputs[] of content blocks. Gemini sees a content.parts[]. Vertex multimodalembedding@001 sees one instance with text, image and video fields. The Gateway does that translation, and when a provider genuinely cannot carry a part, you get a 400 that names the modality rather than a vector that quietly ignored your data.

Embed queries and documents in one call

Any entry in input can be an object with content and its own params. The item's params replace the request-level params for that one entry, and plain entries can be mixed in freely:

Only a wire format with one parameter block per entry can honour this, and today that is Gemini's :batchEmbedContents. Every other provider returns a 400 naming the offending item index. We chose that over silently dropping the override, because a RETRIEVAL_DOCUMENT vector served for a RETRIEVAL_QUERY request looks correct and is wrong.

The request-level parameters also grew. task_type, title and auto_truncate are now first-class fields next to dimensions and encoding_format, and the OpenAI-compatible converter now sends only encoding_format and dimensions. That matters more than it sounds: strict providers such as Mistral returned 422 when the whole parameter struct was copied onto the wire, which also meant a Gemini request carrying task_type could never fall back to an OpenAI-shaped provider.

Tell vectors apart in the response

Each entry in data still has index, object and embedding. Three fields were added, all optional, all omitted when a provider does not need them:

• modality: text, image, audio, video or audio_video. Set only when a provider returns separate vectors per modality.
• video_segment: { "start_offset_sec", "end_offset_sec" } on video vectors that cover a slice of the clip.
• encoding_format: float, int8, uint8, binary, ubinary or base64, so that when a provider returns several representations of one input you can tell them apart. Without it, int8 and binary are indistinguishable once decoded.

Vertex multimodalembedding@001 uses the first two. Send it a text part and a video part with a video_config, and it returns one text vector plus one vector per video segment:

encoding_format: "base64" now works everywhere. Where the model has a native base64 output (OpenAI, Cohere v4) it is forwarded. Where it does not (Titan, Cohere v3, Nova, Vertex), Bifrost encodes the float vector as little-endian float32 base64 itself, which is the same layout OpenAI uses, so a client that decodes OpenAI base64 embeddings decodes these too.

Model by model

The canonical shape is the same everywhere; what each model accepts is not.

Text-only models, such as gemini-embedding-001, the Vertex text models and Titan Text, are in the full provider table, along with each model's limits.

Keep using the Google GenAI, Cohere and Bedrock SDKs

The native dialects got the same treatment so that existing Google GenAI, Cohere and AWS SDKs keep working:

• GenAI :embedContent and :batchEmbedContents. The Google GenAI SDK repeats taskType and outputDimensionality on every batch entry. Bifrost now hoists identical per-entry params to the request level, so these requests work against OpenAI, Vertex, Bedrock and Cohere models too. When entries differ, each keeps its own params and Gemini honours them. :embedContent with a non-text part that the target cannot carry now returns the conversion error instead of silently dropping the part.
• /cohere/v2/embed returns Cohere's native envelope (embeddings, id, response_type, meta.billed_units) for every provider, so the Cohere SDK parses it whether the model behind it is Cohere, OpenAI or Bedrock.
• Bedrock invoke recognises inputImage, embeddingConfig and SINGLE_EMBEDDING as embedding bodies and rebuilds the matching Titan, Nova or Cohere envelope.

Go SDK migration

This is the one breaking change. EmbeddingInput was a struct with Text, Texts, Embedding and Embeddings fields; it is now []EmbeddingInputItem, each holding an EmbeddingContent slice of typed parts and an optional Params override.

The HTTP API is backward compatible. Strings, string arrays and token arrays decode to the same wire bytes they always did. The one exception is Titan Text: a batch now returns a 400 instead of being silently collapsed into one vector.

Get started today

Update to Bifrost v2.3.0 or later. Examples for every part type and provider are in the multimodal embeddings guide, with the Go equivalents in the Go SDK guide.