Ollama Provider on Bifrost

Ollama provider summary

Ollama is a self-hosted inference engine with identical request/response format to OpenAI. Bifrost routes to Ollama with full streaming, embeddings, and tool calling support. Typically runs locally on http://localhost:11434.

Supported operations

Ollama supports 5 major operations. Chat completions, responses API, and text completions support full streaming. Embeddings can customize dimensionality.

Parameter handling

Ollama accepts identical request format to OpenAI with streaming responses. Embeddings support custom dimension specification via dimensions parameter. Tool calling and function definitions fully supported.

Supported Ollama parameters

Quick reference of OpenAI-compatible parameters accepted when routing through Bifrost to Ollama.

[
  "stream",
  "temperature",
  "top_p",
  "top_k",
  "max_tokens",
  "stop"
]

Popular models

Common Ollama models for local inference. Pull with ollama pull <model> before routing through Bifrost with the ollama/ prefix. See Popular models in Bifrost docs.

Context windows vary by model (for example Llama 3.1 70B supports up to 128K tokens). Use stream: true for better UX with larger models; Ollama uses GPU acceleration when available.

API reference by operation

OpenAI-compatible endpoints for self-hosted Ollama instances.

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ollama/llama3.1:latest",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

ResponsesRequest → ChatRequest → ChatCompletion → ResponsesResponse

curl -X POST http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ollama/llama3.1:latest",
    "input": "Hello",
    "max_output_tokens": 1024
  }'

curl -X POST http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ollama/mistral:latest",
    "prompt": "Hello, my name is",
    "max_tokens": 50
  }'

curl -X POST http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ollama/mxbai-embed-large:latest",
    "input": "Hello world",
    "dimensions": 1024
  }'

curl http://localhost:8080/v1/models

Implementation caveats