Memory Management & Performance Tuning

​

Overview

• Concurrency (concurrency): Controls the number of simultaneous requests to each provider.
• Request Buffering (buffer_size): Defines the queue size for pending requests for each provider.
• Object Pooling (initial_pool_size): Pre-allocates memory for request/response objects to reduce garbage collection overhead.

​

1. Concurrency Control (concurrency)

• What it is: The maximum number of simultaneous requests Bifrost will make to a single provider’s API.
• Impact: Directly controls the throughput for each provider.
• Trade-offs:
  • Higher Concurrency: Increases throughput but also increases the risk of hitting API rate limits. Consumes more memory and CPU for in-flight requests.
  • Lower Concurrency: Reduces the risk of rate limiting and consumes fewer resources, but may limit throughput.
• Configuration: This is configured on a per-provider basis. Go Package - Concurrency Configuration Concurrency is set within the ProviderConfig returned by your Account implementation.
  // In your Account implementation
  func (a *MyAccount) GetConfigForProvider(provider schemas.ModelProvider) (*schemas.ProviderConfig, error) {
      // ...
      return &schemas.ProviderConfig{
          ConcurrencyAndBufferSize: schemas.ConcurrencyAndBufferSize{
              Concurrency: 10, // 10 concurrent workers for this provider
              BufferSize:  50,
          },
          // ...
      }, nil
  }
  
  HTTP Transport - Concurrency Configuration Concurrency is set in your config.json under each provider’s concurrency_and_buffer_size.
  {
      "providers": {
          "openai": {
              // ...
              "concurrency_and_buffer_size": {
                  "concurrency": 10,
                  "buffer_size": 50
              }
          }
      }
  }
  

​

2. Request Queuing (buffer_size)

• What it is: The number of requests that can be queued for a provider before new requests either block or are dropped.
• Impact: Helps Bifrost absorb traffic bursts without losing requests.
• Trade-offs:
  • Larger Buffer: Can handle larger bursts of traffic, preventing blocking. However, it consumes more memory to hold the queued request objects.
  • Smaller Buffer: Consumes less memory but may cause requests to block or be dropped during traffic spikes if workers can’t keep up.
• dropExcessRequests: If the buffer is full, the behavior depends on the global dropExcessRequests setting (Go package only).
  • false (default): New requests will block until space is available in the queue.
  • true: New requests are immediately dropped with an error.
  Go Package - Buffer Configuration The buffer size is set alongside concurrency in ProviderConfig.
  // In your Account implementation
  func (a *MyAccount) GetConfigForProvider(provider schemas.ModelProvider) (*schemas.ProviderConfig, error) {
      // ...
      return &schemas.ProviderConfig{
          ConcurrencyAndBufferSize: schemas.ConcurrencyAndBufferSize{
              Concurrency: 10,
              BufferSize:  50, // Queue up to 50 requests
          },
          // ...
      }, nil
  }
  
  // Global config for dropping excess requests
  bifrost, err := bifrost.Init(schemas.BifrostConfig{
      //...
      DropExcessRequests: true, // Drop requests when queue is full
  })
  
  HTTP Transport - Buffer Configuration The buffer size is set in your config.json. Note that dropExcessRequests is not configurable for the HTTP transport and defaults to false (blocking).
  {
      "providers": {
          "openai": {
              // ...
              "concurrency_and_buffer_size": {
                  "concurrency": 10,
                  "buffer_size": 50
              }
          }
      }
  }
  

​

3. Object Pooling (initial_pool_size)

• What it is: A global setting that pre-allocates a specified number of objects for requests, responses, and errors.
• Impact: Significantly reduces memory allocation and GC pressure during high-traffic scenarios.
• Trade-offs:
  • Larger Pool: Improves performance under heavy load by minimizing allocations. Increases the initial memory footprint of Bifrost.
  • Smaller Pool: Lower initial memory usage, but may lead to more GC activity and higher latency under load.
• Configuration: This is a global setting. For the Go package, it is set in BifrostConfig. For the HTTP transport, it’s configured via command-line flags or environment variables, not in config.json. Go Package - Object Pool Configuration

// Global config for object pooling
bifrost, err := bifrost.Init(schemas.BifrostConfig{
    Account:        myAccount,
    InitialPoolSize: 1000, // Pre-allocate 1000 objects of each type
    // ...
})

bifrost-http -config config.json -port 8080 -pool-size 1000

docker run -p 8080:8080 \
  -v $(pwd)/config.json:/app/config/config.json \
  -e APP_POOL_SIZE=1000 \
  -e OPENAI_API_KEY \
  maximhq/bifrost

​

Future Development

​

Dynamic Scaling

Note: This feature is under active development.

​

Configuration Recommendations

​

Monitoring Memory

• Prometheus Metrics: The HTTP transport exposes metrics at the /metrics endpoint. While there are no specific memory metrics, you can monitor go_memstats_* to observe memory usage.
• Go Profiling (pprof): For detailed memory analysis when using the Go package, use the standard net/http/pprof tool to inspect heap allocations and goroutine counts.

Tip: Start with the Balanced configuration and adjust based on observed performance and resource utilization. For example, if you see requests blocking frequently, increase buffer_size. If your provider rate limits are being hit, decrease concurrency.