Skip to main content

​
Overview

Key Management Features:
  • ✅ Multiple Keys per Provider - Distribute load across multiple API keys
  • ✅ Weighted Distribution - Control traffic distribution with custom weights
  • ✅ Model-Specific Keys - Assign keys to specific models only
  • ✅ Automatic Rotation - Seamless failover when keys are rate-limited
  • ✅ Load Balancing - Intelligent request distribution
  • ✅ Cost Optimization - Use different keys for different cost tiers
Benefits:
  • Higher Rate Limits - Combine multiple keys for increased throughput
  • Improved Reliability - Automatic failover prevents service interruption
  • Cost Control - Route traffic based on budget and usage patterns
  • Zero Downtime - Hot-swap keys without service interruption

​
Basic Key Setup

​
Single Key Configuration

Go Package Usage 
func (a *MyAccount) GetKeysForProvider(provider schemas.ModelProvider) ([]schemas.Key, error) {
    switch provider {
    case schemas.OpenAI:
        return []schemas.Key{
            {
                Value:  os.Getenv("OPENAI_API_KEY"),
                Models: []string{"gpt-4o-mini", "gpt-4o"},
                Weight: 1.0,  // 100% of traffic
            },
        }, nil
    case schemas.Anthropic:
        return []schemas.Key{
            {
                Value:  os.Getenv("ANTHROPIC_API_KEY"),
                Models: []string{"claude-3-5-sonnet-20241022"},
                Weight: 1.0,
            },
        }, nil
    }
    return nil, fmt.Errorf("provider not configured")
}
HTTP Transport Usage Configuration (config.json): 
{
    "providers": {
        "openai": {
            "keys": [
                {
                    "value": "env.OPENAI_API_KEY",
                    "models": ["gpt-4o-mini", "gpt-4o"],
                    "weight": 1.0
                }
            ]
        },
        "anthropic": {
            "keys": [
                {
                    "value": "env.ANTHROPIC_API_KEY",
                    "models": ["claude-3-5-sonnet-20241022"],
                    "weight": 1.0
                }
            ]
        }
    }
}
Environment variables: 
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."

​
Key Distribution Strategies

​
Load Balancing Strategy

Distribute requests evenly across multiple keys for maximum throughput: Go Package - Equal Distribution 
func (a *MyAccount) GetKeysForProvider(provider schemas.ModelProvider) ([]schemas.Key, error) {
    if provider == schemas.OpenAI {
        return []schemas.Key{
            {
                Value:  os.Getenv("OPENAI_KEY_1"),
                Models: []string{"gpt-4o-mini", "gpt-4o"},
                Weight: 0.25,  // 25% each for even distribution
            },
            {
                Value:  os.Getenv("OPENAI_KEY_2"),
                Models: []string{"gpt-4o-mini", "gpt-4o"},
                Weight: 0.25,
            },
            {
                Value:  os.Getenv("OPENAI_KEY_3"),
                Models: []string{"gpt-4o-mini", "gpt-4o"},
                Weight: 0.25,
            },
            {
                Value:  os.Getenv("OPENAI_KEY_4"),
                Models: []string{"gpt-4o-mini", "gpt-4o"},
                Weight: 0.25,
            },
        }, nil
    }
    return nil, fmt.Errorf("provider not configured")
}
HTTP Transport - Equal Distribution 
{
    "providers": {
        "openai": {
            "keys": [
                {
                    "value": "env.OPENAI_KEY_1",
                    "models": ["gpt-4o-mini", "gpt-4o"],
                    "weight": 0.25
                },
                {
                    "value": "env.OPENAI_KEY_2",
                    "models": ["gpt-4o-mini", "gpt-4o"],
                    "weight": 0.25
                },
                {
                    "value": "env.OPENAI_KEY_3",
                    "models": ["gpt-4o-mini", "gpt-4o"],
                    "weight": 0.25
                },
                {
                    "value": "env.OPENAI_KEY_4",
                    "models": ["gpt-4o-mini", "gpt-4o"],
                    "weight": 0.25
                }
            ]
        }
    }
}
Environment setup: 
export OPENAI_KEY_1="sk-1..."
export OPENAI_KEY_2="sk-2..."
export OPENAI_KEY_3="sk-3..."
export OPENAI_KEY_4="sk-4..."

​
Tiered Access Strategy

Use premium keys for expensive models, standard keys for cheaper models: Go Package - Tiered Strategy 
func (a *MyAccount) GetKeysForProvider(provider schemas.ModelProvider) ([]schemas.Key, error) {
    if provider == schemas.OpenAI {
        return []schemas.Key{
            // Standard keys for cheap models
            {
                Value:  os.Getenv("OPENAI_STANDARD_KEY_1"),
                Models: []string{"gpt-4o-mini"},  // Cheap model only
                Weight: 0.4,
            },
            {
                Value:  os.Getenv("OPENAI_STANDARD_KEY_2"),
                Models: []string{"gpt-4o-mini"},
                Weight: 0.3,
            },
            // Premium keys for expensive models
            {
                Value:  os.Getenv("OPENAI_PREMIUM_KEY_1"),
                Models: []string{"gpt-4o", "gpt-4o-mini"},  // All models
                Weight: 0.2,
            },
            {
                Value:  os.Getenv("OPENAI_PREMIUM_KEY_2"),
                Models: []string{"gpt-4o", "gpt-4o-mini"},
                Weight: 0.1,
            },
        }, nil
    }
    return nil, fmt.Errorf("provider not configured")
}
Result: Cost optimization with dedicated premium keys for expensive models HTTP Transport - Tiered Strategy 
{
    "providers": {
        "openai": {
            "keys": [
                {
                    "value": "env.OPENAI_STANDARD_KEY_1",
                    "models": ["gpt-4o-mini"],
                    "weight": 0.4
                },
                {
                    "value": "env.OPENAI_STANDARD_KEY_2",
                    "models": ["gpt-4o-mini"],
                    "weight": 0.3
                },
                {
                    "value": "env.OPENAI_PREMIUM_KEY_1",
                    "models": ["gpt-4o", "gpt-4o-mini"],
                    "weight": 0.2
                },
                {
                    "value": "env.OPENAI_PREMIUM_KEY_2",
                    "models": ["gpt-4o", "gpt-4o-mini"],
                    "weight": 0.1
                }
            ]
        }
    }
}

​
Priority-Based Strategy

Route traffic based on key priority and reliability: Go Package - Priority Strategy 
func (a *MyAccount) GetKeysForProvider(provider schemas.ModelProvider) ([]schemas.Key, error) {
    if provider == schemas.OpenAI {
        return []schemas.Key{
            // Primary key (highest priority)
            {
                Value:  os.Getenv("OPENAI_PRIMARY_KEY"),
                Models: []string{"gpt-4o-mini", "gpt-4o"},
                Weight: 0.6,  // 60% traffic to primary
            },
            // Secondary keys (backup)
            {
                Value:  os.Getenv("OPENAI_BACKUP_KEY_1"),
                Models: []string{"gpt-4o-mini", "gpt-4o"},
                Weight: 0.3,  // 30% to first backup
            },
            {
                Value:  os.Getenv("OPENAI_BACKUP_KEY_2"),
                Models: []string{"gpt-4o-mini", "gpt-4o"},
                Weight: 0.1,  // 10% to second backup
            },
        }, nil
    }
    return nil, fmt.Errorf("provider not configured")
}
HTTP Transport - Priority Strategy 
{
    "providers": {
        "openai": {
            "keys": [
                {
                    "value": "env.OPENAI_PRIMARY_KEY",
                    "models": ["gpt-4o-mini", "gpt-4o"],
                    "weight": 0.6
                },
                {
                    "value": "env.OPENAI_BACKUP_KEY_1",
                    "models": ["gpt-4o-mini", "gpt-4o"],
                    "weight": 0.3
                },
                {
                    "value": "env.OPENAI_BACKUP_KEY_2",
                    "models": ["gpt-4o-mini", "gpt-4o"],
                    "weight": 0.1
                }
            ]
        }
    }
}

​
Advanced Key Patterns

​
Multi-Provider Key Management

Go Package - Cross-Provider Keys 
func (a *MyAccount) GetKeysForProvider(provider schemas.ModelProvider) ([]schemas.Key, error) {
    switch provider {
    case schemas.OpenAI:
        return []schemas.Key{
            {
                Value:  os.Getenv("OPENAI_KEY_1"),
                Models: []string{"gpt-4o-mini", "gpt-4o"},
                Weight: 0.7,
            },
            {
                Value:  os.Getenv("OPENAI_KEY_2"),
                Models: []string{"gpt-4o"},
                Weight: 0.3,
            },
        }, nil
    case schemas.Anthropic:
        return []schemas.Key{
            {
                Value:  os.Getenv("ANTHROPIC_KEY_1"),
                Models: []string{"claude-3-5-sonnet-20241022"},
                Weight: 0.8,
            },
            {
                Value:  os.Getenv("ANTHROPIC_KEY_2"),
                Models: []string{"claude-3-5-sonnet-20241022"},
                Weight: 0.2,
            },
        }, nil
    case schemas.Bedrock:
        return []schemas.Key{
            {
                Value:  os.Getenv("AWS_ACCESS_KEY_ID"),
                Models: []string{"anthropic.claude-3-5-sonnet-20241022-v2:0"},
                Weight: 1.0,
            },
        }, nil
    }
    return nil, fmt.Errorf("provider %s not configured", provider)
}
HTTP Transport - Cross-Provider Keys 
{
    "providers": {
        "openai": {
            "keys": [
                {
                    "value": "env.OPENAI_KEY_1",
                    "models": ["gpt-4o-mini", "gpt-4o"],
                    "weight": 0.7
                },
                {
                    "value": "env.OPENAI_KEY_2",
                    "models": ["gpt-4o"],
                    "weight": 0.3
                }
            ]
        },
        "anthropic": {
            "keys": [
                {
                    "value": "env.ANTHROPIC_KEY_1",
                    "models": ["claude-3-5-sonnet-20241022"],
                    "weight": 0.8
                },
                {
                    "value": "env.ANTHROPIC_KEY_2",
                    "models": ["claude-3-5-sonnet-20241022"],
                    "weight": 0.2
                }
            ]
        },
        "bedrock": {
            "keys": [
                {
                    "value": "env.AWS_ACCESS_KEY_ID",
                    "models": ["anthropic.claude-3-5-sonnet-20241022-v2:0"],
                    "weight": 1.0
                }
            ],
            "meta_config": {
                "region": "us-east-1",
                "secret_access_key": "env.AWS_SECRET_ACCESS_KEY"
            }
        }
    }
}

​
Dynamic Key Selection

Go Package - Runtime Key Selection 
type DynamicAccount struct {
    keyRotationInterval time.Duration
    lastRotation        time.Time
    currentKeyIndex     int
    keys                map[schemas.ModelProvider][]schemas.Key
}

func (a *DynamicAccount) GetKeysForProvider(provider schemas.ModelProvider) ([]schemas.Key, error) {
    // Rotate keys every hour
    if time.Since(a.lastRotation) > a.keyRotationInterval {
        a.rotateKeys()
        a.lastRotation = time.Now()
    }

    if keys, exists := a.keys[provider]; exists {
        return keys, nil
    }
    return nil, fmt.Errorf("provider not configured")
}

func (a *DynamicAccount) rotateKeys() {
    // Implement key rotation logic
    // Could fetch new keys from secret management system
    log.Info("Rotating API keys...")
}
HTTP Transport - Hot Key Reload This feature is under development.

​
Key Selection Algorithm

Bifrost uses weighted random selection for key distribution: 
Key Selection Process:
1. Filter keys by requested model
2. Calculate total weight of available keys
3. Generate random number between 0 and total weight
4. Select key based on weighted probability
5. Fallback to next available key if selected key fails
Example with 3 keys:
KeyWeightProbabilityTraffic Distribution
Key A0.550%~50% of requests
Key B0.330%~30% of requests
Key C0.220%~20% of requests

​
Best Practices

​
Security Best Practices

Environment Variable Management Recommended approach: 
# Use descriptive naming
export OPENAI_PRIMARY_KEY="sk-..."
export OPENAI_FALLBACK_KEY="sk-..."
export ANTHROPIC_PRODUCTION_KEY="sk-ant-..."

# Avoid hardcoding in config files
# ❌ Bad
{
  "value": "sk-actual-key-here"
}

# ✅ Good
{
  "value": "env.OPENAI_API_KEY"
}
Key Rotation Schedule Recommended rotation schedule: 
• Production keys: Every 30 days
• Development keys: Every 90 days
• Backup keys: Every 60 days
• Emergency keys: Keep fresh, rotate every 14 days
Implementation: 
// Track key age and force rotation
type KeyWithMetadata struct {
    schemas.Key
    CreatedAt time.Time
    LastUsed  time.Time
}

func (k *KeyWithMetadata) ShouldRotate() bool {
    return time.Since(k.CreatedAt) > 30*24*time.Hour // 30 days
}

​
Performance Optimization

Weight Optimization High-throughput scenario: 
{
    "providers": {
        "openai": {
            "keys": [
                {
                    "value": "env.OPENAI_HIGH_LIMIT_KEY",
                    "models": ["gpt-4o-mini"],
                    "weight": 0.8
                },
                {
                    "value": "env.OPENAI_STANDARD_KEY",
                    "models": ["gpt-4o-mini"],
                    "weight": 0.2
                }
            ]
        }
    }
}
Cost-optimized scenario: 
{
    "providers": {
        "openai": {
            "keys": [
                {
                    "value": "env.OPENAI_CHEAP_KEY",
                    "models": ["gpt-4o-mini"],
                    "weight": 0.9
                },
                {
                    "value": "env.OPENAI_PREMIUM_KEY",
                    "models": ["gpt-4o"],
                    "weight": 0.1
                }
            ]
        }
    }
}

​
Next Steps

TaskDocumentation
Configure providersProviders
Set up networkingNetworking
Optimize performanceMemory Management
Handle failuresError Handling
Tip: Use weights that sum to 1.0 for easier percentage calculations, but Bifrost automatically normalizes weights if they don’t sum to 1.0.