Key Management

Overview

Key Management Features:

✅ Multiple Keys per Provider - Distribute load across multiple API keys
✅ Weighted Distribution - Control traffic distribution with custom weights
✅ Model-Specific Keys - Assign keys to specific models only
✅ Automatic Rotation - Seamless failover when keys are rate-limited
✅ Load Balancing - Intelligent request distribution
✅ Cost Optimization - Use different keys for different cost tiers

Benefits:

Higher Rate Limits - Combine multiple keys for increased throughput
Improved Reliability - Automatic failover prevents service interruption
Cost Control - Route traffic based on budget and usage patterns
Zero Downtime - Hot-swap keys without service interruption

Basic Key Setup

Single Key Configuration

Go Package Usage

func (a *MyAccount) GetKeysForProvider(provider schemas.ModelProvider) ([]schemas.Key, error) {
    switch provider {
    case schemas.OpenAI:
        return []schemas.Key{
            {
                Value:  os.Getenv("OPENAI_API_KEY"),
                Models: []string{"gpt-4o-mini", "gpt-4o"},
                Weight: 1.0,  // 100% of traffic
            },
        }, nil
    case schemas.Anthropic:
        return []schemas.Key{
            {
                Value:  os.Getenv("ANTHROPIC_API_KEY"),
                Models: []string{"claude-3-5-sonnet-20241022"},
                Weight: 1.0,
            },
        }, nil
    }
    return nil, fmt.Errorf("provider not configured")
}

HTTP Transport Usage Configuration (config.json):

{
    "providers": {
        "openai": {
            "keys": [
                {
                    "value": "env.OPENAI_API_KEY",
                    "models": ["gpt-4o-mini", "gpt-4o"],
                    "weight": 1.0
                }
            ]
        },
        "anthropic": {
            "keys": [
                {
                    "value": "env.ANTHROPIC_API_KEY",
                    "models": ["claude-3-5-sonnet-20241022"],
                    "weight": 1.0
                }
            ]
        }
    }
}

Environment variables:

export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."

Key Distribution Strategies

Load Balancing Strategy

Distribute requests evenly across multiple keys for maximum throughput: Go Package - Equal Distribution

func (a *MyAccount) GetKeysForProvider(provider schemas.ModelProvider) ([]schemas.Key, error) {
    if provider == schemas.OpenAI {
        return []schemas.Key{
            {
                Value:  os.Getenv("OPENAI_KEY_1"),
                Models: []string{"gpt-4o-mini", "gpt-4o"},
                Weight: 0.25,  // 25% each for even distribution
            },
            {
                Value:  os.Getenv("OPENAI_KEY_2"),
                Models: []string{"gpt-4o-mini", "gpt-4o"},
                Weight: 0.25,
            },
            {
                Value:  os.Getenv("OPENAI_KEY_3"),
                Models: []string{"gpt-4o-mini", "gpt-4o"},
                Weight: 0.25,
            },
            {
                Value:  os.Getenv("OPENAI_KEY_4"),
                Models: []string{"gpt-4o-mini", "gpt-4o"},
                Weight: 0.25,
            },
        }, nil
    }
    return nil, fmt.Errorf("provider not configured")
}

HTTP Transport - Equal Distribution

{
    "providers": {
        "openai": {
            "keys": [
                {
                    "value": "env.OPENAI_KEY_1",
                    "models": ["gpt-4o-mini", "gpt-4o"],
                    "weight": 0.25
                },
                {
                    "value": "env.OPENAI_KEY_2",
                    "models": ["gpt-4o-mini", "gpt-4o"],
                    "weight": 0.25
                },
                {
                    "value": "env.OPENAI_KEY_3",
                    "models": ["gpt-4o-mini", "gpt-4o"],
                    "weight": 0.25
                },
                {
                    "value": "env.OPENAI_KEY_4",
                    "models": ["gpt-4o-mini", "gpt-4o"],
                    "weight": 0.25
                }
            ]
        }
    }
}

Environment setup:

export OPENAI_KEY_1="sk-1..."
export OPENAI_KEY_2="sk-2..."
export OPENAI_KEY_3="sk-3..."
export OPENAI_KEY_4="sk-4..."

Tiered Access Strategy

Use premium keys for expensive models, standard keys for cheaper models: Go Package - Tiered Strategy

func (a *MyAccount) GetKeysForProvider(provider schemas.ModelProvider) ([]schemas.Key, error) {
    if provider == schemas.OpenAI {
        return []schemas.Key{
            // Standard keys for cheap models
            {
                Value:  os.Getenv("OPENAI_STANDARD_KEY_1"),
                Models: []string{"gpt-4o-mini"},  // Cheap model only
                Weight: 0.4,
            },
            {
                Value:  os.Getenv("OPENAI_STANDARD_KEY_2"),
                Models: []string{"gpt-4o-mini"},
                Weight: 0.3,
            },
            // Premium keys for expensive models
            {
                Value:  os.Getenv("OPENAI_PREMIUM_KEY_1"),
                Models: []string{"gpt-4o", "gpt-4o-mini"},  // All models
                Weight: 0.2,
            },
            {
                Value:  os.Getenv("OPENAI_PREMIUM_KEY_2"),
                Models: []string{"gpt-4o", "gpt-4o-mini"},
                Weight: 0.1,
            },
        }, nil
    }
    return nil, fmt.Errorf("provider not configured")
}

Result: Cost optimization with dedicated premium keys for expensive models HTTP Transport - Tiered Strategy

{
    "providers": {
        "openai": {
            "keys": [
                {
                    "value": "env.OPENAI_STANDARD_KEY_1",
                    "models": ["gpt-4o-mini"],
                    "weight": 0.4
                },
                {
                    "value": "env.OPENAI_STANDARD_KEY_2",
                    "models": ["gpt-4o-mini"],
                    "weight": 0.3
                },
                {
                    "value": "env.OPENAI_PREMIUM_KEY_1",
                    "models": ["gpt-4o", "gpt-4o-mini"],
                    "weight": 0.2
                },
                {
                    "value": "env.OPENAI_PREMIUM_KEY_2",
                    "models": ["gpt-4o", "gpt-4o-mini"],
                    "weight": 0.1
                }
            ]
        }
    }
}

Priority-Based Strategy

Route traffic based on key priority and reliability: Go Package - Priority Strategy

func (a *MyAccount) GetKeysForProvider(provider schemas.ModelProvider) ([]schemas.Key, error) {
    if provider == schemas.OpenAI {
        return []schemas.Key{
            // Primary key (highest priority)
            {
                Value:  os.Getenv("OPENAI_PRIMARY_KEY"),
                Models: []string{"gpt-4o-mini", "gpt-4o"},
                Weight: 0.6,  // 60% traffic to primary
            },
            // Secondary keys (backup)
            {
                Value:  os.Getenv("OPENAI_BACKUP_KEY_1"),
                Models: []string{"gpt-4o-mini", "gpt-4o"},
                Weight: 0.3,  // 30% to first backup
            },
            {
                Value:  os.Getenv("OPENAI_BACKUP_KEY_2"),
                Models: []string{"gpt-4o-mini", "gpt-4o"},
                Weight: 0.1,  // 10% to second backup
            },
        }, nil
    }
    return nil, fmt.Errorf("provider not configured")
}

HTTP Transport - Priority Strategy

{
    "providers": {
        "openai": {
            "keys": [
                {
                    "value": "env.OPENAI_PRIMARY_KEY",
                    "models": ["gpt-4o-mini", "gpt-4o"],
                    "weight": 0.6
                },
                {
                    "value": "env.OPENAI_BACKUP_KEY_1",
                    "models": ["gpt-4o-mini", "gpt-4o"],
                    "weight": 0.3
                },
                {
                    "value": "env.OPENAI_BACKUP_KEY_2",
                    "models": ["gpt-4o-mini", "gpt-4o"],
                    "weight": 0.1
                }
            ]
        }
    }
}

Advanced Key Patterns

Multi-Provider Key Management

Go Package - Cross-Provider Keys

func (a *MyAccount) GetKeysForProvider(provider schemas.ModelProvider) ([]schemas.Key, error) {
    switch provider {
    case schemas.OpenAI:
        return []schemas.Key{
            {
                Value:  os.Getenv("OPENAI_KEY_1"),
                Models: []string{"gpt-4o-mini", "gpt-4o"},
                Weight: 0.7,
            },
            {
                Value:  os.Getenv("OPENAI_KEY_2"),
                Models: []string{"gpt-4o"},
                Weight: 0.3,
            },
        }, nil
    case schemas.Anthropic:
        return []schemas.Key{
            {
                Value:  os.Getenv("ANTHROPIC_KEY_1"),
                Models: []string{"claude-3-5-sonnet-20241022"},
                Weight: 0.8,
            },
            {
                Value:  os.Getenv("ANTHROPIC_KEY_2"),
                Models: []string{"claude-3-5-sonnet-20241022"},
                Weight: 0.2,
            },
        }, nil
    case schemas.Bedrock:
        return []schemas.Key{
            {
                Value:  os.Getenv("AWS_ACCESS_KEY_ID"),
                Models: []string{"anthropic.claude-3-5-sonnet-20241022-v2:0"},
                Weight: 1.0,
            },
        }, nil
    }
    return nil, fmt.Errorf("provider %s not configured", provider)
}

HTTP Transport - Cross-Provider Keys

{
    "providers": {
        "openai": {
            "keys": [
                {
                    "value": "env.OPENAI_KEY_1",
                    "models": ["gpt-4o-mini", "gpt-4o"],
                    "weight": 0.7
                },
                {
                    "value": "env.OPENAI_KEY_2",
                    "models": ["gpt-4o"],
                    "weight": 0.3
                }
            ]
        },
        "anthropic": {
            "keys": [
                {
                    "value": "env.ANTHROPIC_KEY_1",
                    "models": ["claude-3-5-sonnet-20241022"],
                    "weight": 0.8
                },
                {
                    "value": "env.ANTHROPIC_KEY_2",
                    "models": ["claude-3-5-sonnet-20241022"],
                    "weight": 0.2
                }
            ]
        },
        "bedrock": {
            "keys": [
                {
                    "value": "env.AWS_ACCESS_KEY_ID",
                    "models": ["anthropic.claude-3-5-sonnet-20241022-v2:0"],
                    "weight": 1.0
                }
            ],
            "meta_config": {
                "region": "us-east-1",
                "secret_access_key": "env.AWS_SECRET_ACCESS_KEY"
            }
        }
    }
}

Dynamic Key Selection

Go Package - Runtime Key Selection

type DynamicAccount struct {
    keyRotationInterval time.Duration
    lastRotation        time.Time
    currentKeyIndex     int
    keys                map[schemas.ModelProvider][]schemas.Key
}

func (a *DynamicAccount) GetKeysForProvider(provider schemas.ModelProvider) ([]schemas.Key, error) {
    // Rotate keys every hour
    if time.Since(a.lastRotation) > a.keyRotationInterval {
        a.rotateKeys()
        a.lastRotation = time.Now()
    }

    if keys, exists := a.keys[provider]; exists {
        return keys, nil
    }
    return nil, fmt.Errorf("provider not configured")
}

func (a *DynamicAccount) rotateKeys() {
    // Implement key rotation logic
    // Could fetch new keys from secret management system
    log.Info("Rotating API keys...")
}

HTTP Transport - Hot Key Reload This feature is under development.

Key Selection Algorithm

Bifrost uses weighted random selection for key distribution:

Key Selection Process:
Filter keys by requested model
Calculate total weight of available keys
Generate random number between 0 and total weight
Select key based on weighted probability
Fallback to next available key if selected key fails

Example with 3 keys:

Key	Weight	Probability	Traffic Distribution
Key A	0.5	50%	~50% of requests
Key B	0.3	30%	~30% of requests
Key C	0.2	20%	~20% of requests

Best Practices

Security Best Practices

Environment Variable Management Recommended approach:

# Use descriptive naming
export OPENAI_PRIMARY_KEY="sk-..."
export OPENAI_FALLBACK_KEY="sk-..."
export ANTHROPIC_PRODUCTION_KEY="sk-ant-..."

# Avoid hardcoding in config files
# ❌ Bad
{
  "value": "sk-actual-key-here"
}

# ✅ Good
{
  "value": "env.OPENAI_API_KEY"
}

Key Rotation Schedule Recommended rotation schedule:

• Production keys: Every 30 days
• Development keys: Every 90 days
• Backup keys: Every 60 days
• Emergency keys: Keep fresh, rotate every 14 days

Implementation:

// Track key age and force rotation
type KeyWithMetadata struct {
    schemas.Key
    CreatedAt time.Time
    LastUsed  time.Time
}

func (k *KeyWithMetadata) ShouldRotate() bool {
    return time.Since(k.CreatedAt) > 30*24*time.Hour // 30 days
}

Performance Optimization

Weight Optimization High-throughput scenario:

{
    "providers": {
        "openai": {
            "keys": [
                {
                    "value": "env.OPENAI_HIGH_LIMIT_KEY",
                    "models": ["gpt-4o-mini"],
                    "weight": 0.8
                },
                {
                    "value": "env.OPENAI_STANDARD_KEY",
                    "models": ["gpt-4o-mini"],
                    "weight": 0.2
                }
            ]
        }
    }
}

Cost-optimized scenario:

{
    "providers": {
        "openai": {
            "keys": [
                {
                    "value": "env.OPENAI_CHEAP_KEY",
                    "models": ["gpt-4o-mini"],
                    "weight": 0.9
                },
                {
                    "value": "env.OPENAI_PREMIUM_KEY",
                    "models": ["gpt-4o"],
                    "weight": 0.1
                }
            ]
        }
    }
}

Next Steps

Task	Documentation
Configure providers	Providers
Set up networking	Networking
Optimize performance	Memory Management
Handle failures	Error Handling

Tip: Use weights that sum to 1.0 for easier percentage calculations, but Bifrost automatically normalizes weights if they don’t sum to 1.0.

Introduction

Prompt Engineering

Offline Evals

Online Evals

Tracing

Simulations

Library

Dashboards

Integrations

Settings

CI/CD

Overview

Basic Key Setup

Single Key Configuration

Key Distribution Strategies

Load Balancing Strategy

Tiered Access Strategy

Priority-Based Strategy

Advanced Key Patterns

Multi-Provider Key Management

Dynamic Key Selection

Key Selection Algorithm

Best Practices

Security Best Practices

Performance Optimization

Next Steps

​Overview

​Basic Key Setup

​Single Key Configuration

​Key Distribution Strategies

​Load Balancing Strategy

​Tiered Access Strategy

​Priority-Based Strategy

​Advanced Key Patterns

​Multi-Provider Key Management

​Dynamic Key Selection

​Key Selection Algorithm

​Best Practices

​Security Best Practices

​Performance Optimization

​Next Steps

Overview

Basic Key Setup

Single Key Configuration

Key Distribution Strategies

Load Balancing Strategy

Tiered Access Strategy

Priority-Based Strategy

Advanced Key Patterns

Multi-Provider Key Management

Dynamic Key Selection

Key Selection Algorithm

Best Practices

Security Best Practices

Performance Optimization

Next Steps