Set Up Alerts and Notifications
Learn how to configure notification channels (Slack and PagerDuty) and set up alerts to monitor your AI application’s performance and quality metrics.
Maxim provides a comprehensive alerting system that combines powerful notification channels with flexible alert configurations. Set up Slack or PagerDuty integrations to receive real-time notifications when your AI application’s performance metrics or quality scores exceed specified thresholds.
Notification Channels
Maxim supports multiple notification channels to ensure you stay informed about your alerts. Currently, we support Slack and PagerDuty integrations.
You must set up at least one notification channel before creating alerts. Follow the instructions for your preferred notification channel.
Managing Integrations
You can manage your notification channel integrations in the following ways:
- Edit an integration: Hover over the integration on the Integrations page and click the edit icon
- Delete an integration: Hover over the integration and click the delete icon
- View integration details: Click on the integration to see its configuration and usage
When you delete an integration, any alerts using that integration will no longer send notifications. Make sure to update or remove those alerts first.
Performance Metrics Alerts
Monitor your application’s performance by setting up alerts for latency, token usage, and cost metrics. Create custom thresholds and receive notifications when metrics exceed your specified limits.
Before creating alerts, make sure you have set up at least one notification channel in the Notification Channels section above.
Available Performance Metrics
Set up alerts for:
- Latency: Response times for API calls
- Token Usage: Token consumption per request
- Cost: API usage expenses
Creating Performance Alerts
Navigate to repository
Open the Logs page and select your repository
Access alerts tab
Select the Alerts tab and click Create alert
Configure alert settings
- Select Log metrics as the type of alert
- Select a metric (Latency, Token Usage, or Cost)
- Choose an operator (greater than, less than)
- Enter the threshold value
- Set minimum occurrence count
- Define evaluation time range
Select notification channels
Select your preferred notification channels from the ones you’ve set up in the Notification Channels section.
Save your alert
Click Create alert
Performance Alert Examples
Monitor Response Time
Triggers when response time exceeds 1 second in 5 requests within 5 minutes.
Setting Up Token Consumption Alerts
Triggers when hourly token usage exceeds 1 million.
Monitor Daily Costs
Triggers when daily costs exceed $100.
Quality Metrics Alerts
Monitor your AI application’s quality with alerts for evaluation scores and quality checks. Receive notifications when AI responses don’t meet expected quality standards.
Before creating alerts, make sure you have set up at least one notification channel in the Notification Channels section above.
Available Quality Metrics
Set up alerts for various evaluation scores, such as:
- Bias-check: Monitor potential biases in AI responses
- Toxicity: Check for inappropriate or harmful content
- Clarity: Validate clear and understandable output
- Factual accuracy: Verify generated information accuracy
- Custom evaluators: Monitor your defined evaluation metrics
Creating Quality Alerts
Navigate to repository
Open the Logs page and select the repository you want to monitor.
Access alerts tab
Select the Alerts tab and click Create alert
Select alert type
Select Evaluation scores as the type of alert.
Configure alert settings
- Choose an evaluation metric (e.g., “Bias-check”)
- The violation criteria is based on your evaluator’s type and configuration.
- Specify how many times this should occur
- Set the evaluation time range
Choose notification channels
Select your preferred notification channels from the ones you’ve set up in the Notification Channels section.
Save your alert
Click Create alert to save your configuration.
Quality Alert Examples
Bias Check Alert
Triggers when a response fails the bias check within 15 minutes.
Toxicity Alert
Triggers when a response reaches 0.8 or higher toxicity score.
Managing Alerts
Manage your alerts in the following ways:
- Edit an alert: Click the options icon (three dots) on the alert card and select “Edit alert”
- Delete an alert: Click the options icon and select “Delete alert”
- Pause/Resume an alert: Click the options icon and select “Pause alert” or “Resume alert”
Best practices
- Monitor critical evaluators with immediate alerts for bias and toxicity
- Adjust score thresholds to match application requirements
- Combine evaluation metrics for comprehensive monitoring
- Review and adjust alert criteria based on performance
- Document quality issues and solutions from alerts