Maxim provides a comprehensive alerting system that combines powerful notification channels with flexible alert configurations. Set up Slack or PagerDuty integrations to receive real-time notifications when your AI application’s performance metrics or quality scores exceed specified thresholds.

​
Notification Channels

Maxim supports multiple notification channels to ensure you stay informed about your alerts. Currently, we support Slack and PagerDuty integrations.
You must set up at least one notification channel before creating alerts. Follow the instructions for your preferred notification channel.

​
Managing Integrations

You can manage your notification channel integrations in the following ways:
  • Edit an integration: Hover over the integration on the Integrations page and click the edit icon
  • Delete an integration: Hover over the integration and click the delete icon
  • View integration details: Click on the integration to see its configuration and usage
When you delete an integration, any alerts using that integration will no longer send notifications. Make sure to update or remove those alerts first.

​
Performance Metrics Alerts

Monitor your application’s performance by setting up alerts for latency, token usage, and cost metrics. Create custom thresholds and receive notifications when metrics exceed your specified limits.
Before creating alerts, make sure you have set up at least one notification channel in the Notification Channels section above.

​
Available Performance Metrics

Set up alerts for:
  • Latency: Response times for API calls
  • Token Usage: Token consumption per request
  • Cost: API usage expenses

​
Creating Performance Alerts

1

Navigate to repository

Open the Logs page and select your repository
2

Access alerts tab

Select the Alerts tab and click Create alert
3

Configure alert settings

  • Select Log metrics as the type of alert
  • Select a metric (Latency, Token Usage, or Cost)
  • Choose an operator (greater than, less than)
  • Enter the threshold value
  • Set minimum occurrence count
  • Define evaluation time range
4

Select notification channels

Select your preferred notification channels from the ones you’ve set up in the Notification Channels section.
5

Save your alert

Click Create alert

​
Performance Alert Examples

​
Monitor Response Time

Metric: Latency
Operator: greater than
Threshold: 1000 ms
Minimum occurrences: 5
Time range: 5 minutes
Triggers when response time exceeds 1 second in 5 requests within 5 minutes.

​
Setting Up Token Consumption Alerts

Metric: Token Usage
Operator: greater than
Threshold: 1000000
Minimum occurrences: 1
Time range: 1 hour
Triggers when hourly token usage exceeds 1 million.

​
Monitor Daily Costs

Metric: Cost
Operator: greater than
Threshold: 100 USD
Minimum occurrences: 1
Time range: 24 hours
Triggers when daily costs exceed $100.

​
Quality Metrics Alerts

Monitor your AI application’s quality with alerts for evaluation scores and quality checks. Receive notifications when AI responses don’t meet expected quality standards.
Before creating alerts, make sure you have set up at least one notification channel in the Notification Channels section above.

​
Available Quality Metrics

Set up alerts for various evaluation scores, such as:
  • Bias-check: Monitor potential biases in AI responses
  • Toxicity: Check for inappropriate or harmful content
  • Clarity: Validate clear and understandable output
  • Factual accuracy: Verify generated information accuracy
  • Custom evaluators: Monitor your defined evaluation metrics

​
Creating Quality Alerts

1

Navigate to repository

Open the Logs page and select the repository you want to monitor.
2

Access alerts tab

Select the Alerts tab and click Create alert
3

Select alert type

Select Evaluation scores as the type of alert.
4

Configure alert settings

  • Choose an evaluation metric (e.g., “Bias-check”)
  • The violation criteria is based on your evaluator’s type and configuration.
  • Specify how many times this should occur
  • Set the evaluation time range
5

Choose notification channels

Select your preferred notification channels from the ones you’ve set up in the Notification Channels section.
6

Save your alert

Click Create alert to save your configuration.

​
Quality Alert Examples

​
Bias Check Alert

Metric: Bias-check
Minimum occurrences: 1
Time range: 15 minutes
Triggers when a response fails the bias check within 15 minutes.

​
Toxicity Alert

Metric: Toxicity
Condition: violates pass criteria
Score: 0.8
Minimum occurrences: 1
Time range: 5 minutes
Triggers when a response reaches 0.8 or higher toxicity score.

​
Managing Alerts

Manage your alerts in the following ways:
  • Edit an alert: Click the options icon (three dots) on the alert card and select “Edit alert”
  • Delete an alert: Click the options icon and select “Delete alert”
  • Pause/Resume an alert: Click the options icon and select “Pause alert” or “Resume alert”

​
Best practices

  1. Monitor critical evaluators with immediate alerts for bias and toxicity
  2. Adjust score thresholds to match application requirements
  3. Combine evaluation metrics for comprehensive monitoring
  4. Review and adjust alert criteria based on performance
  5. Document quality issues and solutions from alerts