While Maxim offers several pre-built evaluators in the Store, you might need custom evaluators for specific use cases. Create programmatic evaluators using Javascript or Python with access to standard libraries.

1

Navigate to Create Menu

Select Programmatic from the create menu to start building

Create programmatic evaluator

2

Select Language and Response Type

Choose your programming language and set the Response type (Number or Boolean) from the top bar

Evaluation configuration options

3

Implement the Validate Function

Define a function named validate in your chosen language. This function is required as Maxim uses it during execution.

Code restrictions

Javascript

  • No infinite loops
  • No debugger statements
  • No global objects (window, document, global, process)
  • No require statements
  • No with statements
  • No Function constructor
  • No eval
  • No setTimeout or setInterval

Python

  • No infinite loops
  • No recursive functions
  • No global/nonlocal statements
  • No raise, try, or assert statements
  • No disallowed variable assignments

Code editor for evaluation logic

4

Debug with Console

Monitor your evaluator execution with the built-in console. Add console logs for debugging to track what’s happening during evaluation. All logs will appear in this view.

Console showing debug logs during evaluator execution

5

Configure Pass Criteria

Configure two types of pass criteria:

Pass query Define criteria for individual evaluation metrics

Example: Pass if evaluation score > 0.8

Pass evaluator (%) Set threshold for overall evaluation across multiple entries

Example: Pass if 80% of entries meet the evaluation criteria

Pass criteria configuration

6

Test in Playground

Test your evaluator in the playground before using it in your workflows. The right panel shows input fields for all variables used in your evaluator.

  1. Fill in sample values for each variable
  2. Click Run to see how your evaluator performs
  3. Iterate and improve your evaluator based on the results

Testing an evaluator in the playground with input fields for variables