Create Programmatic Evaluators
Build custom code-based evaluators using Javascript or Python
While Maxim offers several pre-built evaluators in the Store, you might need custom evaluators for specific use cases. Create programmatic evaluators using Javascript or Python with access to standard libraries.
Navigate to Create Menu
Select Programmatic from the create menu to start building
Select Language and Response Type
Choose your programming language and set the Response type (Number or Boolean) from the top bar
Implement the Validate Function
Define a function named validate
in your chosen language. This function is required as Maxim uses it during execution.
Code restrictions
Javascript
- No infinite loops
- No debugger statements
- No global objects (window, document, global, process)
- No require statements
- No with statements
- No Function constructor
- No eval
- No setTimeout or setInterval
Python
- No infinite loops
- No recursive functions
- No global/nonlocal statements
- No raise, try, or assert statements
- No disallowed variable assignments
Debug with Console
Monitor your evaluator execution with the built-in console. Add console logs for debugging to track what’s happening during evaluation. All logs will appear in this view.
Configure Pass Criteria
Configure two types of pass criteria:
Pass query Define criteria for individual evaluation metrics
Example: Pass if evaluation score > 0.8
Pass evaluator (%) Set threshold for overall evaluation across multiple entries
Example: Pass if 80% of entries meet the evaluation criteria
Test in Playground
Test your evaluator in the playground before using it in your workflows. The right panel shows input fields for all variables used in your evaluator.
- Fill in sample values for each variable
- Click Run to see how your evaluator performs
- Iterate and improve your evaluator based on the results