​
Why run comparison experiments

  • Make decisions between Prompt versions and models by comparing output differences.
  • Analyze scores across all test cases in your Dataset for the evaluation metrics that you choose.
  • Side by side comparison views for easy decision making and detailed view for every entry.

​
Run a comparison report

1

Open the Prompt playground

Open the Prompt playground for one of the Prompts you want to compare.

2

Start test configuration

Click the test button to start configuring your experiment.

3

Select Prompt versions

Select the Prompt versions you want to compare it to. These could be totally different Prompts or another version of the same Prompt.

4

Choose test Dataset

Select your Dataset to test it against.

5

Configure context evaluation

Optionally, select the context you want to evaluate if there is a difference in retrieval pipeline that needs comparison.

6

Add Evaluators

Select existing Evaluators or add new ones from the store, then run your test.

7

Review summary results

Once the run is completed, you will see summary details for each Evaluator. Below that, charts show the comparison data for latency, cost and tokens used.

8

Analyze detailed comparison

Each entry has 2 rows one below the other showing the outputs, latency and scores for the entities or versions compared. Deep dive into any entry by clicking the row and looking into the particular messages, evaluation details and logs.

If you want to compare Prompt versions over time (e.g. Last month’s scores and this month’s scores post a Prompt iteration), you can instead generate a comparison report retrospectively under the analyze section.

​
Next steps