TestRun

View module source on GitHub

​

YieldedOutputTokenUsage

@dataclass
class YieldedOutputTokenUsage()

This class represents the token usage of a yielded output. Users can pass custom token usage to the yieldsOutput function.

​

YieldedOutputCost

@dataclass
class YieldedOutputCost()

This class represents the cost of a yielded output. Users can pass custom cost to the yieldsOutput function.

​

YieldedOutputMeta

@dataclass
class YieldedOutputMeta()

This class represents the meta of a yielded output. Users can pass custom meta to the yieldsOutput function.

​

YieldedOutput

@dataclass
class YieldedOutput()

Yielded output represents the output of yieldsOutput function.

​

EvaluatorArgs

@dataclass
class EvaluatorArgs()

This class represents the arguments of an evaluator.

​

EvaluatorConfig

@dataclass
class EvaluatorConfig()

This class represents the config of an evaluator.

​

TestRun

@dataclass
class TestRun()

This class represents a test run.

​

TestRunEntry

@dataclass
class TestRunEntry()

This class represents an entry of a test run.

​

TestRunWithDatasetEntry

@dataclass
class TestRunWithDatasetEntry(TestRun)

This class represents a test run with a dataset entry.

​

TestRunStatus

@dataclass
class TestRunStatus()

This class represents the status of a test run.

​

EvaluatorMeanScore

@dataclass
class EvaluatorMeanScore()

This class represents the mean score of an evaluator. This helps users to specify the score of an custom evaluator.

​

TestRunTokenUsage

@dataclass
class TestRunTokenUsage()

This class represents the token usage of a test run.

​

TestRunCost

@dataclass
class TestRunCost()

This class represents the cost of a test run.

​

TestRunLatency

@dataclass
class TestRunLatency()

This class represents the latency of a test run.

​

TestRunResultObj

@dataclass
class TestRunResultObj()

Object representing a result of a test run.

​

TestRunResult

@dataclass
class TestRunResult()

This class represents the result of a test run.

​

RunResult

@dataclass
class RunResult()

This class represents the result of a comparison test run.

​

TestRunLogger

class TestRunLogger(ABC)

​

info

@abstractmethod
def info(message: str) -> None

Log an informational message.

Arguments:

​

error

@abstractmethod
def error(message: str, e: Optional[Exception] = None) -> None

Log an error message.

Arguments:

​

TestRunConfig

@dataclass
class TestRunConfig(Generic[T])

Configuration for a test run.

Attributes: