View module source on GitHub

​
YieldedOutputTokenUsage

@dataclass
class YieldedOutputTokenUsage()
This class represents the token usage of a yielded output. Users can pass custom token usage to the yieldsOutput function.

​
YieldedOutputCost

@dataclass
class YieldedOutputCost()
This class represents the cost of a yielded output. Users can pass custom cost to the yieldsOutput function.

​
YieldedOutputMeta

@dataclass
class YieldedOutputMeta()
This class represents the meta of a yielded output. Users can pass custom meta to the yieldsOutput function.

​
YieldedOutput

@dataclass
class YieldedOutput()
Yielded output represents the output of yieldsOutput function.

​
EvaluatorArgs

@dataclass
class EvaluatorArgs()
This class represents the arguments of an evaluator.

​
EvaluatorConfig

@dataclass
class EvaluatorConfig()
This class represents the config of an evaluator.

​
TestRun

@dataclass
class TestRun()
This class represents a test run.

​
TestRunEntry

@dataclass
class TestRunEntry()
This class represents an entry of a test run.

​
TestRunWithDatasetEntry

@dataclass
class TestRunWithDatasetEntry(TestRun)
This class represents a test run with a dataset entry.

​
TestRunStatus

@dataclass
class TestRunStatus()
This class represents the status of a test run.

​
EvaluatorMeanScore

@dataclass
class EvaluatorMeanScore()
This class represents the mean score of an evaluator. This helps users to specify the score of an custom evaluator.

​
TestRunTokenUsage

@dataclass
class TestRunTokenUsage()
This class represents the token usage of a test run.

​
TestRunCost

@dataclass
class TestRunCost()
This class represents the cost of a test run.

​
TestRunLatency

@dataclass
class TestRunLatency()
This class represents the latency of a test run.

​
TestRunResultObj

@dataclass
class TestRunResultObj()
Object representing a result of a test run.

​
TestRunResult

@dataclass
class TestRunResult()
This class represents the result of a test run.

​
RunResult

@dataclass
class RunResult()
This class represents the result of a comparison test run.

​
TestRunLogger

class TestRunLogger(ABC)

​
info

@abstractmethod
def info(message: str) -> None
Log an informational message. Arguments:
NameTypeDescription
messagestrThe message to be logged.

​
error

@abstractmethod
def error(message: str, e: Optional[Exception] = None) -> None
Log an error message. Arguments:
NameTypeDescription
messagestrThe error message to be logged.

​
TestRunConfig

@dataclass
class TestRunConfig(Generic[T])
Configuration for a test run. Attributes:
NameTypeDescription
base_urlstrThe base URL for the API.
api_keystrThe API key for authentication.
in_workspace_idstrThe ID of the workspace.
workflow_idOptional[str]The ID of the workflow.
prompt_version_idOptional[str]The ID of the prompt version.
prompt_chain_version_idOptional[str]The ID of the prompt chain version.
namestrThe name of the test run.
data_structureOptional[T]The structure of the test data.
dataOptional[Union[str, DataValue[T], Callable[[int], Optional[DataValue[T]]]]]The test data or a function to retrieve it.
test_config_idOptional[str]The ID of the test configuration.
platform_evaluatorsList[PlatformEvaluatorType[T]]List of platform evaluators to use.