Tune.Comparison (Tune v0.5.0)

Copy Markdown View Source

The paired evidence for an original target and a search winner.

Each row contains the observations for both targets and one change per metric. Changes use the metric's direction. The delta is always the winner's value minus the original value. Aggregate means and deltas use only pairs where both scores are available. Each metric includes its :paired_count. Compare it with the length of case_ids to check coverage. Each evaluation also keeps its own full summaries, model calls, and token usage.

case_ids preserves dataset order. rows is keyed by these IDs. A missing or stopped observation yields an unpaired metric. A failed metric yields an error, while other metrics on the same row can still be compared. complete? means both evaluations finished attempting their selected cases. Individual rows and scores can still have errors.

Summary

Types

classification()

@type classification() :: :improved | :regressed | :tied | :error | :unpaired

t()

@type t() :: %Tune.Comparison{
  baseline: Tune.Evaluation.t(),
  case_ids: [String.t()],
  challenger: Tune.Evaluation.t(),
  complete?: boolean(),
  metrics: %{optional(atom()) => map()},
  rows: %{optional(String.t()) => map()},
  split: :train | :validation | :test
}