The paired evidence for an original target and a search winner.
Each row contains the observations for both targets and one change per
metric. Changes use the metric's direction. The delta is always the winner's
value minus the original value. Aggregate means and deltas use only pairs
where both scores are available. Each metric includes its :paired_count.
Compare it with the length of case_ids to check coverage. Each evaluation
also keeps its own full summaries, model calls, and token usage.
case_ids preserves dataset order. rows is keyed by these IDs. A missing
or stopped observation yields an unpaired metric. A failed metric yields an
error, while other metrics on the same row can still be compared.
complete? means both evaluations finished attempting their selected cases.
Individual rows and scores can still have errors.
Summary
Types
@type classification() :: :improved | :regressed | :tied | :error | :unpaired
@type t() :: %Tune.Comparison{ baseline: Tune.Evaluation.t(), case_ids: [String.t()], challenger: Tune.Evaluation.t(), complete?: boolean(), metrics: %{optional(atom()) => map()}, rows: %{optional(String.t()) => map()}, split: :train | :validation | :test }