Tune.Evaluation (Tune v0.5.0)

Copy Markdown View Source

The eager result of evaluating one target over one dataset.

observations keeps every attempted, failed, interrupted, and skipped case. summaries keeps raw statistics for each named score. selected_score is the higher-is-better value that search strategies compare.

Summary

Functions

Starts an evaluation under the caller's Task.Supervisor.

Returns the summary for one named score.

Types

t()

@type t() :: %Tune.Evaluation{
  calls: %{optional(atom()) => non_neg_integer()},
  complete?: boolean(),
  eligible?: boolean(),
  metadata: map(),
  observations: [Tune.Observation.t()],
  selected_score: number() | nil,
  stop_reason:
    :completed
    | :case_limit
    | :model_call_limit
    | :deadline
    | :cancelled
    | :lost_accounting,
  summaries: %{optional(atom()) => Tune.Score.Summary.t()},
  usage: %{
    input: non_neg_integer(),
    output: non_neg_integer(),
    total: non_neg_integer()
  }
}

Functions

start(supervisor, pipeline, role \\ :test)

@spec start(GenServer.server(), Tune.t(), :train | :validation | :test) :: Task.t()

Starts an evaluation under the caller's Task.Supervisor.

summary(evaluation, name)

@spec summary(t(), atom()) :: Tune.Score.Summary.t() | nil

Returns the summary for one named score.