Lazy evaluation and search pipelines for Elixir values and LLM programs.

Start with new/2, add declarations, and end with preview/2, evaluate/2, tune/1, or compare/3. Builder calls only return a new %Tune{} value.

Summary

Functions

Compares the original target and a search result or saved artifact on an explicit split.

Adds named train, validation, or test data without consuming it.

Builds a versioned artifact from a completed tuning result.

Evaluates the pipeline target on test data, or on an explicit data role.

Adds explicit work limits.

Validates an artifact and applies it to fresh application code.

Creates a lazy Tune pipeline.

Prepares one inspectable prompt without making a model call.

Builds an evaluation study from saved rows with input and output fields.

Adds one uniquely named score.

Selects a search strategy without starting it.

Selects the score policy used to compare candidates.

Runs the selected search and returns one immutable result.

Declares named values that a search strategy can change.

Types

t()

@type t() :: %Tune{
  cancel: (-> boolean()) | nil,
  data: map(),
  limits: map(),
  metadata: map(),
  models: map(),
  on_progress: (map() -> term()) | nil,
  runner: function() | :replay | nil,
  scores: [map()],
  selector: term(),
  strategy: term(),
  target: term(),
  variations: map()
}

Functions

compare(tune, candidate, opts)

@spec compare(t(), Tune.Result.t() | Tune.Artifact.t(), keyword()) ::
  {:ok, Tune.Comparison.t()} | {:error, Exception.t()}

Compares the original target and a search result or saved artifact on an explicit split.

For a saved artifact, Tune loads it into the original target before running the comparison.

data(tune, data)

@spec data(t(), keyword() | map()) :: t()

Adds named train, validation, or test data without consuming it.

dump(result)

@spec dump(Tune.Result.t()) :: {:ok, Tune.Artifact.t()} | {:error, Exception.t()}

Builds a versioned artifact from a completed tuning result.

evaluate(tune, role \\ :test)

@spec evaluate(t(), :train | :validation | :test) ::
  {:ok, Tune.Evaluation.t()} | {:error, Exception.t()}

Evaluates the pipeline target on test data, or on an explicit data role.

limit(tune, limits)

@spec limit(t(), keyword() | map()) :: t()

Adds explicit work limits.

load(artifact, fresh_target)

@spec load(Tune.Artifact.t(), term()) :: {:ok, term()} | {:error, Exception.t()}

Validates an artifact and applies it to fresh application code.

new(target, opts \\ [])

@spec new(term(), keyword()) :: t()

Creates a lazy Tune pipeline.

preview(tune, input)

@spec preview(t(), map() | {String.t(), map()}) ::
  {:ok, Tune.Prepared.t()} | {:error, Exception.t()}

Prepares one inspectable prompt without making a model call.

replay(rows, opts \\ [])

@spec replay(Enumerable.t() | (-> Enumerable.t()), keyword()) :: t()

Builds an evaluation study from saved rows with input and output fields.

Rows may also provide id and expected. Pass an Enumerable or a zero-arity factory. Score saved output with a function that takes (case, output).

score(tune, name, metric, opts \\ [])

@spec score(t(), atom(), function() | module() | struct(), keyword()) :: t()

Adds one uniquely named score.

A one-argument function receives the output. A two-argument function or module receives (case, output). Use argument: :prediction to give a two-argument scorer the prediction instead. The prediction is nil if the runner did not return one.

search(tune, strategy, opts \\ [])

@spec search(t(), atom() | module(), keyword()) :: t()

Selects a search strategy without starting it.

select(tune, name)

@spec select(t(), atom() | keyword()) :: t()

Selects the score policy used to compare candidates.

tune(tune)

@spec tune(t()) :: {:ok, Tune.Result.t()} | {:error, Exception.t()}

Runs the selected search and returns one immutable result.

vary(tune, variations)

@spec vary(t(), keyword() | map()) :: t()

Declares named values that a search strategy can change.