Skip to main content
Eval Datasets loads a task set from JSONL, then turns the runs that scored well into a fine-tuning file — closing the loop from task set to trained model. JSONL sits at both ends — the format Agno, OpenAI’s fine-tuning API, and every dataset tool already speak — so a task set is portable in and the training file is portable out.

​
Quick Start

1

Load a task set

Read a JSONL file into EvalCase objects.
2

Run the suite on the loaded cases

Run the cases and get an EvalReport.
3

Export what passed

Write the passing runs as chat-format training data.

​
Input Format

Each line is one JSON object describing a task. The minimal shape carries a name, an input, and an expected answer.
Aliases are accepted and unrecognised columns fold into metadata.
Blank lines and // comments are skipped.

​
Output Format

Export writes one chat-format JSON object per line — the shape OpenAI’s fine-tuning API expects.
Pass system_prompt to prepend a system message to every row.
The prompt is read from result.record["input"]; the completion is result.actual_output.

​
How It Works

load_cases maps aliases, folds extra columns into metadata, and auto-names rows; export_sft keeps only passing runs at or above min_score. Aliases map friendly column names onto the case fields.

​
Behaviour Worth Knowing

The loader is forgiving where it costs nothing and strict where it matters.
  • Unrecognised columns fold into metadata instead of being dropped, so source / difficulty / id survive into the report.
  • Missing name auto-populates as case_1, case_2, … by position among data rows.
  • Blank lines and // comment lines are skipped; the physical file line is preserved, so file:900 really is line 900 in the editor.
  • A malformed line is reported by file:line, not a generic “invalid JSON”.
  • A non-object line (e.g. [1, 2, 3]) is refused with “must be a JSON object”.
  • An empty file raises DatasetError — an empty task set would otherwise report a perfect pass rate over nothing.
  • Only passed=True runs with score >= min_score are exported.
  • A run missing either a prompt or a completion is skipped, never silently shrinking the set.
  • A zero-record export raises DatasetError rather than writing an empty file that looks like success.
  • DatasetError subclasses ValueError, so existing except ValueError handlers still catch it.

​
Python API

Import the five public helpers from the top-level eval package.

​
load_cases

load_cases(path, *, case_cls=None) -> List[EvalCase] reads a JSONL task set into EvalCase objects. Raises: DatasetError on missing file, malformed line, non-object line, or empty file — with file:line when the fault is inside the file.

​
iter_jsonl

iter_jsonl(path) -> Iterator[Dict[str, Any]] yields one dict per data line, with the same error handling but no aliasing or metadata folding.

​
export_sft

export_sft(report, path, *, min_score=1.0, system_prompt=None) -> int writes passing runs to path and returns the count written. Raises: DatasetError if no run scored high enough, or if report is not an EvalReport / list.

​
sft_records

sft_records(report, *, min_score=1.0, system_prompt=None) -> List[Dict[str, Any]] applies the same selection logic but returns the records in memory instead of writing them.

​
DatasetError

DatasetError(ValueError) is raised for every dataset read or write failure. As a ValueError subclass, existing except ValueError handlers still catch it.

​
Common Patterns

​
Portable in, portable out

Load a Hugging Face-exported JSONL with prompt / answer columns and export a file OpenAI’s fine-tuning API accepts unchanged.

​
Lower the bar for a small dataset

Keep more attempts when the suite is small by lowering min_score — a deliberate trade against SFT quality.

​
Persist a system prompt with each training row

Pass system_prompt= so the fine-tuned model learns the same persona the eval was scored under.

​
Best Practices

The loader keeps source / difficulty / id automatically — leave them in your JSONL so the report carries where each case came from.
A run that barely passed is a fragile teacher. Export above your gate so training data reflects confident wins, not marginal ones.
Regenerate train.jsonl from the eval report instead of editing it, so provenance stays honest and reproducible.
Comment lines are skipped at load time and make the file self-documenting — describe each section inline without breaking the loader.
min_score-filtered export is rejection sampling — it amplifies behaviour the agent already produces and inherits any bias in the judge that decided a run “passed”. It cannot teach behaviour the agent never exhibited; treat it as reinforcing verified wins, not as an oracle.

Evaluation Suite

Run every evaluator you enable as one CI gate

Train Package

Fine-tune on the JSONL that export_sft produces

Dataset Tooling

Prepare and inspect training datasets