Eval Datasets

Load JSONL task sets and export passing runs as fine-tuning data

Eval Datasets loads a task set from JSONL, then turns the runs that scored well into a fine-tuning file — closing the loop from task set to trained model.

JSONL sits at both ends — the format Agno, OpenAI’s fine-tuning API, and every dataset tool already speak — so a task set is portable in and the training file is portable out.

Quick Start

1. Load a task set

   Read a JSONL file into EvalCase objects.

   from praisonaiagents.eval import load_cases
   
   cases = load_cases("tasks.jsonl")

2. Run the suite on the loaded cases

   Run the cases and get an EvalReport.

   report = my_runner.run(cases)

3. Export what passed

   Write the passing runs as chat-format training data.

   from praisonaiagents.eval import export_sft
   
   export_sft(report, "train.jsonl", min_score=0.8)

Input Format

Each line is one JSON object describing a task.

The minimal shape carries a name, an input, and an expected answer.

{"name": "add", "input": "2+2?", "expected": "4"}
{"name": "capital", "input": "capital of France?", "expected": "Paris"}

Aliases are accepted and unrecognised columns fold into metadata.

{"prompt": "2+2?", "answer": "4", "difficulty": "easy", "source": "sheet1"}

Blank lines and // comments are skipped.

// Arithmetic set
{"input": "2+2?", "expected": "4"}

{"input": "3+3?", "expected": "6"}

Output Format

Export writes one chat-format JSON object per line — the shape OpenAI’s fine-tuning API expects.

{"messages":[{"role":"user","content":"2+2?"},{"role":"assistant","content":"4"}]}

Pass system_prompt to prepend a system message to every row.

{"messages":[{"role":"system","content":"Be terse."},{"role":"user","content":"2+2?"},{"role":"assistant","content":"4"}]}

The prompt is read from result.record["input"]; the completion is result.actual_output.

Train on this JSONL directly

praisonai-train llm understands the messages shape natively — point a dataset entry at the exported file, no column rename needed.

# train.yaml
model_name: "unsloth/gemma-2-2b-it-bnb-4bit"
max_seq_length: 2048
dataset:
  - name: "train.jsonl"   # the file export_sft wrote

praisonai-train llm train.yaml

The trainer renames messages → conversations internally and formats each row with the model’s chat template. See Supported dataset shapes.

How It Works

load_cases maps aliases, folds extra columns into metadata, and auto-names rows; export_sft keeps only passing runs at or above min_score.

Aliases map friendly column names onto the case fields.

Behaviour Worth Knowing

The loader is forgiving where it costs nothing and strict where it matters.

• Unrecognised columns fold into metadata instead of being dropped, so source / difficulty / id survive into the report.
• Missing name auto-populates as case_1, case_2, … by position among data rows.
• Blank lines and // comment lines are skipped; the physical file line is preserved, so file:900 really is line 900 in the editor.
• A malformed line is reported by file:line, not a generic “invalid JSON”.
• A non-object line (e.g. [1, 2, 3]) is refused with “must be a JSON object”.
• An empty file raises DatasetError — an empty task set would otherwise report a perfect pass rate over nothing.
• Only passed=True runs with score >= min_score are exported.
• A run missing either a prompt or a completion is skipped, never silently shrinking the set.
• A zero-record export raises DatasetError rather than writing an empty file that looks like success.
• DatasetError subclasses ValueError, so existing except ValueError handlers still catch it.

Python API

Import the five public helpers from the top-level eval package.

from praisonaiagents.eval import (
    load_cases,
    iter_jsonl,
    export_sft,
    sft_records,
    DatasetError,
)

load_cases

load_cases(path, *, case_cls=None) -> List[EvalCase] reads a JSONL task set into EvalCase objects.

Raises: DatasetError on missing file, malformed line, non-object line, or empty file — with file:line when the fault is inside the file.

iter_jsonl

iter_jsonl(path) -> Iterator[Dict[str, Any]] yields one dict per data line, with the same error handling but no aliasing or metadata folding.

export_sft

export_sft(report, path, *, min_score=1.0, system_prompt=None) -> int writes passing runs to path and returns the count written.

Raises: DatasetError if no run scored high enough, or if report is not an EvalReport / list.

sft_records

sft_records(report, *, min_score=1.0, system_prompt=None) -> List[Dict[str, Any]] applies the same selection logic but returns the records in memory instead of writing them.

DatasetError

DatasetError(ValueError) is raised for every dataset read or write failure. As a ValueError subclass, existing except ValueError handlers still catch it.

Common Patterns

Portable in, portable out

Load a Hugging Face-exported JSONL with prompt / answer columns and export a file OpenAI’s fine-tuning API accepts unchanged.

from praisonaiagents.eval import load_cases, export_sft

cases = load_cases("hf_export.jsonl")
report = my_runner.run(cases)
export_sft(report, "train.jsonl", min_score=0.8)

Lower the bar for a small dataset

Keep more attempts when the suite is small by lowering min_score — a deliberate trade against SFT quality.

from praisonaiagents.eval import export_sft

export_sft(report, "train.jsonl", min_score=0.5)

Persist a system prompt with each training row

Pass system_prompt= so the fine-tuned model learns the same persona the eval was scored under.

from praisonaiagents.eval import export_sft

export_sft(report, "train.jsonl", system_prompt="Be terse.")

Best Practices

Related