Dataset Tooling

Generate and quality-check instruction datasets with praisonai-train

Generate an instruction dataset from a teacher LLM, merge parallel batches, then quality-check it before fine-tuning — three commands that compose into a clean training corpus.

generate calls an OpenAI-compatible endpoint to synthesise {instruction, input, output} rows from a recipe; dedup merges parallel batches across files with one shared index; validate filters them with research-backed QC checks. All three are YAML-driven, like praisonai-train llm.

Quick Start

1. Generate a dataset

   One command turns a recipe into a JSONL dataset. Set your teacher endpoint via env vars first.

   pip install praisonai-train
   
   export OPENAI_API_KEY="sk-..."          # or AZURE_OPENAI_KEY + AZURE_OPENAI_ENDPOINT
   export OPENAI_BASE_URL="https://api.openai.com/v1"
   
   praisonai-train generate -r tamil -d gpt-4o -n 1000 -o data/tamil.jsonl

2. Validate and filter

   Quality-check the raw dataset and write the kept rows to a clean file.

   praisonai-train validate data/tamil.jsonl --out data/clean.jsonl

3. Fine-tune on the clean dataset

   Feed the clean JSONL straight into the trainer.

   pip install "praisonai-train[llm]"
   
   praisonai-train llm data/clean.jsonl --model unsloth/gemma-2-2b-it-bnb-4bit

   See praisonai-train Package for the full llm reference.

How It Works

generate fans a recipe’s diversity axes into teacher prompts, streams unique rows to JSONL, and validate scores each row against pluggable QC checks.

generate

Synthesise {instruction, input, output} rows from a recipe, streamed to a JSONL file with dedup, snapshots, and a stop-file circuit-breaker.

CLI flags

YAML-only keys

These have no CLI flag — set them in the config file.

Behaviour worth knowing

• output + num_examples are validated up-front — missing either exits 1 with error: 'output' and 'num_examples' (or --num) are required.
• Atomic write — generate writes to a sibling temp file (via tempfile.mkstemp next to the destination) and os.replaces the destination only when at least one row is produced. A run that fails on credentials, recipe, or every teacher request (no JSON-mode support, all timeouts, etc.) leaves any pre-existing file byte-for-byte untouched — the temp file is unlinked and the CLI still exits 1 with error: no rows generated. A self-referential dedup_from is also read before it would be emptied. Fixed in PraisonAI PR #4239.
• File permissions preserved on overwrite — when the destination already exists, the atomic swap keeps its existing mode. For a fresh file, mode falls back to 0o666 & ~umask (what a plain open(path, "w") would have produced), so downstream validation / training jobs can still read a shared corpus.
• Symlink destinations are followed — --output pointing at a symlink updates the file the link points at, rather than replacing the link itself with a regular file. Consumers reading through the link continue to see the fresh corpus.
• Dedup is on-write and cross-run — the normaliser hashes instruction; rows whose normalised instruction is already seen (from dedup_from or earlier in the run) are silently skipped.
• Zero unique rows exits 1 with error: no rows generated — check endpoint/api_key/deployment and provider JSON-mode support.
• Progress bar: when tqdm is importable the CLI shows a live bar (generating: 42/1000 [00:12<04:38] req kept=39); otherwise it prints ...500/1000 requests (487 kept) every 500 requests. tqdm stays optional — the import is guarded.
• Per-row provenance — each emitted row carries model (the teacher deployment), plus task_type and topic when the recipe stamps them (built-in tamil and any _AxisRecipe subclass do). Downstream QC and category splits key off this metadata instead of brittle text detection. Backward compatible — keys are added only when values are available (see Per-row provenance).

Example config

Mirror setup/generate.yaml.

recipe: tamil
deployment: gpt-4o
azure: true
# endpoint / api_key come from AZURE_OPENAI_ENDPOINT / AZURE_OPENAI_KEY (or OPENAI_*)
num_examples: 5000
concurrency: 50
start_offset: 0
output: data/tamil.jsonl
snapshot_every: 1000
dedup_from:
  - data/existing.jsonl
# stop_file: ~/.praisonai_train_stop   # touch to halt immediately

praisonai-train generate --config generate.yaml

Python API — generate_dataset()

Import and drive the generator yourself.

from praisonai_train.data import generate_dataset

config = {
    "recipe": "tamil",
    "deployment": "gpt-4o",
    "num_examples": 1000,
    # endpoint / api_key read from env if omitted
}

rows = list(generate_dataset(config))

Full signature:

def generate_dataset(
    config: dict,
    on_row: Callable[[dict], None] | None = None,
    progress_callback: Callable[[int, int, int], None] | None = None,
    should_continue: Callable[[], bool] | None = None,
) -> Iterator[dict]

Returns an Iterator[dict] of {"instruction", "input", "output"} — plus model, task_type, and topic when available (see Per-row provenance). The caller drives the run by iterating.

Continuation guard

Halt a run from inside the loop when a billing or quota check flips — a belt-and-suspenders companion to the external stop-file circuit-breaker.

should_continue is a fn() -> bool re-checked every sponsor_check_rows completed requests. When it returns False, the run halts via the same clean shutdown as the stop-file: touch the stop-file → cancel pending futures → break.

The guard is resolved in a fixed order.

Key behaviours:

• Fail-OPEN — azure_sponsorship_guard returns True on any transient error (az CLI missing, network blip, unparseable output). It never halts a healthy run on a flaky check. Pair it with an external watchdog for hard budget enforcement.
• Trigger row is emitted, not dropped — the just-completed row is yielded before the guard runs, so the paid, already-finished request that trips the check is never silently lost.
• sponsor_check_rows coercion — 0 (divide-by-zero) and negative (never-poll) values are corrected to 1; None keeps the 5000 default.

Pass any billing or quota check as a custom callback.

from praisonai_train.data import generate_dataset

def under_budget() -> bool:
    return my_billing_client.remaining_credit_usd() > 10

rows = list(generate_dataset(
    {"recipe": "tamil", "deployment": "gpt-4o", "num_examples": 100_000,
     "sponsor_check_rows": 1000},
    should_continue=under_budget,
))

Auto-wire the built-in Azure sponsorship guard with just a subscription id — no callback needed.

recipe: tamil
deployment: gpt-4o
num_examples: 100000
subscription_id: 00000000-0000-0000-0000-000000000000   # or set AZ_SUBSCRIPTION
sponsor_check_rows: 5000
output: data/tamil.jsonl

praisonai-train generate --config generate.yaml
# The run halts automatically once the Azure subscription's quotaId flips
# away from "Sponsored_2016-01-01" (i.e. sponsorship converts to pay-as-you-go).

Or build the guard directly and pass it as should_continue.

from praisonai_train.data import azure_sponsorship_guard, generate_dataset

guard = azure_sponsorship_guard("sub-123")   # default quota_id="Sponsored_2016-01-01"
rows = list(generate_dataset(config, should_continue=guard))

azure_sponsorship_guard(subscription_id, quota_id="Sponsored_2016-01-01") returns a fn() -> bool that is True while the subscription still reads as sponsored, and fail-OPEN on any error.

Per-row provenance

Emitted rows are self-describing: the generator stamps model (from deployment) and passes task_type / topic through from the recipe when present.

Any axis recipe (tamil, CustomRecipe, or any _AxisRecipe subclass) stamps task_type and topic automatically, so QC and category splits key off metadata instead of brittle text detection.

// Recipe that doesn't stamp (still valid):
{"instruction": "...", "input": "", "output": "..."}

// Built-in tamil / any _AxisRecipe subclass:
{
  "instruction": "...",
  "input": "",
  "output": "...",
  "model": "gpt-4o",
  "task_type": "படைப்பாக்க எழுத்து",
  "topic": "தமிழ் இலக்கியம்"
}

Rows describe themselves — no re-detection needed.

from praisonai_train.data import generate_dataset

rows = list(generate_dataset({
    "recipe": "tamil",
    "deployment": "gpt-4o",
    "num_examples": 10,
}))

for r in rows:
    print(r["model"], r["task_type"], r["topic"])
    # gpt-4o  ஒரு விளக்கம்  அறிவியல்
    # gpt-4o  படைப்பாக்க எழுத்து  தமிழ் இலக்கியம்

Recipes

tamil is the built-in recipe — a Tamil-language instruction recipe crossing five diversity axes (task × topic × style × audience × variant), registered automatically at import.

Define one inline (YAML or dict) without writing Python — pass recipe: as a dict with system, template, and axes (the first two axis keys are the task × topic grid). The template receives {task}, {topic}, {style}, {audience}, {variant} at format time.

recipe:
  name: spanish
  system: "You are a Spanish-language instruction dataset assistant. Write natural Spanish."
  template: |
    Generate a unique Spanish training example.
    Task: {task}
    Topic: {topic}
    Style: {style}
    Audience: {audience}
    Variant: {variant}
    Return only JSON: {{"instruction": "...", "input": "", "output": "..."}}
  axes:
    task: ["question-answer", "explanation", "summary", "translation"]
    topic: ["history", "science", "food", "travel"]
    style: ["simple", "detailed"]
    audience: ["students", "professionals"]
deployment: gpt-4o
num_examples: 1000
output: data/spanish.jsonl

Register a custom Python recipe to reuse it by name from the CLI.

from praisonai_train.data.registry import recipes
from praisonai_train.data.recipes import _AxisRecipe

@recipes.register
class Spanish(_AxisRecipe):
    name = "spanish"
    system = "You are a Spanish-language instruction dataset assistant."
    template = (
        "Generate a unique Spanish training example. Task: {task}\n"
        "Topic: {topic}\nStyle: {style}\nAudience: {audience}\nVariant: {variant}\n"
        'Return only JSON: {{"instruction": "...", "input": "", "output": "..."}}'
    )
    axes = {
        "task": ["question-answer", "explanation", "summary"],
        "topic": ["history", "science", "food"],
        "style": ["simple", "detailed"],
        "audience": ["students", "professionals"],
    }

Once imported anywhere, --recipe spanish (or recipe: spanish in YAML) works from the CLI.

Any _AxisRecipe recipe (including tamil and inline CustomRecipe) automatically stamps task_type (its first axis) and topic (its second axis) onto every emitted row — see Per-row provenance. That metadata is what makes the task-aware QC checks and the metrics.translation by-direction breakdown reliable: they resolve direction from authoritative labels rather than brittle text detection. For the Tamil recipe, the EN_TA_TASK axis value becomes the stamped task_type on translation rows.

validate

Quality-check an instruction dataset — dedup, boilerplate/refusal, script purity, truncation, restates-question — plus dataset-level diversity metrics. Thresholds mirror Self-Instruct / LIMA / AlpaGasus / Deita defaults.

CLI flags

YAML-only keys

Checks

Drop checks remove a row; flag checks keep it but count it.

Report

validate prints a text report to stdout.

─── QC: data/tamil.jsonl ───
  in=500  kept=406 (81.2%)
  drops: {'exact_dup': 12, 'near_dup': 38, 'too_short': 21, 'boilerplate_refusal': 3, 'low_script_purity': 20}
  flags: {'restates_question': 47}
  metrics: {'distinct_2': 0.87, 'length_cv': 0.62, 'top_prefix_share': 0.008, 'translation': {'share': 0.12, 'by_direction': {'en_ta': 43, 'ta_en': 18, 'unknown': 0}, 'exempted': 15}}
  ✓ no diversity warnings
  wrote 406 filtered rows -> data/clean.jsonl

metrics.translation.share is the fraction of kept rows that are translation tasks; by_direction counts them by direction (en_ta, ta_en, unknown); exempted is the number of English-target rows the task-aware exemption actually rescued from the low_script_purity drop. Because both the exemption and the metric use the same detector (translation_intent), the numbers always reconcile.

Malformed JSONL lines are skipped with a ⚠ skipping malformed JSON on line N warning — the run still succeeds on the remaining rows.

Behaviour worth knowing

• Atomic write — validate writes to a sibling temp file next to the destination and os.replaces it only when at least one row survived QC. An --out that aliases the input is never truncated when everything is dropped; the existing corpus stays byte-for-byte intact. File mode and symlink destinations are handled the same as generate/dedup.
• Zero-kept exits 1 — when QC drops every row, validate exits 1 with error: no rows kept — all rows were dropped by QC ({drops}); any existing --out was left intact. The drops dict names the actual reasons — typically low_script_purity for wrong-language corpora. Previously the command silently exited 0 and could truncate an aliased --out to zero bytes. Fixed in PraisonAI PR #4316.

Example config

Mirror setup/validate.yaml.

input: data/tamil.jsonl
output: data/tamil.clean.jsonl
min_output_chars: 20
near_dup: true
near_dup_jaccard: 0.7
script_range: [2944, 3071]      # Tamil block U+0B80-U+0BFF; change per language
script_drop: 0.50
script_flag: 0.70

praisonai-train validate --config validate.yaml

Task-aware script purity

A naive script-purity filter deletes legitimate translation outputs written in the target script — a Tamil→English row’s correct output is English, so a Tamil-purity check would wrongly drop it.

Only a resolved English target exempts a row from the Tamil-purity drop. en→ta, ambiguous, and native rows keep the strict thresholds.

The detector resolves intent in a fixed order: ground-truth task_type metadata is authoritative → instruction-text keyword fallback for external rows → an ambiguous direction yields no exemption (the safe default). That task_type metadata is now stamped automatically by any axis recipe (not just the Tamil recipe’s EN_TA_TASK task), so freshly-generated corpora resolve direction from authoritative labels out of the box.

Translation composition metric

metrics.translation reports {share, by_direction, exempted} — the corpus-level view of how much of the kept dataset is translation.

The metric uses the same translation_intent() detector as the per-row exemption, so the numbers can never disagree with the checks they describe.

from praisonai_train.data import score

rows = [{"instruction": "...", "input": "", "output": "..."}]

result = score(rows, cfg={"near_dup": False})
share = result["metrics"]["translation"]["share"]   # 0.0–1.0

Python API — score() / filter_rows()

Run the same QC from Python.

from praisonai_train.data import score, filter_rows

rows = [{"instruction": "...", "input": "", "output": "..."}]

result = score(rows, cfg={"near_dup": True, "near_dup_jaccard": 0.7})
# result = {"in", "kept", "kept_n", "drops", "flags", "metrics", "warnings"}

clean_rows = filter_rows(rows, cfg={"min_output_chars": 20})   # just the kept list

Diversity warnings fire when distinct_2 < 0.5, length_cv < 0.4, or top_prefix_share > 0.01 — the “healthy dataset” bar.

translation_intent() is the shared detector behind both task-aware purity and the translation metric — call it directly to inspect a single row’s intent.

from praisonai_train.data import translation_intent

intent = translation_intent(
    "Translate the following into English: வணக்கம்",
    task_type=None,  # or the recipe's stamped label
)
# intent == {"is_translation": True, "direction": "ta_en", "expected_script": "english"}

Custom row checks

Register a RowCheck to add a QC rule — no core change.

from praisonai_train.data.registry import checks

@checks.register
class NoUrls:
    name = "no_urls"
    kind = "drop"   # or "flag"

    def triggered(self, instruction, input, output, cfg):
        return "http://" in output or "https://" in output

Once imported, it participates in the next score() / validate run automatically.

from-trials

Export a scored trials report into a trainer-ready SFT dataset — keep the attempts that passed verification, drop the rest, and write a ShareGPT (or Alpaca) JSONL the praisonai-train llm trainer consumes unchanged.

from-trials closes the loop from verification to data generation: verify the agent, keep what passed (rejection sampling), fine-tune on it, then re-verify to measure the gain — one supported pipeline instead of a hand-rolled script. It is invoked under the data group as praisonai-train data from-trials REPORT [OPTIONS].

Quick Start

1. Export the passing attempts

   Take a saved trials report and write the passing, tool-free attempts as a training dataset.

   praisonai-train data from-trials trials.json -o data/train.jsonl

2. Fine-tune on the exported dataset

   Feed the JSONL straight into the trainer — no reshaping needed.

   praisonai-train llm data/train.jsonl --model unsloth/gemma-2-2b-it-bnb-4bit

3. Re-verify to measure the gain

   Re-run the same trials on the fine-tuned model and compare pass rates.

   praisonai-train agents --scenarios trials_cases.json

CLI flags

Behaviour worth knowing

• Selection discipline — every skip is counted, never silent. Four filters run in order, each with its own counter:
• score is None → never a candidate (skipped_unscored; kept under --all, which sets only_passed=False).
• only_passed=True (default) → drop verifier-failed attempts (skipped_failed).
• frontier_only=True (default) → drop cases where pass_rate == 0 or pass_rate == 1 (skipped_saturated) — saturated cases add K near-duplicates of already-mastered behaviour, zero-pass cases have nothing to export.
• Tool-using runs excluded by default (skipped_tool_runs) — exporting only the final text of a run that used tools would train the model to answer without the tools it actually needed.
• Deterministic emission order — case, then attempt, so output files are byte-reproducible.
• Exit code 1 on zero rows written — with a stderr hint pointing at --all / --include-saturated, or to check that the report has passing tool-free attempts.
• Alpaca vs ShareGPT — the (often sole) user prompt is never dropped. With a system turn, instruction = system and input = "\n".join(all user turns); without a system turn, instruction = first user turn and input = "\n".join(remaining user turns). Conversations are truncated at the final assistant turn; assistant content is the raw generated text.
• Pass-threshold handling — resolved in a fixed order: an explicit passed boolean wins; otherwise the attempt’s score is compared against its own threshold (pass_threshold / threshold / min_score), so 0.3 under a pass_threshold: 0.5 counts as a failure; only when no threshold is carried does it fall back to score > 0.

Console output

from-trials prints a summary block naming every counter, then the write and provenance paths.

─── from-trials: trials.json ───
  written=42 skipped_failed=7 skipped_tool_runs=3 skipped_no_text=0 skipped_unscored=2 skipped_saturated=15
  wrote 42 rows -> data/train.jsonl
  provenance -> data/train.jsonl.meta.json

Output format

praisonai-train llm trains both chat column names — conversations and messages — without any rename.

ShareGPT (--format messages, default) — one JSON object per line with a conversations column; the trainer’s process_dataset standardises it directly.

{"conversations":[{"role":"system","content":"..."},{"role":"user","content":"..."},{"role":"assistant","content":"..."}]}

Alpaca (--format alpaca) — flat instruction/input/output rows.

{"instruction":"...","input":"...","output":"..."}

Python API — export_trials()

Import the exporter and its summary dataclass from the friendly top-level namespace.

from praisonai_train.data import export_trials, ExportSummary

summary = export_trials(report, "data/train.jsonl")
print(summary)   # written=42 skipped_failed=7 ...

Full signature:

def export_trials(
    report,                       # dict, object, or path to serialised report JSON
    path,                         # destination .jsonl
    *,
    only_passed: bool = True,
    frontier_only: bool = True,
    format: str = "messages",     # "messages" | "alpaca"
    qc: bool = False,
    qc_cfg: dict | None = None,
) -> ExportSummary

Returns an ExportSummary dataclass — six counter fields plus a to_dict() method and a CLI-friendly __str__ that emits written=X skipped_failed=Y ....

Provenance sidecar

Every run writes {path}.jsonl.meta.json alongside the dataset — every emitted row is traceable to its source case, attempt, and score.

{
  "package": "demo",
  "source_report": "trials.json",
  "options": {"only_passed": true, "frontier_only": true, "format": "messages", "qc": false},
  "summary": {"written": 42, "skipped_failed": 7, "skipped_tool_runs": 3, "...": "..."},
  "lines": [
    {"case_id": "c1", "attempt_index": 0, "score": 0.92},
    {"case_id": "c1", "attempt_index": 2, "score": 0.87}
  ]
}

Export trials with QC (from-trials --qc)

Turn passing agent trials into a training corpus and quality-check them in one pass — --qc reuses the validate filter but auto-disables the target-script check, because agent trajectories are English by construction.

--qc runs dedup, boilerplate/refusal, length, and diversity — not the Tamil script-purity check. The export QC path sets script_drop/script_flag to 0 so nothing falls below the target-script floor, which is why English trajectories survive instead of being dropped wholesale as low_script_purity.

Opt back into a script check for another language by passing qc_cfg to export_trials() with script_range (and/or script_drop / script_flag). Any one of those keys re-enables the target-script check with the QC filter’s own defaults (0.50 / 0.70) — so a documented qc_cfg={"script_range": [...]} no longer silently passes wrong-script rows.

from praisonaiagents import Agent
from praisonaiagents.eval import EvalPackage, EvalCase, run_trials
from praisonai_train.data import export_trials

agent = Agent(name="summariser", instructions="Summarise the input in one sentence.")
package = EvalPackage(name="suite", cases=[
    EvalCase(name="short", input="PraisonAI is an agent framework.",
             verify=lambda output, expected: "agent" in output.lower()),
])

report = run_trials(agent, package, k=8)

# English agent trajectories — QC keeps them (script check auto-disabled).
summary = export_trials(report, "train.jsonl", qc=True)
print(summary)
# written=42 skipped_failed=6 ... skipped_qc=3 qc_drops={'exact_dup': 3}

# Spanish trajectories: opt back into the Latin script check.
summary = export_trials(
    report, "train.jsonl", qc=True,
    qc_cfg={"script_range": (0x0041, 0x024F)},   # Latin block
)

The CLI mirrors this — pass --qc, and when QC drops everything the error names the real reasons instead of guessing.

praisonai-train from-trials trials.json --qc -o data/train.jsonl
# error: no rows exported — QC dropped all rows: {'low_script_purity': 8}

ExportSummary QC fields

export_trials() returns an ExportSummary that now tracks QC removals.

str(summary) appends skipped_qc=N, and qc_drops={...} when non-empty.

Cross-file dedup

Merge many JSONL batch files into one deduped file with a single shared MinHash+LSH index — the cross-file duplicates that per-file validate (whose dedup state resets each call) always misses.

You generated N batches in parallel with disjoint --start-offset values. cat-ing them and re-running validate still leaves near-duplicates across files. dedup catches them because one index spans every file.

praisonai-train dedup batch_*.jsonl --out merged.jsonl

CLI flags

YAML-only keys

Behaviour worth knowing

• Atomic write — the command writes to a sibling temp file and does os.replace() on success, so an --out that aliases an input is never truncated before its rows are read, and a malformed line never leaves a partial file.
• File mode and symlinks handled the same as generate — the atomic swap preserves the destination’s existing mode (or 0o666 & ~umask for a fresh file), and a symlink destination is followed so the swap lands on the target, not the link itself. See PraisonAI PR #4239.
• Exact-dup key = sha1 of the normalised instruction + input + output, removed first; near-dup runs on the survivors.
• Missing path raises FileNotFoundError: dedup source not found: PATH — a clear error, not a cryptic AttributeError.
• Report prints at the end:

─── cross-file dedup: 3 file(s) ───
  in=15000  kept=12873  removed=2127 (14.2% dup)
  wrote 12873 unique rows -> merged.jsonl

Which engine to pick

dedup defaults to minhash; validate/score default to sliding. Pick by scale and intent.

Example config

inputs:
  - data/batch_a.jsonl
  - data/batch_b.jsonl
  - data/batch_c.jsonl
output: data/merged.jsonl
near_dup_method: minhash   # default for dedup; set to 'sliding' for classic window semantics
near_dup_jaccard: 0.7
near_dup: true             # set false for exact-only
minhash_perm: 128
ngram_n: 4
minhash_seed: 1

praisonai-train dedup --config dedup.yaml

Python API — global_dedup()

Deduplicate rows across many JSONL paths or in-memory row lists with one shared exact + LSH index — streaming unique rows in source order.

from praisonai_train.data import global_dedup

merged = list(global_dedup(
    ["data/batch_a.jsonl", "data/batch_b.jsonl", "data/batch_c.jsonl"],
    cfg={"near_dup_jaccard": 0.7},
))
print(f"{len(merged)} unique rows")

Full signature:

def global_dedup(
    sources: Iterable,       # JSONL paths (str) or in-memory row lists — mix freely
    cfg: dict | None = None,
) -> Iterator[dict]

Missing paths raise FileNotFoundError: dedup source not found: PATH.

Python API — near_dedup() and MinHashLSH

Drop near-duplicate rows by instruction over either engine — this is what validate‘s near-dup pass now delegates to.

from praisonai_train.data import near_dedup

rows = [{"instruction": "...", "input": "", "output": "..."}]

kept, dropped = near_dedup(rows, cfg={"near_dup_method": "minhash"})

Full signature:

def near_dedup(
    rows: Iterable[dict],
    cfg: dict | None = None,
    index: "MinHashLSH | None" = None,
) -> tuple[list[dict], int]

Reuse the same index to dedup across a stream of batches — the whole corpus becomes the look-back.

from praisonai_train.data import near_dedup, MinHashLSH

shared = MinHashLSH(threshold=0.7, num_perm=128, seed=1)
cfg = {"near_dup_method": "minhash"}

for batch in stream_of_batches():
    kept, dropped = near_dedup(batch, cfg, index=shared)
    save(kept)
    print(f"batch: kept={len(kept)} dropped={dropped}  index size={len(shared)}")

MinHashLSH(threshold=0.7, num_perm=128, n=4, seed=1) is the pure-stdlib index behind both (no datasketch / numpy).

Common Patterns

Disjoint parallel workers

Split a large run across two shells with disjoint --start-offset values, then dedup on both output files.

# Shell A
praisonai-train generate -r tamil -d gpt-4o -n 5000 --start-offset 0 -o data/a.jsonl

# Shell B (offset past A's slice so prompts never overlap)
praisonai-train generate -r tamil -d gpt-4o -n 5000 --start-offset 5000 -o data/b.jsonl

# each worker's config can dedup against the other's output
dedup_from:
  - data/a.jsonl
  - data/b.jsonl

Merge parallel batches

Two shells generate disjoint slices, then one dedup command merges and cross-dedups them before validate.

# Two shells generate disjoint slices…
praisonai-train generate -r tamil -d gpt-4o -n 5000 --start-offset 0 -o data/a.jsonl
praisonai-train generate -r tamil -d gpt-4o -n 5000 --start-offset 5000 -o data/b.jsonl
# …then one command merges + dedups them.
praisonai-train dedup data/a.jsonl data/b.jsonl --out data/merged.jsonl
praisonai-train validate data/merged.jsonl --out data/clean.jsonl

Live progress in a notebook

Pass a progress_callback to drive a UI in Jupyter or Streamlit.

from praisonai_train.data import generate_dataset

def on_progress(done, total, kept):
    print(f"\r{done}/{total} requests, {kept} kept", end="")

rows = list(generate_dataset(
    {"recipe": "tamil", "deployment": "gpt-4o", "num_examples": 1000},
    progress_callback=on_progress,
))

Circuit-breaker

Halt a rogue run without losing kept rows — queued requests cancel, in-flight ones drain. Two routes trip the same clean shutdown:

• External / manual — touch the stop-file from another shell.
• In-loop / automatic — a should_continue guard (or an auto-wired Azure subscription id) halts the run when a billing or quota check flips, without you touching anything. See Continuation guard.

touch ~/.praisonai_train_stop

from praisonai_train.data import generate_dataset

rows = list(generate_dataset(
    {"recipe": "tamil", "deployment": "gpt-4o", "num_examples": 100_000,
     "subscription_id": "00000000-0000-0000-0000-000000000000"},
))
# Halts automatically once the Azure sponsorship converts to pay-as-you-go.

Custom recipe for another language

Point recipe: at an inline dict — no Python needed. See the Spanish example under Recipes.

Audit a translation corpus

Read metrics.translation before fine-tuning to verify how much of the corpus is translation and in which direction.

from praisonai_train.data import score

result = score(rows, cfg={"near_dup": False})
t = result["metrics"]["translation"]

print(f"translation share: {t['share']:.0%}")
print(f"by direction: {t['by_direction']}")   # {'en_ta': ..., 'ta_en': ..., 'unknown': ...}
print(f"english outputs rescued: {t['exempted']}")

Preference dataset shapes

Preference tuning reads columns by name — give each method the exact shape it needs. See Preference Tuning.

Best Practices

Related