Skip to main content
Teach a model which answer is better, not just what to say — switch from supervised fine-tuning to preference tuning by setting one method key. praisonai-train llm supports four objectives selected by the method config key: sft (default), dpo, orpo, and kto. SFT clones behaviour from single completions; DPO and ORPO learn from chosen vs rejected pairs; KTO learns from per-example thumbs-up / thumbs-down labels.

​
Quick Start

1

Install the training stack

2

Pick a method in config.yaml

Switch from SFT to preference tuning by setting method and pointing dataset at a preference dataset. Everything else stays the same.
Needs prompt, chosen, rejected columns. Uses the frozen base model as an implicit reference (ref_model=None).
3

Train

The trainer validates the dataset columns before the run starts and saves the aligned LoRA to lora_model/ — publishing stays opt-in.

​
Which method should I use?

​
Dataset shapes

Each method fails fast before training if the required columns are missing — the error names both the missing columns and the columns your dataset actually has.
Columns: prompt, chosen, rejected.
Preference datasets are kept as-is — the trainer does not flatten them to a text column the way SFT does. shuffle: true on a dataset entry still applies.

​
Config keys

All preference keys are optional and, apart from method, apply only to preference methods.

​
What happens under the hood

Response-only masking is an SFT concept — preference losses compare whole sequences, so assistant_only_loss is skipped for dpo / orpo / kto.
process_dataset() short-circuits for preference methods: your prompt / chosen / rejected (or prompt / completion / label) columns are passed straight through, un-flattened. SFT keeps its flatten-to-text behaviour.
For preference methods the trainer drops dataset_text_field, packing, and dataset_num_proc, sets max_length = max_seq_length, and defaults max_prompt_length to max_seq_length // 2.
DPO and KTO run with ref_model=None — the trainer uses the frozen base weights as the reference. This works cleanly with a PEFT adapter and avoids doubling VRAM. ORPO needs no reference model at all.
Each trainer / config pair is imported individually. An older TRL missing one method (e.g. KTOTrainer) still lets the others run.

​
Errors you might see

Rename your columns or point at a dataset with the required shape. The check runs before the model loads — you find out in seconds, not minutes.
Upgrade with pip install -U "praisonai-train[llm]".
An unrecognised method raises a ValueError listing every supported method (sft, dpo, orpo, kto) with a one-line summary. method is case-insensitive, so DPO and dpo both work.

Train

Full fine-tuning setup, config reference, and the response-masking section.

Assistant-only Loss

How SFT masks prompts out of the loss (skipped for preference methods).

praisonai-train Package

The standalone training package and its CLI subcommands.

Train CLI

Full flag and config-key reference for praisonai train llm.