method: in config.yaml.
Quick Start
Start with the smallest possible DPO run, then layer on ORPO or KTO.1
Simplest DPO run
DPO trains on
chosen / rejected pairs with a frozen base as the reference model.2
ORPO — no reference model, saves VRAM
ORPO folds the reference into its loss, so there’s no second model to hold in VRAM.
3
KTO — thumbs up/down labels
KTO learns from
prompt / completion / label rows — a single up/down signal per example.Methods
Four methods, selected withmethod: — read from TRAINING_METHODS in trainer.py.
method is case-insensitive and defaults to sft. An unsupported value fails fast with the list of valid methods and their summaries.Which method?
Pick a method from the shape of your dataset and your VRAM budget.Config keys
Preference-specific keys, read fromKNOWN_KEYS in trainer.py — all optional, all backward-compatible.
Dataset-shape validation
PraisonAI validates dataset columns before the run starts, naming exactly which columns are missing — so a shape mismatch fails in seconds rather than minutes-in from a TRL collator traceback. Preference datasets are not flattened to atext column (that’s SFT-only behaviour) — the prompt / chosen / rejected (or prompt / completion / label) columns survive to the trainer.
How it works
Setmethod and a repo, run one command — the trainer resolves the method, validates columns, builds the right TRL trainer, and pushes privately by default.
For DPO and KTO the trainer passes ref_model=None, which reuses the frozen base weights through the PEFT adapter — no second full model in VRAM.
TRL version note — the preference trainers are imported individually and lazily, so an older TRL that lacks e.g.
KTOTrainer still supports the others. If a method is unavailable, the error names both the missing class and tells you to Upgrade trl, or use method: sft.Best Practices
Start with orpo when VRAM is tight
Start with orpo when VRAM is tight
ORPO folds the reference model into its loss, so there’s no second model to hold — the cheapest preference method to run.
Match max_prompt_length to your dataset
Match max_prompt_length to your dataset
The default is
max_seq_length // 2. If your prompts are long, raise max_prompt_length so completions aren’t truncated (and vice-versa).Use dpo's conservative default beta first
Use dpo's conservative default beta first
Leave
beta unset for the first run to inherit TRL’s default, then tune it once you have a baseline.Verify columns before a long run
Verify columns before a long run
Check
datasets.load_dataset(...).column_names locally so a shape mismatch fails in seconds, not minutes.Related
Train
Full fine-tuning flow and config.yaml reference.
Hub Privacy & Upload Options
Every Hub push is private by default — opt in to publish.
Dataset Tooling
Generate and quality-check instruction datasets.