Skip to main content
Continued pretraining teaches a base model a new domain, language, or vocabulary from a plain-text corpus by training the embeddings at their own, much lower learning rate.
CPT is a config.yaml method, not a code import. Set method: cpt, point at a dataset with a text column, and make the embeddings trainable with modules_to_save: [embed_tokens, lm_head].

​
Quick Start

1

Minimal CPT run

Set method: cpt, give it a raw-text dataset, and make the vocabulary trainable.
2

With a custom embedding learning rate

The embedding LR defaults to learning_rate / 10. Set it explicitly to override.

​
How It Works

CPT drives the trainer from config.yaml — the same praisonai-train llm entry point as SFT, switched to UnslothTrainer by method: cpt. CPT is a separate method from SFT because it trains a different part of the model.

​
When to use CPT

Pick the method that matches the change you want to make.
CPT teaches the corpus; SFT teaches the model to follow instructions on it. For a task-ready model, run CPT first, then SFT.

​
Dataset shape

CPT needs a dataset with a text column — each row is one plain-text example.
ShareGPT (conversations) and Alpaca (instruction / input / output) still take priority when present, so a stray text column on those shapes is ignored. Non-string rows become "" and are then dropped. If every row formats to empty text, the trainer names all three shapes so you can fix the columns:

​
Configuration

Two keys turn a run into continued pretraining; a third makes the vocabulary trainable. The copy-paste block for any CPT run:

​
Common Patterns

Domain adaptation — teach a base model a specialised corpus (medical, legal) on its own tokenizer.
Language adaptation — teach a base model a new language (e.g. Tamil) with a tokenizer whose vocab covers the target script.

​
Best Practices

CPT without it trains no embeddings — the run adapts no vocabulary and prints a NOTE. Include both so continued pretraining actually moves the vocab.
The default (learning_rate / 10) is the safe recipe. Training embeddings at the full adapter LR is the known-bad case — that is exactly why UnslothTrainer ships an embedding_learning_rate field.
Older unsloth versions leave UnslothTrainer / UnslothTrainingArguments as None. CPT then fails fast and tells you which package to upgrade — run pip install -U unsloth.
CPT teaches the model the corpus; SFT teaches it to follow instructions on it. Run CPT first, then an SFT pass on instruction data for a task-ready model.

Train

Full fine-tuning config reference and methods.

CLI: train

Run the trainer from the command line.

Multi-GPU Training

Scale a CPT run across every GPU with torchrun.