> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Continued Pretraining (CPT)
> Adapt a base model to a new domain, language, or vocabulary with a raw-text corpus
Continued pretraining teaches a base model a new domain, language, or vocabulary from a plain-text corpus by training the embeddings at their own, much lower learning rate.
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
Corpus[📚 raw-text corpus
text column] --> Trainer[🧠UnslothTrainer
adapter LR + embedding LR]
Trainer --> Model[✅ adapted model]
classDef input fill:#8B0000,stroke:#7C90A0,color:#fff
classDef process fill:#189AB4,stroke:#7C90A0,color:#fff
classDef output fill:#10B981,stroke:#7C90A0,color:#fff
class Corpus input
class Trainer process
class Model output
```
CPT is a `config.yaml` method, not a code import. Set `method: cpt`, point at a dataset with a `text` column, and make the embeddings trainable with `modules_to_save: [embed_tokens, lm_head]`.
## Quick Start
Set `method: cpt`, give it a raw-text dataset, and make the vocabulary trainable.
```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
model_name: unsloth/gemma-2-2b-it-bnb-4bit
max_seq_length: 2048
method: cpt
modules_to_save: [embed_tokens, lm_head]
dataset:
- name: your-domain-corpus.jsonl # rows with a "text" column
```
```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
praisonai-train llm your-domain-corpus.jsonl --model unsloth/gemma-2-2b-it-bnb-4bit
```
The embedding LR defaults to `learning_rate / 10`. Set it explicitly to override.
```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
model_name: unsloth/gemma-2-2b-it-bnb-4bit
max_seq_length: 2048
method: cpt
modules_to_save: [embed_tokens, lm_head]
learning_rate: 2e-4
embedding_learning_rate: 2e-5 # defaults to learning_rate / 10
dataset:
- name: your-domain-corpus.jsonl
```
***
## How It Works
CPT drives the trainer from `config.yaml` — the same `praisonai-train llm` entry point as SFT, switched to `UnslothTrainer` by `method: cpt`.
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
sequenceDiagram
participant User
participant Config as config.yaml
participant CLI as praisonai-train llm
participant Trainer as UnslothTrainer
participant Out as adapted model
User->>Config: method: cpt + text corpus
Config->>CLI: launch
CLI->>Trainer: adapter LR + embedding LR
Trainer-->>Out: save to lora_model/
Out-->>User: domain/language-adapted model
```
CPT is a separate method from SFT because it trains a different part of the model.
| CPT needs | Why |
| ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
| A lower `embedding_learning_rate` | Embeddings train at a fraction of the adapter rate. That field lives on `UnslothTrainer` / `UnslothTrainingArguments`, not `SFTTrainer`. |
| `modules_to_save: [embed_tokens, lm_head]` | Without it, no embeddings are trainable and CPT adapts no vocabulary. |
| A `text` column | CPT trains on raw text directly, not on formatted chat turns. |
***
## When to use CPT
Pick the method that matches the change you want to make.
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph TB
Q{What are you teaching?}
Q -->|New language or script| CPT[method: cpt]
Q -->|New domain vocabulary| CPT
Q -->|Follow specific instructions| SFT[method: sft]
Q -->|Prefer one answer over another| PREF[method: dpo / orpo / kto]
classDef question fill:#F59E0B,stroke:#7C90A0,color:#fff
classDef cpt fill:#189AB4,stroke:#7C90A0,color:#fff
classDef other fill:#6366F1,stroke:#7C90A0,color:#fff
class Q question
class CPT cpt
class SFT,PREF other
```
CPT teaches the corpus; SFT teaches the model to follow instructions on it. For a task-ready model, run CPT first, then SFT.
***
## Dataset shape
CPT needs a dataset with a `text` column — each row is one plain-text example.
```json theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
{"text": "Continued pretraining adapts a model to a new domain."}
{"text": "Each row is one plain-text example the model reads as-is."}
```
ShareGPT (`conversations`) and Alpaca (`instruction` / `input` / `output`) still take priority when present, so a stray `text` column on those shapes is ignored. Non-string rows become `""` and are then dropped.
If every row formats to empty text, the trainer names all three shapes so you can fix the columns:
```text theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
All examples formatted to empty text. The dataset columns are [...]; this trainer understands ShareGPT ('conversations'), Alpaca ('instruction'/'input'/'output') and raw text ('text'). Rename your columns to one of those shapes, or set chat_template to match your data.
```
***
## Configuration
Two keys turn a run into continued pretraining; a third makes the vocabulary trainable.
| Key | Type | Default | Description |
| ------------------------- | ----------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `method` | `str` | `"sft"` | Set to `"cpt"` to enable continued pretraining. |
| `embedding_learning_rate` | `float` | `learning_rate / 10` | Learning rate for `embed_tokens` and `lm_head` when they are in `modules_to_save`. |
| `modules_to_save` | `list[str]` | *unset* | Include `embed_tokens` and `lm_head` to actually move the vocab. Without this, CPT trains no embeddings and prints a NOTE. |
The copy-paste block for any CPT run:
```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
method: cpt
modules_to_save: [embed_tokens, lm_head]
embedding_learning_rate: 2e-5 # defaults to learning_rate / 10
```
***
## Common Patterns
**Domain adaptation** — teach a base model a specialised corpus (medical, legal) on its own tokenizer.
```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
model_name: unsloth/gemma-2-2b-it-bnb-4bit
max_seq_length: 2048
method: cpt
modules_to_save: [embed_tokens, lm_head]
dataset:
- name: medical-corpus.jsonl # rows with a "text" column
```
**Language adaptation** — teach a base model a new language (e.g. Tamil) with a tokenizer whose vocab covers the target script.
```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
model_name: unsloth/gemma-2-2b-it-bnb-4bit
max_seq_length: 2048
method: cpt
chat_template: gemma
modules_to_save: [embed_tokens, lm_head]
dataset:
- name: tamil-corpus.jsonl # rows with a "text" column
```
***
## Best Practices
CPT without it trains no embeddings — the run adapts no vocabulary and prints a NOTE. Include both so continued pretraining actually moves the vocab.
The default (`learning_rate / 10`) is the safe recipe. Training embeddings at the full adapter LR is the known-bad case — that is exactly why `UnslothTrainer` ships an `embedding_learning_rate` field.
Older `unsloth` versions leave `UnslothTrainer` / `UnslothTrainingArguments` as `None`. CPT then fails fast and tells you which package to upgrade — run `pip install -U unsloth`.
CPT teaches the model the corpus; SFT teaches it to follow instructions on it. Run CPT first, then an SFT pass on instruction data for a task-ready model.
***
## Related
Full fine-tuning config reference and methods.
Run the trainer from the command line.
Scale a CPT run across every GPU with torchrun.