Continued Pretraining (CPT)

Adapt a base model to a new domain, language, or vocabulary with a raw-text corpus

Continued pretraining teaches a base model a new domain, language, or vocabulary from a plain-text corpus by training the embeddings at their own, much lower learning rate.

Quick Start

1. Minimal CPT run

   Set method: cpt, give it a raw-text dataset, and make the vocabulary trainable.

   model_name: unsloth/gemma-2-2b-it-bnb-4bit
   max_seq_length: 2048
   method: cpt
   modules_to_save: [embed_tokens, lm_head]
   dataset:
     - name: your-domain-corpus.jsonl   # rows with a "text" column

   praisonai-train llm your-domain-corpus.jsonl --model unsloth/gemma-2-2b-it-bnb-4bit

2. With a custom embedding learning rate

   The embedding LR defaults to learning_rate / 10. Set it explicitly to override.

   model_name: unsloth/gemma-2-2b-it-bnb-4bit
   max_seq_length: 2048
   method: cpt
   modules_to_save: [embed_tokens, lm_head]
   learning_rate: 2e-4
   embedding_learning_rate: 2e-5      # defaults to learning_rate / 10
   dataset:
     - name: your-domain-corpus.jsonl

How It Works

CPT drives the trainer from config.yaml — the same praisonai-train llm entry point as SFT, switched to UnslothTrainer by method: cpt.

CPT is a separate method from SFT because it trains a different part of the model.

When to use CPT

Pick the method that matches the change you want to make.

Dataset shape

CPT needs a dataset with a text column — each row is one plain-text example.

{"text": "Continued pretraining adapts a model to a new domain."}
{"text": "Each row is one plain-text example the model reads as-is."}

ShareGPT (conversations) and Alpaca (instruction / input / output) still take priority when present, so a stray text column on those shapes is ignored. Non-string rows become "" and are then dropped.

If every row formats to empty text, the trainer names all three shapes so you can fix the columns:

All examples formatted to empty text. The dataset columns are [...]; this trainer understands ShareGPT ('conversations'), Alpaca ('instruction'/'input'/'output') and raw text ('text'). Rename your columns to one of those shapes, or set chat_template to match your data.

Configuration

Two keys turn a run into continued pretraining; a third makes the vocabulary trainable.

The copy-paste block for any CPT run:

method: cpt
modules_to_save: [embed_tokens, lm_head]
embedding_learning_rate: 2e-5      # defaults to learning_rate / 10

Common Patterns

Domain adaptation — teach a base model a specialised corpus (medical, legal) on its own tokenizer.

model_name: unsloth/gemma-2-2b-it-bnb-4bit
max_seq_length: 2048
method: cpt
modules_to_save: [embed_tokens, lm_head]
dataset:
  - name: medical-corpus.jsonl   # rows with a "text" column

Language adaptation — teach a base model a new language (e.g. Tamil) with a tokenizer whose vocab covers the target script.

model_name: unsloth/gemma-2-2b-it-bnb-4bit
max_seq_length: 2048
method: cpt
chat_template: gemma
modules_to_save: [embed_tokens, lm_head]
dataset:
  - name: tamil-corpus.jsonl     # rows with a "text" column

Best Practices

Related