> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Model access, PEFT selectors & offline merge
> Gated models, custom-code models, MoE LoRA, offline merged export, and vLLM rollouts for GRPO
Reach gated and custom-code models, LoRA only the layers you want, merge offline into a folder, and turn on fast vLLM rollouts for GRPO — all from `config.yaml`.
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
subgraph "Unsloth Surface"
C[📋 config.yaml] --> A[🔑 Model access]
C --> P[🧩 PEFT selectors]
C --> M[💾 Offline merge]
C --> V[💨 vLLM rollouts]
C --> Q[📦 Quant]
A --> T[✅ Trained model]
P --> T
M --> T
V --> T
Q --> T
end
classDef input fill:#6366F1,stroke:#7C90A0,color:#fff
classDef process fill:#189AB4,stroke:#7C90A0,color:#fff
classDef output fill:#10B981,stroke:#7C90A0,color:#fff
class C input
class A,P,M,V,Q process
class T output
```
These keys were always accepted by Unsloth but never forwarded — set any of them today and the run does exactly what you asked. All are additive; existing configs keep working.
## Quick Start
Train a gated Llama base with a token, pin the revision, and merge offline — no Hub account needed.
Pick the model you want to fine-tune, then reach it from a config.
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent
agent = Agent(instructions="Answer briefly and correctly.")
agent.start("What is the capital of France?")
```
```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# config.yaml
model_name: "meta-llama/Meta-Llama-3.1-8B-Instruct"
max_seq_length: 2048
hf_token: "hf_xxx" # or leave unset and use HF_TOKEN
revision: "main" # pin so a multi-day run is reproducible
trust_remote_code: false
dataset:
- name: "yahma/alpaca-cleaned"
```
```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
merged_save_dir: "./merged-model"
save_method: "merged_16bit" # default
```
```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
pip install "praisonai-train[llm]"
praisonai-train llm config.yaml
```
***
## Model access
`hf_token`, `trust_remote_code`, and `revision` decide **which** weights load and how they are reached.
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
subgraph "Model access"
Cfg[📋 config] --> Tok{gated repo?}
Tok -->|yes| HT[🔑 hf_token → token]
Tok -->|no| Skip[skip]
Cfg --> TR[🧠 trust_remote_code]
Cfg --> Rev[📌 revision pin]
HT --> Load[✅ FastLanguageModel]
TR --> Load
Rev --> Load
end
classDef input fill:#6366F1,stroke:#7C90A0,color:#fff
classDef decision fill:#F59E0B,stroke:#7C90A0,color:#fff
classDef process fill:#189AB4,stroke:#7C90A0,color:#fff
classDef output fill:#10B981,stroke:#7C90A0,color:#fff
class Cfg input
class Tok decision
class HT,TR,Rev,Skip process
class Load output
```
```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# config.yaml
model_name: "meta-llama/Meta-Llama-3.1-8B-Instruct"
max_seq_length: 2048
hf_token: "hf_xxx" # config-level equivalent of HF_TOKEN
dataset:
- name: "yahma/alpaca-cleaned"
```
```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
model_name: "some-org/custom-arch-model"
trust_remote_code: true # required to run the repo's model code
```
```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
model_name: "meta-llama/Meta-Llama-3.1-8B-Instruct"
revision: "main" # a branch, tag, or commit SHA
```
| Key | Type | Default | Description |
| ------------------- | ------ | ------- | ----------------------------------------------------------------------------------------------------------------------- |
| `hf_token` | `str` | — | Forwarded to Unsloth as `token`. Renamed so it can't collide with the trainer's own token handling. Only sent when set. |
| `trust_remote_code` | `bool` | — | Required for custom-code models. Only sent when set. |
| `revision` | `str` | — | Pin a HF branch, tag, or SHA for a reproducible multi-day run. Only sent when set. |
Config `hf_token` is the config-level twin of the `HF_TOKEN` env var. Precedence: `HF_TOKEN` env var wins, then a cached `huggingface-cli login`, then config `hf_token`. See [Hub Privacy & Upload Options](/docs/features/praisonai-train-hub-privacy).
***
## PEFT selectors
Choose exactly which layers and parameters LoRA touches — from the last N transformer blocks to a MoE model's experts.
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph TB
Start["📋 How much can I spend?"] --> Q{Budget?}
Q -->|"full quality"| Full["⚙️ full LoRA
all target modules"]
Q -->|"cheap run"| Last["⚙️ finetune_last_n_layers
last N blocks only"]
Q -->|"MoE model"| Moe["⚙️ target_parameters
expert adapters"]
classDef input fill:#8B0000,stroke:#7C90A0,color:#fff
classDef decision fill:#F59E0B,stroke:#7C90A0,color:#fff
classDef method fill:#6366F1,stroke:#7C90A0,color:#fff
class Start input
class Q decision
class Full,Last,Moe method
```
Tune only the last few transformer blocks to cut memory and time.
```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
model_name: "unsloth/Qwen2.5-32B-Instruct-bnb-4bit"
max_seq_length: 2048
finetune_last_n_layers: 8
dataset:
- name: "yahma/alpaca-cleaned"
```
`target_parameters` is what lets LoRA reach a Mixture-of-Experts model's experts.
```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
model_name: "unsloth/mistral-8x7b-instruct-v0.3-bnb-4bit"
max_seq_length: 2048
target_parameters:
- "mlp.experts" # selector; confirm the exact name in Unsloth's docs
dataset:
- name: "yahma/alpaca-cleaned"
```
```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
layers_to_transform: [0, 1, 2, 3] # explicit layer indices
layers_pattern: "model.layers" # pattern for layer selection
init_lora_weights: "gaussian" # "gaussian", "pissa", "olora", ...
```
| Key | Type | Default | Description |
| ------------------------ | ------------------ | ------- | --------------------------------------------------------------- |
| `finetune_last_n_layers` | `int` | — | Tune only the last N transformer blocks — the cheap-run recipe. |
| `layers_to_transform` | `int \| list[int]` | — | Explicit layer indices to adapt. |
| `layers_pattern` | `str` | — | Regex / pattern selecting which layers to adapt. |
| `target_parameters` | `list[str]` | — | Parameter selectors — required to LoRA MoE experts. |
| `init_lora_weights` | `str \| bool` | — | LoRA init scheme, e.g. `"gaussian"`, `"pissa"`, `"olora"`. |
Each is forwarded to `FastLanguageModel.get_peft_model` only when set, so they compose with the existing `lora_r`, `lora_alpha`, `use_dora`, and `rank_pattern` keys.
***
## Offline merged export
`merged_save_dir` writes merged weights to a folder on this disk — no Hub account, no token, no network.
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
subgraph "Merge routes"
Ad[🧩 Trained adapter] --> R{Where?}
R -->|"merged_save_dir"| Loc[💾 save_pretrained_merged
local folder]
R -->|"hf_model_name"| Hub[🌐 push_to_hub_merged
Hub repo]
end
classDef input fill:#8B0000,stroke:#7C90A0,color:#fff
classDef decision fill:#F59E0B,stroke:#7C90A0,color:#fff
classDef output fill:#10B981,stroke:#7C90A0,color:#fff
class Ad input
class R decision
class Loc,Hub output
```
```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
model_name: "unsloth/gemma-2-2b-it-bnb-4bit"
max_seq_length: 2048
merged_save_dir: "./merged-model"
save_method: "merged_16bit" # default; also "merged_4bit_forced", "lora"
dataset:
- name: "yahma/alpaca-cleaned"
```
| Key | Type | Default | Description |
| ----------------- | ----- | ---------------- | ------------------------------------------------------------------------------------------- |
| `merged_save_dir` | `str` | — | Merge the adapter into this local directory via `save_pretrained_merged`. No-op when unset. |
| `save_method` | `str` | `"merged_16bit"` | Merge precision — `"merged_16bit"`, `"merged_4bit_forced"`, or `"lora"`. |
Every run calls the merge step; it does nothing unless `merged_save_dir` is set. Contrast this with the Hub-push flow in [Export a trained model](/docs/features/praisonai-train-export), which needs a repo id, a write token, and a network.
***
## vLLM fast rollouts (GRPO)
`fast_inference` swaps GRPO's slow HF-generate loop for vLLM rollouts — one line.
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
sequenceDiagram
participant TRL as GRPOTrainer
participant Unsloth
participant Gen as Rollout
TRL->>Unsloth: generate completions
Unsloth->>Gen: fast_inference?
Gen-->>Unsloth: vLLM (fast) or HF-generate (slow)
Unsloth-->>TRL: scored completions
TRL->>TRL: gradient update
```
```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
method: grpo
model_name: "unsloth/gemma-2-2b-it-bnb-4bit"
max_seq_length: 2048
fast_inference: true
gpu_memory_utilization: 0.6
max_lora_rank: 32
num_generations: 8
reward_funcs:
- myproject.rewards:length_penalty
dataset:
- name: "your-org/your-prompts"
```
| Key | Type | Default | Description |
| ------------------------ | ------- | ------- | ------------------------------------------------------------------------------------- |
| `fast_inference` | `bool` | `false` | Turn on vLLM rollouts for GRPO. GRPO previously always ran the slow HF-generate path. |
| `gpu_memory_utilization` | `float` | — | vLLM GPU memory fraction. Only forwarded when `fast_inference` is truthy. |
| `max_lora_rank` | `int` | — | Max LoRA rank vLLM allocates for. Only forwarded when `fast_inference` is truthy. |
`gpu_memory_utilization` and `max_lora_rank` are ignored unless `fast_inference` is on. See [Reward Functions & GRPO](/docs/features/train-reward-functions).
***
## Expanded quant list
Unsloth's exporter accepts 35 quantization methods — 24 standard plus 11 importance-matrix (IQ) quants. The IQ family is what lets a 30B model fit on a laptop.
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph TB
Start["📋 Which quant?"] --> Q{Priority?}
Q -->|"balanced"| Std["📦 q4_k_m
standard"]
Q -->|"best quality"| Big["📦 q8_0
large, accurate"]
Q -->|"fit on a laptop"| IQ["📦 iq4_xs / iq3_m
imatrix"]
classDef input fill:#8B0000,stroke:#7C90A0,color:#fff
classDef decision fill:#F59E0B,stroke:#7C90A0,color:#fff
classDef out fill:#10B981,stroke:#7C90A0,color:#fff
class Start input
class Q decision
class Std,Big,IQ out
```
### Standard (24)
```
bf16 f16 f32 fast_quantized not_quantized quantized
q2_k q2_k_l q3_k_l q3_k_m q3_k_s q3_k_xs
q4_0 q4_1 q4_k q4_k_m q4_k_s
q5_0 q5_1 q5_k q5_k_m q5_k_s
q6_k q8_0
```
### IMatrix — large-model / laptop-fit (11)
```
iq1_m iq1_s iq2_m iq2_s iq2_xs iq2_xxs
iq3_m iq3_s iq3_xxs iq4_nl iq4_xs
```
```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
model_name: "unsloth/Qwen2.5-32B-Instruct-bnb-4bit"
max_seq_length: 2048
quantization_method: iq4_xs # imatrix quant — auto-fetches the matrix
# imatrix_file: "./my.imatrix" # optional; supply your own
dataset:
- name: "yahma/alpaca-cleaned"
```
| Key | Type | Default | Description |
| --------------------- | ----- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `quantization_method` | `str` | `"q4_k_m"` | Any of the 35 valid quants above. Validated up front — a typo fails fast. |
| `imatrix_file` | `str` | — | Path to a precomputed importance matrix; forwarded to `push_to_hub_gguf`. Only sent when set — Unsloth otherwise auto-fetches one from `unsloth/-GGUF`. |
Full quant reference lives in [Export a trained model](/docs/features/praisonai-train-export#valid-quant-values).
***
## Best Practices
Config-level `hf_token` keeps the token out of environment inspection in CI. Outside CI, `export HF_TOKEN=hf_...` is simpler — the env var wins over config.
A branch moves. Set `revision` to a tag or SHA so a run you resume tomorrow loads the same weights.
Tuning the last 8 blocks of a 32B model is a fraction of the memory of full LoRA — start there, widen only if quality needs it.
`merged_save_dir` gives you the merged weights on disk with no Hub round-trip. Inspect them, then push separately only if you want to.
`iq4_xs` / `iq3_m` pack a 30B model small enough to run locally. Unsloth fetches the matrix automatically; supply `imatrix_file` only if you built your own.
***
## Related
Full fine-tuning flow and config.yaml reference.
Every `praisonai train` subcommand and config key.
Publish to HF, GGUF, or Ollama — full 35-quant reference.
Private-by-default pushes and the `hf_token` precedence ladder.
Score completions with Python and turn on vLLM fast rollouts.