Model access, PEFT selectors & offline merge

Gated models, custom-code models, MoE LoRA, offline merged export, and vLLM rollouts for GRPO

Reach gated and custom-code models, LoRA only the layers you want, merge offline into a folder, and turn on fast vLLM rollouts for GRPO — all from config.yaml.

These keys were always accepted by Unsloth but never forwarded — set any of them today and the run does exactly what you asked. All are additive; existing configs keep working.

Quick Start

Train a gated Llama base with a token, pin the revision, and merge offline — no Hub account needed.

1. Start from an Agent

   Pick the model you want to fine-tune, then reach it from a config.

   from praisonaiagents import Agent
   
   agent = Agent(instructions="Answer briefly and correctly.")
   agent.start("What is the capital of France?")

2. Reach a gated base and pin a revision

   # config.yaml
   model_name: "meta-llama/Meta-Llama-3.1-8B-Instruct"
   max_seq_length: 2048
   hf_token: "hf_xxx"          # or leave unset and use HF_TOKEN
   revision: "main"            # pin so a multi-day run is reproducible
   trust_remote_code: false
   dataset:
     - name: "yahma/alpaca-cleaned"

3. Merge into a folder, no Hub

   merged_save_dir: "./merged-model"
   save_method: "merged_16bit"   # default

   pip install "praisonai-train[llm]"
   praisonai-train llm config.yaml

Model access

hf_token, trust_remote_code, and revision decide which weights load and how they are reached.

1. Gated model (Llama / Gemma)

   # config.yaml
   model_name: "meta-llama/Meta-Llama-3.1-8B-Instruct"
   max_seq_length: 2048
   hf_token: "hf_xxx"          # config-level equivalent of HF_TOKEN
   dataset:
     - name: "yahma/alpaca-cleaned"

2. Custom-code model

   model_name: "some-org/custom-arch-model"
   trust_remote_code: true      # required to run the repo's model code

3. Pin a revision for reproducibility

   model_name: "meta-llama/Meta-Llama-3.1-8B-Instruct"
   revision: "main"             # a branch, tag, or commit SHA

PEFT selectors

Choose exactly which layers and parameters LoRA touches — from the last N transformer blocks to a MoE model’s experts.

1. Cheap run — last N layers

   Tune only the last few transformer blocks to cut memory and time.

   model_name: "unsloth/Qwen2.5-32B-Instruct-bnb-4bit"
   max_seq_length: 2048
   finetune_last_n_layers: 8
   dataset:
     - name: "yahma/alpaca-cleaned"

2. MoE expert LoRA

   target_parameters is what lets LoRA reach a Mixture-of-Experts model’s experts.

   model_name: "unsloth/mistral-8x7b-instruct-v0.3-bnb-4bit"
   max_seq_length: 2048
   target_parameters:
     - "mlp.experts"           # selector; confirm the exact name in Unsloth's docs
   dataset:
     - name: "yahma/alpaca-cleaned"

3. Explicit layers and init scheme

   layers_to_transform: [0, 1, 2, 3]     # explicit layer indices
   layers_pattern: "model.layers"        # pattern for layer selection
   init_lora_weights: "gaussian"         # "gaussian", "pissa", "olora", ...

Each is forwarded to FastLanguageModel.get_peft_model only when set, so they compose with the existing lora_r, lora_alpha, use_dora, and rank_pattern keys.

Offline merged export

merged_save_dir writes merged weights to a folder on this disk — no Hub account, no token, no network.

1. Merge into a folder

   model_name: "unsloth/gemma-2-2b-it-bnb-4bit"
   max_seq_length: 2048
   merged_save_dir: "./merged-model"
   save_method: "merged_16bit"   # default; also "merged_4bit_forced", "lora"
   dataset:
     - name: "yahma/alpaca-cleaned"

Every run calls the merge step; it does nothing unless merged_save_dir is set. Contrast this with the Hub-push flow in Export a trained model, which needs a repo id, a write token, and a network.

vLLM fast rollouts (GRPO)

fast_inference swaps GRPO’s slow HF-generate loop for vLLM rollouts — one line.

1. Turn on fast rollouts

   method: grpo
   model_name: "unsloth/gemma-2-2b-it-bnb-4bit"
   max_seq_length: 2048
   fast_inference: true
   gpu_memory_utilization: 0.6
   max_lora_rank: 32
   num_generations: 8
   reward_funcs:
     - myproject.rewards:length_penalty
   dataset:
     - name: "your-org/your-prompts"

Expanded quant list

Unsloth’s exporter accepts 35 quantization methods — 24 standard plus 11 importance-matrix (IQ) quants. The IQ family is what lets a 30B model fit on a laptop.

Standard (24)

bf16  f16  f32  fast_quantized  not_quantized  quantized
q2_k  q2_k_l  q3_k_l  q3_k_m  q3_k_s  q3_k_xs
q4_0  q4_1  q4_k  q4_k_m  q4_k_s
q5_0  q5_1  q5_k  q5_k_m  q5_k_s
q6_k  q8_0

IMatrix — large-model / laptop-fit (11)

iq1_m  iq1_s  iq2_m  iq2_s  iq2_xs  iq2_xxs
iq3_m  iq3_s  iq3_xxs  iq4_nl  iq4_xs

1. Imatrix quant — 30B on a laptop

   model_name: "unsloth/Qwen2.5-32B-Instruct-bnb-4bit"
   max_seq_length: 2048
   quantization_method: iq4_xs      # imatrix quant — auto-fetches the matrix
   # imatrix_file: "./my.imatrix"   # optional; supply your own
   dataset:
     - name: "yahma/alpaca-cleaned"

Full quant reference lives in Export a trained model.

Best Practices

Related