Vision Fine-Tuning

Fine-tune vision-language models with the same LoRA and publish rules as the LLM trainer

praisonai-train fine-tunes vision-language models on image-caption datasets, auto-selecting the vision trainer for any model whose name contains vision, -vl-, or visionmodel.

Quick Start

1. Write a vision config

   Point model_name at a vision model and list an image-caption dataset. A plain config trains locally and publishes nothing.

   model_name: unsloth/Llama-3.2-11B-Vision-Instruct
   load_in_4bit: true
   max_seq_length: 2048
   
   dataset:
     - name: unsloth/Radiology_mini
       split_type: train
   
   vision_instruction: "You are an expert radiographer. Describe accurately what you see in this image."

2. Train

   python -m praisonai_train.train_vision train --config config.yaml

How It Works

The vision trainer converts each image-caption row into a chat conversation, attaches vision-specific LoRA adapters via Unsloth’s FastVisionModel, and trains with TRL’s SFTTrainer.

Configuration Options

These keys are read by the vision trainer. Defaults match the source so an omitted key trains as before.

LoRA & adapter targeting

Run control & publishing

Common Patterns

Train locally, publish nothing — omit every publish key:

model_name: unsloth/Qwen2-VL-7B-Instruct
dataset:
  - name: unsloth/Radiology_mini
    split_type: train

Train and publish a private LoRA — set the flag, its target, and keep the default private visibility:

model_name: unsloth/Llama-3.2-11B-Vision-Instruct
dataset:
  - name: unsloth/Radiology_mini
    split_type: train
huggingface_save: true
hf_model_name: me/my-vision-model

Override the LoRA rank — now honoured on the vision path:

model_name: mistral-community/pixtral-12b
lora_r: 8
lora_alpha: 8
dataset:
  - name: unsloth/Radiology_mini
    split_type: train

Best Practices

Related