Serve & MTP Fast-Inference

Serve a fine-tuned GGUF over OpenAI-compatible HTTP, with optional Multi-Token Prediction for lossless 1.4–2.2× speedup on Gemma-4

Serve any fine-tuned GGUF over an OpenAI-compatible HTTP endpoint. For Gemma-4, add MTP to get lossless 1.4–2.2× faster generation — no drafter retraining, one flag.

Quick Start

1. Serve a trained model

   Point at a GGUF and get an OpenAI-compatible endpoint on 127.0.0.1:8080.

   pip install "praisonai-train[llm]"
   
   praisonai-train serve --gguf model.gguf

2. Use it from any OpenAI client

   The endpoint is OpenAI-compatible, so praisonaiagents (and every other OpenAI SDK) talks to it out of the box.

   from praisonaiagents import Agent
   
   agent = Agent(
       instructions="You are a helpful assistant.",
       llm={
           "model": "openai/local",
           "base_url": "http://127.0.0.1:8080/v1",
           "api_key": "not-used",
       },
   )
   agent.start("Explain MTP in one sentence.")

3. Benchmark and see the MTP win

   --benchmark is a one-shot: it prints tokens-per-second plus draft acceptance for Gemma-4, then exits.

   praisonai-train serve --gguf gemma4-e4b.gguf --benchmark

4. Export the GGUF with its MTP drafter

   Pass --mtp-draft when exporting a Gemma-4 model — the stock drafter is fetched and placed next to the GGUF so serve picks it up automatically.

   praisonai-train export gguf --model-dir lora_model --mtp-draft

How It Works

MTP proposes several tokens with a small drafter, then the fine-tuned model verifies them in one pass — so the output is bitwise identical to plain serving.

The drafter is a separate artifact and is never contained in a plain q4_K_M GGUF — that is why the fetch step exists. Verification is done by the fine-tuned target model, so the generation is lossless: identical output, fewer forward passes.

Configuration Options

Two surfaces: CLI flags on serve, and config-file keys the trainer accepts when you export --mtp-draft.

CLI flags — praisonai-train serve

Provide either --gguf or --model-dir to pick the target.

Config keys — export --mtp-draft

These keys are accepted by the trainer’s config (config.yaml) alongside export.

Common Patterns

Serve any GGUF (non-Gemma-4)

Plain serving works for every fine-tuned model. A non-Gemma family falls back to plain serving with a clear message.

praisonai-train serve --gguf qwen-finetune.gguf

MTP fast inference isn't available for '<model>' (only the Gemma-4 family). Serving without a drafter.

Serve from a model directory

Point at a directory and the target GGUF is auto-detected (drafter files are skipped).

praisonai-train serve --model-dir lora_model

Benchmark before you ship

--benchmark prints raw tokens/sec and (for Gemma-4) draft acceptance in one line — confirm the MTP win on your box before flipping traffic over.

praisonai-train serve --gguf gemma4-e4b.gguf --benchmark

Talk to it from praisonaiagents

Any Agent that already speaks OpenAI works — point base_url at http://<host>:<port>/v1.

from praisonaiagents import Agent

agent = Agent(
    instructions="You are a helpful assistant.",
    llm={
        "model": "openai/local",
        "base_url": "http://127.0.0.1:8080/v1",
        "api_key": "not-used",
    },
)
agent.start("Say hello.")

Supported models

Only Gemma-4 ships a stock MTP drafter. Verified mapping (from _mtp.py):

The drafter precision is Q8_0 by default; bf16 and f16 variants resolve to BF16 / F16 filenames.

Non-Gemma models (Llama / Qwen / Mistral / Phi / DeepSeek) do not ship MTP drafters. serve --gguf still works on them — just without the speculative-decoding pathway. Do not expect a 1.4× on a Qwen fine-tune.

Field notes

Best Practices

Related