The command is named
infer, not generate — generate is dataset generation and reusing it would displace that command.Quick Start
1
Prompt the trained model
Point at a trained adapter and pass a prompt — the reply streams token-by-token.
2
Tune the output length and sampling
Bump
--max-new-tokens for longer replies; lower --temperature for more focused output.How It Works
infer loads the trained model with Unsloth, applies the tokenizer’s chat template with a user role, and streams the generation straight to stdout.
The model is loaded via Unsloth’s FastLanguageModel.from_pretrained + for_inference, and generation streams through TextStreamer(tokenizer, skip_prompt=True).
Configuration Options
If the Unsloth / Transformers stack isn’t installed,
infer prints an error with the remediation pip install "praisonai-train[llm]" and exits 1.Streaming
The prompt is skipped (skip_prompt=True), so only the model’s reply prints — the first token is your “is it alive?” signal, arriving long before the last on a large model.
Choosing between infer, serve, benchmark
Three commands touch a trained model — pick by what you’re trying to answer.infer runs a single local generation; serve stands up an OpenAI-compatible endpoint; benchmark measures throughput on already-deployed remote APIs.
Best Practices
Install the [llm] extra first
Install the [llm] extra first
infer needs the Unsloth / Transformers stack. Run pip install "praisonai-train[llm]" — otherwise the command exits 1 with that exact remediation.Keep the defaults for a sanity check
Keep the defaults for a sanity check
temperature=0.7 and max-new-tokens=256 are deliberately usable defaults. Leave them for a first “did my fine-tune take?” check and only bump --max-new-tokens when you need longer outputs.Disable 4-bit only when you need to
Disable 4-bit only when you need to
Turn off
--load-in-4bit only if the model doesn’t fit in 4-bit or you’re comparing against full-precision output. The 4-bit default keeps memory low.One prompt, one generation
One prompt, one generation
infer is not a chat loop — it sends a single prompt and streams one reply, then exits. For a persistent endpoint use serve instead.Related
Checkpoints CLI
Discover what a training run saved without
ls.Serve & MTP Fast-Inference
Serve a GGUF over an OpenAI-compatible endpoint.
Export a trained model
Publish a trained model to HF, GGUF, or Ollama.
Train CLI
Full flag reference for every subcommand.