Speed Benchmark

Rank LLM deployments by generation speed before you fine-tune or ship

Send the same request N times to each deployment and get a ranked table — fastest first — of throughput and latency.

Compare deployments before you fine-tune — pip install praisonai-train is enough, no GPU or [llm] extras needed.

Quick Start

1. Benchmark two deployments

   Pass one or more --deployment flags. Endpoint and key are read from the AZURE_OPENAI_ / OPENAI_ environment.

   pip install praisonai-train
   
   praisonai-train benchmark -d gpt-4o -d gpt-4o-mini

   Each target prints as it finishes, then a table ranks them by rows/min:

   measured gpt-4o: 87 rows/min, 4123 tok/s, ok=24/24
   measured gpt-4o-mini: 142 rows/min, 5210 tok/s, ok=24/24

2. Drive it from a YAML config

   Put the deployments and knobs in a file and point --config at it — the same idiom as generate and validate.

   praisonai-train benchmark --config benchmark.yaml

   # benchmark.yaml
   deployments:
     - gpt-4o
     - gpt-4o-mini
   n: 24
   concurrency: 8
   azure: true
   api_version: "2024-10-21"
   output: bench_results.json

How It Works

The benchmark reuses the data generator’s own HTTP path (build_chat_request / resolve_cfg), so measurements reflect a real generation workload — no duplicated request code.

Which json_mode?

json_mode sends a response_format={"type":"json_object"} hint. Turn it off for endpoints that don’t support JSON mode so you measure speed, not JSON-mode support.

CLI Flags

YAML-only keys: endpoint, api_key, azure, prompt (a {system, user} dict), and request_timeout (default 180).

Python API

Call benchmark_deployments directly for notebooks, dashboards, or CI gates.

from praisonai_train.data import benchmark_deployments, BenchResult

results: list[BenchResult] = benchmark_deployments(
    ["gpt-4o", "gpt-4o-mini"],
    n=24,
    concurrency=8,
    json_mode=True,
)

for r in results:  # already ranked fastest-first
    print(r.deployment, r.rows_per_min, "rows/min")

BenchResult is a dataclass — serialise any result with .as_dict().

Common Patterns

Benchmark across endpoints in one run

Each target may be a dict overriding endpoint, api_key, azure, or api_version — compare an Azure deployment against a self-hosted OpenAI-compatible server side by side.

from praisonai_train.data import benchmark_deployments

results = benchmark_deployments(
    [
        {"deployment": "gpt-4o", "azure": True},
        {"deployment": "llama-3.1-70b", "endpoint": "https://my-host/v1", "azure": False},
    ],
    n=24,
    json_mode=False,  # the OpenAI-compatible host has no JSON mode
)

Use your real prompt

Set an explicit prompt so the benchmark exercises the exact shape you generate with.

results = benchmark_deployments(
    ["gpt-4o"],
    prompt={"system": "You generate concise answers.",
            "user": "Write one short factual paragraph about photosynthesis."},
)

Stream progress with a callback

on_result fires as each target finishes — ideal for a live dashboard.

benchmark_deployments(
    ["gpt-4o", "gpt-4o-mini"],
    on_result=lambda r: print(f"{r.deployment}: {r.rows_per_min} rows/min"),
)

Best Practices

Related