> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Run Fingerprint
> Refuse to compare eval scores when the model, prompt, or tools moved between runs
A run fingerprint identifies the setup โ model, prompt, tools, evaluators โ an evaluation was scored under, so two scores are never put side by side when the thing being scored has moved.
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
subgraph "Run Fingerprint"
Setup[๐ Model + Prompt + Tools] --> FP[๐ run_fingerprint]
FP --> Result[โ
Stamped Result]
Result --> Check{๐ assert_comparable}
Check -->|Same setup| Compare[โ
Compare scores]
Check -->|Setup moved| Refuse[โ ๏ธ FingerprintMismatch]
end
classDef input fill:#6366F1,stroke:#7C90A0,color:#fff
classDef process fill:#189AB4,stroke:#7C90A0,color:#fff
classDef result fill:#10B981,stroke:#7C90A0,color:#fff
classDef warn fill:#F59E0B,stroke:#7C90A0,color:#fff
class Setup input
class FP process
class Result,Compare result
class Check process
class Refuse warn
```
## Quick Start
Run an `EvalSuite` โ the fingerprint is stamped onto the result. Compare two suite results with `assert_comparable`.
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent
from praisonaiagents.eval import EvalSuite, AccuracyEvaluator, assert_comparable
agent = Agent(name="Assistant", instructions="Be helpful.")
baseline = EvalSuite(evaluators=[
AccuracyEvaluator(agent=agent, input_text="What is 2+2?", expected_output="4"),
]).run()
# ... later, after any change ...
current = EvalSuite(evaluators=[
AccuracyEvaluator(agent=agent, input_text="What is 2+2?", expected_output="4"),
]).run()
assert_comparable(baseline.fingerprint, current.fingerprint) # raises if the setup moved
print(current.overall_score - baseline.overall_score)
```
Compute a fingerprint yourself when you are not using `EvalSuite`.
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent
from praisonaiagents.eval import run_fingerprint
agent = Agent(name="Assistant", instructions="Be helpful.")
fp = run_fingerprint(
model="gpt-4o",
prompt=agent.instructions,
tools=agent.tools,
evaluators=["AccuracyEvaluator"],
)
print(fp) # evalfp1:<16-hex>
```
Use `fingerprint_parts` to see exactly what changed.
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent
from praisonaiagents.eval import fingerprint_parts
agent = Agent(name="Assistant", instructions="Be helpful.")
print(fingerprint_parts(model="gpt-4o", prompt=agent.instructions, tools=agent.tools))
# {"version": "evalfp1", "model": "gpt-4o", "prompt": "...16-hex...", "tools": [...], ...}
```
***
## How It Works
`EvalSuite.run()` reads `model`, `prompt`, and `tools` from each evaluator's agent and stamps the fingerprint onto the result at the start of the run.
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
sequenceDiagram
participant User
participant Suite as EvalSuite
participant FP as _fingerprint()
participant Result as EvalSuiteResult
User->>Suite: run()
Suite->>FP: read model / prompt / tools from each agent
FP-->>Suite: evalfp1:
Suite->>Result: stamp fingerprint
Result-->>User: result.fingerprint
User->>User: assert_comparable(old, new)
Note over User: same โ OK ยท different โ FingerprintMismatch
```
Reading is **best-effort** โ anything unreadable is left out rather than raised, so the fingerprint is never the reason an evaluation fails to run. If nothing can be read, `fingerprint` is `""`.
***
## What Goes Into a Fingerprint
Each input is reduced to a stable shape before hashing.
| Parameter | Type | Behavior |
| ------------ | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `model` | `Any` | Model id string, or any object with a `.model` or `.name` attribute (an `Agent`, an LLM wrapper). Coerced to `str`. |
| `prompt` | `Any` | Hashed, not stored โ SHA-256 first 16 hex chars. The full text is never included. |
| `tools` | `Any` | List/tuple/set of tools, or a single tool. Each reduced to its `.name` / `.__name__` / `str()` fallback. Order-independent โ the list is sorted. |
| `evaluators` | `Iterable[str]` | Sorted list of evaluator class names, e.g. `["AccuracyEvaluator", "HarnessEvaluator"]`. |
| `extra` | `Optional[dict]` | Any additional key/value pairs, stringified and sorted. Escape hatch for anything else that affects the setup. |
### Deliberately excluded
* **Timestamps, run ids, sample order, latency.** Anything that varies run to run without changing what a score *means* is left out. A fingerprint that changed every run would refuse every comparison and be switched off.
* **Tool order.** A reordered tool list is the same setup. A guard that changed on reordering would refuse valid comparisons.
* **The prompt text itself.** Hashed, never stored, so fingerprints can safely land in shared result files.
### Public API
| Symbol | Kind | Purpose |
| ------------------------------------------------ | ------------------------ | ----------------------------------------------------------------------- |
| `run_fingerprint(...)` | function โ `str` | Compute a short, stable id (`"evalfp1:<16-hex>"`) for this setup. |
| `fingerprint_parts(...)` | function โ `dict` | The exact inputs a fingerprint is computed from, for diagnosis. |
| `compare_fingerprints(left, right)` | function โ `bool` | `True` when two runs were made under the same setup. |
| `assert_comparable(left, right, differing=None)` | function | Raise `FingerprintMismatch` unless two runs are comparable. |
| `FingerprintMismatch` | exception (`ValueError`) | Raised by `assert_comparable`; carries `.left`, `.right`, `.differing`. |
| `FINGERPRINT_VERSION` | `str` constant | `"evalfp1"` โ version tag prefixed to every fingerprint. |
**Missing fingerprints are unknown, not equal.** `compare_fingerprints("", "")` returns `False`, and `assert_comparable("", "")` raises `FingerprintMismatch`. Assuming legacy unstamped runs are equal would let every old baseline silently pass the check โ the guard refuses instead.
**Prompts are hashed, never stored.** A fingerprint is written to result files and shared, so the full prompt text is never included.
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.eval import fingerprint_parts
secret = "customer 4111-1111-1111-1111 wants a refund"
parts = fingerprint_parts(prompt=secret)
assert secret not in str(parts)
```
***
## Common Patterns
### Gate CI on a stable baseline
Save `result.fingerprint` alongside the score, then guard before comparing numbers on the next run.
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.eval import assert_comparable
assert_comparable(previous.fingerprint, current.fingerprint)
print(current.overall_score - previous.overall_score)
```
### Persist fingerprints with results
`result.summary` already includes the fingerprint โ write it to disk with the score.
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
import json
from praisonaiagents import Agent
from praisonaiagents.eval import EvalSuite, AccuracyEvaluator
agent = Agent(name="Assistant", instructions="Be helpful.")
result = EvalSuite(evaluators=[
AccuracyEvaluator(agent=agent, input_text="What is 2+2?", expected_output="4"),
]).run()
with open("eval-report.json", "w") as f:
json.dump(result.summary, f, indent=2, default=str) # includes "fingerprint"
```
### Compare setups on purpose
When A/B'ing prompts, skip `assert_comparable` and record both fingerprints so the difference is visible in the graph.
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.eval import run_fingerprint
a = run_fingerprint(model="gpt-4o", prompt="Be terse.")
b = run_fingerprint(model="gpt-4o", prompt="Be verbose.")
print(a, b) # two different fingerprints, compared deliberately
```
***
## Best Practices
A silent side-by-side of two different setups is the failure this guards against. Guard first, then subtract scores.
Old fingerprints will not compare equal to new ones after a bump โ that is intentional, so scores from a changed recipe are never treated as comparable to older runs.
Temperature, seed, retrieval index name โ pass them through `extra=` so they become part of the fingerprint.
The fingerprint code already fails silently at compute time (best-effort). Failing loud at compare time is deliberate โ if two runs are not comparable, that is real.
***
## Related
Run every evaluator as one CI gate
Score test-harness traces
Score loop health and convergence
Export and import suites in the EvalPort open spec