Skip to main content
Media Evaluator scores audio, image, video, and text outputs from specialised agent pipelines, using multimodal LLM-as-judge for images and text and file checks for audio and video.

​
Quick Start

1

Grade an agent's text output

Run an agent, then score its output against a text criteria — no file needed:
from praisonaiagents import Agent
from praisonaiagents.eval import MediaEvaluator

agent = Agent(instructions="You are a friendly greeter")
output = agent.start("Greet a new user")

evaluator = MediaEvaluator(
    media_type="text",
    criteria="Greeting is warm and welcoming",
)
result = evaluator.run(output=output, print_summary=True)
print(result.score, result.passed)
2

Evaluate an image

Score a generated image against a criteria with a multimodal model:
from praisonaiagents.eval import MediaEvaluator

evaluator = MediaEvaluator(
    media_type="image",
    criteria="Image shows a red bicycle on a city street",
)
result = evaluator.run(output="https://example.com/bicycle.png")
print(result.score, result.reasoning)
3

Evaluate audio against expected text

Transcribe a TTS file and compare it to an expected transcript:
from praisonaiagents.eval import MediaEvaluator

evaluator = MediaEvaluator(
    media_type="audio",
    expected_text="Hello world",
)
result = evaluator.run(file_path="/tmp/output.mp3")
print(result.score, result.metadata)
4

Auto-detect from a file extension

Let the evaluator pick the branch from the file extension:
from praisonaiagents.eval import MediaEvaluator

evaluator = MediaEvaluator(media_type="auto")
result = evaluator.run(output="/tmp/render.mp4", print_summary=True)
print(result.media_type, result.passed)

​
How It Works

The evaluator routes the output to one of four branches, either from media_type or by auto-detection.
BranchWhat it checks
audioFile exists and is above min_file_size. With expected_text, transcribes and scores word-overlap similarity (passes at ≥ 0.7).
imageExtracts the image URL. With criteria or expected_content, a multimodal model scores it (passes at ≥ 7.0); otherwise confirms the image was generated.
videoFile exists and is above min_file_size, or a video URL is present.
textWith criteria, an LLM scores the text (passes at ≥ 7.0); otherwise confirms non-empty output.
Auto-detection reads the file extension (.mp3/.wav/.ogg/.flac/.m4a → audio, .png/.jpg/.jpeg/.gif/.webp → image, .mp4/.avi/.mov/.webm → video) and the LiteLLM response class (ImageResponse, HttpxBinaryResponseContent, VideoResponse), falling back to text.

​
Configuration Options

Every field on MediaEvaluator:
OptionTypeDefaultDescription
media_typeLiteral["audio","image","video","text","auto"]"auto"Which branch to run. auto detects from file extension or LiteLLM response class.
criteriaOptional[str]NoneCustom criteria for LLM-as-judge (image and text branches).
expected_textOptional[str]NoneReference transcript for audio comparison.
expected_contentOptional[str]NoneReference description for image/video comparison.
min_file_sizeint100Minimum acceptable file size in bytes.
modelOptional[str]None (→ OPENAI_MODEL_NAME or gpt-4o-mini)LLM for text/image judging.
verboseboolFalseEnable verbose logging.
nameOptional[str]NoneEvaluation name.
save_results_pathOptional[str]NonePath to persist the result JSON.
Methods
MethodReturnsPurpose
run(output=None, file_path=None, print_summary=False)MediaEvaluationResultRoutes to the branch and returns the result.
evaluate(output, file_path=None)MediaEvaluationResultSame routing without the summary print.

Eval Module Reference

Full Python API for the eval package

​
MediaEvaluationResult

run() returns a MediaEvaluationResult with these fields:
FieldTypeDescription
media_typestrBranch that ran (audio, image, video, or text).
passedboolWhether the output met the branch’s pass condition.
scorefloatQuality score on a 1–10 scale.
reasoningstrExplanation for the score.
file_pathOptional[str]Path of the evaluated file, when applicable.
file_sizeOptional[int]Size of the evaluated file in bytes, when applicable.
metadataDict[str, Any]Branch-specific extras (e.g. transcription, image URL).

​
Response Format

The image and text branches ask the model for a SCORE: / REASONING: response and parse it into score and reasoning. See the Judge response format note for the full rules.

​
Common Patterns

Gate an image-generation agent so only on-brief renders pass: 
from praisonaiagents.eval import MediaEvaluator

evaluator = MediaEvaluator(
    media_type="image",
    expected_content="A minimalist logo with a blue circle",
)
result = evaluator.run(output=image_response)
assert result.passed
Auto-detect a mixed pipeline that emits either audio or video: 
from praisonaiagents.eval import MediaEvaluator

evaluator = MediaEvaluator(media_type="auto", min_file_size=1024)
result = evaluator.run(output=pipeline_output_path)
Add media checks to an EvalSuite run: 
from praisonaiagents.eval import MediaEvaluator, EvalSuite

evaluator = MediaEvaluator(
    media_type="text",
    criteria="Summary is concise and accurate",
    name="summary_quality",
)
report = EvalSuite(evaluators=[evaluator], name="media_suite").run(print_summary=True)

​
Best Practices

Set expected_text when you have a known transcript — the audio branch transcribes and scores word-overlap similarity. Use criteria for image and text branches when the check is subjective (“looks professional”, “reads clearly”).
A codec that writes headers still produces a tiny file on failure. Raise min_file_size above that threshold so silent or blank renders fail the file check instead of passing.
Auto-detection reads by file extension and by LiteLLM response class (ImageResponse, HttpxBinaryResponseContent, VideoResponse). When an output has no extension or an unusual one, set media_type explicitly to pick the right branch.
The image and text branches score 1–10 and pass at ≥ 7.0, matching the threshold used across the eval framework, so mixed suites compare cleanly.

Judge

LLM-as-judge for evaluating outputs

Harness Evaluator

Score harness traces into an EvalSuite report

Context Evaluator

Score multi-agent handoff fidelity

Evaluation

Evaluators, suites, and reports