Prompt Optimizer

Automatically optimise an agent's instructions against an eval set and keep the best-scoring version.

Prompt Optimizer rewrites your agent’s instructions and keeps the version that scores highest on your eval.

from praisonaiagents import Agent

agent = Agent(name="Summariser", instructions="Summarise the input.")

result = agent.optimize_instructions(
    evalset=[("Summarise: PraisonAI is an agent framework.", "PraisonAI is a framework for agents.")],
    criteria="concise, faithful, no hallucinations",
)

print(result.best_score, result.improved)
print(agent.instructions)   # winning instructions are written back

Quick Start

1. Optimise with the LLM Judge

   Score each candidate with the built-in Judge using plain-English criteria.

   from praisonaiagents import Agent
   
   agent = Agent(name="Summariser", instructions="Summarise the input.")
   
   result = agent.optimize_instructions(
       evalset=[
           ("Summarise: PraisonAI is an agent framework.", "PraisonAI is a framework for agents."),
           ("Summarise: Skills are grounded reusable tasks.", "Skills are reusable grounded tasks."),
       ],
       criteria="concise, faithful, no hallucinations",
       n_candidates=6,
   )
   
   print(result.best_score, result.improved)

2. Score with a numeric metric

   Supply a metric(output, expected) -> float to replace the Judge with your own gold-data score.

   from praisonaiagents import Agent
   
   def word_overlap(output: str, expected: str) -> float:
       a, b = set(output.lower().split()), set(expected.lower().split())
       return len(a & b) / max(len(b), 1)
   
   agent = Agent(name="Summariser", instructions="Summarise the input.")
   
   result = agent.optimize_instructions(
       evalset=[
           ("Summarise: PraisonAI is an agent framework.", "PraisonAI is a framework for agents."),
           ("Summarise: Skills are grounded reusable tasks.", "Skills are reusable grounded tasks."),
       ],
       metric=word_overlap,
       n_candidates=4,
   )
   
   print(result.best_score)

3. Preview without applying

   Set apply=False to keep the original instructions and inspect the winner first.

   from praisonaiagents import Agent
   
   agent = Agent(name="Summariser", instructions="Summarise the input.")
   
   result = agent.optimize_instructions(
       evalset=[("Summarise X", "gold X"), ("Summarise Y", "gold Y")],
       criteria="concise, faithful",
       apply=False,
   )
   
   print(result.best_instructions)   # inspect before applying
   agent.instructions = result.best_instructions   # apply manually when ready

How It Works

The optimiser runs in five steps:

Configuration Options

agent.optimize_instructions(...) accepts these parameters:

OptimizeResult

optimize_instructions(...) returns an OptimizeResult:

Common Patterns

Judge-based scoring for open-ended tasks

Use criteria when there is no single gold answer.

from praisonaiagents import Agent

agent = Agent(name="Writer", instructions="Write a product blurb.")

result = agent.optimize_instructions(
    evalset=[("Blurb for an AI assistant", None), ("Blurb for a note app", None)],
    criteria="engaging, concise, benefit-led",
    n_candidates=6,
)

Numeric-metric scoring with gold data

Plug in any empirical score — rouge_l, accuracy, latency, or a custom function.

from praisonaiagents import Agent

def accuracy(output: str, expected: str) -> float:
    return 1.0 if expected.lower() in output.lower() else 0.0

agent = Agent(name="Classifier", instructions="Answer yes or no.")

result = agent.optimize_instructions(
    evalset=[("Is 7 prime?", "yes"), ("Is 8 prime?", "no")],
    metric=accuracy,
)

Async usage

from praisonaiagents import Agent

agent = Agent(name="Summariser", instructions="Summarise the input.")

result = await agent.aoptimize_instructions(
    evalset=[("Summarise X", "gold X")],
    criteria="concise, faithful",
)

Dry-run then apply manually

from praisonaiagents import Agent

agent = Agent(name="Summariser", instructions="Summarise the input.")

result = agent.optimize_instructions(
    evalset=[("Summarise X", "gold X")],
    criteria="concise",
    apply=False,
)

if result.improved:
    agent.instructions = result.best_instructions

Best Practices

Related