Multimodal Tool Output

Return images and files from tools so the model can see them

Let your tools hand back screenshots, charts, or rendered pages — the agent sees the picture, not a description of it.

from praisonaiagents import Agent, tool
from praisonaiagents.tools import multimodal_content, text_part, image_part

@tool
def describe_ui() -> dict:
    return multimodal_content([text_part("UI screenshot:"), image_part(b"...png bytes...")])

agent = Agent(name="Reviewer", instructions="Inspect tool images", tools=[describe_ui])
agent.start("What is wrong with this layout?")

The user asks a visual question; tools return images the vision model reads on the next turn.

Quick Start

1. Screenshot tool returning an image

   A @tool-decorated function captures a screenshot and hands the PNG bytes back to the agent. The vision model sees the actual image on the next turn.

   from praisonaiagents import Agent, tool
   from praisonaiagents.tools import multimodal_content, text_part, image_part
   
   @tool
   def take_screenshot() -> dict:
       """Capture a screenshot and return it as an image the agent can see."""
       import subprocess, base64
       result = subprocess.run(
           ["python", "-c",
            "import PIL.ImageGrab; img = PIL.ImageGrab.grab(); "
            "import io; buf=io.BytesIO(); img.save(buf, 'PNG'); "
            "print(buf.getvalue().hex())"],
           capture_output=True, text=True
       )
       png_bytes = bytes.fromhex(result.stdout.strip())
       return multimodal_content(
           text_part("Here is the current screenshot:"),
           image_part(png_bytes),
       )
   
   agent = Agent(
       instructions="You are a screen-reading assistant. Take a screenshot and describe what you see.",
       tools=[take_screenshot],
       llm="gpt-4o",
   )
   
   agent.start("Take a screenshot and describe what is on screen.")

   The user asks for a screenshot summary; the tool returns multimodal content the model can read.

2. Remote image via URL

   No downloading needed — pass url= directly and the model fetches it.

   from praisonaiagents import Agent, tool
   from praisonaiagents.tools import multimodal_content, text_part, image_part
   
   @tool
   def fetch_chart(symbol: str) -> dict:
       """Return a stock chart image for the given symbol."""
       chart_url = f"https://charts.example.com/{symbol}.png"
       return multimodal_content(
           text_part(f"Stock chart for {symbol}:"),
           image_part(url=chart_url),
       )
   
   agent = Agent(
       instructions="You are a financial analyst. Fetch charts and describe trends.",
       tools=[fetch_chart],
       llm="gpt-4o",
   )
   
   agent.start("Show me the chart for AAPL and describe the recent trend.")

3. Bare list (lightweight shape)

   Skip the helper — return a plain list of part-dicts. The runtime normalises it automatically.

   from praisonaiagents import Agent, tool
   
   @tool
   def render_diagram(code: str) -> list:
       """Render a Mermaid diagram and return the image."""
       import base64, io
       # ... render to PNG bytes ...
       png_bytes = b""  # replace with real rendering
       b64 = base64.b64encode(png_bytes).decode()
       return [
           {"type": "text", "text": "Rendered diagram:"},
           {"type": "image", "data": b64, "mime": "image/png"},
       ]
   
   agent = Agent(
       instructions="Render diagrams and explain them.",
       tools=[render_diagram],
       llm="gpt-4o",
   )
   
   agent.start("Render a simple flowchart and explain what it shows.")

—Let your tools hand back screenshots, charts, or rendered pages — the agent sees the picture, not a description of it.

from praisonaiagents import Agent, tool
from praisonaiagents.tools import multimodal_content, text_part, image_part

@tool
def show_chart() -> dict:
    return multimodal_content(text_part("Chart:"), image_part(b"...png bytes..."))

agent = Agent(name="analyst", instructions="Interpret visuals from tools.", tools=[show_chart])
agent.start("What does the chart show?")

The user asks a visual question; the tool returns image bytes the vision model reads on the next turn.

The runtime emits two messages automatically:

1. A tool message satisfying the tool_call_id contract (containing a short text fallback).
2. A follow-up user message carrying the image_url parts.

This two-step delivery is required because most providers reject image_url parts inside tool-role messages. You don’t need to do anything — it’s automatic. When an assistant turn contains multiple tool calls, all tool replies stay consecutive and the media user message is appended only after the full batch flushes.

Choosing the Return Shape

All four shapes are auto-detected by the runtime — pick whichever feels natural:

Configuration Options

There is no separate config class. Import the helpers from praisonaiagents.tools:

from praisonaiagents.tools import multimodal_content, text_part, image_part, file_part, ToolResult

image_part accepts any of:

• Raw bytes
• Base64 str
• A data: URI string
• A remote url= keyword argument

Common Patterns

Screenshot tool

from praisonaiagents import Agent, tool
from praisonaiagents.tools import multimodal_content, text_part, image_part
import io

@tool
def screenshot(region: str = "full") -> dict:
    """Capture a screenshot and return it so the agent can see the screen."""
    try:
        from PIL import ImageGrab, Image
        img = ImageGrab.grab()
        # Resize to keep under 5 MB limit
        img.thumbnail((1280, 800), Image.LANCZOS)
        buf = io.BytesIO()
        img.save(buf, format="PNG", optimize=True)
        png_bytes = buf.getvalue()
        return multimodal_content(
            text_part(f"Screenshot ({region}):"),
            image_part(png_bytes),
        )
    except Exception as e:
        return {"error": str(e)}

agent = Agent(
    instructions="You are a UI assistant. Screenshot the screen and describe the UI.",
    tools=[screenshot],
    llm="gpt-4o",
)
agent.start("What application is currently open?")

Chart renderer (matplotlib → PNG)

from praisonaiagents import Agent, tool
from praisonaiagents.tools import multimodal_content, text_part, image_part
import io

@tool
def plot_chart(data: str) -> dict:
    """Plot a bar chart from comma-separated values and return the image."""
    import matplotlib
    matplotlib.use("Agg")
    import matplotlib.pyplot as plt

    values = [float(x) for x in data.split(",")]
    fig, ax = plt.subplots(figsize=(6, 4))
    ax.bar(range(len(values)), values)
    buf = io.BytesIO()
    fig.savefig(buf, format="png", dpi=100, bbox_inches="tight")
    plt.close(fig)
    png_bytes = buf.getvalue()

    return multimodal_content(
        text_part(f"Bar chart for values: {data}"),
        image_part(png_bytes),
    )

agent = Agent(
    instructions="Plot charts and analyse trends in the data.",
    tools=[plot_chart],
    llm="gpt-4o",
)
agent.start("Plot 10,25,15,40,30 and describe the trend.")

MCP image passthrough

MCP servers often return image blocks with a mimeType key. Return the block as-is — the runtime normalises mimeType → mime automatically.

from praisonaiagents import Agent, tool

@tool
def mcp_screenshot() -> dict:
    """Return an MCP-style image content block directly."""
    import base64
    # MCP servers return blocks like this:
    mcp_block = {
        "type": "image",
        "data": base64.b64encode(b"...png bytes...").decode(),
        "mimeType": "image/png",   # note: mimeType, not mime
    }
    return mcp_block   # runtime normalises it — no extra wrapping needed

agent = Agent(
    instructions="Use the MCP screenshot tool and describe what you see.",
    tools=[mcp_screenshot],
    llm="gpt-4o",
)
agent.start("Take a screenshot using MCP and describe the screen.")

Best Practices

from PIL import Image
import io

def resize_to_safe(img: Image.Image, max_bytes: int = 4_000_000) -> bytes:
    """Resize image until it fits under the byte limit."""
    quality = 90
    scale = 1.0
    while True:
        buf = io.BytesIO()
        w, h = int(img.width * scale), int(img.height * scale)
        resized = img.resize((w, h), Image.LANCZOS)
        resized.save(buf, "PNG", optimize=True)
        data = buf.getvalue()
        if len(data) <= max_bytes or scale < 0.2:
            return data
        scale *= 0.8

agent = Agent(
    instructions="...",
    tools=[take_screenshot],
    llm="gpt-4o",   # vision-capable
)

return multimodal_content(
    text_part("Screenshot of the dashboard at 14:32:"),   # ← always include
    image_part(png_bytes),
)

from pdf2image import convert_from_bytes
import io

def pdf_to_image(pdf_bytes: bytes) -> bytes:
    """Convert first page of PDF to PNG."""
    images = convert_from_bytes(pdf_bytes, first_page=1, last_page=1)
    buf = io.BytesIO()
    images[0].save(buf, "PNG")
    return buf.getvalue()

return multimodal_content(
    text_part("First page of the PDF document:"),
    image_part(pdf_to_image(pdf_bytes)),   # image, not file_part
)

Security

Related