Skip to main content
Let your tools hand back screenshots, charts, or rendered pages — the agent sees the picture, not a description of it. 
from praisonaiagents import Agent, tool
from praisonaiagents.tools import multimodal_content, text_part, image_part

@tool
def describe_ui() -> dict:
    return multimodal_content([text_part("UI screenshot:"), image_part(b"...png bytes...")])

agent = Agent(name="Reviewer", instructions="Inspect tool images", tools=[describe_ui])
agent.start("What is wrong with this layout?")
The user asks a visual question; tools return images the vision model reads on the next turn.

​
Quick Start

1

Screenshot tool returning an image

A @tool-decorated function captures a screenshot and hands the PNG bytes back to the agent. The vision model sees the actual image on the next turn.
from praisonaiagents import Agent, tool
from praisonaiagents.tools import multimodal_content, text_part, image_part

@tool
def take_screenshot() -> dict:
    """Capture a screenshot and return it as an image the agent can see."""
    import subprocess, base64
    result = subprocess.run(
        ["python", "-c",
         "import PIL.ImageGrab; img = PIL.ImageGrab.grab(); "
         "import io; buf=io.BytesIO(); img.save(buf, 'PNG'); "
         "print(buf.getvalue().hex())"],
        capture_output=True, text=True
    )
    png_bytes = bytes.fromhex(result.stdout.strip())
    return multimodal_content(
        text_part("Here is the current screenshot:"),
        image_part(png_bytes),
    )

agent = Agent(
    instructions="You are a screen-reading assistant. Take a screenshot and describe what you see.",
    tools=[take_screenshot],
    llm="gpt-4o",
)

agent.start("Take a screenshot and describe what is on screen.")
The user asks for a screenshot summary; the tool returns multimodal content the model can read.
2

Remote image via URL

No downloading needed — pass url= directly and the model fetches it.
from praisonaiagents import Agent, tool
from praisonaiagents.tools import multimodal_content, text_part, image_part

@tool
def fetch_chart(symbol: str) -> dict:
    """Return a stock chart image for the given symbol."""
    chart_url = f"https://charts.example.com/{symbol}.png"
    return multimodal_content(
        text_part(f"Stock chart for {symbol}:"),
        image_part(url=chart_url),
    )

agent = Agent(
    instructions="You are a financial analyst. Fetch charts and describe trends.",
    tools=[fetch_chart],
    llm="gpt-4o",
)

agent.start("Show me the chart for AAPL and describe the recent trend.")
3

Bare list (lightweight shape)

Skip the helper — return a plain list of part-dicts. The runtime normalises it automatically.
from praisonaiagents import Agent, tool

@tool
def render_diagram(code: str) -> list:
    """Render a Mermaid diagram and return the image."""
    import base64, io
    # ... render to PNG bytes ...
    png_bytes = b""  # replace with real rendering
    b64 = base64.b64encode(png_bytes).decode()
    return [
        {"type": "text", "text": "Rendered diagram:"},
        {"type": "image", "data": b64, "mime": "image/png"},
    ]

agent = Agent(
    instructions="Render diagrams and explain them.",
    tools=[render_diagram],
    llm="gpt-4o",
)

agent.start("Render a simple flowchart and explain what it shows.")
---Let your tools hand back screenshots, charts, or rendered pages — the agent sees the picture, not a description of it. 
from praisonaiagents import Agent, tool
from praisonaiagents.tools import multimodal_content, text_part, image_part

@tool
def show_chart() -> dict:
    return multimodal_content(text_part("Chart:"), image_part(b"...png bytes..."))

agent = Agent(name="analyst", instructions="Interpret visuals from tools.", tools=[show_chart])
agent.start("What does the chart show?")
The user asks a visual question; the tool returns image bytes the vision model reads on the next turn. The runtime emits two messages automatically:
  1. A tool message satisfying the tool_call_id contract (containing a short text fallback).
  2. A follow-up user message carrying the image_url parts.
This two-step delivery is required because most providers reject image_url parts inside tool-role messages. You don’t need to do anything — it’s automatic. When an assistant turn contains multiple tool calls, all tool replies stay consecutive and the media user message is appended only after the full batch flushes.

​
Choosing the Return Shape

All four shapes are auto-detected by the runtime — pick whichever feels natural:
ShapeWhen to use
multimodal_content(*parts)Recommended path — explicit, readable, type-safe
image_part(url=...)Remote image you don’t want to download locally
Bare list of part-dictsLighter-weight; good for simple cases or MCP passthrough
Single part-dictOne image with no caption
MCP-style block (mimeType key)Runtime normalises mimeType → mime automatically

​
Configuration Options

There is no separate config class. Import the helpers from praisonaiagents.tools: 
from praisonaiagents.tools import multimodal_content, text_part, image_part, file_part, ToolResult
HelperArgumentsReturns
multimodal_content(*parts, output=None, success=True)One or more part-dictsToolResult with is_multimodal=True
text_part(text)text: str{"type": "text", "text": ...}
image_part(data=None, *, mime="image/png", name=None, url=None)bytes / base64 str / data URI in data, or remote url=image part-dict
file_part(data=None, *, mime="application/octet-stream", name=None, url=None)Same as image_partfile part-dict
ToolResult(output, success=True, error=None, metadata=None, content=None)Direct constructionToolResult; set content list to enable multimodal
image_part accepts any of:
  • Raw bytes
  • Base64 str
  • A data: URI string
  • A remote url= keyword argument

​
Common Patterns

​
Screenshot tool

from praisonaiagents import Agent, tool
from praisonaiagents.tools import multimodal_content, text_part, image_part
import io

@tool
def screenshot(region: str = "full") -> dict:
    """Capture a screenshot and return it so the agent can see the screen."""
    try:
        from PIL import ImageGrab, Image
        img = ImageGrab.grab()
        # Resize to keep under 5 MB limit
        img.thumbnail((1280, 800), Image.LANCZOS)
        buf = io.BytesIO()
        img.save(buf, format="PNG", optimize=True)
        png_bytes = buf.getvalue()
        return multimodal_content(
            text_part(f"Screenshot ({region}):"),
            image_part(png_bytes),
        )
    except Exception as e:
        return {"error": str(e)}

agent = Agent(
    instructions="You are a UI assistant. Screenshot the screen and describe the UI.",
    tools=[screenshot],
    llm="gpt-4o",
)
agent.start("What application is currently open?")

​
Chart renderer (matplotlib → PNG)

from praisonaiagents import Agent, tool
from praisonaiagents.tools import multimodal_content, text_part, image_part
import io

@tool
def plot_chart(data: str) -> dict:
    """Plot a bar chart from comma-separated values and return the image."""
    import matplotlib
    matplotlib.use("Agg")
    import matplotlib.pyplot as plt

    values = [float(x) for x in data.split(",")]
    fig, ax = plt.subplots(figsize=(6, 4))
    ax.bar(range(len(values)), values)
    buf = io.BytesIO()
    fig.savefig(buf, format="png", dpi=100, bbox_inches="tight")
    plt.close(fig)
    png_bytes = buf.getvalue()

    return multimodal_content(
        text_part(f"Bar chart for values: {data}"),
        image_part(png_bytes),
    )

agent = Agent(
    instructions="Plot charts and analyse trends in the data.",
    tools=[plot_chart],
    llm="gpt-4o",
)
agent.start("Plot 10,25,15,40,30 and describe the trend.")

​
MCP image passthrough

MCP servers often return image blocks with a mimeType key. Return the block as-is — the runtime normalises mimeType → mime automatically. 
from praisonaiagents import Agent, tool

@tool
def mcp_screenshot() -> dict:
    """Return an MCP-style image content block directly."""
    import base64
    # MCP servers return blocks like this:
    mcp_block = {
        "type": "image",
        "data": base64.b64encode(b"...png bytes...").decode(),
        "mimeType": "image/png",   # note: mimeType, not mime
    }
    return mcp_block   # runtime normalises it — no extra wrapping needed

agent = Agent(
    instructions="Use the MCP screenshot tool and describe what you see.",
    tools=[mcp_screenshot],
    llm="gpt-4o",
)
agent.start("Take a screenshot using MCP and describe the screen.")

​
Best Practices

The runtime enforces a 5 MB cap per image part (MULTIMODAL_IMAGE_BYTE_LIMIT = 5_000_000). Images over this limit are silently dropped with a warning — the model only sees the text fallback.Resize before returning:
from PIL import Image
import io

def resize_to_safe(img: Image.Image, max_bytes: int = 4_000_000) -> bytes:
    """Resize image until it fits under the byte limit."""
    quality = 90
    scale = 1.0
    while True:
        buf = io.BytesIO()
        w, h = int(img.width * scale), int(img.height * scale)
        resized = img.resize((w, h), Image.LANCZOS)
        resized.save(buf, "PNG", optimize=True)
        data = buf.getvalue()
        if len(data) <= max_bytes or scale < 0.2:
            return data
        scale *= 0.8
Image parts are only perceived by models that support vision. Use:
  • gpt-4o / gpt-4o-mini (OpenAI)
  • claude-3-5-sonnet-20241022 or any Claude 3+ model (Anthropic)
  • gemini-1.5-pro / gemini-1.5-flash (Google)
Plain text models only see the text fallback string — they do not raise an error.
agent = Agent(
    instructions="...",
    tools=[take_screenshot],
    llm="gpt-4o",   # vision-capable
)
A caption improves grounding and is the only signal a non-vision model gets. Always pair image_part with text_part:
return multimodal_content(
    text_part("Screenshot of the dashboard at 14:32:"),   # ← always include
    image_part(png_bytes),
)
This also helps when the image is dropped due to the 5 MB cap — the agent still gets useful context.
file_part is currently surfaced to the model as a text annotation [file: name (mime)]. Most LLM providers cannot ingest arbitrary binary files inline.For PDFs or other documents you want the model to see, rasterise to images first:
from pdf2image import convert_from_bytes
import io

def pdf_to_image(pdf_bytes: bytes) -> bytes:
    """Convert first page of PDF to PNG."""
    images = convert_from_bytes(pdf_bytes, first_page=1, last_page=1)
    buf = io.BytesIO()
    images[0].save(buf, "PNG")
    return buf.getvalue()

return multimodal_content(
    text_part("First page of the PDF document:"),
    image_part(pdf_to_image(pdf_bytes)),   # image, not file_part
)

​
Security

Text parts returned by external or untrusted tools are automatically wrapped in a prompt-injection fence — the same protection used for text-only tool output. You don’t need to do anything, but know that if you mark a tool as external, its text parts will be fenced before the model sees them.

Multimodal Agents

Send images to the agent via Task(images=[...]) or agent.chat(attachments=[...]) — the symmetric direction.

Image Generation

Generate new images with AI models and use them in agent workflows.