Multimodal Agents - PraisonAI

Multimodal agents process text, images, and video in a single workflow — attach media to tasks or pass ephemeral attachments at chat time.

from praisonaiagents import Agent

agent = Agent(
    name="VisionAnalyst",
    instructions="Analyse images and video and describe what you see.",
    llm="gpt-4o-mini",
)
agent.start("Describe the landmark in this image.")

The user attaches media; the agent returns a single analysis covering text and visuals.

How It Works

Quick Start

Code
No Code

Install Package

First, install the PraisonAI Agents package:

pip install praisonaiagents opencv-python moviepy

Set API Key

Set your OpenAI API key as an environment variable in your terminal:

export OPENAI_API_KEY="${OPENAI_API_KEY:?Set OPENAI_API_KEY in your shell}"

Create a file

Create a new file app.py with the basic setup:

from praisonaiagents import Agent, Task, AgentTeam

# Create Vision Analysis Agent
vision_agent = Agent(
    name="VisionAnalyst",
    role="Computer Vision Specialist",
    goal="Analyze images and videos to extract meaningful information",
    backstory="""You are an expert in computer vision and image analysis.
    You excel at describing images, detecting objects, and understanding visual content.""",
    llm="gpt-4o-mini",
    reflection=False
)

# Create tasks with different media types
task = Task(
    name="analyze_landmark",
    description="Describe this famous landmark and its architectural features.",
    expected_output="Detailed description of the landmark's architecture and significance",
    agent=vision_agent,
    images=["https://upload.wikimedia.org/wikipedia/commons/b/bf/Krakow_-_Kosciol_Mariacki.jpg"]
)

# Run the agents
agents = AgentTeam(
    agents=[vision_agent],
    tasks=[task],
    process="sequential",
    
)

agents.start()

Start Agents

Type this in your terminal to run your agents:

python app.py

Install Package

Install the PraisonAI package:

pip install praisonai opencv-python moviepy

Set API Key

Set your OpenAI API key as an environment variable in your terminal:

export OPENAI_API_KEY="${OPENAI_API_KEY:?Set OPENAI_API_KEY in your shell}"

Create a file

Create a new file agents.yaml with the basic setup:

framework: praisonai
process: sequential
topic: analyze landmark image
agents:  # Canonical: use 'agents' instead of 'roles'
  vision_analyst:
    name: VisionAnalyst
    role: Computer Vision Specialist
    goal: Analyze images and videos to extract meaningful information
    instructions:  # Canonical: use 'instructions' instead of 'backstory' |
      You are an expert in computer vision and image analysis.
      You excel at describing images, detecting objects, and understanding visual content.
    llm: gpt-4o-mini
    self_reflect: false
    tasks:
      analyze_landmark:
        description: Describe this famous landmark and its architectural features.
        expected_output: Detailed description of the landmark's architecture and significance
        images:
          - https://upload.wikimedia.org/wikipedia/commons/b/bf/Krakow_-_Kosciol_Mariacki.jpg

Start Agents

Type this in your terminal to run your agents:

praisonai agents.yaml

Requirements

Python 3.10 or higher
OpenAI API key with vision model access
Basic understanding of Python and media handling

One-shot CLI

Attach an image to a single praisonai run — the fastest way to ask about a screenshot, mockup, or diagram without writing any code.

praisonai run "Describe this screenshot" --image bug.png
praisonai run "Compare these two mockups" --image a.png --image b.png
praisonai run "What does this diagram show?" --image https://example.com/arch.png

Repeat --image for multiple images (never comma-separate one value). The run picks a vision-capable model, defaulting to gpt-4o; override with --model. See Run — Attach an image for the full option, guards, and behaviour.

Understanding Multimodal Agents

What are Multimodal Agents?

Multimodal agents are designed to:

Process multiple types of data (text, images, videos)
Understand context across different modalities
Generate insights from diverse media sources
Handle complex multimedia tasks

Features

Vision Processing

Analyze images, detect objects, and understand visual content.

Video Analysis

Process video content for events and actions.

Text Extraction

Extract and analyze text from images and documents.

Cross-Modal Understanding

Integrate insights across different media types.

Need the other direction — tools that hand images back to the agent? See Multimodal Tool Output.

Multi-Agent Media Processing

Code
No Code

from praisonaiagents import Agent, Task, AgentTeam

# Create first agent for image analysis
vision_agent = Agent(
    role="Image Analyst",
    goal="Analyze visual content and extract key information",
    backstory="Expert in visual analysis and image understanding",
    llm="gpt-4o-mini",
    reflection=False
)

# Create second agent for content writing
writer_agent = Agent(
    role="Content Writer",
    goal="Create engaging content based on image analysis",
    backstory="Expert in creating compelling content from visual insights",
    llm="gpt-4o-mini"
)

# Create tasks for different media types
document_task = Task(
    description="Extract and summarize text from this document image",
    expected_output="Structured text content with key information highlighted",
    agent=vision_agent,
    images=["document.jpg"]
)

writing_task = Task(
    description="Create engaging content based on image analysis",
    expected_output="Compelling article incorporating visual insights",
    agent=writer_agent
)

# Create and start the agents
agents = AgentTeam(
    agents=[vision_agent, writer_agent],
    tasks=[document_task, writing_task],
    process="sequential"
)

result = agents.start()

framework: praisonai
process: sequential
topic: document analysis and content creation
agents:  # Canonical
  vision_analyst:
    role: Image Analyst
    goal: Analyze visual content and extract key information
    instructions:  # Canonical: use 'instructions' instead of 'backstory' Expert in visual analysis and image understanding
    llm: gpt-4o-mini
    self_reflect: false
    tasks:
      document_task:
        description: Extract and summarize text from this document image
        expected_output: Structured text content with key information highlighted
        images:
          - document.jpg

  content_writer:
    role: Content Writer
    goal: Create engaging content based on image analysis
    instructions:  # Canonical: use 'instructions' instead of 'backstory' Expert in creating compelling content from visual insights
    llm: gpt-4o-mini
    tasks:
      writing_task:
        description: Create engaging content based on image analysis
        expected_output: Compelling article incorporating visual insights

Configuration Options

# Create an agent with multimodal configuration
agent = Agent(
    role="Media Analyst",
    goal="Process multiple types of media",
    backstory="Expert in multimedia analysis",
    llm="gpt-4o-mini",  # Must support vision capabilities
      # Enable detailed logging
    reflection=False  # Optional: disable self-reflection
)

# Task with media requirements
task = Task(
    description="Analyze media content",
    expected_output="Comprehensive analysis",
    agent=agent,
    images=[  # Support multiple media sources
        "https://example.com/image1.jpg",
        "path/to/local/image.jpg",
        "path/to/video.mp4"
    ]
)

Best Practices

Choose vision-capable models

Set llm to a model with vision support (e.g. gpt-4o-mini) when attaching images or video.

Write clear task descriptions

Break complex analyses into steps and specify expected output format in each task.

Prefer ephemeral attachments for large media

Use attachments= on agent.chat(), agent.start() (streaming or not), or ephemeral() so image bytes do not fill chat history.

Ephemeral Attachments

Send files to the agent for analysis without storing them in chat history. Essential for preventing context window overflow when processing multiple files. attachments= now accepts images, PDFs, audio, video, text files, URLs, data: URIs, and pathlib.Path objects — not only images. Each entry is routed by its type and the model’s capability.

The full attachment router — the raise-vs-warn split, the capability helpers, and the degradation labels — is documented in Attachments.

Attachments Parameter
Streaming
Ephemeral Context Manager

from praisonaiagents import Agent

agent = Agent(
    instructions="You analyze images and remember context",
    memory=True
)

# Image is analyzed but NOT stored in history
response = agent.chat(
    prompt="What's in this image?",     # ← Stored in history
    attachments=["photo.jpg"],           # ← NOT stored (ephemeral)
)

# Agent remembers the question, not the image data
response = agent.chat("What did I ask about earlier?")
# Agent: "You asked 'What's in this image?' and I told you..."

from praisonaiagents import Agent

agent = Agent(
    instructions="Describe images the user shares.",
    llm="gpt-4o",
)

for chunk in agent.start(
    "What is in this image?",
    stream=True,
    attachments=["photo.jpg"],   # ← reaches the vision model, not stored in history
):
    print(chunk, end="", flush=True)

agent.start(stream=True, attachments=[...]) honours attachments the same way agent.chat() does — this is the call the Desktop app makes for every dropped image.

from praisonaiagents import Agent

agent = Agent(instructions="Analyze images", memory=True)

# Pre-image conversation
agent.chat("Hello, I have some photos to show you")

# Ephemeral block - nothing stored permanently
with agent.ephemeral():
    response = agent.chat(
        "Analyze this",
        attachments=["image1.jpg"]
    )
    followup = agent.chat("What about the colors?")

# After block, history is restored - images NOT persisted
agent.chat("What have we discussed?")  # Only remembers pre-image chat

History Management Methods

Clean up chat history after image analysis sessions:

from praisonaiagents import Agent

agent = Agent(instructions="Image analyst", memory=True)

# After image analysis, clean up history
agent.prune_history(keep_last=5)           # Keep only last 5 messages
agent.delete_history(-1)                    # Delete last message
agent.delete_history_matching("[IMAGE]")   # Delete all image-related messages

# Check history size
print(f"History size: {agent.get_history_size()}")

Method	Description
`prune_history(keep_last=N)`	Keep only last N messages
`delete_history(index)`	Delete message by index
`delete_history_matching(pattern)`	Delete messages containing pattern
`get_history_size()`	Get current history length
`ephemeral()`	Context manager for temporary conversations

Use attachments= for one-time image analysis — on agent.chat() or agent.start(stream=True) — or ephemeral() for multi-turn image conversations that shouldn’t persist.

Example Use Cases

Document Analysis

Extract and analyze text from document images.

Security Monitoring

Monitor security feeds for suspicious activity.

Medical Imaging

Analyze medical scans for abnormalities.

Architectural Analysis

Study architectural features and designs.

Attachments

Attach PDFs, audio, text, video, URLs and Path objects — no silent drops.

Multimodal Tool Output

Return images and media from tools back to the agent.

Media Pipelines

Chain AudioAgent, ImageAgent, and VideoAgent in YAML workflows.

For optimal results, ensure your media files are in supported formats and sizes for processing.

​How It Works

​Quick Start

​One-shot CLI

​Understanding Multimodal Agents

What are Multimodal Agents?

​Features

Vision Processing

Video Analysis

Text Extraction

Cross-Modal Understanding

​Multi-Agent Media Processing

​Configuration Options

​Best Practices

​Ephemeral Attachments

​History Management Methods

​Example Use Cases

Document Analysis

Security Monitoring

Medical Imaging

Architectural Analysis

​Related

Attachments

Multimodal Tool Output

Media Pipelines

How It Works

Quick Start

One-shot CLI

Understanding Multimodal Agents

Features

Multi-Agent Media Processing

Configuration Options

Best Practices

Ephemeral Attachments

History Management Methods

Example Use Cases

Related