Multimodal Agents

Guide for creating and using multimodal AI agents in PraisonAI for processing images, videos, and other media types

Multimodal agents process text, images, and video in a single workflow — attach media to tasks or pass ephemeral attachments at chat time.

python

from praisonaiagents import Agent

agent = Agent(
    name="VisionAnalyst",
    instructions="Analyse images and video and describe what you see.",
    llm="gpt-4o-mini",
)
agent.start("Describe the landmark in this image.")

The user attaches media; the agent returns a single analysis covering text and visuals.

Diagram

How It Works

Diagram

Quick Start

Code

Install Package
First, install the PraisonAI Agents package:
bash
```
pip install praisonaiagents opencv-python moviepy
```
Set API Key
Set your OpenAI API key as an environment variable in your terminal:
bash
```
export OPENAI_API_KEY="${OPENAI_API_KEY:?Set OPENAI_API_KEY in your shell}"
```

Create a file

Create a new file app.py with the basic setup:

python

from praisonaiagents import Agent, Task, AgentTeam

# Create Vision Analysis Agent
vision_agent = Agent(
    name="VisionAnalyst",
    role="Computer Vision Specialist",
    goal="Analyze images and videos to extract meaningful information",
    backstory="""You are an expert in computer vision and image analysis.
    You excel at describing images, detecting objects, and understanding visual content.""",
    llm="gpt-4o-mini",
    reflection=False
)

# Create tasks with different media types
task = Task(
    name="analyze_landmark",
    description="Describe this famous landmark and its architectural features.",
    expected_output="Detailed description of the landmark's architecture and significance",
    agent=vision_agent,
    images=["https://upload.wikimedia.org/wikipedia/commons/b/bf/Krakow_-_Kosciol_Mariacki.jpg"]
)

# Run the agents
agents = AgentTeam(
    agents=[vision_agent],
    tasks=[task],
    process="sequential",

)

agents.start()

Start Agents
Type this in your terminal to run your agents:
bash
```
python app.py
```

No Code

Install Package
Install the PraisonAI package:
bash
```
pip install praisonai opencv-python moviepy
```
Set API Key
Set your OpenAI API key as an environment variable in your terminal:
bash
```
export OPENAI_API_KEY="${OPENAI_API_KEY:?Set OPENAI_API_KEY in your shell}"
```

Create a file

Create a new file agents.yaml with the basic setup:

yaml

framework: praisonai
process: sequential
topic: analyze landmark image
agents:  # Canonical: use 'agents' instead of 'roles'
  vision_analyst:
    name: VisionAnalyst
    role: Computer Vision Specialist
    goal: Analyze images and videos to extract meaningful information
    instructions:  # Canonical: use 'instructions' instead of 'backstory' |
      You are an expert in computer vision and image analysis.
      You excel at describing images, detecting objects, and understanding visual content.
    llm: gpt-4o-mini
    self_reflect: false
    tasks:
      analyze_landmark:
        description: Describe this famous landmark and its architectural features.
        expected_output: Detailed description of the landmark's architecture and significance
        images:
          - https://upload.wikimedia.org/wikipedia/commons/b/bf/Krakow_-_Kosciol_Mariacki.jpg

Start Agents
Type this in your terminal to run your agents:
bash
```
praisonai agents.yaml
```

One-shot CLI

Attach an image to a single praisonai run — the fastest way to ask about a screenshot, mockup, or diagram without writing any code.

bash

praisonai run "Describe this screenshot" --image bug.png
praisonai run "Compare these two mockups" --image a.png --image b.png
praisonai run "What does this diagram show?" --image https://example.com/arch.png

Repeat --image for multiple images (never comma-separate one value). The run picks a vision-capable model, defaulting to gpt-4o; override with --model.

See Run — Attach an image for the full option, guards, and behaviour.

Understanding Multimodal Agents

What are Multimodal Agents?

Multimodal agents are designed to: – Process multiple types of data (text, images, videos) – Understand context across different modalities – Generate insights from diverse media sources – Handle complex multimedia tasks

Features

Vision Processing

Analyze images, detect objects, and understand visual content.

Video Analysis

Process video content for events and actions.

Text Extraction

Extract and analyze text from images and documents.

Cross-Modal Understanding

Integrate insights across different media types.

Multi-Agent Media Processing

Code

python

from praisonaiagents import Agent, Task, AgentTeam

# Create first agent for image analysis
vision_agent = Agent(
    role="Image Analyst",
    goal="Analyze visual content and extract key information",
    backstory="Expert in visual analysis and image understanding",
    llm="gpt-4o-mini",
    reflection=False
)

# Create second agent for content writing
writer_agent = Agent(
    role="Content Writer",
    goal="Create engaging content based on image analysis",
    backstory="Expert in creating compelling content from visual insights",
    llm="gpt-4o-mini"
)

# Create tasks for different media types
document_task = Task(
    description="Extract and summarize text from this document image",
    expected_output="Structured text content with key information highlighted",
    agent=vision_agent,
    images=["document.jpg"]
)

writing_task = Task(
    description="Create engaging content based on image analysis",
    expected_output="Compelling article incorporating visual insights",
    agent=writer_agent
)

# Create and start the agents
agents = AgentTeam(
    agents=[vision_agent, writer_agent],
    tasks=[document_task, writing_task],
    process="sequential"
)

result = agents.start()

No Code

yaml

framework: praisonai
process: sequential
topic: document analysis and content creation
agents:  # Canonical
  vision_analyst:
    role: Image Analyst
    goal: Analyze visual content and extract key information
    instructions:  # Canonical: use 'instructions' instead of 'backstory' Expert in visual analysis and image understanding
    llm: gpt-4o-mini
    self_reflect: false
    tasks:
      document_task:
        description: Extract and summarize text from this document image
        expected_output: Structured text content with key information highlighted
        images:
          - document.jpg

  content_writer:
    role: Content Writer
    goal: Create engaging content based on image analysis
    instructions:  # Canonical: use 'instructions' instead of 'backstory' Expert in creating compelling content from visual insights
    llm: gpt-4o-mini
    tasks:
      writing_task:
        description: Create engaging content based on image analysis
        expected_output: Compelling article incorporating visual insights

Configuration Options

python

# Create an agent with multimodal configuration
agent = Agent(
    role="Media Analyst",
    goal="Process multiple types of media",
    backstory="Expert in multimedia analysis",
    llm="gpt-4o-mini",  # Must support vision capabilities
      # Enable detailed logging
    reflection=False  # Optional: disable self-reflection
)

# Task with media requirements
task = Task(
    description="Analyze media content",
    expected_output="Comprehensive analysis",
    agent=agent,
    images=[  # Support multiple media sources
        "https://example.com/image1.jpg",
        "path/to/local/image.jpg",
        "path/to/video.mp4"
    ]
)

Best Practices

Choose vision-capable models

Set llm to a model with vision support (e.g. gpt-4o-mini) when attaching images or video.

Write clear task descriptions

Break complex analyses into steps and specify expected output format in each task.

Prefer ephemeral attachments for large media

Use attachments= on agent.chat(), agent.start() (streaming or not), or ephemeral() so image bytes do not fill chat history.

Ephemeral Attachments

Send files to the agent for analysis without storing them in chat history. Essential for preventing context window overflow when processing multiple files.

attachments= now accepts images, PDFs, audio, video, text files, URLs, data: URIs, and pathlib.Path objects — not only images. Each entry is routed by its type and the model’s capability.

Attachments Parameter

python

from praisonaiagents import Agent

agent = Agent(
    instructions="You analyze images and remember context",
    memory=True
)

# Image is analyzed but NOT stored in history
response = agent.chat(
    prompt="What's in this image?",     # ← Stored in history
    attachments=["photo.jpg"],           # ← NOT stored (ephemeral)
)

# Agent remembers the question, not the image data
response = agent.chat("What did I ask about earlier?")
# Agent: "You asked 'What's in this image?' and I told you..."

Streaming

python

from praisonaiagents import Agent

agent = Agent(
    instructions="Describe images the user shares.",
    llm="gpt-4o",
)

for chunk in agent.start(
    "What is in this image?",
    stream=True,
    attachments=["photo.jpg"],   # ← reaches the vision model, not stored in history
):
    print(chunk, end="", flush=True)

agent.start(stream=True, attachments=[...]) honours attachments the same way agent.chat() does — this is the call the Desktop app makes for every dropped image.

Ephemeral Context Manager

python

from praisonaiagents import Agent

agent = Agent(instructions="Analyze images", memory=True)

# Pre-image conversation
agent.chat("Hello, I have some photos to show you")

# Ephemeral block - nothing stored permanently
with agent.ephemeral():
    response = agent.chat(
        "Analyze this",
        attachments=["image1.jpg"]
    )
    followup = agent.chat("What about the colors?")

# After block, history is restored - images NOT persisted
agent.chat("What have we discussed?")  # Only remembers pre-image chat

History Management Methods

Clean up chat history after image analysis sessions:

python

from praisonaiagents import Agent

agent = Agent(instructions="Image analyst", memory=True)

# After image analysis, clean up history
agent.prune_history(keep_last=5)           # Keep only last 5 messages
agent.delete_history(-1)                    # Delete last message
agent.delete_history_matching("[IMAGE]")   # Delete all image-related messages

# Check history size
print(f"History size: {agent.get_history_size()}")

Method	Description
`prune_history(keep_last=N)`	Keep only last N messages
`delete_history(index)`	Delete message by index
`delete_history_matching(pattern)`	Delete messages containing pattern
`get_history_size()`	Get current history length
`ephemeral()`	Context manager for temporary conversations