Multimodal Support

Multimodal support lets Mem0 extract facts from images alongside regular text. Add screenshots, receipts, or product photos and Mem0 will store the insights as searchable memories so agents can recall them later.

Feature anatomy

• Vision processing: Mem0 runs the image through a vision model that extracts text and key details.
• Memory creation: Extracted information is stored as standard memories so search, filters, and analytics continue to work.
• Context linking: Visual and textual turns in the same conversation stay linked, giving agents richer context.
• Flexible inputs: Accept publicly accessible URLs or base64-encoded local files in both Python and JavaScript SDKs.

Configure it

Add image messages from URLs

client = Memory()

messages = [ {"role": "user", "content": "Hi, my name is Alice."}, { "role": "user", "content": { "type": "image_url", "image_url": { "url": "https://example.com/menu.jpg" } } } ]

client.add(messages, user_id="alice")

```ts TypeScript
import { Memory } from "mem0ai";

const client = new Memory();

const messages = [
  { role: "user", content: "Hi, my name is Alice." },
  {
    role: "user",
    content: {
      type: "image_url",
      image_url: { url: "https://example.com/menu.jpg" }
    }
  }
];

await client.add(messages, { userId: "alice" });

Upload local images as base64

def encode_image(image_path): with open(image_path, "rb") as image_file: return base64.b64encode(image_file.read()).decode("utf-8")

client = Memory() base64_image = encode_image("path/to/your/image.jpg")

messages = [ { "role": "user", "content": [ {"type": "text", "text": "What's in this image?"}, { "type": "image_url", "image_url": { "url": f"data:image/jpeg;base64,{base64_image}" } } ] } ]

client.add(messages, user_id="alice")

```ts TypeScript
import fs from "fs";
import { Memory } from "mem0ai";

function encodeImage(imagePath: string) {
  const buffer = fs.readFileSync(imagePath);
  return buffer.toString("base64");
}

const client = new Memory();
const base64Image = encodeImage("path/to/your/image.jpg");

const messages = [
  {
    role: "user",
    content: [
      { type: "text", text: "What's in this image?" },
      {
        type: "image_url",
        image_url: {
          url: `data:image/jpeg;base64,${base64Image}`
        }
      }
    ]
  }
];

await client.add(messages, { userId: "alice" });

See it in action

Restaurant menu memory

from mem0 import Memory

client = Memory()

messages = [
    {
        "role": "user",
        "content": "Help me remember which dishes I liked."
    },
    {
        "role": "user",
        "content": {
            "type": "image_url",
            "image_url": {
                "url": "https://example.com/restaurant-menu.jpg"
            }
        }
    },
    {
        "role": "user",
        "content": "I’m allergic to peanuts and prefer vegetarian meals."
    }
]

result = client.add(messages, user_id="user123")
print(result)

Document capture

messages = [
    {
        "role": "user",
        "content": "Store this receipt information for expenses."
    },
    {
        "role": "user",
        "content": {
            "type": "image_url",
            "image_url": {
                "url": "https://example.com/receipt.jpg"
            }
        }
    }
]

client.add(messages, user_id="user123")

Error handling

client = Memory()

try: messages = [{ "role": "user", "content": { "type": "image_url", "image_url": {"url": "https://example.com/image.jpg"} } }]

client.add(messages, user_id="user123")
print("Image processed successfully")

except InvalidImageError: print("Invalid image format or corrupted file") except FileSizeError: print("Image file too large") except Exception as exc: print(f"Unexpected error: {exc}")

```ts TypeScript
import { Memory } from "mem0ai";

const client = new Memory();

try {
  const messages = [{
    role: "user",
    content: {
      type: "image_url",
      image_url: { url: "https://example.com/image.jpg" }
    }
  }];

  await client.add(messages, { userId: "user123" });
  console.log("Image processed successfully");
} catch (error: any) {
  if (error.type === "invalid_image") {
    console.log("Invalid image format or corrupted file");
  } else if (error.type === "file_size_exceeded") {
    console.log("Image file too large");
  } else {
    console.log(`Unexpected error: ${error.message}`);
  }
}

Verify the feature is working

• After calling add, inspect the returned memories and confirm they include image-derived text (menu items, receipt totals, etc.).
• Run a follow-up search for a detail from the image; the memory should surface alongside related text.
• Monitor image upload latency—large files should still complete under your acceptable response time.
• Log file size and URL sources to troubleshoot repeated failures.

Best practices

1. Ask for intent: Prompt users to explain why they sent an image so the memory includes the right context.
2. Keep images readable: Encourage clear photos without heavy filters or shadows for better extraction.
3. Split bulk uploads: Send multiple images as separate add calls to isolate failures and improve reliability.
4. Watch privacy: Avoid uploading sensitive documents unless your environment is secured for that data.
5. Validate file size early: Check file size before encoding to save bandwidth and time.

Troubleshooting