docs/open-source/features/multimodal-support.mdx
Multimodal support lets Mem0 extract facts from images alongside regular text. Add screenshots, receipts, or product photos and Mem0 will store the insights as searchable memories so agents can recall them later.
<Info> **You’ll use this when…** - Users share screenshots, menus, or documents and you want the details to become memories. - You already collect text conversations but need visual context for better answers. - You want a single workflow that handles both URLs and local image files. </Info> <Warning> Mem0 passes images straight to your configured vision model, so per-image size and resolution limits come from that provider (for example, OpenAI caps images at 20 MB). Compress or resize large files to stay within your provider's limits and keep processing fast. </Warning>client = Memory()
messages = [ {"role": "user", "content": "Hi, my name is Alice."}, { "role": "user", "content": { "type": "image_url", "image_url": { "url": "https://example.com/menu.jpg" } } } ]
client.add(messages, user_id="alice")
```ts TypeScript
import { Memory } from "mem0ai/oss";
const client = new Memory();
const messages = [
{ role: "user", content: "Hi, my name is Alice." },
{
role: "user",
content: {
type: "image_url",
image_url: { url: "https://example.com/menu.jpg" }
}
}
];
await client.add(messages, { userId: "alice" });
def encode_image(image_path): with open(image_path, "rb") as image_file: return base64.b64encode(image_file.read()).decode("utf-8")
client = Memory() base64_image = encode_image("path/to/your/image.jpg")
messages = [ { "role": "user", "content": [ {"type": "text", "text": "What's in this image?"}, { "type": "image_url", "image_url": { "url": f"data:image/jpeg;base64,{base64_image}" } } ] } ]
client.add(messages, user_id="alice")
```ts TypeScript
import fs from "fs";
import { Memory } from "mem0ai/oss";
function encodeImage(imagePath: string) {
const buffer = fs.readFileSync(imagePath);
return buffer.toString("base64");
}
const client = new Memory();
const base64Image = encodeImage("path/to/your/image.jpg");
const messages = [
{
role: "user",
content: [
{ type: "text", text: "What's in this image?" },
{
type: "image_url",
image_url: {
url: `data:image/jpeg;base64,${base64Image}`
}
}
]
}
];
await client.add(messages, { userId: "alice" });
from mem0 import Memory
client = Memory()
messages = [
{
"role": "user",
"content": "Help me remember which dishes I liked."
},
{
"role": "user",
"content": {
"type": "image_url",
"image_url": {
"url": "https://example.com/restaurant-menu.jpg"
}
}
},
{
"role": "user",
"content": "I’m allergic to peanuts and prefer vegetarian meals."
}
]
result = client.add(messages, user_id="user123")
print(result)
messages = [
{
"role": "user",
"content": "Store this receipt information for expenses."
},
{
"role": "user",
"content": {
"type": "image_url",
"image_url": {
"url": "https://example.com/receipt.jpg"
}
}
}
]
client.add(messages, user_id="user123")
client = Memory()
try: messages = [{ "role": "user", "content": { "type": "image_url", "image_url": {"url": "https://example.com/image.jpg"} } }]
client.add(messages, user_id="user123")
print("Image processed successfully")
except ValueError as exc: # Raised when an image_url part is missing its url print(f"Invalid image message: {exc}") except Exception as exc: # Raised when the image can't be downloaded or the vision model call fails print(f"Could not process image: {exc}")
```ts TypeScript
import { Memory } from "mem0ai/oss";
const client = new Memory();
try {
const messages = [{
role: "user",
content: {
type: "image_url",
image_url: { url: "https://example.com/image.jpg" }
}
}];
await client.add(messages, { userId: "user123" });
console.log("Image processed successfully");
} catch (error: any) {
// A missing image_url, a failed download, or a vision model error surfaces here
console.log(`Could not process image: ${error.message}`);
}
add, inspect the returned memories and confirm they include image-derived text (menu items, receipt totals, etc.).search for a detail from the image; the memory should surface alongside related text.add calls to isolate failures and improve reliability.| Issue | Cause | Fix |
|---|---|---|
| Upload rejected | Image exceeds your vision provider's size limit | Compress or resize before sending. |
| Memory missing image data | Low-quality or blurry image | Retake the photo with better lighting. |
| Invalid format error | Unsupported file type | Convert to JPEG or PNG first. |
| Slow processing | High-resolution images | Downscale or compress to under 5 MB. |
| Base64 errors | Incorrect prefix or encoding | Ensure data:image/<type>;base64, is present and the string is valid. |