Sign in
v1 modal

Query user memories with multimodal input

Query user's stored memories using multimodal inputs (video, image, or audio).

This endpoint accepts video, image, or audio content as base64-encoded strings and 
searches for relevant memories. The AI will:
1. Understand the content of the multimodal input
2. Search for related episodic memories
3. Return formatted results with context

**Entity Resolution:**
- user_id (str, required): Always required - the main user identifier
- persona_id (str, optional): If provided, query uses persona's context instead of user
- project_id (str, optional): If provided, query uses project's context (inherits from user)

Priority: persona_id > project_id > user_id

**Request Parameters:**
- video_base64 (str, optional): Base64 encoded video content
- image_base64 (str, optional): Base64 encoded image content
- audio_base64 (str, optional): Base64 encoded audio content (supports webm, wav, mp3, mp4, and other formats)
- session_id (str, optional): Session identifier for conversation context

**Note:** At least one multimodal input (video, image, or audio) is required.
Audio will be automatically converted to WAV format for processing.

**Response:**
- new_prompt (str): Formatted string containing retrieved memories
- raw_results (dict): Raw results from the memory retrieval
- image_base64 (str, optional): Base64 encoded image - the original image or a representative frame from video
- success (bool): True if query succeeded

**Example:**
```json
{
    "user_id": "user-123",
    "persona_id": null,
    "project_id": null,
    "video_base64": "base64_encoded_video..."
}
```

Returns 200 OK with memory data. Requires JWT authentication.
post/v1/modal/multimodal-query

Request body

user_idstring required

Unique identifier for the user (always required)

persona_idstring nullable

Optional persona ID. If provided, query is scoped to this persona instead of the user

project_idstring nullable

Optional project ID. If provided, query is scoped to this project (inherits from user)

video_base64string nullable

Base64 encoded video content

image_base64string nullable

Base64 encoded image content

audio_base64string nullable

Base64 encoded audio content (supports webm, wav, mp3, mp4, and other formats)

session_idstring nullable

Optional session identifier for conversation context

Example request

{
  "limit": 10,
  "session_id": "session_123",
  "text_query": "Show me memories related to this video",
  "user_id": "123e4567-e89b-12d3-a456-426614174000",
  "video_base64": "base64_encoded_video_content..."
}

Response

Successful Response

new_promptstring required

Formatted string containing retrieved memories

raw_resultsobject

Raw results from the memory retrieval

image_base64string nullable

Base64 encoded image - either the original image or a representative frame from video

entity_imagesobject nullable

Reference images for matched entities (entity_name -> base64 image)

successboolean

Whether the query was processed successfully

Example response

{
  "image_base64": "iVBORw0KGgoAAAANSUhEUgAAAAUA...",
  "new_prompt": "I want to cook pasta",
  "raw_results": {
    "episodes": [
      {
        "content": "You cooked pasta last week",
        "episode_id": "ep_123",
        "timestamp": "2024-01-15T18:30:00Z"
      }
    ],
    "identity": [
      {
        "content": "I am a person",
        "identity_id": "identity_123",
        "timestamp": "2024-01-15T18:30:00Z"
      }
    ],
    "preferences": [
      {
        "content": "I prefer cooking pasta",
        "preference_id": "pref_123",
        "timestamp": "2024-01-15T18:30:00Z"
      }
    ]
  },
  "success": true
}

Changes