Partially Update Bucket

Schema definition for bucket objects.

IMPORTANT: The bucket schema defines what fields your bucket objects will have. This schema is REQUIRED if you want to:

1. Create collections that use input_mappings to process your bucket data
2. Validate object structure before ingestion
3. Enable type-safe data pipelines

The schema defines the custom fields that will be used in:

• Blob properties (e.g., "content", "thumbnail", "transcript")
• Object metadata structure
• Blob data structures

Example workflow:

1. Create bucket WITH schema defining your data structure
2. Upload objects that conform to that schema
3. Create collections that map schema fields to feature extractors

Without a bucket_schema, collections cannot use input_mappings.

Configuration for bucket unique key enforcement.

Enables automatic uniqueness enforcement on one or more fields from the bucket schema. Supports both single field and compound (multi-field) uniqueness constraints.

When configured, the bucket will maintain a lookup table mapping unique key values to document IDs, enabling efficient upsert operations and preventing duplicates.

Impact on Collection Trigger/Re-processing: When a collection is triggered (POST /collections/{id}/trigger), the unique_key determines whether documents are overwritten or duplicated:

• WITH unique_key: Documents get deterministic IDs → re-triggering OVERWRITES existing docs
• WITHOUT unique_key: Documents get random IDs → re-triggering CREATES DUPLICATES

For idempotent pipelines where re-triggering is safe, configure a unique_key.

Relationship to Extractor position_fields: The unique_key (bucket-level) and position_fields (extractor-level) work together to generate deterministic document IDs:

• unique_key: Identifies unique SOURCE OBJECTS in the bucket (e.g., video_id)
• position_fields: Identifies unique OUTPUT DOCUMENTS from a single object (e.g., start_time, end_time)

Document ID Formula: document_id = hash(source_object_key + extractor_id + collection_id + position_field_values)

Example - Processing a 60-second video with 10-second segments: - Bucket unique_key: ["video_id"] → Identifies the source video - Extractor position_fields: ["start_time", "end_time"] → Identifies each segment - Result: 6 unique document IDs (one per segment), all deterministic

Without position_fields, all segments would get the SAME document_id and overwrite each other. Without unique_key, reprocessing would create DUPLICATE documents instead of updating.

Requirements: - fields: REQUIRED - Array of field names from bucket schema to use as unique constraint - default_policy: OPTIONAL - Bucket-level default insertion policy (can be overridden per request) - All specified fields must exist in the bucket schema - All fields must be scalar types (string, integer, float, uuid) - Field values cannot be null or empty in uploaded objects - Cannot be changed after bucket creation (v1 limitation)

Use Cases: - Single field uniqueness: ["video_id"], ["product_sku"], ["user_email"] - Compound uniqueness: ["sensor_id", "timestamp"], ["product_id", "size", "color"] - With default policy: Enables idempotent ingestion without per-request policy - Without default: Requires explicit policy on each upload (safer, more intentional)

Insertion Policies: - 'insert': Fail with 409 Conflict if key exists (prevents accidental overwrites) - 'update': Fail with 404 Not Found if key doesn't exist (updates only) - 'upsert': Update if exists, insert if not (idempotent ingestion)

Policy Resolution (when uploading objects): 1. Use request-level ?policy= parameter if provided (highest priority) 2. Fall back to bucket-level default_policy if configured 3. Return 400 Bad Request if neither is specified (prevents accidental operations)

Examples: Single field with upsert default (idempotent video ingestion): { "fields": ["video_id"], "default_policy": "upsert" }

Single field with insert default (prevent duplicate products):
    {
        "fields": ["product_sku"],
        "default_policy": "insert"
    }

Compound fields with upsert (time-series sensor data):
    {
        "fields": ["sensor_id", "timestamp"],
        "default_policy": "upsert"
    }

Compound fields without default (explicit policy required):
    {
        "fields": ["user_id", "session_id"]
    }

Where a bucket's extraction outputs are written: the customer's own storage.

Names a storage connection with write access enabled, the same shape as a collection export destination. Outputs and derived assets (segments, thumbnails, transcodes, renders) are written under the prefix, each carrying its lineage as object metadata, and Mixpeek's storage keeps no copy.

A pinned saved retriever every candidate object runs through before commit.

The filter matches a candidate when the retriever returns at least one document for it. The candidate is offered to the retriever as a seed document (its mapped fields at the root, the source record under source) and as inputs (each declared input filled from the candidate field of the same name, and one media-typed input given the bytes as base64).

Schema definition for bucket objects.

IMPORTANT: The bucket schema defines what fields your bucket objects will have. This schema is REQUIRED if you want to:

1. Create collections that use input_mappings to process your bucket data
2. Validate object structure before ingestion
3. Enable type-safe data pipelines

The schema defines the custom fields that will be used in:

• Blob properties (e.g., "content", "thumbnail", "transcript")
• Object metadata structure
• Blob data structures

Example workflow:

1. Create bucket WITH schema defining your data structure
2. Upload objects that conform to that schema
3. Create collections that map schema fields to feature extractors

Without a bucket_schema, collections cannot use input_mappings.

Configuration for bucket unique key enforcement.

Enables automatic uniqueness enforcement on one or more fields from the bucket schema. Supports both single field and compound (multi-field) uniqueness constraints.

When configured, the bucket will maintain a lookup table mapping unique key values to document IDs, enabling efficient upsert operations and preventing duplicates.

Impact on Collection Trigger/Re-processing: When a collection is triggered (POST /collections/{id}/trigger), the unique_key determines whether documents are overwritten or duplicated:

• WITH unique_key: Documents get deterministic IDs → re-triggering OVERWRITES existing docs
• WITHOUT unique_key: Documents get random IDs → re-triggering CREATES DUPLICATES

For idempotent pipelines where re-triggering is safe, configure a unique_key.

Relationship to Extractor position_fields: The unique_key (bucket-level) and position_fields (extractor-level) work together to generate deterministic document IDs:

• unique_key: Identifies unique SOURCE OBJECTS in the bucket (e.g., video_id)
• position_fields: Identifies unique OUTPUT DOCUMENTS from a single object (e.g., start_time, end_time)

Document ID Formula: document_id = hash(source_object_key + extractor_id + collection_id + position_field_values)

Example - Processing a 60-second video with 10-second segments: - Bucket unique_key: ["video_id"] → Identifies the source video - Extractor position_fields: ["start_time", "end_time"] → Identifies each segment - Result: 6 unique document IDs (one per segment), all deterministic

Without position_fields, all segments would get the SAME document_id and overwrite each other. Without unique_key, reprocessing would create DUPLICATE documents instead of updating.

Requirements: - fields: REQUIRED - Array of field names from bucket schema to use as unique constraint - default_policy: OPTIONAL - Bucket-level default insertion policy (can be overridden per request) - All specified fields must exist in the bucket schema - All fields must be scalar types (string, integer, float, uuid) - Field values cannot be null or empty in uploaded objects - Cannot be changed after bucket creation (v1 limitation)

Use Cases: - Single field uniqueness: ["video_id"], ["product_sku"], ["user_email"] - Compound uniqueness: ["sensor_id", "timestamp"], ["product_id", "size", "color"] - With default policy: Enables idempotent ingestion without per-request policy - Without default: Requires explicit policy on each upload (safer, more intentional)

Insertion Policies: - 'insert': Fail with 409 Conflict if key exists (prevents accidental overwrites) - 'update': Fail with 404 Not Found if key doesn't exist (updates only) - 'upsert': Update if exists, insert if not (idempotent ingestion)

Policy Resolution (when uploading objects): 1. Use request-level ?policy= parameter if provided (highest priority) 2. Fall back to bucket-level default_policy if configured 3. Return 400 Bad Request if neither is specified (prevents accidental operations)

Examples: Single field with upsert default (idempotent video ingestion): { "fields": ["video_id"], "default_policy": "upsert" }

Single field with insert default (prevent duplicate products):
    {
        "fields": ["product_sku"],
        "default_policy": "insert"
    }

Compound fields with upsert (time-series sensor data):
    {
        "fields": ["sensor_id", "timestamp"],
        "default_policy": "upsert"
    }

Compound fields without default (explicit policy required):
    {
        "fields": ["user_id", "session_id"]
    }

Where a bucket's extraction outputs are written: the customer's own storage.

Names a storage connection with write access enabled, the same shape as a collection export destination. Outputs and derived assets (segments, thumbnails, transcodes, renders) are written under the prefix, each carrying its lineage as object metadata, and Mixpeek's storage keeps no copy.

A pinned saved retriever every candidate object runs through before commit.

The filter matches a candidate when the retriever returns at least one document for it. The candidate is offered to the retriever as a seed document (its mapped fields at the root, the source record under source) and as inputs (each declared input filled from the candidate field of the same name, and one media-typed input given the bytes as base64).

Statistics about batches in a bucket.

Statistics about object storage in a bucket.