Create Collection

Configuration for collection source (bucket(s) or collection).

Collections can process data from two types of sources:

1. Bucket Source: Process raw objects from one or more buckets (first-stage processing)

   • Use this to create your initial collections from uploaded data
   • Can specify multiple buckets to consolidate data from different sources
   • All buckets must have compatible schemas (validated at creation)
   • Example: Videos from multiple regions → Frame extraction collection

2. Collection Source: Process documents from another collection (decomposition trees)

   • Use this to create multi-stage processing pipelines
   • Example: Frames collection → Scene detection collection

Multi-Bucket Requirements:

• All buckets must have compatible schemas (same fields, types, and required status)
• Schema compatibility is validated when the collection is created
• Documents track which specific bucket they came from via root_bucket_id
• Useful for consolidating data from multiple regions, teams, or environments

The source determines:

• What data the feature extractor receives as input
• The input_schema available for input_mappings and field_passthrough
• The lineage tracking in output documents

Examples: Single bucket: {"type": "bucket", "bucket_ids": ["bkt_products"]} Multi-bucket: {"type": "bucket", "bucket_ids": ["bkt_us", "bkt_eu", "bkt_asia"]} Collection: {"type": "collection", "collection_id": "col_frames"}

Schema definition for bucket objects.

IMPORTANT: The bucket schema defines what fields your bucket objects will have. This schema is REQUIRED if you want to:

1. Create collections that use input_mappings to process your bucket data
2. Validate object structure before ingestion
3. Enable type-safe data pipelines

The schema defines the custom fields that will be used in:

• Blob properties (e.g., "content", "thumbnail", "transcript")
• Object metadata structure
• Blob data structures

Example workflow:

1. Create bucket WITH schema defining your data structure
2. Upload objects that conform to that schema
3. Create collections that map schema fields to feature extractors

Without a bucket_schema, collections cannot use input_mappings.

Configuration for a feature extractor with field passthrough support.

A feature extractor processes source data (from buckets or collections) and produces features (embeddings, extracted text, detected objects, etc.).

With field passthrough, you can also include selected source fields in the output documents alongside the computed features.

Core Concepts: 1. Feature Extraction: Extractors compute features from input data (e.g., text → embeddings, image → detections, video → scenes) 2. Field Passthrough: Selectively preserve source fields in output (e.g., title, category, campaign_id from source → output documents) 3. Output Schema: Combination of passed-through fields + extractor outputs (e.g., {title, category, text_embedding} all in one document)

How Field Passthrough Works: 1. Define which source fields to include via field_passthrough list 2. During processing, these fields are extracted from source 3. They appear in output documents at root level 4. Combine with extractor outputs for complete documents 5. Use target_path to rename fields for cleaner schemas

Field Selection Modes: - Explicit (field_passthrough + include_all=False): Only listed fields pass through. Clean, controlled output. Example: passthrough=[title, category] → output has ONLY title, category, embedding

- **Inclusive** (include_all=True):
  All source fields pass through, field_passthrough for renaming.
  Example: source has 10 fields → output has all 10 + embedding

- **None** (no field_passthrough):
  Only extractor outputs in documents.
  Example: → output has ONLY embedding (no source fields)

Use Cases: - Preserve Identifiers: Keep campaign_id, product_sku, order_id for tracking - Enable Filtering: Pass category, status, department for query filters - Maintain Context: Include title, description for display - Track Metadata: Preserve author, created_at, source for lineage - Business Logic: Keep priority, region, type for application logic

Common Patterns: 1. Minimal Passthrough (recommended): field_passthrough=[{"source_path": "id"}], include_all=False → Clean output, only ID + extractor features

2. **Metadata Preservation**:
   field_passthrough=[
       {"source_path": "title"},
       {"source_path": "category"},
       {"source_path": "created_at"}
   ]
   → Document has context for display and filtering

3. **Field Renaming**:
   field_passthrough=[
       {"source_path": "doc_title", "target_path": "title"},
       {"source_path": "metadata.author", "target_path": "author"}
   ]
   → Cleaner output schema with flattened fields

4. **Required Fields**:
   field_passthrough=[
       {"source_path": "campaign_id", "required": True},
       {"source_path": "priority", "default": 0}
   ]
   → Ensures critical fields always present

Requirements: - feature_extractor_name: REQUIRED - name of the extractor - version: REQUIRED - extractor version (e.g., "v1") - parameters: NOT REQUIRED - extractor-specific config (model, thresholds, etc.) - input_mappings: NOT REQUIRED - maps extractor inputs to source fields - field_passthrough: NOT REQUIRED - which source fields to preserve (default: none) - include_all_source_fields: NOT REQUIRED - preserve all fields (default: false)

Schedule configuration for automatic collection re-processing.

Attaches a cron or interval schedule to a collection, which creates a COLLECTION_TRIGGER trigger behind the scenes. This is a DX convenience so users don't need to create triggers manually.

Examples: Daily re-crawl at 2am UTC: {"trigger_type": "cron", "schedule_config": {"cron_expression": "0 2 * * *"}}

Every 6 hours:
    {"trigger_type": "interval", "schedule_config": {"interval_seconds": 21600}}

Optional taxonomy applications to automatically enrich documents in this collection. Each taxonomy will classify/enrich documents based on configured retriever matches.

Optional cluster applications to automatically execute when batch processing completes. Each cluster enriches documents with cluster assignments (cluster_id, cluster_label, etc.).

Optional alert applications to automatically execute when documents are ingested. Each alert runs a retriever against new documents and sends notifications if matches are found. Supports both ON_INGEST (triggered per batch) and SCHEDULED (periodic) execution modes.

Optional retriever enrichments to run on documents during post-processing. Each enrichment executes a retriever pipeline and writes selected result fields back to the document. Use for: LLM classification, cross-collection joins, multi-stage enrichment at ingestion time.

Whole-collection transform: run a retriever pipeline over this collection's source and write its output as the collection's documents.

Name a saved retriever with retriever_id, and pin its version with retriever_version: the collection keeps running that version after the retriever changes, until a PATCH moves the pin. Inline stages still work and are deprecated. A collection is either extractor-based or transform-based. The source may be a collection, or a bucket, in which case the transform reads the bucket's objects through its system collection. Re-runs update documents in place: a document's id comes from unique_id_field, the pipeline's group_by field, or the source document's id. write_back_fields (optional) picks the output fields; without it the pipeline's output defines them.

Schema definition for bucket objects.

IMPORTANT: The bucket schema defines what fields your bucket objects will have. This schema is REQUIRED if you want to:

1. Create collections that use input_mappings to process your bucket data
2. Validate object structure before ingestion
3. Enable type-safe data pipelines

The schema defines the custom fields that will be used in:

• Blob properties (e.g., "content", "thumbnail", "transcript")
• Object metadata structure
• Blob data structures

Example workflow:

1. Create bucket WITH schema defining your data structure
2. Upload objects that conform to that schema
3. Create collections that map schema fields to feature extractors

Without a bucket_schema, collections cannot use input_mappings.

Schema definition for bucket objects.

IMPORTANT: The bucket schema defines what fields your bucket objects will have. This schema is REQUIRED if you want to:

1. Create collections that use input_mappings to process your bucket data
2. Validate object structure before ingestion
3. Enable type-safe data pipelines

The schema defines the custom fields that will be used in:

• Blob properties (e.g., "content", "thumbnail", "transcript")
• Object metadata structure
• Blob data structures

Example workflow:

1. Create bucket WITH schema defining your data structure
2. Upload objects that conform to that schema
3. Create collections that map schema fields to feature extractors

Without a bucket_schema, collections cannot use input_mappings.

Configuration for a feature extractor with field passthrough support.

A feature extractor processes source data (from buckets or collections) and produces features (embeddings, extracted text, detected objects, etc.).

With field passthrough, you can also include selected source fields in the output documents alongside the computed features.

Core Concepts: 1. Feature Extraction: Extractors compute features from input data (e.g., text → embeddings, image → detections, video → scenes) 2. Field Passthrough: Selectively preserve source fields in output (e.g., title, category, campaign_id from source → output documents) 3. Output Schema: Combination of passed-through fields + extractor outputs (e.g., {title, category, text_embedding} all in one document)

How Field Passthrough Works: 1. Define which source fields to include via field_passthrough list 2. During processing, these fields are extracted from source 3. They appear in output documents at root level 4. Combine with extractor outputs for complete documents 5. Use target_path to rename fields for cleaner schemas

Field Selection Modes: - Explicit (field_passthrough + include_all=False): Only listed fields pass through. Clean, controlled output. Example: passthrough=[title, category] → output has ONLY title, category, embedding

- **Inclusive** (include_all=True):
  All source fields pass through, field_passthrough for renaming.
  Example: source has 10 fields → output has all 10 + embedding

- **None** (no field_passthrough):
  Only extractor outputs in documents.
  Example: → output has ONLY embedding (no source fields)

Use Cases: - Preserve Identifiers: Keep campaign_id, product_sku, order_id for tracking - Enable Filtering: Pass category, status, department for query filters - Maintain Context: Include title, description for display - Track Metadata: Preserve author, created_at, source for lineage - Business Logic: Keep priority, region, type for application logic

Common Patterns: 1. Minimal Passthrough (recommended): field_passthrough=[{"source_path": "id"}], include_all=False → Clean output, only ID + extractor features

2. **Metadata Preservation**:
   field_passthrough=[
       {"source_path": "title"},
       {"source_path": "category"},
       {"source_path": "created_at"}
   ]
   → Document has context for display and filtering

3. **Field Renaming**:
   field_passthrough=[
       {"source_path": "doc_title", "target_path": "title"},
       {"source_path": "metadata.author", "target_path": "author"}
   ]
   → Cleaner output schema with flattened fields

4. **Required Fields**:
   field_passthrough=[
       {"source_path": "campaign_id", "required": True},
       {"source_path": "priority", "default": 0}
   ]
   → Ensures critical fields always present

Requirements: - feature_extractor_name: REQUIRED - name of the extractor - version: REQUIRED - extractor version (e.g., "v1") - parameters: NOT REQUIRED - extractor-specific config (model, thresholds, etc.) - input_mappings: NOT REQUIRED - maps extractor inputs to source fields - field_passthrough: NOT REQUIRED - which source fields to preserve (default: none) - include_all_source_fields: NOT REQUIRED - preserve all fields (default: false)

Configuration for collection source (bucket(s) or collection).

Collections can process data from two types of sources:

1. Bucket Source: Process raw objects from one or more buckets (first-stage processing)

   • Use this to create your initial collections from uploaded data
   • Can specify multiple buckets to consolidate data from different sources
   • All buckets must have compatible schemas (validated at creation)
   • Example: Videos from multiple regions → Frame extraction collection

2. Collection Source: Process documents from another collection (decomposition trees)

   • Use this to create multi-stage processing pipelines
   • Example: Frames collection → Scene detection collection

Multi-Bucket Requirements:

• All buckets must have compatible schemas (same fields, types, and required status)
• Schema compatibility is validated when the collection is created
• Documents track which specific bucket they came from via root_bucket_id
• Useful for consolidating data from multiple regions, teams, or environments

The source determines:

• What data the feature extractor receives as input
• The input_schema available for input_mappings and field_passthrough
• The lineage tracking in output documents

Examples: Single bucket: {"type": "bucket", "bucket_ids": ["bkt_products"]} Multi-bucket: {"type": "bucket", "bucket_ids": ["bkt_us", "bkt_eu", "bkt_asia"]} Collection: {"type": "collection", "collection_id": "col_frames"}

NOT REQUIRED (auto-computed). Lineage chain showing complete processing history. Each entry contains: source_config, feature_extractor, output_schema for one tier. Length indicates processing depth (1 = tier 1, 2 = tier 2, etc.). Use for: Understanding multi-tier pipelines, visualizing decomposition trees.

NOT REQUIRED (auto-computed). Vector indexes for semantic search. Populated from feature_extractor.required_vector_indexes on extractor-backed collections, and from the namespace vector config on promoted or direct-upsert (BYO) collections, where each entry carries source='namespace_vector_config'. Defines which embeddings are indexed, their dimensions and distance metric. Use for understanding search capabilities and debugging vector queries. An entry's source distinguishes a namespace-declared index from an extractor-derived one; extractor-derived entries omit it. The nested index object carries the storage name, dimensions, distance and inference_name.

NOT REQUIRED (auto-computed from extractor + namespace). Payload indexes for filtering. Enables efficient filtering on metadata fields, timestamps, IDs. Populated from: extractor requirements + namespace defaults. Use for: Understanding which fields support fast filtering.

Why the platform disabled a collection, when it did.

Schedule configuration for automatic collection re-processing.

Attaches a cron or interval schedule to a collection, which creates a COLLECTION_TRIGGER trigger behind the scenes. This is a DX convenience so users don't need to create triggers manually.

Examples: Daily re-crawl at 2am UTC: {"trigger_type": "cron", "schedule_config": {"cron_expression": "0 2 * * *"}}

Every 6 hours:
    {"trigger_type": "interval", "schedule_config": {"interval_seconds": 21600}}

Retired: stored and never enforced. Writes are refused; set tiering instead. Kept on reads so existing records still parse.

A namespace's or collection's override of its data class's tiering.

NOT REQUIRED. List of taxonomies to apply to documents in this collection. Each entry specifies: taxonomy_id, optional target_collection_id, optional filters. Enrichments are materialized (persisted to documents) during ingestion. Empty/null if no taxonomies attached. Use for: Categorization, hierarchical classification.

NOT REQUIRED. List of clusters to automatically execute when batch processing completes. Each entry specifies: cluster_id, auto_execute_on_batch, min_document_threshold, cooldown_seconds. Clusters enrich source documents with cluster assignments (cluster_id, cluster_label, etc.). Empty/null if no clusters attached. Use for: Segmentation, grouping, pattern discovery.

NOT REQUIRED. List of alerts to automatically execute when documents are ingested. Each entry specifies: alert_id, execution_mode, input_mappings, execution_phase, priority. Alerts run retrievers on ingested documents and send notifications when matches are found. Empty/null if no alerts attached. Use for: Content monitoring, safety detection, compliance alerts.

NOT REQUIRED. List of retriever enrichments to run on documents during post-processing. Each entry specifies: retriever_id, input_mappings, write_back_fields, execution_phase, priority. Retriever enrichments execute a retriever pipeline and write selected result fields back to documents. Empty/null if no enrichments attached. Use for: LLM classification, cross-collection joins, enrichment.

Whole-collection transform: run a retriever pipeline over this collection's source and write its output as the collection's documents.

Name a saved retriever with retriever_id, and pin its version with retriever_version: the collection keeps running that version after the retriever changes, until a PATCH moves the pin. Inline stages still work and are deprecated. A collection is either extractor-based or transform-based. The source may be a collection, or a bucket, in which case the transform reads the bucket's objects through its system collection. Re-runs update documents in place: a document's id comes from unique_id_field, the pipeline's group_by field, or the source document's id. write_back_fields (optional) picks the output fields; without it the pipeline's output defines them.