Clone Collection

Configuration for collection source (bucket(s) or collection).

Collections can process data from two types of sources:

1. Bucket Source: Process raw objects from one or more buckets (first-stage processing)

   • Use this to create your initial collections from uploaded data
   • Can specify multiple buckets to consolidate data from different sources
   • All buckets must have compatible schemas (validated at creation)
   • Example: Videos from multiple regions → Frame extraction collection

2. Collection Source: Process documents from another collection (decomposition trees)

   • Use this to create multi-stage processing pipelines
   • Example: Frames collection → Scene detection collection

Multi-Bucket Requirements:

• All buckets must have compatible schemas (same fields, types, and required status)
• Schema compatibility is validated when the collection is created
• Documents track which specific bucket they came from via root_bucket_id
• Useful for consolidating data from multiple regions, teams, or environments

The source determines:

• What data the feature extractor receives as input
• The input_schema available for input_mappings and field_passthrough
• The lineage tracking in output documents

Examples: Single bucket: {"type": "bucket", "bucket_ids": ["bkt_products"]} Multi-bucket: {"type": "bucket", "bucket_ids": ["bkt_us", "bkt_eu", "bkt_asia"]} Collection: {"type": "collection", "collection_id": "col_frames"}

Configuration for a feature extractor with field passthrough support.

A feature extractor processes source data (from buckets or collections) and produces features (embeddings, extracted text, detected objects, etc.).

With field passthrough, you can also include selected source fields in the output documents alongside the computed features.

Core Concepts: 1. Feature Extraction: Extractors compute features from input data (e.g., text → embeddings, image → detections, video → scenes) 2. Field Passthrough: Selectively preserve source fields in output (e.g., title, category, campaign_id from source → output documents) 3. Output Schema: Combination of passed-through fields + extractor outputs (e.g., {title, category, text_embedding} all in one document)

How Field Passthrough Works: 1. Define which source fields to include via field_passthrough list 2. During processing, these fields are extracted from source 3. They appear in output documents at root level 4. Combine with extractor outputs for complete documents 5. Use target_path to rename fields for cleaner schemas

Field Selection Modes: - Explicit (field_passthrough + include_all=False): Only listed fields pass through. Clean, controlled output. Example: passthrough=[title, category] → output has ONLY title, category, embedding

- **Inclusive** (include_all=True):
  All source fields pass through, field_passthrough for renaming.
  Example: source has 10 fields → output has all 10 + embedding

- **None** (no field_passthrough):
  Only extractor outputs in documents.
  Example: → output has ONLY embedding (no source fields)

Use Cases: - Preserve Identifiers: Keep campaign_id, product_sku, order_id for tracking - Enable Filtering: Pass category, status, department for query filters - Maintain Context: Include title, description for display - Track Metadata: Preserve author, created_at, source for lineage - Business Logic: Keep priority, region, type for application logic

Common Patterns: 1. Minimal Passthrough (recommended): field_passthrough=[{"source_path": "id"}], include_all=False → Clean output, only ID + extractor features

2. **Metadata Preservation**:
   field_passthrough=[
       {"source_path": "title"},
       {"source_path": "category"},
       {"source_path": "created_at"}
   ]
   → Document has context for display and filtering

3. **Field Renaming**:
   field_passthrough=[
       {"source_path": "doc_title", "target_path": "title"},
       {"source_path": "metadata.author", "target_path": "author"}
   ]
   → Cleaner output schema with flattened fields

4. **Required Fields**:
   field_passthrough=[
       {"source_path": "campaign_id", "required": True},
       {"source_path": "priority", "default": 0}
   ]
   → Ensures critical fields always present

Requirements: - feature_extractor_name: REQUIRED - name of the extractor - version: REQUIRED - extractor version (e.g., "v1") - parameters: NOT REQUIRED - extractor-specific config (model, thresholds, etc.) - input_mappings: NOT REQUIRED - maps extractor inputs to source fields - field_passthrough: NOT REQUIRED - which source fields to preserve (default: none) - include_all_source_fields: NOT REQUIRED - preserve all fields (default: false)

OPTIONAL. Override taxonomy applications. If omitted, copies from source collection.

Collection model with single feature extractor and deterministic schema.

Collections are processing pipelines that transform source data (from buckets or other collections) into searchable documents with features. Each collection represents ONE transformation step with ONE feature extractor.

Key Concepts: - Input Schema: What goes IN (from bucket or upstream collection) - Feature Extractor: The transformation logic (text embedding, video processing, etc.) - Field Passthrough: User-selected source fields to include in output - Output Schema: What comes OUT (field_passthrough + extractor outputs)

Architecture (Task 9): - ONE feature extractor per collection (not a list) - output_schema computed at creation time (deterministic, no waiting) - Multiple extractors = multiple collections - Enables clean decomposition trees and clear lineage

Use Cases: - Create embeddings from text/images/video - Build multi-tier pipelines (bucket → frames → scenes) - Apply feature extraction with metadata passthrough - Organize data processing into clear stages

Requirements: - collection_name: REQUIRED, unique within namespace - source: REQUIRED, bucket or collection source - feature_extractor: REQUIRED, single extractor configuration - input_schema: Auto-computed from source (bucket_schema or upstream output_schema) - output_schema: Auto-computed at creation (field_passthrough + extractor output)