Update Storage Connection

OPTIONAL. New name for the connection. Must be unique within the organization if provided. Format: 1-100 characters.

OPTIONAL. New description for the connection. Set to empty string to clear existing description. Format: Up to 500 characters.

Enumeration of task statuses for tracking asynchronous operations.

Task statuses indicate the current state of asynchronous operations like batch processing, object ingestion, clustering, and taxonomy execution.

Status Categories: Operation Statuses: Track progress of async operations Lifecycle Statuses: Track entity state (buckets, collections, namespaces)

Values: PENDING: Task is queued but has not started processing yet IN_PROGRESS: Task is currently being executed PROCESSING: Task is actively processing data (similar to IN_PROGRESS) COMPLETED: Task finished successfully with no errors COMPLETED_WITH_ERRORS: Task finished but some items failed (partial success) FAILED: Task encountered an error and could not complete CANCELED: Task was manually canceled by a user or system UNKNOWN: Task status could not be determined SKIPPED: Task was intentionally skipped DRAFT: Task is in draft state and not yet submitted

ACTIVE: Entity is active and operational (for buckets, collections, etc.)
ARCHIVED: Entity has been archived
SUSPENDED: Entity has been temporarily suspended

Terminal Statuses: COMPLETED, COMPLETED_WITH_ERRORS, FAILED, CANCELED are terminal statuses. Once a task reaches these states, it will not transition to another state.

Partial Success Handling: COMPLETED_WITH_ERRORS indicates that the operation completed but some documents/items failed. The task result includes: - List of successful items - List of failed items with error details - Success rate percentage This allows clients to handle partial success scenarios appropriately.

Polling Guidance: - Poll tasks in PENDING, QUEUED, IN_PROGRESS, or PROCESSING states - Stop polling when task reaches COMPLETED, COMPLETED_WITH_ERRORS, FAILED, or CANCELED - Use exponential backoff (1s → 30s) when polling

OPTIONAL. Quick boolean flag for filtering. True when status is ACTIVE, False otherwise. Automatically maintained when status changes.

Unique identifier for the storage connection. Auto-generated with 'conn_' prefix followed by secure random token. Format: conn_{15-character alphanumeric}. Used for API operations and audit trails.

REQUIRED. Organization internal identifier for multi-tenancy scoping. All connection operations are scoped to this organization. Format: int_{24-character secure token}.

Supported external storage providers for ingestion and sync.

Mixpeek can connect to external storage providers to automatically ingest objects and keep them synchronized with your namespaces.

Providers: GOOGLE_DRIVE: Google Drive and Google Workspace shared drives. - Authentication: Service account or OAuth2 - Features: Shared drive support, real-time sync, metadata preservation - Use cases: Marketing assets, team documents, knowledge bases - Limitations: Rate limits apply (10,000 requests/100 seconds per user)

S3: Amazon S3 and S3-compatible storage (MinIO, DigitalOcean Spaces, etc).
    - Authentication: Access keys or IAM role assumption
    - Features: Bucket notifications, prefix filtering, versioning support
    - Use cases: Data lakes, video archives, ML datasets, backups
    - Limitations: IAM role assumption preferred over access keys

SNOWFLAKE: Snowflake data warehouse tables.
    - Authentication: Key pair or username/password
    - Features: Incremental sync via watermarks, row-level mapping, schema introspection
    - Use cases: Customer data tables, product catalogs, transaction logs, metadata tables
    - Limitations: Each row becomes one object; large tables require incremental column

SHAREPOINT: Microsoft SharePoint and OneDrive for Business.
    - Authentication: Azure AD OAuth2 (client credentials or delegated)
    - Features: Site/drive selection, folder sync, delta queries for incremental sync
    - Use cases: Enterprise documents, team collaboration files, compliance archives
    - Limitations: Requires Azure AD app registration; throttling limits apply

Connection Requirements: - Valid credentials with read access to target files/buckets - Network connectivity from Mixpeek infrastructure - Appropriate IAM policies or share permissions configured

Examples: - Use GOOGLE_DRIVE for syncing team marketing materials - Use S3 for ingesting video archives from data lakes - Use S3 with IAM role for secure production deployments - Use SHAREPOINT for syncing enterprise SharePoint document libraries

TIGRIS: Tigris Data globally distributed object storage (S3-compatible).
    - Authentication: Access keys (same format as S3)
    - Features: S3-compatible API, global distribution, zero egress fees
    - Use cases: Globally distributed media, low-latency content delivery
    - Endpoint: https://fly.storage.tigris.dev

POSTGRESQL: PostgreSQL relational database.
    - Authentication: Username/password
    - Features: SQL queries, incremental sync via watermarks, row-level mapping
    - Use cases: Customer data tables, product catalogs, transaction logs
    - Limitations: Each row becomes one object; large tables require incremental column

INSTAGRAM: Instagram Business/Creator accounts via Meta Graph API.
    - Authentication: OAuth 2.0 via Meta Developer Console
    - Features: Media sync (posts, reels, carousels), engagement metrics, captions
    - Use cases: Social media content analysis, talent scouting, competitor monitoring
    - Limitations: Requires Instagram Business/Creator account; long-lived tokens (60 days) need refresh

TIKTOK: TikTok accounts via TikTok Content API.
    - Authentication: OAuth 2.0 via TikTok Login Kit
    - Features: Video sync, engagement metrics (likes, views, shares), descriptions
    - Use cases: Social media content analysis, creator discovery, trend monitoring
    - Limitations: Access tokens expire in 24h (refresh_token flow); rate limits apply

RSS: RSS/Atom feed entries.
    - Authentication: Optional HTTP headers (most feeds are public)
    - Features: Entry-level sync with title, author, categories, content
    - Use cases: News monitoring, blog ingestion, content aggregation
    - Limitations: No pagination; feeds are fetched in full each poll

HTTP_API: Arbitrary REST/HTTP JSON APIs.
    - Authentication: Optional HTTP headers (API keys, Bearer tokens, etc.)
    - Features: Configurable JSONPath to items array, dedup via item ID field,
      incremental sync via timestamp field, GET or POST methods, JSON or JSONL responses
    - Use cases: Public APIs (Hacker News, GitHub), private APIs (Stripe, internal services),
      any JSON endpoint that returns a list of items
    - Limitations: No built-in pagination; API is fetched in full each poll

BOX: Box cloud content management and file sharing.
    - Authentication: OAuth 2.0 (JWT or Client Credentials Grant with CCG)
    - Features: Folder sync, enterprise content management, metadata, versioning
    - Use cases: Enterprise document management, compliance archives, collaboration files
    - Limitations: Rate limits apply (10 API calls per second per user)

BRIGHTDATA: BrightData web data platform for dataset collection and web scraping.
    - Authentication: API token from BrightData dashboard
    - Features: Pre-built datasets (LinkedIn, Amazon, etc.), custom scrapers,
      scheduled collection, geo-targeting, output format selection
    - Use cases: Competitor monitoring, market research, lead generation,
      e-commerce pricing, social media data ingestion
    - Limitations: Dataset availability depends on subscription; rate limits vary

BACKBLAZE: Backblaze B2 Cloud Storage (S3-compatible object storage).
    - Authentication: Application key ID + application key from Backblaze console
    - Features: S3-compatible API, auto-discovers regional endpoint via B2 auth,
      full bucket/prefix sync, include/exclude patterns, modified_since filtering
    - Use cases: Cold storage sync, media archives, backup ingestion
    - Limitations: Buckets are region-specific; key must have listFiles/readFiles

MUX: Mux video infrastructure platform.
    - Authentication: Access token ID + access token secret (HTTP Basic Auth)
    - Features: Video asset listing, static rendition download (MP4), metadata sync
    - Use cases: Video library ingestion, media asset processing, content analysis
    - Limitations: Assets must be in 'ready' status; static renditions must be enabled

EMAIL: Inbound email connector (push-based).
    - Authentication: Per-connection webhook signing secret (HMAC-SHA256)
    - Features: MIME parsing, attachment extraction, sender allowlist, raw .eml archival
    - Use cases: Document intake via email forwarding, compliance mailbox, support inbox
    - Limitations: Requires external inbound email service (SES/Postmark/CloudMailin)

SUPABASE: Supabase Storage (S3-compatible object storage built on open source).
    - Authentication: Dedicated S3 access keys (recommended) OR service_role JWT as
      session token (access_key_id=project_ref, secret=anon_key, session_token=service_role)
    - Features: S3-compatible API, per-project endpoint auto-derived from project_ref,
      bucket-scoped sync, include/exclude patterns, modified_since incremental sync
    - Use cases: Syncing user-uploaded assets from Supabase-backed apps, unified
      search across Supabase Storage + other sources
    - Limitations: Endpoint is project-specific (https://<ref>.storage.supabase.co);
      S3 keys must be generated from the Supabase dashboard (Storage → S3 Access Keys)

GCS: Google Cloud Storage via the native google-cloud-storage SDK (service-account JSON).
    - Authentication: Service account JSON key (from a service account granted the
      Storage Object Viewer role on the target bucket)
    - Features: Native GCS API (not the S3-compatible XML API), bucket + prefix sync,
      include/exclude patterns, min/max size + mime_type filtering, modified_since
      incremental sync, md5_hash change detection
    - Use cases: Ingesting media/data-lake objects from GCS buckets, unified search
      across GCS + other sources without minting HMAC interop keys
    - Limitations: Requires a service account JSON key with read access to the bucket;
      the key's private_key is redacted on read but is NOT encrypted at rest
      (CSFLE is not active in this deployment; stored plaintext in Mongo)

AZURE_BLOB: Azure Blob Storage via the native azure-storage-blob SDK.
    - Authentication: Storage account key (account_name + account_key) or Azure
      managed identity (DefaultAzureCredential, optional client_id). Azure Blob has
      no S3-compatible surface, so this native connector is the only path.
    - Features: Native Blob API, container + prefix sync, include/exclude patterns,
      min/max size + mime_type filtering, modified_since incremental sync, etag
      change detection
    - Use cases: Ingesting media/data-lake objects from Azure Blob containers,
      unified search across Azure + other sources
    - Limitations: For managed-identity auth the workload must run with an Azure
      identity that has read access (Storage Blob Data Reader) on the container;
      the account_key is redacted on read but is NOT encrypted at rest
      (CSFLE is not active in this deployment; stored plaintext in Mongo)

RTSP: Live RTSP video streams (IP cameras, encoders, NVR re-streams).
    - Authentication: none, or RTSP basic auth (username + password). Credentials
      are held as separate fields rather than embedded in the URL, so the URL can
      be logged and displayed without leaking them.
    - Features: the stream is cut into fixed-length segments, each segment is one
      object. A saved retriever configured on the bucket sync runs per segment and
      its OUTPUTS are what land in the bucket, so an unchanging camera does not
      pay extraction on every segment.
    - Use cases: traffic and city cameras, site/security monitoring, broadcast
      contribution feeds, any always-on camera whose interesting minutes are a
      small fraction of its wall-clock hours.
    - Limitations: capture is windowed, not gapless. Each sync poll opens the
      stream, captures one window, and closes it, so footage between polls is not
      retained. A gapless consumer is separate work; until it lands, treat an RTSP
      connection as a sampler of the stream and not an archive of it.

REQUIRED. Human-readable connection name for identification. Displayed in dashboards, sync logs, and API responses. Must be unique within the organization for clarity. Format: 1-100 characters, descriptive of the connection's purpose.

NOT REQUIRED. Optional description explaining the connection's purpose and scope. Helpful for team collaboration and documentation. Format: Up to 500 characters.

Enumeration of task statuses for tracking asynchronous operations.

Task statuses indicate the current state of asynchronous operations like batch processing, object ingestion, clustering, and taxonomy execution.

Status Categories: Operation Statuses: Track progress of async operations Lifecycle Statuses: Track entity state (buckets, collections, namespaces)

Values: PENDING: Task is queued but has not started processing yet IN_PROGRESS: Task is currently being executed PROCESSING: Task is actively processing data (similar to IN_PROGRESS) COMPLETED: Task finished successfully with no errors COMPLETED_WITH_ERRORS: Task finished but some items failed (partial success) FAILED: Task encountered an error and could not complete CANCELED: Task was manually canceled by a user or system UNKNOWN: Task status could not be determined SKIPPED: Task was intentionally skipped DRAFT: Task is in draft state and not yet submitted

ACTIVE: Entity is active and operational (for buckets, collections, etc.)
ARCHIVED: Entity has been archived
SUSPENDED: Entity has been temporarily suspended

Terminal Statuses: COMPLETED, COMPLETED_WITH_ERRORS, FAILED, CANCELED are terminal statuses. Once a task reaches these states, it will not transition to another state.

Partial Success Handling: COMPLETED_WITH_ERRORS indicates that the operation completed but some documents/items failed. The task result includes: - List of successful items - List of failed items with error details - Success rate percentage This allows clients to handle partial success scenarios appropriately.

Polling Guidance: - Poll tasks in PENDING, QUEUED, IN_PROGRESS, or PROCESSING states - Stop polling when task reaches COMPLETED, COMPLETED_WITH_ERRORS, FAILED, or CANCELED - Use exponential backoff (1s → 30s) when polling

Quick boolean flag for filtering active connections in queries. True when status is ACTIVE, False for SUSPENDED/FAILED/ARCHIVED. Maintained automatically when status changes. Use for efficient filtering: db.connections.find({'is_active': True})

NOT REQUIRED. UTC timestamp of the most recent successful sync operation. Updated automatically after each successful file sync/list operation. None if connection has never been used. Useful for identifying stale connections and usage analytics.

NOT REQUIRED. Most recent error message from failed health check or sync. Populated when authentication fails, network errors occur, or permissions denied. None when connection is healthy. Format: Error message truncated to 1000 characters. Used for diagnostics and troubleshooting.

Counter tracking consecutive failed health checks or sync attempts. Incremented on each failure, reset to 0 on success. Used to implement automatic connection suspension. Auto-suspend after 5 consecutive failures to prevent account lockout. Range: 0 to infinity (typically 0-10).

UTC timestamp when the connection was created. Auto-generated using shared.utilities.helpers.current_time(). Immutable after creation. Format: ISO 8601 datetime.

UTC timestamp of the most recent update to the connection. Updated automatically on any field modification. Tracks configuration changes, status updates, and credential refreshes. Format: ISO 8601 datetime.

REQUIRED. User identifier of the user who created this connection. Used for audit trails and permission checks. Format: usr_{15-character alphanumeric}. Immutable after creation.