Collections

Export Collection

Changed on

Export collection documents to JSON, CSV, Parquet or WebDataset format.

**Export Formats:**
- **JSON**: Line-delimited JSON (JSONL) format. Good for streaming.
- **CSV**: Comma-separated values. Best for spreadsheets.
- **PARQUET**: Columnar format (default). Best for data pipelines.
- **WEBDATASET**: Tar shards for training loaders. One sample per document,
  carrying `<key>.json` and, with `include_media=True`, the source object's
  bytes. The response returns every shard plus the brace pattern loaders take.

**Vector Export:**
Vectors are large and exported separately. When `include_vectors=True`,
a separate file is created for vectors with document_id mapping.

**Field Selection:**
Use `select_fields` to export only specific fields, reducing file size.

**Lineage:**
Set `include_lineage=True` to add `lineage_chain`, `source_content_hash` and
`document_created_at` to every row, so each exported document traces back to its
source object and processing steps.

**Filtering:**
Apply filters to export a subset of documents.

**Destination (your own storage):**
Set `destination` to a storage connection you own plus a prefix, and the files and a
`manifest.json` are written there as well as to the download copy. The connection
needs `write_enabled` set, which is off by default, and its credentials need the
write permission on your side (`s3:PutObject`, or `storage.objects.create` for GCS).
A connection without it is refused with the permission named.

**Response:**
Returns presigned download URLs valid for 1 hour.

**Limits:**
- Large exports may take time. Consider using `sample_size` for testing.
- Vector exports significantly increase processing time.
post/v1/collections/{collection_identifier}/export

Request

  • Base URL: https://api.mixpeek.com
  • URL: https://api.mixpeek.com/v1/collections/{collection_identifier}/export
  • Auth: HTTP bearer and API key in header X-Namespace

Path parameters

collection_identifierstring required

The ID or name of the collection to export

Request body

format'json' | 'csv' | 'parquet' | 'webdataset'

Supported export formats for collection data.

include_vectorsboolean

Whether to include vectors in the export. Vectors are exported to a separate file due to their large size. This significantly increases export time and file size.

include_lineageboolean

Add lineage columns to every row: lineage_chain (the processing steps from source object to document), source_content_hash (SHA-256 of the source content) and document_created_at (ISO 8601). Kept even when select_fields is set. In CSV and Parquet, lineage_chain is a JSON-encoded string.

select_fieldsstring[] nullable

Specific fields to include in the export. If not provided, all fields are exported. Supports dot notation for nested fields (e.g., 'metadata.title', 'metadata.author').

sample_sizeinteger nullable

Maximum number of documents to export. If not provided, exports all documents. Useful for testing exports or creating sample datasets.

include_mediaboolean

WebDataset only. Put the source object's bytes in each sample alongside its row, resolved through the document's root object. A document whose object is gone, or which never had one, ships with its row alone and is counted in the response's media summary. This moves the full media set, so expect a much larger export and a longer run.

samples_per_shardinteger

WebDataset only. Documents per tar shard. Smaller shards parallelize better across training workers; larger shards mean fewer files to move.

Example request

{
  "select_fields": [
    "document_id",
    "metadata.title",
    "metadata.category"
  ],
  "filters": {
    "AND": [
      {
        "field": "name",
        "operator": "eq",
        "value": "John"
      },
      {
        "field": "age",
        "operator": "gte",
        "value": 30
      }
    ],
    "OR": [
      {
        "field": "status",
        "operator": "eq",
        "value": "active"
      },
      {
        "field": "role",
        "operator": "eq",
        "value": "admin"
      }
    ],
    "NOT": [
      {
        "field": "department",
        "operator": "eq",
        "value": "HR"
      },
      {
        "field": "location",
        "operator": "eq",
        "value": "remote"
      }
    ],
    "case_sensitive": true
  }
}

Response

Successful Response

download_urlstring required

Presigned URL for downloading the exported file. Valid for 1 hour.

s3_pathstring required

Full S3 path where the export is stored (for internal reference). For webdataset this is the prefix holding the shards, not a single file.

format'json' | 'csv' | 'parquet' | 'webdataset' required

Supported export formats for collection data.

document_countinteger required

Number of documents included in the export.

file_size_bytesinteger required

Size of the exported file in bytes. For webdataset this is every shard added together.

exported_atstring date-time required

Timestamp when the export was completed.

vectors_download_urlstring nullable

Presigned URL for downloading the vectors file (if include_vectors=True). Vectors are exported separately due to their large size.

vectors_s3_pathstring nullable

Full S3 path for the vectors file (if include_vectors=True).

shard_countinteger nullable

WebDataset only. Number of tar shards written. Always exact, even when the shards list below is capped.

shard_patternstring nullable

WebDataset only. Brace pattern naming every shard, e.g. documents-{000000..000007}.tar. Most loaders take this directly.

Example response

{
  "document_count": 10000,
  "download_url": "https://s3.amazonaws.com/bucket/org_xxx/ns_xxx/api_collections_export/col_xxx/v1/documents.parquet?...",
  "exported_at": "2024-01-15T10:30:00Z",
  "file_size_bytes": 5242880,
  "format": "parquet",
  "s3_path": "s3://bucket/org_xxx/ns_xxx/api_collections_export/col_xxx/v1/documents.parquet"
}

Changes

  • 61a175e8eb9aRevision changes
    • warning

      removed the optional property from the response with the status

    • info

      added the non-success response with the status

    • info

      added the non-success response with the status

    • info

      added the optional property to the response with the status

    • info

      added the required property to the response with the status

    • info

      added the required property to the response with the status

  • 5231c1d891e8Revision changes
    • info

      added the new words enum value to the request property //////

    • info

      added the new words enum value to the request property //////

    • info

      added the new words enum value to the request property //////