---
title: "Create a dataset from file upload, HuggingFace, Kaggle, Google Sheets, or a public S3 or Cloud Storage object"
method: POST
path: "/api/v1/datasets"
tags: ["Dataset"]
---

# Create a dataset from file upload, HuggingFace, Kaggle, Google Sheets, or a public S3 or Cloud Storage object

`POST /api/v1/datasets`

## Request

- The document declares no server URL.
- Auth: HTTP bearer
- TypeScript: <https://skmtc.dev/adaptionlabs/apis/adaption-api/docs/api/v1/datasets/post.ts> (a function with typed inputs; its response is checked against the schema)

Unified ingest endpoint. Pass source.url to start an async import — the provider is inferred from the URL host (HuggingFace, Kaggle, Google Sheets, or one public S3 or Cloud Storage object ending in .csv, .json, .jsonl, or .parquet). Omit it to get upload instructions for a presigned S3 PUT. Adapt the result with [POST /datasets/{dataset_id}/adapt](/api/python/resources/datasets/methods/adapt), or send it straight to training with [POST /autoscientist](/api/python/resources/autoscientist/methods/create) when it already holds prompt and completion columns.

## Request body

- CreateDatasetDto
  - `source` DatasetSourceDto, required
    - `url` string: HuggingFace, Kaggle, Google Sheets, or public S3 or Cloud Storage object URL. Provide it to import from that source; omit it to upload a local file instead. An object URL is one https file ending in .csv, .json, .jsonl, or .parquet.
    - `files` string[]: File paths to download from the source. Required with HuggingFace/Kaggle `url`. Not used for Google Sheets — use `sheet_names` (or `sheet_name`) instead — or for a public S3 or Cloud Storage object URL.
    - `sheet_names` string[]: Google Sheets tab titles to import. Only for Google Sheets `url`. Omit or pass `[]` to import every tab; pass one or more titles to import only those tabs.
    - `sheet_name` string: Legacy single Google Sheets tab title. Ignored when `sheet_names` is present (including `[]`).
    - `access` 'oauth' | 'public': Google Sheets only. `public` imports a link-shared spreadsheet without a connected Google account. `oauth` (default) uses the connected account and may be unavailable on some accounts.
    - `name` string: Name for the dataset. Required for file uploads, where it may double as a filename ("sales-data.csv") to supply the format; derived from the URL for provider imports.
    - `file_format` 'csv' | 'json' | 'jsonl' | 'parquet' | 'pdf' | 'docx' | 'pptx' | 'xlsx' | 'html' | 'zip' | 'txt': Format of the file being uploaded. Optional when `name` ends in a supported extension — it is inferred from there, and an explicit value overrides it.
    - `processing_mode` 'adapt' | 'raw': How the data is ingested, for both uploads and provider imports. `adapt` (default) runs it through the adaptation pipeline. `raw` materializes the data straight to a trainable dataset with no augmentation, and needs a column_mapping plus tabular input — `file_format` for an upload, or `files` for an import. Imported files are read as one dataset, so they must share a single format and schema. Either can then be fine-tuned or run through AutoScientist.
    - `column_mapping` PassthroughColumnMappingDto
      - `prompt` string, required: Name of the raw column holding the prompt / input text
      - `completion` string: Name of the raw column holding the completion / target text. Required unless chosen and rejected are given.
      - `chosen` string: Name of the raw column holding the preferred response. Give it together with rejected to ingest a preference dataset.
      - `rejected` string: Name of the raw column holding the dispreferred response. Give it together with chosen.
      - `context` string[]: Names of raw columns to fold in as context alongside the prompt. Each is validated against the uploaded file at ingestion; an unknown name fails the request listing the available columns. The columns themselves are kept in the dataset as well as folded.

## Response `200`

- CreateDatasetResponseDto
  - `dataset_id` string, required: ID of the newly created dataset
  - `status` string, required: Current dataset status
  - `upload_instructions` UploadInstructionsDto
    - `url` string, required: Pre-signed URL for uploading the file
    - `method` string, required: HTTP method to use
    - `s3_key` string, required: S3 object key — pass this back in the complete request if needed for verification

## Changes

- **2026-10-20** `7398ff1eeeba`: 1 warning
  - removed the request property `defer_adaption`
- **2026-09-18** `8fb9be26396b`: 3 info
  - added the new optional request property `source/allOf[#/components/schemas/DatasetSourceDto]/column_mapping/allOf[#/components/schemas/PassthroughColumnMappingDto]/chosen`
  - added the new optional request property `source/allOf[#/components/schemas/DatasetSourceDto]/column_mapping/allOf[#/components/schemas/PassthroughColumnMappingDto]/rejected`
  - the request property `source/allOf[#/components/schemas/DatasetSourceDto]/column_mapping/allOf[#/components/schemas/PassthroughColumnMappingDto]/completion` became optional

[Operation history](https://skmtc.dev/adaptionlabs/apis/adaption-api/changes/api/v1/datasets/post.md)

---

[API](https://skmtc.dev/adaptionlabs/apis/adaption-api.md) - [All operations](https://skmtc.dev/adaptionlabs/apis/adaption-api/llms.txt) - [OpenAPI document](https://skmtc.dev/adaptionlabs/apis/adaption-api/revisions/d4cf908c19a0?raw)