Sign in
felix
datasets

Pull Dataset From Hub

Pull a dataset from HuggingFace Hub and save it locally.

User-authenticated requests require hf_token. Sandbox run keys omit it and import public repos anonymously. Gated or private repos must be imported in the Pioneer UI with a user token. Optionally accepts session_id for SSE log streaming.

post/felix/datasets/pull-from-hub

Request body

repo_idstring required

HuggingFace repo ID to pull from

config_namestring nullable

HuggingFace dataset configuration, distinct from the local Pioneer name

revisionstring nullable

Specific revision/branch to pull

namestring nullable

Name for the local dataset (defaults to repo name)

hf_tokenstring nullable

HuggingFace API token. Required for user-authenticated imports. Sandbox run keys must omit this field and can only import public repos. Gated or private repos must be imported in the Pioneer UI.

session_idstring nullable

Session ID for SSE log streaming

column_mappingobject nullable

Column mapping from source to standard names

dataset_typestring nullable

Dataset type (ner, classification, custom)

type'training' | 'evaluation' | 'benchmark' nullable

Dataset purpose: 'training' (trainable), 'evaluation' (not trainable), 'benchmark' (system-managed, evaluation-only; cannot be created via this endpoint). This model is also bound to POST /felix/datasets/preview-from-hub, which ignores this field entirely -- nothing is persisted on a preview.

Response

Successful Response

idstring required
user_idstring required
dataset_namestring required
dataset_pathstring required
dataset_typestring required
sizeinteger nullable
sample_sizeinteger nullable
train_rationumber nullable

Train split ratio for this dataset version. Left-to-right split with no shuffle; validation is the tail.

created_atstring required
updated_atstring required
version_numberstring
root_dataset_idstring nullable
project_idstring nullable
schemaobject nullable
schema_warningsstring[] nullable
validationobject nullable
annotation_status'none' | 'in_progress' | 'completed' nullable
annotation_configobject nullable
annotation_progressobject nullable
status'initialized' | 'uploading' | 'converting' | 'validating' | 'ready' | 'failed' | 'generating' | 'queued' nullable

Dataset status: initialized/uploading/converting/validating/ready/failed/generating/queued

processing_errorstring nullable

Error message if status is failed

typestring

Dataset purpose tag: 'training', 'evaluation', or 'benchmark'

visibilitystring

Dataset visibility: 'private' or 'public'

is_competitionboolean

Whether this dataset is a competition benchmark

labelsstring[] nullable

Label names (entity types for NER, class labels for classification)

generation_typestring nullable

Canonical operation that created this dataset version.

is_seedboolean nullable

Whether this dataset is a seed dataset (small set for review before full expansion)

synthesis_session_idstring nullable

UUID of the synthesis log session for this dataset, used to restore creation workflow on resume

column_mappingobject nullable

Column mapping from original to standard names

Changes