PIpioneer
generate

Create Generation Job

Changed on

Create an async dataset generation job.

Returns immediately with a job_id. Poll GET /generate/jobs/{job_id} for status and results.

The task_type field in the request body determines what kind of dataset is generated:

  • ner: Named-entity recognition dataset. Requires labels.
  • classification: Text classification dataset. Requires labels.
  • custom: Free-form prompt-based dataset. Requires prompt.
  • decoder: Instruction-tuning (chat format) dataset. Requires domain_description.

Dispatches via SQS with idempotency deduplication; falls back to an in-process background task if SQS is unavailable.

Args: request: Incoming HTTP request (used by SlowAPI rate limiter). generation_request: Unified generation parameters including task_type. auth: Authenticated user context. is_seed: Whether this is a small seed dataset for UI preview. synthesis_session_id: Optional synthesis log session UUID for resume.

Returns: GenerateAsyncResponse with job_id for polling and initial status.

Raises: HTTPException: 422 if required task-specific fields are missing.

post/v1/generate

Request

Query parameters

is_seedboolean

Whether this is a seed generation

synthesis_session_idstring nullable

Synthesis log session ID for resume support

Request body

task_type'ner' | 'classification' | 'custom' | 'decoder' | 'records' | 'fields' required

Type of generation task

dataset_namestring required

Name for the generated dataset

num_examplesinteger

Number of examples to generate

domain_descriptionstring nullable

Domain description for generation context (required for decoder)

temperaturenumber

Generation temperature

qualitystring nullable

Generation quality: 'light', 'medium', or 'heavy'

generation_profile'auto' | 'fast' | 'balanced' | 'quality'

Queue/runtime execution profile. 'auto' selects a task-aware default.

session_idstring nullable

Session ID for log streaming

config_num_examplesinteger

Number of examples to use for config generation

seedinteger nullable

Random seed for reproducibility

labelsstring[] nullable

Entity/classification labels (required for ner and classification)

multi_labelboolean

Enable multi-label classification (classification only)

class_balanceobject nullable

Optional class distribution map (classification only)

batch_sizeinteger

Batch size for generation API calls

negative_ratiointeger nullable

Percentage of rows with no labels/entities (negative samples). Applies to ner, and to classification only when multi_label=True — a no-op for single-label classification, which already expresses 'none of these apply' via its own label. For ner, note bench_use_cases.py's weekly quality benchmark separately caps the empty-entity row ratio at ner_empty_entity_row_ratio_max (0.35 by default); values above ~35 risk a dataset the platform's own quality gate rejects.

promptstring nullable

Natural language prompt describing the task (required for custom)

output_formatobject nullable

Expected output schema for custom generation

infer_output_formatboolean

Infer output format from prompt when output_format is absent

instructionstring nullable

System instruction for decoder chat format

include_reasoning_traceboolean

Whether to include reasoning traces (<think> blocks) in generated outputs for decoder tasks. Ignored when task_type is not 'decoder'.

reasoning_effortstring nullable

Reasoning effort for the underlying LLM (e.g., 'low', 'medium', 'high')

use_meta_synthesizerboolean

Use meta-synthesis to auto-generate diversity criteria

min_criteriainteger

Minimum diversity criteria to generate

target_choicesinteger

Number of choices to expand for diversity

project_idstring uuid nullable

Project ID to assign the dataset to

typestring

Dataset type tag: 'training', 'evaluation', or 'split'

visibilitystring

Dataset visibility: 'private' or 'public'

output_style'prose' | 'code' | 'mixed' nullable

Decoder-only formatting policy: 'prose', 'code', or 'mixed'. Rejected when task_type is not decoder.

Response

Successful Response

job_idstring required

Job ID (same as dataset_id) for polling status

statusstring

Initial job status: queued or generating

dataset_namestring required

Name of the dataset being generated

task_typestring required

Generation task type: ner, classification, or custom

is_seedboolean

Whether this is a seed generation job

messagestring

Human-readable status message

Changes