Create Training Job
Create a new training job.
The provider is resolved from the registry based on (base_model, training_type).
Args: request: Incoming FastAPI request (required by SlowAPI key function). training_request: Training configuration with dataset references and base_model. auth: Authenticated user context.
Request body
User-friendly name for the trained model
HuggingFace model identifier (e.g. 'fastino/gliner2-base-v1', 'deepseek-ai/DeepSeek-V4-Flash', or 'nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16').
Training type: 'full' or 'lora'
Fraction of data held out for validation.
Maximum training epochs. With early stopping enabled, training typically terminates well before this ceiling.
Peak learning rate for AdamW. When omitted, the trainer's default applies — the per-model catalog rate (2e-4) for decoder LoRA on Modal, 2e-5 elsewhere. Pinning the old 2e-5 default on a decoder LoRA run under-trains the adapter by an order of magnitude.
Training batch size. Prefer omitting this field so the training service applies the catalog default for base_model. Explicit values equal to this Field default (4) that exceed the model's safe maximum are treated as legacy unset clients and clamped to the catalog default at the training service boundary; any other oversize value is rejected.
Optional reproducibility seed for Modal decoder or GLiNER2 (encoder) training. Requests must pin provider_name to 'modal'; Fireworks and other unsupported architectures reject this field. When omitted, encoder and decoder jobs share the same trainer default (3407). Decoder contract: A pinned seed governs LoRA initialisation and the trainer's own RNG (dataloader shuffle order and dropout). It does not select the train/validation split: that partition uses a dedicated split seed so two runs that differ only in seed are scored on the same held-out rows. It does not make runs bit-identical: GPU reduction order stays non-deterministic, so metrics can differ between otherwise identical runs. Across six observed same-config decoder pairs, final validation loss agreed to within 6.8% relative and two pairs agreed exactly. Treat that as an observed envelope from production history, not a guaranteed bound. Encoder contract: A pinned seed governs dataset shuffle and auto-sizing downsample order, and the trainer's own weight-initialisation and dropout RNG. It does not select the train/validation split on a Brain-dispatched run: that partition is a fixed left-to-right split derived from validation_data_percentage (validation is the tail), independent of seed, so two runs that differ only in seed are scored on the same held-out rows. It does not make runs bit-identical: GPU reduction order stays non-deterministic (cuBLAS GEMM split-k and, unless the embedding-backward scatter/index_add path below applies), so metrics can still differ between identically-configured runs. Before encoder seed control existed, three identically-configured launches measured classification macro-F1 ranging 0.38–0.63 — treat pinning a seed as removing one real, measured source of that noise, not as a guaranteed bound on the rest. Every GLiNER2 job unconditionally pins cuDNN's own algorithm selection (the same flags GLiNER2's bundled Trainer sets) and enables torch.use_deterministic_algorithms in warn-only mode -- this is a fixed container default, not a per-request knob. The cuDNN pin currently costs nothing and changes nothing on this backbone: cuDNN governs convolution/pooling/RNN kernels, and this encoder's DeBERTa-v2 backbone has none, so pinning it has nothing to pin there; the throughput trade-off only materializes if a future backbone adds conv/pooling/RNN layers. The deterministic-algorithms half is not a no-op: it makes embedding-backward scatter/index_add deterministic on CUDA, narrowing -- but, since CUBLAS_WORKSPACE_CONFIG is not set, not closing -- the GPU-reduction-order gap above. It never raises instead of running (warn-only), so it is safe to always leave on, but for the same reason it does not guarantee bit-identical runs.
Save checkpoint every N steps
Enable structured training profiling for this run and persist a training_profile.json artifact.
Optional W&B API key for logging
Project ID to associate with this training job. When omitted, the job is anchored to the caller's auto-managed "Default" project so it is always deployable and fleet-eligible.
LoRA rank. When omitted, the trainer's default applies: the per-model catalog rank for decoder LoRA on Modal (including 32 for the qualified Nemotron 3.5 Lightning profile), or 16 elsewhere.
LoRA alpha. When omitted, the trainer's default applies: the per-model catalog alpha for decoder LoRA on Modal, or 32 elsewhere.
LoRA dropout. When omitted, the trainer's default applies: the per-model catalog dropout for decoder LoRA on Modal, or 0.1 elsewhere.
Pack multiple short examples into one training sequence. Applies to decoder dense-LoRA training only, which is the one backend that receives it. None uses the base model's catalog default, so omitting it leaves existing behaviour unchanged.
Decoder SFT loss masking knob. Dense decoder LoRA currently rejects true until assistant-only loss masking is supported by the active trainer. Defaults false to preserve the stock recipe.
Fraction of total training steps for linear LR warmup. Ignored when warmup_steps is set. When omitted, the trainer's default applies — the per-model catalog warmup (0.03) for decoder LoRA on Modal, none elsewhere.
Absolute number of linear LR warmup steps. When set, takes priority over warmup_ratio.
LR decay schedule after warmup: 'constant', 'linear', or 'cosine'.
AdamW weight decay (L2 penalty). 0 disables weight decay. Honoured by Modal decoder/dense-LoRA training paths. Not forwarded to GLiNER2 Modal training, which uses its own container default.
Validation epochs without improvement before stopping. Requires validation_data_percentage > 0. Set to 0 to disable.
Minimum validation loss improvement to count as progress. Prevents early stopping from triggering on noise.
Pin training to a specific provider (e.g. 'modal'). Bypasses automatic provider selection.
Canonical system prompt written to every decoder training row. When populated, it is persisted on training_jobs.system_prompt and re-injected by inference providers for API-direct callers that omit the system message (train/serve alignment), and prefills the inference-page system-prompt editor. Leave null for PAFT datasets, mixed-prompt uploads, or any case where no single prompt should be pinned at serve time. Ignored for non-decoder tasks.
GLiNER only: learning rate applied to encoder parameters. When omitted, falls back to learning_rate.
GLiNER only: learning rate applied to task-head parameters. When omitted, falls back to learning_rate.
Accumulate gradients over N mini-batches before each optimizer step. Effective batch size = batch_size * N. Honoured by GLiNER and dense decoder LoRA training; when omitted, dense LoRA applies the per-model catalog default (8 for the H200 Nemotron 3.5 profiles, whose per-device batch is pinned to 1).
GLiNER only. Opt-in: when true, downsample each training dataset to min(max_samples_per_dataset, max(min_samples_per_dataset, samples_per_label * num_labels)). When omitted or false, the full provided dataset is used (no silent downsampling). Defaults to false in the Modal container.
GLiNER only: lower bound for auto-sized dataset cap.
GLiNER only: upper bound for auto-sized dataset cap.
GLiNER only: scaling factor used when computing the auto-sized per-dataset cap.
GLiNER only: minimum number of optimizer steps; raises epoch count if the provided nr_epochs would yield fewer steps.
Training algorithm: 'sft' (default), 'grpo', or 'dpo'. GRPO and DPO are dispatched to the Modal RL entrypoint. GRPO requires rl_config.reward_type from the built-in menu; DPO requires {prompt, chosen, rejected} columns and optional rl_config.dpo_beta / loss_type.
Algorithm-specific hyperparameters for RL training. Supported keys (all optional unless noted, TRL-aligned defaults applied container-side): max_steps, kl_beta, group_size, sampling_temperature, max_completion_length, reward_type (GRPO; required, one of the built-in reward function names — see rl_training.BUILTIN_REWARDS); dpo_beta, loss_type (DPO); logging_steps (both; defaults to 25, lower for short smoke runs). When reward_type == 'llm_as_judge' (GRPO only) the judge call is routed through brain's '/v1/chat/completions' API authenticated with a per-run pio_sk* key minted by ModalTrainingHandler._launch_and_monitor immediately before spawning the Modal function (the user never supplies the key — minted in the workqueue handler so the cleartext value never enters the SQS message body, injected into the Modal payload at spawn time, revoked from the post-training cleanup hook on terminal status). Additional knobs: llm_judge_model (HuggingFace model id, default 'claude-haiku-4-5' — must resolve to a brain catalog entry via resolve_catalog_model_id), llm_judge_rubric (template string with {prompt}/{completion}/{answer} placeholders; falls back to a generic faithfulness/quality rubric scored 1-10 when absent), llm_judge_score_scale (raw max score for normalisation to [0,1], default 10), llm_judge_timeout_s (HTTP timeout per judge call, default 30), llm_judge_max_concurrent (parallelism cap on judge HTTP calls, default 8), llm_judge_max_retries (per-row retry budget on transient HTTP errors, default 1), llm_judge_retry_backoff_s (sleep between retries, default 2.0).
Response
Successful Response
Project ID this training job is associated with
Experiment whose agent trained this adapter, when one did. A composite foreign key pins it to the same project as project_id, so it is never an Experiment from elsewhere. Null for a job dispatched outside an Experiment -- a direct API call, or a job predating the column -- which means the owning Experiment is unknown, not that there is none. Adapter-scoped UI hand-offs read this to open the thread that produced the adapter instead of whichever of the project's Experiments happens to be the most recently active (ENG-7287).
Effective reproducibility seed for Modal decoder or GLiNER2 (encoder) training. Null for Fireworks, unknown, and other providers/architectures that cannot honor this contract, and also null for a legacy Modal decoder/encoder job created before seed provenance was recorded (ENG-6970): its seed is unknown, not reconstructed from the default the migration backfilled. A non-null value is always a genuinely recorded seed. Decoder contract: A pinned seed governs LoRA initialisation and the trainer's own RNG (dataloader shuffle order and dropout). It does not select the train/validation split: that partition uses a dedicated split seed so two runs that differ only in seed are scored on the same held-out rows. It does not make runs bit-identical: GPU reduction order stays non-deterministic, so metrics can differ between otherwise identical runs. Across six observed same-config decoder pairs, final validation loss agreed to within 6.8% relative and two pairs agreed exactly. Treat that as an observed envelope from production history, not a guaranteed bound. Encoder contract: A pinned seed governs dataset shuffle and auto-sizing downsample order, and the trainer's own weight-initialisation and dropout RNG. It does not select the train/validation split on a Brain-dispatched run: that partition is a fixed left-to-right split derived from validation_data_percentage (validation is the tail), independent of seed, so two runs that differ only in seed are scored on the same held-out rows. It does not make runs bit-identical: GPU reduction order stays non-deterministic (cuBLAS GEMM split-k and, unless the embedding-backward scatter/index_add path below applies), so metrics can still differ between identically-configured runs. Before encoder seed control existed, three identically-configured launches measured classification macro-F1 ranging 0.38–0.63 — treat pinning a seed as removing one real, measured source of that noise, not as a guaranteed bound on the rest. Every GLiNER2 job unconditionally pins cuDNN's own algorithm selection (the same flags GLiNER2's bundled Trainer sets) and enables torch.use_deterministic_algorithms in warn-only mode -- this is a fixed container default, not a per-request knob. The cuDNN pin currently costs nothing and changes nothing on this backbone: cuDNN governs convolution/pooling/RNN kernels, and this encoder's DeBERTa-v2 backbone has none, so pinning it has nothing to pin there; the throughput trade-off only materializes if a future backbone adds conv/pooling/RNN layers. The deterministic-algorithms half is not a no-op: it makes embedding-backward scatter/index_add deterministic on CUDA, narrowing -- but, since CUBLAS_WORKSPACE_CONFIG is not set, not closing -- the GPU-reduction-order gap above. It never raises instead of running (warn-only), so it is safe to always leave on, but for the same reason it does not guarantee bit-identical runs.
Immutable snapshot of what this job actually trained with, taken at dispatch: LoRA rank/alpha/dropout, learning rate, warmup ratio, gradient accumulation, packing, precision, attention backend, reasoning parser, container image, runtime profile, max sequence length, and seed, for every seed-capable provider/model. Null for jobs dispatched before the snapshot existed, and for providers that resolve no catalog recipe -- except Modal encoder and Modal RL strategies, which have no dense-LoRA catalog recipe but still return a non-null (seed) snapshot, since seed has no other persisted column. Read this rather than re-deriving from the catalog: the catalog reports what a job dispatched today would get, which is a different question.
HuggingFace repo id (e.g. 'username/model-name') set only after a successful push_training_job_to_hub call. This is the sole source of truth for whether a checkpoint has been pushed to the Hub -- it does not mean the repo is reachable by anyone other than the pusher: see hub_model_private for that. trained_model_path is an internal storage key and must never be parsed to infer a Hub repo id.
Whether the pushed Hub repo is private, as recorded at push time. Null for a job that has never been pushed, and for a push recorded before this field existed (ENG-6761) -- this means the visibility was not recorded, not that either visibility applies. The request schema (HuggingFacePushModelRequest.private) defaults to True, but the only real caller (the CLI) always sends it explicitly, so that default is unreachable in practice -- null here means the visibility was never recorded, not that a caller omitted it. Render null as a neutral 'pushed, visibility unknown' state rather than assuming either PUBLISHED or PRIVATE.
Canonical status alias for compatibility handling (requested, running, complete, deployed, failed, cancelled)
Whether this status is terminal for polling loops
Task type derived from training datasets: 'ner', 'classification', 'custom', or 'decoder'
Raw training method as persisted: 'lora', 'qlora', or 'full'.
Normalized fine-tune kind: 'lora' for adapters (lora/qlora) or 'full' for merged weights. Null when the persisted training type is unrecognised -- clients must not claim a kind in that case.
Whether an artifact location is recorded, so there is something to serve.
Whether a provider is already serving this artifact. False is not a deployment blocker: promotion provisions or re-warms a provider.
Whether this job passes server-side deployability validation for its own project. Authoritative -- the same check the deployment endpoints enforce.
Why the job is not deployable (e.g. 'job_incomplete', 'missing_artifact', 'provider_incompatible'). Null when deployable.
Merged labels from training datasets (entity types for NER, class labels for classification)
Sample text to pre-load into inference input
Training and evaluation metrics dictionary. Contains final_training_loss, final_validation_loss, best_validation_loss from training logs, and optional evaluation metrics (f1_score, precision_score, recall_score, accuracy) if an evaluation has been run.
Version number for this training job (e.g., '1', '2', '3')
ID of the original/root training job this version derives from
Provider-specific deployment metadata written by the training monitor, keyed by provider: {"modal": {...}}.
Training provider that handled this job (e.g. 'modal'). Jobs predating a provider removal carry an 'archived_<provider>' label.
Overall training completion percentage (0-100). Updated live during training.
Epoch currently in progress (1-indexed). Updated live during training.
Deprecated. Always returns None -- deployment_status no longer exists.
Kept for backward compat with clients that read this field.