Sign in
RL

Sample

Submits a sample operation that will asynchronously generate text completions with logprobs.

post/rl/training-sessions/{session_id}/operations/sample

Path parameters

session_idstring required

Training session ID

Request body

num_samplesinteger

Number of completions to generate per prompt

prompt_logprobsboolean

When true, also compute teacher-forced log-probabilities for the model input tokens and return them in SampleResult.prompt_logprobs.

topk_prompt_logprobsinteger

Number of most likely alternative tokens to return per model input token in SampleResult.topk_prompt_logprobs. 0 disables top-k prompt log-probabilities. Maximum 20.

return_routed_expertsboolean

When true, capture the mixture-of-experts routing decisions made while generating and return them in SampledSequence.routed_experts, so training can reuse the same expert selection. Only available on mixture-of-experts models; ignored otherwise. The captured buffer scales with sequence length, so leave it off unless you replay routing during training.

return_routed_experts_object_uriboolean

When true together with return_routed_experts, return each routing capture as a backend-owned object_uri plus shape instead of inline base64 data. Clients that do not opt in keep the legacy inline response.

Example request

{
  "model_inputs": [
    {
      "chunks": [
        {
          "encoded_text": {
            "tokens": [
              123,
              456,
              789
            ]
          }
        }
      ]
    }
  ],
  "sampling_params": {
    "max_tokens": 512,
    "temperature": 1,
    "top_p": 1,
    "top_k": -1,
    "stop": [
      "\n",
      "END"
    ]
  },
  "num_samples": 1
}

Response

idstring required

Operation ID

status'TRAINING_OPERATION_STATUS_UNSPECIFIED' | 'TRAINING_OPERATION_STATUS_PENDING' | 'TRAINING_OPERATION_STATUS_RUNNING' | 'TRAINING_OPERATION_STATUS_COMPLETED' | 'TRAINING_OPERATION_STATUS_FAILED' required

Example response

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "output": {
    "results": [
      {
        "sequences": [
          {
            "tokens": [
              "123",
              "456",
              "789"
            ],
            "logprobs": [
              -0.5,
              -1.2,
              -0.3
            ],
            "prompt_cache_hit_tokens": 1024,
            "routed_experts": {
              "shape": [
                "512",
                "64",
                "8"
              ]
            }
          }
        ],
        "prompt_logprobs": [
          0,
          -0.5,
          -1.2
        ],
        "topk_prompt_logprobs": [
          {
            "token_ids": [
              123,
              456
            ],
            "logprobs": [
              -0.5,
              -2.1
            ]
          }
        ],
        "policy_segments": [
          {
            "version": 5
          }
        ]
      }
    ]
  },
  "error": {
    "message": "Operation timed out"
  }
}

Changes

Changed in 5 of the 15 revisions of this API.4415

  • 3ad4ab32894812See the full diff
    • ▲

      the response property output/results/items/sequences/items/routed_experts/data became optional for the status 200

      response-property-became-optional

    • ○

      added the new optional request property return_routed_experts_object_uri

      new-optional-request-property

    • ○

      added the optional property output/results/items/sequences/items/routed_experts/object_uri to the response with the 200 status

      response-optional-property-added

  • f26a23fab8e02See the full diff
    • ○

      added the new optional request property return_routed_experts

      new-optional-request-property

    • ○

      added the optional property output/results/items/sequences/items/routed_experts to the response with the 200 status

      response-optional-property-added

  • ebd4756a41e41See the full diff
    • ●

      added the new TRAINING_OPERATION_ERROR_CODE_NON_FINITE_LOSS enum value to the error/code response property for the response status 200

      response-property-enum-value-added

  • 9ccc57e5e4b9233See the full diff
    • ▲

      added the new required request property model_inputs

      new-required-request-property

    • ▲

      removed the required property output/rollouts from the response with the 200 status

      response-required-property-removed

    • ●

      removed the request property prompts

      request-property-removed

    • ●

      removed the request property sampling_params/return_prompt_logprobs

      request-property-removed

    • ●

      removed the optional property output/policy_segments from the response with the 200 status

      response-optional-property-removed

    • ○

      added the new optional request property prompt_logprobs

      new-optional-request-property

    • ○

      added the new optional request property topk_prompt_logprobs

      new-optional-request-property

    • ○

      added the required property output/results to the response with the 200 status

      response-required-property-added

    This revision also has 11 changes that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog

  • 0c4fdad7743518See the full diff
    • ▲

      the output/rollouts/items/sequences/items/logprobs/items/ response's property type/format changed from number/double to number/ for status 200

      response-property-type-changed

    • ○

      added the new optional request property sampling_params/return_prompt_logprobs

      new-optional-request-property

    • ○

      the max_tokens request property default value changed from 100 to 512

      request-property-default-value-changed

    • ○

      the num_samples request property type/format was generalized from integer/int64 to integer/

      request-property-type-generalized

    • ○

      the sampling_params/max_tokens request property type/format was generalized from integer/int32 to integer/

      request-property-type-generalized

    • ○

      the sampling_params/temperature request property type/format was generalized from number/float to number/

      request-property-type-generalized

    • ○

      the sampling_params/top_k request property type/format was generalized from integer/int32 to integer/

      request-property-type-generalized

    • ○

      the sampling_params/top_p request property type/format was generalized from number/float to number/

      request-property-type-generalized

    • ○

      added the optional property output/rollouts/items/prompt_logprobs to the response with the 200 status

      response-optional-property-added

    This revision also has 2 changes that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog