evaluations

Add rollouts to an open evaluation

Add rollouts to an OPEN evaluation (one submitted with open: true): the same parameters as submit_rollouts, attached to an existing set. An identity already in the evaluation is refused (an evaluation never holds two rows of one identity). Call seal_evaluation when the last batch is in.

post/preview/evaluations/{name}/rollouts

Request

  • Base URL: https://autoresearch.sfcompute.com
  • URL: https://autoresearch.sfcompute.com/preview/evaluations/{name}/rollouts
  • Auth: HTTP bearer

Path parameters

namestring nullable required

The evaluation to attach to (created if new; refused if sealed or owned by another workspace).

Query parameters

workspacestring

Act in this workspace (its slug). Omitted: the token's bound workspace, then the organization's default. Never ambient - there is no switch-workspace mutation on this door.

Request body

chipstring nullable

The runner's chip for a task-kind submit ("h100", "cpu-2"). Default: the catalog container's, else h100. Ignored for sandbox kind.

commandstring nullable

The runner command (default: the image's). Task kind only.

egressstring nullable

Egress for an openenv bake: "open" (default) or "none".

envobject

Environment variables for the runner (a policy endpoint's key rides here, sealed, and never reaches a sandbox). Task kind only.

environmentunknown required
gpustring nullable

Alias of chip.

groupinteger nullable

Attested group of N children from one state (sandbox kind: N forks of the snapshot; requires --no-memo under eval). Seeds must number N.

idempotency_keystring nullable

Replay protection: the same key returns the first submit's evaluation rather than creating a second set.

judgestring required

The judge, pinned: "goal-region@v1", or "swe-tests@sha256:<digest>" to bind the label to code, or "upstream:<name>@<image digest>" for an environment's own reward. Reports never average rows whose judges differ.

labelsunknown
lease_batchinteger nullable

Rollouts one lease hands the runner (default 32; up to 8192 for a vectorized simulator; openenv pools default 16, max 64 - each granted rollout forks a sandbox).

max_duration_minutesinteger nullable

The runner attempt's max_duration_minutes (default 240). Task kind.

memoboolean nullable

Memoization on/off (default follows the profile; true with "train" is refused). A memo hit returns the stored verdict, unbilled.

missionstring nullable

The mission this work rolls up into.

openboolean nullable

Leave the evaluation open so add_rollouts can append later.

policystring nullable

The policy label, part of the identity. REQUIRED for a sandbox-kind eval submit with memoization on: the model driving the sandbox is outside the envelope, so without it the next checkpoint would be served this one's verdicts. What we cannot see, we cannot tell apart.

profilestring nullable

"eval" (default: memoization on, seeded-stochastic) or "train" (memoization off - a memoized training sample collapses your sample diversity - and unrepeatable).

repro_classstring nullable

"bit-exact", "seeded-stochastic" or "unrepeatable". Never defaults to bit-exact: we stamp the class you declare, and qualify_rollout is how you find out whether it is true.

runnersinteger nullable

Machines that may work the pool at once (default 4, max 32); started as the backlog warrants, so a ceiling on spend, not a reservation.

seedsunknown
specunknown
state_expires_afterstring nullable

Expiry for states these rollouts declare ("30d", "never"). Default: the org's snapshot default, else never.

Response

The capability's JSON result.

objectstring

The resource type, when the result is one resource.

Changes

No recorded changes to this operation across all 1 revision of this API.