Sign in
Eval runs

Create an eval case

Adds one case to the suite. Both title and a non-empty steps array are required — steps is optional on the shared case shape so PATCH can be partial, but a case persisted without executable steps could never run. When models is omitted the suite's configured model is used.

post/projects/{projectId}/eval-suites/{suiteId}/cases

Path parameters

projectIdstring required

ID of the hosted project that contains the server.

suiteIdstring required

Eval suite ID, as returned by POST /eval-runs.

Request body

titlestring required
idstring

Stable declared id for the case — the id it answers to in a suite file, an import or a CLI argument. Minted for you when omitted, and unique within the suite. Distinct from the id on the response, which is the case's platform row id.

expectedOutputstring
iterationsinteger
repetitionsinteger

Per-case repetitions under verdict policy 2; omitted cases inherit the suite default.

passThresholdnumber

Per-case pass threshold fraction under verdict policy 2.

isNegativeboolean
scenariostring
intentstring

Optional authored analytics grouping label. Must be already trimmed.

matchOptionsobject nullable

null clears the case override and inherits the suite's.

Response

The created case.

idstring required
declaredIdstring

The case's effective declared id. Absent on cases authored before declared identity existed.

titlestring required
expectedOutputstring
iterationsinteger required
repetitionsinteger

Trials this case runs under verdict policy 2, overriding the suite default. Absent means the case inherits it. NOT a second spelling of iterations: that one is the legacy count, which the legacy resolver reads as a FLOOR (max(iterations, suite.minimumIterations)) and which a policy-2 case still reports for compatibility. This one is exact.

passThresholdnumber

Fraction of this case's trials that must pass, overriding the suite default. Absent means the case inherits it. Never derived from the suite's minimumAccuracy, which is a PERCENT under a different resolver.

isNegativeboolean required

When true, the case passes if NO tools are called.

scenariostring
intentstring

Optional authored analytics grouping label. Must be already trimmed; absent means unlabelled.

matchOptionsobject

Absent when the case sets none — omitted from the response rather than sent as null.

createdAtnumber nullable
updatedAtnumber nullable

Changes

Changed in 5 of the 71 revisions of this API.11

  • d9a00becd4762See the full diff
    • ○

      added the new optional request property intent

      new-optional-request-property

    • ○

      added the optional property intent to the response with the 201 status

      response-optional-property-added

  • e81e8ba8baae2See the full diff
    • ○

      added the new optional request property import

      new-optional-request-property

    • ○

      added the optional property import to the response with the 201 status

      response-optional-property-added

  • c479eeceb7a34See the full diff
    • ○

      added the new optional request property passThreshold

      new-optional-request-property

    • ○

      added the new optional request property repetitions

      new-optional-request-property

    • ○

      added the optional property passThreshold to the response with the 201 status

      response-optional-property-added

    • ○

      added the optional property repetitions to the response with the 201 status

      response-optional-property-added

  • 879ca349c8042See the full diff
    • ○

      added the new optional request property id

      new-optional-request-property

    • ○

      added the optional property declaredId to the response with the 201 status

      response-optional-property-added

  • c5f6167955e41See the full diff
    • ○

      endpoint added

      endpoint-added