Evals

Changes to Launch an eval

post/v1/eval

Oct 30, 20268f5039399f77 Revision changes
- breaking
  the request property became not nullable
- breaking
  the request property became not nullable
- breaking
  the request property // became not nullable
- breaking
  the request property /// became not nullable
- breaking
  the request property // became not nullable
- breaking
  the request property // became not nullable
- breaking
  the request property // became not nullable
- breaking
  the request property /// became not nullable
- breaking
  the request property // became not nullable
- breaking
  the request property // became not nullable
- breaking
  the request property /// became not nullable
- breaking
  the request property became not nullable
- breaking
  the request property became not nullable
- breaking
  the request property became not nullable
- breaking
  the request property became not nullable
- breaking
  the request property / became not nullable
- breaking
  the request property / became not nullable
- breaking
  the request property ////// became not nullable
- breaking
  the request property /////// became not nullable
- breaking
  the request property ////// became not nullable
- breaking
  the request property //// became not nullable
- breaking
  the request property ///// became not nullable
- breaking
  the request property ///// became not nullable
- breaking
  the request property ///// became not nullable
- breaking
  the request property ///// became not nullable
- breaking
  the request property ///// became not nullable
- breaking
  the request property ///// became not nullable
- breaking
  the request property ///// became not nullable
- breaking
  the request property ///// became not nullable
- breaking
  the request property ///// became not nullable
- breaking
  the request property /////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property /////////// became not nullable
- breaking
  the request property /////////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property ///////// became not nullable
- breaking
  the request property /////////// became not nullable
- breaking
  the request property //////////// became not nullable
- breaking
  the request property //////////////// became not nullable
- breaking
  the request property ////////////// became not nullable
- breaking
  the request property /////////// became not nullable
- breaking
  the request property /////////// became not nullable
- breaking
  the request property /////////// became not nullable
- breaking
  the request property /////////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property ///////////// became not nullable
- breaking
  the request property ///////////// became not nullable
- breaking
  the request property ///////////// became not nullable
- breaking
  the request property /////////////// became not nullable
- breaking
  the request property /////////////// became not nullable
- breaking
  the request property ///////////// became not nullable
- breaking
  the request property ///////////// became not nullable
- breaking
  the request property ///////////// became not nullable
- breaking
  the request property ///////////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property /////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property /////////// became not nullable
- breaking
  the request property /////////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property ///////// became not nullable
- breaking
  the request property /////////// became not nullable
- breaking
  the request property //////////// became not nullable
- breaking
  the request property //////////////// became not nullable
- breaking
  the request property ////////////// became not nullable
- breaking
  the request property /////////// became not nullable
- breaking
  the request property /////////// became not nullable
- breaking
  the request property /////////// became not nullable
- breaking
  the request property /////////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property ///////////// became not nullable
- breaking
  the request property ///////////// became not nullable
- breaking
  the request property ///////////// became not nullable
- breaking
  the request property /////////////// became not nullable
- breaking
  the request property /////////////// became not nullable
- breaking
  the request property ///////////// became not nullable
- breaking
  the request property ///////////// became not nullable
- breaking
  the request property ///////////// became not nullable
- breaking
  the request property ///////////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property /////// became not nullable
- breaking
  the request property became not nullable
- breaking
  the request property became not nullable
- breaking
  the request property // became not nullable
- breaking
  the request property /// became not nullable
- breaking
  the request property /// became not nullable
- breaking
  the request property ////// became not nullable
- breaking
  the request property ////// became not nullable
- breaking
  the request property /// became not nullable
- breaking
  the request property //// became not nullable
- breaking
  the request property ////// became not nullable
- breaking
  the request property /////// became not nullable
- breaking
  the request property /////////// became not nullable
- breaking
  the request property ///////// became not nullable
- breaking
  the request property ////// became not nullable
- breaking
  the request property ////// became not nullable
- breaking
  the request property ////// became not nullable
- breaking
  the request property ////// became not nullable
- breaking
  the request property /// became not nullable
- breaking
  the request property /// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property ////////// became not nullable
- breaking
  the request property ////////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property /// became not nullable
- breaking
  the request property /// became not nullable
- breaking
  the request property // became not nullable
- breaking
  the request property /// became not nullable
- breaking
  the request property ////// became not nullable
- breaking
  the request property ////// became not nullable
- breaking
  the request property /// became not nullable
- breaking
  the request property //// became not nullable
- breaking
  the request property ////// became not nullable
- breaking
  the request property /////// became not nullable
- breaking
  the request property /////////// became not nullable
- breaking
  the request property ///////// became not nullable
- breaking
  the request property ////// became not nullable
- breaking
  the request property ////// became not nullable
- breaking
  the request property ////// became not nullable
- breaking
  the request property ////// became not nullable
- breaking
  the request property /// became not nullable
- breaking
  the request property /// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property ////////// became not nullable
- breaking
  the request property ////////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property /// became not nullable
- breaking
  the request property /// became not nullable
- breaking
  the request property // became not nullable
- breaking
  the request property became not nullable
- breaking
  the request property became not nullable
- breaking
  removed the enum value undefined of the request property ///////
- breaking
  removed the enum value undefined of the request property ///////////
- breaking
  removed the enum value undefined of the request property ///////
- breaking
  removed the enum value undefined of the request property ///////////////
- breaking
  removed the enum value undefined of the request property ////////////
- breaking
  removed the enum value undefined of the request property ///////
- breaking
  removed the enum value undefined of the request property ///////////////
- breaking
  removed the enum value undefined of the request property ////////////
- breaking
  removed the enum value undefined of the request property //
- breaking
  removed the enum value undefined of the request property //////
- breaking
  removed the enum value undefined of the request property //
- breaking
  removed the enum value undefined of the request property //////////
- breaking
  removed the enum value undefined of the request property ///////
- breaking
  removed the enum value undefined of the request property //
- breaking
  removed the enum value undefined of the request property //////////
- breaking
  removed the enum value undefined of the request property ///////
Oct 29, 20266409b3f2bde5 Revision changes
- info
  added the new optional request property //
Oct 3, 2026877ab1fd2a22 Revision changes
- info
  added the new optional request property ///////////
- info
  added the new optional request property ///////////
- info
  added the new optional request property //////
- info
  added the new optional request property //////
Sep 22, 20262e0e9e00d0bd Revision changes
- info
  added to the /////////////// request property anyOf list
- info
  added to the /////////////// request property anyOf list
- info
  added to the ////////// request property anyOf list
- info
  added to the ////////// request property anyOf list
Aug 21, 20262ddd4c0bfd2c Revision changes
- info
  added subschema #3: preprocessor_inline to the //////// request property anyOf list
- info
  added subschema #3: preprocessor_inline to the //////// request property anyOf list
- info
  added subschema #3: preprocessor_inline to the /// request property anyOf list
- info
  added subschema #3: preprocessor_inline to the /// request property anyOf list
- info
  added subschema #4: experiment_name to the request property anyOf list
Aug 13, 2026e05ef145579d Revision changes
- info
  added the new optional request property
- info
  added the new optional request property /////////
- info
  added the new optional request property /////////
- info
  added the new optional request property ////////
- info
  added the new optional request property /////////////////
- info
  added the new optional request property //////////////////
- info
  added the new optional request property /////////////////
- info
  added the new optional request property /////////////////
- info
  added the new optional request property /////////////////
- info
  added the new optional request property /////////////////
- info
  added the new optional request property /////////////////
- info
  added the new optional request property /////////
- info
  added the new optional request property /////////
- info
  added the new optional request property ////////
- info
  added the new optional request property /////////////////
- info
  added the new optional request property //////////////////
- info
  added the new optional request property /////////////////
- info
  added the new optional request property /////////////////
- info
  added the new optional request property /////////////////
- info
  added the new optional request property /////////////////
- info
  added the new optional request property /////////////////
- info
  added the new optional request property ////
- info
  added the new optional request property ////
- info
  added the new optional request property ///
- info
  added the new optional request property ////////////
- info
  added the new optional request property /////////////
- info
  added the new optional request property ////////////
- info
  added the new optional request property ////////////
- info
  added the new optional request property ////////////
- info
  added the new optional request property ////////////
- info
  added the new optional request property ////////////
- info
  added the new optional request property ////
- info
  added the new optional request property ////
- info
  added the new optional request property ///
- info
  added the new optional request property ////////////
- info
  added the new optional request property /////////////
- info
  added the new optional request property ////////////
- info
  added the new optional request property ////////////
- info
  added the new optional request property ////////////
- info
  added the new optional request property ////////////
- info
  added the new optional request property ////////////
May 21, 20267c3461755f69 Revision changes
- info
  added the new optional request property /////////////
- info
  added the new optional request property /////////////
- info
  added the new optional request property ////////
- info
  added the new optional request property ////////
Apr 25, 2026d51a7016439c Revision changes
- info
  added the new optional request property //
- info
  added the new optional request property //
- info
  added the new optional request property //
- info
  added the new optional request property //
Apr 8, 2026a21b724fece9 Revision changes
- info
  added the new optional request property ///////
- info
  added the new optional request property //
Mar 4, 202679e3efbd784c Revision changes
- info
  added the new optional request property
- info
  added the new optional request property
- info
  added the new sandbox enum value to the request property ///////
- info
  added the new sandbox enum value to the request property ///////
- info
  added the new sandbox enum value to the request property ///////////////
- info
  added the new sandbox enum value to the request property ////////////
- info
  added the new sandbox enum value to the request property ///////
- info
  added the new sandbox enum value to the request property ///////////////
- info
  added the new sandbox enum value to the request property ////////////
- info
  added the new sandbox enum value to the request property //
- info
  added the new sandbox enum value to the request property //
- info
  added the new sandbox enum value to the request property //////////
- info
  added the new sandbox enum value to the request property ///////
- info
  added the new sandbox enum value to the request property //
- info
  added the new sandbox enum value to the request property //////////
- info
  added the new sandbox enum value to the request property ///////
Feb 10, 20264918f9bd942e Revision changes
- info
  added the new parameters enum value to the request property ///////
- info
  added the new parameters enum value to the request property ///////
- info
  added the new parameters enum value to the request property ///////////////
- info
  added the new parameters enum value to the request property ////////////
- info
  added the new parameters enum value to the request property ///////
- info
  added the new parameters enum value to the request property ///////////////
- info
  added the new parameters enum value to the request property ////////////
- info
  added the new parameters enum value to the request property //
- info
  added the new parameters enum value to the request property //
- info
  added the new parameters enum value to the request property //////////
- info
  added the new parameters enum value to the request property ///////
- info
  added the new parameters enum value to the request property //
- info
  added the new parameters enum value to the request property //////////
- info
  added the new parameters enum value to the request property ///////
Feb 2, 20268dda984bb502 Revision changes
- breaking
  added subschema #2 to the ///////// request property allOf list
- breaking
  added subschema #2 to the ///////// request property allOf list
- breaking
  added subschema #2 to the //// request property allOf list
- breaking
  added subschema #2 to the //// request property allOf list
- breaking
  removed subschema #1: function subschema #2: global from the ///////// request property anyOf list
- breaking
  removed subschema #1: function subschema #2: global from the ///////// request property anyOf list
- breaking
  removed subschema #1: function subschema #2: global from the //// request property anyOf list
- breaking
  removed subschema #1: function subschema #2: global from the //// request property anyOf list
Jan 23, 202679b5f945907c Revision changes
- info
  added the new optional request property /////////
- info
  added the new optional request property /////////
- info
  added the new optional request property /////////
- info
  added the new optional request property /////////
- info
  added the new optional request property ////
- info
  added the new optional request property ////
- info
  added the new optional request property ////
- info
  added the new optional request property ////
- info
  the request property ///////// became optional
- info
  the request property ///////// became optional
- info
  the request property //// became optional
- info
  the request property //// became optional
- info
  added the new classifier enum value to the request property ///////
- info
  added the new classifier enum value to the request property ///////
- info
  added the new classifier enum value to the request property ///////////
- info
  added the new classifier enum value to the request property ///////
- info
  added the new classifier enum value to the request property ///////////
- info
  added the new classifier enum value to the request property //
- info
  added the new classifier enum value to the request property //
- info
  added the new classifier enum value to the request property //////
- info
  added the new classifier enum value to the request property //
- info
  added the new classifier enum value to the request property //////
- info
  added the new tag enum value to the request property ///////
- info
  added the new tag enum value to the request property ///////
- info
  added the new tag enum value to the request property ///////////
- info
  added the new tag enum value to the request property ///////
- info
  added the new tag enum value to the request property ///////////
- info
  added the new tag enum value to the request property //
- info
  added the new tag enum value to the request property //
- info
  added the new tag enum value to the request property //////
- info
  added the new tag enum value to the request property //
- info
  added the new tag enum value to the request property //////
Jan 13, 202697c5791b3df1 Revision changes
- breaking
  the request property /////// became not nullable
- breaking
  the request property /////////// became not nullable
- breaking
  the request property /////////// became not nullable
- breaking
  the request property // became not nullable
- breaking
  the request property ////// became not nullable
- breaking
  the request property ////// became not nullable
- info
  the request property default value scorer was added
- info
  the request property default value scorer was added
- info
  the request property default value scorer was added
- info
  the request property default value scorer was added
- info
  the request property default value scorer was added
- info
  the request property default value scorer was added
- info
  the request property default value scorer was added
- info
  the request property default value scorer was added
- info
  the request property default value scorer was added
- info
  the request property default value scorer was added
Jan 8, 2026c7496daaaa37 Revision changes
- info
  added the new optional request property ///////
- info
  added the new optional request property ////////
- info
  added the new optional request property ///////////
- info
  added the new optional request property ////////
- info
  added the new optional request property ///////////
- info
  added the new optional request property //
- info
  added the new optional request property ///
- info
  added the new optional request property //////
- info
  added the new optional request property ///
- info
  added the new optional request property //////
- info
  added the new facet enum value to the request property ///////
- info
  added the new facet enum value to the request property ///////
- info
  added the new facet enum value to the request property //
- info
  added the new facet enum value to the request property //
- info
  added the new preprocessor enum value to the request property ///////
- info
  added the new preprocessor enum value to the request property ///////
- info
  added the new preprocessor enum value to the request property //
- info
  added the new preprocessor enum value to the request property //
- info
  added the new quickjs enum value to the request property ////////
- info
  added the new quickjs enum value to the request property ///
Dec 4, 2025b6967d956e2f Revision changes
- info
  added the new optional request property
- info
  added the new optional request property ////////
- info
  added the new optional request property ////////
- info
  added the new optional request property ///
- info
  added the new optional request property ///
- info
  added to the /////////////// request property anyOf list
- info
  added to the /////////////// request property anyOf list
- info
  added to the ////////// request property anyOf list
- info
  added to the ////////// request property anyOf list
- info
  added the new browser enum value to the request property ////////
- info
  added the new browser enum value to the request property ///
- info
  added the new custom_view enum value to the request property ///////
- info
  added the new custom_view enum value to the request property ///////
- info
  added the new custom_view enum value to the request property //
- info
  added the new custom_view enum value to the request property //
- info
  added the new none enum value to the request property ///////////
- info
  added the new none enum value to the request property ///////////
- info
  added the new none enum value to the request property //////
- info
  added the new none enum value to the request property //////
Nov 18, 2025ae80b8fa951b Revision changes
- info
  added the new optional request property ///////////
- info
  added the new optional request property ///////////
- info
  added the new optional request property ///////////
- info
  added the new optional request property ///////////
- info
  added the new optional request property ///////////
- info
  added the new optional request property ///////////
- info
  added the new optional request property ///////////
- info
  added the new optional request property ///////////
- info
  added the new optional request property ///////////
- info
  added the new optional request property ///////////
- info
  added the new optional request property ///////////
- info
  added the new optional request property ///////////
- info
  added the new optional request property ///////////
- info
  added the new optional request property ///////////
- info
  added the new optional request property ///////////
- info
  added the new optional request property ///////////
- info
  added the new optional request property //////
- info
  added the new optional request property //////
- info
  added the new optional request property //////
- info
  added the new optional request property //////
- info
  added the new optional request property //////
- info
  added the new optional request property //////
- info
  added the new optional request property //////
- info
  added the new optional request property //////
- info
  added the new optional request property //////
- info
  added the new optional request property //////
- info
  added the new optional request property //////
- info
  added the new optional request property //////
- info
  added the new optional request property //////
- info
  added the new optional request property //////
- info
  added the new optional request property //////
- info
  added the new optional request property //////
Sep 29, 2025167acf59c7c3 Revision changes
- info
  added the new optional request property ///////////
- info
  added the new optional request property ///////////
- info
  added the new optional request property ///////////
- info
  added the new optional request property ///////////
- info
  added the new optional request property //////
- info
  added the new optional request property //////
- info
  added the new optional request property //////
- info
  added the new optional request property //////
Aug 15, 2025a3fce07a60cf Revision changes
- breaking
  added subschema #2 to the / request property allOf list
- breaking
  added subschema #2 to the request property allOf list
- breaking
  removed subschema #1 from the ////////// request property anyOf list
- breaking
  removed subschema #1 from the ////////// request property anyOf list
- breaking
  removed subschema #1: function_id subschema #2: project_slug subschema #3: global_function subschema #4: prompt_session_id subschema #5: inline_code subschema #6: inline_function subschema #7: inline_prompt from the / request property anyOf list
- breaking
  removed subschema #1: span_parent_struct subschema #2 from the request property anyOf list
- breaking
  the request property // became not nullable
- breaking
  the request property // became not nullable
- info
  added to the ////////// request property anyOf list
- info
  added to the ////////// request property anyOf list
- info
  the request property default value 10 was added
Aug 7, 20257b003086a539 Revision changes
- info
  added the new optional request property ///////
- info
  added the new optional request property ///////
- info
  added the new optional request property //////
- info
  added the new optional request property //////
- info
  added the new minimal enum value to the request property ///////
- info
  added the new minimal enum value to the request property ///////
- info
  added the new minimal enum value to the request property //////
- info
  added the new minimal enum value to the request property //////
Jul 16, 202587d9da616538 Revision changes
- info
  added the new optional request property ///
- info
  added the new optional request property ///
- info
  added the new optional request property //
- info
  added the new optional request property //
Jul 16, 202529c3853d6db5 Revision changes
- info
  added subschema #6: developer to the /////// request property anyOf list
- info
  added subschema #6: developer to the /////// request property anyOf list
- info
  added subschema #6: developer to the ////// request property anyOf list
- info
  added subschema #6: developer to the ////// request property anyOf list
Jun 17, 202577a2d5797d94 Revision changes
- breaking
  removed from the ////////// request property anyOf list
- breaking
  removed subschema #3: nullableVariant from the /// request property anyOf list
- breaking
  the request property ////// became not nullable
- breaking
  the request property //////// became not nullable
- breaking
  the request property //////// became required
- breaking
  the ////// request property type changed from object to no type
- warning
  removed subschema #2 from the / request property allOf list
- warning
  removed the request property ///////
- info
  added the new optional request property
- info
  added the new optional request property
- info
  added the new optional request property //
- info
  added the new optional request property //
- info
  added the new optional request property
- info
  added the new optional request property
- info
  added the new optional request property
- info
  added the new optional request property
- info
  added the new optional request property
- info
  added the new optional request property
- info
  added the new optional request property
- info
  added the new optional request property
- info
  added the new optional request property
- info
  added the new optional request property //////
- info
  added the new optional request property //////
- info
  added the new optional request property ////////
- info
  added the new optional request property
- info
  added the new optional request property
- info
  added subschema #1 to the ////////// request property anyOf list
- info
  added subschema #1: function_id subschema #2: project_slug subschema #3: global_function subschema #4: prompt_session_id subschema #5: inline_code subschema #6: inline_function subschema #7: inline_prompt to the / request property anyOf list
- info
  added subschema #1: json_object subschema #2: json_schema subschema #3: text subschema #4 to the ////// request property anyOf list
- info
  added subschema #3 to the /// request property anyOf list
- info
  added subschema #3: dataset_rows to the request property anyOf list
- info
  added subschema #3: required to the ////// request property anyOf list
- info
  added subschema #6: inline_function to the request property anyOf list
- info
  the request property //////// became nullable
- info
  the request property default value was removed
- info
  the request property default value was removed
- info
  the request property default value was removed
- info
  request property //////// list-of-types was widened by adding types array to media type application/json
- info
  request property //////// list-of-types was widened by adding types array and null to media type application/json
- info
  request property //////// list-of-types was widened by adding types array to media type application/json
Sep 30, 2024ac2727014d93 Revision changes
- breaking
  removed subschema #1 subschema #2 from the request property anyOf list
- breaking
  removed subschema #1 subschema #2 subschema #3 subschema #4 subschema #5 subschema #6 from the ///// request property anyOf list
- breaking
  removed subschema #1 subschema #2 subschema #3 subschema #4 subschema #5 subschema #6 from the request property anyOf list
- info
  added subschema #1: dataset_id subschema #2: project_dataset_name to the request property anyOf list
- info
  added subschema #1: function_id subschema #2: project_slug subschema #3: global_function subschema #4: prompt_session_id subschema #5: inline_code subschema #6: inline_prompt to the ///// request property anyOf list
- info
  added subschema #1: function_id subschema #2: project_slug subschema #3: global_function subschema #4: prompt_session_id subschema #5: inline_code subschema #6: inline_prompt to the request property anyOf list
- info
  the request property / became nullable
Sep 25, 202468955d9009ff Revision changes
- breaking
  added subschema #2 to the / request property allOf list
- breaking
  removed subschema #1 subschema #2 subschema #3 subschema #4 subschema #5 subschema #6 from the / request property anyOf list
Sep 24, 2024a774951c03d7 Revision changes
- breaking
  removed subschema #5 from the / request property anyOf list
- breaking
  removed subschema #5 from the request property anyOf list
- info
  added subschema #5 subschema #6 to the / request property anyOf list
- info
  added subschema #5 subschema #6 to the request property anyOf list
Sep 21, 20241e677afe2f7b Revision changes
- info
  added subschema #5 to the / request property anyOf list
- info
  added subschema #5 to the request property anyOf list
Jul 30, 20249635568dd298 Revision changes
- info
  endpoint added