Create Eval
Request body
Example request
{
"messages": [
{
"content": "The weather in San Francisco is sunny.",
"toolCalls": [
{
"name": "get_weather"
}
]
}
],
"name": "Verified User Flow Eval",
"description": "This eval checks if the user flow is verified.",
"type": "chat.mockConversation"
}Response
Example response
{
"messages": [
{
"content": "The weather in San Francisco is sunny.",
"toolCalls": [
{
"name": "get_weather"
}
]
}
],
"name": "Verified User Flow Eval",
"description": "This eval checks if the user flow is verified.",
"type": "chat.mockConversation"
}Changes
Changed in 10 of the 57 revisions of this API.9577
- ●
added the new
gemini-3-flash-previewenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/allOf[AssistantMessageJudgePlanAI]/model/oneOf[subschema #3: EvalGoogleModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gemini-3-flash-previewenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/oneOf[subschema #1: AssistantMessageJudgePlanAI]/model/oneOf[subschema #3: EvalGoogleModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gemini-3-flash-previewenum value to themessages/items/oneOf[subschema #6: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #3: EvalGoogleModel]/modelresponse property for the response status201response-property-enum-value-added
- ○
added the new
gemini-3-flash-previewenum value to the request propertymessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/allOf[AssistantMessageJudgePlanAI]/model/oneOf[subschema #3: EvalGoogleModel]/modelrequest-property-enum-value-added
- ○
added the new
gemini-3-flash-previewenum value to the request propertymessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/oneOf[subschema #1: AssistantMessageJudgePlanAI]/model/oneOf[subschema #3: EvalGoogleModel]/modelrequest-property-enum-value-added
- ○
added the new
gemini-3-flash-previewenum value to the request propertymessages/items/oneOf[subschema #6: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #3: EvalGoogleModel]/modelrequest-property-enum-value-added
- ●
- ●
added the new
gpt-4.1-2025-04-14:germanywestcentralenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/allOf[AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-2025-04-14:germanywestcentralenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/oneOf[subschema #1: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-2025-04-14:germanywestcentralenum value to themessages/items/oneOf[subschema #6: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-2025-04-14:polandcentralenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/allOf[AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-2025-04-14:polandcentralenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/oneOf[subschema #1: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-2025-04-14:polandcentralenum value to themessages/items/oneOf[subschema #6: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-2025-04-14:spaincentralenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/allOf[AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-2025-04-14:spaincentralenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/oneOf[subschema #1: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-2025-04-14:spaincentralenum value to themessages/items/oneOf[subschema #6: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-2025-04-14:westeuropeenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/allOf[AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-2025-04-14:westeuropeenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/oneOf[subschema #1: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-2025-04-14:westeuropeenum value to themessages/items/oneOf[subschema #6: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-mini-2025-04-14:germanywestcentralenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/allOf[AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-mini-2025-04-14:germanywestcentralenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/oneOf[subschema #1: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-mini-2025-04-14:germanywestcentralenum value to themessages/items/oneOf[subschema #6: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-mini-2025-04-14:polandcentralenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/allOf[AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-mini-2025-04-14:polandcentralenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/oneOf[subschema #1: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-mini-2025-04-14:polandcentralenum value to themessages/items/oneOf[subschema #6: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-mini-2025-04-14:spaincentralenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/allOf[AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-mini-2025-04-14:spaincentralenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/oneOf[subschema #1: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-mini-2025-04-14:spaincentralenum value to themessages/items/oneOf[subschema #6: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-mini-2025-04-14:westeuropeenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/allOf[AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-mini-2025-04-14:westeuropeenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/oneOf[subschema #1: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4.1-mini-2025-04-14:westeuropeenum value to themessages/items/oneOf[subschema #6: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4o-2024-11-20:germanywestcentralenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/allOf[AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4o-2024-11-20:germanywestcentralenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/oneOf[subschema #1: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4o-2024-11-20:germanywestcentralenum value to themessages/items/oneOf[subschema #6: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4o-2024-11-20:polandcentralenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/allOf[AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4o-2024-11-20:polandcentralenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/oneOf[subschema #1: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4o-2024-11-20:polandcentralenum value to themessages/items/oneOf[subschema #6: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4o-2024-11-20:spaincentralenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/allOf[AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4o-2024-11-20:spaincentralenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/oneOf[subschema #1: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4o-2024-11-20:spaincentralenum value to themessages/items/oneOf[subschema #6: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4o-2024-11-20:westeuropeenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/allOf[AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4o-2024-11-20:westeuropeenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/oneOf[subschema #1: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-4o-2024-11-20:westeuropeenum value to themessages/items/oneOf[subschema #6: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-5-chat-latestenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/allOf[AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-5-chat-latestenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/oneOf[subschema #1: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-5-chat-latestenum value to themessages/items/oneOf[subschema #6: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-5.2enum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/allOf[AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-5.2enum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/oneOf[subschema #1: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-5.2enum value to themessages/items/oneOf[subschema #6: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-5.2-chat-latestenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/allOf[AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-5.2-chat-latestenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/oneOf[subschema #1: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-5.2-chat-latestenum value to themessages/items/oneOf[subschema #6: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
This revision's changelog hit the recording limit, so it may not list every change to this endpoint.
- ●
- ●
added the new
gpt-5.1enum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/allOf[AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-5.1enum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/oneOf[subschema #1: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-5.1enum value to themessages/items/oneOf[subschema #6: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-5.1-chat-latestenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/allOf[AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-5.1-chat-latestenum value to themessages/items/oneOf[subschema #4: ChatEvalToolResponseMessageEvaluation]/judgePlan/oneOf[subschema #1: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
- ●
added the new
gpt-5.1-chat-latestenum value to themessages/items/oneOf[subschema #6: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #1: EvalOpenAIModel]/modelresponse property for the response status201response-property-enum-value-added
This revision's changelog hit the recording limit, so it may not list every change to this endpoint.
- ●
- ▲
added
ChatEvalToolResponseMessageEvaluationto themessages/items/response propertyoneOflist for the response status201response-property-one-of-added
- ○
added
ChatEvalToolResponseMessageEvaluationto themessages/items/request propertyoneOflistrequest-property-one-of-added
- ▲
- ▲
the
descriptionresponse property's minLength was decreased from1to0for the response status201response-property-min-length-decreased
- ●
added the new
claude-haiku-4-5-20251001enum value to themessages/items/oneOf[subschema #5: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #2: EvalAnthropicModel]/modelresponse property for the response status201response-property-enum-value-added
- ○
added the new
claude-haiku-4-5-20251001enum value to the request propertymessages/items/oneOf[subschema #5: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #2: EvalAnthropicModel]/modelrequest-property-enum-value-added
- ○
the
descriptionrequest property's minLength was decreased from1to0request-property-min-length-decreased
- ▲
- ▲
the
messages/items/oneOf[subschema #2: ChatEvalSystemMessageMock]/contentresponse property's maxLength was unset from100000000for the response status201response-property-max-length-unset
- ▲
the
messages/items/oneOf[subschema #3: ChatEvalToolResponseMessageMock]/contentresponse property's maxLength was unset from1000for the response status201response-property-max-length-unset
- ●
added the new
claude-sonnet-4-5-20250929enum value to themessages/items/oneOf[subschema #5: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #2: EvalAnthropicModel]/modelresponse property for the response status201response-property-enum-value-added
This revision's changelog hit the recording limit, so it may not list every change to this endpoint.
This revision also has 3 changes that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog
- ▲
- ▲
removed the enum value
claude-sonnet-4-5-20250929of the request propertymessages/items/oneOf[subschema #5: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #2: EvalAnthropicModel]/modelrequest-property-enum-value-removed
This revision's changelog hit the recording limit, so it may not list every change to this endpoint.
- ▲
- ●
added the new
claude-sonnet-4-5-20250929enum value to themessages/items/oneOf[subschema #5: ChatEvalAssistantMessageEvaluation]/judgePlan/oneOf[subschema #3: AssistantMessageJudgePlanAI]/model/oneOf[subschema #2: EvalAnthropicModel]/modelresponse property for the response status201response-property-enum-value-added
This revision's changelog hit the recording limit, so it may not list every change to this endpoint.
This revision also has 2 changes that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog
- ●
- ▲
added the new required request property
messagesnew-required-request-property
- ▲
added the new required request property
typenew-required-request-property
- ▲
removed
CreateEvalDTOfrom the request bodyoneOflistrequest-body-one-of-removed
- ▲
the request's body type/format changed from
/toobject/request-body-type-changed
This revision's changelog hit the recording limit, so it may not list every change to this endpoint.
- ▲
- ○
endpoint added
endpoint-added
- ○
Of the 57 revisions, 2 have no diff computed and 9 have a changelog that could not be searched.