--- title: "Create the structure of an evaluation that can be used to test a model's performance. An evaluation is a set of testing criteria and a datasource. After creating an evaluation, you can run it on different models and model parameters. We support several types of graders and datasources. For more information, see the [Evals guide](/docs/guides/evals)." method: POST path: "/evals" tags: ["Evals"] --- # Create the structure of an evaluation that can be used to test a model's performance. An evaluation is a set of testing criteria and a datasource. After creating an evaluation, you can run it on different models and model parameters. We support several types of graders and datasources. For more information, see the [Evals guide](/docs/guides/evals). `POST /evals` ## Request body - CreateEvalRequest - `name` string — The name of the evaluation. - `metadata` Metadata, nullable — Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters. - `data_source_config` union, required — The configuration for the data source used for the evaluation runs. - object — A CustomDataSourceConfig object that defines the schema for the data source used for the evaluation runs. This schema is used to define the shape of the data that will be: - Used to define your testing criteria and - What data is required when creating a run - `type` 'custom', required — The type of data source. Always `custom`. - `item_schema` object, required — The json schema for each row in the data source. - `include_sample_schema` boolean — Whether the eval should expect you to populate the sample namespace (ie, by generating responses off of your data source) - object — A data source config which specifies the metadata property of your stored completions query. This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc. - `type` 'logs', required — The type of data source. Always `logs`. - `metadata` object — Metadata filters for the logs data source. - `testing_criteria` union[], required — A list of graders for all eval runs in this group. - union - CreateEvalLabelModelGrader — A LabelModelGrader object which uses a model to assign labels to each item in the evaluation. - `type` 'label_model', required — The object type, which is always `label_model`. - `name` string, required — The name of the grader. - `model` string, required — The model to use for the evaluation. Must support structured outputs. - `input` CreateEvalItem[], required — A list of chat messages forming the prompt or context. May include variable references to the "item" namespace, ie {{item.name}}. - union — A chat message that makes up the prompt or context. May include variable references to the "item" namespace, ie {{item.name}}. - object - … - object — A message input to the model with a role indicating instruction following hierarchy. Instructions given with the `developer` or `system` role take precedence over instructions given with the `user` role. Messages with the `assistant` role are presumed to have been generated by the model in previous interactions. - … - `labels` string[], required — The labels to classify to each item in the evaluation. - `passing_labels` string[], required — The labels that indicate a passing result. Must be a subset of labels. - EvalStringCheckGrader — A StringCheckGrader object that performs a string comparison between input and reference using a specified operation. - `type` 'string_check', required — The object type, which is always `string_check`. - `name` string, required — The name of the grader. - `input` string, required — The input text. This may include template strings. - `reference` string, required — The reference text. This may include template strings. - `operation` 'eq' | 'ne' | 'like' | 'ilike', required — The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`. - EvalTextSimilarityGrader — A TextSimilarityGrader object which grades text based on similarity metrics. - `type` 'text_similarity', required — The type of grader. - `name` string — The name of the grader. - `input` string, required — The text being graded. - `reference` string, required — The text being graded against. - `pass_threshold` number, required — A float score where a value greater than or equal indicates a passing grade. - `evaluation_metric` 'fuzzy_match' | 'bleu' | 'gleu' | 'meteor' | 'rouge_1' | 'rouge_2' | 'rouge_3' | 'rouge_4' | 'rouge_5' | 'rouge_l', required — The evaluation metric to use. One of `fuzzy_match`, `bleu`, `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`, or `rouge_l`. - EvalPythonGrader — A PythonGrader object that runs a python script on the input. - `type` 'python', required — The object type, which is always `python`. - `name` string, required — The name of the grader. - `source` string, required — The source code of the python script. - `pass_threshold` number — The threshold for the score. - `image_tag` string — The image tag to use for the python script. - EvalScoreModelGrader — A ScoreModelGrader object that uses a model to assign a score to the input. - `type` 'score_model', required — The object type, which is always `score_model`. - `name` string, required — The name of the grader. - `model` string, required — The model to use for the evaluation. - `sampling_params` object — The sampling parameters for the model. - `input` EvalItem[], required — The input text. This may include template strings. - `role` 'user' | 'assistant' | 'system' | 'developer', required — The role of the message input. One of `user`, `assistant`, `system`, or `developer`. - `content` union, required — Text inputs to the model - can contain template strings. - string — A text input to the model. - InputTextContent — A text input to the model. - … - object — A text output from the model. - … - `type` 'message' — The type of the message input. Always `message`. - `pass_threshold` number — The threshold for the score. - `range` number[] — The range of the score. Defaults to `[0, 1]`. ## Response `201` OK - Eval — An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like: - Improve the quality of my chatbot - See how well my chatbot handles customer support - Check if o3-mini is better at my usecase than gpt-4o - `object` 'eval', required — The object type. - `id` string, required — Unique identifier for the evaluation. - `name` string, required — The name of the evaluation. - `data_source_config` union, required — Configuration of data sources used in runs of the evaluation. - object — A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces. The response schema defines the shape of the data that will be: - Used to define your testing criteria and - What data is required when creating a run - `type` 'custom', required — The type of data source. Always `custom`. - `schema` object, required — The json schema for the run data source items. Learn how to build JSON schemas [here](https://json-schema.org/). - object — A StoredCompletionsDataSourceConfig which specifies the metadata property of your stored completions query. This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc. The schema returned by this data source config is used to defined what variables are available in your evals. `item` and `sample` are both defined when using this data source config. - `type` 'stored_completions', required — The type of data source. Always `stored_completions`. - `metadata` Metadata, nullable — Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters. - `schema` object, required — The json schema for the run data source items. Learn how to build JSON schemas [here](https://json-schema.org/). - `testing_criteria` union[], required — A list of testing criteria. - union - EvalLabelModelGrader — A LabelModelGrader object which uses a model to assign labels to each item in the evaluation. - `type` 'label_model', required — The object type, which is always `label_model`. - `name` string, required — The name of the grader. - `model` string, required — The model to use for the evaluation. Must support structured outputs. - `input` EvalItem[], required - `role` 'user' | 'assistant' | 'system' | 'developer', required — The role of the message input. One of `user`, `assistant`, `system`, or `developer`. - `content` union, required — Text inputs to the model - can contain template strings. - string — A text input to the model. - InputTextContent — A text input to the model. - … - object — A text output from the model. - … - `type` 'message' — The type of the message input. Always `message`. - `labels` string[], required — The labels to assign to each item in the evaluation. - `passing_labels` string[], required — The labels that indicate a passing result. Must be a subset of labels. - EvalStringCheckGrader — A StringCheckGrader object that performs a string comparison between input and reference using a specified operation. - `type` 'string_check', required — The object type, which is always `string_check`. - `name` string, required — The name of the grader. - `input` string, required — The input text. This may include template strings. - `reference` string, required — The reference text. This may include template strings. - `operation` 'eq' | 'ne' | 'like' | 'ilike', required — The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`. - EvalTextSimilarityGrader — A TextSimilarityGrader object which grades text based on similarity metrics. - `type` 'text_similarity', required — The type of grader. - `name` string — The name of the grader. - `input` string, required — The text being graded. - `reference` string, required — The text being graded against. - `pass_threshold` number, required — A float score where a value greater than or equal indicates a passing grade. - `evaluation_metric` 'fuzzy_match' | 'bleu' | 'gleu' | 'meteor' | 'rouge_1' | 'rouge_2' | 'rouge_3' | 'rouge_4' | 'rouge_5' | 'rouge_l', required — The evaluation metric to use. One of `fuzzy_match`, `bleu`, `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`, or `rouge_l`. - EvalPythonGrader — A PythonGrader object that runs a python script on the input. - `type` 'python', required — The object type, which is always `python`. - `name` string, required — The name of the grader. - `source` string, required — The source code of the python script. - `pass_threshold` number — The threshold for the score. - `image_tag` string — The image tag to use for the python script. - EvalScoreModelGrader — A ScoreModelGrader object that uses a model to assign a score to the input. - `type` 'score_model', required — The object type, which is always `score_model`. - `name` string, required — The name of the grader. - `model` string, required — The model to use for the evaluation. - `sampling_params` object — The sampling parameters for the model. - `input` EvalItem[], required — The input text. This may include template strings. - `role` 'user' | 'assistant' | 'system' | 'developer', required — The role of the message input. One of `user`, `assistant`, `system`, or `developer`. - `content` union, required — Text inputs to the model - can contain template strings. - string — A text input to the model. - InputTextContent — A text input to the model. - … - object — A text output from the model. - … - `type` 'message' — The type of the message input. Always `message`. - `pass_threshold` number — The threshold for the score. - `range` number[] — The range of the score. Defaults to `[0, 1]`. - `created_at` integer, required — The Unix timestamp (in seconds) for when the eval was created. - `metadata` Metadata, nullable, required — Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters. ## Changes - **2025-04-29** `6a6c681b1820` — 1 info - endpoint added - **2024-08-23** `5856f8b0c850` — 1 breaking - api path removed without deprecation [Change history](https://skmtc.dev/openai/apis/openapi/changes/evals/post.md) --- [API](https://skmtc.dev/openai/apis/openapi.md) · [All operations](https://skmtc.dev/openai/apis/openapi/llms.txt) · [OpenAPI document](https://skmtc-service-production.skmtc.workers.dev/v1/apis/openai/openapi/revisions/6a6c681b1820/schema)