--- title: "Run an LLM inference" method: POST path: "/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/inference" tags: ["LLM Observability"] --- # Run an LLM inference `POST /api/v2/llm-obs/v1/integrations/{integration}/{account_id}/inference` ## Request - Base URL: `https://api.datadoghq.com` - URL: `https://api.datadoghq.com/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/inference` - Auth: API key in header `DD-API-KEY` and API key in header `DD-APPLICATION-KEY` Run an LLM inference request through the specified integration and account, returning the model response and token usage. ## Path parameters - `integration` 'openai' | 'amazon_bedrock' | 'anthropic' | 'azure_openai' | 'vertex_ai' | 'llmproxy', required: The name of a supported LLM provider integration. - `account_id` string, required ## Request body - LLMObsIntegrationInferenceRequest: Parameters for an LLM inference request. - `anthropic_metadata` LLMObsAnthropicMetadata: Anthropic-specific metadata for an inference request. - `effort` 'low' | 'medium' | 'high' | 'max', nullable: The effort level for Anthropic inference. - `thinking` LLMObsAnthropicThinkingConfig: Configuration for Anthropic extended thinking feature. - `budget_tokens` integer, nullable: Maximum token budget for extended thinking. Required when type is `enabled`. - `type` 'enabled' | 'disabled' | 'adaptive', required: The thinking mode for Anthropic extended thinking. - `azure_openai_metadata` LLMObsAzureOpenAIMetadata: Azure OpenAI-specific metadata for an integration account or inference request. - `deployment_id` string: The Azure OpenAI deployment ID. - `model_version` string: The model version deployed in Azure. - `resource_name` string: The Azure OpenAI resource name. - `bedrock_metadata` LLMObsBedrockMetadata: Amazon Bedrock-specific metadata for an inference request. - `region` string: The AWS region for the Bedrock request. - `frequency_penalty` number, double, nullable: Penalty for token frequency to reduce repetition. - `json_schema` string, nullable: JSON schema for structured output, if supported by the model. - `max_completion_tokens` integer, nullable: Maximum number of completion tokens to generate (alternative to max_tokens for some providers). - `max_tokens` integer, nullable: Maximum number of tokens to generate. - `messages` LLMObsInferenceMessage[], required: List of messages in an inference conversation. - `content` string: Plain text content of the message. - `contents` LLMObsInferenceContent[]: List of structured content blocks in a message. - `type` string, required: The content block type. - `value` LLMObsInferenceContentValue, required: The typed value of a message content block. - `text` string: Plain text content. - `tool_call` LLMObsInferenceToolCall: A tool call made during LLM inference. - `arguments` object: The arguments passed to the tool. - `name` string: The name of the tool being called. - `tool_id` string: Unique identifier for the tool call. - `type` string: The type of tool call. - `tool_call_result` LLMObsInferenceToolResult: The result returned by a tool call during LLM inference. - `name` string: The name of the tool that produced this result. - `result` string: The result content returned by the tool. - `tool_id` string: Identifier matching the corresponding tool call. - `type` string: The type of tool result. - `id` string: Unique identifier for the message. - `role` string: The role of the message author. - `tool_calls` LLMObsInferenceToolCall[]: List of tool calls in a message. - `arguments` object: The arguments passed to the tool. - `name` string: The name of the tool being called. - `tool_id` string: Unique identifier for the tool call. - `type` string: The type of tool call. - `tool_results` LLMObsInferenceToolResult[]: List of tool results in a message. - `name` string: The name of the tool that produced this result. - `result` string: The result content returned by the tool. - `tool_id` string: Identifier matching the corresponding tool call. - `type` string: The type of tool result. - `model_id` string, required: The model identifier to use for inference. - `openai_metadata` LLMObsOpenAIMetadata: OpenAI-specific metadata for an inference request. - `reasoning_effort` 'none' | 'low' | 'medium' | 'high' | 'xhigh', nullable: The reasoning effort level for OpenAI models that support it. - `reasoning_summary` 'auto' | 'concise' | 'detailed', nullable: The verbosity of the reasoning summary. - `presence_penalty` number, double, nullable: Penalty for token presence to encourage topic diversity. - `temperature` number, double, nullable: Sampling temperature between 0 and 2. Higher values produce more random output. - `tools` LLMObsInferenceTool[]: List of tools available to the model. - `function` LLMObsInferenceFunction, required: A function definition for a tool available to the model. - `description` string: A description of what the function does. - `name` string, required: The name of the function. - `parameters` object, required: JSON schema describing the function parameters. - `type` string, required: The type of tool. - `top_k` integer, nullable: Top-K sampling parameter. - `top_p` number, double, nullable: Nucleus sampling probability mass. - `vertex_ai_metadata` LLMObsVertexAIMetadata: Vertex AI-specific metadata for an integration account or inference request. - `location` string: The Vertex AI region. - `project` string: The Google Cloud project ID. - `project_ids` string[]: List of Google Cloud project IDs available to the service account. ## Response `200` OK - LLMObsIntegrationInferenceResponse: The result of an LLM inference request, including input parameters and the model response. - `anthropic_metadata` LLMObsAnthropicMetadata: Anthropic-specific metadata for an inference request. - `effort` 'low' | 'medium' | 'high' | 'max', nullable: The effort level for Anthropic inference. - `thinking` LLMObsAnthropicThinkingConfig: Configuration for Anthropic extended thinking feature. - `budget_tokens` integer, nullable: Maximum token budget for extended thinking. Required when type is `enabled`. - `type` 'enabled' | 'disabled' | 'adaptive', required: The thinking mode for Anthropic extended thinking. - `azure_openai_metadata` LLMObsAzureOpenAIMetadata: Azure OpenAI-specific metadata for an integration account or inference request. - `deployment_id` string: The Azure OpenAI deployment ID. - `model_version` string: The model version deployed in Azure. - `resource_name` string: The Azure OpenAI resource name. - `bedrock_metadata` LLMObsBedrockMetadata: Amazon Bedrock-specific metadata for an inference request. - `region` string: The AWS region for the Bedrock request. - `error_response` LLMObsInferenceErrorResponse: Error details returned when an inference provider returns an error. - `message` string, required: A human-readable description of the error. - `type` string, required: The provider-specific error type. - `frequency_penalty` number, double, nullable: Frequency penalty that was applied. - `json_schema` string, nullable: JSON schema that was applied for structured output. - `max_completion_tokens` integer, nullable: Maximum number of completion tokens that were configured. - `max_tokens` integer, nullable: Maximum number of tokens that were configured. - `messages` LLMObsInferenceMessage[], required: List of messages in an inference conversation. - `content` string: Plain text content of the message. - `contents` LLMObsInferenceContent[]: List of structured content blocks in a message. - `type` string, required: The content block type. - `value` LLMObsInferenceContentValue, required: The typed value of a message content block. - `text` string: Plain text content. - `tool_call` LLMObsInferenceToolCall: A tool call made during LLM inference. - `arguments` object: The arguments passed to the tool. - `name` string: The name of the tool being called. - `tool_id` string: Unique identifier for the tool call. - `type` string: The type of tool call. - `tool_call_result` LLMObsInferenceToolResult: The result returned by a tool call during LLM inference. - `name` string: The name of the tool that produced this result. - `result` string: The result content returned by the tool. - `tool_id` string: Identifier matching the corresponding tool call. - `type` string: The type of tool result. - `id` string: Unique identifier for the message. - `role` string: The role of the message author. - `tool_calls` LLMObsInferenceToolCall[]: List of tool calls in a message. - `arguments` object: The arguments passed to the tool. - `name` string: The name of the tool being called. - `tool_id` string: Unique identifier for the tool call. - `type` string: The type of tool call. - `tool_results` LLMObsInferenceToolResult[]: List of tool results in a message. - `name` string: The name of the tool that produced this result. - `result` string: The result content returned by the tool. - `tool_id` string: Identifier matching the corresponding tool call. - `type` string: The type of tool result. - `model_id` string, required: The model identifier used for inference. - `openai_metadata` LLMObsOpenAIMetadata: OpenAI-specific metadata for an inference request. - `reasoning_effort` 'none' | 'low' | 'medium' | 'high' | 'xhigh', nullable: The reasoning effort level for OpenAI models that support it. - `reasoning_summary` 'auto' | 'concise' | 'detailed', nullable: The verbosity of the reasoning summary. - `presence_penalty` number, double, nullable: Presence penalty that was applied. - `response` LLMObsInferenceRunResult, required: The output of a completed LLM inference call. - `assessment` string, nullable, required: An optional assessment of the inference output quality. - `content` string, required: The text content of the model response. - `finish_reason` string, required: The reason the model stopped generating tokens. - `inference_codes` LLMObsInferenceCode[], required: List of generated code snippets for the inference configuration. - `code` string, required: The generated code content. - `id` string, required: Unique identifier for the code snippet. - `type` string, required: The programming language or SDK type of the code snippet. - `input_tokens` integer, required: Number of input tokens consumed. - `internal_reasoning` LLMObsInternalReasoning: The model's internal reasoning or thinking output, if available. - `reasoning_tokens` integer, nullable: Number of tokens used for internal reasoning. - `text` string, required: The reasoning text produced by the model. - `latency` integer, required: Request latency in milliseconds. - `output_tokens` integer, required: Number of output tokens generated. - `tools` LLMObsInferenceTool[], required: List of tools available to the model. - `function` LLMObsInferenceFunction, required: A function definition for a tool available to the model. - `description` string: A description of what the function does. - `name` string, required: The name of the function. - `parameters` object, required: JSON schema describing the function parameters. - `type` string, required: The type of tool. - `total_tokens` integer, required: Total tokens used (input plus output). - `temperature` number, double, nullable: Sampling temperature that was used. - `tools` LLMObsInferenceTool[]: List of tools available to the model. - `function` LLMObsInferenceFunction, required: A function definition for a tool available to the model. - `description` string: A description of what the function does. - `name` string, required: The name of the function. - `parameters` object, required: JSON schema describing the function parameters. - `type` string, required: The type of tool. - `top_k` integer, nullable: Top-K sampling parameter that was used. - `top_p` number, double, nullable: Nucleus sampling parameter that was used. - `vertex_ai_metadata` LLMObsVertexAIMetadata: Vertex AI-specific metadata for an integration account or inference request. - `location` string: The Vertex AI region. - `project` string: The Google Cloud project ID. - `project_ids` string[]: List of Google Cloud project IDs available to the service account. ## Other responses - `400`: Bad Request - `401`: Unauthorized - `403`: Forbidden - `429`: Too many requests - `500`: Internal Server Error --- [API](https://skmtc.dev/datadog/apis/datadog-api-v2-collection.md) - [All operations](https://skmtc.dev/datadog/apis/datadog-api-v2-collection/llms.txt) - [OpenAPI document](https://skmtc.dev/datadog/apis/datadog-api-v2-collection/revisions/da68bf029e4c?raw)