--- title: "Run an LLM inference" method: POST path: "/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/inference" tags: ["LLM Observability"] --- # Run an LLM inference `POST /api/v2/llm-obs/v1/integrations/{integration}/{account_id}/inference` Run an LLM inference request through the specified integration and account, returning the model response and token usage. ## Path parameters - `integration` 'openai' | 'amazon_bedrock' | 'anthropic' | 'azure_openai' | 'vertex_ai' | 'llmproxy', required — The name of a supported LLM provider integration. - `account_id` string, required ## Request body - LLMObsIntegrationInferenceRequest — Parameters for an LLM inference request. - `anthropic_metadata` LLMObsAnthropicMetadata — Anthropic-specific metadata for an inference request. - `effort` 'low' | 'medium' | 'high' | 'max', nullable — The effort level for Anthropic inference. - `thinking` LLMObsAnthropicThinkingConfig — Configuration for Anthropic extended thinking feature. - `budget_tokens` integer, nullable — Maximum token budget for extended thinking. Required when type is `enabled`. - `type` 'enabled' | 'disabled' | 'adaptive', required — The thinking mode for Anthropic extended thinking. - `azure_openai_metadata` LLMObsAzureOpenAIMetadata — Azure OpenAI-specific metadata for an integration account or inference request. - `deployment_id` string — The Azure OpenAI deployment ID. - `model_version` string — The model version deployed in Azure. - `resource_name` string — The Azure OpenAI resource name. - `bedrock_metadata` LLMObsBedrockMetadata — Amazon Bedrock-specific metadata for an inference request. - `region` string — The AWS region for the Bedrock request. - `frequency_penalty` number, double, nullable — Penalty for token frequency to reduce repetition. - `json_schema` string, nullable — JSON schema for structured output, if supported by the model. - `max_completion_tokens` integer, nullable — Maximum number of completion tokens to generate (alternative to max_tokens for some providers). - `max_tokens` integer, nullable — Maximum number of tokens to generate. - `messages` LLMObsInferenceMessage[], required — List of messages in an inference conversation. - `content` string — Plain text content of the message. - `contents` LLMObsInferenceContent[] — List of structured content blocks in a message. - `type` string, required — The content block type. - `value` LLMObsInferenceContentValue, required — The typed value of a message content block. - `text` string — Plain text content. - `tool_call` LLMObsInferenceToolCall — A tool call made during LLM inference. - `arguments` object — The arguments passed to the tool. - `name` string — The name of the tool being called. - `tool_id` string — Unique identifier for the tool call. - `type` string — The type of tool call. - `tool_call_result` LLMObsInferenceToolResult — The result returned by a tool call during LLM inference. - `name` string — The name of the tool that produced this result. - `result` string — The result content returned by the tool. - `tool_id` string — Identifier matching the corresponding tool call. - `type` string — The type of tool result. - `id` string — Unique identifier for the message. - `role` string — The role of the message author. - `tool_calls` LLMObsInferenceToolCall[] — List of tool calls in a message. - `arguments` object — The arguments passed to the tool. - `name` string — The name of the tool being called. - `tool_id` string — Unique identifier for the tool call. - `type` string — The type of tool call. - `tool_results` LLMObsInferenceToolResult[] — List of tool results in a message. - `name` string — The name of the tool that produced this result. - `result` string — The result content returned by the tool. - `tool_id` string — Identifier matching the corresponding tool call. - `type` string — The type of tool result. - `model_id` string, required — The model identifier to use for inference. - `openai_metadata` LLMObsOpenAIMetadata — OpenAI-specific metadata for an inference request. - `reasoning_effort` 'none' | 'low' | 'medium' | 'high' | 'xhigh', nullable — The reasoning effort level for OpenAI models that support it. - `reasoning_summary` 'auto' | 'concise' | 'detailed', nullable — The verbosity of the reasoning summary. - `presence_penalty` number, double, nullable — Penalty for token presence to encourage topic diversity. - `temperature` number, double, nullable — Sampling temperature between 0 and 2. Higher values produce more random output. - `tools` LLMObsInferenceTool[] — List of tools available to the model. - `function` LLMObsInferenceFunction, required — A function definition for a tool available to the model. - `description` string — A description of what the function does. - `name` string, required — The name of the function. - `parameters` object, required — JSON schema describing the function parameters. - `type` string, required — The type of tool. - `top_k` integer, nullable — Top-K sampling parameter. - `top_p` number, double, nullable — Nucleus sampling probability mass. - `vertex_ai_metadata` LLMObsVertexAIMetadata — Vertex AI-specific metadata for an integration account or inference request. - `location` string — The Vertex AI region. - `project` string — The Google Cloud project ID. - `project_ids` string[] — List of Google Cloud project IDs available to the service account. ## Response `200` OK - LLMObsIntegrationInferenceResponse — The result of an LLM inference request, including input parameters and the model response. - `anthropic_metadata` LLMObsAnthropicMetadata — Anthropic-specific metadata for an inference request. - `effort` 'low' | 'medium' | 'high' | 'max', nullable — The effort level for Anthropic inference. - `thinking` LLMObsAnthropicThinkingConfig — Configuration for Anthropic extended thinking feature. - `budget_tokens` integer, nullable — Maximum token budget for extended thinking. Required when type is `enabled`. - `type` 'enabled' | 'disabled' | 'adaptive', required — The thinking mode for Anthropic extended thinking. - `azure_openai_metadata` LLMObsAzureOpenAIMetadata — Azure OpenAI-specific metadata for an integration account or inference request. - `deployment_id` string — The Azure OpenAI deployment ID. - `model_version` string — The model version deployed in Azure. - `resource_name` string — The Azure OpenAI resource name. - `bedrock_metadata` LLMObsBedrockMetadata — Amazon Bedrock-specific metadata for an inference request. - `region` string — The AWS region for the Bedrock request. - `error_response` LLMObsInferenceErrorResponse — Error details returned when an inference provider returns an error. - `message` string, required — A human-readable description of the error. - `type` string, required — The provider-specific error type. - `frequency_penalty` number, double, nullable — Frequency penalty that was applied. - `json_schema` string, nullable — JSON schema that was applied for structured output. - `max_completion_tokens` integer, nullable — Maximum number of completion tokens that were configured. - `max_tokens` integer, nullable — Maximum number of tokens that were configured. - `messages` LLMObsInferenceMessage[], required — List of messages in an inference conversation. - `content` string — Plain text content of the message. - `contents` LLMObsInferenceContent[] — List of structured content blocks in a message. - `type` string, required — The content block type. - `value` LLMObsInferenceContentValue, required — The typed value of a message content block. - `text` string — Plain text content. - `tool_call` LLMObsInferenceToolCall — A tool call made during LLM inference. - `arguments` object — The arguments passed to the tool. - `name` string — The name of the tool being called. - `tool_id` string — Unique identifier for the tool call. - `type` string — The type of tool call. - `tool_call_result` LLMObsInferenceToolResult — The result returned by a tool call during LLM inference. - `name` string — The name of the tool that produced this result. - `result` string — The result content returned by the tool. - `tool_id` string — Identifier matching the corresponding tool call. - `type` string — The type of tool result. - `id` string — Unique identifier for the message. - `role` string — The role of the message author. - `tool_calls` LLMObsInferenceToolCall[] — List of tool calls in a message. - `arguments` object — The arguments passed to the tool. - `name` string — The name of the tool being called. - `tool_id` string — Unique identifier for the tool call. - `type` string — The type of tool call. - `tool_results` LLMObsInferenceToolResult[] — List of tool results in a message. - `name` string — The name of the tool that produced this result. - `result` string — The result content returned by the tool. - `tool_id` string — Identifier matching the corresponding tool call. - `type` string — The type of tool result. - `model_id` string, required — The model identifier used for inference. - `openai_metadata` LLMObsOpenAIMetadata — OpenAI-specific metadata for an inference request. - `reasoning_effort` 'none' | 'low' | 'medium' | 'high' | 'xhigh', nullable — The reasoning effort level for OpenAI models that support it. - `reasoning_summary` 'auto' | 'concise' | 'detailed', nullable — The verbosity of the reasoning summary. - `presence_penalty` number, double, nullable — Presence penalty that was applied. - `response` LLMObsInferenceRunResult, required — The output of a completed LLM inference call. - `assessment` string, nullable, required — An optional assessment of the inference output quality. - `content` string, required — The text content of the model response. - `finish_reason` string, required — The reason the model stopped generating tokens. - `inference_codes` LLMObsInferenceCode[], required — List of generated code snippets for the inference configuration. - `code` string, required — The generated code content. - `id` string, required — Unique identifier for the code snippet. - `type` string, required — The programming language or SDK type of the code snippet. - `input_tokens` integer, required — Number of input tokens consumed. - `internal_reasoning` LLMObsInternalReasoning — The model's internal reasoning or thinking output, if available. - `reasoning_tokens` integer, nullable — Number of tokens used for internal reasoning. - `text` string, required — The reasoning text produced by the model. - `latency` integer, required — Request latency in milliseconds. - `output_tokens` integer, required — Number of output tokens generated. - `tools` LLMObsInferenceTool[], required — List of tools available to the model. - `function` LLMObsInferenceFunction, required — A function definition for a tool available to the model. - `description` string — A description of what the function does. - `name` string, required — The name of the function. - `parameters` object, required — JSON schema describing the function parameters. - `type` string, required — The type of tool. - `total_tokens` integer, required — Total tokens used (input plus output). - `temperature` number, double, nullable — Sampling temperature that was used. - `tools` LLMObsInferenceTool[] — List of tools available to the model. - `function` LLMObsInferenceFunction, required — A function definition for a tool available to the model. - `description` string — A description of what the function does. - `name` string, required — The name of the function. - `parameters` object, required — JSON schema describing the function parameters. - `type` string, required — The type of tool. - `top_k` integer, nullable — Top-K sampling parameter that was used. - `top_p` number, double, nullable — Nucleus sampling parameter that was used. - `vertex_ai_metadata` LLMObsVertexAIMetadata — Vertex AI-specific metadata for an integration account or inference request. - `location` string — The Vertex AI region. - `project` string — The Google Cloud project ID. - `project_ids` string[] — List of Google Cloud project IDs available to the service account. ## Other responses - `400` — Bad Request - `401` — Unauthorized - `403` — Forbidden - `429` — Too many requests - `500` — Internal Server Error --- [API](https://skmtc.dev/datadog/apis/api-v2.md) · [All operations](https://skmtc.dev/datadog/apis/api-v2/llms.txt) · [OpenAPI document](https://skmtc-service-production.skmtc.workers.dev/v1/apis/datadog/api-v2/revisions/da68bf029e4c/schema)