Generate conversational responses with streaming support and multimodal capabilities.

This endpoint generates chat completions using language models, supporting both text-only conversations and vision inputs. Includes support for function calling, streaming responses, and automatic conversation tracking.

Request Body:

• model: Model ID (required). Use your aligned model id (e.g. model_01kmqm4nrn9fw6r); base text models (e.g. model_01kmqm4nrn9fw6r) also work if you have not aligned a model yet. For reserved capacity pass ptm_id:model_id
• messages: Array of message objects (required, minimum 1), each containing:
  • role: Message role (system, user, or assistant)
  • content: Text string OR array of content objects (for multimodal)
    • Text: {"type": "text", "text": "your message"}
    • Image URL: {"type": "image_url", "image_url": {"url": "https://..."}}
    • Image Base64: {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}
  • name (optional): Author name (a-z, A-Z, 0-9, underscores, max 64 chars)
• max_tokens (optional): Maximum tokens to generate in completion
• prompt_truncate_len (optional): Size to truncate chat prompts (default: 1500)
• temperature (optional): Sampling temperature between 0 and 2 (default: 1)
• stream (optional): Stream the response as server-sent events (default: true). Set stream: false for one JSON response.

  Tip: For prompts where you expect a long output (e.g., max_tokens > 500), keep streaming on to avoid HTTP 504 gateway timeouts on slow or large responses.

• tools (optional): List of tools/functions available to the model
• tool_choice (optional): Tool selection mode (auto, none, or specific tool)
• top_p (optional): Nucleus sampling parameter
• top_k (optional): Top-k sampling parameter
• n (optional): Number of completions to generate (default: 1)
• reasoning (optional): Reasoning configuration object with:
  • effort: Reasoning effort level (xhigh, high, medium, low, minimal, none)
  • max_tokens: Token limit for reasoning
  • exclude: Set true to exclude reasoning tokens from response
  • enabled: Enable reasoning with default parameters

Optional Headers:

• X-Session-ID: Session identifier for multi-turn conversation tracking

Returns:

Non-streaming mode -

• id: Unique identifier for the response
• object: Object type (always chat.completion)
• created: Unix timestamp when completion was created
• model: Model used for chat completion
• choices: List of completion choices, each containing:
  • index: Index of the choice
  • message: Response message with:
    • role: Role of the author (always assistant)
    • content: Generated response text
    • tool_calls (optional): Tool calls made by the model (for function calling)
  • finish_reason: Reason model stopped (stop for natural stop, length if max tokens reached)
• usage: Token usage statistics:
  • prompt_tokens: Number of tokens in the prompt
  • completion_tokens: Number of tokens generated
  • total_tokens: Total tokens used (prompt + completion)
• confidence_score (optional): Confidence score, 0-100, on aligned models; null on base models

Streaming mode - ChatCompletionChunk stream with:

• id: Unique identifier
• created: Timestamp
• model: Model ID
• choices: List of chunk choices with:
  • index: ChatCompletionChoice index
  • delta: Delta content with role and content
  • finish_reason: Reason for stopping (only in final chunk)
• usage (optional): Only present in final chunk

Example Request (Text Chat):

POST /api/v3/inference/chat/completions
Headers: {"Authorization": "Bearer <api_key>"}

{
  "model": "model_01kmqm4nrn9fw6r",
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant."
    },
    {
      "role": "user",
      "content": "What is the capital of France?"
    }
  ],
  "max_tokens": 500,
  "temperature": 0.7,
  "stream": false
}

Example Response (Text Chat):

{
  "id": "nugen-abc123",
  "object": "chat.completion",
  "created": 1704123600.0,
  "model": "model_01kmqm4nrn9fw6r",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The capital of France is Paris."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 20,
    "completion_tokens": 8,
    "total_tokens": 28
  },
  "confidence_score": 89.5454
}

Example Request (Using Domain-Aligned Model):

POST /api/v3/inference/chat/completions
Headers: {"Authorization": "Bearer <api_key>"}

{
  "model": "model_01kmqm4nrn9fw6r",
  "messages": [
    {
      "role": "user",
      "content": "How do I return a product?"
    }
  ],
  "max_tokens": 500,
  "temperature": 0.7,
  "stream": false
}

Example Response (Domain-Aligned Model):

{
  "id": "nugen-abc456",
  "object": "chat.completion",
  "created": 1704123700.0,
  "model": "model_01kmqm4nrn9fw6r",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "To return a product, please visit our returns portal within 30 days of purchase with your order number and receipt."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 15,
    "completion_tokens": 25,
    "total_tokens": 40
  },
  "confidence_score": 95.8
}

How to get your aligned model ID:

1. Create an alignment project using POST /api/v3/alignment-projects/create
2. Poll GET /api/v3/alignment-projects/{id}/status until status is READY
3. Use the model_id field from the response (e.g., "model_01kmqm4nrn9fw6r")
4. Deploy the model using POST /api/v3/models/{model_id}/deployment (optional - for evaluation/production use)
5. Use the model_id in the model parameter for inference requests

Example Request (Vision - Image URL):

POST /api/v3/inference/chat/completions
Headers: {"Authorization": "Bearer <api_key>"}

{
  "model": "model_01kmqm4nrn9fw6r",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Describe what you see in this image."
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "https://example.com/image.jpg"
          }
        }
      ]
    }
  ],
  "max_tokens": 1000,
  "stream": false
}

Example Request (Vision - Base64 Image):

POST /api/v3/inference/chat/completions
Headers: {"Authorization": "Bearer <api_key>"}

{
  "model": "model_01kmqm4nrn9fw6r",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What's in this image?"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAYABgAAD..."
          }
        }
      ]
    }
  ],
  "stream": false
}

Example Request (Function Calling):

POST /api/v3/inference/chat/completions
Headers: {"Authorization": "Bearer <api_key>"}

{
  "model": "model_01kmqm4nrn9fw6r",
  "messages": [
    {"role": "user", "content": "What is the status of INC-4821?"}
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_incident",
        "description": "Get incident status",
        "parameters": {
          "type": "object",
          "properties": {
            "incident_id": {"type": "string"}
          }
        }
      }
    }
  ],
  "tool_choice": "auto",
  "stream": false
}

Example Request (Streaming):

POST /api/v3/inference/chat/completions
Headers: {"Authorization": "Bearer <api_key>"}

{
  "model": "model_01kmqm4nrn9fw6r",
  "messages": [{"role": "user", "content": "Tell me a story"}],
  "stream": true
}

Example Response (Streaming):

data: {"id":"nugen-abc123","created":1704123600.0,"model":"model_01kmqm4nrn9fw6r","choices":[{"index":0,"delta":{"role":"assistant","content":"Once"},"finish_reason":null}]}

data: {"id":"nugen-abc123","created":1704123600.0,"model":"model_01kmqm4nrn9fw6r","choices":[{"index":0,"delta":{"content":" upon"},"finish_reason":null}]}

data: {"id":"nugen-abc123","created":1704123600.0,"model":"model_01kmqm4nrn9fw6r","choices":[{"index":0,"delta":{"content":" a time"},"finish_reason":"stop"}],"usage":{"prompt_tokens":10,"completion_tokens":5,"total_tokens":15}}

data: [DONE]

Provisioned throughput:

Reserve inference capacity for production workloads. Pass your provisioned id in model as ptm_id:model_id in place of the plain model_id. The request and response surface is identical, so your application code stays as it is once capacity is reserved. Available for aligned models on the Enterprise Edition.

Notes:

• Supports both text-only and vision/multimodal inputs depending on the model
• Using Domain-Aligned Models: Pass the model_id from your completed alignment project as the model parameter. Get your aligned model IDs from GET /api/v3/alignment-projects/{id}/status or GET /api/v3/models/aligned
• Function calling enables models to invoke external tools via tools and tool_choice parameters
• confidence_score (0-100) comes from inference-time alignment: uncertainty is quantified across the generation trajectory, flagging any drift from your domain. Available on aligned models only; always null for base models. In streaming mode, confidence arrives as its own {"object": "confidence", ...} events interleaved with content chunks, one per generated span, rather than a single value on the final chunk
• For streaming, usage statistics are included only in the final chunk
• Conversations are automatically saved to S3 if enabled in user settings
• Include X-Session-ID header for multi-turn conversation tracking
• Use prompt_truncate_len to control context window usage for long conversations
• The reasoning parameter enables advanced reasoning capabilities for supported models

{
  "model": "model_01kmqm4nrn9fw6r"
}

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "I'm here to help you with anything you need!",
        "role": "assistant"
      }
    }
  ],
  "confidence_score": 89.5454,
  "created": 16285464635.5454,
  "id": "nugen-1234",
  "model": "model_01kmqm4nrn9fw6r",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 7,
    "prompt_tokens": 5,
    "total_tokens": 12
  }
}