Create an LLM

Creates a connection to an external Large Language Model (LLM) for Retrieval Augmented Generation (RAG) and chat. You can connect OpenAI API-compatible models from providers like Anthropic, Azure, Google, or custom-hosted endpoints. After creation, reference your custom LLM by name in query generation parameters.

• Connect external LLMs that use the OpenAI-compatible API format
• Configure multiple LLM providers for different use cases
• Override the platform's built-in LLMs with your own models
• Use custom models for RAG, chat, and document summarization

Example providers:

OpenAI

Type: openai-compatible Models: GPT-4o, GPT-5 Auth: Bearer token

{
  "type": "openai-compatible",
  "name": "my-gpt5",
  "model": "gpt-5",
  "uri": "https://api.openai.com/v1/chat/completions",
  "auth": {
    "type": "bearer",
    "token": "sk-..."
  }
}

OpenAI Responses API

Type: openai-responses Models: o1-preview, o1-mini, o3-mini (reasoning models) Auth: Bearer token Note: For reasoning models that don't support streaming

{
  "type": "openai-responses",
  "name": "my-o1",
  "model": "o1-preview",
  "uri": "https://api.openai.com/v1/responses",
  "auth": {
    "type": "bearer",
    "token": "sk-..."
  }
}

Anthropic Claude

Type: anthropic Models: claude-4-opus, claude-4-5-haiku, claude-4-5-sonnet Auth: Bearer token (direct API); AWS or GCP credentials for Bedrock / Vertex

{
  "type": "anthropic",
  "name": "my-claude",
  "model": "claude-sonnet-4-5-20250929",
  "auth": {
    "type": "bearer",
    "token": "sk-ant-..."
  }
}

Azure OpenAI

Type: openai-compatible Models: GPT-3.5, GPT-4 (Azure-deployed versions) Auth: Custom header (api-key)

{
  "type": "openai-compatible",
  "name": "my-azure-gpt4",
  "model": "gpt-4",
  "uri": "https://YOUR-RESOURCE.openai.azure.com/openai/deployments/YOUR-DEPLOYMENT/chat/completions?api-version=2024-02-15-preview",
  "auth": {
    "type": "header",
    "header": "api-key",
    "value": "your-azure-key"
  }
}

Google Vertex AI (Gemini) — Service Account

Type: vertex-ai Models: gemini-2.5-pro, gemini-2.5-flash Auth: Service account

{
  "type": "vertex-ai",
  "name": "my-gemini",
  "model": "gemini-2.5-flash",
  "uri": "https://us-central1-aiplatform.googleapis.com/v1/projects/YOUR-PROJECT/locations/us-central1",
  "auth": {
    "type": "service_account",
    "key_json": "{...service account JSON...}"
  }
}

Google AI Studio (Gemini) — API Key

Type: vertex-ai Models: gemini-2.5-pro, gemini-2.5-flash Auth: API key

{
  "type": "vertex-ai",
  "name": "my-gemini",
  "model": "gemini-2.5-flash",
  "uri": "https://generativelanguage.googleapis.com/v1beta",
  "auth": {
    "type": "api_key",
    "api_key": "your-google-api-key"
  }
}

The uri field is flexible — you can provide a base URI or a full URL copied from Google docs (including model path and :generateContent suffix). The platform normalizes it automatically.

Custom OpenAI-Compatible

Type: openai-compatible Models: Any self-hosted or custom LLM, such as OpenRouter. Auth: Bearer or custom header

{
  "type": "openai-compatible",
  "name": "my-custom-llm",
  "model": "llama-3-70b",
  "uri": "https://my-llm-endpoint.com/v1/chat/completions",
  "auth": {
    "type": "bearer",
    "token": "custom-token"
  }
}

{
  "name": "Claude 3.7 Sonnet",
  "description": "The Anthropic Claude 3.7 Sonnet model",
  "model": "claude-3-7-sonnet-20250219",
  "uri": "https://api.anthropic.com/v1/chat/completions",
  "auth": {
    "type": "bearer",
    "token": "abcdef......"
  },
  "idle_timeout_seconds": 300,
  "test_model_parameters": {
    "max_tokens": 512
  }
}

{
  "id": "llm_1021844",
  "name": "Claude 3.7 Sonnet",
  "description": "The Anthropic Claude 3.7 Sonnet model.",
  "ownership": "platform",
  "type": "openai-compatible",
  "idle_timeout_seconds": 300,
  "auth": {
    "type": "bearer",
    "token": "abcdef......"
  }
}