H2h2o
Models

Send a message and get a response from an LLM.

Send a message and get a response from an LLM. Note: This method is only recommended if you are passing a chat conversation or for low-volume testing. For general chat with an LLM, use "POST /chats/{session_id}/completions" endpoint.

post/models/{model_name}/answer_question

Request

Path parameters

model_namestring required

Name of LLM. Use auto, when you are not interested in particular model.

Request body

text_context_liststring[]

List of raw text strings to be summarized.

system_promptstring

Text sent to models which support system prompts. Gives the model overall context in how to respond. Use auto for the model default or None for h2oGPTe defaults. Defaults to '' for no system prompt.

llm_argsChatCompletionRequestLLMArgs

A map of arguments sent to LLM with query.

  • temperature (type=double, default=0.0) - A value used to modulate the next token probabilities. 0 is the most deterministic and 1 is most creative.
  • top_k (type=integer, default=1) - A number of highest probability vocabulary tokens to keep for top-k-filtering.
  • top_p (type=double, default=0.0) - If set to a value < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for generation.
  • seed (type=integer, default=0) - A seed for the random number generator when sampling during generation (if temp>0 or top_k>1 or top_p<1), seed=0 picks a random seed.
  • repetition_penalty (type=double, default=1.07) - A parameter for repetition penalty. 1.0 means no penalty.
  • max_new_tokens (type=double, default=1024) - A maximum number of new tokens to generate. This limit applies to each (map+reduce) step during summarization and each (map) step during extraction.
  • min_max_new_tokens (type=integer, default=512) - A minimum value for max_new_tokens when auto-adjusting for content of prompt, docs, etc.
  • response_format (type=enum[text, json_object, json_code], default=text) - An output type of LLM
  • guided_json (type=map) - If specified, the output will follow the JSON schema.
  • guided_regex (type=string) - If specified, the output will follow the regex pattern. Only for models that support guided generation.
  • guided_choice (type=array[string]) - If specified, the output will be exactly one of the choices. Only for models that support guided generation.
  • guided_grammar (type=string) - If specified, the output will follow the context free grammar. Only for models that support guided generation.
  • guided_whitespace_pattern (type=string) - If specified, will override the default whitespace pattern for guided json decoding. Only for models that support guided generation.
  • enable_vision (type=enum[on, off, auto], default=auto) - Controls vision mode, send images to the LLM in addition to text chunks.
  • visible_vision_models (type=array[string], default=[auto]) - Controls which vision model to use when processing images. Must provide exactly one model. [auto] for automatic.
  • images_num_max (type=integer, default=None) - Maximum number of images to process.
  • json_preserve_system_prompt (type=boolean, default=None) - Whether to preserve system prompt in JSON response.
  • client_metadata (type=string, default=None) - Additional metadata to send with the request.
  • min_chars_per_yield (type=integer, default=1) - Minimum characters to yield in streaming response.
  • reasoning_effort (type=integer, default=0) - Level of reasoning effort for the model (higher values = deeper reasoning, e.g., 10000-65000). Use for models that support chain-of-thought reasoning. 0 means no additional reasoning effort.
  • cost_controls (type=map) A map with cost controls settings:
    • max_cost (type=double) - Sets the maximum allowed cost in USD per LLM call when doing Automatic model routing. If the estimated cost based on input and output token counts is higher than this limit, the request will fail as early as possible.
    • max_cost_per_million_tokens (type=double) - Only consider models that cost less than this value in USD per million tokens when doing automatic routing. Using the max of input and output cost.
    • model (type=array[string]) - Optional subset of models to consider when doing automatic routing. If not specified, all models are considered.
    • willingness_to_pay (type=double) - Controls the willingness to pay extra for a more accurate model for every LLM call when doing automatic routing, in units of USD per +10% increase in accuracy. We start with the least accurate model. For each more accurate model, we accept it if the increase in estimated cost divided by the increase in estimated accuracy is no more than this value divided by 10%, up to the upper limit specified above. Lower values will try to keep the cost as low as possible, higher values will approach the cost limit to increase accuracy. 0 means unlimited.
    • willingness_to_wait (type=double) - Controls the willingness to wait longer for a more accurate model for every LLM call when doing automatic routing, in units of seconds per +10% increase in accuracy. We start with the least accurate model. For each more accurate model, we accept it if the increase in estimated time divided by the increase in estimated accuracy is no more than this value divided by 10%. Lower values will try to keep the time as low as possible, higher values will take longer to increase accuracy. 0 means unlimited.
  • use_agent (type=boolean, default=False) - If True, use the AI agent (with access to tools) to generate the response.
  • agent_accuracy (type=string, default="standard") - Effort level by the agent. Only if use_agent=True. One of ["quick", "basic", "standard", "maximum"].
  • agent_max_turns (type=union[string, integer], default="auto") - Optional max. number of back-and-forth turns with the agent. Only if use_agent=True. Either "auto" or an integer.
  • agent_tools (type=union[string, array[string]], default="auto") - Either "auto", "all", "any" to enable all available tools, or a specific list of tools to use. Only if use_agent=True.
  • agent_type (type=string, default="auto") - Type of agent to use for task processing.
  • agent_original_files (type=array[string], default=None) - List of file paths for agent to process.
  • agent_timeout (type=integer, default=None) - Timeout in seconds for each agent turn.
  • agent_total_timeout (type=integer, default=3600) - Total timeout in seconds for all agent processing.
  • agent_code_writer_system_message (type=string, default=None) - System message for agent code writer.
  • agent_num_executable_code_blocks_limit (type=integer, default=1) - Maximum number of executable code blocks.
  • agent_system_site_packages (type=boolean, default=True) - Whether agent has access to system site packages.
  • agent_main_model (type=string, default=None) - Main model to use for agent.
  • agent_max_stream_length (type=integer, default=None) - Maximum stream length for agent response.
  • agent_max_memory_usage (type=integer, default=16*10243)** - Maximum memory usage for agent in bytes (16GB default).
  • agent_main_reasoning_effort (type=integer, default=None) - Effort level for main reasoning.
  • agent_advanced_reasoning_effort (type=integer, default=None) - Effort level for advanced reasoning.
  • agent_max_confidence_level (type=integer, default=None) - Maximum confidence level for agent responses.
  • agent_planning_forced_mode (type=boolean, default=None) - Whether to force planning mode for agent.
  • agent_too_soon_forced_mode (type=boolean, default=None) - Whether to force "too soon" mode for agent.
  • agent_critique_forced_mode (type=integer, default=None) - Whether to force critique mode for agent.
  • agent_query_understanding_parallel_calls (type=integer, default=None) - Number of parallel calls for query understanding.
  • task_tracking (type=boolean, default=None) - Enable task tracking for agent.
  • task_update_steps (type=integer, default=None) - Number of steps for task update tracking (0-5, None for auto).
  • tool_building_mode (type=string, default=None) - Mode for tool building configuration.
  • agent_stream_files (type=boolean, default=True) - Whether to stream files from agent.
  • final_answer_guidelines_mode (type=string, default="auto") - Mode for formatting agent final answers. Only applies when use_agent=True. Options: "auto" (chooses detailed, may later adapt), "detailed" (full final answer guidelines), "detailed_no_file_links", "detailed_no_image_links", "detailed_no_links", "file_template" (use custom template), "raw_LLM" (bypass final answer guidelines). Available modes may vary by deployment.
timeoutinteger

Timeout in seconds.

pre_prompt_querystring

Text that is prepended before the contextual document chunks in text_context_list. Only used if text_context_list is provided.

prompt_querystring

Text that is appended after the contextual document chunks in text_context_list. Only used if text_context_list is provided.

questionstring required

Text query to send to the LLM.

Example request

{
  "guardrails_settings": {
    "disallowed_regex_patterns": [
      "secret_disallowed_word",
      "(?!0{3})(?!6{3})[0-8]\\d{2}-(?!0{2})\\d{2}-(?!0{4})\\d{4}"
    ],
    "presidio_labels_to_flag": [
      "IBAN_CODE",
      "US_SSN"
    ],
    "pii_labels_to_flag": [
      "IBAN",
      "SSN"
    ],
    "prompt_guard_labels_to_flag": [
      "JAILBREAK"
    ],
    "guardrails_labels_to_flag": [
      "Violent Crimes",
      "Non-Violent Crimes",
      "Intellectual Property",
      "Code Interpreter Abuse"
    ],
    "guardrails_llm": "h2oai/h2o-danube3-4b-chat",
    "guardrails_safe_category": "Safe",
    "column_redaction_custom_entities_to_flag": [
      "Mother's Maiden Name"
    ],
    "column_redaction_custom_entities": {
      "Mother's Maiden Name": "Mother's maiden name"
    },
    "custom_pii_entities_to_flag": [
      "Mother's Maiden Name"
    ],
    "custom_pii_entities": {
      "Mother's Maiden Name": "Mother's maiden name"
    }
  }
}

Response

Successful operation

contentstring required
errorstring required
prompt_rawstring
llmstring required
input_tokensinteger
output_tokensinteger
originstring

Changes

No recorded changes to this operation across all 1 revision of this API.