---
title: "Summarize a video file"
method: POST
path: "/summarize"
tags: ["Summarization"]
---

# Summarize a video file

`POST /summarize`

Run video file summarization. For stream summarization use POST /v1/generate_captions and POST /v1/stream_summarize instead.

## Request body

- SummarizationQuery — Summarization Query Request Fields.
  - `min_tokens` integer — Minimum number of tokens to generate before the model is allowed to stop. Used with ignore_eos for fixed-length generation.
  - `ignore_eos` boolean — If true, ignore end-of-sequence token and continue generating until max_tokens. Useful for benchmarking with fixed output length.
  - `id` union — Unique ID or list of IDs of the file(s)/live-stream(s) to summarize
    - string, uuid
    - string[]
  - `url` string, nullable — URL of the video to summarize
  - `system_prompt` string — System prompt for the VLM. To enable reasoning with Cosmos Reason1, add <think></think> and <answer></answer> tags to the system prompt.
  - `prompt` string — Prompt for summary generation
  - `model` string, required — Model to use for this query.
  - `max_tokens` integer — The maximum number of tokens to generate in any given call.
  - `temperature` number — The sampling temperature to use for text generation. The higher the temperature value is, the less deterministic the output text will be.
  - `top_p` number — The top-p sampling mass used for text generation. The top-p value determines the probability mass that is sampled at sampling time.
  - `top_k` number — The number of highest probability vocabulary tokens to keep for top-k-filtering
  - `seed` integer — Seed value
  - `chunk_duration` integer — Chunk videos into `chunkDuration` seconds. Set `0` for no chunking
  - `chunk_overlap_duration` integer — Chunk Overlap Duration Time in Seconds. Set `0` for no overlap
  - `summary_duration` integer — Summarize every `summaryDuration` seconds of the video. Applicable to live streams only.
  - `media_info` union — Provide Start and End times offsets for processing part of a video file. Not applicable for live-streaming.
    - MediaInfoOffset — Media information using offset for files.
      - `type` 'offset', required — Information about a segment of media with start and end offsets.
      - `start_offset` integer — Segment start offset in seconds from the beginning of the media.
      - `end_offset` integer — Segment end offset in seconds from the beginning of the media.
    - MediaInfoTimeStamp — Media information using offset for live-streams.
      - `type` 'timestamp', required — Information about a segment of live-stream with start and end timestamp.
      - `start_timestamp` string — Timestamp in the video to start processing from
      - `end_timestamp` string — Timestamp in the video to stop processing at
  - `num_frames_per_chunk` integer — DEPRECATED: Use num_frames_per_second_or_fixed_frames_chunk instead. Number of frames per chunk to use for the VLM.
  - `vlm_input_width` integer — VLM Input Width
  - `vlm_input_height` integer — VLM Input Height
  - `enable_audio` boolean — Enable transcription of the audio stream in the media
  - `enable_reasoning` boolean — Enable reasoning for VLM captions generation
  - `num_frames_per_second_or_fixed_frames_chunk` number — Number of frames per second (if use_fps_for_chunking=true) or fixed number of frames per chunk (if use_fps_for_chunking=false).
  - `use_fps_for_chunking` boolean — If true, use num_frames_per_second_or_fixed_frames_chunk as FPS. If false, use it as a fixed frame count per chunk.
  - `creation_time` string — Creation time of the media in ISO 8601 format (e.g. 2024-06-09T18:32:11.123Z). If provided, offsets frame timestamps in the response.
  - `alert_category` string — Alert category for VLM captions (e.g. 'Worker PPE Violation').
  - `mm_processor_kwargs` object — Additional keyword arguments for the multimodal processor (e.g., size, shortest_edge, longest_edge).
  - `custom_metadata` object — Custom metadata to be added to the summarization request. This is a JSON object with key-value pairs. Custom metadata is supported only with user managed milvus db collections.
  - `delete_external_collection` boolean — Delete the external collection at the end of the summarization request
  - `schema` string — JSON schema for structured output extraction from video content
  - `batch_response_method` string — Method for batch response processing
  - `scenario` string, required — Scenario or use case context for the summarization
  - `events` string[], required — List of events to detect or extract from the video
  - `auto_generate_prompt` boolean — Enable automatic prompt generation based on schema and events
  - `override_vlm_prompt` boolean — Override the VLM prompt with the user supplied prompt
  - `enable_vlm_structured_output` boolean — Enable VLM structured output
  - `objects_of_interest` string[] — List of objects of interest to detect or extract from the video

## Response `200`

Successful Response.

- CompletionResponse — Represents a summarization/chat completion response.
  - `id` string, uuid, required — Unique ID for the query
  - `video_id` string, uuid, required — Unique ID for the video
  - `choices` CompletionResponseChoice[], required — A list of chat completion choices. Can be more than one if `n` is greater than 1.
    - `finish_reason` 'stop' | 'length' | 'content_filter' | 'tool_calls', required — The reason the model stopped generating tokens.
    - `index` integer, required — The index of the choice in the list of choices.
    - `message` ChatCompletionResponseMessage, required — A chat completion message generated by the model.
      - `content` string, nullable, required — The contents of the message. For VLM captions API, this field contains a combined response with timestamps for each chunk.
      - `tool_calls` ChatCompletionMessageToolCall[]
        - `type` 'alert', required — Types of tools supported by VIA.
        - `alert` ChatCompletionMessageAlertTool, required — Alert trigerred by VIA.
          - `name` string, required — Name for the alert that was triggered.
          - `ntpTimestamp` string, nullable — NTP timestamp of when the event occurred (for live-streams).
          - `offset` integer — Offset in seconds in the video file when the event occurred (for files).
          - `detectedEvents` string[], required — List of events detected.
          - `details` string, required — Details of the alert.
      - `role` 'assistant', required — The role of the author of this message.
  - `created` integer, required — The Unix timestamp (in seconds) of when the chat completion/summary request was created.
  - `model` string, required — The model used for the chat completion/summarization.
  - `media_info` union, required — Part of the file / live-stream for which this response is applicable.
    - MediaInfoTimeStamp — Media information using offset for live-streams.
      - `type` 'timestamp', required — Information about a segment of live-stream with start and end timestamp.
      - `start_timestamp` string — Timestamp in the video to start processing from
      - `end_timestamp` string — Timestamp in the video to stop processing at
    - MediaInfoOffset — Media information using offset for files.
      - `type` 'offset', required — Information about a segment of media with start and end offsets.
      - `start_offset` integer — Segment start offset in seconds from the beginning of the media.
      - `end_offset` integer — Segment end offset in seconds from the beginning of the media.
  - `object` 'chat.completion' | 'summarization.completion' | 'summarization.progressing' | 'vlm_captions.completion' | 'vlm_captions.progressing', required — Completion object type.
  - `usage` CompletionUsage — An optional field that will only be present when you set `stream_options: {"include_usage": true}` in your request. When present, it contains a null value except for the last chunk which contains the token usage statistics for the entire request.
    - `query_processing_time` integer, required — Summarization Query Processing Time in seconds.
    - `total_chunks_processed` integer, required — Total Number of chunks processed.
    - `summary_tokens` integer — Total Number of tokens used for summary.
    - `aggregation_tokens` integer — Total Number of tokens used for aggregation.
    - `summary_requests` integer — Total Number of requests for summary.
    - `summary_latency` number — Total latency for summary.
    - `aggregation_latency` number — Total latency for aggregation.

## Other responses

- `400` — Bad Request. The server could not understand the request due to invalid syntax.
- `401` — Unauthorized request.
- `422` — Failed to process request.
- `429` — Rate limiting exceeded.
- `500` — Internal Server Error.
- `503` — Server is busy processing another file. Client may try again in some time.

---

[API](https://skmtc.dev/nvidia/apis/long-video-summarization-api.md) · [All operations](https://skmtc.dev/nvidia/apis/long-video-summarization-api/llms.txt) · [OpenAPI document](https://skmtc-service-production.skmtc.workers.dev/v1/apis/nvidia/long-video-summarization-api/revisions/2ce997975112/schema)