---
title: "Detect UI elements"
method: POST
path: "/boxes/{boxId}/actions/elements/detect"
tags: ["UI Action"]
---

# Detect UI elements

`POST /boxes/{boxId}/actions/elements/detect`

Detect and identify interactive UI elements in the current screen. Note: This feature currently only supports element detection within a running browser. If the browser is not running, the Elements array will be empty.

## Path parameters

- `boxId` string, required

## Request body

- DetectElements — Detect UI elements action configuration
  - `screenshot` DetectElementsScreenshotOptions — Detect elements screenshot options
    - `outputFormat` 'base64' | 'storageKey' — Type of the URI. default is base64.
    - `presignedExpiresIn` string — Presigned url expires in. Only takes effect when outputFormat is storageKey. Supported time units: ms (milliseconds), s (seconds), m (minutes), h (hours) Example formats: "500ms", "30s", "5m", "1h" Default: 30m

## Response `200`

- DetectedElementsResult — Result containing original screenshot, annotated screenshot, and detected elements
  - `screenshot` DetectedElementsScreenshot, required — Detected elements screenshot
    - `source` ScreenshotResult, required — Result of screenshot capture action
      - `uri` string, required — URL of the screenshot
      - `presignedUrl` string — Presigned url of the screenshot
    - `marked` ScreenshotResult, required — Result of screenshot capture action
      - `uri` string, required — URL of the screenshot
      - `presignedUrl` string — Presigned url of the screenshot
  - `elements` DetectedElement[], required — Detected UI elements
    - `id` string, required — Element id
    - `source` string, required — Element source
    - `type` string, required — Element type
    - `path` string, required — Element path
    - `width` number, required — Element width
    - `height` number, required — Element height
    - `x` number, required — Element x coordinate relative to screen
    - `y` number, required — Element y coordinate relative to screen
    - `centerX` number, required — Element center x coordinate relative to screen
    - `centerY` number, required — Element center y coordinate relative to screen
    - `label` string, required — A human-readable identifier generated from the element's visible attributes to help understand what this element represents. For images, it uses alt text or filename; for links, it uses text content or href; for buttons, it uses text content or aria-label; for inputs, it uses placeholder or value; etc.

## Changes

- **2025-09-24** `4040181e1ef5` — 1 breaking, 1 info
  - removed the success response with the status `201`
  - added the success response with the status `200`
- **2025-09-24** `8bbae399f00e` — 2 warning, 1 info
  - removed the request property `outputFormat`
  - removed the request property `presignedExpiresIn`
  - added the new optional request property `screenshot`
- **2025-09-24** `4c3b603d0ddc` — 1 info
  - endpoint added

[Change history](https://skmtc.dev/babelcloud/apis/gbox-open-api/changes/boxes/:boxId/actions/elements/detect/post.md)

---

[API](https://skmtc.dev/babelcloud/apis/gbox-open-api.md) · [All operations](https://skmtc.dev/babelcloud/apis/gbox-open-api/llms.txt) · [OpenAPI document](https://skmtc-service-production.skmtc.workers.dev/v1/apis/babelcloud/gbox-open-api/revisions/d655577fa915/schema)