---
title: "Segment an image Asset into per-region masks (non-minting)"
method: POST
path: "/v1/image/segment"
tags: ["Utilities"]
---

# Segment an image Asset into per-region masks (non-minting)

`POST /v1/image/segment`

## Request

- Base URL: `http://localhost:9998`
- URL: `http://localhost:9998/v1/image/segment`
- Auth: HTTP bearer

The segment API's non-minting REST door (the `image_segment` MCP verb's
sibling): runs the SAME capability-checked segmentation seam — the
user-selectable model registry (Apple Vision foreground-instance floor
by default, per-call override via `model`) with typed rejections for an
unknown model id or a prompt mode the selected entry does not declare —
but writes NO Asset. Every produced region mask returns inline as
base64 PNG, primary first (highest selection score: prompt coverage for
point/box, area fraction for auto on the Apple Vision floor, the model's
own mask confidence for auto on a SAM3-class entry — where that ordering
carries no claim about which region matters). Minting stays verb-only:
call the `image_segment` tool to record `mask` Assets with provenance.

## Request body

- object
  - `asset` string, required: Id of the input `image` Asset to segment.
  - `mode` 'auto' | 'point' | 'box' | 'text': Segmentation mode: `text` (every instance matching a noun-phrase concept — requires a text-capable segmentation model, a typed 400 otherwise; the concept must be a CONCRETE noun such as `person` or `red car`, since abstract prompts like `object`, `subject` or `foreground` match nothing), `point` (instances covering the given points), `box` (instances overlapping the rectangle), or `auto` (no prompt: foreground instances on the Apple Vision floor, and on SAM3-class entries an EXHAUSTIVE DECOMPOSITION via automatic mask generation over a dense point grid. It partitions the scene into regions of everything and frequently does not contain the subject at all, returned in no meaningful order — the score is the model's confidence in its own mask, which favours large flat areas and is a poor proxy for importance. Use `auto` to enumerate a scene and consume the whole set; prefer `text` / `point` / `box` to find a subject.)
  - `points` object[]: Point prompts for `point` mode, normalized top-left-origin coordinates. Each point carries an optional `label`, so a click set can both select and correct.
    - `x` number, required: Normalized x in 0-1.
    - `y` number, required: Normalized y in 0-1.
    - `label` 0 | 1: What this click asserts: `1` foreground ("select this"), `0` background ("not this" — excludes the region under the click). OPTIONAL, and an ABSENT label means `1`, which is exactly the behaviour of every build shipped before the field existed. At least one foreground point is required; background points correct a selection, they cannot make one.
  - `box` object: Box prompt for `box` mode, normalized top-left-origin coordinates.
    - `x` number, required
    - `y` number, required
    - `width` number, required
    - `height` number, required
  - `text` string: Noun-phrase concept for `text` mode (e.g. `red car`).
  - `previousMask` object: OPTIONAL refinement input for `point` mode: the mask of the region being refined, exactly as this door returned it on the previous call (the response's own `bytesBase64`). With it the decoder conditions on the interpretation already chosen and answers with ONE refined region instead of the whole/part/subpart triple, so a selection stops flipping granularity as clicks accumulate. Omit on a first click. ABSENT means exactly the behaviour of every build shipped before the field existed; a value sent to such a build is accepted and ignored. Read only under `point`; every other mode, and entries without a refinement channel, ignore it. Any resolution up to 64,000,000 pixels is accepted (the server resamples to the decoder's mask-input edge), so a downsampled copy is enough; past that it is `input_too_large` (400), refused before the pixels are decoded.
    - `bytesBase64` string, required: Single-channel matte PNG (white = region), base64-encoded.
  - `model` string: Optional segmentation model id for this call, overriding the `segmentationModelId` setting (default `vision-foreground-instance`). An unknown id is a typed 400 naming the registered entries, never a silent fallback.

## Response `200`

Per-region masks, primary first; nothing is written to the Asset graph.

- object
  - `masks` object[], required: One entry per segmented region, primary first. That ordering is meaningful under a PROMPTED mode (`point` / `box` / `text`), where the score measures the match against the prompt; under `auto` on a SAM3-class entry it is not, because the score is the model's own mask confidence rather than a ranking of importance. Read the whole array in that case. Capped at 32 regions per call. The cap is a TRANSPORT CEILING, not a selection of the best 32: `auto` on a SAM3-class entry is automatic mask generation over a dense point grid, which can propose more regions than a caller — or the Asset graph, on the minting `image_segment` verb — should absorb from one request. The cap applies after duplicate suppression, so it drops genuine extra regions rather than near-duplicates of one object; when it binds, the regions it dropped are not distinguishable in quality from the ones it kept.
    - `bytesBase64` string, required: Single-channel matte PNG, base64-encoded (white = region).
    - `mimeType` 'image/png', required
    - `bytes` integer, required: Decoded PNG byte count.
    - `score` number, required: Selection score in 0-1, the primary-first ordering key. Meaning is per entry: prompt coverage or area fraction on the Apple Vision floor, the model's own confidence on SAM3-class entries.
    - `bbox` object: OPTIONAL, additive. The region's extent, normalized 0-1 of the source image with a top-left origin — the same space the request's `box` takes, so it feeds straight back as a box prompt. Tight to the covered pixels at the matte's own resolution; zero width and height when nothing is covered. Absent on builds that predate the field.
      - `x` number, required
      - `y` number, required
      - `width` number, required
      - `height` number, required
    - `rle` object: OPTIONAL, additive. The region's coverage as row-major run lengths over its own grid, so a consumer never has to rasterize the PNG to hit-test it. `counts` alternates UNCOVERED then COVERED runs, starting with an uncovered run (0 when the first sample is covered), and sums to `width * height`. The grid is the producer's — the decoder's native mask resolution, or a nearest-sample reduction of a larger matte — with a long edge of at most 512 samples, and every mask of one response shares it. A sample counts as covered at or above the 8-bit level 128, the same rule a client decode of the PNG applies. Absent on builds that predate the field.
      - `width` integer, required
      - `height` integer, required
      - `counts` integer[], required
  - `count` integer, required: Number of region masks returned.
  - `width` integer, required: Source image pixel width.
  - `height` integer, required: Source image pixel height.
  - `mode` 'auto' | 'point' | 'box' | 'text', required
  - `model` string, required: The segmentation model id that produced the masks.

## Other responses

- `400`: Invalid request
- `404`: Input Asset not found (or its bytes are unreadable)
- `422`: No region matched (nothing detected, or no instance matched the prompt)
- `503`: Asset store unavailable

---

[API](https://skmtc.dev/modelpiper/apis/toolpiper-api.md) · [All operations](https://skmtc.dev/modelpiper/apis/toolpiper-api/llms.txt) · [OpenAPI document](https://skmtc.dev/modelpiper/apis/toolpiper-api/revisions/6722bf56385c?raw)