Webhooks

Create a webhook subscription

Changed on

Subscribe a webhook to one event on a prompt, evaluator, evaluation task, or space. To deliver several events to the same webhook, create one subscription per event. A SPACE subscription covers every evaluation task in the space, including tasks created later.

Payload Requirements

  • webhook_id, source_type, source_id, and event are required.
  • webhook_id must be a webhook in the source's organization; unknown webhooks yield a 404.
  • event must belong to the source type: prompt events for PROMPT sources, evaluator events for EVALUATOR sources, and ONLINE_TASK_RUN_FAILED for ONLINE_TASK and SPACE sources. Other combinations are rejected with a 422.
  • An ONLINE_TASK source must be an evaluation task; other task types are rejected with a 422.
  • A webhook can subscribe to a given event on a given source only once (409 on conflict).
  • At most 200 webhooks may subscribe to the same event on a source; requests that would exceed this limit are rejected with a 422.

Creating a subscription requires PROMPT_UPDATE for a prompt source, EVALUATOR_UPDATE for an evaluator source, PROJECT_EVAL_TASK_UPDATE (or EXPERIMENT_EVAL_TASK_UPDATE for a dataset task) for a task source, or a space, organization, or account admin role for a space source, since a space subscription covers every task in the space, including tasks in restricted projects. Callers who can read the source (PROMPT_READ, EVALUATOR_READ, PROJECT_EVAL_TASK_READ or EXPERIMENT_EVAL_TASK_READ) but lack the required permission receive a 403; sources the caller cannot read yield a 404.

Checks run in this order, and the first failure is returned: request body validation (422), source access (404 or 403), source task type (422), webhook lookup (404), then duplicate subscription (409) and the per-source webhook limit (422).

<Warning>This endpoint is in alpha, read more here.</Warning>

post/v2/webhook-subscriptions

Request

  • Base URL
    https://api.arize.com
  • URL
    https://api.arize.com/v2/webhook-subscriptions
  • Auth
    HTTP bearer

Request body

webhook_idstring required

A universally unique identifier (base64-encoded opaque string).

source_type'PROMPT' | 'EVALUATOR' | 'ONLINE_TASK' | 'SPACE' required

The kind of resource a webhook subscription is attached to.

  • PROMPT — source_id is a prompt id. Prompt events apply.
  • EVALUATOR — source_id is an evaluator id. Evaluator events apply.
  • ONLINE_TASK — source_id is an evaluation task id. Task run events apply.
  • SPACE — source_id is a space id. Task run events apply to every evaluation task in the space, including tasks created later.
source_idstring required

A universally unique identifier (base64-encoded opaque string).

event'PROMPT_VERSION_CREATED' | 'PROMPT_VERSION_LABELED' | 'PROMPT_VERSION_UNLABELED' | 'EVALUATOR_VERSION_CREATED' | 'ONLINE_TASK_RUN_FAILED' required

An event that a webhook can subscribe to.

  • PROMPT_VERSION_CREATED — a new version of the prompt was created.
  • PROMPT_VERSION_LABELED — a label was added to a version of the prompt.
  • PROMPT_VERSION_UNLABELED — a label was removed from a version of the prompt.
  • EVALUATOR_VERSION_CREATED — a new version of the evaluator was created.
  • ONLINE_TASK_RUN_FAILED — a run of an evaluation task failed, or wrote some evaluations and then stopped on an error. Sent once when a task starts failing and again each time the cause changes.

Prompt events are valid only for PROMPT sources, evaluator events only for EVALUATOR sources, and task run events only for ONLINE_TASK and SPACE sources.

Example request

{
  "webhook_id": "RW50aXR5OjEyMzQ1",
  "source_id": "RW50aXR5OjEyMzQ1"
}

Response

The created webhook subscription

idstring required

A universally unique identifier (base64-encoded opaque string).

webhook_idstring required

A universally unique identifier (base64-encoded opaque string).

source_type'PROMPT' | 'EVALUATOR' | 'ONLINE_TASK' | 'SPACE' required

The kind of resource a webhook subscription is attached to.

  • PROMPT — source_id is a prompt id. Prompt events apply.
  • EVALUATOR — source_id is an evaluator id. Evaluator events apply.
  • ONLINE_TASK — source_id is an evaluation task id. Task run events apply.
  • SPACE — source_id is a space id. Task run events apply to every evaluation task in the space, including tasks created later.
source_idstring required

A universally unique identifier (base64-encoded opaque string).

event'PROMPT_VERSION_CREATED' | 'PROMPT_VERSION_LABELED' | 'PROMPT_VERSION_UNLABELED' | 'EVALUATOR_VERSION_CREATED' | 'ONLINE_TASK_RUN_FAILED' required

An event that a webhook can subscribe to.

  • PROMPT_VERSION_CREATED — a new version of the prompt was created.
  • PROMPT_VERSION_LABELED — a label was added to a version of the prompt.
  • PROMPT_VERSION_UNLABELED — a label was removed from a version of the prompt.
  • EVALUATOR_VERSION_CREATED — a new version of the evaluator was created.
  • ONLINE_TASK_RUN_FAILED — a run of an evaluation task failed, or wrote some evaluations and then stopped on an error. Sent once when a task starts failing and again each time the cause changes.

Prompt events are valid only for PROMPT sources, evaluator events only for EVALUATOR sources, and task run events only for ONLINE_TASK and SPACE sources.

created_atstring date-time required

Timestamp for when the subscription was created

Example response

{
  "id": "RW50aXR5OjEyMzQ1",
  "webhook_id": "RW50aXR5OjEyMzQ1",
  "source_id": "RW50aXR5OjEyMzQ1"
}

Changes

  • 74723b1c4847Revision changes
    • warning

      added the new ONLINE_TASK enum value to the //// response property for the response status

    • warning

      added the new ONLINE_TASK_RUN_FAILED enum value to the //// response property for the response status

    • warning

      added the new SPACE enum value to the //// response property for the response status

    • info

      added the new ONLINE_TASK enum value to the request property ////

    • info

      added the new ONLINE_TASK_RUN_FAILED enum value to the request property ////

    • info

      added the new SPACE enum value to the request property ////

  • 2a4c8244d07bRevision changes
    • info

      endpoint added