--- title: "Create a backfill job" method: POST path: "/v1/jobs/backfills" tags: ["Jobs"] --- # Create a backfill job `POST /v1/jobs/backfills` ## Request - Base URL: `https://docs.grepr.ai` (the document declares no server; this is the origin it was published from) - URL: `https://docs.grepr.ai/v1/jobs/backfills` - Auth: one of: - API key in header `GREPR-API-KEY` - OpenID Connect (`https://grepr-prod.us.auth0.com/.well-known/openid-configuration`) Creates a logs or spans backfill batch job from backfill parameters (time window, query, vendor sinks, and optional type-specific fields). The `dataType` selects the built-in logs or spans template. This is an asynchronous batch job: the response returns an identifier (field: `id`) that can be used to query the status of the job using the `GET` jobs endpoint. The server builds the backfill job graph from the parameters, so callers do not assemble it themselves. ## Request body - union: Parameters for creating a logs or spans backfill batch job. - object: Parameters for a logs backfill. - `dataType` 'logs', required: Selects the built-in logs backfill template. - `datasetId` string, required: The dataset the backfill reads from and the dedup sinks track delivery in. - `end` string, date-time, required: End of the time interval to backfill. - `limit` integer: Maximum number of rows to backfill. Default and max is 500,000. For spans this applies independently to each vendor source. A negative value means no limit. - `name` string, required: Name of the backfill job. Must match the job name pattern [a-z0-9_]{1,128}. - `query` union, required: Predicate applied to rows before analytical aggregation. - object - `queries` EventPredicate[], required: List of predicates that must all match for this predicate to match. - `type` 'and-predicate', required: A predicate that matches when every child predicate matches. - `query` string - object - `query` string, required: The datadog query used by the predicate. See https://docs.datadoghq.com/tracing/trace_explorer/query_syntax - `type` 'datadog-query', required: A predicate that checks if an event matches a datadog query. - object - `filterTags` object: The set of tags to query for. If unspecified, all tags are queried. - `metricNames` string[], required: The names of the metrics to query. Must not be empty. - `query` string, required - `type` 'metric-predicate', required: A predicate that checks if an event matches a metrics query. - object - `query` string, required: The New Relic query used by the predicate. See https://docs.newrelic.com/docs/logs/ui-data/query-syntax-logs/ - `type` 'newrelic-query', required: A predicate that checks if an event matches a New Relic query. - object - `query` string, required: The Nrql query used by the predicate. See https://docs.newrelic.com/docs/nrql - `strict` boolean: Whether the query should be parsed in strict mode. - `type` 'nrql-query', required: A predicate that checks if an event matches an NRQL query. - `sinks` Operation[], required: The vendor sink operations that deliver the backfilled events. Passed through verbatim, matching the sinks a rule action would carry. - union: Base class of all Operations - object - `filter` union: Predicate applied to rows before analytical aggregation. - object - … - object - … - object - … - object - … - object - … - `maxAllowedRate` number, double: Max allowed rate in messages per second. The max rate used by the token rate limiter to control the sampling rate of events. The rate is distributed across parallel subtasks. - `maxBurstLimit` integer: Maximum size of acceptable burst used by the token bucket rate limiter to handle bursty traffic. - `name` string, required - `type` 'logs-event-sampler', required: Samples log events using a token based rate limiter. Throttles log events based on max rate accepting up to max burst limit for bursty traffic. Can selectively sample events matching a filter predicate. Rate limiting is distributed across parallel subtasks. - object - `name` string, required - `predicate` union, required: Predicate applied to rows before analytical aggregation. - object - … - object - … - object - … - object - … - object - … - `type` 'logs-branch', required: Splits events based on a predicate. Positive matches go to the main output. Negative matches go to the 'else' output. - object - `inverted` boolean: Whether to invert the predicate logic - `maxLateEventTimestampDelta` string, ISO-8601: Maximum allowed time delta between received and event timestamps. Events with a delta exceeding this threshold will be filtered out. ISO-8601 duration format. - `name` string, required - `predicate` union, required: Predicate applied to rows before analytical aggregation. - object - … - object - … - object - … - object - … - object - … - `type` 'logs-filter', required: Filters events based on a predicate. - object - `extractAttribute` union: Optional attribute to extract. Does nothing if it does not refer to a string attribute. A dotted string, or an array of literal keys. - string - string[] - `grokHelperRules` string[] - `grokParsingRules` string[]: The Grok parser enriches log events by parsing unstructured text into structured attributes. - `name` string, required - `predicate` union: Predicate applied to rows before analytical aggregation. - object - … - object - … - object - … - object - … - object - … - `type` 'grok-parser', required: Processes unstructured logs and extracts fields from them using Grok patterns. - object - `liftFieldNames` string[]: Object-valued JSON fields whose children are lifted into the containing parsed object. An empty list disables lifting. Lifting repeats recursively when a lifted object contains another configured field. Colliding maps are merged recursively. For leaf values and all other type combinations, the lifted child replaces the existing value; arrays are not combined. If multiple configured fields contain the same child, the later field wins according to these rules. - `maxNestedDepthForFields` integer - `maxNestedJsonIterations` integer: Maximum number of iterations to perform when resolving nested JSON fields. - `name` string, required - `preserveJsonMessageField` boolean: The processor tries to deserialize the string in the 'message' field as JSON, putting all values into the 'attributes' field. If this is enabled, the processor will keep the JSON string in 'message'. When disabled, the string will be cleared. - `resolveNestedJsonFields` array[]: If you have JSON that is double serialized, e.g. '{"field": "{\"nested\": \"value\"}"}', you might want to deserialize the nested JSON fields as well. This option allows you to specify the paths within the attributes where we should look for nested JSON fields. All the paths are tried in every iteration, and the deserialized fields are added to the root of attributes. Use this for singly serialized JSON attribute fields as well. - string[]: If you have JSON that is double serialized, e.g. '{"field": "{\"nested\": \"value\"}"}', you might want to deserialize the nested JSON fields as well. This option allows you to specify the paths within the attributes where we should look for nested JSON fields. All the paths are tried in every iteration, and the deserialized fields are added to the root of attributes. Use this for singly serialized JSON attribute fields as well. - `type` 'json-log-processor', required: Deserializes the 'message' field as JSON merging the result into the 'attributes' map. Can also deserialize attributes that contain JSON strings. - object - `attributeRemappingRules` AttributeRemappingRule[]: List of attribute-to-attribute remapping rules. Each rule moves the value from a source attribute path to a target attribute path, removing the source unless the rule's preserveSource flag is set. An optional predicate filter controls when each rule is applied. - `predicate` union: Predicate applied to rows before analytical aggregation. - … - `preserveSource` boolean: When true, the source attribute is preserved (copied) instead of being removed (moved). Defaults to false (source is removed after remapping). - `source` union, required: Source attribute path: a dotted string ("resource.service" is resource → service), or an array of literal keys (["http", "user.agent"]). - … - `target` union, required: Target attribute path: a dotted string, or an array of literal keys. - … - `hostPreserveSource` boolean: When true, preserves the source host attribute after remapping instead of removing it. Defaults to true (source is preserved). - `hostReservedAttributePaths` array[]: Paths to search for host values - string[]: Paths to search for host values - `hostReservedAttributes` string[]: Attributes to search for host values - `messagePreserveSource` boolean: When true, preserves the source message attribute after remapping instead of removing it. Defaults to false (source is removed after remapping). - `messageReservedAttributePaths` array[]: Paths to search for message values - string[]: Paths to search for message values - `messageReservedAttributes` string[]: Attributes to search for message values. The used attribute will be removed once remapped. - `name` string, required - `servicePreserveSource` boolean: When true, preserves the source service attribute after remapping instead of removing it. Defaults to true (source is preserved). - `serviceReservedAttributePaths` array[]: Paths to search for service values - string[]: Paths to search for service values - `serviceReservedAttributes` string[]: Attributes to search for service values - `statusMapping` StatusMapping: Maps raw status values to severities. - `debug` StatusValues: Raw status strings that map to one severity. Matching is case-sensitive. - … - `error` StatusValues: Raw status strings that map to one severity. Matching is case-sensitive. - … - `fatal` StatusValues: Raw status strings that map to one severity. Matching is case-sensitive. - … - `info` StatusValues: Raw status strings that map to one severity. Matching is case-sensitive. - … - `numberRanges` StatusNumberRange[]: Inclusive ranges mapping numeric status values to severities. - … - `trace` StatusValues: Raw status strings that map to one severity. Matching is case-sensitive. - … - `warn` StatusValues: Raw status strings that map to one severity. Matching is case-sensitive. - … - `statusPreserveSource` boolean: When true, preserves the source status attribute after remapping instead of removing it. Defaults to true (source is preserved). - `statusReservedAttributePaths` array[]: Attributes to search for status values - string[]: Attributes to search for status values - `statusReservedAttributes` string[]: Attributes to search for status values - `tagMappingPaths` object: Map from tag key to an ordered list of attribute paths to search for values. All paths will be tried, and all found values will be added. If no value is found, the tag will not be changed. The attribute paths are a list of paths. This does not remove existing tag values. - `tagMappings` object: Map from tag key to an ordered list of attribute paths to search for values. All paths will be tried, and all found values will be added. If no value is found, the tag will not be changed. The attribute paths are dot-separated. This does not remove existing tag values. - `tagMappingsLocationPaths` array[]: Possible locations of a map of string to string or to collection of strings that has tags. Added to tags, without replacement. - string[]: Possible locations of a map of string to string or to collection of strings that has tags. Added to tags, without replacement. - `tagMappingsLocationPreserveSource` boolean: When true, preserves the source tag map location attributes after remapping instead of removing them. Defaults to false (source is removed after remapping). - `tagMappingsLocations` string[]: Possible locations of a map of string to string or to collection of strings that has tags. Added to tags, without replacement. - `tagMappingsPreserveSource` boolean: When true, preserves the source tag mappings attributes after remapping instead of removing them. Defaults to false (source is removed). - `timestampPreserveSource` boolean: When true, preserves the source timestamp attribute after remapping instead of removing it. Defaults to true (source is preserved). - `timestampReservedAttributePaths` array[]: Paths to search for timestamp values - string[]: Paths to search for timestamp values - `timestampReservedAttributes` string[]: Attributes to search for timestamp values - `traceIdPreserveSource` boolean: When true, preserves the source traceId attribute after remapping instead of removing it. Defaults to true (source is preserved). - `traceIdReservedAttributePaths` array[]: Paths to search for traceId values - string[]: Paths to search for traceId values - `traceIdReservedAttributes` string[]: Attributes to search for traceId values - `type` 'log-attributes-remapper', required: Maps attributes to top-level fields and tags. - object - `attributeMergeStrategyEntries` AttributesMergeStrategyEntry[]: List of attribute paths and their corresponding merge strategies. If no strategy is defined for a path, the terminal merge strategy will be used. - `attributePath` string[], required: The attribute path, represented as an array of strings (e.g., ["user", "id"]). - `strategy` union, required: Default merge strategy to use for merging leaf-valued attributes. This is used when no specific merge strategy is defined for an attribute path. - … - `attributeTokenization` 'KEYS_ONLY' | 'KEYS_AND_VALUES': Use attribute shape only, or shape and whole-value similarity tokens. - `dedupThreshold` integer: Number of duplicates we send for a pattern before we start deduplication. - `defaultAttributesMergeStrategy` union: Default merge strategy to use for merging leaf-valued attributes. This is used when no specific merge strategy is defined for an attribute path. - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - `delimiters` string[], required: Characters to use for splitting tokens. Default is missing '@' to avoid splitting email addresses. Also missing '.' to avoid splitting identifiers using it. - `emitNewPatternSignals` boolean: Whether this operation may emit LogPatternSignal events to the agent-service when new patterns are detected or replayed. Must be enabled before an agent can subscribe to this pipeline. - `enabledMasks` string[], required: Masks that are enabled for tokenizing messages. These are specified as the names of the masks to use. The default enabled masks are: ipport, timestamp, uuid, number. - `excludedAttributePrefixes` array[]: Excluded attribute prefixes as arrays of literal key segments. Does not affect partition-by values or output attributes. - string[]: Excluded attribute prefixes as arrays of literal key segments. Does not affect partition-by values or output attributes. - `includedAttributePrefixes` array[]: Included attribute prefixes as arrays of literal key segments. Empty selects all. Array elements are traversed without indices. - string[]: Included attribute prefixes as arrays of literal key segments. Empty selects all. Array elements are traversed without indices. - `integrationExceptionConfigs` IntegrationExceptionConfig[]: Exception configs that define which exceptions to use from the ones imported from the observability vendor integrations. - `autoSync` boolean: When enabled, the exceptions list will be updated automatically based on the integration's exceptions. - `autosyncPassthroughRateThreshold` number, double: Maximum impact percentage threshold for autosync (0.0 to 100.0). When set, only exceptions with impact percentage <= threshold are auto-included. - `datasetId` string: Dataset against which the auto-sync impact threshold is evaluated. Required for threshold filtering to apply; when null, no exceptions bypass via auto-sync. - `ids` string[] - `integrationId` string, required: The ID of the integration associated with this exception configuration. - `logReducerExceptions` EventPredicate[]: Predicate for skipping events. Logs matching these predicates won't be aggregated. - union: Predicate applied to rows before analytical aggregation. - … - `masks` array[], required: Set of masks that can be applied to messages before tokenizing them. These masks are specified as pairs of (name, regex). The default masks are: ipport, timestamp, uuid, number, awsarn, awstoken. - string[]: Set of masks that can be applied to messages before tokenizing them. These masks are specified as pairs of (name, regex). The default masks are: ipport, timestamp, uuid, number, awsarn, awstoken. - `name` string, required - `partitionByAttributePaths` array[]: Attribute paths as arrays of keys. Use for unambiguous nested vs flat attributes. Example: [["user", "id"]] for nested user.id, [["user.id"]] for flat attribute named 'user.id' - string[]: Attribute paths as arrays of keys. Use for unambiguous nested vs flat attributes. Example: [["user", "id"]] for nested user.id, [["user.id"]] for flat attribute named 'user.id' - `partitionByAttributes` string[] - `partitionByTags` string[] - `reductionTimeWindow` string, ISO-8601: Time window in which logs are considered for deduplication. Should not be modified without explicit guidance. - `repeatCountInSummaryMessageEnabled` boolean: Whether to include the repeat count prefix in the summary message. Grepr will add a prefix to the summary message with the number messages aggregated. However, this might break downstream parsing if the message is expected to be in a specific format. If you want to include the repeat count, set this to true. - `samplingConfig` union: Determines how often samples should be emitted - object - … - object - … - `similarityThreshold` number, double: Controls how close a set of tokens are to a pattern before it is considered a match. Calculated as the percent of matching tokens between a pattern and a message. - `tokenizationInput` 'MESSAGE_ONLY' | 'ATTRIBUTES_ONLY' | 'MESSAGE_AND_ATTRIBUTES' | 'AUTO': Selects message text, attributes, both, or message text with an attributes fallback. - `type` 'log-reducer', required: Reduces logs by combining messages with similar patterns. - object - `name` string, required - `transforms` EventAction[] - union: Base class of all Event Actions - … - `type` 'log-transform', required: Applies transformations to log events. - object - `attributeMasks` AttributeMask[], required: Masks to apply to attribute values. Each entry names an attribute path (as a list of segments) and the masks to apply to the value at that path. Only string values at the exact path are masked; missing or non-string attributes are skipped. Paths must be unique. The same matching rules as messageMasks apply. - `masks` object, required: Masks to apply to the value at this path, as a map of label to mask. The same matching rules as messageMasks apply. - `path` string[], required: Attribute path to mask, as a list of segments. - `messageMasks` object, required: Masks to apply to the log message, as a map of label to mask. A mask may be written as a bare regex string, in which case each match is replaced with the label in square brackets — a match of the regex labeled "email" becomes "[email]". Labels may be up to 64 characters. Overlapping matches resolve to the leftmost match first, then the longest; declaration order breaks remaining ties, so declare more specific patterns first. - `name` string, required - `predicate` union: Predicate applied to rows before analytical aggregation. - object - … - object - … - object - … - object - … - object - … - `type` 'masking-operator', required: Masks sensitive substrings in log events by replacing regex matches with decorated labels. - object - `attributeTokenization` 'KEYS_ONLY' | 'KEYS_AND_VALUES': Use attribute shape only, or shape and whole-value similarity tokens. - `delimiters` string[], required: Characters to use for splitting tokens. - `emitNewPatternSignals` boolean: Whether this operation may emit LogPatternSignal events to the agent-service when new patterns are detected or replayed. Must be enabled before an agent can subscribe to this pipeline. - `enabledMasks` string[], required: Masks that are enabled for tokenizing messages. - `excludedAttributePrefixes` array[]: Excluded attribute prefixes as arrays of literal key segments. Does not affect partition-by values or output attributes. - string[]: Excluded attribute prefixes as arrays of literal key segments. Does not affect partition-by values or output attributes. - `includedAttributePrefixes` array[]: Included attribute prefixes as arrays of literal key segments. Empty selects all. Array elements are traversed without indices. - string[]: Included attribute prefixes as arrays of literal key segments. Empty selects all. Array elements are traversed without indices. - `masks` array[], required: Set of masks that can be applied to messages before tokenizing them. These masks are specified as pairs of (name, regex). - string[]: Set of masks that can be applied to messages before tokenizing them. These masks are specified as pairs of (name, regex). - `name` string, required - `partitionByAttributePaths` array[]: Attribute paths as arrays of keys. Use for unambiguous nested vs flat attributes. Example: [["user", "id"]] for nested user.id, [["user.id"]] for flat attribute named 'user.id' - string[] - `partitionByAttributes` string[] - `partitionByTags` string[] - `similarityThreshold` number, double: Controls how close a set of tokens are to a pattern before it is considered a match. Calculated as the percent of matching tokens between a pattern and a message. - `tokenizationInput` 'MESSAGE_ONLY' | 'ATTRIBUTES_ONLY' | 'MESSAGE_AND_ATTRIBUTES' | 'AUTO': Selects message text, attributes, both, or message text with an attributes fallback. - `type` 'pattern-matcher', required: Matches log patterns using an iterative feedback loop. Events with known patterns skip the global reducer for improved efficiency. - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'logs-iceberg-table-sink', required: Writes logs to an Iceberg Table. - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'pattern-lookup-iceberg-table-sink', required: Writes event id to pattern id mappings to an Iceberg Table. - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'event-dedup-iceberg-table-sink', required: Writes event IDs to an Iceberg table to keep track of which events have been sent to a particular sink. - `vendorSinkId` string, required - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'vendorlog-event-dedup-iceberg-table-sink', required: Writes confirmed-delivered event IDs to an Iceberg table to keep track of which events have actually been sent to a particular sink. - `vendorSinkId` string, required - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'spans-dedup-iceberg-table-sink', required: Writes span identity tuples to an Iceberg table to keep track of which spans have been sent to a particular vendor sink. - `vendorSinkId` string, required - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'spans-iceberg-table-sink', required: Writes individual spans to the raw spans Iceberg table. - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'logs-iceberg-table-source', required: Reads events from an Iceberg table for processing. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'agent-sessions-iceberg-table-source', required: Selects grouped agent session summaries from the logs Iceberg table. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'agent-session-events-iceberg-table-source', required: Reads every event of the agent sessions whose events match the query. - `variables` object: Variables that can be used to modify the query while parsed - object - `aggSpec` MetricAggSpec, required: The metrics aggregation to apply to the source data. - `groupByTags` string[]: The tags to group the results by. If unspecified, all series are grouped together. Only applicable when spaceAggregation is set. - `spaceAggregation` 'LATEST' | 'SUM' | 'AVG' | 'MAX': The space aggregation to use for the query. If null, no grouping is done and the timeseries are returned with full tags. - `timeAggregation` 'LATEST' | 'SUM' | 'AVG' | 'MAX': The time aggregation to use for the query. If this is set, timeGranularity must also be provided. If null, no time aggregation is done and the raw datapoints are returned. - `timeGranularity` string, ISO-8601: The time granularity to use for the query. Only applicable when timeAggregation is set. - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'metrics-iceberg-table-source', required: Reads metrics out of an Iceberg Table for processing. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `hasError` boolean: Filter by error status - `isRootSpan` boolean: Filter by parent span ID (null for root spans only) - `limit` integer: The maximum number of rows to process - `maxDuration` integer: Filter by maximum duration in nanoseconds - `minDuration` integer: Filter by minimum duration in nanoseconds - `name` string, required - `offset` integer: Offset on the number of rows to process - `operationNames` string[]: Filter by operation names (exact match) - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `serviceNames` string[]: Filter by service names (exact match) - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `traceIds` string[]: Filter by trace IDs (32-character hex format) - `traceSignatureIds` string[]: Filter by trace signature ids (exact match) - `traceSignatures` string[]: Filter by trace signature text (supports wildcards with *) - `type` 'traces-iceberg-table-source', required: Reads traces out of an Iceberg Table for processing. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'spans-backfill-iceberg-table-source', required: Reads spans from an Iceberg table for backfill, skipping spans already sent to the given vendor sinks (idempotent backfill source). - `variables` object: Variables that can be used to modify the query while parsed - `vendorSinkIntegrationIds` string[] - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'metrics-iceberg-table-sink', required: Writes metrics to an Iceberg table. - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'metricdata-raw-iceberg-table-sink', required: Writes MetricData (one row per datapoint, attributes auto-flattened to columns) to an Iceberg table. - object - `filterQuery` string: The Datadog query filter to use when reading logs from Datadog. This is concatenated with the one from the integration. - `integrationId` string, required: The Datadog integration id. - `name` string, required - `type` 'datadog-log-cloud-source', required: Pulls log events from the Datadog cloud service for processing using a specified query. - object - `hostEnrichmentEnabled` boolean - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'datadog-log-agent-source', required: Reads log events from Grepr's Datadog collection agent. This is used to pre-process logs before being sent to Datadog. - object - `discardStats` boolean: Whether stats should be discarded instead of being forwarded to Datadog - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'datadog-trace-agent-source', required: Reads dd_trace protos from Datadog Agent. This is used to process traces from Datadog. - object - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `type` string, required - object - `additionalTags` object: Additional tags to add to the metrics while loading. - `metrics` string[]: The list of metrics to load. Will load all possible metrics if nothing is specified - `name` string, required - `type` 'grepr-metrics-source', required: Reads metric events from Grepr for processing. - object - `additionalTags` string[]: Additional tags to add to the logs when writing to Datadog. Gets combined with the integration's additionalTags. - `browserLogGroupByAttributes` string[]: Group by attributes to use when writing logs to Datadog browser intake.Defaults to ["http.useragent", "network.client.ip"]. - `hostFieldNames` string[]: Top-level JSON field names to write the host to. Datadog only maps one of these onto the log's host if the name is in the destination organization's "Preprocessing for JSON logs -> Host attributes" list, so an organization that has removed the default entry needs this overridden. Defaults to ["hostname"]. - `integrationId` string, required: The Datadog integration id. - `name` string, required - `type` 'datadog-log-sink', required: Writes processed events to Datadog. - object: Converts CompleteSpan objects to Datadog's native trace format and sends them to the Datadog platform. - `additionalAttributes` object: Additional attributes to add to traces when writing to Datadog. Gets combined with the integration's additionalAttributes. - `integrationId` string, required: Integration ID for Datadog connection - `name` string, required - `type` 'datadog-trace-sink', required: Writes processed traces to Datadog in native format. - object - `additionalAttributes` object: Additional attributes to add to the logs when writing to New Relic. Gets combined with the integration's additionalAttributes. - `integrationId` string, required: The New Relic integration id. - `name` string, required - `type` 'newrelic-log-sink', required: Writes logs to NewRelic. - object - `additionalTags` string[]: Additional tags to add to the logs when writing to Splunk. Gets combined with the integration's additionalTags. - `defaultIndex` string: Default index to be set when writing to Splunk if no index is set. - `indexOverride` string: Override index to be set when writing to Splunk - `integrationId` string, required: The Splunk integration id. - `name` string, required - `type` 'splunk-log-sink', required: Writes processed events to splunk. - object - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'sumologic-log-agent-source', required: Reads log events from SumoLogic HTTP Source collector. This is used to pre-process logs before being sent to SumoLogic. - object - `additionalAttributes` object: Additional attributes to add to the logs when writing to SumoLogic. Gets combined with the integration's additionalAttributes. - `integrationId` string, required: The New Relic integration id. - `name` string, required - `type` 'sumologic-log-sink', required: Writes processed logs to Sumo Logic. - object - `additionalAttributes` object: Additional attributes to add to the logs when writing to OTLP destination. Gets combined with the integration's additionalAttributes. - `integrationId` string, required: The OTLP integration id. - `name` string, required - `type` 'otlp-log-sink', required: Writes processed logs to any OTLP-compatible destination. - object - `additionalAttributes` object: Additional attributes to add to the logs when writing to OTLP destination. Gets combined with the integration's additionalAttributes. - `integrationId` string, required: The OTLP integration id. - `name` string, required - `type` 'otlp-trace-sink', required: Writes processed logs to any OTLP-compatible destination. - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'logs-iceberg-replay-source', required: Replays log events from an Iceberg table at their original ingestion rate using receivedTimestamp for timing. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'logs-backfill-iceberg-table-source', required: Backfills events out of an Iceberg Table, returning events with the sinks they have already been sent to. - `variables` object: Variables that can be used to modify the query while parsed - `vendorSinkIntegrationIds` string[] - object - `conditions` object, required: Map of condition Id to condition objects. The ids are a unique name for the conditions. - `name` string, required - `triggers` object, required: Map of trigger Id to trigger objects. The ids are a unique name for the trigger. - `type` 'rule-engine', required: Describes rules to execute on log events. See the guide for the Rule Engine for more details. - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `processedLogsTableName` string: The name of the table containing processed logs. - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `reductionInterval` string, ISO-8601, required: The interval used by the Log Reducer for the query. Uses ISO 8601 format for intervals. - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'reducer-logs-iceberg-table-source', required: Queries raw and processed logs from a reducer pipeline. - `variables` object: Variables that can be used to modify the query while parsed - object - `name` string, required - `type` 'logs-sync-sink', required: Sink for returning log results to the caller. - object - `name` string, required - `type` 'agent-sessions-sync-sink', required: Sink for returning agent session summaries to the caller. - object - `name` string, required - `type` 'metrics-sync-sink', required: Sink for returning metric results to the caller. - object - `name` string, required - `type` 'variant-sync-sink', required: Sink for returning arbitrary JSON results to the caller. - object - `name` string, required - `type` 'spans-sync-sink', required: Sink for returning results to the caller. - object - `columns` string[]: Ordered CSV column names; required only for CSV artifacts - `format` 'COMPRESSED_JSON' | 'JSON_LINES' | 'CSV', required: Artifact serialization format - `integrationId` string, required: The integration to use for writing to the file - `name` string, required - `path` string: Destination path relative to the selected warehouse root - `type` 'query-sink', required: Write a bounded artifact to a warehouse provided by the given integration id. - object: Emits a generic pipeline signal for each input log event. - `context` string: Optional author-provided context included with every emitted signal. - `name` string, required - `targetAgentIds` string[], required: Agent ids that should investigate every signal emitted by this sink. - `type` 'agent-signal-sink', required: Emits generic pipeline signals to selected agents. - object: Sink that discards all of its input. - `name` string, required - `type` 'discarding-sink', required: Sink that discards all of its input. - object - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'grepr-pipe-log-source', required: Reads log events that other pipelines write into a Grepr pipe. - object: Writes log events into a Grepr pipe integration. - `integrationId` string, required: Destination Grepr pipe integration in the pipeline's organization. - `name` string, required - `type` 'grepr-pipe-log-sink', required - object - `additionalTags` string[]: Additional tags to add to the logs when writing to Datadog. Gets combined with the integration's additionalTags. - `integrationId` string, required: The Datadog integration id. - `name` string, required - `type` 'datadog-metrics-sink', required: Writes processed metrics to Datadog. - object - `additionalTags` string[]: Additional tags to add to the metrics when writing to Datadog. Gets combined with the integration's additionalTags. - `integrationId` string, required: The Datadog integration id. - `name` string, required - `type` 'legacy-datadog-metrics-sink', required: Writes processed metrics (flat MetricDatapoint model) to Datadog. Transitional; use datadog-metrics-sink for new pipelines. - object - `name` string, required - `query` AnalyticsQuery, required: Complete analytical query to execute. - `end` string, date-time, required: End of the time interval to query. - `from` string, required: Logical table alias to query. - `groupBy` string[]: Grepr SQL expressions or projection aliases used for grouping. An expression may reference projection aliases. - `having` string: Optional Grepr SQL expression applied after aggregation. The expression may reference projection aliases. - `limit` integer: Maximum number of result rows. - `orderBy` AnalyticsOrderBy[]: Ordering applied to the analytical results. - … - `predicate` union, required: Predicate applied to rows before analytical aggregation. - … - `select` AnalyticsProjection[], required: Expressions returned by the query. - … - `start` string, date-time, required: Start of the time interval to query. - `variables` object: Variables available while parsing the predicate. - `window` AnalyticsTimeWindow: Optional time window that buckets the results. Adds leading 'window_start' and 'window_end' result columns, which are always grouped. - … - `type` string, required - `scanBudgetBytes` integer: Maximum number of bytes this query may scan. - `tables` AnalyticsTableBinding[], required: Dataset tables available to the query under logical aliases. - `alias` string, required: Logical table name used by the analytical query. - `datasetId` string, required: ID of the dataset that owns the table. - `table` 'logs_raw' | 'logs_dedup' | 'logs_pattern_lookup' | 'metrics_raw' | 'metricdata_raw' | 'spans_raw' | 'spans_dedup', required: Dataset-owned table exposed under the logical alias. - `type` 'athena-analytics-source', required: Executes an analytical query against dataset tables using Athena. - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `processedLogsTableName` string: The name of the table containing processed logs. - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `reductionInterval` string, ISO-8601, required: The interval used by the Log Reducer for the query. Uses ISO 8601 format for intervals. - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'grepr-reducer-log-source', required: Reads events using Grepr's internal query system out of an Iceberg Table for processing. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'athena-agent-sessions-source', required: Returns grouped agent session summaries through Athena. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'grepr-raw-log-source', required: Reads raw log events using Grepr's internal query system from an Iceberg Table without pattern enrichment. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'grepr-raw-span-source', required: Reads spans using Grepr's internal query system out of an Iceberg Table for processing. - `variables` object: Variables that can be used to modify the query while parsed - object - `name` string, required - `query` AnalyticsQuery, required: Complete analytical query to execute. - `end` string, date-time, required: End of the time interval to query. - `from` string, required: Logical table alias to query. - `groupBy` string[]: Grepr SQL expressions or projection aliases used for grouping. An expression may reference projection aliases. - `having` string: Optional Grepr SQL expression applied after aggregation. The expression may reference projection aliases. - `limit` integer: Maximum number of result rows. - `orderBy` AnalyticsOrderBy[]: Ordering applied to the analytical results. - … - `predicate` union, required: Predicate applied to rows before analytical aggregation. - … - `select` AnalyticsProjection[], required: Expressions returned by the query. - … - `start` string, date-time, required: Start of the time interval to query. - `variables` object: Variables available while parsing the predicate. - `window` AnalyticsTimeWindow: Optional time window that buckets the results. Adds leading 'window_start' and 'window_end' result columns, which are always grouped. - … - `type` string, required - `scanBudgetBytes` integer: Maximum number of bytes this query may process. - `tables` AnalyticsTableBinding[], required: Dataset tables available to the query under logical aliases. - `alias` string, required: Logical table name used by the analytical query. - `datasetId` string, required: ID of the dataset that owns the table. - `table` 'logs_raw' | 'logs_dedup' | 'logs_pattern_lookup' | 'metrics_raw' | 'metricdata_raw' | 'spans_raw' | 'spans_dedup', required: Dataset-owned table exposed under the logical alias. - `type` 'trino-analytics-source', required: Executes an analytical query using a Trino query engine. - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `processedLogsTableName` string: The name of the table containing processed logs. - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `reductionInterval` string, ISO-8601, required: The interval used by the Log Reducer for the query. Uses ISO 8601 format for intervals. - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'trino-reducer-log-source', required: Reads events using Grepr's internal query system out of an Iceberg Table for processing, executed against a Trino query engine. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'trino-raw-log-source', required: Reads raw log events out of an Iceberg Table without pattern enrichment, executed against a Trino query engine. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'trino-raw-span-source', required: Reads spans using Grepr's internal query system out of an Iceberg Table for processing, executed against a Trino query engine. - `variables` object: Variables that can be used to modify the query while parsed - object - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'splunk-log-agent-source', required: Reads log events from Splunk HEC collector. This is used to pre-process logs before being sent to Splunk. - object - `filterQuery` string: The Splunk query filter to use when reading logs from Splunk. This is concatenated with the one from the integration. - `integrationId` string, required: The Splunk integration id. - `name` string, required - `type` 'splunk-log-http-source', required: Pulls log events from Splunk server for processing using a specified query. - object - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'newrelic-log-agent-source', required: Reads log events from Grepr's New Relic collection agent. This is used to pre-process logs before being sent to New Relic. - object - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'otlp-log-agent-source', required: Reads log events from Open Telemetry collector. This is used to pre-process logs before being sent to an Otel sink. - object - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'otlp-trace-agent-source', required: Reads trace events from an OpenTelemetry collector. This is used to process traces from OTLP trace ingestion. - object - `name` string, required - `sampleDurationMillis` integer: How long (in milliseconds) a sampled trace ID remains marked for as sampled without new messages before being declared expired. After this duration with no more events for trace, subsequent events for the same trace will be re-evaluated for sampling if seen again. - `sampleRate` number, double, required: The probability (0.0 to 1.0) of sampling a newly seen trace ID. - `selector` union: Predicate applied to rows before analytical aggregation. - object - … - object - … - object - … - object - … - object - … - `traceIdAttributePaths` array[]: List of attribute paths (each path is a list of segments) to extract trace IDs from. The first non-null value found will be used as the trace ID. - string[]: List of attribute paths (each path is a list of segments) to extract trace IDs from. The first non-null value found will be used as the trace ID. - `traceIdTagKeys` string[]: List of tag keys to extract trace IDs from. The first non-null value found will be used as the trace ID (checked after attribute paths). - `traceSamplerTagKey` string: The key for the tag added to events of sampled traces to signal downstream operations differentiated processing. - `traceSamplerTagValue` string: The value for the tag added to events of sampled traces to signal downstream operations differentiated processing. - `type` 'trace-sampler', required: Samples traces based on their ID and marks associated events that are part of the selected traces. This can be used to bypass downstream processing such as message reduction, ensuring complete unmodified logs for those selected traces. This operation can add some overhead since it might require an additional shuffle in the pipeline. Messages that have trace IDs will be output on the "traced" output, while messages that do not have any trace IDs be output on the "untraced" output. - object: Overlays a fixed set of attributes onto every span passing through this operation. Used by the trace draft pipeline to stamp stage tags on previewed spans. - `attributes` object, required: Attributes to overlay onto each span's attributes map. - `name` string, required - `type` 'span-tagger', required: Overlays a fixed set of attributes onto every span passing through. - object - `fileReadingStrategy` FileReadingStrategy: The file reading strategy will determine how files are enumerated. Timestamp reading strategy will only read files created after the specified timestamp. Latest reading strategy will only read files created after the job start time. - `type` string, required - `format` FileFormat, required: Describes the file format to read from. - `mapper` LogEventMapper: Selects the mapper used to decode raw file records into LogEvents. - … - `type` string, required - `name` string, required - `pathPrefix` string: The s3 path under the provided bucket to monitor. Optional.In the case of CloudTrail logs, this is the path to the AWSLogs/ directory. - `s3IntegrationId` string, required: Integration ID for the S3 bucket where Grepr will be reading data. This corresponds to the 's3-data-warehouse' integration type. - `streamingPollInterval` string, ISO-8601: The interval at which Grepr will check for new files in the S3 bucket. This is only applicable in streaming mode. - `type` 's3-file-source', required: Reads generic data from an S3 path. Each record will need to be mapped to the Grepr data model before being usable. - object - `accountIds` string[], required: The list of account IDs from which to read logs. - `fileReadingStrategy` FileReadingStrategy: The file reading strategy will determine how files are enumerated. Timestamp reading strategy will only read files created after the specified timestamp. Latest reading strategy will only read files created after the job start time. - `type` string, required - `format` JsonFileFormat: JSON file format. - `charset` string: Character set name such as UTF-8, UTF-16, etc. - `mapper` JsonLogEventMapper: JSON-to-LogEvent mapper. - `recordsPath` string[]: Path to the list of log records. - `streamFormat` StreamFormatAny - … - `name` string, required - `pathPrefix` string: The s3 path under the provided bucket to monitor. Optional.In the case of CloudTrail logs, this is the path to the AWSLogs/ directory. - `regions` string[]: The list of regions from which to read logs from. If empty or null, reads logs from all regions. - `s3IntegrationId` string, required: Integration ID for the S3 bucket where Grepr will be reading data. This corresponds to the 's3-data-warehouse' integration type. - `streamingPollInterval` string, ISO-8601: The interval at which Grepr will check for new files in the S3 bucket. This is only applicable in streaming mode. - `type` 'cloudtrail-file-source', required: Reads logs from an S3 bucket at a CloudTrail path. Only reads logs in folders at or after the specified start time. The path prefix is optional. Each record will need to be mapped to the Grepr data model before being usable. - object - `format` FileFormat, required: Describes the file format to read from. - `mapper` LogEventMapper: Selects the mapper used to decode raw file records into LogEvents. - … - `type` string, required - `name` string, required - `type` 'grepr-uploaded-log-file-source', required: Reads log files uploaded to Grepr using environment credentials. Intended for file uploads in draft mode. - `uploadId` string, required: Unique identifier for the upload. The full S3 path will be constructed as uploaded_files/{organizationId}/{uploadId} to ensure customers can only access their own uploads. Use the uploadId returned by the file upload endpoint. - object - `draftMode` boolean: Set as a variable in the script scope, this input tells the script to run in 'draft' mode, which should enable additional outputs from various parts of the built pipeline, according to the Template's defined draft outputs. - `name` string, required - `templateId` string, required: The ID of the template to use for this operation. - `templateInputs` object: Inputs to the template. Each entry in the map is added to the scope of the template javascript as a variable and can be accessed directly. In addition to the inputs, a 'user' object is also bound providing information about the current user. - `templateVersion` integer: The template's version. Required for a template your organization owns. Built-in templates ignore it and run the version that ships with the pipeline's Grepr version. - `type` 'template-operation', required: Runs a template script to generate a subgraph running in place of the operation in the parent graph. - object: Processes data through a sequence of SQL statements executed in order. Supports VIEW statements that create temporary tables, OUTPUT statements that produce DataStream results, and IO statements that perform side effects. - `availableDatasets` string[]: IDs of datasets whose tables can be referenced in SQL statements. These will be registered as queryable tables in the SQL environment. The tables from each dataset will appear in a separate namespace based on the dataset name. - `globalStateTtl` string, ISO-8601: Global time-to-live for operation state. When calculating aggregations or joins in streaming mode, Grepr pipelines will keep track of some state (such as the count and sum of values for an averaging aggregation). To prevent the size of the state from growing indefinitely, Grepr will set a default time-to-live value for any state. If state does not receive updates after this TTL, it will be considered idle and the state will be cleared. Aggregations for will be reset for the idle group for example. - `inputs` object: Input table names that can be referenced in SQL statements. Each input table corresponds to an input stream that is connected to this operation in the job graph. All connected input types must be compatible with the SQL operations. Each type specifies what the schema of the resulting table that is available for the SQL statements looks like. For `COMPLETE_SPAN`, the table schema uses SQL row field names, which differ from the REST JSON property names in the `CompleteSpan` schema. - `name` string, required - `statements` SqlExecutable[], required: List of SQL statements to execute in order. Each statement can reference tables created by previous statements, the input tables, and any tables from the available datasets. - union: Base class for SQL statements - … - `type` 'sql-operation', required: Processes data through a sequence of SQL statements. - `watermarkDelay` string, ISO-8601: Delay for watermark generation. Default is 5 seconds. Watermarks are used to track event time progress in streaming jobs. The watermark is usually the maximum event time seen by a processing task minus this delay. Watermarks are used to trigger time-based operations like windowing. This delay defines how far behind the system can be in processing events before it considers the event time to be 'late'. Late events would not be included in windowing operations that might have already been closed (ie where the data is already sent). This also means that the results of windowing operations will be delayed by this delay, as the system waits for the watermark to catch up. For more information, see the Flink documentation on watermarks. - object: Reduces trace volume by applying a uniform trace head sampling rate and configurable span head sampling rules with optional tail sampling, then emits the retained spans. - `actions` TraceAction[]: Actions to execute on traces. Actions process spans after sampling decisions have been made and can trigger operations like logs backfill. - `actionId` string, required: Unique identifier for this action within the TraceReducer. Must be alphanumeric with hyphens and underscores only. - `type` string, required - `criticalPathAnalysis` CriticalPathAnalysisConfig: Opt-in critical-path analysis settings. Computed during trace assembly; requires tail sampling. The maxSpans and sampleRate levers bound the per-trace cost. - `maxSpans` integer: Cap on how many spans are analyzed per trace; the critical chain is kept first. Null means no cap beyond the global per-trace assembly limit. - `sampleRate` number, double: Fraction of assembled traces to analyze, in [0.0, 1.0], selected deterministically by trace id. Defaults to 1.0 (analyze every trace). - `datasetId` string: Dataset ID used for naming the Iceberg table (traces_raw_{datasetId}). If not provided, traces will not be persisted to Iceberg. - `name` string, required - `siblingMergeRules` SiblingMergeRule[]: Rules that merge same-signature sibling spans matching a predicate, after tail sampling decisions. The earliest-start match is kept with its children; the other matching same-signature siblings are removed and counted under grepr.merged_siblings. Requires tail sampling to be configured. - `predicate` string, required: Boolean SQL expression evaluated per span over the CompleteSpan schema. Uses the same fields and functions as head sampling predicates: top-level columns such as servicename, nested fields such as span.operationName, and VARIANT_VALUE(...) for span/resource attributes. - `spanDropRules` SpanDropRule[]: Rules that drop matching spans and their descendants after tail sampling decisions. Requires tail sampling to be configured. - `predicate` string, required: Boolean SQL expression evaluated per span over the CompleteSpan schema. Uses the same fields and functions as head sampling predicates: top-level columns such as servicename, nested fields such as span.operationName, and VARIANT_VALUE(...) for span/resource attributes. - `spanHeadSamplingRules` SpanHeadSamplingRule[]: Span head sampling rules. Matched spans are retained as SPAN_HEAD_SAMPLED. If tail sampling is configured, matched spans participate in trace assembly before output; otherwise, matched spans are emitted directly. Order is significant; first match wins for metric attribution. Trace head sampling wins when both match the same span. - `filter` HeadSamplingFilter: Predicate that determines whether a span matches a head sampling rule. - … - `rate` number, double, required: Fraction of matching spans/traces to retain, in [0.0, 1.0]. - `tailSampling` TailSamplingConfig: Optional configuration for histogram-based tail sampling. When provided, enables intelligent sampling based on trace duration percentiles calculated over sliding windows. Uses hierarchical trace signatures to group similar traces and applies configurable sampling rates based on performance tiers (fast, medium, slow, outliers). - `bootstrapThreshold` integer: Number of traces to sample during bootstrap period before applying percentile-based sampling rules. This allows the system to gather initial data on trace durations before applying more complex sampling logic. This number will be multiplied by the parallelism of the pipeline, so the bootstrap traces may be higher as the pipeline expands for new trace signatures. - `errorSamplingRate` number, double: Sampling rate for traces that contain errors or failures. Error traces are often the most important for debugging and should typically have high sampling rates. Applied before quantile-based sampling. - `incompleteSamplingRate` number, double: Sampling rate for incomplete/partial traces. Partial traces occur when some spans are missing, which can indicate data loss or system issues. Applied before quantile-based sampling. - `keyAttributePaths` array[]: Array of attribute paths to extract key attributes from spans for hierarchical signature generation. Each inner array represents a sequence of keys to search for in the attributes map. These attributes are used to group traces by their structural patterns. - … - `longRunningTraceSamplingRate` number, double: Sampling rate for traces force-emitted because they exceeded the configured maxSessionLength. Routed through a dedicated 'long_running' percentile bucket — distinct from the structural 'incomplete' bucket — so operators can monitor and tune separately. - `maxSessionLength` string, ISO-8601: Upper bound on how long a single trace assembly session may accumulate spans in memory before being force-emitted. Protects against long-running traces that would otherwise hold spans in the assembly buffer indefinitely. When the cap is hit, the trace is classified as long-running and later spans form a new session. Must be strictly greater than maxTraceDelay. - `maxTraceDelay` string, ISO-8601: The maximum time to wait for all spans of a trace to arrive before assembling the trace. This determines the size of the tumbling window used for trace assembly. Traces that span longer than this duration may be assembled as partial traces. Should be set based on the expected maximum trace duration in your system. Large values may lead to memory issues if traces are too long and will delay sending traces to the sink by that amount. The pipeline may also fail if configured too high. - `quantileSamplingTiers` QuantileSamplingTier[]: List of quantile sampling tiers for performance-based sampling. Each tier defines a percentile threshold, sampling rate, and label. Traces ABOVE each quantile get the corresponding sampling rate and are labeled with the tier's label. Evaluated in descending quantile order. A tier at quantile 0.0 is a catch-all that matches unconditionally, giving the fastest traces a baseline rate; without one, traces below the lowest configured quantile are dropped. - … - `traceHeadSamplingRate` number, double: Uniform trace head sampling rate in [0.0, 1.0]. When set, all spans sharing a traceId are retained together as HEAD_SAMPLED at this probability. Mutually exclusive with traceHeadSamplingRules (scalar wins when both present). - `traceHeadSamplingRules` TraceHeadSamplingRule[]: Deprecated: use traceHeadSamplingRate instead. Kept for backwards compatibility with stored configurations. When traceHeadSamplingRate is absent, the effective rate is max(rate) over this list (filters are ignored). - `filter` HeadSamplingFilter: Predicate that determines whether a span matches a head sampling rule. - … - `rate` number, double, required: Fraction of matching spans/traces to retain, in [0.0, 1.0]. - `type` 'trace-reducer', required: Reduces trace volume by sampling CompleteSpan instances. Combines a uniform trace head sampling rate (whole-trace retention) and per-span head sampling rules with optional quantile-based tail sampling. - object: Reduces metric volume by aggregating per-object time series into per-cohort summaries while preserving the original signal when anomaly rules fire. Series that don't belong to a monitored object flow through unchanged; series matching a drop rule are discarded. - `anomalies` AnomalyConfig[]: Anomaly rules. - `anomalyDuration` string, ISO-8601, required: START..END lifecycle window after the last triggering datapoint. - `condition` string, required: Scalar SQL condition that fires the rule. - `from` object, required: Alias → aggregation declaration. - `joinOn` string[]: Join expressions linking aliases (Calcite-parsed SQL). - `name` string, required: Unique rule name. - `anomalyActions` AnomalyActions: Actions applied when an object is flagged anomalous. - `objectMetricsPassthrough` ObjectMetricsPassthrough: When an object is anomalous, de-aggregate its raw datapoints alongside the per-cohort aggregate output, under a suffixed metric name. - … - `cohorts` CohortsConfig: Cohort-identity + space-aggregation configuration. - `identifyingAttributes` string[]: Level-prefixed attribute keys that group objects into cohorts. - `metadataAttributes` string[]: Level-prefixed attribute keys describing cohorts. - `numShardBits` integer: Parallelism multiplier for cohort outputs (anomaly rules ignore this). Splits each cohort across 2^n CohortCalc instances by the low n bits of tsIdLo, trading parallelism for partial per-shard aggregation. 0 = one instance per cohort. - `outputs` CohortOutputDecl[]: Ordered cohort-output entries (first-match-wins). - … - `dropRules` DropRule[]: Drop rules — matching series are discarded outright. - `attributeFilters` object: Level-prefixed attribute key → glob pattern. All entries must match. - `metricName` MetricNameFilter: Glob-pattern filter on metric names. - … - `emitInterval` string, ISO-8601, required: How often the reducer flushes cohort-level output. - `hostMetadata` HostMetadataReporting: Keep a vendor's host inventory in sync with the metric reduction. - `aggregateCohortHosts` boolean: Collapse each cohort's real hosts into a single virtual cohort host. - `integrationIds` string[], required: Datadog integration IDs that receive the reported host metadata. - `rehydrateAnomalousHosts` boolean: While an object is anomalous, re-report its real host to Datadog. - `lateArrivalWaitDuration` string, ISO-8601, required: Watermark delay applied to give late-arriving data a chance to be aggregated. - `name` string, required - `objects` ObjectsConfig, required: Object-identity configuration. - `identifyingAttributes` string[], required: Level-prefixed attribute keys that identify an object. Each key must have a 'resource.', 'scope.', or 'series.' prefix. - `metadataAttributes` string[]: Level-prefixed attribute keys that describe an object but may change. - `metricsFilter` FilterPrimitive: Combined metric-name + attribute-value filter. - … - `spaceAggOverrides` object: Per-metric-name overrides for space (cross-object) aggregation. - `timeGranularity` TimeGranularity, required: Time-aggregation window and per-metric agg-fn overrides. - `aggFnOverrides` object: Per-metric-name overrides for time aggregation. - `duration` string, ISO-8601, required: Time-aggregation window; also the slide interval used by every aggregation. - `type` 'metric-reducer', required: Metric reducer. - object: Forwards Datadog stats payloads to the Datadog platform for accurate RED metrics. Stats are pre-computed by the Datadog agent and bypass all processing. - `additionalTags` object: Additional tags to add to stats when writing to Datadog. Gets combined with the integration's and global additionalTags. - `integrationId` string, required: Integration ID for Datadog connection - `name` string, required - `type` 'datadog-stats-sink', required: Writes Datadog stats payloads to Datadog for accurate RED metrics. - object - `loop` boolean: If true, the source will loop over the values indefinitely when processing is STREAMING. In BATCH mode, this flag is ignored. - `name` string, required - `type` 'logs-values-source', required: Generates log events with constant values. - `values` LogEvent[]: List of log events to be generated by the source. - `attributes` object: Attributes associated with the event, a map from attribute name to attribute value, which may be a nested map or list. - `eventTimestamp` integer, required: Event timestamp that is in the record. Milliseconds from epoch. - `id` string, required: Id of the event. - `message` string: The log message. - `receivedTimestamp` integer, required: Timestamp when the event was received by Grepr. Milliseconds from epoch. - `severity` integer: The log severity. Higher numbers mean higher severity, following the OTel format 1-24. - `tags` object: Tags associated with the event, a map from tag name to set of tag values. - object - `globalRules` PatternRuleConfig[]: Global rules applied to all log events regardless of pattern (e.g., log-based alert rules). - `enabled` boolean - `order` integer - `name` string, required - `rules` object: Per-pattern rules keyed by pattern ID (TSID string). - `type` 'log-rules-application', required: Applies per-pattern rules (masking, time-series spike detection) to log events. - object - `capacity` integer: Maximum number of concurrent trigger handler calls. - `name` string, required - `timeoutSeconds` integer: Timeout for trigger handler execution in seconds. - `type` 'trigger-action', required: Dispatches trigger actions to rule-type-specific handlers. - object - `name` string, required - `ruleSets` MultiGrokRuleSet[]: Ordered list of grok rule sets. Each rule set has its own predicate, optional extract attribute, parsing rules, and helper rules. Rule sets extracting from enriched attributes should be ordered after rule sets that populate those attributes. - `extractAttribute` union: Optional attribute to extract from. If null, extracts from the log message. Rules that extract from enriched attributes should be ordered after rules that populate those attributes. A dotted string, or an array of literal keys. - … - `grokHelperRules` string[]: Helper rules scoped to this rule set - `grokParsingRules` string[]: Grok parsing rules for this rule set - `predicate` union: Predicate applied to rows before analytical aggregation. - … - `type` 'multi-grok-parser', required: Processes unstructured logs and extracts fields from them using Grok patterns. - object - `analysisJobId` string, required - `analysisVersion` string, required - `datasetId` string, required - `embeddingCapacity` integer - `embeddingDimensions` integer - `embeddingModel` string, required - `end` string, date-time, required - `labelModel` string, required - `maxRetries` integer - `maximumNamedGroups` integer - `modelIntegrationId` string, required - `name` string, required - `query` union, required: Predicate applied to rows before analytical aggregation. - object - … - object - … - object - … - object - … - object - … - `requestTimeout` string, ISO-8601, required: Duration in ISO-8601 format (e.g., 'PT20.345S' for 20.345 seconds). - `retainedAnalyticsMemoryLimitBytes` integer - `sessionCap` integer - `start` string, date-time, required - `type` 'agent-session-analytics', required - object: Processes native agent-harness OTel logs into canonical agent LogEvents. - `name` string, required - `type` 'agent-log-processor', required: Processes native agent-harness OTel logs. - object: Normalizes agent span attributes and projects observed transcript content as canonical agent LogEvents in one pass. - `name` string, required - `type` 'agent-span-processor', required: Processes native agent-harness OTel spans. - `start` string, date-time, required: Start of the time interval to backfill, inclusive. - `tags` object: Custom tags added to the backfill job for ease of search. - `teamIds` string[]: The team IDs that the backfill job is associated with. - `variables` object: Variables used to modify the query while it is parsed. - `vendorSinkIntegrationIds` string[]: Vendor integration IDs, if any, whose prior delivery the backfill skips: exactly the integrations the sinks send to, or empty. Also gates concurrent overlapping backfills. Empty (the default) does no per-vendor dedup join and never blocks on other backfills. - object: Parameters for a spans backfill. - `dataType` 'spans', required: Selects the built-in spans backfill template. - `datasetId` string, required: The dataset the backfill reads from and the dedup sinks track delivery in. - `end` string, date-time, required: End of the time interval to backfill. - `limit` integer: Maximum number of rows to backfill. Default and max is 500,000. For spans this applies independently to each vendor source. A negative value means no limit. - `name` string, required: Name of the backfill job. Must match the job name pattern [a-z0-9_]{1,128}. - `query` union, required: Predicate applied to rows before analytical aggregation. - object - `queries` EventPredicate[], required: List of predicates that must all match for this predicate to match. - `type` 'and-predicate', required: A predicate that matches when every child predicate matches. - `query` string - object - `query` string, required: The datadog query used by the predicate. See https://docs.datadoghq.com/tracing/trace_explorer/query_syntax - `type` 'datadog-query', required: A predicate that checks if an event matches a datadog query. - object - `filterTags` object: The set of tags to query for. If unspecified, all tags are queried. - `metricNames` string[], required: The names of the metrics to query. Must not be empty. - `query` string, required - `type` 'metric-predicate', required: A predicate that checks if an event matches a metrics query. - object - `query` string, required: The New Relic query used by the predicate. See https://docs.newrelic.com/docs/logs/ui-data/query-syntax-logs/ - `type` 'newrelic-query', required: A predicate that checks if an event matches a New Relic query. - object - `query` string, required: The Nrql query used by the predicate. See https://docs.newrelic.com/docs/nrql - `strict` boolean: Whether the query should be parsed in strict mode. - `type` 'nrql-query', required: A predicate that checks if an event matches an NRQL query. - `sinks` Operation[], required: The vendor sink operations that deliver the backfilled events. Passed through verbatim, matching the sinks a rule action would carry. - union: Base class of all Operations - object - `filter` union: Predicate applied to rows before analytical aggregation. - object - … - object - … - object - … - object - … - object - … - `maxAllowedRate` number, double: Max allowed rate in messages per second. The max rate used by the token rate limiter to control the sampling rate of events. The rate is distributed across parallel subtasks. - `maxBurstLimit` integer: Maximum size of acceptable burst used by the token bucket rate limiter to handle bursty traffic. - `name` string, required - `type` 'logs-event-sampler', required: Samples log events using a token based rate limiter. Throttles log events based on max rate accepting up to max burst limit for bursty traffic. Can selectively sample events matching a filter predicate. Rate limiting is distributed across parallel subtasks. - object - `name` string, required - `predicate` union, required: Predicate applied to rows before analytical aggregation. - object - … - object - … - object - … - object - … - object - … - `type` 'logs-branch', required: Splits events based on a predicate. Positive matches go to the main output. Negative matches go to the 'else' output. - object - `inverted` boolean: Whether to invert the predicate logic - `maxLateEventTimestampDelta` string, ISO-8601: Maximum allowed time delta between received and event timestamps. Events with a delta exceeding this threshold will be filtered out. ISO-8601 duration format. - `name` string, required - `predicate` union, required: Predicate applied to rows before analytical aggregation. - object - … - object - … - object - … - object - … - object - … - `type` 'logs-filter', required: Filters events based on a predicate. - object - `extractAttribute` union: Optional attribute to extract. Does nothing if it does not refer to a string attribute. A dotted string, or an array of literal keys. - string - string[] - `grokHelperRules` string[] - `grokParsingRules` string[]: The Grok parser enriches log events by parsing unstructured text into structured attributes. - `name` string, required - `predicate` union: Predicate applied to rows before analytical aggregation. - object - … - object - … - object - … - object - … - object - … - `type` 'grok-parser', required: Processes unstructured logs and extracts fields from them using Grok patterns. - object - `liftFieldNames` string[]: Object-valued JSON fields whose children are lifted into the containing parsed object. An empty list disables lifting. Lifting repeats recursively when a lifted object contains another configured field. Colliding maps are merged recursively. For leaf values and all other type combinations, the lifted child replaces the existing value; arrays are not combined. If multiple configured fields contain the same child, the later field wins according to these rules. - `maxNestedDepthForFields` integer - `maxNestedJsonIterations` integer: Maximum number of iterations to perform when resolving nested JSON fields. - `name` string, required - `preserveJsonMessageField` boolean: The processor tries to deserialize the string in the 'message' field as JSON, putting all values into the 'attributes' field. If this is enabled, the processor will keep the JSON string in 'message'. When disabled, the string will be cleared. - `resolveNestedJsonFields` array[]: If you have JSON that is double serialized, e.g. '{"field": "{\"nested\": \"value\"}"}', you might want to deserialize the nested JSON fields as well. This option allows you to specify the paths within the attributes where we should look for nested JSON fields. All the paths are tried in every iteration, and the deserialized fields are added to the root of attributes. Use this for singly serialized JSON attribute fields as well. - string[]: If you have JSON that is double serialized, e.g. '{"field": "{\"nested\": \"value\"}"}', you might want to deserialize the nested JSON fields as well. This option allows you to specify the paths within the attributes where we should look for nested JSON fields. All the paths are tried in every iteration, and the deserialized fields are added to the root of attributes. Use this for singly serialized JSON attribute fields as well. - `type` 'json-log-processor', required: Deserializes the 'message' field as JSON merging the result into the 'attributes' map. Can also deserialize attributes that contain JSON strings. - object - `attributeRemappingRules` AttributeRemappingRule[]: List of attribute-to-attribute remapping rules. Each rule moves the value from a source attribute path to a target attribute path, removing the source unless the rule's preserveSource flag is set. An optional predicate filter controls when each rule is applied. - `predicate` union: Predicate applied to rows before analytical aggregation. - … - `preserveSource` boolean: When true, the source attribute is preserved (copied) instead of being removed (moved). Defaults to false (source is removed after remapping). - `source` union, required: Source attribute path: a dotted string ("resource.service" is resource → service), or an array of literal keys (["http", "user.agent"]). - … - `target` union, required: Target attribute path: a dotted string, or an array of literal keys. - … - `hostPreserveSource` boolean: When true, preserves the source host attribute after remapping instead of removing it. Defaults to true (source is preserved). - `hostReservedAttributePaths` array[]: Paths to search for host values - string[]: Paths to search for host values - `hostReservedAttributes` string[]: Attributes to search for host values - `messagePreserveSource` boolean: When true, preserves the source message attribute after remapping instead of removing it. Defaults to false (source is removed after remapping). - `messageReservedAttributePaths` array[]: Paths to search for message values - string[]: Paths to search for message values - `messageReservedAttributes` string[]: Attributes to search for message values. The used attribute will be removed once remapped. - `name` string, required - `servicePreserveSource` boolean: When true, preserves the source service attribute after remapping instead of removing it. Defaults to true (source is preserved). - `serviceReservedAttributePaths` array[]: Paths to search for service values - string[]: Paths to search for service values - `serviceReservedAttributes` string[]: Attributes to search for service values - `statusMapping` StatusMapping: Maps raw status values to severities. - `debug` StatusValues: Raw status strings that map to one severity. Matching is case-sensitive. - … - `error` StatusValues: Raw status strings that map to one severity. Matching is case-sensitive. - … - `fatal` StatusValues: Raw status strings that map to one severity. Matching is case-sensitive. - … - `info` StatusValues: Raw status strings that map to one severity. Matching is case-sensitive. - … - `numberRanges` StatusNumberRange[]: Inclusive ranges mapping numeric status values to severities. - … - `trace` StatusValues: Raw status strings that map to one severity. Matching is case-sensitive. - … - `warn` StatusValues: Raw status strings that map to one severity. Matching is case-sensitive. - … - `statusPreserveSource` boolean: When true, preserves the source status attribute after remapping instead of removing it. Defaults to true (source is preserved). - `statusReservedAttributePaths` array[]: Attributes to search for status values - string[]: Attributes to search for status values - `statusReservedAttributes` string[]: Attributes to search for status values - `tagMappingPaths` object: Map from tag key to an ordered list of attribute paths to search for values. All paths will be tried, and all found values will be added. If no value is found, the tag will not be changed. The attribute paths are a list of paths. This does not remove existing tag values. - `tagMappings` object: Map from tag key to an ordered list of attribute paths to search for values. All paths will be tried, and all found values will be added. If no value is found, the tag will not be changed. The attribute paths are dot-separated. This does not remove existing tag values. - `tagMappingsLocationPaths` array[]: Possible locations of a map of string to string or to collection of strings that has tags. Added to tags, without replacement. - string[]: Possible locations of a map of string to string or to collection of strings that has tags. Added to tags, without replacement. - `tagMappingsLocationPreserveSource` boolean: When true, preserves the source tag map location attributes after remapping instead of removing them. Defaults to false (source is removed after remapping). - `tagMappingsLocations` string[]: Possible locations of a map of string to string or to collection of strings that has tags. Added to tags, without replacement. - `tagMappingsPreserveSource` boolean: When true, preserves the source tag mappings attributes after remapping instead of removing them. Defaults to false (source is removed). - `timestampPreserveSource` boolean: When true, preserves the source timestamp attribute after remapping instead of removing it. Defaults to true (source is preserved). - `timestampReservedAttributePaths` array[]: Paths to search for timestamp values - string[]: Paths to search for timestamp values - `timestampReservedAttributes` string[]: Attributes to search for timestamp values - `traceIdPreserveSource` boolean: When true, preserves the source traceId attribute after remapping instead of removing it. Defaults to true (source is preserved). - `traceIdReservedAttributePaths` array[]: Paths to search for traceId values - string[]: Paths to search for traceId values - `traceIdReservedAttributes` string[]: Attributes to search for traceId values - `type` 'log-attributes-remapper', required: Maps attributes to top-level fields and tags. - object - `attributeMergeStrategyEntries` AttributesMergeStrategyEntry[]: List of attribute paths and their corresponding merge strategies. If no strategy is defined for a path, the terminal merge strategy will be used. - `attributePath` string[], required: The attribute path, represented as an array of strings (e.g., ["user", "id"]). - `strategy` union, required: Default merge strategy to use for merging leaf-valued attributes. This is used when no specific merge strategy is defined for an attribute path. - … - `attributeTokenization` 'KEYS_ONLY' | 'KEYS_AND_VALUES': Use attribute shape only, or shape and whole-value similarity tokens. - `dedupThreshold` integer: Number of duplicates we send for a pattern before we start deduplication. - `defaultAttributesMergeStrategy` union: Default merge strategy to use for merging leaf-valued attributes. This is used when no specific merge strategy is defined for an attribute path. - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - `delimiters` string[], required: Characters to use for splitting tokens. Default is missing '@' to avoid splitting email addresses. Also missing '.' to avoid splitting identifiers using it. - `emitNewPatternSignals` boolean: Whether this operation may emit LogPatternSignal events to the agent-service when new patterns are detected or replayed. Must be enabled before an agent can subscribe to this pipeline. - `enabledMasks` string[], required: Masks that are enabled for tokenizing messages. These are specified as the names of the masks to use. The default enabled masks are: ipport, timestamp, uuid, number. - `excludedAttributePrefixes` array[]: Excluded attribute prefixes as arrays of literal key segments. Does not affect partition-by values or output attributes. - string[]: Excluded attribute prefixes as arrays of literal key segments. Does not affect partition-by values or output attributes. - `includedAttributePrefixes` array[]: Included attribute prefixes as arrays of literal key segments. Empty selects all. Array elements are traversed without indices. - string[]: Included attribute prefixes as arrays of literal key segments. Empty selects all. Array elements are traversed without indices. - `integrationExceptionConfigs` IntegrationExceptionConfig[]: Exception configs that define which exceptions to use from the ones imported from the observability vendor integrations. - `autoSync` boolean: When enabled, the exceptions list will be updated automatically based on the integration's exceptions. - `autosyncPassthroughRateThreshold` number, double: Maximum impact percentage threshold for autosync (0.0 to 100.0). When set, only exceptions with impact percentage <= threshold are auto-included. - `datasetId` string: Dataset against which the auto-sync impact threshold is evaluated. Required for threshold filtering to apply; when null, no exceptions bypass via auto-sync. - `ids` string[] - `integrationId` string, required: The ID of the integration associated with this exception configuration. - `logReducerExceptions` EventPredicate[]: Predicate for skipping events. Logs matching these predicates won't be aggregated. - union: Predicate applied to rows before analytical aggregation. - … - `masks` array[], required: Set of masks that can be applied to messages before tokenizing them. These masks are specified as pairs of (name, regex). The default masks are: ipport, timestamp, uuid, number, awsarn, awstoken. - string[]: Set of masks that can be applied to messages before tokenizing them. These masks are specified as pairs of (name, regex). The default masks are: ipport, timestamp, uuid, number, awsarn, awstoken. - `name` string, required - `partitionByAttributePaths` array[]: Attribute paths as arrays of keys. Use for unambiguous nested vs flat attributes. Example: [["user", "id"]] for nested user.id, [["user.id"]] for flat attribute named 'user.id' - string[]: Attribute paths as arrays of keys. Use for unambiguous nested vs flat attributes. Example: [["user", "id"]] for nested user.id, [["user.id"]] for flat attribute named 'user.id' - `partitionByAttributes` string[] - `partitionByTags` string[] - `reductionTimeWindow` string, ISO-8601: Time window in which logs are considered for deduplication. Should not be modified without explicit guidance. - `repeatCountInSummaryMessageEnabled` boolean: Whether to include the repeat count prefix in the summary message. Grepr will add a prefix to the summary message with the number messages aggregated. However, this might break downstream parsing if the message is expected to be in a specific format. If you want to include the repeat count, set this to true. - `samplingConfig` union: Determines how often samples should be emitted - object - … - object - … - `similarityThreshold` number, double: Controls how close a set of tokens are to a pattern before it is considered a match. Calculated as the percent of matching tokens between a pattern and a message. - `tokenizationInput` 'MESSAGE_ONLY' | 'ATTRIBUTES_ONLY' | 'MESSAGE_AND_ATTRIBUTES' | 'AUTO': Selects message text, attributes, both, or message text with an attributes fallback. - `type` 'log-reducer', required: Reduces logs by combining messages with similar patterns. - object - `name` string, required - `transforms` EventAction[] - union: Base class of all Event Actions - … - `type` 'log-transform', required: Applies transformations to log events. - object - `attributeMasks` AttributeMask[], required: Masks to apply to attribute values. Each entry names an attribute path (as a list of segments) and the masks to apply to the value at that path. Only string values at the exact path are masked; missing or non-string attributes are skipped. Paths must be unique. The same matching rules as messageMasks apply. - `masks` object, required: Masks to apply to the value at this path, as a map of label to mask. The same matching rules as messageMasks apply. - `path` string[], required: Attribute path to mask, as a list of segments. - `messageMasks` object, required: Masks to apply to the log message, as a map of label to mask. A mask may be written as a bare regex string, in which case each match is replaced with the label in square brackets — a match of the regex labeled "email" becomes "[email]". Labels may be up to 64 characters. Overlapping matches resolve to the leftmost match first, then the longest; declaration order breaks remaining ties, so declare more specific patterns first. - `name` string, required - `predicate` union: Predicate applied to rows before analytical aggregation. - object - … - object - … - object - … - object - … - object - … - `type` 'masking-operator', required: Masks sensitive substrings in log events by replacing regex matches with decorated labels. - object - `attributeTokenization` 'KEYS_ONLY' | 'KEYS_AND_VALUES': Use attribute shape only, or shape and whole-value similarity tokens. - `delimiters` string[], required: Characters to use for splitting tokens. - `emitNewPatternSignals` boolean: Whether this operation may emit LogPatternSignal events to the agent-service when new patterns are detected or replayed. Must be enabled before an agent can subscribe to this pipeline. - `enabledMasks` string[], required: Masks that are enabled for tokenizing messages. - `excludedAttributePrefixes` array[]: Excluded attribute prefixes as arrays of literal key segments. Does not affect partition-by values or output attributes. - string[]: Excluded attribute prefixes as arrays of literal key segments. Does not affect partition-by values or output attributes. - `includedAttributePrefixes` array[]: Included attribute prefixes as arrays of literal key segments. Empty selects all. Array elements are traversed without indices. - string[]: Included attribute prefixes as arrays of literal key segments. Empty selects all. Array elements are traversed without indices. - `masks` array[], required: Set of masks that can be applied to messages before tokenizing them. These masks are specified as pairs of (name, regex). - string[]: Set of masks that can be applied to messages before tokenizing them. These masks are specified as pairs of (name, regex). - `name` string, required - `partitionByAttributePaths` array[]: Attribute paths as arrays of keys. Use for unambiguous nested vs flat attributes. Example: [["user", "id"]] for nested user.id, [["user.id"]] for flat attribute named 'user.id' - string[] - `partitionByAttributes` string[] - `partitionByTags` string[] - `similarityThreshold` number, double: Controls how close a set of tokens are to a pattern before it is considered a match. Calculated as the percent of matching tokens between a pattern and a message. - `tokenizationInput` 'MESSAGE_ONLY' | 'ATTRIBUTES_ONLY' | 'MESSAGE_AND_ATTRIBUTES' | 'AUTO': Selects message text, attributes, both, or message text with an attributes fallback. - `type` 'pattern-matcher', required: Matches log patterns using an iterative feedback loop. Events with known patterns skip the global reducer for improved efficiency. - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'logs-iceberg-table-sink', required: Writes logs to an Iceberg Table. - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'pattern-lookup-iceberg-table-sink', required: Writes event id to pattern id mappings to an Iceberg Table. - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'event-dedup-iceberg-table-sink', required: Writes event IDs to an Iceberg table to keep track of which events have been sent to a particular sink. - `vendorSinkId` string, required - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'vendorlog-event-dedup-iceberg-table-sink', required: Writes confirmed-delivered event IDs to an Iceberg table to keep track of which events have actually been sent to a particular sink. - `vendorSinkId` string, required - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'spans-dedup-iceberg-table-sink', required: Writes span identity tuples to an Iceberg table to keep track of which spans have been sent to a particular vendor sink. - `vendorSinkId` string, required - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'spans-iceberg-table-sink', required: Writes individual spans to the raw spans Iceberg table. - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'logs-iceberg-table-source', required: Reads events from an Iceberg table for processing. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'agent-sessions-iceberg-table-source', required: Selects grouped agent session summaries from the logs Iceberg table. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'agent-session-events-iceberg-table-source', required: Reads every event of the agent sessions whose events match the query. - `variables` object: Variables that can be used to modify the query while parsed - object - `aggSpec` MetricAggSpec, required: The metrics aggregation to apply to the source data. - `groupByTags` string[]: The tags to group the results by. If unspecified, all series are grouped together. Only applicable when spaceAggregation is set. - `spaceAggregation` 'LATEST' | 'SUM' | 'AVG' | 'MAX': The space aggregation to use for the query. If null, no grouping is done and the timeseries are returned with full tags. - `timeAggregation` 'LATEST' | 'SUM' | 'AVG' | 'MAX': The time aggregation to use for the query. If this is set, timeGranularity must also be provided. If null, no time aggregation is done and the raw datapoints are returned. - `timeGranularity` string, ISO-8601: The time granularity to use for the query. Only applicable when timeAggregation is set. - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'metrics-iceberg-table-source', required: Reads metrics out of an Iceberg Table for processing. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `hasError` boolean: Filter by error status - `isRootSpan` boolean: Filter by parent span ID (null for root spans only) - `limit` integer: The maximum number of rows to process - `maxDuration` integer: Filter by maximum duration in nanoseconds - `minDuration` integer: Filter by minimum duration in nanoseconds - `name` string, required - `offset` integer: Offset on the number of rows to process - `operationNames` string[]: Filter by operation names (exact match) - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `serviceNames` string[]: Filter by service names (exact match) - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `traceIds` string[]: Filter by trace IDs (32-character hex format) - `traceSignatureIds` string[]: Filter by trace signature ids (exact match) - `traceSignatures` string[]: Filter by trace signature text (supports wildcards with *) - `type` 'traces-iceberg-table-source', required: Reads traces out of an Iceberg Table for processing. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'spans-backfill-iceberg-table-source', required: Reads spans from an Iceberg table for backfill, skipping spans already sent to the given vendor sinks (idempotent backfill source). - `variables` object: Variables that can be used to modify the query while parsed - `vendorSinkIntegrationIds` string[] - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'metrics-iceberg-table-sink', required: Writes metrics to an Iceberg table. - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'metricdata-raw-iceberg-table-sink', required: Writes MetricData (one row per datapoint, attributes auto-flattened to columns) to an Iceberg table. - object - `filterQuery` string: The Datadog query filter to use when reading logs from Datadog. This is concatenated with the one from the integration. - `integrationId` string, required: The Datadog integration id. - `name` string, required - `type` 'datadog-log-cloud-source', required: Pulls log events from the Datadog cloud service for processing using a specified query. - object - `hostEnrichmentEnabled` boolean - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'datadog-log-agent-source', required: Reads log events from Grepr's Datadog collection agent. This is used to pre-process logs before being sent to Datadog. - object - `discardStats` boolean: Whether stats should be discarded instead of being forwarded to Datadog - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'datadog-trace-agent-source', required: Reads dd_trace protos from Datadog Agent. This is used to process traces from Datadog. - object - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `type` string, required - object - `additionalTags` object: Additional tags to add to the metrics while loading. - `metrics` string[]: The list of metrics to load. Will load all possible metrics if nothing is specified - `name` string, required - `type` 'grepr-metrics-source', required: Reads metric events from Grepr for processing. - object - `additionalTags` string[]: Additional tags to add to the logs when writing to Datadog. Gets combined with the integration's additionalTags. - `browserLogGroupByAttributes` string[]: Group by attributes to use when writing logs to Datadog browser intake.Defaults to ["http.useragent", "network.client.ip"]. - `hostFieldNames` string[]: Top-level JSON field names to write the host to. Datadog only maps one of these onto the log's host if the name is in the destination organization's "Preprocessing for JSON logs -> Host attributes" list, so an organization that has removed the default entry needs this overridden. Defaults to ["hostname"]. - `integrationId` string, required: The Datadog integration id. - `name` string, required - `type` 'datadog-log-sink', required: Writes processed events to Datadog. - object: Converts CompleteSpan objects to Datadog's native trace format and sends them to the Datadog platform. - `additionalAttributes` object: Additional attributes to add to traces when writing to Datadog. Gets combined with the integration's additionalAttributes. - `integrationId` string, required: Integration ID for Datadog connection - `name` string, required - `type` 'datadog-trace-sink', required: Writes processed traces to Datadog in native format. - object - `additionalAttributes` object: Additional attributes to add to the logs when writing to New Relic. Gets combined with the integration's additionalAttributes. - `integrationId` string, required: The New Relic integration id. - `name` string, required - `type` 'newrelic-log-sink', required: Writes logs to NewRelic. - object - `additionalTags` string[]: Additional tags to add to the logs when writing to Splunk. Gets combined with the integration's additionalTags. - `defaultIndex` string: Default index to be set when writing to Splunk if no index is set. - `indexOverride` string: Override index to be set when writing to Splunk - `integrationId` string, required: The Splunk integration id. - `name` string, required - `type` 'splunk-log-sink', required: Writes processed events to splunk. - object - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'sumologic-log-agent-source', required: Reads log events from SumoLogic HTTP Source collector. This is used to pre-process logs before being sent to SumoLogic. - object - `additionalAttributes` object: Additional attributes to add to the logs when writing to SumoLogic. Gets combined with the integration's additionalAttributes. - `integrationId` string, required: The New Relic integration id. - `name` string, required - `type` 'sumologic-log-sink', required: Writes processed logs to Sumo Logic. - object - `additionalAttributes` object: Additional attributes to add to the logs when writing to OTLP destination. Gets combined with the integration's additionalAttributes. - `integrationId` string, required: The OTLP integration id. - `name` string, required - `type` 'otlp-log-sink', required: Writes processed logs to any OTLP-compatible destination. - object - `additionalAttributes` object: Additional attributes to add to the logs when writing to OTLP destination. Gets combined with the integration's additionalAttributes. - `integrationId` string, required: The OTLP integration id. - `name` string, required - `type` 'otlp-trace-sink', required: Writes processed logs to any OTLP-compatible destination. - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'logs-iceberg-replay-source', required: Replays log events from an Iceberg table at their original ingestion rate using receivedTimestamp for timing. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'logs-backfill-iceberg-table-source', required: Backfills events out of an Iceberg Table, returning events with the sinks they have already been sent to. - `variables` object: Variables that can be used to modify the query while parsed - `vendorSinkIntegrationIds` string[] - object - `conditions` object, required: Map of condition Id to condition objects. The ids are a unique name for the conditions. - `name` string, required - `triggers` object, required: Map of trigger Id to trigger objects. The ids are a unique name for the trigger. - `type` 'rule-engine', required: Describes rules to execute on log events. See the guide for the Rule Engine for more details. - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `processedLogsTableName` string: The name of the table containing processed logs. - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `reductionInterval` string, ISO-8601, required: The interval used by the Log Reducer for the query. Uses ISO 8601 format for intervals. - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'reducer-logs-iceberg-table-source', required: Queries raw and processed logs from a reducer pipeline. - `variables` object: Variables that can be used to modify the query while parsed - object - `name` string, required - `type` 'logs-sync-sink', required: Sink for returning log results to the caller. - object - `name` string, required - `type` 'agent-sessions-sync-sink', required: Sink for returning agent session summaries to the caller. - object - `name` string, required - `type` 'metrics-sync-sink', required: Sink for returning metric results to the caller. - object - `name` string, required - `type` 'variant-sync-sink', required: Sink for returning arbitrary JSON results to the caller. - object - `name` string, required - `type` 'spans-sync-sink', required: Sink for returning results to the caller. - object - `columns` string[]: Ordered CSV column names; required only for CSV artifacts - `format` 'COMPRESSED_JSON' | 'JSON_LINES' | 'CSV', required: Artifact serialization format - `integrationId` string, required: The integration to use for writing to the file - `name` string, required - `path` string: Destination path relative to the selected warehouse root - `type` 'query-sink', required: Write a bounded artifact to a warehouse provided by the given integration id. - object: Emits a generic pipeline signal for each input log event. - `context` string: Optional author-provided context included with every emitted signal. - `name` string, required - `targetAgentIds` string[], required: Agent ids that should investigate every signal emitted by this sink. - `type` 'agent-signal-sink', required: Emits generic pipeline signals to selected agents. - object: Sink that discards all of its input. - `name` string, required - `type` 'discarding-sink', required: Sink that discards all of its input. - object - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'grepr-pipe-log-source', required: Reads log events that other pipelines write into a Grepr pipe. - object: Writes log events into a Grepr pipe integration. - `integrationId` string, required: Destination Grepr pipe integration in the pipeline's organization. - `name` string, required - `type` 'grepr-pipe-log-sink', required - object - `additionalTags` string[]: Additional tags to add to the logs when writing to Datadog. Gets combined with the integration's additionalTags. - `integrationId` string, required: The Datadog integration id. - `name` string, required - `type` 'datadog-metrics-sink', required: Writes processed metrics to Datadog. - object - `additionalTags` string[]: Additional tags to add to the metrics when writing to Datadog. Gets combined with the integration's additionalTags. - `integrationId` string, required: The Datadog integration id. - `name` string, required - `type` 'legacy-datadog-metrics-sink', required: Writes processed metrics (flat MetricDatapoint model) to Datadog. Transitional; use datadog-metrics-sink for new pipelines. - object - `name` string, required - `query` AnalyticsQuery, required: Complete analytical query to execute. - `end` string, date-time, required: End of the time interval to query. - `from` string, required: Logical table alias to query. - `groupBy` string[]: Grepr SQL expressions or projection aliases used for grouping. An expression may reference projection aliases. - `having` string: Optional Grepr SQL expression applied after aggregation. The expression may reference projection aliases. - `limit` integer: Maximum number of result rows. - `orderBy` AnalyticsOrderBy[]: Ordering applied to the analytical results. - … - `predicate` union, required: Predicate applied to rows before analytical aggregation. - … - `select` AnalyticsProjection[], required: Expressions returned by the query. - … - `start` string, date-time, required: Start of the time interval to query. - `variables` object: Variables available while parsing the predicate. - `window` AnalyticsTimeWindow: Optional time window that buckets the results. Adds leading 'window_start' and 'window_end' result columns, which are always grouped. - … - `type` string, required - `scanBudgetBytes` integer: Maximum number of bytes this query may scan. - `tables` AnalyticsTableBinding[], required: Dataset tables available to the query under logical aliases. - `alias` string, required: Logical table name used by the analytical query. - `datasetId` string, required: ID of the dataset that owns the table. - `table` 'logs_raw' | 'logs_dedup' | 'logs_pattern_lookup' | 'metrics_raw' | 'metricdata_raw' | 'spans_raw' | 'spans_dedup', required: Dataset-owned table exposed under the logical alias. - `type` 'athena-analytics-source', required: Executes an analytical query against dataset tables using Athena. - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `processedLogsTableName` string: The name of the table containing processed logs. - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `reductionInterval` string, ISO-8601, required: The interval used by the Log Reducer for the query. Uses ISO 8601 format for intervals. - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'grepr-reducer-log-source', required: Reads events using Grepr's internal query system out of an Iceberg Table for processing. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'athena-agent-sessions-source', required: Returns grouped agent session summaries through Athena. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'grepr-raw-log-source', required: Reads raw log events using Grepr's internal query system from an Iceberg Table without pattern enrichment. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'grepr-raw-span-source', required: Reads spans using Grepr's internal query system out of an Iceberg Table for processing. - `variables` object: Variables that can be used to modify the query while parsed - object - `name` string, required - `query` AnalyticsQuery, required: Complete analytical query to execute. - `end` string, date-time, required: End of the time interval to query. - `from` string, required: Logical table alias to query. - `groupBy` string[]: Grepr SQL expressions or projection aliases used for grouping. An expression may reference projection aliases. - `having` string: Optional Grepr SQL expression applied after aggregation. The expression may reference projection aliases. - `limit` integer: Maximum number of result rows. - `orderBy` AnalyticsOrderBy[]: Ordering applied to the analytical results. - … - `predicate` union, required: Predicate applied to rows before analytical aggregation. - … - `select` AnalyticsProjection[], required: Expressions returned by the query. - … - `start` string, date-time, required: Start of the time interval to query. - `variables` object: Variables available while parsing the predicate. - `window` AnalyticsTimeWindow: Optional time window that buckets the results. Adds leading 'window_start' and 'window_end' result columns, which are always grouped. - … - `type` string, required - `scanBudgetBytes` integer: Maximum number of bytes this query may process. - `tables` AnalyticsTableBinding[], required: Dataset tables available to the query under logical aliases. - `alias` string, required: Logical table name used by the analytical query. - `datasetId` string, required: ID of the dataset that owns the table. - `table` 'logs_raw' | 'logs_dedup' | 'logs_pattern_lookup' | 'metrics_raw' | 'metricdata_raw' | 'spans_raw' | 'spans_dedup', required: Dataset-owned table exposed under the logical alias. - `type` 'trino-analytics-source', required: Executes an analytical query using a Trino query engine. - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `processedLogsTableName` string: The name of the table containing processed logs. - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `reductionInterval` string, ISO-8601, required: The interval used by the Log Reducer for the query. Uses ISO 8601 format for intervals. - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'trino-reducer-log-source', required: Reads events using Grepr's internal query system out of an Iceberg Table for processing, executed against a Trino query engine. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'trino-raw-log-source', required: Reads raw log events out of an Iceberg Table without pattern enrichment, executed against a Trino query engine. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'trino-raw-span-source', required: Reads spans using Grepr's internal query system out of an Iceberg Table for processing, executed against a Trino query engine. - `variables` object: Variables that can be used to modify the query while parsed - object - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'splunk-log-agent-source', required: Reads log events from Splunk HEC collector. This is used to pre-process logs before being sent to Splunk. - object - `filterQuery` string: The Splunk query filter to use when reading logs from Splunk. This is concatenated with the one from the integration. - `integrationId` string, required: The Splunk integration id. - `name` string, required - `type` 'splunk-log-http-source', required: Pulls log events from Splunk server for processing using a specified query. - object - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'newrelic-log-agent-source', required: Reads log events from Grepr's New Relic collection agent. This is used to pre-process logs before being sent to New Relic. - object - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'otlp-log-agent-source', required: Reads log events from Open Telemetry collector. This is used to pre-process logs before being sent to an Otel sink. - object - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'otlp-trace-agent-source', required: Reads trace events from an OpenTelemetry collector. This is used to process traces from OTLP trace ingestion. - object - `name` string, required - `sampleDurationMillis` integer: How long (in milliseconds) a sampled trace ID remains marked for as sampled without new messages before being declared expired. After this duration with no more events for trace, subsequent events for the same trace will be re-evaluated for sampling if seen again. - `sampleRate` number, double, required: The probability (0.0 to 1.0) of sampling a newly seen trace ID. - `selector` union: Predicate applied to rows before analytical aggregation. - object - … - object - … - object - … - object - … - object - … - `traceIdAttributePaths` array[]: List of attribute paths (each path is a list of segments) to extract trace IDs from. The first non-null value found will be used as the trace ID. - string[]: List of attribute paths (each path is a list of segments) to extract trace IDs from. The first non-null value found will be used as the trace ID. - `traceIdTagKeys` string[]: List of tag keys to extract trace IDs from. The first non-null value found will be used as the trace ID (checked after attribute paths). - `traceSamplerTagKey` string: The key for the tag added to events of sampled traces to signal downstream operations differentiated processing. - `traceSamplerTagValue` string: The value for the tag added to events of sampled traces to signal downstream operations differentiated processing. - `type` 'trace-sampler', required: Samples traces based on their ID and marks associated events that are part of the selected traces. This can be used to bypass downstream processing such as message reduction, ensuring complete unmodified logs for those selected traces. This operation can add some overhead since it might require an additional shuffle in the pipeline. Messages that have trace IDs will be output on the "traced" output, while messages that do not have any trace IDs be output on the "untraced" output. - object: Overlays a fixed set of attributes onto every span passing through this operation. Used by the trace draft pipeline to stamp stage tags on previewed spans. - `attributes` object, required: Attributes to overlay onto each span's attributes map. - `name` string, required - `type` 'span-tagger', required: Overlays a fixed set of attributes onto every span passing through. - object - `fileReadingStrategy` FileReadingStrategy: The file reading strategy will determine how files are enumerated. Timestamp reading strategy will only read files created after the specified timestamp. Latest reading strategy will only read files created after the job start time. - `type` string, required - `format` FileFormat, required: Describes the file format to read from. - `mapper` LogEventMapper: Selects the mapper used to decode raw file records into LogEvents. - … - `type` string, required - `name` string, required - `pathPrefix` string: The s3 path under the provided bucket to monitor. Optional.In the case of CloudTrail logs, this is the path to the AWSLogs/ directory. - `s3IntegrationId` string, required: Integration ID for the S3 bucket where Grepr will be reading data. This corresponds to the 's3-data-warehouse' integration type. - `streamingPollInterval` string, ISO-8601: The interval at which Grepr will check for new files in the S3 bucket. This is only applicable in streaming mode. - `type` 's3-file-source', required: Reads generic data from an S3 path. Each record will need to be mapped to the Grepr data model before being usable. - object - `accountIds` string[], required: The list of account IDs from which to read logs. - `fileReadingStrategy` FileReadingStrategy: The file reading strategy will determine how files are enumerated. Timestamp reading strategy will only read files created after the specified timestamp. Latest reading strategy will only read files created after the job start time. - `type` string, required - `format` JsonFileFormat: JSON file format. - `charset` string: Character set name such as UTF-8, UTF-16, etc. - `mapper` JsonLogEventMapper: JSON-to-LogEvent mapper. - `recordsPath` string[]: Path to the list of log records. - `streamFormat` StreamFormatAny - … - `name` string, required - `pathPrefix` string: The s3 path under the provided bucket to monitor. Optional.In the case of CloudTrail logs, this is the path to the AWSLogs/ directory. - `regions` string[]: The list of regions from which to read logs from. If empty or null, reads logs from all regions. - `s3IntegrationId` string, required: Integration ID for the S3 bucket where Grepr will be reading data. This corresponds to the 's3-data-warehouse' integration type. - `streamingPollInterval` string, ISO-8601: The interval at which Grepr will check for new files in the S3 bucket. This is only applicable in streaming mode. - `type` 'cloudtrail-file-source', required: Reads logs from an S3 bucket at a CloudTrail path. Only reads logs in folders at or after the specified start time. The path prefix is optional. Each record will need to be mapped to the Grepr data model before being usable. - object - `format` FileFormat, required: Describes the file format to read from. - `mapper` LogEventMapper: Selects the mapper used to decode raw file records into LogEvents. - … - `type` string, required - `name` string, required - `type` 'grepr-uploaded-log-file-source', required: Reads log files uploaded to Grepr using environment credentials. Intended for file uploads in draft mode. - `uploadId` string, required: Unique identifier for the upload. The full S3 path will be constructed as uploaded_files/{organizationId}/{uploadId} to ensure customers can only access their own uploads. Use the uploadId returned by the file upload endpoint. - object - `draftMode` boolean: Set as a variable in the script scope, this input tells the script to run in 'draft' mode, which should enable additional outputs from various parts of the built pipeline, according to the Template's defined draft outputs. - `name` string, required - `templateId` string, required: The ID of the template to use for this operation. - `templateInputs` object: Inputs to the template. Each entry in the map is added to the scope of the template javascript as a variable and can be accessed directly. In addition to the inputs, a 'user' object is also bound providing information about the current user. - `templateVersion` integer: The template's version. Required for a template your organization owns. Built-in templates ignore it and run the version that ships with the pipeline's Grepr version. - `type` 'template-operation', required: Runs a template script to generate a subgraph running in place of the operation in the parent graph. - object: Processes data through a sequence of SQL statements executed in order. Supports VIEW statements that create temporary tables, OUTPUT statements that produce DataStream results, and IO statements that perform side effects. - `availableDatasets` string[]: IDs of datasets whose tables can be referenced in SQL statements. These will be registered as queryable tables in the SQL environment. The tables from each dataset will appear in a separate namespace based on the dataset name. - `globalStateTtl` string, ISO-8601: Global time-to-live for operation state. When calculating aggregations or joins in streaming mode, Grepr pipelines will keep track of some state (such as the count and sum of values for an averaging aggregation). To prevent the size of the state from growing indefinitely, Grepr will set a default time-to-live value for any state. If state does not receive updates after this TTL, it will be considered idle and the state will be cleared. Aggregations for will be reset for the idle group for example. - `inputs` object: Input table names that can be referenced in SQL statements. Each input table corresponds to an input stream that is connected to this operation in the job graph. All connected input types must be compatible with the SQL operations. Each type specifies what the schema of the resulting table that is available for the SQL statements looks like. For `COMPLETE_SPAN`, the table schema uses SQL row field names, which differ from the REST JSON property names in the `CompleteSpan` schema. - `name` string, required - `statements` SqlExecutable[], required: List of SQL statements to execute in order. Each statement can reference tables created by previous statements, the input tables, and any tables from the available datasets. - union: Base class for SQL statements - … - `type` 'sql-operation', required: Processes data through a sequence of SQL statements. - `watermarkDelay` string, ISO-8601: Delay for watermark generation. Default is 5 seconds. Watermarks are used to track event time progress in streaming jobs. The watermark is usually the maximum event time seen by a processing task minus this delay. Watermarks are used to trigger time-based operations like windowing. This delay defines how far behind the system can be in processing events before it considers the event time to be 'late'. Late events would not be included in windowing operations that might have already been closed (ie where the data is already sent). This also means that the results of windowing operations will be delayed by this delay, as the system waits for the watermark to catch up. For more information, see the Flink documentation on watermarks. - object: Reduces trace volume by applying a uniform trace head sampling rate and configurable span head sampling rules with optional tail sampling, then emits the retained spans. - `actions` TraceAction[]: Actions to execute on traces. Actions process spans after sampling decisions have been made and can trigger operations like logs backfill. - `actionId` string, required: Unique identifier for this action within the TraceReducer. Must be alphanumeric with hyphens and underscores only. - `type` string, required - `criticalPathAnalysis` CriticalPathAnalysisConfig: Opt-in critical-path analysis settings. Computed during trace assembly; requires tail sampling. The maxSpans and sampleRate levers bound the per-trace cost. - `maxSpans` integer: Cap on how many spans are analyzed per trace; the critical chain is kept first. Null means no cap beyond the global per-trace assembly limit. - `sampleRate` number, double: Fraction of assembled traces to analyze, in [0.0, 1.0], selected deterministically by trace id. Defaults to 1.0 (analyze every trace). - `datasetId` string: Dataset ID used for naming the Iceberg table (traces_raw_{datasetId}). If not provided, traces will not be persisted to Iceberg. - `name` string, required - `siblingMergeRules` SiblingMergeRule[]: Rules that merge same-signature sibling spans matching a predicate, after tail sampling decisions. The earliest-start match is kept with its children; the other matching same-signature siblings are removed and counted under grepr.merged_siblings. Requires tail sampling to be configured. - `predicate` string, required: Boolean SQL expression evaluated per span over the CompleteSpan schema. Uses the same fields and functions as head sampling predicates: top-level columns such as servicename, nested fields such as span.operationName, and VARIANT_VALUE(...) for span/resource attributes. - `spanDropRules` SpanDropRule[]: Rules that drop matching spans and their descendants after tail sampling decisions. Requires tail sampling to be configured. - `predicate` string, required: Boolean SQL expression evaluated per span over the CompleteSpan schema. Uses the same fields and functions as head sampling predicates: top-level columns such as servicename, nested fields such as span.operationName, and VARIANT_VALUE(...) for span/resource attributes. - `spanHeadSamplingRules` SpanHeadSamplingRule[]: Span head sampling rules. Matched spans are retained as SPAN_HEAD_SAMPLED. If tail sampling is configured, matched spans participate in trace assembly before output; otherwise, matched spans are emitted directly. Order is significant; first match wins for metric attribution. Trace head sampling wins when both match the same span. - `filter` HeadSamplingFilter: Predicate that determines whether a span matches a head sampling rule. - … - `rate` number, double, required: Fraction of matching spans/traces to retain, in [0.0, 1.0]. - `tailSampling` TailSamplingConfig: Optional configuration for histogram-based tail sampling. When provided, enables intelligent sampling based on trace duration percentiles calculated over sliding windows. Uses hierarchical trace signatures to group similar traces and applies configurable sampling rates based on performance tiers (fast, medium, slow, outliers). - `bootstrapThreshold` integer: Number of traces to sample during bootstrap period before applying percentile-based sampling rules. This allows the system to gather initial data on trace durations before applying more complex sampling logic. This number will be multiplied by the parallelism of the pipeline, so the bootstrap traces may be higher as the pipeline expands for new trace signatures. - `errorSamplingRate` number, double: Sampling rate for traces that contain errors or failures. Error traces are often the most important for debugging and should typically have high sampling rates. Applied before quantile-based sampling. - `incompleteSamplingRate` number, double: Sampling rate for incomplete/partial traces. Partial traces occur when some spans are missing, which can indicate data loss or system issues. Applied before quantile-based sampling. - `keyAttributePaths` array[]: Array of attribute paths to extract key attributes from spans for hierarchical signature generation. Each inner array represents a sequence of keys to search for in the attributes map. These attributes are used to group traces by their structural patterns. - … - `longRunningTraceSamplingRate` number, double: Sampling rate for traces force-emitted because they exceeded the configured maxSessionLength. Routed through a dedicated 'long_running' percentile bucket — distinct from the structural 'incomplete' bucket — so operators can monitor and tune separately. - `maxSessionLength` string, ISO-8601: Upper bound on how long a single trace assembly session may accumulate spans in memory before being force-emitted. Protects against long-running traces that would otherwise hold spans in the assembly buffer indefinitely. When the cap is hit, the trace is classified as long-running and later spans form a new session. Must be strictly greater than maxTraceDelay. - `maxTraceDelay` string, ISO-8601: The maximum time to wait for all spans of a trace to arrive before assembling the trace. This determines the size of the tumbling window used for trace assembly. Traces that span longer than this duration may be assembled as partial traces. Should be set based on the expected maximum trace duration in your system. Large values may lead to memory issues if traces are too long and will delay sending traces to the sink by that amount. The pipeline may also fail if configured too high. - `quantileSamplingTiers` QuantileSamplingTier[]: List of quantile sampling tiers for performance-based sampling. Each tier defines a percentile threshold, sampling rate, and label. Traces ABOVE each quantile get the corresponding sampling rate and are labeled with the tier's label. Evaluated in descending quantile order. A tier at quantile 0.0 is a catch-all that matches unconditionally, giving the fastest traces a baseline rate; without one, traces below the lowest configured quantile are dropped. - … - `traceHeadSamplingRate` number, double: Uniform trace head sampling rate in [0.0, 1.0]. When set, all spans sharing a traceId are retained together as HEAD_SAMPLED at this probability. Mutually exclusive with traceHeadSamplingRules (scalar wins when both present). - `traceHeadSamplingRules` TraceHeadSamplingRule[]: Deprecated: use traceHeadSamplingRate instead. Kept for backwards compatibility with stored configurations. When traceHeadSamplingRate is absent, the effective rate is max(rate) over this list (filters are ignored). - `filter` HeadSamplingFilter: Predicate that determines whether a span matches a head sampling rule. - … - `rate` number, double, required: Fraction of matching spans/traces to retain, in [0.0, 1.0]. - `type` 'trace-reducer', required: Reduces trace volume by sampling CompleteSpan instances. Combines a uniform trace head sampling rate (whole-trace retention) and per-span head sampling rules with optional quantile-based tail sampling. - object: Reduces metric volume by aggregating per-object time series into per-cohort summaries while preserving the original signal when anomaly rules fire. Series that don't belong to a monitored object flow through unchanged; series matching a drop rule are discarded. - `anomalies` AnomalyConfig[]: Anomaly rules. - `anomalyDuration` string, ISO-8601, required: START..END lifecycle window after the last triggering datapoint. - `condition` string, required: Scalar SQL condition that fires the rule. - `from` object, required: Alias → aggregation declaration. - `joinOn` string[]: Join expressions linking aliases (Calcite-parsed SQL). - `name` string, required: Unique rule name. - `anomalyActions` AnomalyActions: Actions applied when an object is flagged anomalous. - `objectMetricsPassthrough` ObjectMetricsPassthrough: When an object is anomalous, de-aggregate its raw datapoints alongside the per-cohort aggregate output, under a suffixed metric name. - … - `cohorts` CohortsConfig: Cohort-identity + space-aggregation configuration. - `identifyingAttributes` string[]: Level-prefixed attribute keys that group objects into cohorts. - `metadataAttributes` string[]: Level-prefixed attribute keys describing cohorts. - `numShardBits` integer: Parallelism multiplier for cohort outputs (anomaly rules ignore this). Splits each cohort across 2^n CohortCalc instances by the low n bits of tsIdLo, trading parallelism for partial per-shard aggregation. 0 = one instance per cohort. - `outputs` CohortOutputDecl[]: Ordered cohort-output entries (first-match-wins). - … - `dropRules` DropRule[]: Drop rules — matching series are discarded outright. - `attributeFilters` object: Level-prefixed attribute key → glob pattern. All entries must match. - `metricName` MetricNameFilter: Glob-pattern filter on metric names. - … - `emitInterval` string, ISO-8601, required: How often the reducer flushes cohort-level output. - `hostMetadata` HostMetadataReporting: Keep a vendor's host inventory in sync with the metric reduction. - `aggregateCohortHosts` boolean: Collapse each cohort's real hosts into a single virtual cohort host. - `integrationIds` string[], required: Datadog integration IDs that receive the reported host metadata. - `rehydrateAnomalousHosts` boolean: While an object is anomalous, re-report its real host to Datadog. - `lateArrivalWaitDuration` string, ISO-8601, required: Watermark delay applied to give late-arriving data a chance to be aggregated. - `name` string, required - `objects` ObjectsConfig, required: Object-identity configuration. - `identifyingAttributes` string[], required: Level-prefixed attribute keys that identify an object. Each key must have a 'resource.', 'scope.', or 'series.' prefix. - `metadataAttributes` string[]: Level-prefixed attribute keys that describe an object but may change. - `metricsFilter` FilterPrimitive: Combined metric-name + attribute-value filter. - … - `spaceAggOverrides` object: Per-metric-name overrides for space (cross-object) aggregation. - `timeGranularity` TimeGranularity, required: Time-aggregation window and per-metric agg-fn overrides. - `aggFnOverrides` object: Per-metric-name overrides for time aggregation. - `duration` string, ISO-8601, required: Time-aggregation window; also the slide interval used by every aggregation. - `type` 'metric-reducer', required: Metric reducer. - object: Forwards Datadog stats payloads to the Datadog platform for accurate RED metrics. Stats are pre-computed by the Datadog agent and bypass all processing. - `additionalTags` object: Additional tags to add to stats when writing to Datadog. Gets combined with the integration's and global additionalTags. - `integrationId` string, required: Integration ID for Datadog connection - `name` string, required - `type` 'datadog-stats-sink', required: Writes Datadog stats payloads to Datadog for accurate RED metrics. - object - `loop` boolean: If true, the source will loop over the values indefinitely when processing is STREAMING. In BATCH mode, this flag is ignored. - `name` string, required - `type` 'logs-values-source', required: Generates log events with constant values. - `values` LogEvent[]: List of log events to be generated by the source. - `attributes` object: Attributes associated with the event, a map from attribute name to attribute value, which may be a nested map or list. - `eventTimestamp` integer, required: Event timestamp that is in the record. Milliseconds from epoch. - `id` string, required: Id of the event. - `message` string: The log message. - `receivedTimestamp` integer, required: Timestamp when the event was received by Grepr. Milliseconds from epoch. - `severity` integer: The log severity. Higher numbers mean higher severity, following the OTel format 1-24. - `tags` object: Tags associated with the event, a map from tag name to set of tag values. - object - `globalRules` PatternRuleConfig[]: Global rules applied to all log events regardless of pattern (e.g., log-based alert rules). - `enabled` boolean - `order` integer - `name` string, required - `rules` object: Per-pattern rules keyed by pattern ID (TSID string). - `type` 'log-rules-application', required: Applies per-pattern rules (masking, time-series spike detection) to log events. - object - `capacity` integer: Maximum number of concurrent trigger handler calls. - `name` string, required - `timeoutSeconds` integer: Timeout for trigger handler execution in seconds. - `type` 'trigger-action', required: Dispatches trigger actions to rule-type-specific handlers. - object - `name` string, required - `ruleSets` MultiGrokRuleSet[]: Ordered list of grok rule sets. Each rule set has its own predicate, optional extract attribute, parsing rules, and helper rules. Rule sets extracting from enriched attributes should be ordered after rule sets that populate those attributes. - `extractAttribute` union: Optional attribute to extract from. If null, extracts from the log message. Rules that extract from enriched attributes should be ordered after rules that populate those attributes. A dotted string, or an array of literal keys. - … - `grokHelperRules` string[]: Helper rules scoped to this rule set - `grokParsingRules` string[]: Grok parsing rules for this rule set - `predicate` union: Predicate applied to rows before analytical aggregation. - … - `type` 'multi-grok-parser', required: Processes unstructured logs and extracts fields from them using Grok patterns. - object - `analysisJobId` string, required - `analysisVersion` string, required - `datasetId` string, required - `embeddingCapacity` integer - `embeddingDimensions` integer - `embeddingModel` string, required - `end` string, date-time, required - `labelModel` string, required - `maxRetries` integer - `maximumNamedGroups` integer - `modelIntegrationId` string, required - `name` string, required - `query` union, required: Predicate applied to rows before analytical aggregation. - object - … - object - … - object - … - object - … - object - … - `requestTimeout` string, ISO-8601, required: Duration in ISO-8601 format (e.g., 'PT20.345S' for 20.345 seconds). - `retainedAnalyticsMemoryLimitBytes` integer - `sessionCap` integer - `start` string, date-time, required - `type` 'agent-session-analytics', required - object: Processes native agent-harness OTel logs into canonical agent LogEvents. - `name` string, required - `type` 'agent-log-processor', required: Processes native agent-harness OTel logs. - object: Normalizes agent span attributes and projects observed transcript content as canonical agent LogEvents in one pass. - `name` string, required - `type` 'agent-span-processor', required: Processes native agent-harness OTel spans. - `sqlOperation` SqlOperation: Processes data through a sequence of SQL statements executed in order. Supports VIEW statements that create temporary tables, OUTPUT statements that produce DataStream results, and IO statements that perform side effects. - `availableDatasets` string[]: IDs of datasets whose tables can be referenced in SQL statements. These will be registered as queryable tables in the SQL environment. The tables from each dataset will appear in a separate namespace based on the dataset name. - `globalStateTtl` string, ISO-8601: Global time-to-live for operation state. When calculating aggregations or joins in streaming mode, Grepr pipelines will keep track of some state (such as the count and sum of values for an averaging aggregation). To prevent the size of the state from growing indefinitely, Grepr will set a default time-to-live value for any state. If state does not receive updates after this TTL, it will be considered idle and the state will be cleared. Aggregations for will be reset for the idle group for example. - `inputs` object: Input table names that can be referenced in SQL statements. Each input table corresponds to an input stream that is connected to this operation in the job graph. All connected input types must be compatible with the SQL operations. Each type specifies what the schema of the resulting table that is available for the SQL statements looks like. For `COMPLETE_SPAN`, the table schema uses SQL row field names, which differ from the REST JSON property names in the `CompleteSpan` schema. - `name` string, required - `statements` SqlExecutable[], required: List of SQL statements to execute in order. Each statement can reference tables created by previous statements, the input tables, and any tables from the available datasets. - union: Base class for SQL statements - object: SQL statement that creates a queryable temporary view - `eventTimeAttribute` string: Event time attribute column name for watermark re-establishment after materialization. - `materialized` boolean - `sqlQuery` string, required: The SQL query to execute. Can reference tables created by previous statements by their tableName, as well as the 'traces' input table and available datasets. - `tableName` string, required: Name of the output table that this statement produces. Subsequent statements can reference this table name directly in their SQL. Must be unique within the operation. - `type` string, required - object: SQL statement that produces a DataStream output - `eventTimeAttribute` string - `materialized` boolean - `outputName` string, required: Name of the output stream that this statement produces. Used to identify the output DataStream in the SqlStatementsExecutor results. Must be unique across all OUTPUT statements within the operation. - `outputType` 'VARIANT' | 'LOG_EVENT' | 'COMPLETE_SPAN' | 'METRIC_DATA' | 'WIDE_METRIC_DATA', required: Data types that are acceptable as inputs and outputs to SQL operations. See [SQL transform supported data types](/transforms/sql-transform/data-types) for the SQL row schemas. The `COMPLETE_SPAN` SQL row schema uses SQL field names that differ from the REST JSON property names documented in [CompleteSpan](/apis/api-spec#schema/CompleteSpan). - `sqlQuery` string, required: The SQL query to execute. Can reference tables created by previous statements by their tableName, as well as the 'traces' input table and available datasets. - `type` string, required - object: SQL statement that performs side effects like INSERT INTO - `eventTimeAttribute` string - `materialized` boolean - `sqlQuery` string, required: The SQL query to execute. Can reference tables created by previous statements by their tableName, as well as the 'traces' input table and available datasets. - `type` string, required - `type` 'sql-operation', required: Processes data through a sequence of SQL statements. - `watermarkDelay` string, ISO-8601: Delay for watermark generation. Default is 5 seconds. Watermarks are used to track event time progress in streaming jobs. The watermark is usually the maximum event time seen by a processing task minus this delay. Watermarks are used to trigger time-based operations like windowing. This delay defines how far behind the system can be in processing events before it considers the event time to be 'late'. Late events would not be included in windowing operations that might have already been closed (ie where the data is already sent). This also means that the results of windowing operations will be delayed by this delay, as the system waits for the watermark to catch up. For more information, see the Flink documentation on watermarks. - `start` string, date-time, required: Start of the time interval to backfill, inclusive. - `tags` object: Custom tags added to the backfill job for ease of search. - `teamIds` string[]: The team IDs that the backfill job is associated with. - `variables` object: Variables used to modify the query while it is parsed. ## Response `201` Backfill job submitted successfully - ReadJob: Attributes of the Grepr Job - `createdAt` string, date-time, required: Time the job was accepted for creation - `desiredState` 'CREATED' | 'PENDING' | 'INFRA_UPDATE' | 'INFRA_UPDATE_WAIT' | 'VERIFYING' | 'WAITING' | 'RUNNING' | 'STOPPED' | 'FINISHED' | 'FAILED' | 'CANCELLED' | 'DELETED' | 'UPDATING' | 'STARTING' | 'RESTARTING' | 'STOPPING' | 'ROLLING_BACK' | 'DELETING', required: Desired state of the job - `editedBy` string: User who created this job version, in format 'Full Name (userId)' or 'system' for automated operations - `endedAt` string, date-time: Time the job was ended - `execution` 'SYNCHRONOUS' | 'ASYNCHRONOUS', required: Execution mode of the job - `greprVersion` string: The version of Grepr used to create the job - `id` string, required: Id of the job - `jobGraph` GreprJobGraph, required - `edges` string[], required: Edges that connect the vertices of the Grepr job graph - `vertices` Operation[], required: Operations that make up the vertices of the Grepr job graph - union: Base class of all Operations - object - `filter` union: Predicate applied to rows before analytical aggregation. - object - … - object - … - object - … - object - … - object - … - `maxAllowedRate` number, double: Max allowed rate in messages per second. The max rate used by the token rate limiter to control the sampling rate of events. The rate is distributed across parallel subtasks. - `maxBurstLimit` integer: Maximum size of acceptable burst used by the token bucket rate limiter to handle bursty traffic. - `name` string, required - `type` 'logs-event-sampler', required: Samples log events using a token based rate limiter. Throttles log events based on max rate accepting up to max burst limit for bursty traffic. Can selectively sample events matching a filter predicate. Rate limiting is distributed across parallel subtasks. - object - `name` string, required - `predicate` union, required: Predicate applied to rows before analytical aggregation. - object - … - object - … - object - … - object - … - object - … - `type` 'logs-branch', required: Splits events based on a predicate. Positive matches go to the main output. Negative matches go to the 'else' output. - object - `inverted` boolean: Whether to invert the predicate logic - `maxLateEventTimestampDelta` string, ISO-8601: Maximum allowed time delta between received and event timestamps. Events with a delta exceeding this threshold will be filtered out. ISO-8601 duration format. - `name` string, required - `predicate` union, required: Predicate applied to rows before analytical aggregation. - object - … - object - … - object - … - object - … - object - … - `type` 'logs-filter', required: Filters events based on a predicate. - object - `extractAttribute` union: Optional attribute to extract. Does nothing if it does not refer to a string attribute. A dotted string, or an array of literal keys. - string - string[] - `grokHelperRules` string[] - `grokParsingRules` string[]: The Grok parser enriches log events by parsing unstructured text into structured attributes. - `name` string, required - `predicate` union: Predicate applied to rows before analytical aggregation. - object - … - object - … - object - … - object - … - object - … - `type` 'grok-parser', required: Processes unstructured logs and extracts fields from them using Grok patterns. - object - `liftFieldNames` string[]: Object-valued JSON fields whose children are lifted into the containing parsed object. An empty list disables lifting. Lifting repeats recursively when a lifted object contains another configured field. Colliding maps are merged recursively. For leaf values and all other type combinations, the lifted child replaces the existing value; arrays are not combined. If multiple configured fields contain the same child, the later field wins according to these rules. - `maxNestedDepthForFields` integer - `maxNestedJsonIterations` integer: Maximum number of iterations to perform when resolving nested JSON fields. - `name` string, required - `preserveJsonMessageField` boolean: The processor tries to deserialize the string in the 'message' field as JSON, putting all values into the 'attributes' field. If this is enabled, the processor will keep the JSON string in 'message'. When disabled, the string will be cleared. - `resolveNestedJsonFields` array[]: If you have JSON that is double serialized, e.g. '{"field": "{\"nested\": \"value\"}"}', you might want to deserialize the nested JSON fields as well. This option allows you to specify the paths within the attributes where we should look for nested JSON fields. All the paths are tried in every iteration, and the deserialized fields are added to the root of attributes. Use this for singly serialized JSON attribute fields as well. - string[]: If you have JSON that is double serialized, e.g. '{"field": "{\"nested\": \"value\"}"}', you might want to deserialize the nested JSON fields as well. This option allows you to specify the paths within the attributes where we should look for nested JSON fields. All the paths are tried in every iteration, and the deserialized fields are added to the root of attributes. Use this for singly serialized JSON attribute fields as well. - `type` 'json-log-processor', required: Deserializes the 'message' field as JSON merging the result into the 'attributes' map. Can also deserialize attributes that contain JSON strings. - object - `attributeRemappingRules` AttributeRemappingRule[]: List of attribute-to-attribute remapping rules. Each rule moves the value from a source attribute path to a target attribute path, removing the source unless the rule's preserveSource flag is set. An optional predicate filter controls when each rule is applied. - `predicate` union: Predicate applied to rows before analytical aggregation. - … - `preserveSource` boolean: When true, the source attribute is preserved (copied) instead of being removed (moved). Defaults to false (source is removed after remapping). - `source` union, required: Source attribute path: a dotted string ("resource.service" is resource → service), or an array of literal keys (["http", "user.agent"]). - … - `target` union, required: Target attribute path: a dotted string, or an array of literal keys. - … - `hostPreserveSource` boolean: When true, preserves the source host attribute after remapping instead of removing it. Defaults to true (source is preserved). - `hostReservedAttributePaths` array[]: Paths to search for host values - string[]: Paths to search for host values - `hostReservedAttributes` string[]: Attributes to search for host values - `messagePreserveSource` boolean: When true, preserves the source message attribute after remapping instead of removing it. Defaults to false (source is removed after remapping). - `messageReservedAttributePaths` array[]: Paths to search for message values - string[]: Paths to search for message values - `messageReservedAttributes` string[]: Attributes to search for message values. The used attribute will be removed once remapped. - `name` string, required - `servicePreserveSource` boolean: When true, preserves the source service attribute after remapping instead of removing it. Defaults to true (source is preserved). - `serviceReservedAttributePaths` array[]: Paths to search for service values - string[]: Paths to search for service values - `serviceReservedAttributes` string[]: Attributes to search for service values - `statusMapping` StatusMapping: Maps raw status values to severities. - `debug` StatusValues: Raw status strings that map to one severity. Matching is case-sensitive. - … - `error` StatusValues: Raw status strings that map to one severity. Matching is case-sensitive. - … - `fatal` StatusValues: Raw status strings that map to one severity. Matching is case-sensitive. - … - `info` StatusValues: Raw status strings that map to one severity. Matching is case-sensitive. - … - `numberRanges` StatusNumberRange[]: Inclusive ranges mapping numeric status values to severities. - … - `trace` StatusValues: Raw status strings that map to one severity. Matching is case-sensitive. - … - `warn` StatusValues: Raw status strings that map to one severity. Matching is case-sensitive. - … - `statusPreserveSource` boolean: When true, preserves the source status attribute after remapping instead of removing it. Defaults to true (source is preserved). - `statusReservedAttributePaths` array[]: Attributes to search for status values - string[]: Attributes to search for status values - `statusReservedAttributes` string[]: Attributes to search for status values - `tagMappingPaths` object: Map from tag key to an ordered list of attribute paths to search for values. All paths will be tried, and all found values will be added. If no value is found, the tag will not be changed. The attribute paths are a list of paths. This does not remove existing tag values. - `tagMappings` object: Map from tag key to an ordered list of attribute paths to search for values. All paths will be tried, and all found values will be added. If no value is found, the tag will not be changed. The attribute paths are dot-separated. This does not remove existing tag values. - `tagMappingsLocationPaths` array[]: Possible locations of a map of string to string or to collection of strings that has tags. Added to tags, without replacement. - string[]: Possible locations of a map of string to string or to collection of strings that has tags. Added to tags, without replacement. - `tagMappingsLocationPreserveSource` boolean: When true, preserves the source tag map location attributes after remapping instead of removing them. Defaults to false (source is removed after remapping). - `tagMappingsLocations` string[]: Possible locations of a map of string to string or to collection of strings that has tags. Added to tags, without replacement. - `tagMappingsPreserveSource` boolean: When true, preserves the source tag mappings attributes after remapping instead of removing them. Defaults to false (source is removed). - `timestampPreserveSource` boolean: When true, preserves the source timestamp attribute after remapping instead of removing it. Defaults to true (source is preserved). - `timestampReservedAttributePaths` array[]: Paths to search for timestamp values - string[]: Paths to search for timestamp values - `timestampReservedAttributes` string[]: Attributes to search for timestamp values - `traceIdPreserveSource` boolean: When true, preserves the source traceId attribute after remapping instead of removing it. Defaults to true (source is preserved). - `traceIdReservedAttributePaths` array[]: Paths to search for traceId values - string[]: Paths to search for traceId values - `traceIdReservedAttributes` string[]: Attributes to search for traceId values - `type` 'log-attributes-remapper', required: Maps attributes to top-level fields and tags. - object - `attributeMergeStrategyEntries` AttributesMergeStrategyEntry[]: List of attribute paths and their corresponding merge strategies. If no strategy is defined for a path, the terminal merge strategy will be used. - `attributePath` string[], required: The attribute path, represented as an array of strings (e.g., ["user", "id"]). - `strategy` union, required: Default merge strategy to use for merging leaf-valued attributes. This is used when no specific merge strategy is defined for an attribute path. - … - `attributeTokenization` 'KEYS_ONLY' | 'KEYS_AND_VALUES': Use attribute shape only, or shape and whole-value similarity tokens. - `dedupThreshold` integer: Number of duplicates we send for a pattern before we start deduplication. - `defaultAttributesMergeStrategy` union: Default merge strategy to use for merging leaf-valued attributes. This is used when no specific merge strategy is defined for an attribute path. - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - `delimiters` string[], required: Characters to use for splitting tokens. Default is missing '@' to avoid splitting email addresses. Also missing '.' to avoid splitting identifiers using it. - `emitNewPatternSignals` boolean: Whether this operation may emit LogPatternSignal events to the agent-service when new patterns are detected or replayed. Must be enabled before an agent can subscribe to this pipeline. - `enabledMasks` string[], required: Masks that are enabled for tokenizing messages. These are specified as the names of the masks to use. The default enabled masks are: ipport, timestamp, uuid, number. - `excludedAttributePrefixes` array[]: Excluded attribute prefixes as arrays of literal key segments. Does not affect partition-by values or output attributes. - string[]: Excluded attribute prefixes as arrays of literal key segments. Does not affect partition-by values or output attributes. - `includedAttributePrefixes` array[]: Included attribute prefixes as arrays of literal key segments. Empty selects all. Array elements are traversed without indices. - string[]: Included attribute prefixes as arrays of literal key segments. Empty selects all. Array elements are traversed without indices. - `integrationExceptionConfigs` IntegrationExceptionConfig[]: Exception configs that define which exceptions to use from the ones imported from the observability vendor integrations. - `autoSync` boolean: When enabled, the exceptions list will be updated automatically based on the integration's exceptions. - `autosyncPassthroughRateThreshold` number, double: Maximum impact percentage threshold for autosync (0.0 to 100.0). When set, only exceptions with impact percentage <= threshold are auto-included. - `datasetId` string: Dataset against which the auto-sync impact threshold is evaluated. Required for threshold filtering to apply; when null, no exceptions bypass via auto-sync. - `ids` string[] - `integrationId` string, required: The ID of the integration associated with this exception configuration. - `logReducerExceptions` EventPredicate[]: Predicate for skipping events. Logs matching these predicates won't be aggregated. - union: Predicate applied to rows before analytical aggregation. - … - `masks` array[], required: Set of masks that can be applied to messages before tokenizing them. These masks are specified as pairs of (name, regex). The default masks are: ipport, timestamp, uuid, number, awsarn, awstoken. - string[]: Set of masks that can be applied to messages before tokenizing them. These masks are specified as pairs of (name, regex). The default masks are: ipport, timestamp, uuid, number, awsarn, awstoken. - `name` string, required - `partitionByAttributePaths` array[]: Attribute paths as arrays of keys. Use for unambiguous nested vs flat attributes. Example: [["user", "id"]] for nested user.id, [["user.id"]] for flat attribute named 'user.id' - string[]: Attribute paths as arrays of keys. Use for unambiguous nested vs flat attributes. Example: [["user", "id"]] for nested user.id, [["user.id"]] for flat attribute named 'user.id' - `partitionByAttributes` string[] - `partitionByTags` string[] - `reductionTimeWindow` string, ISO-8601: Time window in which logs are considered for deduplication. Should not be modified without explicit guidance. - `repeatCountInSummaryMessageEnabled` boolean: Whether to include the repeat count prefix in the summary message. Grepr will add a prefix to the summary message with the number messages aggregated. However, this might break downstream parsing if the message is expected to be in a specific format. If you want to include the repeat count, set this to true. - `samplingConfig` union: Determines how often samples should be emitted - object - … - object - … - `similarityThreshold` number, double: Controls how close a set of tokens are to a pattern before it is considered a match. Calculated as the percent of matching tokens between a pattern and a message. - `tokenizationInput` 'MESSAGE_ONLY' | 'ATTRIBUTES_ONLY' | 'MESSAGE_AND_ATTRIBUTES' | 'AUTO': Selects message text, attributes, both, or message text with an attributes fallback. - `type` 'log-reducer', required: Reduces logs by combining messages with similar patterns. - object - `name` string, required - `transforms` EventAction[] - union: Base class of all Event Actions - … - `type` 'log-transform', required: Applies transformations to log events. - object - `attributeMasks` AttributeMask[], required: Masks to apply to attribute values. Each entry names an attribute path (as a list of segments) and the masks to apply to the value at that path. Only string values at the exact path are masked; missing or non-string attributes are skipped. Paths must be unique. The same matching rules as messageMasks apply. - `masks` object, required: Masks to apply to the value at this path, as a map of label to mask. The same matching rules as messageMasks apply. - `path` string[], required: Attribute path to mask, as a list of segments. - `messageMasks` object, required: Masks to apply to the log message, as a map of label to mask. A mask may be written as a bare regex string, in which case each match is replaced with the label in square brackets — a match of the regex labeled "email" becomes "[email]". Labels may be up to 64 characters. Overlapping matches resolve to the leftmost match first, then the longest; declaration order breaks remaining ties, so declare more specific patterns first. - `name` string, required - `predicate` union: Predicate applied to rows before analytical aggregation. - object - … - object - … - object - … - object - … - object - … - `type` 'masking-operator', required: Masks sensitive substrings in log events by replacing regex matches with decorated labels. - object - `attributeTokenization` 'KEYS_ONLY' | 'KEYS_AND_VALUES': Use attribute shape only, or shape and whole-value similarity tokens. - `delimiters` string[], required: Characters to use for splitting tokens. - `emitNewPatternSignals` boolean: Whether this operation may emit LogPatternSignal events to the agent-service when new patterns are detected or replayed. Must be enabled before an agent can subscribe to this pipeline. - `enabledMasks` string[], required: Masks that are enabled for tokenizing messages. - `excludedAttributePrefixes` array[]: Excluded attribute prefixes as arrays of literal key segments. Does not affect partition-by values or output attributes. - string[]: Excluded attribute prefixes as arrays of literal key segments. Does not affect partition-by values or output attributes. - `includedAttributePrefixes` array[]: Included attribute prefixes as arrays of literal key segments. Empty selects all. Array elements are traversed without indices. - string[]: Included attribute prefixes as arrays of literal key segments. Empty selects all. Array elements are traversed without indices. - `masks` array[], required: Set of masks that can be applied to messages before tokenizing them. These masks are specified as pairs of (name, regex). - string[]: Set of masks that can be applied to messages before tokenizing them. These masks are specified as pairs of (name, regex). - `name` string, required - `partitionByAttributePaths` array[]: Attribute paths as arrays of keys. Use for unambiguous nested vs flat attributes. Example: [["user", "id"]] for nested user.id, [["user.id"]] for flat attribute named 'user.id' - string[] - `partitionByAttributes` string[] - `partitionByTags` string[] - `similarityThreshold` number, double: Controls how close a set of tokens are to a pattern before it is considered a match. Calculated as the percent of matching tokens between a pattern and a message. - `tokenizationInput` 'MESSAGE_ONLY' | 'ATTRIBUTES_ONLY' | 'MESSAGE_AND_ATTRIBUTES' | 'AUTO': Selects message text, attributes, both, or message text with an attributes fallback. - `type` 'pattern-matcher', required: Matches log patterns using an iterative feedback loop. Events with known patterns skip the global reducer for improved efficiency. - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'logs-iceberg-table-sink', required: Writes logs to an Iceberg Table. - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'pattern-lookup-iceberg-table-sink', required: Writes event id to pattern id mappings to an Iceberg Table. - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'event-dedup-iceberg-table-sink', required: Writes event IDs to an Iceberg table to keep track of which events have been sent to a particular sink. - `vendorSinkId` string, required - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'vendorlog-event-dedup-iceberg-table-sink', required: Writes confirmed-delivered event IDs to an Iceberg table to keep track of which events have actually been sent to a particular sink. - `vendorSinkId` string, required - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'spans-dedup-iceberg-table-sink', required: Writes span identity tuples to an Iceberg table to keep track of which spans have been sent to a particular vendor sink. - `vendorSinkId` string, required - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'spans-iceberg-table-sink', required: Writes individual spans to the raw spans Iceberg table. - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'logs-iceberg-table-source', required: Reads events from an Iceberg table for processing. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'agent-sessions-iceberg-table-source', required: Selects grouped agent session summaries from the logs Iceberg table. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'agent-session-events-iceberg-table-source', required: Reads every event of the agent sessions whose events match the query. - `variables` object: Variables that can be used to modify the query while parsed - object - `aggSpec` MetricAggSpec, required: The metrics aggregation to apply to the source data. - `groupByTags` string[]: The tags to group the results by. If unspecified, all series are grouped together. Only applicable when spaceAggregation is set. - `spaceAggregation` 'LATEST' | 'SUM' | 'AVG' | 'MAX': The space aggregation to use for the query. If null, no grouping is done and the timeseries are returned with full tags. - `timeAggregation` 'LATEST' | 'SUM' | 'AVG' | 'MAX': The time aggregation to use for the query. If this is set, timeGranularity must also be provided. If null, no time aggregation is done and the raw datapoints are returned. - `timeGranularity` string, ISO-8601: The time granularity to use for the query. Only applicable when timeAggregation is set. - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'metrics-iceberg-table-source', required: Reads metrics out of an Iceberg Table for processing. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `hasError` boolean: Filter by error status - `isRootSpan` boolean: Filter by parent span ID (null for root spans only) - `limit` integer: The maximum number of rows to process - `maxDuration` integer: Filter by maximum duration in nanoseconds - `minDuration` integer: Filter by minimum duration in nanoseconds - `name` string, required - `offset` integer: Offset on the number of rows to process - `operationNames` string[]: Filter by operation names (exact match) - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `serviceNames` string[]: Filter by service names (exact match) - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `traceIds` string[]: Filter by trace IDs (32-character hex format) - `traceSignatureIds` string[]: Filter by trace signature ids (exact match) - `traceSignatures` string[]: Filter by trace signature text (supports wildcards with *) - `type` 'traces-iceberg-table-source', required: Reads traces out of an Iceberg Table for processing. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'spans-backfill-iceberg-table-source', required: Reads spans from an Iceberg table for backfill, skipping spans already sent to the given vendor sinks (idempotent backfill source). - `variables` object: Variables that can be used to modify the query while parsed - `vendorSinkIntegrationIds` string[] - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'metrics-iceberg-table-sink', required: Writes metrics to an Iceberg table. - object - `datasetId` string, required: The id of the dataset to write to - `name` string, required - `type` 'metricdata-raw-iceberg-table-sink', required: Writes MetricData (one row per datapoint, attributes auto-flattened to columns) to an Iceberg table. - object - `filterQuery` string: The Datadog query filter to use when reading logs from Datadog. This is concatenated with the one from the integration. - `integrationId` string, required: The Datadog integration id. - `name` string, required - `type` 'datadog-log-cloud-source', required: Pulls log events from the Datadog cloud service for processing using a specified query. - object - `hostEnrichmentEnabled` boolean - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'datadog-log-agent-source', required: Reads log events from Grepr's Datadog collection agent. This is used to pre-process logs before being sent to Datadog. - object - `discardStats` boolean: Whether stats should be discarded instead of being forwarded to Datadog - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'datadog-trace-agent-source', required: Reads dd_trace protos from Datadog Agent. This is used to process traces from Datadog. - object - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `type` string, required - object - `additionalTags` object: Additional tags to add to the metrics while loading. - `metrics` string[]: The list of metrics to load. Will load all possible metrics if nothing is specified - `name` string, required - `type` 'grepr-metrics-source', required: Reads metric events from Grepr for processing. - object - `additionalTags` string[]: Additional tags to add to the logs when writing to Datadog. Gets combined with the integration's additionalTags. - `browserLogGroupByAttributes` string[]: Group by attributes to use when writing logs to Datadog browser intake.Defaults to ["http.useragent", "network.client.ip"]. - `hostFieldNames` string[]: Top-level JSON field names to write the host to. Datadog only maps one of these onto the log's host if the name is in the destination organization's "Preprocessing for JSON logs -> Host attributes" list, so an organization that has removed the default entry needs this overridden. Defaults to ["hostname"]. - `integrationId` string, required: The Datadog integration id. - `name` string, required - `type` 'datadog-log-sink', required: Writes processed events to Datadog. - object: Converts CompleteSpan objects to Datadog's native trace format and sends them to the Datadog platform. - `additionalAttributes` object: Additional attributes to add to traces when writing to Datadog. Gets combined with the integration's additionalAttributes. - `integrationId` string, required: Integration ID for Datadog connection - `name` string, required - `type` 'datadog-trace-sink', required: Writes processed traces to Datadog in native format. - object - `additionalAttributes` object: Additional attributes to add to the logs when writing to New Relic. Gets combined with the integration's additionalAttributes. - `integrationId` string, required: The New Relic integration id. - `name` string, required - `type` 'newrelic-log-sink', required: Writes logs to NewRelic. - object - `additionalTags` string[]: Additional tags to add to the logs when writing to Splunk. Gets combined with the integration's additionalTags. - `defaultIndex` string: Default index to be set when writing to Splunk if no index is set. - `indexOverride` string: Override index to be set when writing to Splunk - `integrationId` string, required: The Splunk integration id. - `name` string, required - `type` 'splunk-log-sink', required: Writes processed events to splunk. - object - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'sumologic-log-agent-source', required: Reads log events from SumoLogic HTTP Source collector. This is used to pre-process logs before being sent to SumoLogic. - object - `additionalAttributes` object: Additional attributes to add to the logs when writing to SumoLogic. Gets combined with the integration's additionalAttributes. - `integrationId` string, required: The New Relic integration id. - `name` string, required - `type` 'sumologic-log-sink', required: Writes processed logs to Sumo Logic. - object - `additionalAttributes` object: Additional attributes to add to the logs when writing to OTLP destination. Gets combined with the integration's additionalAttributes. - `integrationId` string, required: The OTLP integration id. - `name` string, required - `type` 'otlp-log-sink', required: Writes processed logs to any OTLP-compatible destination. - object - `additionalAttributes` object: Additional attributes to add to the logs when writing to OTLP destination. Gets combined with the integration's additionalAttributes. - `integrationId` string, required: The OTLP integration id. - `name` string, required - `type` 'otlp-trace-sink', required: Writes processed logs to any OTLP-compatible destination. - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'logs-iceberg-replay-source', required: Replays log events from an Iceberg table at their original ingestion rate using receivedTimestamp for timing. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'logs-backfill-iceberg-table-source', required: Backfills events out of an Iceberg Table, returning events with the sinks they have already been sent to. - `variables` object: Variables that can be used to modify the query while parsed - `vendorSinkIntegrationIds` string[] - object - `conditions` object, required: Map of condition Id to condition objects. The ids are a unique name for the conditions. - `name` string, required - `triggers` object, required: Map of trigger Id to trigger objects. The ids are a unique name for the trigger. - `type` 'rule-engine', required: Describes rules to execute on log events. See the guide for the Rule Engine for more details. - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `processedLogsTableName` string: The name of the table containing processed logs. - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `reductionInterval` string, ISO-8601, required: The interval used by the Log Reducer for the query. Uses ISO 8601 format for intervals. - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'reducer-logs-iceberg-table-source', required: Queries raw and processed logs from a reducer pipeline. - `variables` object: Variables that can be used to modify the query while parsed - object - `name` string, required - `type` 'logs-sync-sink', required: Sink for returning log results to the caller. - object - `name` string, required - `type` 'agent-sessions-sync-sink', required: Sink for returning agent session summaries to the caller. - object - `name` string, required - `type` 'metrics-sync-sink', required: Sink for returning metric results to the caller. - object - `name` string, required - `type` 'variant-sync-sink', required: Sink for returning arbitrary JSON results to the caller. - object - `name` string, required - `type` 'spans-sync-sink', required: Sink for returning results to the caller. - object - `columns` string[]: Ordered CSV column names; required only for CSV artifacts - `format` 'COMPRESSED_JSON' | 'JSON_LINES' | 'CSV', required: Artifact serialization format - `integrationId` string, required: The integration to use for writing to the file - `name` string, required - `path` string: Destination path relative to the selected warehouse root - `type` 'query-sink', required: Write a bounded artifact to a warehouse provided by the given integration id. - object: Emits a generic pipeline signal for each input log event. - `context` string: Optional author-provided context included with every emitted signal. - `name` string, required - `targetAgentIds` string[], required: Agent ids that should investigate every signal emitted by this sink. - `type` 'agent-signal-sink', required: Emits generic pipeline signals to selected agents. - object: Sink that discards all of its input. - `name` string, required - `type` 'discarding-sink', required: Sink that discards all of its input. - object - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'grepr-pipe-log-source', required: Reads log events that other pipelines write into a Grepr pipe. - object: Writes log events into a Grepr pipe integration. - `integrationId` string, required: Destination Grepr pipe integration in the pipeline's organization. - `name` string, required - `type` 'grepr-pipe-log-sink', required - object - `additionalTags` string[]: Additional tags to add to the logs when writing to Datadog. Gets combined with the integration's additionalTags. - `integrationId` string, required: The Datadog integration id. - `name` string, required - `type` 'datadog-metrics-sink', required: Writes processed metrics to Datadog. - object - `additionalTags` string[]: Additional tags to add to the metrics when writing to Datadog. Gets combined with the integration's additionalTags. - `integrationId` string, required: The Datadog integration id. - `name` string, required - `type` 'legacy-datadog-metrics-sink', required: Writes processed metrics (flat MetricDatapoint model) to Datadog. Transitional; use datadog-metrics-sink for new pipelines. - object - `name` string, required - `query` AnalyticsQuery, required: Complete analytical query to execute. - `end` string, date-time, required: End of the time interval to query. - `from` string, required: Logical table alias to query. - `groupBy` string[]: Grepr SQL expressions or projection aliases used for grouping. An expression may reference projection aliases. - `having` string: Optional Grepr SQL expression applied after aggregation. The expression may reference projection aliases. - `limit` integer: Maximum number of result rows. - `orderBy` AnalyticsOrderBy[]: Ordering applied to the analytical results. - … - `predicate` union, required: Predicate applied to rows before analytical aggregation. - … - `select` AnalyticsProjection[], required: Expressions returned by the query. - … - `start` string, date-time, required: Start of the time interval to query. - `variables` object: Variables available while parsing the predicate. - `window` AnalyticsTimeWindow: Optional time window that buckets the results. Adds leading 'window_start' and 'window_end' result columns, which are always grouped. - … - `type` string, required - `scanBudgetBytes` integer: Maximum number of bytes this query may scan. - `tables` AnalyticsTableBinding[], required: Dataset tables available to the query under logical aliases. - `alias` string, required: Logical table name used by the analytical query. - `datasetId` string, required: ID of the dataset that owns the table. - `table` 'logs_raw' | 'logs_dedup' | 'logs_pattern_lookup' | 'metrics_raw' | 'metricdata_raw' | 'spans_raw' | 'spans_dedup', required: Dataset-owned table exposed under the logical alias. - `type` 'athena-analytics-source', required: Executes an analytical query against dataset tables using Athena. - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `processedLogsTableName` string: The name of the table containing processed logs. - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `reductionInterval` string, ISO-8601, required: The interval used by the Log Reducer for the query. Uses ISO 8601 format for intervals. - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'grepr-reducer-log-source', required: Reads events using Grepr's internal query system out of an Iceberg Table for processing. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'athena-agent-sessions-source', required: Returns grouped agent session summaries through Athena. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'grepr-raw-log-source', required: Reads raw log events using Grepr's internal query system from an Iceberg Table without pattern enrichment. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'grepr-raw-span-source', required: Reads spans using Grepr's internal query system out of an Iceberg Table for processing. - `variables` object: Variables that can be used to modify the query while parsed - object - `name` string, required - `query` AnalyticsQuery, required: Complete analytical query to execute. - `end` string, date-time, required: End of the time interval to query. - `from` string, required: Logical table alias to query. - `groupBy` string[]: Grepr SQL expressions or projection aliases used for grouping. An expression may reference projection aliases. - `having` string: Optional Grepr SQL expression applied after aggregation. The expression may reference projection aliases. - `limit` integer: Maximum number of result rows. - `orderBy` AnalyticsOrderBy[]: Ordering applied to the analytical results. - … - `predicate` union, required: Predicate applied to rows before analytical aggregation. - … - `select` AnalyticsProjection[], required: Expressions returned by the query. - … - `start` string, date-time, required: Start of the time interval to query. - `variables` object: Variables available while parsing the predicate. - `window` AnalyticsTimeWindow: Optional time window that buckets the results. Adds leading 'window_start' and 'window_end' result columns, which are always grouped. - … - `type` string, required - `scanBudgetBytes` integer: Maximum number of bytes this query may process. - `tables` AnalyticsTableBinding[], required: Dataset tables available to the query under logical aliases. - `alias` string, required: Logical table name used by the analytical query. - `datasetId` string, required: ID of the dataset that owns the table. - `table` 'logs_raw' | 'logs_dedup' | 'logs_pattern_lookup' | 'metrics_raw' | 'metricdata_raw' | 'spans_raw' | 'spans_dedup', required: Dataset-owned table exposed under the logical alias. - `type` 'trino-analytics-source', required: Executes an analytical query using a Trino query engine. - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `processedLogsTableName` string: The name of the table containing processed logs. - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `reductionInterval` string, ISO-8601, required: The interval used by the Log Reducer for the query. Uses ISO 8601 format for intervals. - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'trino-reducer-log-source', required: Reads events using Grepr's internal query system out of an Iceberg Table for processing, executed against a Trino query engine. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'trino-raw-log-source', required: Reads raw log events out of an Iceberg Table without pattern enrichment, executed against a Trino query engine. - `variables` object: Variables that can be used to modify the query while parsed - object - `datasetId` string, required: The ID of the dataset to read data from. - `end` string, date-time, required: End of time interval to query data for. - `limit` integer: The maximum number of rows to process - `name` string, required - `offset` integer: Offset on the number of rows to process - `query` union, required: A query predicate that will be used to filter rows - object - … - object - … - object - … - object - … - object - … - `sortOrder` 'ASCENDING' | 'DESCENDING' | 'UNSORTED': The order in which the rows should be sorted by - `start` string, date-time, required: Start of time interval to query data for. - `type` 'trino-raw-span-source', required: Reads spans using Grepr's internal query system out of an Iceberg Table for processing, executed against a Trino query engine. - `variables` object: Variables that can be used to modify the query while parsed - object - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'splunk-log-agent-source', required: Reads log events from Splunk HEC collector. This is used to pre-process logs before being sent to Splunk. - object - `filterQuery` string: The Splunk query filter to use when reading logs from Splunk. This is concatenated with the one from the integration. - `integrationId` string, required: The Splunk integration id. - `name` string, required - `type` 'splunk-log-http-source', required: Pulls log events from Splunk server for processing using a specified query. - object - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'newrelic-log-agent-source', required: Reads log events from Grepr's New Relic collection agent. This is used to pre-process logs before being sent to New Relic. - object - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'otlp-log-agent-source', required: Reads log events from Open Telemetry collector. This is used to pre-process logs before being sent to an Otel sink. - object - `integrationId` string, required: The integration id for the observability vendor. - `name` string, required - `offsetResetStrategy` 'EARLIEST' | 'LATEST' | 'NONE': Strategy for resetting Kafka consumer offset when no committed offset exists or has expired. EARLIEST reads from oldest available offset (prevents data loss on restart after retention period). LATEST skips to newest offset (current behavior). - `type` 'otlp-trace-agent-source', required: Reads trace events from an OpenTelemetry collector. This is used to process traces from OTLP trace ingestion. - object - `name` string, required - `sampleDurationMillis` integer: How long (in milliseconds) a sampled trace ID remains marked for as sampled without new messages before being declared expired. After this duration with no more events for trace, subsequent events for the same trace will be re-evaluated for sampling if seen again. - `sampleRate` number, double, required: The probability (0.0 to 1.0) of sampling a newly seen trace ID. - `selector` union: Predicate applied to rows before analytical aggregation. - object - … - object - … - object - … - object - … - object - … - `traceIdAttributePaths` array[]: List of attribute paths (each path is a list of segments) to extract trace IDs from. The first non-null value found will be used as the trace ID. - string[]: List of attribute paths (each path is a list of segments) to extract trace IDs from. The first non-null value found will be used as the trace ID. - `traceIdTagKeys` string[]: List of tag keys to extract trace IDs from. The first non-null value found will be used as the trace ID (checked after attribute paths). - `traceSamplerTagKey` string: The key for the tag added to events of sampled traces to signal downstream operations differentiated processing. - `traceSamplerTagValue` string: The value for the tag added to events of sampled traces to signal downstream operations differentiated processing. - `type` 'trace-sampler', required: Samples traces based on their ID and marks associated events that are part of the selected traces. This can be used to bypass downstream processing such as message reduction, ensuring complete unmodified logs for those selected traces. This operation can add some overhead since it might require an additional shuffle in the pipeline. Messages that have trace IDs will be output on the "traced" output, while messages that do not have any trace IDs be output on the "untraced" output. - object: Overlays a fixed set of attributes onto every span passing through this operation. Used by the trace draft pipeline to stamp stage tags on previewed spans. - `attributes` object, required: Attributes to overlay onto each span's attributes map. - `name` string, required - `type` 'span-tagger', required: Overlays a fixed set of attributes onto every span passing through. - object - `fileReadingStrategy` FileReadingStrategy: The file reading strategy will determine how files are enumerated. Timestamp reading strategy will only read files created after the specified timestamp. Latest reading strategy will only read files created after the job start time. - `type` string, required - `format` FileFormat, required: Describes the file format to read from. - `mapper` LogEventMapper: Selects the mapper used to decode raw file records into LogEvents. - … - `type` string, required - `name` string, required - `pathPrefix` string: The s3 path under the provided bucket to monitor. Optional.In the case of CloudTrail logs, this is the path to the AWSLogs/ directory. - `s3IntegrationId` string, required: Integration ID for the S3 bucket where Grepr will be reading data. This corresponds to the 's3-data-warehouse' integration type. - `streamingPollInterval` string, ISO-8601: The interval at which Grepr will check for new files in the S3 bucket. This is only applicable in streaming mode. - `type` 's3-file-source', required: Reads generic data from an S3 path. Each record will need to be mapped to the Grepr data model before being usable. - object - `accountIds` string[], required: The list of account IDs from which to read logs. - `fileReadingStrategy` FileReadingStrategy: The file reading strategy will determine how files are enumerated. Timestamp reading strategy will only read files created after the specified timestamp. Latest reading strategy will only read files created after the job start time. - `type` string, required - `format` JsonFileFormat: JSON file format. - `charset` string: Character set name such as UTF-8, UTF-16, etc. - `mapper` JsonLogEventMapper: JSON-to-LogEvent mapper. - `recordsPath` string[]: Path to the list of log records. - `streamFormat` StreamFormatAny - … - `name` string, required - `pathPrefix` string: The s3 path under the provided bucket to monitor. Optional.In the case of CloudTrail logs, this is the path to the AWSLogs/ directory. - `regions` string[]: The list of regions from which to read logs from. If empty or null, reads logs from all regions. - `s3IntegrationId` string, required: Integration ID for the S3 bucket where Grepr will be reading data. This corresponds to the 's3-data-warehouse' integration type. - `streamingPollInterval` string, ISO-8601: The interval at which Grepr will check for new files in the S3 bucket. This is only applicable in streaming mode. - `type` 'cloudtrail-file-source', required: Reads logs from an S3 bucket at a CloudTrail path. Only reads logs in folders at or after the specified start time. The path prefix is optional. Each record will need to be mapped to the Grepr data model before being usable. - object - `format` FileFormat, required: Describes the file format to read from. - `mapper` LogEventMapper: Selects the mapper used to decode raw file records into LogEvents. - … - `type` string, required - `name` string, required - `type` 'grepr-uploaded-log-file-source', required: Reads log files uploaded to Grepr using environment credentials. Intended for file uploads in draft mode. - `uploadId` string, required: Unique identifier for the upload. The full S3 path will be constructed as uploaded_files/{organizationId}/{uploadId} to ensure customers can only access their own uploads. Use the uploadId returned by the file upload endpoint. - object - `draftMode` boolean: Set as a variable in the script scope, this input tells the script to run in 'draft' mode, which should enable additional outputs from various parts of the built pipeline, according to the Template's defined draft outputs. - `name` string, required - `templateId` string, required: The ID of the template to use for this operation. - `templateInputs` object: Inputs to the template. Each entry in the map is added to the scope of the template javascript as a variable and can be accessed directly. In addition to the inputs, a 'user' object is also bound providing information about the current user. - `templateVersion` integer: The template's version. Required for a template your organization owns. Built-in templates ignore it and run the version that ships with the pipeline's Grepr version. - `type` 'template-operation', required: Runs a template script to generate a subgraph running in place of the operation in the parent graph. - object: Processes data through a sequence of SQL statements executed in order. Supports VIEW statements that create temporary tables, OUTPUT statements that produce DataStream results, and IO statements that perform side effects. - `availableDatasets` string[]: IDs of datasets whose tables can be referenced in SQL statements. These will be registered as queryable tables in the SQL environment. The tables from each dataset will appear in a separate namespace based on the dataset name. - `globalStateTtl` string, ISO-8601: Global time-to-live for operation state. When calculating aggregations or joins in streaming mode, Grepr pipelines will keep track of some state (such as the count and sum of values for an averaging aggregation). To prevent the size of the state from growing indefinitely, Grepr will set a default time-to-live value for any state. If state does not receive updates after this TTL, it will be considered idle and the state will be cleared. Aggregations for will be reset for the idle group for example. - `inputs` object: Input table names that can be referenced in SQL statements. Each input table corresponds to an input stream that is connected to this operation in the job graph. All connected input types must be compatible with the SQL operations. Each type specifies what the schema of the resulting table that is available for the SQL statements looks like. For `COMPLETE_SPAN`, the table schema uses SQL row field names, which differ from the REST JSON property names in the `CompleteSpan` schema. - `name` string, required - `statements` SqlExecutable[], required: List of SQL statements to execute in order. Each statement can reference tables created by previous statements, the input tables, and any tables from the available datasets. - union: Base class for SQL statements - … - `type` 'sql-operation', required: Processes data through a sequence of SQL statements. - `watermarkDelay` string, ISO-8601: Delay for watermark generation. Default is 5 seconds. Watermarks are used to track event time progress in streaming jobs. The watermark is usually the maximum event time seen by a processing task minus this delay. Watermarks are used to trigger time-based operations like windowing. This delay defines how far behind the system can be in processing events before it considers the event time to be 'late'. Late events would not be included in windowing operations that might have already been closed (ie where the data is already sent). This also means that the results of windowing operations will be delayed by this delay, as the system waits for the watermark to catch up. For more information, see the Flink documentation on watermarks. - object: Reduces trace volume by applying a uniform trace head sampling rate and configurable span head sampling rules with optional tail sampling, then emits the retained spans. - `actions` TraceAction[]: Actions to execute on traces. Actions process spans after sampling decisions have been made and can trigger operations like logs backfill. - `actionId` string, required: Unique identifier for this action within the TraceReducer. Must be alphanumeric with hyphens and underscores only. - `type` string, required - `criticalPathAnalysis` CriticalPathAnalysisConfig: Opt-in critical-path analysis settings. Computed during trace assembly; requires tail sampling. The maxSpans and sampleRate levers bound the per-trace cost. - `maxSpans` integer: Cap on how many spans are analyzed per trace; the critical chain is kept first. Null means no cap beyond the global per-trace assembly limit. - `sampleRate` number, double: Fraction of assembled traces to analyze, in [0.0, 1.0], selected deterministically by trace id. Defaults to 1.0 (analyze every trace). - `datasetId` string: Dataset ID used for naming the Iceberg table (traces_raw_{datasetId}). If not provided, traces will not be persisted to Iceberg. - `name` string, required - `siblingMergeRules` SiblingMergeRule[]: Rules that merge same-signature sibling spans matching a predicate, after tail sampling decisions. The earliest-start match is kept with its children; the other matching same-signature siblings are removed and counted under grepr.merged_siblings. Requires tail sampling to be configured. - `predicate` string, required: Boolean SQL expression evaluated per span over the CompleteSpan schema. Uses the same fields and functions as head sampling predicates: top-level columns such as servicename, nested fields such as span.operationName, and VARIANT_VALUE(...) for span/resource attributes. - `spanDropRules` SpanDropRule[]: Rules that drop matching spans and their descendants after tail sampling decisions. Requires tail sampling to be configured. - `predicate` string, required: Boolean SQL expression evaluated per span over the CompleteSpan schema. Uses the same fields and functions as head sampling predicates: top-level columns such as servicename, nested fields such as span.operationName, and VARIANT_VALUE(...) for span/resource attributes. - `spanHeadSamplingRules` SpanHeadSamplingRule[]: Span head sampling rules. Matched spans are retained as SPAN_HEAD_SAMPLED. If tail sampling is configured, matched spans participate in trace assembly before output; otherwise, matched spans are emitted directly. Order is significant; first match wins for metric attribution. Trace head sampling wins when both match the same span. - `filter` HeadSamplingFilter: Predicate that determines whether a span matches a head sampling rule. - … - `rate` number, double, required: Fraction of matching spans/traces to retain, in [0.0, 1.0]. - `tailSampling` TailSamplingConfig: Optional configuration for histogram-based tail sampling. When provided, enables intelligent sampling based on trace duration percentiles calculated over sliding windows. Uses hierarchical trace signatures to group similar traces and applies configurable sampling rates based on performance tiers (fast, medium, slow, outliers). - `bootstrapThreshold` integer: Number of traces to sample during bootstrap period before applying percentile-based sampling rules. This allows the system to gather initial data on trace durations before applying more complex sampling logic. This number will be multiplied by the parallelism of the pipeline, so the bootstrap traces may be higher as the pipeline expands for new trace signatures. - `errorSamplingRate` number, double: Sampling rate for traces that contain errors or failures. Error traces are often the most important for debugging and should typically have high sampling rates. Applied before quantile-based sampling. - `incompleteSamplingRate` number, double: Sampling rate for incomplete/partial traces. Partial traces occur when some spans are missing, which can indicate data loss or system issues. Applied before quantile-based sampling. - `keyAttributePaths` array[]: Array of attribute paths to extract key attributes from spans for hierarchical signature generation. Each inner array represents a sequence of keys to search for in the attributes map. These attributes are used to group traces by their structural patterns. - … - `longRunningTraceSamplingRate` number, double: Sampling rate for traces force-emitted because they exceeded the configured maxSessionLength. Routed through a dedicated 'long_running' percentile bucket — distinct from the structural 'incomplete' bucket — so operators can monitor and tune separately. - `maxSessionLength` string, ISO-8601: Upper bound on how long a single trace assembly session may accumulate spans in memory before being force-emitted. Protects against long-running traces that would otherwise hold spans in the assembly buffer indefinitely. When the cap is hit, the trace is classified as long-running and later spans form a new session. Must be strictly greater than maxTraceDelay. - `maxTraceDelay` string, ISO-8601: The maximum time to wait for all spans of a trace to arrive before assembling the trace. This determines the size of the tumbling window used for trace assembly. Traces that span longer than this duration may be assembled as partial traces. Should be set based on the expected maximum trace duration in your system. Large values may lead to memory issues if traces are too long and will delay sending traces to the sink by that amount. The pipeline may also fail if configured too high. - `quantileSamplingTiers` QuantileSamplingTier[]: List of quantile sampling tiers for performance-based sampling. Each tier defines a percentile threshold, sampling rate, and label. Traces ABOVE each quantile get the corresponding sampling rate and are labeled with the tier's label. Evaluated in descending quantile order. A tier at quantile 0.0 is a catch-all that matches unconditionally, giving the fastest traces a baseline rate; without one, traces below the lowest configured quantile are dropped. - … - `traceHeadSamplingRate` number, double: Uniform trace head sampling rate in [0.0, 1.0]. When set, all spans sharing a traceId are retained together as HEAD_SAMPLED at this probability. Mutually exclusive with traceHeadSamplingRules (scalar wins when both present). - `traceHeadSamplingRules` TraceHeadSamplingRule[]: Deprecated: use traceHeadSamplingRate instead. Kept for backwards compatibility with stored configurations. When traceHeadSamplingRate is absent, the effective rate is max(rate) over this list (filters are ignored). - `filter` HeadSamplingFilter: Predicate that determines whether a span matches a head sampling rule. - … - `rate` number, double, required: Fraction of matching spans/traces to retain, in [0.0, 1.0]. - `type` 'trace-reducer', required: Reduces trace volume by sampling CompleteSpan instances. Combines a uniform trace head sampling rate (whole-trace retention) and per-span head sampling rules with optional quantile-based tail sampling. - object: Reduces metric volume by aggregating per-object time series into per-cohort summaries while preserving the original signal when anomaly rules fire. Series that don't belong to a monitored object flow through unchanged; series matching a drop rule are discarded. - `anomalies` AnomalyConfig[]: Anomaly rules. - `anomalyDuration` string, ISO-8601, required: START..END lifecycle window after the last triggering datapoint. - `condition` string, required: Scalar SQL condition that fires the rule. - `from` object, required: Alias → aggregation declaration. - `joinOn` string[]: Join expressions linking aliases (Calcite-parsed SQL). - `name` string, required: Unique rule name. - `anomalyActions` AnomalyActions: Actions applied when an object is flagged anomalous. - `objectMetricsPassthrough` ObjectMetricsPassthrough: When an object is anomalous, de-aggregate its raw datapoints alongside the per-cohort aggregate output, under a suffixed metric name. - … - `cohorts` CohortsConfig: Cohort-identity + space-aggregation configuration. - `identifyingAttributes` string[]: Level-prefixed attribute keys that group objects into cohorts. - `metadataAttributes` string[]: Level-prefixed attribute keys describing cohorts. - `numShardBits` integer: Parallelism multiplier for cohort outputs (anomaly rules ignore this). Splits each cohort across 2^n CohortCalc instances by the low n bits of tsIdLo, trading parallelism for partial per-shard aggregation. 0 = one instance per cohort. - `outputs` CohortOutputDecl[]: Ordered cohort-output entries (first-match-wins). - … - `dropRules` DropRule[]: Drop rules — matching series are discarded outright. - `attributeFilters` object: Level-prefixed attribute key → glob pattern. All entries must match. - `metricName` MetricNameFilter: Glob-pattern filter on metric names. - … - `emitInterval` string, ISO-8601, required: How often the reducer flushes cohort-level output. - `hostMetadata` HostMetadataReporting: Keep a vendor's host inventory in sync with the metric reduction. - `aggregateCohortHosts` boolean: Collapse each cohort's real hosts into a single virtual cohort host. - `integrationIds` string[], required: Datadog integration IDs that receive the reported host metadata. - `rehydrateAnomalousHosts` boolean: While an object is anomalous, re-report its real host to Datadog. - `lateArrivalWaitDuration` string, ISO-8601, required: Watermark delay applied to give late-arriving data a chance to be aggregated. - `name` string, required - `objects` ObjectsConfig, required: Object-identity configuration. - `identifyingAttributes` string[], required: Level-prefixed attribute keys that identify an object. Each key must have a 'resource.', 'scope.', or 'series.' prefix. - `metadataAttributes` string[]: Level-prefixed attribute keys that describe an object but may change. - `metricsFilter` FilterPrimitive: Combined metric-name + attribute-value filter. - … - `spaceAggOverrides` object: Per-metric-name overrides for space (cross-object) aggregation. - `timeGranularity` TimeGranularity, required: Time-aggregation window and per-metric agg-fn overrides. - `aggFnOverrides` object: Per-metric-name overrides for time aggregation. - `duration` string, ISO-8601, required: Time-aggregation window; also the slide interval used by every aggregation. - `type` 'metric-reducer', required: Metric reducer. - object: Forwards Datadog stats payloads to the Datadog platform for accurate RED metrics. Stats are pre-computed by the Datadog agent and bypass all processing. - `additionalTags` object: Additional tags to add to stats when writing to Datadog. Gets combined with the integration's and global additionalTags. - `integrationId` string, required: Integration ID for Datadog connection - `name` string, required - `type` 'datadog-stats-sink', required: Writes Datadog stats payloads to Datadog for accurate RED metrics. - object - `loop` boolean: If true, the source will loop over the values indefinitely when processing is STREAMING. In BATCH mode, this flag is ignored. - `name` string, required - `type` 'logs-values-source', required: Generates log events with constant values. - `values` LogEvent[]: List of log events to be generated by the source. - `attributes` object: Attributes associated with the event, a map from attribute name to attribute value, which may be a nested map or list. - `eventTimestamp` integer, required: Event timestamp that is in the record. Milliseconds from epoch. - `id` string, required: Id of the event. - `message` string: The log message. - `receivedTimestamp` integer, required: Timestamp when the event was received by Grepr. Milliseconds from epoch. - `severity` integer: The log severity. Higher numbers mean higher severity, following the OTel format 1-24. - `tags` object: Tags associated with the event, a map from tag name to set of tag values. - object - `globalRules` PatternRuleConfig[]: Global rules applied to all log events regardless of pattern (e.g., log-based alert rules). - `enabled` boolean - `order` integer - `name` string, required - `rules` object: Per-pattern rules keyed by pattern ID (TSID string). - `type` 'log-rules-application', required: Applies per-pattern rules (masking, time-series spike detection) to log events. - object - `capacity` integer: Maximum number of concurrent trigger handler calls. - `name` string, required - `timeoutSeconds` integer: Timeout for trigger handler execution in seconds. - `type` 'trigger-action', required: Dispatches trigger actions to rule-type-specific handlers. - object - `name` string, required - `ruleSets` MultiGrokRuleSet[]: Ordered list of grok rule sets. Each rule set has its own predicate, optional extract attribute, parsing rules, and helper rules. Rule sets extracting from enriched attributes should be ordered after rule sets that populate those attributes. - `extractAttribute` union: Optional attribute to extract from. If null, extracts from the log message. Rules that extract from enriched attributes should be ordered after rules that populate those attributes. A dotted string, or an array of literal keys. - … - `grokHelperRules` string[]: Helper rules scoped to this rule set - `grokParsingRules` string[]: Grok parsing rules for this rule set - `predicate` union: Predicate applied to rows before analytical aggregation. - … - `type` 'multi-grok-parser', required: Processes unstructured logs and extracts fields from them using Grok patterns. - object - `analysisJobId` string, required - `analysisVersion` string, required - `datasetId` string, required - `embeddingCapacity` integer - `embeddingDimensions` integer - `embeddingModel` string, required - `end` string, date-time, required - `labelModel` string, required - `maxRetries` integer - `maximumNamedGroups` integer - `modelIntegrationId` string, required - `name` string, required - `query` union, required: Predicate applied to rows before analytical aggregation. - object - … - object - … - object - … - object - … - object - … - `requestTimeout` string, ISO-8601, required: Duration in ISO-8601 format (e.g., 'PT20.345S' for 20.345 seconds). - `retainedAnalyticsMemoryLimitBytes` integer - `sessionCap` integer - `start` string, date-time, required - `type` 'agent-session-analytics', required - object: Processes native agent-harness OTel logs into canonical agent LogEvents. - `name` string, required - `type` 'agent-log-processor', required: Processes native agent-harness OTel logs. - object: Normalizes agent span attributes and projects observed transcript content as canonical agent LogEvents in one pass. - `name` string, required - `type` 'agent-span-processor', required: Processes native agent-harness OTel spans. - `name` string, required: Name of the job - `organizationId` string, required: Organization ID of the job - `pipelineStatus` PipelineStatus: Current operational status and health of the pipeline - `error` ErrorDetails - `exception` string - `message` string - `health` 'HEALTHY' | 'STABILIZING' | 'UNHEALTHY' | 'UNKNOWN' - `isRolledBack` boolean - `message` string - `processing` 'BATCH' | 'STREAMING', required: Processing method of the job - `startedAt` string, date-time: Time the job was started - `state` 'CREATED' | 'PENDING' | 'INFRA_UPDATE' | 'INFRA_UPDATE_WAIT' | 'VERIFYING' | 'WAITING' | 'RUNNING' | 'STOPPED' | 'FINISHED' | 'FAILED' | 'CANCELLED' | 'DELETED' | 'UPDATING' | 'STARTING' | 'RESTARTING' | 'STOPPING' | 'ROLLING_BACK' | 'DELETING', required: State of the job - `tags` object, required: Custom tags attached to the job for ease of search - `teamIds` string[]: The team IDs that this job is associated with. - `updatedAt` string, date-time, required: Time the job was last updated - `version` integer, required: Version of the job ## Other responses - `400`: Bad Request - `401`: Unauthorized --- [API](https://skmtc.dev/grepr/apis/grepr-api.md) - [All operations](https://skmtc.dev/grepr/apis/grepr-api/llms.txt) - [OpenAPI document](https://skmtc.dev/grepr/apis/grepr-api/revisions/2a5ce03a0bf0?raw)