Marketing run-latency stats for the scope, split queue vs analysis.
Three latencies over the same set of runs, so they decompose one journey:
- queue_* = analysis_started_at - run_time -- created until a worker
actually began. This is dispatch/queue delay, not work.
- analysis_* = analysis_completed_at - analysis_started_at -- file prep
plus analysis, not the analysis alone. The worker stamps
analysis_started_at before it blocks waiting for uploads to finish
processing, so this span can contain minutes of file-pipeline wait. A spike
here does not on its own implicate the analysis or the model.
- end_to_end_* = analysis_completed_at - run_time -- what the person who
submitted the run waited, the two above together.
Three statistics per latency, because one is not enough. P50 and P95
describe the bulk and the tail; queue_max_seconds catches what percentiles
structurally cannot. A handful of runs stalled for hours out of a hundred
normal ones does not reach the 95th percentile, so P95 can sit flat across a
week containing a multi-hour stall while the max moves by orders of
magnitude. Conversely the max is one row and says nothing about how
widespread a delay is -- queued_over_threshold (runs whose queue wait
exceeded queue_threshold_seconds) is the breadth signal. Read them
together; each is blind to a different failure.
queued_over_threshold is a count and must stay one. Every other metric
on the reliability page carries a companion percentage, so the pull to
normalise this one is strong -- but dividing it by runs inverts the ranking
whenever volume moves. On the incident week, breaches rose in absolute terms
while the rate fell below the neighbouring quiet weeks, purely because run
volume roughly doubled. As a rate the worst week reads as the best. Report the
count; put the denominator beside it if context is needed.
Maxima are per-column and so may come from three different runs: unlike the
percentiles they do not decompose, and queue_max + analysis_max is no
particular run's end-to-end. Use the slowest-runs drill-down to see one run's
real split.
runs is the measurement denominator, and it is not the number of runs
that happened. Every figure here is over successful runs carrying both
analysis timestamps in a consistent order. Both timestamps are written in one
update when results land, so a run that failed or is still queued carries
neither -- it contributes nothing, and its wait is not measurable at all.
These are survivor latencies and the true tail is worse than shown. Compare
runs against the same week's WeeklyFailurePoint.total to see how much of
the week was measurable. Every latency field is null when runs is zero (no
measurement), never zero.