experiments

Get First Week Holdout

Changed on

Did the first-week ladder move the habit rate? Treated versus held out.

Reads the durable experiment record the state job writes onto each customer, rather than recomputing the arm from the hash. Recomputing would silently re-label everyone the moment the salt changed, and any drift between the two implementations would read as a real effect.

Only closed windows count. Someone three days past signup cannot yet have four active days, so counting them draws a collapse at the right edge that makes the treated arm look worse the faster it grows.

And only windows the programme could actually reach. A cohort whose window closed before the ladder's first delivery received nothing in EITHER arm, which makes it an A/A test: the true difference is known to be zero, and whatever it reports is noise. Pooling those into the headline is not a rounding error. On Parrot the 2026-08-23 cohort -- 233 treated, 52 control, zero pushes between them -- read +6.25pp, and pooling it held the headline up at -2.08pp when the reachable cohorts alone read -4.87pp. The card presented that as a measured lift for four days.

So the floor is the programme's own first delivery, and cohorts below it are still RETURNED, under exposure.excluded_zero_exposure, so the caller can grey them out rather than silently lose them. exposure.coverage_pct reports how much of the treated arm actually received a message, because an intent-to-treat estimate diluted by a third is indistinguishable from a null unless the dilution is on the page.

Expect not_yet for a while. At ~350 signups a day and a 20% holdout, a two point lift needs about forty days and a one point lift is not resolvable at this volume at all. That is a property of the question, not a fault in the answer.

get/api/v1/agents/first-week/holdout

Request

  • Base URL
    https://api.minro.com (the document declares no server; this is the origin it was published from)
  • URL
    https://api.minro.com/api/v1/agents/first-week/holdout
  • Auth
    none declared

Headers

authorizationstring nullable

Bearer <token>

Response

Successful Response

dataobject required

Changes

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      the response's property default value {"request_id":"","timestamp":"2026-09-16T23:27:36.856600"} was removed for the status

    • info

      the response's property default value 2026-09-16T23:27:36.856600 was removed for the status