List Drift

Why two runs of the same collateral disagreed about one requirement.

Three kinds, kept apart because they have three different fixes and one "disagreement" count would point at none of them.

What happened when one requirement met one piece of collateral.

Stored rather than derived. "We checked 34 requirements and 3 failed" cannot be reconstructed from the risks alone: a requirement that produced no finding is indistinguishable in document_risks from one that was never evaluated, and that difference is the whole feature.

NOT_DETERMINED is why the model is asked to answer for every rule rather than only to report failures. Silence has two meanings — "the collateral satisfies this" and "I cannot tell from what the collateral shows" — and a panel that reads the first from the absence of a finding claims a pass the model never gave. The judging contract therefore makes each rule land in exactly one of these three, and a rule the model omits altogether lands here too.

What happened when one requirement met one piece of collateral.

Stored rather than derived. "We checked 34 requirements and 3 failed" cannot be reconstructed from the risks alone: a requirement that produced no finding is indistinguishable in document_risks from one that was never evaluated, and that difference is the whole feature.

NOT_DETERMINED is why the model is asked to answer for every rule rather than only to report failures. Silence has two meanings — "the collateral satisfies this" and "I cannot tell from what the collateral shows" — and a panel that reads the first from the absence of a finding claims a pass the model never gave. The judging contract therefore makes each rule land in exactly one of these three, and a rule the model omits altogether lands here too.

Which way one (collateral, requirement) pair flipped in the reviewer's view.

The strict agreement figure counts a pair as agreeing only when both runs judged the same revision of the rule and returned the identical verdict, which is the right denominator for tuning — DriftKind splits the disagreements three ways because they have three different fixes. It is the wrong denominator for the question a rollout decision actually turns on: would the customer have seen a different report? Met against not-conclusive reads as drift and changes nothing on screen; a rule that only reached the judge in one run reads as presence drift and changes everything if that run found it unmet.

So this projects both sides through RISK_RAISING_OUTCOMES — unmet, or not — and names the two directions that survive. Never summed with DriftKind counts: this is the same set of disagreements read a second way, not more of them.

Named A_ONLY / B_ONLY rather than lost/gained on purpose. The two trial runs are the same path run twice in sequence, so neither is the "after"; and the regression half of this dashboard already means something specific by a lost finding — a risk the customer's reviewer flagged that the new path stopped producing. A skew between these two is worth a look on its own (one run reaching more of the library than the other), but neither direction is a verdict.