Research · published in full
Nine datasets where the obvious reading of the rows is the wrong one. Elbi’s verification gates get all nine right. claude-sonnet-5, given a 300-row sample of each and told which check applies, returns a different verdict on 6.
Two of those 6 come with a caveat. On simpsons_ab_reversal the model sent back an empty response, which scores as inconclusive. On multiverse_joint_fragile the right answer, fragile, wasn’t one of the verdicts it was allowed to give. confounding_freight_rate is still a candidate trap, so treat that row as provisional.
The other 3 are struck through. A bare model already gets them right, and they’re queued for removal.
| Trap | Oracleverdict | claude-sonnet-5verdict | Advantage |
|---|---|---|---|
| berkson_collider | unsound | unsound | none |
| confounding_freight_rate provisional | unsound | sound | gate |
| confounding_ice_cream | unsound | sound | gate |
| leakage_churn | unsound | unsound | none |
| multiverse_joint_fragile | fragile | unsound | gate |
| multiverse_noise | inconclusive | sound | gate |
| rtm_selected_cohort | unsound | unsound | none |
| selection_ambiguous_role | inconclusive | unsound | gate |
| simpsons_ab_reversal | unsound | inconclusive | gate |
python -m benchmark.run --refresh-llm.Reproduce it yourself with uv run pytest benchmark/tests