- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks whether an eval protocol can discriminate between behaviors corresponding to different values of a target estimand. It gives the definition of identification over some finite behavior class, and returns a collision witness whenever 2 alternatives have different estimands but an identical observed response.
Measurement selection and repair are, thus, reduced to hitting-set problems over the measurements that differentiate pairs. Solver-labeled reasoning problems are used to create experiments with base, target, and sham worlds, multiple possible label mappings, and flexible response readouts.
For a class of 86 model derived signatures, a frozen 15 cell support resolves all derivation collisions and leaves a single held out collision which is resolved by an additional measurement. A separate matched control analysis finds higher sham preservation than target fidelity among base-correct responses. Limitations explicitly bound conclusions to the controlled testbed.
Critical strengths
-
It turns a vague concern into something checkable: The phrase “This eval might not measure what it claims” is hard to act on - but here, when a protocol fails, the audit names the 2 behaviors it confuses + the measurements that'd separate them. Appendix C shows this pretty well: remove any single control condition and the audit points to a specific collision. As a good example: without target edits, a model that never updates looks identical to one that reasons correctly - so that gives designers a concrete reason for each control.
-
The reporting is careful. The paper uses exhaustive enumeration where feasible, reports full random-baseline distributions, checks all 26 minimum designs, admits tie-break sensitivity and validates oracle labels with blinded human reviewers (k around 0.94) - I trust the numbers as a description of the testbed
Critical weaknesses
-
The method is closer to existing work than the paper suggests: picking the fewest attributes that still separate every pair of differently-labeled items is a known problem - in rough set theory these are decision-relative reducts, which are computed as hitting sets over a discernibility matrix whose entries match the paper’s D {h,h’} sets (Pawlak , 1982, Skowron & Rauszer, 1992). Min features selection (Almuallim & Dietterich 1991) and minimum test collection (Garey & Johnson 1979) are fairly close as well. The paper claims novelty because it separates behaviors relative to an estimand (Section 2.3, Table 4) - but these methods already separate relative to labels - so the contribution is mainly an application of known tools, which then puts more weight on showing practical value.
-
The experiments mostly confirm what was built into them: the 7 behaviors were designed so that each control is needed - and the larger synthetic sets were generated so that specific measurements become necessary - so random selection failing is expected and it tells us little about real protocols
The model-derived design only covers patterns already seen. In Appendix H shows it breaks once unseen patterns are added, with ~6.6k failures after 5k additions - and unseen behaviors are exactly the most important ones in this case. Simple greedy selection also comes within 1 pair of the exact solution (Table 2) - so exact synthesis doesn't look necessary
The model finding has a simple alternative explanation: in a sham edit, the correct answer stays the same - so a model that tends to repeat its earlier answer will look good on shams, and bad on targets. The paper doesn't rule this out and this analysis doesn't use the audit framework in any case. The models are also small models from 2024, with base accuracy near 0.5
To be convincing I think the paper needs at least 1 case where the audit finds a significant collision in an existing benchmark and changes how a published result should be read
Questions for authors
-
How does your audit & synthesis procedure differ from decision-relative reducts in rough set theory or from min-features selection? Because both already find the smallest attribute set that separates every pair of differently labeled items, so if there's a property your setting handles that these methods don't - please state it clearly.
-
Can you show 1 case where the audit finds a significant collision in an existing benchmark or published eval, and (most importantly) that that collision changes how a reported result should be read? I think a single convincing example would do much more for the paper’s significance than additional synthetic classes.
-
In a sham edit, the correct answer stays the same - so a model that tends to repeat its base answer will score well on shams and poorly on targets. Among base-correct cases where the target answer is wrong: how often does it simply match the base answer? Does the asymmetry hold up once you account for this?
2 comments
Thanks for the thorough response - the added experiments and narrower claims improve the paper, but my concerns about novelty and practical impact remain, so I am keeping my score
Response to tpAz: benchmark consequence, classical scope, and persistence
Thank you for the careful review. We have added a public-benchmark collision and a frozen new-theory repair validation, while clarifying the classical-method correspondence and the limits of the matched-response analysis.
Locations: ARC witness, Appendix O.3 (p. 27); frozen repair, Tables 19–20 (p. 25); classical methods, Section 2.3/Appendix B; persistence, Table 23 (p. 28).
Beyond hand-designed H7–H50 stress cases, Appendix N evaluates a repair frozen before 96 new original-text ProofWriter theories (192 units). Complete-target recovery improves from 45/192 (original 12-cell pipeline) to 72/192 (repaired 17-cell pipeline); covered errors fall from 37 to 7, with 113 abstentions (Table 19). This compares complete pipelines, not a pure selector effect. Holding the enriched class/lookup fixed, adding five measurements improves recovery from 34/192 to 72/192. At equal 17-cell count, exact, coverage-greedy, conditional entropy and target information gain all recover 72/192, compared with a 32-seed random mean of 50.09/192 (Table 20). This equal-budget replay is secondary and retrospective, and effective heuristics can match exact synthesis.
Class-relative limitations remain: Appendix H shows that unseen alternatives can break a frozen certificate; unknown projections may abstain, but unseen behaviors can also match known projections incorrectly. The added Qwen3-8B panel does not establish broad natural-task or frontier-model transfer.
Decision-relative reducts (W1/Q1). We agree that finite deterministic support selection is mathematically the same separation problem as decision-relative reduct/minimum-feature selection. Section 2.3 and Appendix B now state this correspondence, distinguish minimum-cardinality from inclusion-minimal solutions, and attribute the underlying mathematics to prior work. We do not claim a new hitting-set theorem. Our evaluation contribution is an executable audit-to-repair procedure: specify the estimand and alternatives; expose a cross-estimand collision when measurements are insufficient; identify the distinguishing measurements; and augment existing support. The ARC witness and frozen validation examine the usefulness of this operational workflow.
Answer persistence (W3/Q3). We have quantified repetition rather than attributing asymmetry to a hidden mechanism. Among base-correct target errors, repetitions are 232/266 (balanced Llama3.1) and 89/198 (balanced Qwen2.5), versus 104/120 and 62/102 on the second source. Appendix P decomposes the sham-minus-target contrast on the original base-correct denominator: nonrepeat contributions are -3.72, +5.12, +7.69, and +12.21 points across those four model/source combinations (Table 23); a positive nonrepeat contribution is not universal.
This decomposition is descriptive, not a persistence-adjusted causal effect. Repetition is defined using the target response, so removing repeated errors or conditioning on nonrepetition selects on outcomes. Separate world queries do not eliminate a stable persistence tendency. We therefore restrict the matched result to an observed conditional contrast, separate from the class-relative identification certificates, and make no claim about an identified internal reasoning mechanism.
We hope the concrete ARC witness and independently frozen repair validation address the practical-value concern.