- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces DECOY, which consists of 155 families of four physics prompts that differ only in a single cue. A symbolic solver certifies that the cue cannot change the answer, and each family is labelled by the largest drop in the teacher-forced log-probability of the certified answer. On these families, Qwen2.5-7B-Instruct is sensitive on 50% and Llama-3.1-8B-Instruct on 23%. A linear probe on Qwen’s hidden state predicts this sensitivity before generation, and knocking out a small set of attention heads at layer 16 cancels the effect on a Qwen panel. The contributions are a certified measure of confidence shifts under answer-preserving edits, a pre-generation monitor for that sensitivity, and a causal localisation of the effect in one model.
Critical strengths
The line of the work is clear. The core idea is that sensitivity to irrelevant conditions corresponds to unstable confidence. The related work shows that each dimension is grounded and reflects the authors’ own improvements: every paragraph first states prior practice, then points to a specific gap, and that gap is exactly a step addressed later in the paper. The paper also sets up many experiments to rule out confounds and to probe the problem in depth, and the logic is tight.
Critical weaknesses
One issue in the methodology is that Cue C is not the same as Cue A and Cue B. A and B are irrelevant information, whereas C is more about the model’s unit conversion, and in some cases different conventions may correspond to different units. This is completely different from A and B, and C should not be pooled with A and B in the later analysis. In addition, the 155 DECOY families are mainly short calculations that can be symbolically certified, covering mechanics, circuits, and thermodynamics, but they do not include problems that require formula selection, multi-step derivation, or backtracking, which are more closely tied to confidence. The paper shows that simple formula-evaluation items make it clear whether irrelevant information interferes, but these plug-in-the-numbers questions may also be less susceptible to irrelevant information than real theoretical physics derivations, so using only short calculations is somewhat limited. A further point is that the contribution currently stops at certified measurement. The paper does not show how what it proves and measures can be used back in physics problem solving, for example whether it can improve accuracy, calibration, or robustness during calculation or reasoning.
Questions for authors
Please report sensitivity rates, mean drops, and probe results separately for Cue A, Cue B, and Cue C. Do the headline claims still hold if Cue C is excluded from the aggregate? The current families are short, symbolically certified calculations. Can answer invariance still be certified for items that require formula selection, multi-step derivation, or backtracking? If so, do you have any such results, or is this left to future work? Beyond measurement, does flagging sensitive families or suppressing the localised heads improve accuracy, calibration, or robustness on physics calculations? If this was not tested, which of these uses do the authors see as the immediate next step?
Thank you for your review and for your comments on the benchmark design and the broader direction of the work.
We agree that Cue C should not be treated as the same phenomenon as Cues A and B. Equivalent representations of a relevant physical quantity raise different questions from genuinely irrelevant information, and combining them in the same analysis makes the interpretation less clear. We plan to separate these cases and study them under distinct evaluation protocols.
Your observation about the limited complexity of the current physics problems is also important. The short calculation-based families gave us a controlled starting point, but they do not adequately represent problems involving formula selection, multistep derivations, or more involved physical reasoning. Extending the benchmark in these directions, while retaining reliable verification of the underlying physics, is a major part of our ongoing work.
We also agree that the study should go beyond detecting sensitivity. Whether the identified behavior can be used to improve actual problem-solving accuracy, calibration, or robustness remains an open question, and we intend to investigate this rather than assume that better sensitivity measurements will translate into practical improvements.
We have decided to withdraw this submission and continue the research with these extensions and corrections in mind.
Thank you again for the encouraging and constructive feedback. Your suggestions are closely aligned with several of the directions we want to pursue next.