- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether using a signed score versus a magnitude-only score in CoT perturbation evaluation changes the evaluation conclusions. It finds that the two scoring rules can reverse matched model rankings and can also select different top reasoning spans, and that they should be reported separately in concrete evaluations.
Critical strengths
The paper describes the specific experimental methods relatively clearly. The specific experimental methods are concise, and some experiments are rigorous.
Critical weaknesses
- The scope of the experiments is relatively narrow. Considering the performance of the models under evaluation, the existing core controlled evidence is concentrated on short-answer QA, with insufficient validation on complex long-chain reasoning tasks and open-ended tasks.
- The description of the value of this research is not clear enough. The average of +3 and -3 is 0, while the average of their absolute values is 3; these are obviously inconsistent. The experiment verifies the theoretically predicted inconsistency, but I do not see much beyond that. If this paper argues that both direction and magnitude are important in scoring, it could identify which existing evaluation methods ignore direction and provide empirical evidence that incorporating direction would substantially reverse existing ranking results.
Questions for authors
- Why is the continuation limited to a maximum of 60 tokens? In particular, this truncation may affect the conclusions on BoolQ.
- Appendix C.1 says the screening criterion is at least 6 words. Does this imply that all analyses in this paper are based on extremely short reasoning chains? The authors should add a statistic on the length distribution.