acceptodds
Withdrawn from ICLR 2027

When Conclusions Reverse: Signed vs. Magnitude Effects in Chain-of-Thought Evaluation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

As reasoning models are increasingly evaluated by their chain of thought (CoT), perturbation tests are a common way to probe how much the chain affects the model’s answer by changing part of it and measuring how the answer shifts. The resulting effect can be scored in two ways: a signed score that keeps both the size and the direction of the change in the reference answer’s probability, or a magnitude-only score that keeps only the size. We ask whether this scoring choice alone can change the conclusions drawn from a CoT evaluation when everything else is held fixed. Across about 79,000 perturbation records from nine models and six question-answering datasets, we find that it can. (1) In our main comparison, the two scores rank two models in the opposite order over the 132 questions they share (out of 144 and 150); and (2) they rank a different part of the reasoning first for 40–56% of questions in three of four model–dataset combinations spanning two comparisons, one on a multiple-choice dataset and one on a yes/no dataset. In the broader descriptive analysis, model rankings change in most comparisons, while others show little or no disagreement between the two scores. These results show that signed and magnitude-only scores answer different questions and should be reported separately when both are relevant.

Primary area:
interpretability and explainable AI
Keywords:
chain-of-thought, large language models, interpretability, reasoning evaluation, perturbation analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sqeCOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether using a signed score versus a magnitude-only score in CoT perturbation evaluation changes the evaluation conclusions. It finds that the two scoring rules can reverse matched model rankings and can also select different top reasoning spans, and that they should be reported separately in concrete evaluations.

Critical strengths

The paper describes the specific experimental methods relatively clearly. The specific experimental methods are concise, and some experiments are rigorous.

Critical weaknesses
  • The scope of the experiments is relatively narrow. Considering the performance of the models under evaluation, the existing core controlled evidence is concentrated on short-answer QA, with insufficient validation on complex long-chain reasoning tasks and open-ended tasks.
  • The description of the value of this research is not clear enough. The average of +3 and -3 is 0, while the average of their absolute values is 3; these are obviously inconsistent. The experiment verifies the theoretically predicted inconsistency, but I do not see much beyond that. If this paper argues that both direction and magnitude are important in scoring, it could identify which existing evaluation methods ignore direction and provide empirical evidence that incorporating direction would substantially reverse existing ranking results.
Questions for authors
  1. Why is the continuation limited to a maximum of 60 tokens? In particular, this truncation may affect the conclusions on BoolQ.
  2. Appendix C.1 says the screening criterion is at least 6 words. Does this imply that all analyses in this paper are based on extremely short reasoning chains? The authors should add a statistic on the length distribution.
Reviewer_nUeLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how signed and magnitude-based scoring affect conclusions in chain-of-thought evaluation. It perturbs reasoning traces and estimates the change in correct-answer probability, comparing the signed effect XX with its magnitude ∣X∣|X|. Primary experiments on BoolQ and OpenBookQA, together with a broader analysis covering nine model checkpoints and six datasets, show that the two scores can reverse model orderings and select different top-ranked spans. The broader analysis finds reversals in 36 of 47 matched model comparisons, while an external step-level reanalysis finds different top-step selections in approximately 37% of traces. These differences arise from how the scores treat perturbations that improve correctness.

Critical strengths
  1. The motivation and research question is clear and sensible.
  2. I think the harmful/helpful decomposition explains how reversals arise which is interesting.
Critical weaknesses

Major:

  1. The span-ranking results are not adequately calibrated against sampling noise. The authors should compare the observed disagreement with a null baseline in which perturbations have no true effect, using the same sampling and selection procedure. Experiments varying the sampling budget nn and repeating the selection with independent samples would help establish whether the selected spans have reliably distinguishable effects. Robustness to alternative candidate-span choices should also be assessed.
  2. The writing could be improved, particularly because the interpretation of the results depends on the perturbation method. The main text should explain more clearly how the concrete perturbation design affects the interpretation of the results. A worked example, together with an explicit discussion of single-word perturbations and their possible semantic and formatting effects, would make the findings easier to assess.
  3. Given the mathematical formulation, it is quite expected that signed and magnitude scores capture different properties and can produce different rankings. Disagreement alone does not establish that either gives a misleading evaluation. The paper would benefit from interventions with independently characterized semantic roles, such as meaning-preserving edits, error insertion, and error correction, to clarify when each score provides a useful interpretation.
  4. Relatedly, I find the current analysis somewhat shallow and the contribution limited. Beyond just establishing disagreement, I would expect a deeper analysis of how the choice of score interacts with different perturbation types, along with clearer guidance on which combinations of scoring rules and perturbations are appropriate for different goals in CoT evaluation.

Minor:

  1. typo at line 220
Questions for authors

See weaknesses.

Reviewer_oRVFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how retaining or discarding the direction of perturbation effects can influence conclusions in chain-of-thought (CoT) evaluation. Specifically, it compares a signed score, X=pref(clean)−pref(perturbed)X = p_{\mathrm{ref}}(\mathrm{clean}) - p_{\mathrm{ref}}(\mathrm{perturbed}), with its magnitude-only counterpart, ∣X∣|X|, while holding the underlying perturbation records fixed.

Across approximately 79,000 perturbation records from nine models and six QA datasets, the authors show that the two scoring rules can reverse model rankings and select different reasoning spans. The main OpenBookQA comparison demonstrates a statistically robust ranking reversal, primarily driven by differences in how strongly perturbations increase reference-answer support. The paper also provides an algebraic decomposition of these effects, additional robustness checks, and an external reanalysis of previously released reasoning-step measurements.

Overall, the paper presents a careful empirical audit of an important measurement choice in CoT evaluation, although its contribution is primarily methodological rather than a new approach to measuring CoT faithfulness.

Critical strengths
  • The paper compares signed and magnitude-only scores using exactly the same realized perturbation records within each model, holding the reasoning traces, perturbations, sampled continuations, and parsing procedures fixed. Cross-model comparisons are further restricted to shared eligible items. This paired design is a major strength because it isolates the consequences of the scoring transformation from other experimental factors.

  • The main OpenBookQA reversal is supported by matched-item bootstrap confidence intervals, a two-stage bootstrap accounting for shared clean continuation pools, leave-one-item-out analyses, and temperature sensitivity checks. The decomposition into harmful and helpful contributions also clarifies that the reversal is primarily associated with differences in the magnitude, rather than merely the frequency, of helpful perturbations.

Critical weaknesses
  1. The main findings are partly consequences of the scoring definitions, which limits the incremental scientific insight.

The model-ordering reversal follows directly from the algebraic relationship between harmful and helpful contributions. Appendix A establishes that a reversal occurs precisely when the absolute between-model difference in helpful contributions exceeds that in harmful contributions. Therefore, the more informative empirical finding is not the reversal itself, but that helpful perturbations occur frequently and differ substantially in magnitude across models. A similar issue affects the reasoning-span results. When all four perturbations increase reference-answer support, signed scoring selects the least helpful perturbation, while magnitude-only scoring selects the largest change. Unless all effects are tied, these criteria necessarily select different spans. Table 11 shows that disagreement reaches 54–96% in all-helpful cases, compared with only 6–22% when at least one harmful effect is present. These observations are valid, but I am less convinced that the headline disagreement rates provide substantial new insight beyond the underlying effect-sign distributions. The paper would be stronger if it more clearly separated disagreement arising directly from the definitions from additional empirical findings, particularly those that remain informative under mixed-sign conditions.

  1. The practical implications for existing CoT evaluation methods remain insufficiently established, and the results are sensitive to the evaluation protocol.

Although the paper is motivated by existing CoT evaluation methods, its primary model-ordering endpoint is defined specifically for this study. Prior methods such as Lanham et al., Thought Anchors, and Thought Branches use different interventions and evaluation quantities. The paper does not demonstrate that applying signed versus magnitude-only scoring would reverse an actual published model-level conclusion. The external reanalysis of Du et al. provides useful additional evidence for step selection, but does not establish a corresponding change in the original study's conclusions. Moreover, most non-GSM8K interventions modify individual words, followed by 60-token continuations. These interventions are substantially different from sentence-level resampling or broader reasoning interventions. Changes in reference-answer support may reflect local wording, output formatting, or answer-generation behavior rather than changes in reasoning content.

The output diagnostics reinforce this concern. In BoolQ, 83–91% of Gemma-2's helpful gains are associated with empty output termination. In OpenBookQA, the authors' valid-output analysis preserves the reversal, but only 616 of 5,000 bootstrap replicates are defined. Model-specific eligibility and the available headroom in clean reference-answer support may further influence the frequency and magnitude of helpful effects. The authors acknowledge several of these limitations, which I appreciate. Nevertheless, they collectively restrict how broadly the observed reversals can be interpreted as consequential changes to CoT evaluation in practice.

Questions for authors
  • Since much of the reported top-span disagreement is concentrated in all-helpful cases, could the authors quantify how much of the overall 40–56% disagreement is attributable to this stratum versus other sign patterns, including zero-effect cases? Could they also assess the contribution of finite continuation-sampling noise to the top-span selection endpoint? This would help distinguish stable empirical differences from effects of the scoring definitions and sampling variability.

  • Could the authors report the distribution of pref(clean)p_{\mathrm{ref}}(\mathrm{clean}) in the main OpenBookQA comparison and examine whether the ranking reversal persists after stratifying by clean reference-answer support? This seems particularly relevant because items with lower clean support have more room for beneficial perturbation effects, whereas the GSM8K results suggest that limited headroom may substantially reduce disagreement.

  • Could the authors provide a more direct example showing how the scoring choice changes a substantive conclusion from an established CoT evaluation setting, rather than only the model-ordering and top-span endpoints constructed for this study? A comparison using a more representative reasoning-level intervention, or a reanalysis of an existing published evaluation, would help establish the practical significance and generalizability of the findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.