- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes () to distinguish the procedural value of a reasoning trace from its formatting effects. It compares how much a trace helps on problems requiring its procedure versus format-matched problems requiring another procedure. The paper develops a theoretical interpretation and shows that () provides information about in-context transfer beyond correctness and raw codelength reduction.
Critical strengths
- The paper distinguishes useful methods from correct execution. The core construct is well motivated and clearly distinguished from correctness, raw compression, process reward, and influence-based signals. The format-residualized contrast directly targets a real confound in reasoning-trace evaluation.
- The paper provides a useful information-theoretic framework. Codelength gives a concrete way to measure how a trace changes a model’s predictions. The theoretical analysis connects these changes to beliefs about underlying procedures. The KL interpretation and baseline-invariance result further clarify the metric. The theoretical treatment is unusually complete for an empirical measurement paper.
- Targeted empirical controls and transparent reporting. The empirical program is broad and carefully controlled: three domains, five scorers, templated corruption ladders, two real benchmarks, shape-matched analyses, cross-scorer checks, negative-knowledge controls, and out-of-sample transfer prediction with an independent transfer model.
Critical weaknesses
- Format cancellation requires a clearer assumption. The mixture model allows a trace’s effect to vary across queries. Equal re-weighting across procedures within one query does not guarantee that format effects cancel between (P) and (N). Corollary 1 therefore needs an explicit condition connecting the updates across these two sets.
- The capacity claim goes beyond the bound. Corollary 2 bounds an individual score, not the score gap between valid and wrong traces. A wrong trace can still receive a negative score when a valid trace has little room for improvement. The bound therefore does not establish that saturated scorers cannot distinguish traces or explain reversed rankings.
- Null construction remains a source of uncertainty. Different procedures can produce the same answer, so different procedure labels do not guarantee (\epsilon)-separation. The real-data null sets also retain either procedure overlap or format mismatch. Moreover, (R) can increase by hurting null tasks without helping same-procedure tasks.
- Practical benefits remain uncertain. The strongest results concern transfer prediction. In-context selection uses designed corruptions, the PRM comparison covers only arithmetic, and fine-tuning selection shows no benefit. Whether (R) improves selection among natural traces or helps train a better reward model remains unclear.
Questions for authors
-
Clarify when format effects cancel. What must hold across queries for Corollary 1 to apply? Does sharing a surface format suffice, or must the normalized posterior updates satisfy an additional condition? Please state this condition explicitly and explain how the experiments support it.
-
Justify the claim about scorer saturation. Can you bound ), the quantity used to measure trace discrimination? What additional assumptions make this gap vanish as the scorer saturates? Please also explain what happens to the term in this limit.
-
Test sensitivity to null choice. How stable are trace rankings across alternative null sets? Please quantify how often high (R) comes mainly from harming null tasks rather than helping same-procedure tasks. What evidence supports -separation beyond assigning different procedure labels?
-
Show a concrete use of (R). How would you incorporate (R) into a correctness-based selector or reward model? A small comparison against correctness alone and correctness plus raw codelength reduction would help. Naturally generated candidates and held-out tasks would make this informative without requiring a full RL experiment.
4 comments
Response to Reviewer p9Ru (2/2)
W3 and Q3. The choice of null set, and ε-separation.
How often a high R comes from hurting the null problems. We split R into its two parts.
- On the real data with Llama-3.1-8B, the gap of 0.474 between correct and wrong-method traces is +0.446 from the problems that need the method and −0.028 from the null problems.
- A correct trace has R > 0 without making the problems that need the method easier in 4.5% of groups. With Mistral-7B this happens in 0.4% of groups, and in none on the code-tracing and templated data.
- A stricter score that sets R to zero unless the trace makes those problems easier leaves the main result at 0.468 (p = 8e-18).
How stable trace rankings are across null sets. We compared the two real null sets, which are biased in opposite directions.
- With Llama-3.1-8B, the median Kendall τ is 0.667, 78.4% of trace pairs keep their order, and 80.4% of groups agree on whether the correct trace beats the wrong-method trace.
- With Mistral-7B the numbers are 0.333, 72.1% and 79.0%.
- With the original and an alternate null on the templated benchmark, they are 0.810, 88.2% and 94.4%.
When we replace a fraction α of the null set with problems that need the trace's own method, the gap shrinks to about 1 − α of its value (0.890, 0.741 and 0.489 at α = 0.1, 0.25 and 0.5) and never changes sign.
Evidence for ε-separation beyond method labels. We agree that separation depends on answers, not labels, and the paper now says so right after the definition. We checked directly how often different methods give the same answer.
- Running every other method in the frame on every templated problem reproduces the correct answer for only 1.8% of problems. This keeps the error term below log₂(1 + ε̂) = 0.026 bits.
- On the code-tracing data, different methods never give the same answer, by construction.
- On the real data, 28.4% of the intermediate-quantity null's answers equal an intermediate number from the trace's own solution. This is why we call that null cautious.
- Removing the problems where answers coincide makes every result slightly stronger. On the templated data the gap goes from 0.684 to 0.698 with Llama-3.1-8B and from 0.209 to 0.214 with Mistral-7B. On the shape-matched real data it goes from 0.251 to 0.264 and from 0.122 to 0.129 (Appendix C.16).
W4 and Q4. A concrete use of R together with correctness. We ran exactly the comparison you suggested, on traces that models actually wrote and on new problems.
We used 121 real GSM-Symbolic problem groups with 1,410 chains of thought from three models. Each method picks one trace as a worked example for a separate model, and we measure that model's accuracy on twelve new problems per group. Compared with correctness alone (a random correct trace), here is what each addition gives.
- Correctness plus the plain loss drop adds 0.6 points (p = 0.15).
- Correctness plus the process reward model adds 1.5 points (p = 0.010).
- Correctness plus R adds 2.7 points (CI 1.1 to 4.3, p = 8e-4).
- Correctness plus the sum of the standardized R and PRM scores adds 3.8 points (p = 4e-6).
The last one is a simple combined reward of the kind you describe. It shows that R and the PRM add different information on real traces, as they did on our designed traces. When picking the best of n candidates, R's lead over the plain loss drop grows from 0.7 points with two candidates to 2.2 with twelve. The gain also holds when each template is counted once, with a second transfer model, with a second scoring model, and on harder problems of a different kind (Section 6.5, Table 2, Appendix C.12).
A full reinforcement learning experiment did not fit in the rebuttal period, and the paper does not claim one.
Thank you again. Your comments made the theory more correct and easier to check.
Thank you for the additional information which has explained my doubts, I am appreciative of the work done by all of you and I will keep my rating the same.
Thank you
Thank you for reading our rebuttal and for the constructive review. We are glad the additional details cleared up your questions.
Response to Reviewer p9Ru (1/2)
Thank you for a precise and technical review. Both of your theory points were correct, so we fixed the statements instead of defending them. Each of your questions now has a measured answer.
W1 and Q1. When do format effects cancel? You are right. Equal reweighting within one query does not mean the effects cancel across the different queries in P and N.
The revised Corollary 1 writes a format-only update as π(z | t, x) = w_t(x) π(z | c₀, x) for every procedure z in the frame. The factor w_t(x) may depend on the query. Theorem 1 then gives
R*(t) = (average of log₂ w_t(x) over P) − (average of log₂ w_t(x) over N) + O(ε).
So the format effect cancels exactly when its average is the same on the two problem sets. This holds if w_t depends on x only through surface features φ(x) that have the same distribution in P and N. Our matching on answer length, word form and story aims at exactly that. If log₂ w_t is L-Lipschitz in φ, then |R*(t)| ≤ L · W₁(φ_P, φ_N) + O(ε). The proof is in Appendix B.
The experiments support the condition because the leftover format effect can be measured. A fluent trace that computes nothing is the closest thing we can observe to a format-only trace. Its R on the real data is +0.019 with Llama-3.1-8B (CI −0.053 to 0.088), 4% of the gap. With Mistral-7B it is +0.043, 19% of the gap.
There is one exception, and we report it. On the templated data the short non-sequitur scores −0.333, because it makes the problems that need the method harder. So it is not a format-only trace there and cannot be used for this check (Appendix C.16).
W2 and Q2. A bound on the gap, and the log₂(1/γ) term. You are right again. Corollary 2 bounds a single score, and a wrong trace can still score below zero. We added Corollary 3.
Call the scoring model κ-saturated on a group if π(z(x) | c, x) ≥ 1 − κ for every context c in {c₀, t⁺, t⁻} and every related problem x, where z(x) is the procedure that generated x. Then
|R*(t⁺) − R*(t⁻)| ≤ 2 log₂(1/(1 − κ)) + O(ε).
On the null problems, π(z_N | t, x) ≥ 1 − κ, so γ ≥ 1 − κ and log₂(1/γ) ≤ log₂(1/(1 − κ)). Both terms go to zero as κ goes to zero.
The condition can be checked. Because p_M(a | x, c) ≤ (1 + ε) π(z | c, x), the model's own probability of the correct answer gives a lower bound on its belief in the right procedure.
We checked all fifteen model-and-domain cells, using the 10th percentile of that probability as a soft estimate of 1 − κ. The gap stays inside the bound in fourteen cells, including the small reversed Qwen cells. The exception is Mistral-7B on list tracing (0.208 against a bound of 0.174), which a percentile estimate can allow (Appendix C.15).
As a test fixed in advance, two reasoning models that already solve most GSM-Symbolic problems separate traces only weakly there, and clearly on the harder p2 version (Section 6.3).
The bound explains why a saturated model's gaps are small. It does not explain why some of them reverse. We removed the sentence that claimed it did, and we changed the caption of Table 3 in the same way.