acceptodds
Under review as a conference paper at ICLR 2027

Procedure or Format? Measuring What a Reasoning Trace Teaches with Format-Residualized Codelength

Abstract

A chain-of-thought trace can lower a model's loss on related problems for two different reasons: it can teach a procedure that transfers, or it can merely prime the surface format those problems share. A score based on that loss reduction cannot tell the two apart, and we prove that no function of it can. We therefore measure the reduction twice, once on sibling problems that need the trace's procedure and once on a matched null set in the same format that needs a different one, and take the difference. Format priming is shared by both sets and cancels; we call what remains the format-residualized differential codelength R. Under the standard mixture view of in-context learning, R is, up to a separation term, the expected log Bayes factor the trace induces between its own procedure and the null's. On 224 clusters built from GSM-Symbolic, R separates valid from wrong-method reasoning by 0.251 bits per token (p = 1e-5) with trace length and answer shape controlled, and where the raw reduction ranks a fluent trace that performs no computation above a valid one, R ranks them correctly (p = 9e-7). On 216 controlled clusters in three domains under five open models, R falls with corruption severity and predicts an independent model's out-of-distribution transfer beyond length, raw reduction, and correctness (partial rho = 0.27, p = 2e-15). The result is a label-free, forward-pass-only test of whether a trace teaches the procedure a problem family needs. It requires a scorer that has not already solved the task, and it measures in-context value, which does not carry over to selecting fine-tuning data. Code and data are available at https://anonymous.4open.science/r/format-residualized-codelength-8D79.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
chain-of-thought, reasoning-value estimation, codelength, Bayes factor, in-context learning, format priming, process reward model, GSM-Symbolic, out-of-distribution transfer, identifiability, mixture models, teacher-forced evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_p9RuOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes (RR) to distinguish the procedural value of a reasoning trace from its formatting effects. It compares how much a trace helps on problems requiring its procedure versus format-matched problems requiring another procedure. The paper develops a theoretical interpretation and shows that (RR) provides information about in-context transfer beyond correctness and raw codelength reduction.

Critical strengths
  1. The paper distinguishes useful methods from correct execution. The core construct is well motivated and clearly distinguished from correctness, raw compression, process reward, and influence-based signals. The format-residualized contrast directly targets a real confound in reasoning-trace evaluation.
  2. The paper provides a useful information-theoretic framework. Codelength gives a concrete way to measure how a trace changes a model’s predictions. The theoretical analysis connects these changes to beliefs about underlying procedures. The KL interpretation and baseline-invariance result further clarify the metric. The theoretical treatment is unusually complete for an empirical measurement paper.
  3. Targeted empirical controls and transparent reporting. The empirical program is broad and carefully controlled: three domains, five scorers, templated corruption ladders, two real benchmarks, shape-matched analyses, cross-scorer checks, negative-knowledge controls, and out-of-sample transfer prediction with an independent transfer model.
Critical weaknesses
  1. Format cancellation requires a clearer assumption. The mixture model allows a trace’s effect to vary across queries. Equal re-weighting across procedures within one query does not guarantee that format effects cancel between (P) and (N). Corollary 1 therefore needs an explicit condition connecting the updates across these two sets.
  2. The capacity claim goes beyond the bound. Corollary 2 bounds an individual score, not the score gap between valid and wrong traces. A wrong trace can still receive a negative score when a valid trace has little room for improvement. The bound therefore does not establish that saturated scorers cannot distinguish traces or explain reversed rankings.
  3. Null construction remains a source of uncertainty. Different procedures can produce the same answer, so different procedure labels do not guarantee (\epsilon)-separation. The real-data null sets also retain either procedure overlap or format mismatch. Moreover, (R) can increase by hurting null tasks without helping same-procedure tasks.
  4. Practical benefits remain uncertain. The strongest results concern transfer prediction. In-context selection uses designed corruptions, the PRM comparison covers only arithmetic, and fine-tuning selection shows no benefit. Whether (R) improves selection among natural traces or helps train a better reward model remains unclear.
Questions for authors
  1. Clarify when format effects cancel. What must hold across queries for Corollary 1 to apply? Does sharing a surface format suffice, or must the normalized posterior updates satisfy an additional condition? Please state this condition explicitly and explain how the experiments support it.

  2. Justify the claim about scorer saturation. Can you bound R(t+)−R(t−)R(t^{+})-R(t^{-})), the quantity used to measure trace discrimination? What additional assumptions make this gap vanish as the scorer saturates? Please also explain what happens to the log⁡2(1/γ)\log_2(1/\gamma) term in this limit.

  3. Test sensitivity to null choice. How stable are trace rankings across alternative null sets? Please quantify how often high (R) comes mainly from harming null tasks rather than helping same-procedure tasks. What evidence supports ϵ\epsilon-separation beyond assigning different procedure labels?

  4. Show a concrete use of (R). How would you incorporate (R) into a correctness-based selector or reward model? A small comparison against correctness alone and correctness plus raw codelength reduction would help. Naturally generated candidates and held-out tasks would make this informative without requiring a full RL experiment.

4 comments
AuthorsOpenReview ↗

Response to Reviewer p9Ru (1/2)

Thank you for a precise and technical review. Both of your theory points were correct, so we fixed the statements instead of defending them. Each of your questions now has a measured answer.

W1 and Q1. When do format effects cancel? You are right. Equal reweighting within one query does not mean the effects cancel across the different queries in P and N.

The revised Corollary 1 writes a format-only update as π(z | t, x) = w_t(x) π(z | c₀, x) for every procedure z in the frame. The factor w_t(x) may depend on the query. Theorem 1 then gives

R*(t) = (average of log₂ w_t(x) over P) − (average of log₂ w_t(x) over N) + O(ε).

So the format effect cancels exactly when its average is the same on the two problem sets. This holds if w_t depends on x only through surface features φ(x) that have the same distribution in P and N. Our matching on answer length, word form and story aims at exactly that. If log₂ w_t is L-Lipschitz in φ, then |R*(t)| ≤ L · W₁(φ_P, φ_N) + O(ε). The proof is in Appendix B.

The experiments support the condition because the leftover format effect can be measured. A fluent trace that computes nothing is the closest thing we can observe to a format-only trace. Its R on the real data is +0.019 with Llama-3.1-8B (CI −0.053 to 0.088), 4% of the gap. With Mistral-7B it is +0.043, 19% of the gap.

There is one exception, and we report it. On the templated data the short non-sequitur scores −0.333, because it makes the problems that need the method harder. So it is not a format-only trace there and cannot be used for this check (Appendix C.16).

W2 and Q2. A bound on the gap, and the log₂(1/γ) term. You are right again. Corollary 2 bounds a single score, and a wrong trace can still score below zero. We added Corollary 3.

Call the scoring model κ-saturated on a group if π(z(x) | c, x) ≥ 1 − κ for every context c in {c₀, t⁺, t⁻} and every related problem x, where z(x) is the procedure that generated x. Then

|R*(t⁺) − R*(t⁻)| ≤ 2 log₂(1/(1 − κ)) + O(ε).

On the null problems, π(z_N | t, x) ≥ 1 − κ, so γ ≥ 1 − κ and log₂(1/γ) ≤ log₂(1/(1 − κ)). Both terms go to zero as κ goes to zero.

The condition can be checked. Because p_M(a | x, c) ≤ (1 + ε) π(z | c, x), the model's own probability of the correct answer gives a lower bound on its belief in the right procedure.

We checked all fifteen model-and-domain cells, using the 10th percentile of that probability as a soft estimate of 1 − κ. The gap stays inside the bound in fourteen cells, including the small reversed Qwen cells. The exception is Mistral-7B on list tracing (0.208 against a bound of 0.174), which a percentile estimate can allow (Appendix C.15).

As a test fixed in advance, two reasoning models that already solve most GSM-Symbolic problems separate traces only weakly there, and clearly on the harder p2 version (Section 6.3).

The bound explains why a saturated model's gaps are small. It does not explain why some of them reverse. We removed the sentence that claimed it did, and we changed the caption of Table 3 in the same way.

AuthorsOpenReview ↗

Response to Reviewer p9Ru (2/2)

W3 and Q3. The choice of null set, and ε-separation.

How often a high R comes from hurting the null problems. We split R into its two parts.

  • On the real data with Llama-3.1-8B, the gap of 0.474 between correct and wrong-method traces is +0.446 from the problems that need the method and −0.028 from the null problems.
  • A correct trace has R > 0 without making the problems that need the method easier in 4.5% of groups. With Mistral-7B this happens in 0.4% of groups, and in none on the code-tracing and templated data.
  • A stricter score that sets R to zero unless the trace makes those problems easier leaves the main result at 0.468 (p = 8e-18).

How stable trace rankings are across null sets. We compared the two real null sets, which are biased in opposite directions.

  • With Llama-3.1-8B, the median Kendall τ is 0.667, 78.4% of trace pairs keep their order, and 80.4% of groups agree on whether the correct trace beats the wrong-method trace.
  • With Mistral-7B the numbers are 0.333, 72.1% and 79.0%.
  • With the original and an alternate null on the templated benchmark, they are 0.810, 88.2% and 94.4%.

When we replace a fraction α of the null set with problems that need the trace's own method, the gap shrinks to about 1 − α of its value (0.890, 0.741 and 0.489 at α = 0.1, 0.25 and 0.5) and never changes sign.

Evidence for ε-separation beyond method labels. We agree that separation depends on answers, not labels, and the paper now says so right after the definition. We checked directly how often different methods give the same answer.

  • Running every other method in the frame on every templated problem reproduces the correct answer for only 1.8% of problems. This keeps the error term below log₂(1 + ε̂) = 0.026 bits.
  • On the code-tracing data, different methods never give the same answer, by construction.
  • On the real data, 28.4% of the intermediate-quantity null's answers equal an intermediate number from the trace's own solution. This is why we call that null cautious.
  • Removing the problems where answers coincide makes every result slightly stronger. On the templated data the gap goes from 0.684 to 0.698 with Llama-3.1-8B and from 0.209 to 0.214 with Mistral-7B. On the shape-matched real data it goes from 0.251 to 0.264 and from 0.122 to 0.129 (Appendix C.16).

W4 and Q4. A concrete use of R together with correctness. We ran exactly the comparison you suggested, on traces that models actually wrote and on new problems.

We used 121 real GSM-Symbolic problem groups with 1,410 chains of thought from three models. Each method picks one trace as a worked example for a separate model, and we measure that model's accuracy on twelve new problems per group. Compared with correctness alone (a random correct trace), here is what each addition gives.

  • Correctness plus the plain loss drop adds 0.6 points (p = 0.15).
  • Correctness plus the process reward model adds 1.5 points (p = 0.010).
  • Correctness plus R adds 2.7 points (CI 1.1 to 4.3, p = 8e-4).
  • Correctness plus the sum of the standardized R and PRM scores adds 3.8 points (p = 4e-6).

The last one is a simple combined reward of the kind you describe. It shows that R and the PRM add different information on real traces, as they did on our designed traces. When picking the best of n candidates, R's lead over the plain loss drop grows from 0.7 points with two candidates to 2.2 with twelve. The gain also holds when each template is counted once, with a second transfer model, with a second scoring model, and on harder problems of a different kind (Section 6.5, Table 2, Appendix C.12).

A full reinforcement learning experiment did not fit in the rebuttal period, and the paper does not claim one.

Thank you again. Your comments made the theory more correct and easier to check.

Thank you for the additional information which has explained my doubts, I am appreciative of the work done by all of you and I will keep my rating the same.

Thank you for reading our rebuttal and for the constructive review. We are glad the additional details cleared up your questions.

Reviewer_mePUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether a reasoning example helps a model learn a method or simply makes the correct answer more likely through familiar wording and formatting. It scores an example by measuring how much it lowers correct-answer loss on new questions using the same method, then subtracting how much it helps on similar-looking questions using a different method. The authors test this score on arithmetic, deduction, and code-tracing tasks. They also test whether it predicts how much an example improves another model’s answer accuracy. Selecting fine-tuning examples with the score does not improve performance.

Critical strengths

The paper studies an interesting problem with a simple approach. Some experiments are useful, particularly the subtraction using comparison questions and the evaluation of generated-answer accuracy on a separate model. These help examine whether the score captures more than familiar wording.

Critical weaknesses
  1. Lower correct-answer loss does not establish better problem-solving.

An explanation could make the model more likely to produce an answer or a number without helping it choose the correct one. The subtraction is intended to remove this effect, but depends on it affecting both question groups equally. The paper should test whether the correct answer gains probability relative to plausible wrong answers

  1. The demonstrated scope and practical application are limited.

The main experiments use small models that are not dedicated reasoning models, and mainly evaluate reasoning as worked examples placed before another question. They do not establish whether the findings extend to reasoning models and their own generated reasoning traces. Familiar wording improving answer likelihood is also not especially surprising for models trained on next-token prediction. The paper therefore needs stronger evidence of what the proposed score adds to our understanding or improves in practice. The unsuccessful fine-tuning selection experiment further limits its demonstrated application.

  1. The writing is hard to follow.

There are too many technical terms and long sentences combining several claims. This makes a simple idea unnecessarily difficult to understand. The paper would be clearer with plain language.

Questions for authors
  1. Does an explanation increase the probability of the correct answer more than it increases the probability of plausible wrong answers ? For example, when the answer is “30,” does it help the model prefer “30” over reasonable alternatives, or does it simply make numbers generally more likely ? This would help distinguish better problem-solving from preparation to output an answer.

  2. What practical benefit does the score provide over ordinary loss reduction, given the mixed results across tasks and the lack of improvement in fine-tuning selection ?

Suggestions for a stronger version of the paper: Testing recent reasoning models and their generated traces would broaden the evidence. It would also be interesting to investigate whether the fine-tuning failure comes from selecting examples with one objective while training with another, and to test the score directly as an RL reward.

These may require more work than is feasible during rebuttal. I do not expect all these experiments to be completed, but the limited scope and demonstrated usefulness of the current results are central to my assessment; I currently lean toward rejection.

5 comments
AuthorsOpenReview ↗

Response to Reviewer mePU (1/2)

Thank you for a clear review and for suggesting concrete experiments. We ran the wrong-answer test you asked for and the reasoning-model study, and we rewrote the hardest passages.

W1 and Q1. Does the trace make the right answer more likely than plausible wrong answers, or does it just make numbers more likely?

This is the right test. For every related problem we built eight wrong answers with exactly the same number of digits as the right answer.

  • The answer a wrong method gives on that problem.
  • Intermediate numbers from the worked solution.
  • Numbers that appear in the problem.
  • Near misses, such as the right answer plus or minus 1, 2, 5 or 10.

We then measured how much the trace raises the right answer above these wrong answers, compared with no trace. Every candidate has the same number of digits, so a trace that just makes numbers more likely cannot score well on this test.

Results on the 224 real problem groups.

  • With Llama-3.1-8B, a correct trace raises the right answer above the wrong ones by 0.714 bits more than a wrong-method trace does (p = 4e-19). With Mistral-7B the difference is 0.386 bits (p = 2e-14).
  • When the wrong answers are only those the wrong method itself produces, the difference is 0.762 bits (p = 1e-18).
  • A correct trace moves the right answer to the top of the list on 43.8% of the problems where it was not already on top. A wrong-method trace does so on only 12.1%.

The case you describe, a trace that just makes the model readier to output an answer, is exactly our fluent non-sequitur trace. It raises the right answer by about the same amount on both problem sets (0.342 and 0.328 bits), so its R is 0.014 (CI −0.015 to 0.043). The correct trace beats it by 0.414 bits (p = 6e-14).

Our theory also says this new score measures the same thing as R, namely how much the trace shifts the model's belief toward the right method. In practice the two agree at Spearman 0.742 (Section 3, Section 6.2, Appendix C.13).

W2. Reasoning models and the traces they write.

Traces written by reasoning models. We collected 968 traces from DeepSeek-R1-Distill-Qwen-7B, Qwen3-8B in thinking mode, OpenThinker-7B and Qwen3-4B-Thinking-2507. We scored each one in full (median 418 words) and as its final solution only (median 56 words).

  • The thinking text lowers the plain loss by a further 0.224 bits per token. It raises R by only 0.038 (p = 4e-12 for the difference).
  • The thinking text does not make the trace a better worked example either (+0.082 against +0.079, p = 0.48).
  • R also tells correct from incorrect reasoning traces better than the plain loss drop (AUC 0.694 against 0.578).

So on the long traces that reasoning models write, R ignores extra text that the plain loss drop rewards.

Reasoning models as the scoring model. Our capacity result predicts that a model that already solves a task cannot tell traces apart on it. We tested this prediction, fixed in advance.

  • DeepSeek-R1-Distill-Qwen-7B and Qwen3-8B are rarely unsure on GSM-Symbolic (budgets of 0.48 and 0.61 bits per token). They separate traces only weakly (+0.052, p = 0.07, and +0.068, p = 0.04).
  • On the harder p2 version, their budgets rise to 1.52 and 1.69 bits, and the separation comes back (+0.294 and +0.328, p < 1e-7).
  • Llama-3.1-8B separates traces on both versions (+0.474 and +0.612).

So what matters is how much the scoring model still has to learn about the task, not how strong it is (Section 6.3, Appendix C.14).

AuthorsOpenReview ↗

Response to Reviewer mePU (2/2)

W2. "It is not surprising that familiar wording makes answers more likely." We agree, and we see that we did not state our finding clearly. That familiar wording helps is our starting point, not our result. Our result is that the plain loss drop, which is used to score and pick traces, ranks a fluent trace that computes nothing above a correct trace in 76% of real problem groups. We prove that no adjustment of the plain loss drop alone can fix this. R prefers the correct trace in 63% of groups and favours it significantly (p = 9e-7). The new natural-trace experiment shows what this means in practice.

W2 and Q2. What does R give that the plain loss drop does not, and what about fine-tuning? We now report three concrete benefits.

  1. Better choice among traces that models actually write. Among correct traces from three models, using the trace with the highest R as the worked example raises a separate model's accuracy by 2.7 points over a random correct trace (p = 8e-4). Picking by the plain loss drop adds only 0.6 points, which is not significant. R combined with a process reward model adds 3.8 points (new Section 6.5).
  2. Fluent traces that do no computation cannot fool it, as shown above and in Section 6.2.
  3. It stays useful when the new problems differ in kind. When the new problems combine the method with a new step, the plain loss drop points the wrong way, while R still adds useful information (Appendix C.9).

On fine-tuning, we tested your idea that the problem was a mismatch between the model that picks the data and the model that is trained. We re-scored the training pool with the student model itself, Qwen2.5-1.5B, so that the same model picks and learns.

  • With trace length controlled, picking by the student's own R gives 0.565 accuracy, against 0.583 for a random correct trace (p = 0.38). This is about the same as the original 0.562.
  • The student's R and the original scoring model's R agree only moderately (0.482).

So the mismatch is not the cause. What helps as a worked example and what helps as training data are different things. R is useful for choosing worked examples, not fine-tuning data, and the paper now says this plainly.

Testing R as a reward for reinforcement learning is more than we can do in the rebuttal period. The closest test we could run is picking the best of n candidates by R (Appendix C.12). There, R's lead over the plain loss drop grows as more candidates are compared.

W3. Writing. Thank you, we agree. In the revision we made three changes.

  • We rewrote the problem statement in the introduction in plain language.
  • We added a short paragraph that says what R is and what it is not. R is a label-free score of how much a trace teaches when shown as an example. It is not a correctness checker, it is not a stronger detector of wrong methods than the plain loss drop, and it does not pick fine-tuning data.
  • We shortened the densest passages and moved technical checks to the appendix.

All changes are in blue. We would be grateful to hear about any passage that is still hard to follow.

We hope the wrong-answer test, the reasoning-model study and the natural-trace results address the concerns about scope and usefulness that were central to your assessment.

Thank you for the additional experiments. They address some specific questions, but the main concerns remain. The new evidence strengthens the case within a restricted setting rather than establishing broader applicability.

  • The reasoning-model experiments still evaluate traces as examples supplied for another problem. Comparing full thinking with the final solution tests which makes a better demonstration, not whether the thinking helps the original model solve its problem. Likewise, scoring answers after an empty thinking block does not evaluate the reasoning model’s normal reasoning process. Including reasoning models in this way therefore does not fully address my concern about relevance to their own reasoning during problem-solving.

  • The wrong-answer test is useful, but matching digit counts does not necessarily match token counts. It also restricts the comparison to one numerical format, without showing whether the result holds for equivalent answers such as “30” and “thirty,” or non-numerical answers. This leaves the broader interpretation insufficiently supported.

  • My fine-tuning hypothesis concerned different objectives: selection rewards a trace for helping on other problems, whereas fine-tuning trains the student to reproduce the trace token by token. Using the student itself as scorer changes which model measures usefulness, but leaves this objective mismatch intact. The new experiment therefore does not test the explanation I proposed.

The additions strengthen the paper, but its demonstrated scope and usefulness remain below the ICLR acceptance bar in my assessment. I maintain my score.

Reviewer_4EEmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes format-residualized differential codelength (R) to measure whether a chain-of-thought (CoT) trace conveys a transferable reasoning procedure rather than merely priming the surface format of related problems. The proposed metric computes the reduction in gold-answer codelength on procedure-matched sibling problems and subtracts the corresponding reduction on format-matched problems requiring a different procedure. The authors provide a theoretical interpretation of R as an approximate log Bayes factor under a Bayesian-mixture view of in-context learning.

The method is evaluated on controlled reasoning benchmarks spanning arithmetic, relational deduction, and list tracing, as well as GSM-Symbolic and CRUXEval, using multiple open-weight language models. The results suggest that R can distinguish valid reasoning from wrong-method or format-only traces and provides additional predictive information about out-of-distribution transfer beyond raw codelength reduction, trace length, and correctness. The paper also examines limitations related to scorer capacity and fine-tuning data selection.

Critical strengths
  1. A well-motivated formulation for disentangling procedural information from format priming.

The paper identifies an important limitation of using raw codelength reduction to evaluate reasoning traces: improvements in answer likelihood may arise from surface-format familiarity rather than transferable reasoning. The proposed residualization provides a conceptually simple way to address this confound. The theoretical analysis, particularly the Bayes-factor interpretation and identifiability result, further motivates why the contrast between procedure-matched and procedure-mismatched problems is meaningful.

  1. Careful experimental design with multiple controls and complementary evaluations.

The paper evaluates the proposed metric across multiple reasoning domains, scorer families, and controlled trace corruptions. The inclusion of fluent non-sequitur traces is particularly valuable, as it directly tests whether the metric can distinguish procedural information from superficial fluency. The authors also investigate answer-shape confounds, alternative null sets, and correlations with realized transfer on independent models. These experiments provide useful evidence beyond simply demonstrating that the metric correlates with trace correctness.

Critical weaknesses
  1. The reliance on carefully constructed procedure-matched and format-matched problem sets limits the general applicability of the method.

The effectiveness of R critically depends on constructing two sets of problems that share a surface format while requiring different reasoning procedures. However, satisfying these conditions simultaneously is challenging outside controlled benchmarks. In particular, Appendix C.2 acknowledges that the intermediate-quantity null in GSM-Symbolic partially overlaps with the demonstrated procedure, while the cross-template null has relatively weak surface similarity. These discrepancies raise concerns about how reliably the residual isolates procedural information in more general settings.

Moreover, computing R requires reference answers and knowledge of which problems share or differ in their underlying procedures. Although the method does not require explicit trace-quality labels, these additional requirements may limit its practicality on open-ended reasoning tasks where procedural structure is not predefined. This weakens the broader applicability of the proposed measurement framework.

  1. The practical benefit of R for selecting useful reasoning traces remains insufficiently established.

Although R shows promising correlations with realized transfer, its incremental predictive value varies considerably across domains and scorer models. For example, Table 1 shows that its partial correlation with transfer is essentially zero in arithmetic after controlling for raw codelength reduction, length, and correctness. More importantly, the trace-selection experiments in Appendix C.10 show that the scorer-averaged R selector performs worse than random selection in arithmetic and list tracing, despite performing well in relational deduction.

The authors provide a plausible explanation based on scorer saturation, but these results indicate that obtaining a reliable selection signal requires careful scorer choice. Furthermore, much of the transfer-prediction and selection evidence relies on controlled trace corruptions rather than diverse, naturally generated reasoning traces. It therefore remains unclear whether the observed benefits translate into robust improvements in realistic trace-selection scenarios.

Questions for authors
  1. Regarding the construction and generalization of the null sets (Weakness 1):

Could the authors provide additional evidence that R remains reliable when the procedure-matched and null sets cannot be precisely controlled? In particular, how sensitive are the rankings and transfer correlations to imperfect format matching or partial procedural overlap? A clearer discussion or evaluation of how such sets could be constructed for previously unseen, open-ended reasoning tasks, including the required reference answers and procedural information, would substantially strengthen the practical relevance of the method.

  1. Regarding the practical effectiveness of trace selection (Weakness 2):

Could the authors further evaluate R as a selector among diverse, naturally generated reasoning traces, rather than primarily among predefined corruption variants? In light of the mixed selection results in Appendix C.10, it would be particularly useful to understand whether the method consistently improves realized transfer over random selection and raw codelength-based selection across domains, and whether an appropriate non-saturated scorer can be identified without relying on labeled validation data.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 4EEm (1/2)

Thank you for a careful review and for your kind words about the idea and the controls. You raised two concerns, how hard it is to build the null set and how useful R is for choosing traces. We ran new experiments for both, all fixed in advance.

W1 and Q1. Does R still work when the null set is not perfect? We agree this is the main practical cost of the method, so we measured it directly.

Overlap with the trace's method only makes R smaller. We replaced a fraction α of each null set with problems that need the trace's own method. On the 224 real groups (Llama-3.1-8B), the gap between correct and wrong-method traces falls to 0.890, 0.741 and 0.489 of its first value at α = 0.1, 0.25 and 0.5. Theory predicts 1 − α. The sign never flips. The null set you mention shares an intermediate step with the trace's method in 28.4% of its answers. This test shows that such overlap only makes R smaller and does not mislead it.

Two very different null sets rank traces in much the same way. We compared the null built from intermediate quantities with the null built from other templates.

  • With Llama-3.1-8B, the median Kendall τ between the two rankings is 0.667. 78.4% of trace pairs keep their order, and 80.4% of groups agree on whether the correct trace beats the wrong-method trace.
  • With Mistral-7B the agreement is weaker (τ = 0.333, 72.1% of pairs, 79.0% of groups). This fits its smaller gaps.
  • On the templated benchmark, the original and an alternate null give τ = 0.810, with 88.2% of pairs and 94.4% of groups agreeing.

A bad format match can be detected without labels. Prompted by Reviewer p9Ru, Corollary 1 now states exactly when format effects cancel. This also gives a simple check. A fluent trace that computes nothing measures the leftover format effect directly. On the real data that leftover is +0.019 bits per token with Llama-3.1-8B (CI −0.053 to 0.088), about 4% of the gap. With Mistral-7B it is +0.043, about 19% of the gap.

How to build the sets for a new task. Appendix C.16 now gives a short recipe.

  • The related problems are rewrites of the original problem with the same template.
  • The null problems keep the same story and ask for a different quantity. Our real-data pipeline already creates these automatically.
  • Before trusting R, run the format check above, check that different methods do not give the same answers, and choose the scoring model with the label-free rule described in part 2.

W1. Reference answers. R needs correct answers for the related problems. It never needs a label on the trace being scored. We tested whether answers produced by a model are good enough. We replaced every reference answer with the majority vote of eight answers from Qwen2.5-72B-Instruct, which is right on 94.6%, 93.8% and 96.2% of the three problem sets. Every main result stays inside its original confidence interval.

  • Correct versus wrong-method traces, 0.458 instead of 0.474.
  • The same comparison on the shape-matched subset, 0.244 instead of 0.251.
  • Correct versus non-sequitur traces, 0.089 instead of 0.092.

Over all scores, R computed with model answers and R computed with reference answers agree at Spearman 0.964.

AuthorsOpenReview ↗

Response to Reviewer 4EEm (2/2)

W2 and Q2. Is R useful for choosing traces?

Choosing among traces that models actually write. You asked for exactly the right experiment, and we ran it. We used 121 real GSM-Symbolic problem groups with 1,410 chains of thought written by Qwen3-8B, Qwen2.5-7B-Instruct and SmolLM3-3B. Each method picks one trace per group. A separate model then uses it as a worked example on twelve new problems from the same group, and we measure its accuracy over eight sampled answers and one greedy answer per problem.

The table shows the change in accuracy compared with picking a correct trace at random.

How the trace is chosenChange in accuracyp
Any trace at random (no correctness check)−6.2 points
Correct trace with the largest plain loss drop+0.6 points (CI −0.5 to 1.7)0.15
Correct trace with the highest PRM score+1.5 points0.010
Correct trace with the highest R+2.7 points (CI 1.1 to 4.3)8e-4
Correct trace with the highest R and PRM together+3.8 points4e-6
  • Picking by R closes 26.7% of the gap to the best possible correct trace.
  • R's lead over the plain loss drop grows with the number of candidates, from 0.7 points with two candidates to 2.2 with twelve.
  • The gain holds when each template is counted once (+2.5, p = 4e-3), with a second transfer model, Mistral-7B-Instruct (+2.3, p = 4e-3), with Mistral-7B as the scoring model (+1.9, p = 0.02), and on harder problems of a different kind (+1.5, p = 0.03).
  • Among correct traces, R predicts how much a trace helps even after accounting for its length and its plain loss drop (partial Spearman +0.187, p = 2e-3).

So among real traces, R beats both random choice and the plain loss drop. Its limits are clear. Without the correctness check, R alone does 2.8 points worse than the check. R works best together with correctness, not instead of it. This study covers arithmetic only, and the paper says so.

The mixed results in Appendix C.10 (now C.11) and the near-zero arithmetic correlation. You read both correctly. In arithmetic, the plain loss drop already carries the same signal as R, so R adds nothing to transfer prediction there (Table 1). What R adds in arithmetic is that fluent traces with no computation cannot fool it. On real arithmetic traces, R now helps where the plain loss drop does not.

The average over scoring models does worse than random in arithmetic and list tracing because it includes Qwen models that already solve those tasks. Our capacity result says such models cannot tell traces apart.

Choosing a good scoring model without labelled data. We fixed two rules in advance. Both use only how uncertain each scoring model is about the related problems' answers, which we call its budget. Neither uses labels on traces or on transfer.

  • Rule A picks the scoring model with the largest budget in each domain. It picks Llama-3.1-8B in arithmetic (2.61 bits) and list tracing (2.22 bits), the two domains where the average fails, and Qwen2.5-1.5B in deduction, where that model separates traces strongly.
  • Rule B weights each scoring model by its budget. Compared with the plain average, it does better in arithmetic (−0.182 against −0.340) and list tracing (−0.204 against −0.414), equally well in deduction, and beats random overall by +0.110 (p = 0.024).

To be open about this, the rule turns into a formula the advice our capacity result already gave at submission. Llama-3.1-8B's gain over random (+0.106, p = 0.028, at or above random in every domain) was already in the paper.

The paper now has a second label-free check. The scoring model's own probability of the correct answer shows how close it is to already knowing the task (new Corollary 3, checked in all fifteen model-and-domain cells in Appendix C.15).

We hope these results address your concerns about how widely R applies and how useful it is. We would be glad to answer further questions.

Thank the authors for providing additional results and analysis, which address my main concerns. Therefore, I have raised my score from 2 to 3.

Thank you for reading the revision and raising your score. We are glad our new experiments and analysis addressed your main concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all three reviewers for careful and helpful reviews. All three liked the basic idea and the controls, especially the fluent "non-sequitur" trace (a trace that reads well but does no real computation) and the use of a separate model to measure transfer. The reviews ask for four main things. We ran new experiments or fixed the theory for each one. We fixed every new experiment's prediction, main test and decision rule in our repository before running it, and we report every result, including the weak ones. All new or changed text in the revised PDF is in blue. The main text now uses the extra tenth page allowed for revisions.

1. Does R help in practice, on traces that models actually write? (all reviewers) We used 121 real GSM-Symbolic problem groups with 1,410 chains of thought written by three models (1,014 correct, 396 wrong). Each method picked one trace per group. A separate model (Llama-3.2-3B-Instruct, chosen before the runs) then used that trace as a worked example on twelve new problems.

  • Among the correct traces, picking the one with the highest R raises accuracy by 2.7 points over picking a correct trace at random (95% CI 1.1 to 4.3, p = 8e-4).
  • Picking by the plain loss drop adds only 0.6 points, which is not significant.
  • Picking by a process reward model (PRM) adds 1.5 points. Using R and the PRM together adds 3.8 points (p = 4e-6).
  • The gain also holds when we count each problem template once, with a second transfer model, with a second scoring model, and on harder problems of a different kind (new Section 6.5 and Table 2).

2. Does the trace favour the right answer over plausible wrong ones? (mePU) For every problem we also scored eight wrong answers with exactly the same number of digits as the right answer.

  • A correct trace raises the right answer above these wrong answers by 0.714 bits more than a wrong-method trace does (p = 4e-19).
  • A fluent trace that computes nothing helps the right answer equally on both problem sets, so its R stays at zero (Section 6.2, Appendix C.13).

3. Reasoning models (mePU). We scored traces written by four reasoning models, and we also used two reasoning models as the scoring model.

  • The long "thinking" text lowers the plain loss by a further 0.224 bits per token. It raises R by only 0.038, and it does not make the trace a better worked example.
  • As scoring models, DeepSeek-R1-Distill-Qwen-7B and Qwen3-8B already solve most GSM-Symbolic problems, so they barely tell traces apart. On the harder p2 version they tell them apart clearly, as our capacity result predicts (Sections 6.3 and 6.5, Appendix C.14).

4. How much does R depend on the comparison set, and does it need reference answers? (4EEm, p9Ru) R compares the trace's help on problems that need its method with its help on a "null" set of similar-looking problems that need a different method. Details are in Section 6.1 and Appendix C.16.

  • If part of the null set is replaced by problems that need the same method, R shrinks in proportion and never changes sign.
  • Two very different null sets give largely the same ranking of traces.
  • The effect comes from helping the problems that need the method (0.446 of the 0.474 gap), not from hurting the null problems.
  • Different methods give the same answer on only 1.8% of templated problems. Removing those problems makes every result slightly stronger.
  • Answers produced by a stronger model by majority vote work in place of reference answers and give the same results.

5. Theory (p9Ru). Both of Reviewer p9Ru's theory points were correct, and we fixed them (Section 3, Appendix B, Appendix C.15).

  • Corollary 1 now states the extra condition it needs across the two problem sets.
  • A new Corollary 3 limits the gap between two traces when the scoring model already knows the task. This condition can be checked from the model's own outputs.
  • We removed the sentence that used the old bound to explain reversed rankings.

6. Clarity (mePU and all).

  • We rewrote the problem statement in the introduction in plain language.
  • We added a short paragraph on what R is and what it is not.
  • We shortened the densest passages and moved technical checks to the appendix.
  • The abstract has one new sentence with the natural-trace result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.