- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies when redundant reasoning is more accurate than stating each intermediate result only once. The authors model a reasoning chain as noisy load-bearing steps: sharp writes each step’s result once; flat re-derives before using the previous step’s result and adopts the re-derived value on conflict. All three parameters are measured directly on the model: the stepwise accuracy cost under no noise, and the probabilities and that the two procedures repair injected errors. The paper proves that as per-step error rate rises, flat can exceed sharp if and only if , and gives a closed-form solution for the crossover location. This condition reduces to the old formula when .
Experiments cover modular arithmetic, stacks, automata, permutations, code execution, kinship relations, and GSM linear chains. Eight of nine calibration groups are modular arithmetic; crossover locations predicted from measured parameters match observations with . A preregistered “no crossover” prediction on permutation composition is confirmed by scanning. The authors also give the sample size needed to detect a crossover and retract crossovers that cannot be resolved at that sample size.
Critical strengths
-
The paper separates three questions: whether a redundant procedure can beat a fragile one, at what noise level crossover occurs, and whether existing sample size is enough to observe crossover. All three quantities come from direct measurement on the model; predictions do not rely on post hoc fitting of crossover points.
-
Older theory assumed a state-once procedure cannot recover from errors, so redundancy is almost always better in the end. This paper shows that recovery rate is usually not zero and changes whether redundancy is worthwhile.
Critical weaknesses
-
Eight of nine calibration groups are modular arithmetic; five of six full scans are run first and then checked with the new formula. The only preregistered case is a task where crossover does not occur. Positive cases across tasks do not yield resolvable reversals, so evidence is still weak on whether the law still locates crossover when the fragile procedure itself corrects many errors.
-
The necessary sample sizes given do not guarantee that a crossover can actually be observed, and a model of independent per-step errors can overstate gains from redundancy. Most other task families received only small-scale screening. Open-ended reasoning without stepwise verifiers remains outside the scope claimed in the paper.
Questions for authors
-
The stack family remains unresolvable at while the necessary sample size is about 114. Is the margin too thin, are errors correlated, or does flat’s re-derivation not act on load-bearing state?
-
In code execution, kinship, or GSM, can you add a full scan where is clearly greater than 0 and the predicted crossover falls in a scannable range? This directly affects whether I treat the law as holding across tasks.
-
Activation noise and sampling noise other than semantic substitution show no significant flat advantage. Does the existence condition apply only to noise of the form “replace a committed intermediate value”?
2 comments
Follow-up to Reviewer 5oqZ
Thank you again for your review. The revision adds the full scans you asked for in Q2 where they can succeed, namely six pre-registered cells with between 0.15 and 0.48 across five models, all resolved and all predicted within their measured intervals, and a full GSM sweep in which the predicted absence of a crossover held at every noise level. It also explains the unresolved stack cell of Q1 through measured interference between adjacent corruptions, after which the cell resolves at the predicted location. Reviewer mGGV found an error in one of our new experiments, and a recheck found one more. Both are now reported as failed predictions, so that of fourteen predictions recorded before their sweeps twelve held, as described in our reply to that reviewer. A single new table lists every condition with its prediction and outcome. If your questions are now answered, we would be grateful if you would consider updating your assessment.
Response to Reviewer 5oqZ
We thank the reviewer for an accurate summary and for questions that each shaped a new experiment.
W1. Arithmetic concentration, retrodiction and a single pre-registered null. We agree that the submission did not test location where the brittle procedure itself corrects many errors, and the revision adds that test, run prospectively (general response and Table 1). In six pre-registered cells with cs between 0.15 and 0.48, across five models and two task families, every predicted crossover lies inside the measured interval (mean absolute error 0.014), and the cs=0 prediction lies inside none (mean absolute error 0.106). We also note that the pre-registered permutation null is more than an existence test. Fed the measured parameters, the cs=0 form predicts a crossover at p∗≈0.25, inside the swept range, so this was the cell where the two forms disagree most sharply, and the data side with the generalized law. GSM is a second such case (Q2).
W2. Necessity of (R5), independence, small screens and scope. We now calibrate (R5) prospectively rather than only stating that it is necessary. All eight verdicts recorded before the prospective sweeps were correct, and no cell resolved with fewer samples than the bound requires. Subsampling the resolved cells shows that a reversal resolves with probability 0.12 at N=Nreq and 0.84 at 2Nreq, and the multiple needed for probability 0.8 lies between 1.84 and 2.25 across six cells, so collecting twice the bound is a measured planning rule. The independence assumption is tested directly in the new Section 4.4 (see Q1). The code and kinship rows of Table 2 are re-measured at N=200 per schema, and GSM is swept in full. Open-ended reasoning remains outside the scope, since the recovery probe needs an exact step verifier, and we state this as the main limitation.
Q1. Why the stack did not resolve at N=600. Of the three explanations, the data point to the third, in a specific form. Correlated errors are ruled out, since the error-process screen classifies the stack as independent, and the margin is not unusually thin (M=0.200). Full-curve predictions from single-corruption parameters hold up to one corruption per trace and then increasingly overpredict flat (0.072 observed against 0.209 predicted at p=0.5). A new probe injects two corruptions at controlled positions. When they are separated, the flat schema's joint recovery equals the product of its single recoveries (R2/P2=0.97 [0.91,1.04]), and when they are adjacent it halves (0.49 [0.44,0.55]). The flat stack schema rebuilds each state from its predecessor, so a corrupted predecessor leaves nothing correct to rebuild from. With this interference factor measured and included, the largest flat advantage is overestimated by a factor of 1.3 rather than 3.3, and the (R5) bound rises from 114 to 752, which explains why 600 problems per noise level were too few. The pre-registered retest at N=2400 resolves at 0.171 [0.144,0.202], containing the predicted 0.156 (Section 4.4 and Appendix E).
Q2. A full scan in code execution, kinship or GSM. We ran this experiment where the measured parameters say it can succeed, and we report what they say for the three named families. GSM, re-measured on all 800 eligible problems, fails the existence condition (cf=0.618<κcs=1.349×0.584=0.788). The law therefore predicts no crossover at any noise level, while the cs=0 form predicts one at p∗=0.534. Swept to p=0.9, the sharp schema is significantly better at all ten noise levels, including the four beyond 0.534. Code execution satisfies the condition, but its predicted crossover (0.542) lies beyond the swept range, its errors are correlated (overdispersion 4.7×), and only 64 problems admit a verifiable chain. On longer GSM chains (L=6) the condition holds (cf=0.624>κcs=0.599) and a sweep resolves at 0.298 [0.242,0.365] against a predicted 0.284, which is the case the question asks for. Kinship floors on a 7B model, and on Qwen2.5-32B its bound exceeds the 600 problems of a control sweep, which did not resolve, as predicted. Table 1 adds six more such cells.
Q3. Noise other than value substitution. The condition is stated per kernel (Proposition 3), and the revision now tests it beyond substitution (Section 4.6). Under a kernel that substitutes the operation rather than the value, the law predicts p∗=0.096 [0.081,0.114] and the sweep resolves at 0.102 [0.082,0.126]. Under sampling noise we measured cs=0.742 and cf=0.812, so the condition holds, but the model absorbs most resampled steps, and the effective error reaches only 0.041 per unit of the noise knob, an order of magnitude below the budget q∗=0.43. Sampling noise therefore cannot reach the budget, as the measured parameters predict.