acceptodds
Under review as a conference paper at ICLR 2027

When Does Redundant Reasoning Pay? A Measurable Condition for Error-Tolerant Reasoning in Language Models

Abstract

Language models are often made more reliable by having them re-derive their own intermediate results. This redundancy can repair a corrupted step, but it also costs accuracy when nothing goes wrong, so it sometimes helps and sometimes hurts, and it is unclear in advance which. We model a reasoning chain as a sequence of noisy steps in which both a brittle procedure (sharp, which states each result once) and a redundant one (flat, which re-derives the previous result before using it) can repair a corrupted step. The model has three parameters, each measured directly on the language model: the two procedures' per-step recovery rates and , found by injecting errors into correct reasoning traces, and the per-step accuracy cost of redundancy . We prove that the redundant procedure can overtake the brittle one if and only if , and give in closed form the per-step error rate beyond which it does. The usual assumption that a brittle chain never recovers () makes this condition trivially true, yet on six of our seven task families the brittle procedure repairs 10% to 63% of corrupted steps unaided. On nine settings spanning three models and two chain lengths, eight of them modular arithmetic, predictions from the measured parameters alone, with nothing fitted to the crossover, match the observed crossover with (mean absolute error ). The condition also agrees with all six full noise sweeps, including a pre-registered prediction of no crossover that held. In practice this is a check to run before spending compute on redundancy: a few short measurements say whether it can pay off at all, a 64-problem screen flags tasks where it cannot because the model's errors recur at every step, and the same parameters bound the sample size needed to detect the gain. Code: https://anonymous.4open.science/r/reasoning-noise-budget-DF74.

Primary area:
learning theory
Keywords:
reasoning chains, re-derivation, redundancy, noisy step-channel, error correction, recovery rate, crossover condition, sample size, arithmetic reasoning, stack machine, pre-registered null, comparative statics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5oqZOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when redundant reasoning is more accurate than stating each intermediate result only once. The authors model a reasoning chain as LL noisy load-bearing steps: sharp writes each step’s result once; flat re-derives before using the previous step’s result and adopts the re-derived value on conflict. All three parameters are measured directly on the model: the stepwise accuracy cost κ\kappa under no noise, and the probabilities csc_s and cfc_f that the two procedures repair injected errors. The paper proves that as per-step error rate rises, flat can exceed sharp if and only if cf>κcsc_f>\kappa c_s, and gives a closed-form solution for the crossover location. This condition reduces to the old formula when cs=0c_s=0.

Experiments cover modular arithmetic, stacks, automata, permutations, code execution, kinship relations, and GSM linear chains. Eight of nine calibration groups are modular arithmetic; crossover locations predicted from measured parameters match observations with R2=0.839R^2=0.839. A preregistered “no crossover” prediction on permutation composition is confirmed by scanning. The authors also give the sample size needed to detect a crossover and retract crossovers that cannot be resolved at that sample size.

Critical strengths
  1. The paper separates three questions: whether a redundant procedure can beat a fragile one, at what noise level crossover occurs, and whether existing sample size is enough to observe crossover. All three quantities come from direct measurement on the model; predictions do not rely on post hoc fitting of crossover points.

  2. Older theory assumed a state-once procedure cannot recover from errors, so redundancy is almost always better in the end. This paper shows that recovery rate is usually not zero and changes whether redundancy is worthwhile.

Critical weaknesses
  1. Eight of nine calibration groups are modular arithmetic; five of six full scans are run first and then checked with the new formula. The only preregistered case is a task where crossover does not occur. Positive cases across tasks do not yield resolvable reversals, so evidence is still weak on whether the law still locates crossover when the fragile procedure itself corrects many errors.

  2. The necessary sample sizes given do not guarantee that a crossover can actually be observed, and a model of independent per-step errors can overstate gains from redundancy. Most other task families received only small-scale screening. Open-ended reasoning without stepwise verifiers remains outside the scope claimed in the paper.

Questions for authors
  1. The stack family remains unresolvable at N=600N=600 while the necessary sample size is about 114. Is the margin too thin, are errors correlated, or does flat’s re-derivation not act on load-bearing state?

  2. In code execution, kinship, or GSM, can you add a full scan where csc_s is clearly greater than 0 and the predicted crossover falls in a scannable range? This directly affects whether I treat the law as holding across tasks.

  3. Activation noise and sampling noise other than semantic substitution show no significant flat advantage. Does the existence condition apply only to noise of the form “replace a committed intermediate value”?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5oqZ

We thank the reviewer for an accurate summary and for questions that each shaped a new experiment.

W1. Arithmetic concentration, retrodiction and a single pre-registered null. We agree that the submission did not test location where the brittle procedure itself corrects many errors, and the revision adds that test, run prospectively (general response and Table 1). In six pre-registered cells with csc_s between 0.150.15 and 0.480.48, across five models and two task families, every predicted crossover lies inside the measured interval (mean absolute error 0.0140.014), and the cs=0c_s{=}0 prediction lies inside none (mean absolute error 0.1060.106). We also note that the pre-registered permutation null is more than an existence test. Fed the measured parameters, the cs=0c_s{=}0 form predicts a crossover at p∗≈0.25p^* \approx 0.25, inside the swept range, so this was the cell where the two forms disagree most sharply, and the data side with the generalized law. GSM is a second such case (Q2).

W2. Necessity of (R5), independence, small screens and scope. We now calibrate (R5) prospectively rather than only stating that it is necessary. All eight verdicts recorded before the prospective sweeps were correct, and no cell resolved with fewer samples than the bound requires. Subsampling the resolved cells shows that a reversal resolves with probability 0.120.12 at N=NreqN = N_{\mathrm{req}} and 0.840.84 at 2Nreq2N_{\mathrm{req}}, and the multiple needed for probability 0.80.8 lies between 1.841.84 and 2.252.25 across six cells, so collecting twice the bound is a measured planning rule. The independence assumption is tested directly in the new Section 4.4 (see Q1). The code and kinship rows of Table 2 are re-measured at N=200N = 200 per schema, and GSM is swept in full. Open-ended reasoning remains outside the scope, since the recovery probe needs an exact step verifier, and we state this as the main limitation.

Q1. Why the stack did not resolve at N=600N = 600. Of the three explanations, the data point to the third, in a specific form. Correlated errors are ruled out, since the error-process screen classifies the stack as independent, and the margin is not unusually thin (M=0.200M = 0.200). Full-curve predictions from single-corruption parameters hold up to one corruption per trace and then increasingly overpredict flat (0.0720.072 observed against 0.2090.209 predicted at p=0.5p = 0.5). A new probe injects two corruptions at controlled positions. When they are separated, the flat schema's joint recovery equals the product of its single recoveries (R2/P2=0.97R_2/P_2 = 0.97 [0.91,1.04][0.91, 1.04]), and when they are adjacent it halves (0.490.49 [0.44,0.55][0.44, 0.55]). The flat stack schema rebuilds each state from its predecessor, so a corrupted predecessor leaves nothing correct to rebuild from. With this interference factor measured and included, the largest flat advantage is overestimated by a factor of 1.31.3 rather than 3.33.3, and the (R5) bound rises from 114114 to 752752, which explains why 600600 problems per noise level were too few. The pre-registered retest at N=2400N = 2400 resolves at 0.1710.171 [0.144,0.202][0.144, 0.202], containing the predicted 0.1560.156 (Section 4.4 and Appendix E).

Q2. A full scan in code execution, kinship or GSM. We ran this experiment where the measured parameters say it can succeed, and we report what they say for the three named families. GSM, re-measured on all 800800 eligible problems, fails the existence condition (cf=0.618<κcs=1.349×0.584=0.788c_f = 0.618 < \kappa c_s = 1.349 \times 0.584 = 0.788). The law therefore predicts no crossover at any noise level, while the cs=0c_s{=}0 form predicts one at p∗=0.534p^* = 0.534. Swept to p=0.9p = 0.9, the sharp schema is significantly better at all ten noise levels, including the four beyond 0.5340.534. Code execution satisfies the condition, but its predicted crossover (0.5420.542) lies beyond the swept range, its errors are correlated (overdispersion 4.7×4.7\times), and only 6464 problems admit a verifiable chain. On longer GSM chains (L=6L = 6) the condition holds (cf=0.624>κcs=0.599c_f = 0.624 > \kappa c_s = 0.599) and a sweep resolves at 0.2980.298 [0.242,0.365][0.242, 0.365] against a predicted 0.2840.284, which is the case the question asks for. Kinship floors on a 7B model, and on Qwen2.5-32B its bound exceeds the 600600 problems of a control sweep, which did not resolve, as predicted. Table 1 adds six more such cells.

Q3. Noise other than value substitution. The condition is stated per kernel (Proposition 3), and the revision now tests it beyond substitution (Section 4.6). Under a kernel that substitutes the operation rather than the value, the law predicts p∗=0.096p^* = 0.096 [0.081,0.114][0.081, 0.114] and the sweep resolves at 0.1020.102 [0.082,0.126][0.082, 0.126]. Under sampling noise we measured cs=0.742c_s = 0.742 and cf=0.812c_f = 0.812, so the condition holds, but the model absorbs most resampled steps, and the effective error reaches only 0.0410.041 per unit of the noise knob, an order of magnitude below the budget q∗=0.43q^* = 0.43. Sampling noise therefore cannot reach the budget, as the measured parameters predict.

AuthorsOpenReview ↗

Follow-up to Reviewer 5oqZ

Thank you again for your review. The revision adds the full scans you asked for in Q2 where they can succeed, namely six pre-registered cells with csc_s between 0.15 and 0.48 across five models, all resolved and all predicted within their measured intervals, and a full GSM sweep in which the predicted absence of a crossover held at every noise level. It also explains the unresolved stack cell of Q1 through measured interference between adjacent corruptions, after which the cell resolves at the predicted location. Reviewer mGGV found an error in one of our new experiments, and a recheck found one more. Both are now reported as failed predictions, so that of fourteen predictions recorded before their sweeps twelve held, as described in our reply to that reviewer. A single new table lists every condition with its prediction and outcome. If your questions are now answered, we would be grateful if you would consider updating your assessment.

Reviewer_HkTwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when redundant reasoning can offset its cost in clean accuracy. It incorporates the recovery capabilities of both reasoning procedures into a per-step noise model, derives the crossover condition and location, and independently measures recovery through error-injection probes. Experiments vary the reasoning procedure, model, chain length, and task family to evaluate agreement between predictions and observations, while also examining whether crossovers are observable with finite samples.

Critical strengths

The paper provides a testable quantitative framework for the trade-off between the accuracy cost of redundancy and its recovery benefits. Predicting crossover locations from independently measured parameters is a valuable experimental design.

Critical weaknesses

1.The compositionality of recovery remains insufficiently validated. Recovery rates measured under single corruptions are used in a multi-step multiplicative model, although multiple repairs may share state or interfere with one another. The zero-injection error screen places limited constraints on these dependencies, leaving the use of single-corruption measurements to predict behavior under multiple corruptions without direct validation.

2.Validation of the generalized criterion remains limited in its most informative regimes. Of the nine distinct conditions used to calibrate crossover locations in Figure 1(c), eight are arithmetic. The stack task provides supporting evidence, although a significant reversal remains unresolved. The GSM case, where stronger recovery is insufficient to offset the accuracy cost, has undergone parameter screening but lacks a full noise sweep.

3.Inconsistent figure quality is a minor presentation issue. Figure 1 juxtaposes a low-resolution raster schematic with vector plots, creating noticeable differences in text and line clarity that affect readability and visual consistency.

Questions for authors

Do existing experiments with multiple corruptions support the recovery compositionality assumption and explain the overestimated gains on the stack task?

4 comments
AuthorsOpenReview ↗

Response to Reviewer HkTw

We thank the reviewer for a precise reading and in particular for the compositionality question, which led to the most informative new result in the revision.

W1. Compositionality of recovery. The reviewer is right that the submission never tested composition, and that the zero-injection screen constrains errors but not repairs. The revision tests composition in two ways (Section 4.4). First, at high noise a trace carries several corruptions, so predicting the whole accuracy curve from single-corruption parameters is itself a test. On arithmetic (three depths, three models), permutation and the automaton, the predicted curves stay within 0.0460.046 of the measured ones at every noise level above the accuracy floor, and the residual shows no trend with the number of corruptions per trace, up to about four. Second, a new probe injects two or three load-bearing corruptions at controlled positions into clean traces and compares the joint recovery RkR_k with the product PkP_k of single recoveries at the same positions, over 10001000 instances per condition. Its single-corruption arm reproduces the paper's csc_s and cfc_f within their intervals. On arithmetic, Rk/PkR_k/P_k lies between 0.990.99 and 1.021.02 for two and three corruptions, adjacent or separated, and on permutation it is 1.031.03. On the stack, recovery composes when corruptions are separated (0.970.97 [0.91,1.04][0.91, 1.04]) but halves when they are adjacent (0.490.49 [0.44,0.55][0.44, 0.55]), and three adjacent corruptions give 0.230.23, close to the square of the pairwise factor. Repairs therefore do share state on the stack, exactly as the reviewer anticipated, and only there. The flat stack schema rebuilds each state from its predecessor, so a corrupted predecessor defeats the repair. We model this with one measured interference factor per schema, through a two-state transfer matrix (Appendix E). The extended model reduces to the original one when the factor is 11 and departs from it only at second order in qq, since interference needs two consecutive corrupted steps. Applied to the original stack sweep, which makes it a retrodiction, it lowers the flat-curve RMSE from 0.0780.078 to 0.0220.022.

W2. Arithmetic concentration, the unresolved stack and the unswept GSM family. We agree, and the revision addresses this directly (general response). Six pre-registered cells with csc_s between 0.150.15 and 0.480.48, across five models and two task families, all resolve, and every generalized prediction lies inside the measured interval while the cs=0c_s{=}0 prediction lies inside none. The stack cell resolves on fresh seeds at N=2400N = 2400, at p∗=0.171p^* = 0.171 [0.144,0.202][0.144, 0.202] against the predicted 0.1560.156. GSM is now swept in full on all 800800 eligible problems. It fails the existence condition (cf=0.618<κcs=0.788c_f = 0.618 < \kappa c_s = 0.788), and the sharp schema is significantly better at all ten noise levels up to p=0.9p = 0.9, although the cs=0c_s{=}0 form predicts a crossover at 0.5340.534. Like permutation, this cell therefore discriminates between the two forms in favor of the generalized law. At L=6L = 6 the re-measured GSM parameters satisfy the condition (cf=0.624>κcs=0.599c_f = 0.624 > \kappa c_s = 0.599), and a sweep resolves at 0.2980.298 [0.242,0.365][0.242, 0.365] against a predicted 0.2840.284, so existence switches on with depth as the parameters measured at that depth say.

W3. Figure 1 quality. We agree. Panel (a) is now vector art in the same typeface, sizes and colours as panels (b) and (c), so the whole figure is equally sharp at any zoom.

Question. Do multi-corruption experiments support compositionality and explain the stack overestimate? Yes on both counts. Recovery composes on arithmetic, permutation and the automaton, both in the full curves and in the controlled multi-corruption probe. On the stack it fails only for adjacent corruptions, and including the measured interference factor reduces the overestimate of the largest flat advantage from a factor of 3.33.3 to 1.31.3. It also raises the (R5) bound from 114114 to 752752, which accounts for the failure to resolve at N=600N = 600 and the success of the pre-registered retest at N=2400N = 2400.

Thank you for the detailed response and additional experiments. Most of my original concerns have been substantially addressed. However, I still have reservations about the overall contribution and the numerical inconsistency raised by Reviewer mGGV. Therefore, I have decided to maintain my current rating.

Thank you for taking the time to check the revision and follow up. We appreciate your feedback on the experiments.

AuthorsOpenReview ↗

Follow-up to Reviewer HkTw

Thank you for the follow-up and for confirming that most of your concerns are addressed. Reviewer mGGV's numerical point is correct. The printed prediction for the third sharp-schema variant does not follow from its printed parameters under Eq. (6), and a recheck of every prediction in the paper found the same problem in the GSM L=6L = 6 cell. Both are now reported in the revised PDF as failed predictions, and our reply to Reviewer mGGV gives the full parameters. Every other prediction reproduces from its printed parameters, including all six pre-registered cells, the two pre-registered nulls, the composition results that answer your compositionality question, and the resolvability calibration. A new Table 1 lists every condition with its parameters, prediction, outcome and timing, so each number can be checked directly against Eq. (6).

On the overall contribution, the paper gives a condition, cf>κcsc_f > \kappa c_s, that can be checked with two short probes before any noise sweep and that says whether adding redundancy to a reasoning procedure can pay at all. The usual idealization that the brittle procedure never recovers always answers yes, yet on six of our seven families the brittle procedure repairs 10 to 58 percent of corrupted steps. Of the fourteen predictions recorded before their sweeps, twelve held. These include all six located crossovers, which the idealization missed in every case, both families where the condition fails and the redundant procedure never overtook, and the sample-size bound, which was calibrated prospectively into a usable planning rule. We hope this clarifies the contribution and would be grateful if you would reconsider your assessment in light of the corrected revision.

Reviewer_mGGVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies when redundant reasoning can outperform standard reasoning under step-level noise. The authors derive a theoretical condition for the crossover and evaluate its predictive accuracy primarily on synthetic arithmetic tasks.

Critical strengths

The paper investigates an interesting question of when redundant verification can benefit LLM reasoning under step-level noise.

Critical weaknesses
  1. This paper is extremely difficult to follow and suffers from serious presentation issues. The authors repeatedly refer to an “earlier version” or “unpublished version,” making the manuscript read more like a revision history than a scientific paper. Section 4.4, titled “Negative Results and Withdrawn Claims,” is particularly confusing. The overall organization and presentation fall well below the standard expected of a conference submission.
  2. On arithmetic tasks, the generalized and earlier formulations differ in their predicted crossover thresholds by only 0.0013 at L=8 and 0.0016 at L=12, at least 38 times smaller than the empirical confidence interval widths. These experiments therefore cannot meaningfully distinguish the two formulations. The stack-machine experiment, where the two formulations make substantially different quantitative predictions, fails to exhibit a statistically resolved crossover. Moreover, five of the six full noise sweeps are retrospective, and the resolvability bound was derived after certain experiments failed. While retrospective validation is not inherently problematic, it provides limited evidence for the prospective predictive capability emphasized as a central contribution of the paper.
Questions for authors
  1. How can the authors substantiate the claimed predictive accuracy of the crossover threshold, given that the generalized and earlier formulations make nearly identical predictions on arithmetic tasks, while the stack-machine crossover remains statistically unresolved?
  2. Can the authors validate the retrospectively derived resolvability bound on new, prospectively designed experiments?
4 comments
AuthorsOpenReview ↗

Response to Reviewer mGGV

We thank the reviewer for a direct review. Both weaknesses are fair, and we have changed the paper and the evidence in response.

W1. Presentation and the revision-history framing. We agree, and we have rewritten the paper accordingly. The revision contains no reference to any earlier or unpublished version of the work. The formula with cs=0c_s{=}0 is introduced as what it is, the special case in which the brittle procedure is assumed never to recover, and the generalized law is presented directly. The former Section 4.4 is replaced by Section 4.6, "Predicted nulls and the resolvability bound", which presents the cells where the law predicts no crossover, or no observable one, and tests those predictions. The stricter criterion for calling a reversal real is described once, in Section 3, as methodology, and the two point-estimate crossings it rejects are documented in Appendix D. The results section now follows four lines of evidence in order, namely retrospective calibration, prospective tests, composition and predicted nulls.

W2. Arithmetic cannot discriminate, the stack was unresolved, and five of six sweeps were retrospective. Each of these statements is correct, and we have changed both the evidence and how it is presented. The submission conflated two claims that different cells test. The arithmetic calibration tests whether the budget can be predicted from independently measured tax and recovery without fitting, and, as the reviewer says, it cannot test the csc_s generalization. The revision says so explicitly (Section 4.1). The generalization is tested where csc_s is far from zero, and we now test it prospectively. In six pre-registered cells with csc_s between 0.150.15 and 0.480.48, every generalized prediction lies inside the measured interval (mean absolute error 0.0140.014), and the cs=0c_s{=}0 prediction lies inside none (mean absolute error 0.1060.106). The selection rule, predictions and falsifiers were all recorded before any sweep. The stack cell resolves on fresh seeds at N=2400N = 2400, at 0.1710.171 [0.144,0.202][0.144, 0.202], which contains the generalized 0.1560.156 and excludes the cs=0c_s{=}0 form's 0.0470.047. We also test by construction the one comparative static that is new to the generalized law. Three sharp schemas that differ only in how much state they restate move csc_s from 0.2840.284 through 0.4540.454 to 0.6120.612, and the measured crossover rises from 0.1040.104 to 0.1710.171 to 0.2420.242, each inside the interval of its prediction (0.0930.093, 0.1560.156 and 0.2350.235), while the tax ratio falls across the series.

Q1. How the predictive accuracy is substantiated. By evidence that does not depend on arithmetic. First, the six prospective cells above. Second, two pre-registered nulls where the forms disagree sharply. On permutation the cs=0c_s{=}0 form predicts a crossover at p∗≈0.25p^* \approx 0.25 and on GSM at 0.5340.534, both inside the swept range, while the generalized condition fails on both. The sharp schema is significantly better at seven of nine noise levels on permutation and at all ten on GSM, and flat is never significantly better. We report the nine retrospective, arithmetic-dominated cells separately as a test of the location law, not of the generalization.

Q2. Prospective validation of the resolvability bound. Yes. Before the new sweeps we recorded, for each cell, the sample size (R5) requires and a verdict on whether the reversal would resolve at the sample size we would collect. Two of these cells were chosen to fall below the bound and were predicted not to resolve. All eight verdicts were correct, and no cell resolved with fewer samples than the bound requires. Subsampling the resolved cells calibrates the bound rather than only stating that it is necessary. A reversal resolves with probability 0.120.12 at N=NreqN = N_{\mathrm{req}} and 0.840.84 at 2Nreq2N_{\mathrm{req}}, and the multiple at which it first resolves with probability 0.80.8 lies between 1.841.84 and 2.252.25 across six cells on three task structures and three models. With the interference correction of Section 4.4, the stack cell, which the uncorrected bound misjudged, falls in the same range.

Thank you for the detailed response and additional experiments. However, the paper remains difficult to follow. The Introduction presents multiple technical results before clearly defining the key variables, and the evidence is spread across Tables 1 to 3, making it hard to tell which experiments were conducted before the theory was developed and which were designed to test it afterward. A shorter Introduction and a unified table showing all experimental conditions and when the predictions were made would help. I also have a concern about the new comparative static experiment in Section 4.2. Using the reported parameters (cf=0.640c_f = 0.640 for the fixed flat schema, cs=0.612c_s = 0.612, κ=1.0224\kappa = 1.0224), Eq. (6) gives an effective crossover threshold of approximately 0.61. Since the noise-knob threshold is obtained by dividing the effective threshold by a probability that cannot exceed 1, the predicted threshold should be at least 0.61, rather than the reported 0.235. Could the authors clarify the parameters used and report the flat recovery rate, sharp recovery rate, tax ratio, and effective error probability for each of the three sharp-schema variants? This inconsistency raises concerns about a key experiment in the revised paper, so I maintain my current rating.

AuthorsOpenReview ↗

Thank you for your follow-up

Thank you for taking the time to read our rebuttal and revision, and for your thoughtful follow-up feedback throughout the review process.

AuthorsOpenReview ↗

Follow-up to Reviewer mGGV

Thank you for checking this so carefully. You are right, and we apologize for the error. With the parameters we reported for the third variant (cs=0.612c_s = 0.612, cf=0.640c_f = 0.640, κ=1.0224\kappa = 1.0224), Eq. (6) gives q∗=0.0224/0.0367=0.61q^* = 0.0224 / 0.0367 = 0.61. Since the knob threshold divides this by a probability no larger than one, the prediction is at least 0.610.61, beyond the swept range. The printed value of 0.2350.235 is wrong, and the agreement we claimed for that variant does not stand.

The parameters for the three variants are below. The flat schema is the same in all three, and every prediction uses the sharp schema's measured effective error probability on this task, qeff=0.82q_{\mathrm{eff}} = 0.82.

sharp variantcsc_scfc_fκ\kappaqeffq_{\mathrm{eff}}q∗q^* from Eq. (6)predicted p∗p^*measured p∗p^*
writes the new state only0.2840.6401.02870.820.0760.0930.104 [0.082, 0.128]
standard0.4540.6401.02560.820.1280.1560.171 [0.144, 0.202]
also restates the operands0.6120.6401.02240.820.6100.74, beyond the grid0.242 [0.208, 0.281]

The first two variants agree with the law, with each prediction inside its measured interval and the two intervals separated. For the third, the law predicts no crossover inside the swept range, and its net recovery cf−κcs=0.014c_f - \kappa c_s = 0.014 would make any reversal very hard to resolve, so the measured crossover at 0.2420.242 contradicts the prediction. The revised PDF now reports this variant as a failed prediction that we cannot explain, and the comparative-static claim rests on the first two variants.

After your comment we rechecked every prediction in the paper against Eq. (6). We found the same problem in one other cell, GSM at L=6L = 6, where the reported parameters (κ=1.104\kappa = 1.104, cs=0.542c_s = 0.542, cf=0.624c_f = 0.624) give q∗=0.80q^* = 0.80 and so cannot produce the printed prediction of 0.2840.284. That cell is now also reported as a failed prediction. Every other prediction reproduces from its printed parameters, including the six pre-registered cells, the permutation and GSM L=2L = 2 nulls, the two new noise kernels and the resolvability calibration. Of the fourteen predictions recorded before their sweeps, twelve held and two failed, and the paper now says so.

On presentation, we followed both suggestions. The Introduction is shorter, with three positioning paragraphs moved to the related-work appendix, and it now defines qq, κ\kappa, csc_s and cfc_f before stating any result. The former Tables 1 and 2 are merged into a single Table 1 that lists every experimental condition with its measured parameters, its prediction and its outcome, grouped by whether the prediction was fixed after the sweep, as a consistency check, or recorded before it. The two failed predictions are marked in that table. We thank the reviewer for both comments, which made the paper more accurate and easier to follow.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the three reviewers for careful and constructive reviews. All three raise the same central concern. The calibration evidence sat in iterated arithmetic, where the generalized law and its cs=0c_s{=}0 special case predict almost the same budget, the one family where the two differ (the stack machine) did not resolve, and five of the six sweeps were retrodictions. We agree that this was the right objection, and the revision answers it with new prospective experiments rather than with argument. All new material is in the revised PDF.

Prospective tests where the brittle procedure recovers (Section 4.2, Table 1). We ran the procedure the paper proposes forward. Eleven candidate cells were screened on fresh seeds, and a cell was swept only if it met a selection rule fixed before screening. The rule required cs≥0.10c_s \ge 0.10, existence at the interval bounds of the measured parameters, a predicted crossover at or below p=0.35p = 0.35, a separation of at least 0.050.05 between the two forms, and a feasible sample size under (R5). Six cells qualified, across five models (Qwen2.5-7B and 14B, Llama-3.1-8B, Mistral-7B and Phi-4) and two task families (stack machine and register automaton), with csc_s between 0.150.15 and 0.480.48. Predictions, resolvability verdicts and falsifiers were recorded before any sweep was run. All six cells resolve a sustained reversal. Every generalized prediction lies inside the measured 95%95\% interval, with mean absolute error 0.0140.014, while the cs=0c_s{=}0 prediction lies inside none of the six intervals, with mean absolute error 0.1060.106. The generalized prediction is the closer of the two in all six cells (one-sided sign test, p=0.016p = 0.016). The stack cell that did not resolve in the submission resolves on fresh seeds at N=2400N = 2400, at p∗=0.171p^* = 0.171 [0.144,0.202][0.144, 0.202], which contains the generalized prediction 0.1560.156 and excludes the cs=0c_s{=}0 prediction 0.0470.047.

The new comparative static, tested by construction (Section 4.2). Three sharp schemas that differ only in how much of the prior state each line restates move csc_s from 0.2840.284 through 0.4540.454 to 0.6120.612, with the task, model and flat schema fixed. The measured crossover rises from 0.1040.104 to 0.1710.171 to 0.2420.242, each inside the interval of its prediction (0.0930.093, 0.1560.156 and 0.2350.235), with the endpoints separated at 95%95\%. The tax ratio falls across the series, so it is the brittle procedure's own recovery that moves the budget.

Two discriminating nulls (Section 4.6). On permutation tracking the cs=0c_s{=}0 form, fed the measured parameters, predicts a crossover at p∗≈0.25p^* \approx 0.25, inside the swept range, whereas the generalized condition fails. The pre-registered sweep found sharp significantly better at seven of nine noise levels and never significantly worse. GSM chains, now swept in full on all 800800 eligible problems, behave the same way. The cs=0c_s{=}0 form predicts a crossover at p∗=0.534p^* = 0.534, the generalized condition fails, and sharp is significantly better at all ten noise levels up to p=0.9p = 0.9.

Composition of recovery (Section 4.4, Appendix E). Following Reviewer HkTw, we tested whether recovery measured one corruption at a time composes across several corruptions. It does on arithmetic, permutation and the automaton. On the stack it fails only when two corruptions are adjacent, a mechanism we measure and model with one interference factor, which also explains why the stack did not resolve at N=600N = 600.

A calibrated resolvability bound (Section 4.6). All eight resolvability verdicts recorded before the prospective sweeps were correct, and no cell resolved with fewer samples than the bound requires. Subsampling shows that collecting twice the bound resolves a reversal with probability above 0.80.8, with the required multiple between 1.841.84 and 2.252.25 across six cells.

Presentation. Following Reviewer mGGV, the revision no longer refers to any earlier version of the work. The cs=0c_s{=}0 formula is introduced as the special case in which the brittle procedure never recovers, and the former Section 4.4 is replaced by a section on predicted nulls and the resolvability bound. The stricter criterion for calling a reversal real is described once, in Section 3, and the two point-estimate crossings it rejects are documented in Appendix D. Following Reviewer HkTw, Figure 1(a) is now vector art that matches the other two panels.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.