acceptodds
Under review as a conference paper at ICLR 2027

From Local Prediction Error to Final Outputs: A Theory of Expert Sharing in Multi-Step Reasoning

Abstract

Mixture-of-experts (MoE) models can share experts across layers or computational steps, pooling training information from different contexts. This can reduce local prediction error, but multi-step systems are ultimately evaluated by their final outputs. We study when these local gains survive composition. In specified linear regression models, we derive exact sharing and merging criteria by balancing variance savings against bias from sharing across roles with different target behavior. Under a common fixed linear propagation map, we characterize expected squared final-output risk, retaining error directions and correlations. We construct cases where sharing lowers average local error yet increases final-output error. Within this setting, we also identify when orthogonal full pooling with isotropic noise preserves every local improvement at the final output. For general routed computations, we derive conditional end-to-end error bounds that account for distribution shifts, errors in surrounding components, and changes in expert selection. Separately, we show that correct predictions and route labels alone do not establish that individual experts implement the attributed operations. Controlled experiments realize a strict local-to-final risk reversal. In low-data symbolic settings, trained Transformers exhibit statistically supported teacher-forced one-step gains without statistically resolved free-running final-output gains. Separate intervention tests distinguish successful multi-layer actions from single-expert attribution and generalization to unseen transitions. Together, these results distinguish when sharing improves local estimation, when those gains survive to the final output, and what evidence is needed to attribute reusable operations to individual experts.

Primary area:
learning theory
Keywords:
MoE, Reasoning, Reuse, Expert Sharing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_F8RXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formalizes expert sharing across layers or computational roles in multi-step reasoning. It defines role-level event losses, derives exact expected risks for separate fitting, full pooling, partial pooling, and arbitrary partitions, and shows that pooling is beneficial when the variance saved by removing an expert exceeds the functional heterogeneity (sharing bias) between roles. It then composes the fitted role estimates through a fixed linear map and derives an exact terminal-risk expression that depends on the propagation geometry and error covariance. This yields a strict local-to-terminal reversal: a sharing rule can lower average local error while increasing final-output error. For general routed programs, the paper gives conditional error certificates that include implementation mismatch, state propagation, and route changes. Finally, it argues that observational route labels do not identify expert-level operations and proposes intervention-based audits.

Validation consists of controlled Monte Carlo estimation experiments, a three-event linear composition experiment, and small Transformer experiments on modular arithmetic. The controlled experiments match the analytic risks and reproduce the reversal. The Transformers show lower teacher-forced one-step error for tied experts in a low-data sweep, but no statistically resolved free-running final-output improvement; intervention-supervised models achieve high compound-action fidelity, while single-layer attribution and unseen-transition generalization remain limited.

Critical strengths
  1. A precise local-to-final distinction. The paper does more than state that composition can matter: it gives an exact risk formula and a concrete reversal where pooling improves local risk but increases terminal risk. This is consequential because many evaluations of expert sharing report only teacher-forced or per-call metrics.

  2. Careful treatment of attribution. The proposition that correct route labels and natural predictions do not imply causal reuse is important, and the intervention protocol makes the missing evidence explicit. This prevents a common over-interpretation of routing visualizations.

Critical weaknesses
  1. The empirical task is too limited to establish broad practical impact. The learned-model evidence is confined to a small modular-arithmetic task. At one end, the short-horizon models are nearly perfect; at the other, the longer-horizon models are close to failure. This leaves limited evidence about expert sharing in a realistic, non-saturated multi-step reasoning regime.

  2. The main theory-to-practice link remains conditional. The strongest exact claims concern controlled linear propagation, while the general routed-program result is an error certificate under additional assumptions. The paper therefore establishes that local-to-final reversals are possible, but does not yet show how reliably the proposed criteria predict final performance in a trained Transformer.

Questions for authors
  1. Can the authors provide an end-to-end comparison in a non-saturated multi-step setting where both separate and shared models achieve meaningful but imperfect terminal accuracy?

  2. Which assumptions of the general certificate can be checked in the Transformer experiments, and does the certificate make a quantitatively useful prediction of the observed final-output comparison?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer F8RX (1/2): Q1 and W1

Your main concerns are that the trained evidence sat at the two extremes and that the paper did not say how reliably its criteria predict final performance. The revised PDF adds comparisons between chance and saturation (Q1) and reports, level by level, what the theory predicted and what it did not (part 2); W1 states the scope of the claim.

Q1. Can the authors provide an end-to-end comparison in a non-saturated setting where both separate and shared models achieve meaningful but imperfect terminal accuracy?

Answer: yes, at 128 traces. Separate and shared experts reach 45.2% and 47.1% at horizon 8 and, in a new experiment, 80.2% and 83.5% at horizon 12. At 96 traces both are still near chance (3.2%), and at 192 both are near saturation.

HorizonTraining tracesSeparate expertsShared across depthDense shared block
8968.7%7.4%31.0%
812845.2%47.1%91.5%
819297.5%97.0%99.9%
12 (new)9610.0%6.4%86.6%
12 (new)12880.2%83.5%98.3%
12 (new)19299.0%98.9%100.0%

The entries are final-output accuracy with learned routing. Horizon 8 is the paper's ten-seed study (Table 19); a rerun on different hardware gives 8.3 / 7.5 / 31.2%, 49.3 / 51.8 / 92.3% and 98.0 / 96.6 / 99.6%. Horizon 12 has five seeds (66 new runs), with budgets chosen by a rule fixed before the additional seeds were run (§R.9).

  1. Paired evaluation. Both endpoints of every pair are evaluated on the same 32,768 traces. A contrast is "resolved" when its simultaneous 95% interval excludes zero.
  2. Shared versus separate at horizon 8, 128 traces. The final contrast is resolved in 9 of 10 seeds, 6 in favour of sharing and 3 against: with learned routing there is no consistent winner (the means differ by 1.9 pp, the seed standard deviation is about 20 pp). The dense block has no router, so its 46.2 pp gain over separate experts combines sharing with the absence of routing errors; item 5 and part 2 separate the two.
  3. Local versus final. At horizon 8, 51 of the 120 contrasts are resolved at both endpoints, and all 51 have the same sign (13 of them shared versus separate; 47 of 47 in the rerun). At horizon 12: 15 of 15. The other contrasts are unresolved at one endpoint or both, mostly at chance (64 traces) or near saturation (192 traces).
  4. Size. In these models the direction of a resolved local gain is inherited, but its size is not. For the dense block at horizon 8, a one-step gain of 9.3 pp at 128 traces becomes 46.2 pp at the output.
  5. With the router supervised, some sharing is better than none. On a ladder of five sharing strengths (125 new runs, five seeds, §R.6, Table 39), mean final accuracy at 96 traces is 40.1% with no sharing, 54.2% at strength 0.5 and 47.2% with full sharing. At 96 and 128 traces, no sharing is the final-error minimizer in 0 of 10 seeds.

W1. The empirical task is too limited to establish broad practical impact; short-horizon models are nearly perfect and longer-horizon models are close to failure.

Answer. We agree that one family of symbolic tasks cannot establish broad practical impact, and the paper does not claim it: the abstract limits the trained evidence to "low-data symbolic settings", and the revised §8 now states this limit. The contribution is the theory; the experiments test it where its quantities are known. What carries over to an MoE without declared targets is a procedure: select a sharing choice on held-out data (Proposition 11), confirm it with the paired final-output audit (Proposition 45), and check routing before attributing a final-output deficit to sharing. In our models the sign of a resolved one-step gain was always inherited, so the audit mattered for the size of a gain and for unresolved contrasts. Within this setting we added:

  1. Sharing patterns. All five patterns analysed in Theorem 19 are now trained (80 new runs). Of 400 contrasts, 145 are resolved at both endpoints, and all 145 have the same sign.
  2. A second task (exploratory, §R.9). In a product-state task (69 new runs), the roles differ in how strongly their errors reach the output (0.939 against 0.833). Local and final signs agree in all 21 resolved pairs.
  3. Targets that we do not declare (§R.7). On the residuals that trained layers actually write, the same-data plug-in rule of Theorem 1 transfers only partly: it is correct in 76 of 107 cells (46 of 48 at n ≥ 256), but no better than always fitting separately (102 of 107). Held-out selection is the rule we recommend.
  4. Why a symbolic task. The tests need the role, the target operation and the pre-action state of every call, forced experts compared with a declared operator, and exact transition holdouts. The symbolic task provides all of these.

The H12 and H16 results of Table 9 are length-extrapolation checks of models trained at shorter horizons. No claim rests on them, and the revised §G.3 says so.

AuthorsOpenReview ↗

Reply to Reviewer F8RX (2/2): W2, Q2 and summary

W2. The theory-to-practice link remains conditional: the paper shows that reversals are possible, not how reliably the criteria predict final performance in a trained Transformer.

Answer. Agreed, and the answer differs by level. On trained Transformers, held-out one-step selection found the final-error minimizer in 10 of 10 seeds between chance and saturation, and every one-step gain resolved at both endpoints kept its sign. The rules that plug in the paper's own quantities add little: the plug-in rule of Theorem 1 is correct in 76 of 107 cells against 102 for always fitting separately, and selection with the terminal geometry gains 0.29 pp on average, with opposite signs across layers. What the theory adds to plain validation is when a one-step comparison can be trusted (row 1), a constructed case where it fails (row 3) and the term to check first, routing (row 4).

Level of the theoryPredictionTest on trained TransformersOutcome
Uniform propagation (analogue of §N.4, not an application of it; measured one-step error, not B and C)A one-step gain resolved at both endpoints keeps its sign, given uniform propagation and composing step errors, both measured (§R.3)Five sharing patterns; sharing-strength ladder; horizon 12Same sign in 145 of 145, 24 of 24 and 15 of 15 such pairs (of 400, 250 and 60 contrasts)
Validation selection (Proposition 11; router supervised)Held-out one-step selection is near-optimal for the local loss (guaranteed); that it also finds the best final strength is an empirical findingLadder: 5 strengths, 5 budgets, 5 seedsMinimizer in 10 of 10 seeds at 96 and 128 traces; 20 of 25 cells overall
Exact frontier, uneven propagation (Theorem 46, §N.4)For a target constructed from the measured geometry, pooling lowers local risk and raises terminal riskLinear maps with synthetic noise at the four depths of six trained networks (terminal = final hidden state)Reversal in 6 of 6 with routes fixed. Natural rerouting: 3 of 6 (terminal risk falls in the other three); optimum within one grid step in 7 of 18 settings. No two trained models show a reversal
Certificate of Theorem 4, per-call terms (Eqs. 19–20; finite-class terms of Eq. 22 not evaluated)With routes on plan it tracks the final error; otherwise the routing term can decide the comparisonMeasure the per-call terms; supervise the router onlyRatio 0.89 to 1.22 at final error 2–20% (ten runs); a deficit of 23.6 and 44.3 pp becomes an advantage of 21.1 and 4.4 pp

Q2. Which assumptions of the general certificate can be checked in the Transformer experiments, and does it make a quantitatively useful prediction of the final-output comparison?

Answer. As a measured diagnostic, yes when routes follow the plan and no under learned routing; it is not a prediction made before training. Its per-call terms are measured at the model's own states at 160 checkpoints (§R.4).

Assumptions.

  1. Not checkable here. Theorem 4 assumes fixed finite classes, probe laws fixed independently of the fitting data, matched ERMs and Assumption 3. Jointly trained Transformers are not fitted this way (§B.2), so the finite-class terms of Eq. (22) are not evaluated; §R.4 measures the recursion below Eq. (20).
  2. The per-step inequality p_j ≤ a_j + r_j. No violation in any record.
  3. Routing discrepancy r_j of Eq. (20), measured by forcing another expert at the same pre-call input. The nearest substitute does less harm than the farthest in 6 of 6 full-budget checkpoints (Table 12) and in 19 of 24 low-budget ones (Table 13; 2 of 6 at 96 traces; the five exceptions have natural error of at least 85%).

What was measured.

  1. Routes on plan. The certificate is 3.17% against a final error of 3.17% (shared, router supervised) and 8.26% against 7.72% (dense block) at 128 traces. The certificate is then a sum of per-step errors, so this closeness shows that the unrolling loses little. Over the ten runs with final error between 2% and 20% the ratio is 0.89 to 1.22 (1.3 and 1.6 in the 96-trace means, above 1 at chance); the four values below 1 are within 1.7 standard errors. It orders these two models as the final error does in 16 of 16 resolved pairs.
  2. Learned routing. The certificate exceeds 1. The implementation and routing discrepancies are 0.34 and 0.21 per call at 96 traces, and 0.21 and 0.19 at 128. Supervising the router, which also adds route labels as a training signal, turns the deficit of the tied model against the dense block into an advantage.

Summary. The revised PDF adds the comparisons between chance and saturation (§R.2, §R.6, §R.9) and the measured certificate (§R.4), and reports what held on trained Transformers and what did not. The contribution is the theory: exact results, with complete proofs, on when local sharing gains survive composition and on what attribution requires. We hope this lets you reassess the paper.

Reviewer_XPxZOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether expert sharing that reduces local prediction error also reduces final-output error. Its exact results are confined to linear estimators under a common fixed propagation map; in the general routed setting, it provides upper-bound certificates that neither order true risks nor bound answer accuracy. The experiments mainly verify constructed analytic examples and evaluate small Transformers on four externally supplied operations over Z31\mathbb Z_{31}. They do not apply the proposed criteria to an open sharing decision in a trained model, demonstrate a statistically significant local-to-final reversal in learned systems, or evaluate realistic MoE models and benchmarks. Consequently, the paper does not establish that its framework is practically useful for choosing expert-sharing architectures, while the main fixed-map results appear close to standard shrinkage under a changed quadratic loss.

Critical strengths
  1. Within the linear model the risks are exact.
  2. Local estimation, final-output risk, and causal attribution are kept separate, and the appendix measures the third.
Critical weaknesses
  1. The criteria are never applied to an open sharing decision. Same-data plug-in of B^\hat B can still raise risk. No experiment estimates B,CB,C or the terminal coefficients aH,bHa_H,b_H and checks the predicted tied-versus-local ordering on the trained models.
  2. Theorem 4 is a Lipschitz unrolling plus a finite-class deviation bound and the paper states that this certificate does not order true risks or bound answer accuracy, and that the margin diagnostic is not a numerical certificate for it. The exact terminal result stays in the fixed-map case, where both models share GG. Jointly trained routers lie outside it.
  3. The controlled simulation verifies the derived quadratic risk curves, but does not test whether the theory predicts an unknown sharing decision. Moreover, the only strict reversal is constructed; none of the trained-model comparisons establishes statistically supported opposite signs for local and terminal effects.
  4. The main text has only one figure. At 64 traces the one-step gain is 3.49–7.72 pp, while the final-output gap is -0.19 to +0.19 pp, every interval crosses zero, and both models are at chance. No displayed budget reverses sign. The full budget is saturated. Operation tokens are given externally, and there is no language-model, benchmark, or comparison with the cited sharing systems.
Questions for authors
  1. On the Z31\mathbb{Z}_{31} models, at a budget that is neither chance nor saturated, does an estimate of B and C — or validation-selected λ\lambda, with the bounded-loss restriction of Proposition 11 observed — predict the tied-versus-local final-output ordering?
  2. Theorem 46 appears to be a quadratic-risk decomposition under the terminal metric G^\top G, and §N.4 an elementary spectral characterization. Could the authors clarify the novelty relative to prior shrinkage results under general quadratic or matrix loss?
6 comments
AuthorsOpenReview ↗

Reply to Reviewer XPxZ (1/4): W1 and Q1

Your main concern is that the criteria, although exact in the linear model, were never shown to predict an open sharing decision on trained models. The revised PDF now tests this with the analysis proposed in your Q1: held-out selection passes the test, and the plug-in rule transfers only partly. Parts 1 to 4 cover W1 and Q1, W2, W3 and W4, and Q2; "§R.k" is Appendix R of the revised PDF (new text in blue).

W1 and Q1. Are the criteria applied to an open sharing decision on the trained models? At a budget that is neither at chance nor saturated, does validation selection predict the tied-versus-local final-output ordering? Can B, C, a_H and b_H be estimated?

Answer: yes for validation-selected λ; only partly for the plug-in estimate of B and C. At the two budgets between chance and saturation (96 and 128 traces), selection on held-out one-step loss picks the final-error-minimizing sharing strength in 10 of 10 seeds, including an interior optimum at 96 traces, and it agrees with the final ordering in every resolved tied-versus-local case. The same-data plug-in rule agrees with the empirical decision at larger samples but does not improve on always fitting separately (item 3); §4 and §D.4 already caution that the exact frontier does not cover a same-data plug-in.

  1. Sharing-strength ladder (new, 125 runs, §R.6). The strength λ interpolates between layer-local experts (λ = 0) and experts tied across depth (λ = 1). We train five values at five budgets (64 to 384 traces), five seeds each. The router is supervised, so the comparison isolates the sharing decision. Selecting λ on held-out one-step loss is Proposition 11 with a five-element family and a loss in [0, 1], the fraction of wrong steps on 4,096 validation traces. Proposition 11 guarantees the local loss of the selected λ; that this λ also minimizes the final error is an empirical finding.
    • At 96 and 128 traces the selected λ is the final-error minimizer in 10 of 10 seeds. Over all 25 cells, which include budgets at chance (64) and near saturation (192, 384), it is the minimizer in 20, with regret at most 0.104 pp.
    • The decision is open. At 96 traces the mean final error is 59.9% at λ = 0, 45.8% at λ = 0.5 and 52.8% at λ = 1, and selection finds the minimizer in 5 of 5 seeds (λ = 0.5, 1, 0.75, 0.5, 0.5).
  2. Tied versus local, ten seeds, 96 and 128 traces.
One-step contrast resolvedSelection favours tied / localFinal contrast also resolvedFinal ordering agrees
Original record18 of 2011 / 71313 of 13
Replication12 of 207 / 599 of 9

Mean final accuracy of the two models lies between 7% and 52% in these cells. Both endpoints of each pair are evaluated on the same 32,768 traces, with simultaneous 95% intervals. Over all three architecture pairs the agreement is 51 of 51 (replication 47 of 47). Inheritance is what uniform propagation implies, and uniformity was a criterion fixed before the runs (a first error reaches the output with probability at least 0.95 for every operation and step; observed at least 0.965, §R.3). This is the discrete analogue of the common-gain case of §N.4, not an application of that theorem.

  1. B and C (§R.7). A jointly trained network has no declared role targets, so we fitted role-wise linear surrogates to the residuals each trained layer writes and applied the same-data plug-in form of Theorem 1. The rule transfers only partly. Its decision is correct in 76 of 107 resolved cells: 46 of 48 at n ≥ 256, but 5 of 23 at n ≤ 64, where it recommends pooling too often. Separate fitting is empirically better in 102 of the 107 cells, so the rule does not improve on always fitting separately, and in the five cells where pooling wins it recommends separate fitting. After the estimates converge (n = 128 and 192) it is still wrong in 6 of 24 cells, so the linear surrogate is misspecified for these targets. Inside the specified linear model, same-data shrinkage saves 9% to 86% of the separate-fit risk (Table 29). On real targets we recommend held-out selection (item 1), which needs no estimate of B and C.
  2. a_H and b_H (§R.5). Theorem 46 needs one propagation map common to both comparators, so the coefficients cannot be read off two separately trained networks. They can be measured inside one. In six trained Transformers we estimated W_H from the measured Jacobian and inserted linear residual maps at the four depths; the estimates carry synthetic isotropic noise, and the terminal quantity is the final hidden state. With routes held fixed, the frontier predicts the terminal risk to within 10% in 54 of 54 checks, and it predicts which target reverses and which inherits in 6 of 6 checkpoints each. Under natural rerouting the reversal remains in 3 of 6 checkpoints, and the predicted optimum is within one grid step of the measured one in only 7 of 18 settings, which marks the limit of the fixed-map formula (§N.3).

The tables are in §R.2 and §R.6.

AuthorsOpenReview ↗

Reply to Reviewer XPxZ (2/4): W2

W2. Theorem 4 is a Lipschitz unrolling with a finite-class bound. It does not order true risks or bound answer accuracy, and jointly trained routers lie outside the exact result.

Answer. We agree on reviewer's comment that an upper bound alone cannot order true risks, but we do not agree that this is weakness of theory. The concern is that the theorem then says nothing quantitative, but its per-call discrepancies can be measured (the finite-class terms of the bound are not evaluated for these networks), and the revised PDF reports them at 160 checkpoints (§R.4). With routes on plan and small errors their sum is close to the final error. With learned routing it is vacuous, and the measurement points to the routes, which item 5 tests.

  1. Per-step inequality. p_j ≤ a_j + r_j holds in every record, with no violation at 160 checkpoints.
  2. Routes on plan. With the router supervised, the measured routing term is exactly zero in all 40 runs, and the dense block has no router. In this case the certificate reduces to a sum of per-step errors at the model's own states, so its closeness to the final error shows that the unrolling loses little, and no more than that. The certificate Σ_j (a_j + r_j) then equals 3.17% against a final error of 3.17% (tied, 128 traces), 8.26% against 7.72% (dense, 128 traces), and 0.63% against 0.61% and 0.44% against 0.43% at 192 traces. Over the ten runs with final error between 2% and 20%, the ratio lies between 0.89 and 1.22. At 96 traces the mean certificate is 1.3 and 1.6 times the mean final error (Table 37), and at 64 traces, at chance, it exceeds 1: a sum of per-step errors is tight only when errors are small. Four of the ten ratios fall slightly below 1. This is consistent with sampling error: the certificate was estimated from about 2,000 sites per step, and all four lie within 1.7 standard errors of the final error.
  3. Ordering. The certificate orders the tied model (router supervised) against the dense model as the final error does, in 16 of 16 resolved pairs.
  4. Learned routing. The certificate is vacuous (above 1): the implementation and routing discrepancies (a_j and r_j of Eqs. 19–20) are 0.34 and 0.21 per call at 96 traces, and 0.21 and 0.19 at 128. They stay at 0.17 at 192 traces, where the final error is 3.4%: the learned routes do not follow the declared plan, so these terms measure mismatch with the plan, not error.
  5. The prediction that follows, tested (ten seeds, final-output error). This is the informative use of the theorem. If the loss sits in these two terms and not in the statistical effect of tying, then fixing the routes alone should remove it.
Training tracesTied, learned routingTied, router supervisedDense
9692.6% (92.5%)47.8% (45.9%)69.0% (68.8%)
12852.9% (48.2%)4.1% (3.2%)8.5% (7.7%)
1923.0% (3.4%)0.4% (0.6%)0.1% (0.4%)

The first number is the original record, and the number in parentheses is the replication (Table 35). Architecture and parameter count are identical in the two tied columns. With learned routing, the tied model is worse than the dense model by 23.6 and 44.3 pp at 96 and 128 traces, in 10 of 10 seeds each. With the router supervised it is better by 21.1 pp (9 of 10 seeds) and 4.4 pp (8 of 10). In the replication the deficits are 23.7 and 40.5 pp (10 of 10 each), and the advantages are 22.8 pp (9 of 10) and 4.5 pp (7 of 10). Router supervision also adds route labels as a training signal, so the test shows that correcting the routes removes the deficit, not that tying is free of cost in general.

  1. Jointly trained and hard routers. They lie outside the exact fixed-map result, as you say. Theorem 4 is the statement that covers them. The discrepancy r_j changes only the plan at the same current state, and the theorem does not assume that a hard router is Lipschitz (below Eq. 18). The router enters through ρ_j, which is bounded by architecture (Table 31), by margins (Theorems 28 to 30) and by conditional moments under route selection (Proposition 53). Support stability is checked on trained routers: 150,095 perturbation pairs inside the half-margin tube of Lemma 27, with no change of support (Table 14). As the paper states, this is a sampled local diagnostic and not a numerical certificate.
  2. Ordering of true risks and answer accuracy. The certificate does not provide these, as you say. Companion results do: a two-sided interval for the true terminal risk under nonlinear and rerouted execution (Proposition 49 with Theorem 50), and the paired held-out audit that certifies actual improvement (Proposition 45), which every interval in our trained comparisons implements. Eq. (226) bounds answer disagreement by the readout margin and the state error; we did not evaluate it numerically.

Items 1 to 5 are in §R.2 and §R.4 of the revised PDF, and §7 summarises them.

AuthorsOpenReview ↗

Reply to Reviewer XPxZ (3/4): W3 and W4

W3. The simulation verifies the derived curves but does not predict an unknown decision, and the only strict reversal is constructed.

Answer. Proposition 47 is a construction, and no comparison between two trained sharing choices shows opposite local and final signs. The revised PDF adds two results that go beyond verifying derived curves: the frontier is executed with a propagation map we did not design, the measured Jacobian of trained networks, and a decision that was not known in advance is predicted on trained models.

  1. Reversal with the propagation map of trained Transformers (§R.5, Table 38). In six fully trained checkpoints, the propagation map is the network's measured Jacobian. From it the theory predicts one target that reverses and one that inherits. The targets are still chosen by us, and the estimates are the declared coefficients plus synthetic isotropic noise; what is no longer constructed is the propagation map. The experiment thus tests the exact frontier with a learned map, not two trained sharing choices. Composition runs across the four depths of one forward pass, and terminal risk is the error of the final hidden state. Both outcomes are observed in 6 of 6 checkpoints: pooling lowers local risk by 0.003 to 0.006 in both cases, and changes terminal risk by +0.061 to +0.184 for the first target and by −0.041 to −0.142 for the second, each at more than 97 standard errors over 128 paired noise draws. These numbers hold routes fixed, as Theorem 46 assumes. Under natural rerouting the second target is still inherited in 6 of 6, and the reversal remains in 3 of 6.
  2. Unknown decisions (part 1). The final-error-minimizing sharing strength was not known before the 125 ladder runs. Held-out selection finds it in 10 of 10 seeds at 96 and 128 traces, the two budgets between chance and saturation, and in 20 of 25 cells overall.
  3. The construction is generic (§N.4). Whenever G restricted to the contrast space is nonzero and not a scaled isometry, the proof exhibits a target with a strict local gain and a strict terminal loss. In the trained networks the measured anisotropy is 4.7 to 6.1, so this condition is met.
  4. It survives the natural repair. Role weights π = (1/4, 1/2, 1/4), which are proportional to |G|, still give local risk 15/16 < 1 and terminal risk 9 > 6.

W4. One main-text figure. Chance at 64 traces and saturation at the full budget. No displayed budget reverses sign. Operation tokens are external. No language model, benchmark or comparison with the cited sharing systems.

Answer. We add in the revised version: non-saturated budgets, horizon 12, five sharing patterns §7 now reports the trained-model evidence in three new paragraphs. Not added: An LLM benchmark, as stated in General Response, this is far beyond learning theory track paper's experimental requirement. The core contribution and criteria should be theoretical instead of experimental.

  1. Between chance and saturation. At horizon 8 the ten-seed study spans final accuracy from 3% to 99.9%. At horizon 12 (new, 66 runs) the accuracies of local, tied and dense models are 10.0%, 6.4% and 86.6% at 96 traces, and 80.2%, 83.5% and 98.3% at 128.
  2. Five sharing patterns. The five patterns that Theorem 19 and Table 30 analyse are now all trained in our task (80 new runs): layer-local experts, tying across depth, one shared block per depth (no router), additive depth conditioning and one global pool. In 145 of 145 resolved pairs the local and the final contrast have the same sign. At 64 traces every shared pattern has a lower one-step error than the layer-local one (0.65 to 0.71 against 0.77), and by 192 traces, where all one-step errors approach zero, the five patterns differ by less than 0.7 pp, consistent with the 1/n variance term of Theorem 1.
  3. Sign agreement. Local and final signs agree in every resolved pair, also in the new families (24 of 24 on the ladder, 15 of 15 at horizon 12). This is what uniform propagation implies: a first wrong state leads to a wrong final state with probability at least 0.965 for every operation and every step, the discrete analogue of the common-gain case of §N.4.
  4. A second task (new, exploratory, 69 runs, §R.9). In a product-state task, errors of two roles reach the output with probability 0.939 and errors of the other two with 0.833; local and final signs agree in all 21 resolved pairs.
  5. External operation tokens. They make the role, the target operator and the pre-action state known, which is what allows forced-action fidelity (Proposition 5) and exact transition holdouts (§G.8) to be measured against ground truth. §R.7 uses the residuals that trained layers actually write as targets. There the same-data plug-in rule transfers only partly (part 1, item 3).
Reviewer_cpdSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies when sharing experts across computational roles improves local prediction and whether those improvements survive multi-step composition. It derives exact sharing and merging criteria in specified linear estimation models, then incorporates downstream geometry and cross-call error covariance into terminal-risk comparisons. A controlled construction demonstrates that sharing can reduce local error while increasing final-output error. The paper also distinguishes predictive and routing-label accuracy from causal evidence that individual experts implement reusable operations. Small symbolic Transformer experiments show significant teacher-forced one-step improvements without statistically resolved free-running final-output improvements.

Critical strengths
  1. The terminal-risk analysis retains error directions and correlations that average local error cannot capture. Proposition 47 provides a transparent strict reversal, while Appendix N.4 characterizes when every local full-pooling improvement is preserved under orthogonal pooling and isotropic noise. Together, these results explain both the failure of local evaluation and the conditions under which it is sufficient.
  2. The paper shows why correct predictions and route labels do not establish expert-level operations, and evaluates this distinction through forced-routing interventions. Separating single-call effects, multi-layer compound actions, and unseen-transition generalization provides a useful framework for assessing claims of expert specialization.
Critical weaknesses
  1. At 64 training traces, sharing improves one-step error by 3.49–7.72 percentage points, but both models have approximately 96.8% final-state error, close to chance. These results establish unresolved terminal gains, rather than a statistically supported reversal. Moreover, local and final evaluations use different prefix distributions. Consequently, the experiments do not isolate the directional amplification or correlation mechanism from teacher-forcing distribution shift and generally poor rollout performance.
  2. The supplementary artifact does not fully cover the manuscript’s central empirical evidence. It provides a substantive implementation of the primary experiments and controlled linear analysis, but omits the low-data ten-seed follow-up underlying Figure 1 and the strict transition-holdout and larger-state-space studies. The supplied training entry point enforces a fixed configuration whitelist, so these experiments cannot be reproduced simply by changing the data budget or seed. Checkpoints and observed result summaries are also excluded. This limits independent verification of the main empirical claims, despite the useful core implementation.
Questions for authors
  1. Can an existing trained-model setting with final performance meaningfully above chance demonstrate a statistically supported local-to-final reversal? Alternatively, can a controlled neural experiment separate the effects of error direction and cross-call correlation from teacher-forcing distribution shift?
  2. For one trained setting, can estimated terminal geometry and error covariance predict the held-out final-risk ordering or preferred sharing strength after accounting for nonlinear and routing corrections? If a useful certificate is currently unavailable, which term is the main obstacle?
  3. Intervention supervision adds both counterfactual targets and an extra forward pass. Can it be compared with a control receiving the same additional transition supervision and comparable computation, without enforcing expert–operation alignment? This would clarify the specific contribution of the intervention objective.
3 comments
AuthorsOpenReview ↗

Reply to Reviewer cpdS (1/3): W1 and Q1

Your main concern is that the trained-model evidence did not separate a real local-to-final effect from chance-level accuracy and teacher-forcing shift. The revised PDF now separates them, with the three experiments you proposed in Q1 to Q3. Part 1 covers W1 and Q1, part 2 Q2 and Q3, and part 3 W2 and the presentation. "§R.k" refers to Appendix R of the revised PDF, where new text is blue.

W1. At 64 traces both models are near chance, so the result is an unresolved terminal gain and not a supported reversal. Local and final evaluation use different prefix distributions.

Answer. Agreed: the 64-trace panel shows an unresolved final comparison, not a reversal. §7 of the revised PDF now states this explicitly, in the wording of the abstract ("without statistically resolved free-running final-output gains"). We have now measured the prefix shift directly: above chance its median is 4.7% of the step error and its maximum 13.6%.

  1. Decomposition (160 checkpoints, §R.3). The gap between final error and teacher-forced step error splits exactly into three terms: prefix shift (the same step evaluated on the model's own prefix), composition over the eight calls, and dependence between calls. The figures above cover the 54 checkpoints with trace error between 5% and 95%. At 128 traces with layer-local experts the terms are −0.002, +0.415 and −0.001.
  2. Composition. The free-running trace error is within a factor 0.948 to 1.036 of the independent composition of the on-policy step errors (146 checkpoints). Against the composed teacher-forced step error it is 0.921 to 1.062 (horizons 4 to 8, 718 evaluations).
  3. At 64 traces your caution applies. There the shift term is −0.09 to −0.12, so we draw no conclusion about direction from that panel.

Q1. Can a trained setting above chance show a supported reversal? Alternatively, can a controlled neural experiment separate direction and cross-call correlation from teacher-forcing shift?

Answer. Yes, through the controlled neural experiment you propose as the alternative. Among the trained comparisons above chance none shows a reversal: every resolved pair inherits the local sign (items 5 and 6). In the controlled experiment, with the propagation map measured in trained Transformers, both predictions of §N.4 are observed: the target built for uneven propagation reverses, and the control target is inherited.

  1. Design (§R.5). In six fully trained checkpoints (three with layer-local experts, three tied) we insert declared linear residual maps at the four depths, which act as the roles, and execute the downstream network. Both endpoints use the same reference prefixes, so there is no teacher-forcing shift. The estimates are the declared coefficients plus isotropic Gaussian noise, so the error covariance is prescribed, not estimated. The composition runs across the four depths of one forward pass, not across reasoning steps, and terminal risk is the error of the final hidden state. The propagation map is the network's own measured Jacobian. Its anisotropy on the contrast space is 4.7 to 6.1, far from the common-gain case.
  2. Direction. For the target that §N.4 constructs from this measured geometry, pooling changes local risk by −0.0032 to −0.0062 and terminal risk by +0.061 to +0.184. Both changes exceed 97 standard errors (over 128 paired noise draws) in 6 of 6 checkpoints. Relative to separate fits, full pooling lowers local risk by 40.2–44.8% but raises terminal risk by 91.0–109.0% (§R.5). For a control target on which the theory predicts inheritance, terminal risk falls by 0.041 to 0.142, also in 6 of 6.
  3. Cross-call correlation. When the same error is reused at all four depths instead of drawn independently, the ratio of the two terminal risks is predicted to within 2% in 6 of 6 checkpoints.
  4. Natural rerouting. Items 2 and 3 hold routes fixed, as Theorem 46 assumes. When routes respond to the perturbation, the control is still inherited in 6 of 6 checkpoints and the strict reversal remains in 3 of 6 (Table 38), where local risk falls by 40.2–43.4% and terminal risk rises by 20.8–39.5%; part 2 (Q2) discusses this case.
  5. Inheritance in the architecture comparisons. Every pair resolved at both endpoints has the same local and final sign: 51 of 51 in the paper's ten-seed study (47 of 47 in the replication), 145 of 145 over all five sharing patterns, 24 of 24 on the sharing-strength ladder, 15 of 15 at horizon 12.
  6. Why. In these models a first wrong state leads to a wrong final state with probability at least 0.965 for every operation and every step, and the trace error is the independent composition of the step errors (W1, item 2). The final error is then close to a monotone function of the step error, so a resolved local gain keeps its sign. This is the discrete analogue of the common-gain case of §N.4, not an application of that theorem. The inserted maps of item 1 perturb residual directions, which propagate unevenly.
AuthorsOpenReview ↗

Reply to Reviewer cpdS (2/3): Q2 and Q3

Q2. For one trained setting, can estimated terminal geometry and error covariance predict the held-out final-risk ordering or the preferred sharing strength after nonlinear and routing corrections? If no useful certificate is available, which term is the main obstacle?

Answer. With routes held fixed, yes for the ordering, with the geometry estimated and the error covariance prescribed (the estimates carry synthetic isotropic noise); for the strength the test is weak (item 2). After natural rerouting, only partly, and the experiment identifies the obstacle: rerouting together with the nonlinear remainder. Teacher-forcing shift is not the obstacle: this experiment has none by design, and above chance its median is 4.7% of the step error (§R.3).

  1. Ordering, routes fixed (§R.5). The geometry W_H is estimated from the network's measured Jacobian on 8,192 contexts and tested on 2,048 held-out contexts. It predicts which target reverses and which inherits, and both predictions hold in 6 of 6 checkpoints (part 1, Q1).
  2. Risk and strength, routes fixed. The predicted terminal risk is within 10% of the measured risk in 54 of 54 checks. The measured minimizer is the predicted terminal optimum in 13 of 18 settings and within 0.01 of it in all 18. For these targets the terminal and local optima differ by at most 0.03, so item 1 is the sharper test.
  3. After natural rerouting. The control target is still inherited in 6 of 6 checkpoints. The strict reversal remains in 3 of 6: those in which the geometry measured under natural routing agrees with the fixed-route geometry (observed after the runs, not registered). The measured optimum is within one grid step (0.1) of the linear prediction in only 7 of 18 settings. With the remainder correction of Proposition 49, the measured terminal risk lies inside the predicted interval in 54 of 54 checks (§R.5), but the interval does not pin down the optimum. This is the case §N.3 describes: the route is selected by the same error that is propagated, so a fixed-G formula no longer applies.
  4. The certificate of Theorem 4 (§R.4). With routes on plan the routing term is exactly zero and the certificate reduces to a sum of per-step errors. In the ten runs with final error between 2% and 20% it is 0.89 to 1.22 times the final error (1.3 and 1.6 times in the 96-trace means), and it orders the two models as the final error does in 16 of 16 resolved pairs. With learned routing it is vacuous: the implementation and routing discrepancies of Eqs. 19–20 are 0.34 and 0.21 per call at 96 traces, and 0.21 and 0.19 at 128.
  5. Real residual targets (§R.7). Here the surrogate fits and the geometry are both estimated. Selecting the sharing strength with the measured terminal geometry instead of the local loss lowers the final error by 0.29 pp on average (SE 0.07), but not uniformly: −0.88 pp in layer 3 and +0.38 pp in layer 2. The same-data plug-in decision transfers only partly (76 of 107 cells correct, against 102 for always fitting separately).

In short, the main obstacle is rerouting, with the nonlinear remainder second.

Q3. Intervention supervision adds counterfactual targets and an extra forward pass. Can it be compared with a control that receives the same supervision and computation without enforcing expert–operation alignment?

Answer. Yes. We ran this control (60 new runs: four budgets, five seeds). It supports the attribution claims of the paper: only enforced alignment gives forced-action fidelity near 1, while the extra transition targets alone already give high accuracy.

Design (§R.8). The control draws the same counterfactual step and action as intervention supervision (I), receives the same transition target and makes the same second forward pass. The action is delivered through the operation token under natural routing, and no expert is forced. TR adds this to route supervision (R), and TB adds it to the base objective.

64 tracesFinal accuracyForced-action fidelityRoute agreement
R3.5%0.371.00
I87.2%0.991.00
TR (control, routes supervised)99.7%0.911.00
TB (control, learned routes)99.7%0.360.46
  1. Fidelity. TR has route agreement 1 and still has lower forced-action fidelity than I in 5 of 5 seeds at 64 and at 96 traces (0.91 and 0.93 against 0.99 and 1.00), and TB stays at or below 0.36.
  2. Accuracy. Both controls match or exceed I at every budget (64 traces: 99.7% against 87.2%; 96 traces: 99.95% and 99.32% against 98.25%). The low-data gain of I over R is therefore due to the added targets, and at 64 traces I is 12.5 pp below the matched controls.
  3. What this means for the paper. Its statements about I concern attribution (Table 20), and they are unchanged. TB is a new instance of Proposition 5: 99.7% accuracy with fidelity 0.36. §R.8 (Table 41 for 64 and 96 traces, Fig. 5 for all four budgets) states that the accuracy gain of I comes from the added targets.
AuthorsOpenReview ↗

Reply to Reviewer cpdS (3/3): W2, presentation and summary

W2. The supplement omits the ten-seed follow-up behind Fig. 1 and the transition-holdout and larger-state-space studies. The training entry point enforces a configuration whitelist. Checkpoints and result summaries are excluded.

Answer. You are right on each point. The original package covered only the fifteen primary runs and the two controlled studies. The revised supplementary archive closes these gaps.

  1. Ten-seed follow-up behind Fig. 1. run_ext1.py and run_ext2.py take budgets and seeds as arguments, and analysis/fig1_local_final.py rebuilds Fig. 1 from the result record.
  2. Transition holdout and larger state spaces. run_ext5.py (sixteen-pair holdout), run_ext6.py (capacity and holdout-size controls) and run_ext9.py (moduli 31, 97 and 251). README.md points to the commands for every study (RUNNING.md, docs/FIRST_ADDENDUM_ASSEMBLY_README.md, REBUTTAL_REPRODUCE.md), and PAPER_MAP.md maps each table and figure of the paper to its record and script.
  3. Whitelist. The whitelist is unchanged and binds only the fifteen primary runs to their reported configurations. The follow-up scripts import the same model, data and objective code directly (extension/ext_common.py), without passing through it.
  4. Result summaries. delivery/RESULTS.json holds the original record, delivery/rebuttal/RESULTS_REGENERATED.json the retrained ten-seed study, and delivery/rebuttal/RESULTS_REBUTTAL.json the nine new studies. CPU-only scripts rebuild every table and figure from them. README.md notes where an archive table uses a different convention from the paper (raw against relabelled route agreement; closed-form against measured local risk).
  5. Checkpoints. The checkpoint files are not included because of their size: several hundred models of 14 to 42 MB each, more than 10 GB in total, which is far beyond the 100 MB limit for supplementary material on OpenReview. We will release them on a public platform (GitHub or Hugging Face) once the anonymity period ends. Until then, each model is regenerated by one command; the median training time of the 237 runs of the data-budget sweep is 87 seconds on a single GPU.
  6. An end-to-end test of the package. All 609 runs of this rebuttal were produced from the released scripts on different hardware. The rerun is not bit-identical: our registered single-run check (tolerance 3 pp) is classified DIFFERENT, with a dense final error of 0.011 against 0.108 at 128 traces and seed 0, and local and tied differing by 2.8 and 2.6 pp (delivery/rebuttal/REPRODUCTION.md; §R.2). As registered, every new comparison uses rerun checkpoints only. Within one environment training is deterministic: 40 runs that repeat earlier configurations reproduce them exactly. The ten-seed conclusions replicate: 51 of 51 same-sign pairs in the original record against 47 of 47 in the rerun, and a routing deficit of 23.6 and 44.3 pp against 23.7 and 40.5 pp (§R.2).

Presentation.

Answer. Improved in the revised PDF, within edits that leave every statement and its numbering unchanged. The paper states many results, and the main text did not make clear enough which ones carry the argument and which experiment tests which claim. The paper remains long; the changes below, all in blue, make the main line easier to follow.

  1. §5 opens with a short map: the three kinds of statement (exact comparison, certificate, audit), the result that makes each, and the experiment that tests it.
  2. §7 is corrected and extended. The 64-trace result is worded as in the abstract, a resolved one-step gain with an unresolved final comparison, and the text states that it is not a reversal. Three new paragraphs report what is resolved above chance, the reversal obtained with the propagation map of trained Transformers, and the selection, certificate and routing tests.
  3. Appendix R presents the new studies in one place, each with its design, its table and the statement it tests (Table 33).
  4. §2 positions the terminal comparison relative to shrinkage under general quadratic loss.
  5. The matched control is in §R.8, with the attribution of the accuracy gain stated (part 2, Q3).
  6. Three typographical errors in §4 and §6 are corrected.

Summary. The revised PDF contains the three experiments you proposed (§R.5, §R.4, §R.8), the shift decomposition (§R.3) and the presentation changes above, and the supplementary archive covers every study. Your questions led to the two additions we consider most valuable: the controlled reversal experiment in trained networks and the matched control.

Remark. The experiments are sized to test these predictions where ground truth is known, and they now cover each experiment you proposed.

The core contribution and criteria of this Learning Theory track paper is theoretical. As you do not hold any theoretical concerns in your review comment, we would be grateful if you would reassess the paper.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response (to all reviewers and the AC)

The three reviews ask how far the theory is supported in trained models and propose concrete experiments. We ran them: nine new studies and 609 training runs, with every pass criterion fixed before the runs. The results are in Appendix R of the revised PDF (new text in blue; "§R.k" below) and in the supplementary archive, negative outcomes included.

Overview

RequestExperimentResult
A controlled neural experiment that separates direction and cross-call correlation from teacher-forcing shift (cpdS Q1)Linear residual maps with synthetic noise at the four depths of six trained Transformers; the propagation map is the measured Jacobian (§R.5)With routes fixed, a strict reversal in 6 of 6 checkpoints: pooling lowers local risk and raises terminal risk (the error of the final hidden state). Under natural rerouting, in 3 of 6
Teacher-forcing shift versus composition (cpdS W1)Exact three-term decomposition at 160 checkpoints (§R.3)Prefix shift is 4.7% of the step error (median, trace error 5–95%); the free-running trace error is 0.95 to 1.04 times the composed on-policy step errors
Does the certificate predict the final comparison, and which term is the obstacle? (cpdS Q2, F8RX Q2, XPxZ W2)The per-call discrepancies of Theorem 4 (Eqs. 19–20) measured at 160 checkpoints (§R.4)Routes on plan: certificate 0.89 to 1.22 times the final error where it is 2–20% (ten runs), looser above; same ordering in 16 of 16 pairs. Learned routes: vacuous (discrepancies of 0.17 to 0.34 per call)
An open sharing decision on trained models (XPxZ W1, Q1)Sharing strength at five values, five budgets and five seeds (§R.6)At 96 and 128 traces, between chance and saturation, held-out selection picks the final-error minimizer in 10 of 10 seeds (20 of 25 cells over all budgets)
A non-saturated end-to-end comparison (F8RX Q1, W1; XPxZ W4)Horizon 12; all five sharing patterns of Theorem 19 (§R.9)Final accuracy 10.0 / 6.4 / 86.6% and 80.2 / 83.5 / 98.3% (local / tied / dense); every resolved pair has the same local and final sign (15 of 15; 145 of 145)
A matched control for intervention supervision (cpdS Q3)Same extra targets and forward pass, no forced expert (§R.8)Only enforced alignment gives forced-action fidelity near 1 (0.99, against 0.91 and 0.36); the accuracy gain comes from the added targets (99.7% against 87.2%)
B and C on real targets (XPxZ Q1)Same-data plug-in rule on the residuals that trained layers write (§R.7)Partial: correct in 76 of 107 cells (46 of 48 at n ≥ 256), but no better than always fitting separately (102)

Replication. All new studies and the paper's ten-seed study were run on different hardware with the released scripts. Single runs are not bit-identical (a registered single-run check differs by up to 9.7 pp); the ten-seed conclusions are unchanged. Same-sign pairs: 51 of 51 originally, 47 of 47 in the replication. Deficit of the tied model (learned routing) against the dense model: 23.6 and 44.3 pp against 23.7 and 40.5 pp (§R.2). Of 31 directional predictions fixed before the runs, 26 passed their registered test; §R.1 discusses each of the five that did not. A check of the ladder endpoints also exceeded its threshold (§R.6).

How the results line up with the theory.

  1. Role-dependent propagation admits a reversal, for a target that the proof constructs (Theorem 46, Proposition 47, §N.4). This is now observed with the propagation map of trained Transformers, for synthetic estimates and fixed routes.
  2. Uniform propagation gives inheritance (§N.4, for a fixed linear map). The trained task is its discrete analogue: a first wrong state leads to a wrong final state in at least 96.5% of cases for every operation and step, and step errors compose nearly independently (§R.3). Every resolved pair has the same local and final sign; none reverses.
  3. Theorem 4 separates the statistical effect of sharing from assembly and routing terms. With routes on plan and small errors the measured discrepancies sum to the final error, and supervising the router (which also adds route labels) removes the deficit of the tied model.

Remark. We would like to emphasise the core contribution of this paper submitted in Learning Theory track is theoretical rather than experimental. The experiments that the quantities of the theory allow are now added in the paper. Extensive LLM benchmark is far beyond common theory paper's experimental requirement(reply to XPxZ, W4), and the revised §8 states this limit. The core contribution is theorems with complete proofs.

AuthorsOpenReview ↗

Additional experiments: more budgets between chance and saturation, and selection under learned routing

We ran three further studies with the updated released code. Their criteria were fixed before the runs, and all seven directional criteria passed. They extend two earlier answers that carried a qualifier ("at 128 traces", "router supervised"). The results are in the new §R.10 of the updated revision (Tables 43 and 44) and in delivery/followup/ of the updated supplementary archive.

  1. More budgets between chance and saturation (F8RX Q1; XPxZ Q1). Ten seeds at 112, 144, 160 and 176 traces (120 runs, learned routing).
Traces112144160176
Final accuracy, local / tied / dense21.4 / 25.0 / 72.8%81.1 / 76.9 / 97.1%92.8 / 92.9 / 99.6%98.3 / 98.5 / 100.0%
Tied versus local: seeds with the final contrast resolved for tied / for local6 / 33 / 63 / 51 / 0

Of 120 contrasts, 43 are resolved at both endpoints, and all 43 have the same sign (12 of them tied versus local). As at 128 traces, tied and local have no consistent winner. The free-running trace error is 0.95 to 1.04 times the independent composition of the teacher-forced step error (100 runs).

  1. Sharing strength with the router supervised, denser grid (XPxZ W1, Q1; F8RX W2). Five budgets from 96 to 128 traces and ten seeds: 50 cells, 40 of them new (200 new runs). The mean final error per budget and strength lies between 2% and 62%. Selecting the strength on held-out one-step error picks the final-error minimizer in 43 of 50 cells (33 of the 40 new ones) and is within the simultaneous radius (2.5 pp) of it in 49 of 50; the largest regret is 2.8 pp. The minimizer is λ = 0.25, 0.5, 0.75 and 1 in 3, 14, 13 and 20 cells, and never λ = 0. Mean final error is 17.1% at the selected strength, against 20.3% for the best fixed strength (λ = 1) and 31.0% without sharing.

  2. The same ladder with learned routing (cpdS Q2; XPxZ W2). Five budgets from 112 to 176 traces and five seeds: 25 cells (125 runs), no route labels. Held-out selection picks the final-error minimizer in 22 of 25 cells and is within the radius in 25 of 25 (largest regret 0.6 pp). Mean final error is 24.4% at the selected strength, against 31.7% for the best fixed strength (λ = 0.25). No sharing is the minimizer in 10 of 25 cells: with learned routing, sharing does not help consistently, in line with item 1, yet the held-out choice still finds the best strength. Resolved pairs with the same local and final sign: 100 of 100 (item 2) and 82 of 82.

AuthorsOpenReview ↗

Summary and Reminder for the Area Chair and Reviewers.

Dear Area Chair and Reviewers,

Thank you for your time and engagement during the discussion. We have posted point-by-point replies, a general response and a note on additional experiments. For ease of reference, we highlight three central questions from the reviews and point to the evidence that addresses each in the revised manuscript.

  • 1. Is the theory tested in trained models, between chance and saturation? (cpdS W1, Q1; XPxZ W3; F8RX Q1) With the propagation map of trained Transformers and synthetic estimates, the strict reversal of Proposition 47 is observed in 6 of 6 checkpoints with routes fixed, and in 3 of 6 under natural rerouting. Between trained models, every pair resolved at both endpoints keeps its local sign (51 of 51 in the paper's ten-seed study, 47 of 47 in its replication), and none reverses. (§R.5, Table 38; §R.2; §R.10.)
  • 2. Can the theory guide an open sharing decision? (XPxZ W1, W2, Q1; cpdS Q2; F8RX W2, Q2) Held-out selection of the sharing strength finds the final-error minimizer in 43 of 50 cells with the router supervised and in 22 of 25 with learned routing, an empirical finding. The measured certificate of Theorem 4 tracks the final error when routes are on plan and errors are small, and is vacuous under learned routes. (§R.6; §R.10, Table 44; §R.4.)
  • 3. Is validation on a symbolic task enough for a learning-theory paper? (XPxZ W4 and follow-up; F8RX W1) The contribution is a set of theorems with complete proofs, and no review points to an error in a statement or proof. The symbolic task makes the quantities in the theorems (roles, target operators, pre-action states) observable, whole a language-model benchmark does not define them, and §8 states this limit. Within this setting we ran the experiments the reviews proposed and report the outcomes that did not pass. (§R.1, Table 33; §8.)

We thank Reviewer XPxZ for reassessing the paper after the rebuttal. The remaining question the reviewer raised for discussion with the Area Chair, whether validation at this scale is enough for a learning-theory paper, is addressed in the third point above.

As a reminder for Reviewers cpdS and F8RX, we would be grateful if you could give response to our reply before the public discussion period ends, and we hope our additional evidence and reply allows the paper to be reassessed. We would be happy to clarify any remaining questions before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.