acceptodds
Desk rejected at ICLR 2027

Read Together, Write Apart: Decoupling Rollout Access from Update Ownership in Multi-Role LLM Training

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

When one language model plays solver, verifier, and reviser, “parameter sharing” conflates two choices: which adapters a role generates from, and which adapter receives its gradient. A shared adapter couples reads and writes; role isolation severs both. We introduce Dual Routing, which makes the two routes explicit: the verifier and reviser generate from a common adapter pool, while each role's update trains only the adapter it owns. With seven independently trained Llama-3.1-8B checkpoints per method, the downstream workflows of shared training and of role isolation erase accuracy from their own first answers in every replicate, whereas Dual Routing's does not, and Dual Routing beats capacity-matched role isolation by 26.7 points in every replicate. A preregistered route factorial shows the gain is learned: read exactly as isolated adapters are read, Dual-trained adapters still win by 30.0 points. Further controls identify the mechanism: training downstream roles on reads at the merged pool's reduced scale is what keeps the workflow from collapsing. Of all trained variants, Dual Routing's reviser is the only one that improves on the untrained model both at repairing wrong answers and at preserving correct ones. Collaborative reads with role-owned writes are a simple design point for training multi-role LLM workflows that adds no forward cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-role LLM training, parameter-efficient fine-tuning, LoRA, adapter routing, gradient interference, reinforcement learning from verifiable rewards, self-correction, modular fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Spg3OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose Dual Routing, a multi-LoRA training interface for solver–verifier–reviser workflows where downstream roles generate from a 0.5/0.5 factor-wise merge of their adapters while each role's gradient trains only its own adapter. The key mechanism is that the merged read pool reduces the effective adapter scale during training, preventing downstream roles from destructively overwriting the solver's correct answers — the paper's own controls show that training role isolation at this reduced scale recovers most of the gain (34.66% vs. 38.99%). Experiments on 1,472 synthetic program-reasoning items with Llama-3.1-8B across 7 seeds show Dual Routing beats role isolation by 26.7 points and is the only method with positive system gain, but on Qwen2.5-7B all methods are within 2 points. The statistical analysis is thorough (sign tests, Wilcoxon, bootstrap, Holm correction), though the underlying evaluation scope is narrow — four deterministic task templates with no natural-language reasoning benchmarks.

Critical strengths

Exemplary experimental controls. The preregistered route factorial (Table 6, Appendix E–G) systematically decomposes the contribution of reads, writes, and scale into separate controls — this is the level of ablation design more multi-role papers should aspire to.

Unusual transparency. The paper explicitly proves PCGrad-A/B reduce to role isolation under disjoint adapters (line 228–236), reports that Qwen results do not transfer (Appendix J), discloses that 3/7 Dual seeds were re-run after scoring (Appendix K), and provides first-run sensitivity analysis — the honesty test is passed decisively.

Rigorous statistical methodology. Seed-level inference with four complementary procedures (sign, Wilcoxon, t, bootstrap) with Holm correction, plus explicit separation of record-level vs. seed-level uncertainty (Table 13), sets a high standard for RL training papers that typically report a single-seed pooled mean.

Critical weaknesses
  1. Mechanism undermines motivation. The paper's title and narrative emphasize "collaborative reads," but the own-half control (Table 17) shows that simply halving the adapter scale — with no cross-role reading at all — recovers 86% of the gap (34.66% vs. 38.99% for full Dual, from 7.64% isolated). The actual cross-role reading contributes only +4.33 points, making the headline contribution primarily an implicit regularization effect, not a collaboration effect.

  2. Evaluation scope is critically narrow. All main results come from 4 synthetic task templates on a single backbone (Llama-3.1-8B); the one additional backbone tested (Qwen2.5-7B) shows the final-accuracy result does not transfer (Table 22, all methods within 1.6 points). No real-world benchmarks (math, code, open-domain QA), no model scale variation, no variation of the fixed 0.5/0.5 merge ratio or topology (e.g., solver reading downstream adapters) — the generality claimed in the conclusion far exceeds what the experiments support.

  3. Effective baseline count is two. PCGrad-A and PCGrad-B are confirmed no-ops (inner product is exactly zero, projection never fires), collapsing five methods to three (Shared, Isolated, Dual). The Shared comparison is "unresolved" at seed level (CI [−5.82, +14.32]). So the paper's reliable claim is against one real baseline (role isolation), on one backbone, on synthetic tasks.

Questions for authors
  1. The Qwen result shows no final-accuracy transfer. Have you investigated why, beyond "the solver is already strong"?

  2. Why are no real-world benchmarks included?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Spg3 (Part 1/4)

We thank Reviewer Spg3 for the constructive review and for noting our controlled route ablations, statistical analysis, and transparency. Taking up your comments on presentation, mechanism, and scope (W1–W3, Q1–Q2), we reorganized the Revised Manuscript around the algebraic rollout-scale mechanism (Abstract, §1, §2.3, §4.1, §5–§7, Limitations), added Untrained (0 updates, ref.) as row 1 of Main Tables 3–4, labeled PCGrad-A/B as "(isolated rep. A/B)", and ran 19 fresh 100-step H200 RL runs alongside fixed-solver transplants, exact rank-128 zero-cross-term readouts, and 54-checkpoint evaluations (>60,000 trajectories) on GSM8K (n = 500) math and container-tested APPS (n = 500) code.

The Dual Routing training and inference procedure is unchanged. Because the 0.5/0.5 merge sets s = 0.25 by construction on each role's own delta, Dual Routing keeps the LoRA rollout perturbation in the stable regime while role-owned writes prevent cross-role gradient interference.

Summary of Revisions

Critique / QuestionNew Experimental EvidenceRevised PDF Pointers
W1 (+ Clarity): Mechanism vs. motivation (own-half recovers 86%; regularization vs. collaboration)Rebuilt the narrative around forward rollout readout scale under role-owned writes: (i) explicit algebra (§2.3 Eqs. 4–5) shows 0.5/0.5 factor merging weights own-adapter deltas by s = 0.25; (ii) single-seed ladder (seed 52) locates a collapse cliff at s = 1.00 (0.61%) vs. 39.54%–43.95% across s ∈ [0.25, 0.50] (untrained reference s = 0: 42.66%); (iii) 5-pair zero-rerun confirmation (s53–57) gives cross-scale s = 0.25 at 36.64% vs. Dual at 32.07% (−4.58 pp, t95 [−13.86,+4.70]; Always-revise +0.12 pp), matching explicit s = 0.25 scaling within seed variance (paired SD ≈ 7.47 pp); (iv) 3-seed Low-LR (2.5×10⁻⁶, s_train = 1.0) still collapses (21.33% final, −11.89 pp gain, 4.80% stop), isolating forward readout scale from optimizer step size; (v) rank-128 zero-cross-term readout matches Dual within 0.5 pp (31.60% vs. 32.07%). With a fixed base solver, Dual VR leads cross-scale s = 0.25 VR on 5/5 paired seeds (29.52% vs. 28.15%, +1.37 pp).Abstract (p. 1), §1 (pp. 1–2), §2.3 Eqs. 4–5 (p. 3), §4.1 Tables 3–4 (p. 6), §5 & §7 (pp. 8–9), App. M–N, Fig. 5 (p. 20), Tables 24–25 (p. 21), Fig. 6 (p. 22), Table 26 (p. 23)
W2, W3, Q2: Real-world benchmarks, scale/topology sweeps, & baselinesEvaluated 6 Llama-3.1-8B arms (23 ckpts) and 7 Qwen2.5-7B arms (31 ckpts) on GSM8K and APPS, added Untrained to Main Tables 3–4, and contrasted with 3-checkpoint MALT in §6. On same-seed Llama (s45–48), Dual achieves +5.20 pp gain on GSM8K (27.45% vs. 26.15% Shared and 13.35% Role-Iso, 4/4 seeds ahead of Role-Iso) and 7.75% on APPS (+6.90 pp over Role-Iso 0.85%, t95 [+4.54,+9.26], 4/4 seeds). Fixed-solver Base S + Dual VR maintains untouched base-solver accuracy on the 1,472-item main panel (29.52% vs. 29.42%; 28.58% on s45–48), Llama GSM8K (67.12% vs. 65.60%), and Qwen GSM8K (80.56% vs. 80.20%).§4.1 (Tables 3–4, p. 6), §6 (p. 9), App. M–O, Table 24 (p. 21), Fig. 6b,c (p. 22), Tables 27–28 (p. 23), Table 29 (p. 24)
Q1: Why Qwen2.5-7B shows +0.91 pp in-domainBreak-even decomposition plus perturbation-threshold test: at s = 1.0 Qwen's isolated reviser preserves 94.7% of correct answers (5.3% break rate vs. Llama's 91.6%), predicting (19.56% − 5.02%) × 0.053 = +0.77 pp against +0.91 pp observed on the main evaluation panel. On OOD GSM8K, Dual exceeds Qwen Role-Iso by +38.52 pp (71.96% vs. 33.44%, 5/5 seeds). Concatenating two full-scale adapters (dualconcat, 2×1.0×) crosses Qwen's critical threshold and collapses to 5.66% vs. 60.43% on the 1,024-item checkpoint-validation panel (+54.77 pp, 5/5 seeds), 0.84% vs. 71.96% on GSM8K, and 1.60% vs. 13.52% on APPS.§5 (pp. 8–9), App. N–O, Fig. 6a,c (p. 22), Table 26 (p. 23), Table 29 (p. 24)
AuthorsOpenReview ↗

Response to Reviewer Spg3 (Part 2/4)

W1 (+ Presentation and clarity): Rollout Readout Scale Under Role-Owned Writes

"The paper's title and narrative emphasize 'collaborative reads,' but the own-half control (Table 17) shows that simply halving the adapter scale ... with no cross-role reading at all ... recovers 86% of the gap (34.66% vs. 38.99%)... making the headline contribution primarily an implicit regularization effect, not a collaboration effect."

Table 17 pointed directly to readout scale as the candidate driver. We tested the competing mechanistic hypotheses and centered the revised paper's explanation on forward readout scale under role-owned writes (Abstract, §1, §2.3, §5, §7, Limitations), tracing how rollout-scale attenuation prevents downstream entropy collapse:

  1. Factor-merge algebra and cross-term test (§2.3 Eqs. 4–5, p. 3; Main Tables 3–4, p. 6): Expanding (0.5 B_v + 0.5 B_r)(0.5 A_v + 0.5 A_r) = ¼(B_v A_v + B_r A_r + B_v A_r + B_r A_v) weights each downstream role's own adapter by s = 0.25. Scoring Dual checkpoints with exact rank-128 linear readout 0.25 B_v A_v + 0.25 B_r A_r (zero cross-terms) yields 31.60% vs. 32.07% on the 1,472-item main panel, 38.80% vs. 38.48% on Llama GSM8K (n = 500), and 72.08% vs. 71.96% on Qwen GSM8K (Appendix N, Tables 27 & 29), bounding the bilinear cross-term effect below 0.5 pp. Main Tables 3–4 also include Untrained workflow (0 updates, ref.; 41.75%) and label PCGrad-A/B as "(isolated rep. A/B)".
  2. 6-point readout-scale ladder (Table 24, p. 21) and 5-pair zero-rerun confirmation (Table 25, p. 21): On single seed 52, own-only readout collapses at s = 1.00 (0.61%) and stays in a stable regime below it: 33.15% at s = 0.75, 39.54% at s = 0.50, 43.95% at s = 0.25, 36.01% at s = 0.10, and 42.66% for the untrained reference (s = 0), with variations over s ∈ [0.25, 0.50] lying within single-seed variance. Across 10 fresh zero-rerun runs on seeds 53–57 (bitwise-identical 1,344-tensor LoRA initialization per pair), cross-scale s = 0.25 reaches 36.64% Final and Dual Routing 32.07% (Dual − cross-scale s = 0.25: −4.58 pp, t95 [−13.86,+4.70]; Always-revise +0.12 pp, [−3.38,+3.62]; 7-seed own-half at s = 0.5 gives −0.67 pp). Dual Routing and explicit s = 0.25 own-only reads perform within seed variance (paired SD 7.47 pp).
  3. Optimizer learning-rate control (Appendix N, Figure 6a, p. 22): Testing whether s = 0.25 behaves like a 4× smaller step size, we cut AdamW LR 4× to 2.5×10⁻⁶ while keeping s_train = 1.0 across 3 fresh runs (s53–55). Downstream collapse persists (21.33% final vs. 33.22% solver = −11.89 pp gain, 4.80% stop; behind both Dual and cross-scale s = 0.25 on 3/3 paired seeds), whereas attenuating only the evaluation readout to s_eval = 0.25 restores a 41.51% stop rate and moves GSM8K gain from −3.27 pp to +2.27 pp. Separating read (R_τ) and write (W_τ) routes thus isolates forward rollout scale under disjoint role-owned writes as the governing stability lever, and with solver drift removed (Base Solver + Trained VR, Figure 6b), Dual VR (29.52%) finishes ahead of cross-scale s = 0.25 VR (28.15%) on 5/5 paired seeds (+1.37 pp, t95 [+0.24,+2.50]).
AuthorsOpenReview ↗

Response to Reviewer Spg3 (Part 3/4)

W2, W3 & Q2: Real-World Math (GSM8K) & Code (APPS), Topologies, and Baselines

"Why are no real-world benchmarks included? ... No real-world benchmarks (math, code, open-domain QA), no variation of the fixed 0.5/0.5 merge ratio or topology ... Effective baseline count is two (Shared, Isolated)."

  1. 54-Checkpoint Cross-Domain and Cross-Family Evaluations (Appendix O, Tables 27–29, Figure 6c, pp. 22–24): We evaluated six Llama-3.1-8B arms (23 checkpoints, including same-seed s45–48 Dual and fixed-solver transplants) and seven Qwen2.5-7B arms (31 checkpoints, matched seeds 42–46) on held-out GSM8K (n = 500) natural-language math (zero prose math in training) and held-out APPS (n = 500) competitive programming (100 intro / 300 interview / 100 competition, container-executed against official unit tests):

    Workflow Arm / Transplant (n = 500 items/benchmark)GSM8K Strict Final (Lenient)GSM8K GainGSM8K FRcond\mathrm{FR}_{\mathrm{cond}}APPS Strict Final (Test-avg)APPS GainAPPS FRcond\mathrm{FR}_{\mathrm{cond}}
    Llama Untrained (0 updates, ref.)65.60% (68.00%)-9.20 pp24.1%11.40% (24.83%)-1.00 pp30.6%
    Llama Shared (s=1.0s=1.0, s45–48)26.15% (26.25%)-9.35 pp85.1%5.85% (11.14%)-3.70 pp84.2%
    Llama Role-Isolated (s=1.0s=1.0, s45–48)13.35% (13.45%)-9.40 pp99.3%0.85% (1.72%)-4.40 pp100.0%
    Llama Dual Routing (0.50.5 merge, s45–48)27.45% (29.90%)+5.20 pp36.5%7.75% (13.39%)-0.35 pp28.5%
    Llama Cross-scale s=0.25s=0.25 (s53–57)26.44% (28.04%)+4.12 pp39.8%7.60% (13.47%)-0.28 pp32.5%
    Llama Dual Routing (0.50.5 merge, s53–57)38.48% (40.44%)+1.68 pp31.8%6.76% (12.03%)+0.40 pp38.5%
    Llama Fixed Base Solver (74.80%) + Dual VR67.12% (69.44%)-7.68 pp28.9%------
    Qwen2.5-7B Untrained (0 updates, ref.)80.20% (solver 20.80%)+59.40 pp14.4%12.40% (26.17%)(Table 29)46.1%
    Qwen2.5-7B Shared (s=1.0s=1.0, s42–46)54.04% (solver 28.00%)+26.04 pp51.3%13.72% (29.23%)(Table 29)51.5%
    Qwen2.5-7B Role-Isolated (s=1.0s=1.0, s42–46)33.44% (77.40% at seval=0.25s_{\mathrm{eval}}{=}0.25)+12.80 pp85.7%15.12% (31.09%)(Table 29)66.8%
    Qwen2.5-7B dualconcat (2×1.0×2\times 1.0\times, s42–46)0.84% (solver 20.64%)-19.80 pp98.1%1.60% (2.71%)(Table 29)94.1%
    Qwen2.5-7B Dual Routing (0.50.5 merge, s42–46)71.96% (Base S + Dual VR: 80.56%)+49.20 pp14.2%13.52% (29.15%)(Table 29)40.3%

    For Qwen rows the parenthetical gives strict-format solver accuracy, except Role-Isolated (accuracy at s_eval = 0.25) and Dual (Base S + Dual VR). Zero-shot Qwen2.5-7B often omits the required <answer> tag on first-pass prose math (20.80% strict solver), and the reviser stage formats and repairs answers to 80.20% strict Final for the untrained workflow (Table 29 caption); the identical strict-tag scorer applies across all arms.

    • Expanded baseline suite (W3): Alongside Shared and Role-Isolated (s=1.0s=1.0), the revised paper compares against zero-shot Untrained (Main Tables 3–4, Tables 27–29), scale-matched cross-scale s=0.25s=0.25 and own-half (Tables 24–25, 27–28), Low-LR (2.5×10−62.5\times 10^{-6}) Role-Isolated (Appendix N, Figure 6a), shared-downstream-writes (Table 16), unattenuated dualconcat (2×1.0×2\times 1.0\times, Table 29), and fixed-solver transplants (Tables 27 & 29), and contrasts single-backbone LoRA routing with 3-checkpoint MALT in §6 (p. 9). Same-seed Llama (s45–48) Dual beats Role-Isolated on 4/4 seeds on both GSM8K (+14.10 pp, item CI95 [+11.60,+16.60]\mathrm{CI}_{95}\ [+11.60,+16.60], +5.20 pp system gain) and APPS (+6.90 pp, seed t95 [+4.54,+9.26]t_{95}\ [+4.54,+9.26]), while fixed-solver Base S + Dual VR maintains untouched base-solver accuracy across all three settings (29.52% vs. 29.42% on the 1,472-item main panel; 67.12% vs. 65.60% Llama GSM8K; 80.56% vs. 80.20% Qwen GSM8K), compared with 0.29%, 44.75%, and 33.48% for Role-Iso VR.
  2. Merge-scale and operator variations (W2): Together with the 2×2 route factorial, solver-read, and shared-downstream-write topologies (Table 6 p. 8; Appendices E–G pp. 14–18), Table 24 (p. 21) sweeps s∈{0.00,0.10,0.25,0.50,0.75,1.00}s \in \{0.00, 0.10, 0.25, 0.50, 0.75, 1.00\}, and Tables 27 & 29 (pp. 23–24) compare exact rank-128 linear 0.25×0.25\times readout, 2×1.0×2\times 1.0\times concatenation (dualconcat), and 0.5/0.50.5/0.5 factor merging.

Reviewer_S6SqOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the "two-sided sharing problem" in training multi-role LLM workflows (e.g., solver-verifier-reviser). It argues that "parameter sharing" conflates two independent dimensions: read routes (what a role reads during generation) and write routes (what a role's gradients update). To resolve this, the authors propose Dual Routing, which decouples reads and writes. Each role updates only its own LoRA module (isolated writes), while the verifier and reviser share a factor-wise 0.5/0.5 merged read pool (shared reads) during generation, allowing cross-role collaboration without mixing training objectives.

Evaluated on a controlled 1,472-item program-reasoning panel across 7 seeds, Dual Routing is the only method with a positive system gain (+1.99). It outperforms capacity-matched role isolation by 26.74 points in all replicates and trains the only reviser that improves upon the untrained backbone at both repairing wrong answers and preserving correct ones. Through preregistered controls, the authors attribute this gain primarily to "read scale": the factor-wise merge shrinks the effective delta norm to 0.43–0.61 of a single adapter's delta, and training downstream roles at this reduced scale prevents the workflow collapse observed in standard role isolation. The paper further validates the topology via a 2x2 route factorial, write-route controls, and code-level verification that PCGrad is a no-op under role-owned writes.

Critical strengths
  1. Conceptual Clarity: Decoupling Reads and Writes The paper offers a valuable reframing of multi-role LLM training by decomposing "parameter sharing" into two orthogonal axes: read routes (generation-time visibility) and write routes (gradient update ownership).

    The proposed Dual Routing—shared reads via factor-wise merge with role-isolated writes—provides a principled way to avoid both objective interference (shared adapters) and workflow collapse (strict role isolation).

  2. Rigorous Mechanistic Validation The empirical design is notably rigorous. Using a 1,472-item panel across 7 seeds with strict capacity and rank matching, the authors employ preregistered controls (route factorials, write-route and read-scale ablations) to isolate specific mechanisms.

    The code-level verification that PCGrad is a no-op under role-owned writes further strengthens their argument. This level of transparency makes the findings highly informative and substantiates the proposed method's effectiveness.

Critical weaknesses
  1. Causal Attribution Lacks Training Diagnostics In the main text, the paper attributes the catastrophic collapse of role isolation (Final 7.73%, Table 3) to the read route/read scale mechanism (Section 4.2, Section 5). However, an extreme drop from a 25.00% solver accuracy to a 7.73% final accuracy is counter-intuitive.

    The main text provides no training-process diagnostics (e.g., loss curves, gradient norms, policy entropy) for the downstream roles to rule out a major alternative explanation: that the isolated verifier/reviser training simply failed to optimize or converge. Without this, claiming read scale is the definitive root cause is premature.

  2. Missing Critical Baselines To substantiate the severe degradation in role isolation, the main text lacks a critical control: there is no end-to-end baseline evaluating a frozen backbone across the full three-role workflow (Final/Stop/FA/FR).

    While Section 4.1 notes the untrained reviser's repair rate (5.0%) and preservation rate (74.6%), it does not report the untrained model's end-to-end Final accuracy. This makes it impossible to assess from the main text whether the isolated downstream training is fundamentally broken compared to a zero-shot prompt baseline.

Questions for authors
  1. Role isolation drops drastically from 25.00% solver accuracy to 7.73% final accuracy (Table 3), yet the paper attributes this entirely to read routing/scale. How do the authors rule out training instability or optimization failure as the cause of this collapse, and what evidence demonstrates that the isolated downstream models were trained normally?

  2. Section 4.1 notes the untrained backbone achieves a 74.6% preservation rate, but the paper lacks an end-to-end performance report for it. What is the final accuracy and system gain if the verifier and reviser use the zero-shot untrained backbone (without any trained LoRA adapters) in the full workflow? Does the isolated workflow (7.73%) perform worse than this zero-shot baseline?

  3. The paper concludes that the factor-wise merge (0.43–0.61 effective delta norm) is the key mechanism preventing collapse. Does final accuracy monotonically improve if the read scale is further decreased (e.g., swept from 0.1 to 1.0), or is 0.5 uniquely optimal? Additionally, were the retrained role-isolation controls evaluated on the same seeds (42–48) as the main table to rule out seed variance?

5 comments
AuthorsOpenReview ↗

Response to Reviewer S6Sq (Part 1/4)

We thank Reviewer S6Sq for the thoughtful review and for noting our decoupling of rollout read routes (RτR_\tau) from gradient write routes (WτW_\tau) and our controlled route ablations. To address your diagnostic and baseline questions (W1–W2, Q1–Q3), the Revised Manuscript adds continuous training telemetry and per-role endpoint diagnostics, 3-seed Low-LR (2.5×10−62.5\times 10^{-6}) optimizer controls, a full end-to-end zero-shot Untrained workflow baseline (Final, Stop, FA, FR) alongside 15-seed fixed-solver transplants, a 6-point readout-scale sweep paired with a 15-pair zero-rerun Llama confirmation (30 runs) and 5-pair Qwen confirmation, and 79-checkpoint evaluations on GSM8K (n=500n = 500) and APPS (n=500n = 500).

The Dual Routing procedure is unchanged. Across the new controls, averaging the two active adapters' factors (0.5/0.5) places each role's own delta at forward readout weight s=0.25s = 0.25, keeping the LoRA rollout perturbation in the stable range while role-owned writes prevent cross-role gradient interference.

Overview of New Experiments and Manuscript Updates

Question / ConcernNew Experimental & Diagnostic EvidenceRevised PDF Pointers
W1 & Q1: Optimization failure vs. rollout-scale collapse (25.00% → 7.73%)Continuous training telemetry (Fig. 5 right, seed 45) and per-role endpoint diagnostics (App. M, Table 24, seed 52) verify that all arms start with finite gradients (0.022/0.009) and healthy entropy (4.39/2.05); s=1.0s = 1.0 then undergoes mid-run entropy collapse (4.39 → 1.30, 2.05 → 0.82; parse 95.92% → 70.31%). A 3-seed Low-LR (2.5×10−62.5\times 10^{-6}, strain=1.0s_{\mathrm{train}} = 1.0) control (Fig. 6a) still collapses (21.33% final vs. 33.22% solver = -11.89 pp, 4.80% stop, trailing Dual and cross-scale s=0.25s = 0.25 on 3/3 seeds), whereas setting seval=0.25s_{\mathrm{eval}} = 0.25 on those exact checkpoints restores a 41.51% stop rate (+2.27 pp gain on GSM8K).§5 (pp. 8–9), App. M–N, Fig. 5 (p. 20), Table 24 (p. 21), Fig. 6a (p. 23), Table 26 (p. 23)
W2 & Q2: Zero-shot Untrained workflow & comparison to 7.73%Unscaled Role-Isolated (7.73%) finishes 34.02 pp below the 41.75% zero-shot workflow. Added Untrained workflow (0 updates, ref.) as row 1 of Main Tables 3–4 (41.75% Final, 45.55% Solver, -3.80 pp Gain, 59.38% Stop, 31.25% FA, 17.43% FR, 36.72% Redo). Fixed-solver transplants (Base Solver + Trained VR) isolate downstream verification and revision from first-pass solver drift: Base S + Dual VR maintains untouched base-solver accuracy on the 1,472-item main panel (29.18% vs. 29.42% across 15 seeds s53–67 under matched T=1.0T = 1.0 rollout, +0.62 pp ahead of s=0.25s = 0.25 VR 28.56%, 9/15 seeds; 29.52% vs. 28.15% on s53–57; 28.58% on s45–48), Llama GSM8K (66.03% vs. 65.60%), Llama APPS (10.60% vs. 11.40%), and Qwen GSM8K (80.56% vs. 80.20%).§4.1 (Tables 3–4, p. 6), §5 (pp. 8–9), Limitations (p. 10), App. N–O, Fig. 6b,c (p. 23), Tables 27–28 (p. 24), Table 29 (p. 25)
Q3: Monotonicity over 0.1–1.0 & same-seed pairingSingle-seed ladder on seed 52 (Table 24, p. 21): 0.61% (s=1.00s = 1.00) → 33.15% (0.75) → 39.54% (0.50) → 43.95% (0.25) → 36.01% (0.10), vs. 42.66% untrained (s=0s = 0), showing a sharp cliff at s=1.00s = 1.00 and non-monotonic drop at s=0.10s = 0.10. Paired on identical seeds: own-half (s=0.5s = 0.5) on s45–48 (+4.33 pp) and s45–51 (-0.67 pp), 30 fresh zero-rerun bitwise-matched runs across 15 Llama pairs s53–67 (Table 25: Dual 33.14% vs. cross-scale s=0.25s = 0.25 32.55%, Δ=+0.58 pp\Delta = +0.58\text{ pp}, t95t_{95} [-4.48, +5.65]; +0.69 pp Always-revise), 5 matched Qwen s=0.25s = 0.25 pairs on s42–46 (56.74% vs. 60.43% Dual), and same-seed s45–48 Dual on GSM8K (+5.20 pp gain) and APPS (+6.90 pp, t95t_{95} [+4.54, +9.26]).Abstract (p. 1), §1 (pp. 1–2), §2.3 (p. 3), §5 (pp. 8–9), App. M & O, Fig. 5 (p. 20), Tables 24–25 (pp. 21–22), Tables 27–29 (pp. 24–25)
AuthorsOpenReview ↗

Response to Reviewer S6Sq (Part 2/4)

W1 & Q1: Ruling Out Broken Gradients and Step-Size Effects (Appendices M–N, Table 24, Figures 5–6)

"How do the authors rule out training instability or optimization failure as the cause of this collapse (25.00% → 7.73%), and what evidence demonstrates that the isolated downstream models were trained normally?"

Figure 5 (right, p. 20), Table 24 (p. 21), Figure 6a (p. 22), and Appendices M–N (pp. 19–23) test optimization health directly via continuous telemetry and a 4×4\times learning-rate reduction:

  1. Healthy initialization and early updates across all arms (Figure 5 right and Appendix M): Over 100 updates on seed 45 (Figure 5 right), Role-Isolated (s=1.0s = 1.0), own-half (s=0.5s = 0.5), Shared, and Dual Routing all start with finite policy-gradient losses, well-scaled global gradient norms, healthy logged entropy, and zero non-finite updates. Per-role telemetry on seed 52 (Appendix M) confirms healthy initial adapter gradient norms (0.022 checker / 0.009 reviser), policy entropy (4.39 / 2.05), and verdict parse (>95%), ruling out NaNs, broken reward wiring, or inactive optimizers.
  2. Mid-training entropy collapse at s=1.0s = 1.0 vs. stable training across s∈[0.25,0.5]s \in [0.25, 0.5] (Tables 24 & 26): Across updates 25–100 under s=1.0s = 1.0 isolation, adapter gradient norms climb (0.022/0.009 → 0.077/0.042) alongside synchronous entropy collapse in the isolated checker (4.39 → 1.30) and reviser (2.05 → 0.82). Seed-52 verdict parse drops to 70.31% (vs. 95.92% untrained), and 7-seed reviser preservation drops to 8.4% (91.6% break rate; Table 26, p. 23). Changing only the scalar readout multiplier to s=0.50s = 0.50 or s=0.25s = 0.25 (under the identical AdamW LR of 10−510^{-5}, batching, and bitwise-identical initial LoRA tensors across all 15 seed pairs 53–67) keeps verdict parse at 93.95%–94.63% and raises seed-52 Final accuracy from 0.61% (s=1.00s = 1.00) to 39.54% (s=0.50s = 0.50) and 43.95% (s=0.25s = 0.25; Table 24).
  3. 3-seed Low-LR (2.5×10−62.5\times 10^{-6}, strain=1.0s_{\mathrm{train}} = 1.0) optimizer control (Appendix N, Figure 6a, p. 23): Testing whether s=0.25s = 0.25 acts like a 4×4\times smaller step size, we trained three fresh Role-Isolated seeds (53–55) with AdamW LR cut 4×4\times to 2.5×10−62.5\times 10^{-6} at strain=1.0s_{\mathrm{train}} = 1.0. Downstream collapse persists (21.33% ±\pm 2.31% final vs. 33.22% solver = -11.89 pp, 4.80% ±\pm 1.32% stop; trailing Dual on 3/3 paired seeds by +11.75 pp, t95t_{95} [+6.32, +17.18], and cross-scale s=0.25s = 0.25 on 3/3 by +14.36 pp). Evaluating those same Low-LR checkpoints at seval=0.25s_{\mathrm{eval}} = 0.25 lifts stop rate from 4.80% to 41.51% (26.52% final) and shifts held-out GSM8K (n=500n = 500) system gain from -3.27 pp (28.60%) to +2.27 pp (34.13%), showing that forward-pass readout scale drives the collapse.
AuthorsOpenReview ↗

Response to Reviewer S6Sq (Part 3/4)

W2 & Q2: Zero-Shot Untrained Workflow Baseline and Fixed-Solver Transplants (Main Tables 3–4, Appendices N–O, Tables 27–29, Figure 6b)

"To anchor the severity of role isolation's degradation, the main text lacks a critical control: running the full 3-role workflow with the frozen backbone, reporting Final, Stop, FA, FR... Does the isolated workflow (7.73%) perform worse than this zero-shot baseline?"

Yes: unscaled role isolation (7.73%) scores 34.02 pp below the zero-shot Untrained workflow (41.75%). We added Untrained workflow (0 updates, ref.) as row 1 of Main Tables 3 and 4 (p. 6) with all requested behavioral metrics, and evaluated both zero-shot and Fixed-Solver Transplant (Untrained Base Solver + Trained VR) baselines across all 15 Llama pairs (seeds 53–67) and 5 Qwen pairs (seeds 42–46) on the 1,472-item main panel, held-out GSM8K (n=500n = 500), and APPS (n=500n = 500), reported in Tables 27–29 (pp. 24–25):

Full 3-Role Workflow / Probe (1,472-item main panel, Main Tables 3–4 & Table 25)FinalSolverSystem GainRedo Acc.StopFAFRParsed
Untrained workflow (0 updates, ref., T=0T = 0)41.75%45.55%-3.80 pp36.72%59.38%31.25%17.43%96.30%
Shared adapter (s=1.0s = 1.0, 7s)30.21%46.47%-16.26 pp28.55%12.64%6.50%40.33%98.78%
Role-isolated (s=1.0s = 1.0, 7s)7.73%25.00%-17.27 pp5.03%6.85%4.07%22.21%97.78%
Dual Routing (0.5 merge, 7s, Main Table 3)34.46%32.47%+1.99 pp39.52%59.37%38.29%11.39%94.88%
Cross-scale s=0.25s = 0.25 (own-only, s53–67, 15s, Table 25)32.55%31.21%+1.35 pp37.58%60.29%32.16%11.23%--
Dual Routing (0.5 merge, s53–67, 15s, Table 25)33.14%30.98%+2.16 pp38.27%57.91%31.32%11.74%--
  1. Zero-shot erosion vs. Dual Routing's positive system gain: Even before training, the untrained Llama-3.1-8B three-role workflow loses 3.80 pp on the main panel (45.55% → 41.75%), 9.20 pp on GSM8K (74.80% → 65.60%), and 1.00 pp on APPS (12.40% → 11.40%). Unscaled s=1.0s = 1.0 RL pushes conditional false rejection to 84.2%–100.0% (7.73% main panel, 13.35% GSM8K, 0.85% APPS), whereas Dual Routing achieves positive system gain (+1.99 pp on Main Table 3 and +2.16 pp across s53–67; +5.20 pp on s45–48 and +3.07 pp on s53–67 GSM8K; +0.15 pp on s53–67 APPS).
  2. Fixed-Solver Transplants (Figure 6b, p. 23; Tables 27–29, pp. 23–25): Pairing the untouched base solver with each method's trained verifier-reviser (VR) evaluates downstream modules directly. Base S + Dual VR maintains untouched base-solver accuracy within seed variance across all evaluations:
    • 1,472-item main panel (matched T=1.0T = 1.0 rollout, 36.48% base solver): 29.18% ±\pm 0.92% vs. 29.42% untrained across 15 seeds s53–67 (+0.62 pp ahead of cross-scale s=0.25s = 0.25 VR 28.56%, t95t_{95} [-0.10, +1.35], 9/15 seeds; 29.52% vs. 28.15% on s53–57, +1.37 pp; 28.58% on s45–48), while Role-Iso VR drops to 0.29% and Shared VR to 6.56%.
    • Llama GSM8K & APPS (s53–67, 15s, Tables 27–28): 66.03% ±\pm 1.99% strict (68.51% lenient) vs. 65.60% (68.00%) untrained on GSM8K (s=0.25s = 0.25 VR 66.12%, Shared VR 56.75%, Role-Iso VR 44.75%) and 10.60% strict (24.68% test-avg) vs. 11.40% (24.83%) untrained on APPS (s=0.25s = 0.25 VR 11.13%).
    • Qwen GSM8K & APPS (Table 29): 80.56% ±\pm 0.70% vs. 80.20% untrained on GSM8K (s=0.25s = 0.25 VR 81.20%, Shared VR 58.68%, Role-Iso VR 33.48%, dualconcat VR 0.96%) and 13.52% vs. 12.40% untrained on APPS (s=0.25s = 0.25 VR 13.84%).
Reviewer_LSShOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

When one LLM plays solver , verifier and reviser with separate LoRA adapters, paper separates which adapters a role reads from and which one its gradient updates. In Dual Routing, the verifier and reviser generate from a merge of their adapters but update only their own. With seven Llama-3.1-8B seeds per arm on 1,472 in-house items, it beats role isolation by 26.7 points ; its 4.25-point edge over shared training isn't significant. The paper's own controls show that most of the gain comes from training at a reduced adapter scale.

Critical strengths

The statistics are unusually rigorous, and the controls actually locate the mechanism. The authors also disclose results that cut against their own story, which makes the paper easy to audit.

Critical weaknesses

The baseline it beats collapses to near zero under a single untuned setting. Training isolation at half scale recovers about 89% of the gap, which leaves roughly 4 points for "reading together". No trained workflow beats the shared solver on its own, and the effect vanishes on Qwen. Three Dual seeds were retrained after their results had been seen, and with the original runs the component effects turn negative.

Questions for authors

What does the untrained model score? Would a tuned isolation baseline close the gap? Why were the seeds retraiend? For future work, test on standard benchmarks and against methods like MALT.

3 comments
AuthorsOpenReview ↗

Response to Reviewer LSSh (Part 1/3)

We thank Reviewer LSSh for the rigorous review and for recognizing our statistical analysis and empirical transparency. Your critique and four questions directly shaped our revision experiments:

  1. "What does the untrained model score?"
  2. "Would a tuned isolation baseline close the gap?"
  3. "Why were the seeds retrained?"
  4. "For future work, test on standard benchmarks and against methods like MALT."

Following suggestion (4), we evaluated 79 checkpoints (>95,000 trajectories) on GSM8K (n=500n = 500) and container-scored APPS (n=500n = 500) across Llama-3.1-8B and Qwen2.5-7B alongside 44 fresh 100-step H200 RL runs (6-point seed-52 sweep, 30 zero-rerun Llama runs across 15 pairs, 3 Low-LR 2.5×10−62.5\times 10^{-6} controls, and 5 Qwen2.5-7B cross-scale s=0.25s = 0.25 runs), fixed-solver transplants, and rank-128 zero-cross-term readouts. Dual Routing is unchanged: its 0.5/0.5 merge sets s=0.25s = 0.25 by construction on each role's own delta under disjoint role-owned writes.

Overview of New Experiments and Manuscript Updates

Question / CritiqueNew Experimental EvidenceRevised PDF Pointers
Q1 & W1: Untrained score & isolating downstream value from solver driftAdded Untrained (0 updates, ref.) as row 1 of Main Tables 3–4 (41.75% Final, 45.55% Solver, -3.80 pp Gain; Role-Iso 7.73% is 34.02 pp below zero-shot). Unanchored 100-step GDPO at LR 10−510^{-5} shifts first-pass solver accuracy (45.55% → 32.47% for Dual Solver, 34.46% Dual Final); pairing the untouched base solver with trained verifier-reviser modules (Base S + Dual VR) isolates downstream control and maintains untouched base-solver accuracy across 15 seeds s53–67 on the 1,472-item main panel (29.18% ±\pm 0.92% vs. 29.42% under matched T=1.0T = 1.0 rollout, +0.62 pp ahead of s=0.25s = 0.25 VR 28.56%, 9/15 positive; 29.52% vs. 28.15% on s53–57; 28.58% on s45–48), Llama GSM8K (66.03% vs. 65.60%), Llama APPS (10.60% vs. 11.40%), and Qwen GSM8K (80.56% vs. 80.20%), whereas Role-Iso VR drops by 20.85 to 46.72 pp (0.29%, 44.75%, 33.48%).§4.1 (Tables 3–4, p. 6), §5 (pp. 8–9), Limitations (p. 10), App. N–O, Fig. 6b (p. 23), Tables 27–28 (p. 24), Table 29 (p. 25)
Q2 & W1: Tuned isolation (cross-scale s=0.25s = 0.25 vs. Low-LR) & Qwen regimeExplicit readout scaling (cross-scale s=0.25s = 0.25) matches Dual on parity, whereas a 4×4\times lower learning rate (Low-LR 2.5×10−62.5\times 10^{-6}, strain=1.0s_{\mathrm{train}} = 1.0) still collapses (-11.89 pp gain, 21.33% final, 4.80% stop). Seed 52 shows a collapse cliff at s=1.00s = 1.00 (0.61%) vs. 39.54%–43.95% across s∈[0.25,0.50]s \in [0.25, 0.50] (untrained s=0s = 0: 42.66%), and 15-pair zero-rerun Llama s53–67 (df=14df = 14) gives Dual 33.14% (+2.16 pp gain) vs. cross-scale s=0.25s = 0.25 32.55% (+1.35 pp gain), Δ=+0.58 pp\Delta = +0.58\text{ pp} (t95t_{95} [-4.48, +5.65], item CI95\mathrm{CI}_{95} [-0.23, +1.40]; Always-revise +0.69 pp). Qwen's +0.91 pp at s=1.0s = 1.0 reflects 94.7% reviser preservation (+0.77 pp predicted); Qwen cross-scale s=0.25s = 0.25 scores 56.74% vs. 60.43% Dual on 1,024-item validation, whereas 2×1.0×2\times 1.0\times dualconcat collapses to 5.66% (+54.77 pp, 5/5 seeds) and on OOD GSM8K Dual beats Qwen Role-Iso at s=1.0s = 1.0 by +38.52 pp (71.96% vs. 33.44%, 5/5 seeds).Abstract (p. 1), §1 (pp. 1–2), §2.3 Eqs. 4–5 (p. 3), §5 & §7 (pp. 8–9), App. M–N, Fig. 5 (p. 20), Tables 24–25 (pp. 21–22), Fig. 6 (p. 23), Table 26 (p. 23), Table 29 (p. 25)
Q3 & W1: Seed reruns & component residualDual-45 experienced two mid-run preemptions, and Dual-45, 47, 48 were restarted after scoring (Appendix K, pp. 17–18). Under unedited first runs, Dual beats unscaled Role-Iso by +23.50 pp (6/7 seeds, Table 21, p. 20) while Dual minus own-half is -1.34 pp (Fig. 5a). Thirty outcome-blind, zero-rerun runs across 15 consecutive pairs s53–67 (Table 25, p. 22) give +0.58 pp Final (t95t_{95} [-4.48, +5.65]) and +0.69 pp Always-revise (+3.17 pp on the 10 prospective pairs s58–67), anchoring the readout-scale account.§3 (p. 5), §5 (p. 8), Limitations (p. 10), App. I, K, M, Table 21 & Fig. 5a (p. 20), Tables 23 & 25 (pp. 21–22)
Q4: Standard benchmarks (GSM8K, APPS) & MALTEvaluated 6 Llama-3.1-8B arms (43 ckpts) and 8 Qwen2.5-7B arms (36 ckpts) on GSM8K math and container-tested APPS code (Tables 27–29, Fig. 6c), and added an architectural and memory-footprint comparison with 3-checkpoint MALT (Motwani et al., 2024) in §6.§6 (p. 9), App. O (pp. 24–25), Fig. 6c (p. 23), Tables 27–29 (pp. 24–25)
AuthorsOpenReview ↗

Response to Reviewer LSSh (Part 2/3)

1. Untrained Workflow Baseline and Fixed-Solver Transplants (Main Tables 3–4 p. 6, Appendices N–O, Figure 6b p. 23)

Row 1 of Main Tables 3 and 4 (p. 6) reports the zero-shot Untrained Llama-3.1-8B-Instruct workflow (0 updates, ref.):

  • Zero-shot reference: On the 1,472-item panel, Untrained scores 41.75% Final, 45.55% Solver, -3.80 pp Gain, 36.72% Always-revise, 59.38% Stop, 31.25% FA, and 17.43% FR; on Llama GSM8K (n=500n = 500) 65.60% / 74.80% (-9.20 pp), and on Llama APPS (n=500n = 500) 11.40% / 12.40% (-1.00 pp). Unscaled Role-Isolated (7.73%) drops 34.02 pp below zero-shot, and Shared (46.47% solver) erases 16.26 pp downstream (30.21% final), whereas Dual Routing achieves +1.99 pp positive system gain (34.46% Final; +2.16 pp across seeds 53–67).
  • Fixed-solver transplants (Figure 6b, Tables 27–29): Over 100 online GDPO updates at LR 10−510^{-5} without a KL anchor, first-pass solver accuracy shifts from 45.55% to 32.47% under Dual Routing (§4.1, Limitations p. 10). Pairing the untouched base solver with each method's trained verifier-reviser (VR) evaluates the downstream modules directly: Base S + Dual VR maintains untouched base-solver accuracy within seed variance across all evaluations: 29.18% ±\pm 0.92% vs. 29.42% across 15 seeds s53–67 on the main panel under matched T=1.0T = 1.0 rollout (+0.62 pp ahead of cross-scale s=0.25s = 0.25 VR at 28.56%, t95t_{95} [-0.10, +1.35], 9/15 seeds; 29.52% vs. 28.15% on s53–57; 28.58% on s45–48), 66.03% strict / 68.51% lenient vs. 65.60% / 68.00% untrained on Llama GSM8K (s=0.25s = 0.25 VR 66.12%), 10.60% vs. 11.40% untrained on Llama APPS (s=0.25s = 0.25 VR 11.13%), and 80.56% vs. 80.20% untrained on Qwen GSM8K (s=0.25s = 0.25 VR 81.20%). Under the same transplant, Role-Isolated VR drops by 29.13, 20.85, and 46.72 pp (0.29%, 44.75%, 33.48%) and Shared VR drops to 6.56%, 56.75%, and 58.68%. Out of distribution, full-RL Dual Routing also achieves +5.20 pp system gain on Llama GSM8K (s45–48), 71.96% on Qwen GSM8K, and 13.52% on Qwen APPS.

2. Tuned Isolation Controls (Readout Scale vs. Learning Rate) and Qwen's Perturbation Threshold (Appendices M–N, Tables 24–26 pp. 21–23, Figures 5–6)

Matching readout scale (cross-scale s=0.25s = 0.25) reaches the same stability regime as Dual Routing, whereas lowering the learning rate 4×4\times (Low-LR) still collapses (Abstract, §1, §2.3, §5, §7):

  • Readout-scale matching vs. 4×4\times lower LR: Expanding (0.5Bv+0.5Br)(0.5Av+0.5Ar)(0.5 B_v + 0.5 B_r)(0.5 A_v + 0.5 A_r) weights each role's own adapter by s=0.25s = 0.25 (§2.3 Eq. 4, p. 3). On single seed 52 (Table 24, p. 21), own-only readout collapses at s=1.00s = 1.00 (0.61%) and remains stable at s=0.25s = 0.25 (43.95%) and s=0.50s = 0.50 (39.54%), vs. 36.01% at s=0.10s = 0.10 and 42.66% untrained (s=0s = 0). Across 30 fresh zero-rerun runs on seeds 53–67 (df=14df = 14, Table 25, p. 22), Dual reaches 33.14% and cross-scale s=0.25s = 0.25 reaches 32.55% (Δ=+0.58 pp\Delta = +0.58\text{ pp}, t95t_{95} [-4.48, +5.65], item CI95\mathrm{CI}_{95} [-0.23, +1.40]; Always-revise +0.69 pp; exact rank-128 zero-cross-term 31.60%), narrowing the initial 5-pair s53–57 estimate (-4.58 pp) to near-zero difference. Cutting AdamW LR 4×4\times to 2.5×10−62.5\times 10^{-6} at strain=1.0s_{\mathrm{train}} = 1.0 across 3 fresh seeds (s53–55, Figure 6a) still collapses downstream (21.33% final vs. 33.22% solver = -11.89 pp, 4.80% stop; behind Dual on 3/3 seeds by +11.75 pp and cross-scale s=0.25s = 0.25 on 3/3 by +14.36 pp), while evaluating those Low-LR checkpoints at seval=0.25s_{\mathrm{eval}} = 0.25 restores a 41.51% stop rate (+2.27 pp gain on GSM8K).
  • Qwen across regimes (+0.91 pp in-domain, +38.52 pp OOD GSM8K, +54.77 pp at 2×1.0×2\times 1.0\times; Table 26 p. 23, Table 29 p. 25, Figure 6a): In-domain at s=1.0s = 1.0, Qwen's isolated reviser preserves 94.7% of correct answers (5.3% break rate vs. Llama's 91.6%), predicting +0.77 pp vs. +0.91 pp observed. Across 5 matched seeds (42–46), Qwen cross-scale s=0.25s = 0.25 scores 56.74% (+2.40 pp gain) vs. 60.43% Dual on the 1,024-item validation panel (81.28% vs. 71.96% GSM8K; 14.36% vs. 13.52% APPS). On GSM8K, Role-Isolated at s=1.0s = 1.0 over-rejects 85.7% (33.44% final, jumping +43.96 pp to 77.40% at seval=0.25s_{\mathrm{eval}} = 0.25), while dualconcat (2×1.0×2\times 1.0\times) collapses across all three benchmarks: 5.66% vs. 60.43% (+54.77 pp, 5/5 seeds), 0.84% GSM8K (+71.12 pp), and 1.60% APPS (+11.92 pp).
AuthorsOpenReview ↗

Response to Reviewer LSSh (Part 3/3)

3. Seed Reruns and Zero-Rerun Confirmation (§3 p. 5, App. I & K, App. M Table 25)

As recorded in Appendix K (pp. 17–18), during expansion from 3 to 7 replicates prior to locking the outcome-blind protocol, the initial Dual-45 run hit two mid-training cluster preemptions, and Dual-45, 47, 48 were each restarted once after scoring their first runs (23.71%, 37.70%, 30.30%) on the panel:

  1. Unedited historical first runs (Table 21 & Figure 5a, p. 20): Retaining all original first runs beats unscaled Role-Isolated (+23.50 pp, t95t_{95} [+11.22, +35.77], 6/7 seeds) and PCGrad-A/B (+26.73 / +29.42 pp, 7/7 seeds), with a Dual minus own-half difference of -1.34 pp (t95t_{95} [-12.52, +9.84], Figure 5a), consistent with the 7-seed s45–51 extension (-0.67 pp).
  2. Outcome-blind, zero-rerun 15-pair confirmation (Table 25, p. 22; Limitations, p. 10): All 30 runs across consecutive seeds 53–67 (df=14df = 14) were trained once from bitwise-identical 1,344-tensor LoRA initializations with zero post-evaluation reruns. Expanding the initial 5 pairs (seeds 53–57: -4.58 pp, t95t_{95} [-13.86, +4.70]) with 10 prospective pairs (seeds 58–67: +3.17 pp, t95t_{95} [-3.36, +9.69]) narrows the 95% tt-interval half-width from ±9.28 pp\pm 9.28\text{ pp} to ±5.07 pp\pm 5.07\text{ pp} and yields +0.58 pp on Final (33.14% vs. 32.55%, t95t_{95} [-4.48, +5.65], item CI95\mathrm{CI}_{95} [-0.23, +1.40]; +0.07 pp across seeds 53–62) and +0.69 pp on Always-revise (t95t_{95} [-0.63, +2.01]), directly anchoring our readout-scale mechanism.

4. Standard Benchmarks (GSM8K & APPS) and Comparison to MALT (§6 p. 9, Appendix O Tables 27–29 pp. 24–25)

  1. 79-checkpoint evaluations on held-out GSM8K (n=500n = 500) and container-scored APPS (n=500n = 500), reported in Tables 27–29:
    • Llama-3.1-8B (43 ckpts, Tables 27–28, p. 24): On same-seed s45–48, Dual achieves +5.20 pp system gain on GSM8K (27.45% vs. 26.15% Shared and 13.35% Role-Iso; 4/4 seeds ahead of Role-Iso) and 7.75% strict (13.39% test-avg) on APPS, ahead of Role-Iso (0.85%) by +6.90 pp (t95t_{95} [+4.54, +9.26], 4/4 seeds). Across all 15 zero-rerun pairs on seeds 53–67 (df=14df = 14), Dual (30.29% GSM8K / 6.71% APPS, +0.15 pp APPS gain) and cross-scale s=0.25s = 0.25 (26.64% / 6.56%; paired Δ=+3.65 pp\Delta = +3.65\text{ pp}, t95t_{95} [-4.03, +11.34] on GSM8K and +0.15 pp+0.15\text{ pp}, t95t_{95} [-0.83, +1.13] on APPS) both prevent the 84.2%–100.0% conditional false-rejection collapse of s=1.0s = 1.0 training while preserving base-solver accuracy under transplant (66.03% / 66.12% on GSM8K; 10.60% / 11.13% on APPS).
    • Qwen2.5-7B (36 ckpts, s42–46, Table 29, p. 25): On GSM8K, Dual reaches 71.96% (+49.20 pp gain, 14.2% FRcond\mathrm{FR}_{\mathrm{cond}}), beating Role-Iso (33.44%, 85.7% FRcond\mathrm{FR}_{\mathrm{cond}}) by +38.52 pp (t95t_{95} [+9.20, +67.84], 5/5 seeds) and dualconcat (0.84%) by +71.12 pp (cross-scale s=0.25s = 0.25: 81.28%; Base S + Dual VR 80.56% vs. 81.20% s=0.25s = 0.25 VR and 80.20% untrained). On APPS, Dual reaches 13.52% (29.15% test-avg; cross-scale s=0.25s = 0.25: 14.36%; untrained 12.40%) vs. 1.60% for dualconcat (+11.92 pp, t95t_{95} [+8.86, +14.98]).
  2. Comparison to MALT (§6, p. 9): Section 6 contrasts our single-backbone interface with MALT (Motwani et al., 2024), which trains three separate full-model checkpoints (3×3\times weight memory and serving footprint) with offline trajectory-tree credit assignment. Dual Routing operates online over one frozen backbone at 1×1\times active forward rank, using s∈[0.25,0.5]s \in [0.25, 0.5] readout under role-owned writes to prevent downstream collapse without 3×3\times full checkpoints, and is orthogonal to trajectory-tree credit assignment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Response to All Reviewers

We thank Reviewers Spg3, S6Sq, and LSSh. We ran 44 fresh 100-step H200 RL runs (6-point seed-52 sweep, 30 matched-seed Llama-3.1-8B runs across 15 pairs, 3 Low-LR 2.5×10−62.5\times 10^{-6} controls, and 5 Qwen2.5-7B cross-scale s=0.25s = 0.25 runs), fixed-solver verifier-reviser (VR) transplants, rank-128 zero-cross-term readouts, and 79-checkpoint OOD evaluations (>95,000 trajectories) on GSM8K (n=500n = 500) and APPS (n=500n = 500). Dual Routing is unchanged: its 0.5/0.5 merge sets s=0.25s = 0.25 by construction on each role's own delta under disjoint role-owned writes at 1×1\times active forward rank.

Headline Matched-Seed & Fixed-Solver Comparison

Benchmark / SettingRole-Iso (s=1.0)Cross-scale s=0.25Dual (0.5 merge)Dual - s=0.25
Llama 1,472-item panel (s53–67, 15 pairs)7.73% (s42–48)32.55% (+1.35 gain)33.14% (+2.16 gain)+0.58 pp ([-4.48, +5.65])
Llama Fixed-Solver Base S + VR (15 pairs)0.29% (s45–48)28.56% ±\pm 1.00%29.18% ±\pm 0.92%+0.62 pp (9/15 seeds > 0)
Llama held-out GSM8K (n=500n=500, 15 pairs)13.35% (s45–48)26.64% (Fixed-S: 66.12%)30.29% (Fixed-S: 66.03%)+3.65 pp (Fixed-S: -0.09 pp)
Llama held-out APPS (n=500n=500, 15 pairs)0.85% (s45–48)6.56% (-0.08 gain)6.71% (+0.15 gain)+0.15 pp (Fixed-S: 10.60% / 11.13%)
Qwen 1,024-item panel (s42–46, 5 pairs)5.66% (dualconcat)56.74% (+2.40 gain)60.43% (+2.38 gain)+3.69 pp ([-6.02, +13.40])

Summary of Revisions

  1. Rollout-scale stability (Spg3 W1, S6Sq W1/Q1/Q3, LSSh W1/Q2; Abstract, §1, §2.3 Eqs. 4–5 pp. 1–3; §5, §7 pp. 8–9; App. M–N, Figs. 5–6, Tab. 24–26 pp. 20–23): Seed-52 ladder locates a collapse cliff at s=1.00s = 1.00 (0.61%) vs. 39.54%–43.95% across s∈[0.25,0.50]s \in [0.25, 0.50] (untrained s=0s = 0: 42.66%). Expanding matched-seed Llama pairs 3×3\times to 15 consecutive pairs (s53–67, 30 runs, df=14df = 14) narrows 5-pair s53–57 (-4.58 pp) to +0.07 pp (s53–62), +3.17 pp (s58–67), and +0.58 pp overall (item CI95\mathrm{CI}_{95} [-0.23, +1.40]; Always-revise +0.69 pp; 7-seed own-half -0.67 pp). Low-LR (2.5×10−62.5\times 10^{-6}, strain=1.0s_{\mathrm{train}} = 1.0) still collapses (21.33% vs. 33.22% solver = -11.89 pp, 4.80% stop; +11.75 pp for Dual, 3/3 seeds), whereas seval=0.25s_{\mathrm{eval}} = 0.25 lifts stop rate to 41.51% and GSM8K gain from -3.27 pp to +2.27 pp. Rank-128 zero-cross-term readout matches Dual within 0.5 pp (31.60% main, 38.80% Llama GSM8K, 72.08% Qwen GSM8K).
  2. Zero-shot reference & fixed-solver transplants (S6Sq W2/Q2, LSSh W1/Q1, Spg3 W3; §4.1 Tab. 3–4 p. 6; §5 pp. 8–9; Limitations p. 10; App. N–O, Fig. 6b, Tab. 27–29 pp. 24–25): Added Untrained workflow (0 updates, ref.) to Main Tables 3–4 (41.75% Final, 45.55% Solver, -3.80 pp Gain, 59.38% Stop, 31.25% FA, 17.43% FR; Role-Iso 7.73% is 34.02 pp below zero-shot; Dual 34.46% Final, +1.99 pp Gain). Pairing untouched Base Solver with trained VR (Base S + Dual VR) preserves base-solver accuracy everywhere: main panel 29.18% vs. 29.42% (+1.37 pp on s53–57), Llama GSM8K 66.03% vs. 65.60%, Llama APPS 10.60% vs. 11.40%, Qwen GSM8K 80.56% vs. 80.20% (s=0.25s = 0.25 VR 81.20%), whereas Role-Iso VR drops by 20.85–46.72 pp (0.29%, 44.75%, 33.48%) and Shared VR to 6.56%, 56.75%, and 58.68%.
  3. Held-out math, code, & MALT (Spg3 W2/W3/Q2, S6Sq W2, LSSh Q4; §5–§6 pp. 8–9; App. O, Fig. 6c, Tab. 27–29 pp. 24–25): Across 6 Llama (43 ckpts) and 8 Qwen (36 ckpts) arms, same-seed Llama (s45–48) Dual hits +5.20 pp GSM8K gain (27.45% vs. 26.15% Shared, 13.35% Role-Iso, 4/4 seeds) and 7.75% APPS (+6.90 pp over Role-Iso 0.85%, t95t_{95} [+4.54, +9.26]). Qwen (s42–46) Dual hits 71.96% GSM8K (+38.52 pp over Role-Iso 33.44%) and 13.52% APPS (s=0.25s = 0.25: 81.28% / 14.36%). MALT is contrasted in §6.
  4. Cross-backbone regime & first runs (Spg3 Q1, S6Sq Q3, LSSh W1/Q3; §3 p. 5; §5 pp. 8–9; App. I–K & M–O, Tab. 21–23, 25–26, 29, Fig. 6a pp. 20–25): Qwen's +0.91 pp in-domain gap (74.31% vs. 73.41%) follows FR⋅(1−rpres)\mathrm{FR}\cdot(1 - r_{\mathrm{pres}}): Qwen's s=1.0s = 1.0 reviser preserves 94.7% (5.3% break; +0.77 pp predicted) vs. Llama's 8.4% (91.6% break). Qwen 1,472-item fixed-solver is 69.78% vs. 69.74% (s=0.25s = 0.25), while 2×1.0×2\times 1.0\times dualconcat collapses to 5.66% (1,024-item), 0.84% (GSM8K), and 1.60% (APPS). First runs (+23.50 pp, 6/7 seeds) are in Table 21.

Full point-by-point analyses and tables are in our individual replies below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.