acceptodds
Under review as a conference paper at ICLR 2027

Why Do Efficient Architectures Learn Expensive Reasoning?

Abstract

Hybrid architectures combining linear and softmax attention increasingly undergo reinforcement learning from verifiable rewards (RLVR), yet architecture selection still relies on pre-training metrics. We ask whether long, explicit state-tracking trajectories selected by correctness-only RL reflect architectural requirements or training-induced strategies. We study parameter-matched 126M and 325M attention, linear-attention, and hybrid models under matched pre-training, supervised fine-tuning, and RL on verifiable working-memory tasks. Several hybrids adopt per-step state summaries, increasing output length 2.7× and inference time 2.8–3.1×. Three controlled analyses separate selection from necessity. First, independently trained terse policies match or outperform long policies for several architecture–task pairs, but lose 5–16 percentage points for other hybrids. Second, corrupting summaries reduces accuracy by 36–61 percentage points, showing that dependence within a learned policy does not establish necessity across policies. Third, although summary trajectories are less accurate overall throughout training, they gain an advantage within the mixed-outcome groups that drive policy updates, reinforcing their selection. Stratifying advantages within each (prompt, style) removes this between-style term and prevents switching in all nine runs across two scales and two tasks, with accuracy within one percentage point of the summary-forbidden control. Experiments varying pre-training seeds and task difficulty support the same mechanism: bases with similar pre-training loss differ in state fidelity, and switching depends on terse-policy failures and the conditional advantage of summaries within the training band. These RL-selected trajectories reverse the hybrids’ deployment-cost advantage, raising inference time from 0.6× to 1.6–2.0× that of full attention. Our findings show that efficiency depends on the interaction between architecture and post-training strategy selection, which pre-training metrics alone fail to capture. We release generators, checkpoints, per-item records, and an online diagnostic that predicts switching.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Architectures, Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KD6WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Using 126M/325M models, the paper studies how outcome-reward RL selects terse or state-summary trajectories and changes inference cost. Independently trained terse policies and prefix edits distinguish alternative training paths from a frozen policy's dependence on summaries. Two profiled hybrids lose their measured speed advantage; shortening incurs architecture- and task-dependent accuracy costs, while the proposed style-stratified normalization (SSN) prevents the transition from terse reasoning to long trajectories with repeated state summaries in all nine tested runs.

Critical strengths

The paper addresses an important efficiency question through shared recipes within each scale, several recurrent mixers and additional initialization seeds. Wall-clock measurements at three batch sizes demonstrate a concrete consequence of post-training trajectory choices (Table 1).

Separating independently trained terse policies from frozen-policy prefix interventions is a valuable design choice. FORBID, SFT-style changes and SSN provide complementary evidence of training-path dependence, with failed non-inferiority tests and seed-sensitive results reported transparently.

Critical weaknesses

The distinctive mechanism remains insufficiently isolated. Figure 2 pools VARTRACK and SORT, despite state-style accuracy being 8–26 pp worse on SORT (Appendix C); SSN changes both advantage scaling and gradient sparsity, and positive advantages also occur without switching (§5). These findings do not yet separate the proposed feedback from task composition and strategy selection already studied in Krishnamurthy et al., Select and Improve (2026), which is missing from the related work.

The practical efficiency evidence remains narrow. The 126M/325M results are useful controlled observations, but pretraining histories differ across scales (§2; Appendix E). Higher-accuracy SFT bases at 325M remain terse and attain 94–96% accuracy (§3.1), leaving unresolved whether switching persists beyond a particular competence/difficulty regime. The authors acknowledge that generalization to production models is untested (§8); this limits the deployment significance of the cost reversal without invalidating the small-model findings.

Questions for authors

Task and capability controls: Does the accuracy/advantage reversal persist within each task/difficulty when comparing styles on the same prompts? After matching starting SFT accuracy and relative task difficulty, does switching still differ across architectures/scales? The current cross-scale comparison also changes pretraining history.

Causal validation: Does style-wise centering with the original prompt-level standard deviation still prevent switching? A random-partition control with comparable gradient sparsity and update magnitude would help distinguish the proposed mechanism from reduced or rescaled updates.

Comparative validation: How do L1/LCPO or the solvable-problem second stage of Concise Reasoning compare with SSN/FORBID on the same held-out items, reporting accuracy and wall time with graph setup amortized? Equation (3) does not implement either method.

Reviewer_gCwsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper asks whether hybrid linear-attention architectures retain their inference-efficiency advantage after outcome-reward RL. The authors pre-train approximately parameter-matched models of several architectures, fine-tune them on synthetic tasks containing both terse and state-summary solution styles, and apply a shared GRPO/DAPO-style RL recipe with correctness-based rewards. Under the main experimental setting, some hybrids, notably GDN 7:1 and KDA 3:1, switch to the longer state-summary style, this behavior varies across pre-training initializations.

Critical strengths
  1. The paper studies an important and timely question: whether the inference-efficiency advantage of linear and hybrid architectures survives outcome-reward RL post-training.
  2. The authors report negative results transparently.
  3. The experimental design is carefully controlled. All architectures within a scale share the same tokenizer, data order, optimiser and RL recipe, and are approximately parameter-matched, line 070–083, which makes the comparison of post-RL behaviour across architectures clean
Critical weaknesses
  1. Defensive writing is pervasive. Almost every result is immediately followed by a negative qualification, e.g. line 212, line 284... It is hard to tell what the authors actually claim, and this sits oddly next to a much more assertive abstract. The paper would benefit from stating two or three explicit claims, each with clear evidence and scope.

  2. The paper is poorly written and hard to follow. The introduction is very brief. It devotes little space to the background of hybrid architectures and RL post-training, and does not explain why the question matters before moving quickly into contributions and experiments. Readers may find it difficult to understand the motivation. The description of the experimental setup is compressed. Architectures, data, tasks, SFT, the RL recipe and the interventions are compressed into a few short paragraphs.

  3. The abstract overclaims relative to the main text: The abstract claims that style-conditional advantages predict switching across initializations and difficulty bands. However, Section 3.1 reports cases with positive conditional-advantage windows that do not switch. The paper does not establish a reliable predictive criterion or quantify its predictive performance. The abstract therefore appears to overstate the evidence for this statistic's predictive value.

  4. Architecture-level claims are not sufficiently supported by the current initialization coverage. In the 325M experiments, only GDN 7:1 and KDA 3:1 are evaluated across three independent pre-training initializations. For both architectures, only base seed 0 switches to state-summary reasoning, while seeds 1 and 2 remain terse under the same data, SFT, and RL recipes, despite nearly identical pre-training validation losses. These results demonstrate substantial initialization dependence, while the limited number of pre-training seeds prevents a reliable assessment of architecture-level switching tendencies.

Questions for authors
  1. Since only one of three initialisations switches for GDN 7:1 and KDA 3:1, what evidence supports attributing switching to architecture rather than to the starting checkpoint? Would the non-switching architectures stay terse with additional initialisations?

  2. How do you reconcile the abstract’s claim that the statistic predicts switching with the main-text statements and counterexamples showing that it does not?

Reviewer_SKcSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how outcome-reward RL selects reasoning styles in efficient architectures. Using approximately parameter-matched models trained on shared data, the authors compare terse and state-summary trajectories on synthetic working-memory tasks. Separate training interventions and prefix edits distinguish alternative attainable policies from an existing policy’s dependence on summaries. Results show substantial, configuration-dependent accuracy cost trade-offs. Style-conditioned advantage diagnostics and within-style normalization provide a possible explanation and intervention, although the causal mechanism remains unresolved.

Critical strengths
  1. The experiments make a useful distinction between policy dependence and architectural necessity. Matched training comparisons, prefix interventions, additional initializations, and variations in task difficulty and SFT style collectively provide convincing evidence that reasoning style depends on the training path.

  2. The paper connects these behavioral changes to measured inference costs and reports important negative results, including failed non-inferiority tests and predictions that did not hold. This supports a valuable empirical contribution even where the proposed explanation remains incomplete.

Critical weaknesses
  1. The mechanistic evidence does not isolate why switching occurs. The main advantage diagnostics pool VARTRACK and SORT, despite substantial differences in their style-specific accuracy. SSN also changes normalization scale and gradient sparsity alongside between-style mean advantages. Its success therefore does not establish which change prevents switching.

  2. Some headline claims exceed the evidence. KDA’s small mean accuracy difference does not establish non-inferiority, and positive advantage windows also occur without switching. SSN should additionally be positioned against Stratified GRPO, whose within-stratum normalization closely matches the proposed intervention. These issues narrow the predictive and methodological claims, although the empirical findings remain useful.

Questions for authors
  1. Additional evidence: Does the accuracy–advantage reversal persist when VARTRACK and SORT are analyzed separately, with difficulty controlled? Can these diagnostics distinguish runs that switch from those with positive advantages that remain terse?

  2. Additional evidence: Could you separate SSN’s centering effect from its scaling and sparsity effects, for example by retaining the global standard deviation and comparing controls with matched effective gradient magnitudes and nonzero-advantage rates?

  3. Clarification: How does SSN differ from the unblended within-stratum normalization in Stratified GRPO? Please clarify whether its contribution here is methodological or its use as a diagnostic intervention.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.