acceptodds
Withdrawn from ICLR 2027

T-Router: Learning Thalamic Routing for Reasoning with Parameter-Efficient Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Parameter-efficient reinforcement learning aims to improve reasoning with a compact trainable interface to a pretrained model. We introduce the Thalamic Router (T-Router), which concentrates adaptation on the reuse of completed computations. A compressed, addressable bank preserves block changes; a depth-recurrent controller conditions their selection and relative-scale writeback. This coupling gives thalamic context-dependent routing a concrete computational form: learn which earlier contributions a receiving layer uses, and with what influence. Correctness rewards train the interface while preserving backbone parameters and layer order. On an 8.95B-parameter backbone, T-Router allocates 41.73M parameters—0.466% of the backbone—and achieves MathAvg after GSM8K RL, compared with for full-parameter GRPO across three evaluation rounds. At a comparable parameter budget and with matched retries, it exceeds LoRA's MathAvg, improving all three task families and raising mean AIME accuracy from 48.33 to 60.56. Capacity-controlled comparisons favor addressable block changes and recurrent context; separate search training extends the interface to tool-mediated reasoning. These results establish controlled computation reuse as an effective route to parameter-efficient reasoning reinforcement learning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
parameter-efficient reinforcement learning, large language models, mathematical reasoning, cross-layer routing, computation reuse, thalamic routing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_tMYsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes T-Router, a compact adaptation module for reinforcement learning around a frozen language model. It stores compressed changes from completed decoder blocks. A controller recurrent over depth conditions attention over these records and a signed writeback gate. RMS calibration sets each intervention's magnitude relative to the receiving residual. All backbone layers still execute in order. On Qwen3.5-9B-Base, the module allocates 41.73M parameters, or 0.466% of the backbone. After GSM8K training, it reports 83.64 MathAvg, compared with 73.79 for full-parameter GRPO and 77.28 for LoRA with matched retries. The paper also presents component ablations, mathematical characterizations of the interface, and separately trained search agents.

Critical strengths
  1. The mathematical reasoning results are substantial at a small trainable parameter budget. Table 1 includes a relevant LoRA baseline with approximately the same parameter allocation and the same retry rule. T-Router improves MathAvg by 6.36 points and improves all three task families. Its MATH-500 score increases from 87.93 to 92.73, and mean AIME accuracy increases from 48.33 to 60.56. This is useful evidence for the adaptation interface beyond a comparison against full fine-tuning alone.

  2. The design has an explicit and interpretable computational structure. Equations 2, 5, and 7 separate stored content, recurrent control, and relative writeback magnitude. Appendix B correctly distinguishes a local intervention bound from a guarantee about final model behavior. It also identifies the implemented policy gradient as a single-evaluation, sequence-normalized update rather than claiming that clipping provides an active trust region. The capacity-controlled comparisons in Table 20 provide initial evidence that both stored block changes and recurrent context matter.

Critical weaknesses
  1. The evidence attributing the gain specifically to the proposed organization of adaptation capacity needs stronger experimental control. Tables 1 and 20 report three decoding seeds, while Appendix A.6 and C.12 do not report independent training runs. Table 11 gives a configurable recipe, but does not specify the complete settings and tuning budgets for each baseline. The exact architecture and inputs of the capacity-matched MLP in Table 20 are also not specified. This makes the central recurrent-control comparison difficult to reproduce. Training exposure matters here: removing retries lowers MathAvg from 83.64 to 62.53, while recorded cost falls from 121.23 to 37.92 GPU-hours (Table 21). Matching the retry rule for LoRA is valuable, but the realized generation counts and costs are still missing. The current evidence supports a parameter-count advantage under the reported recipe. It is less decisive about robustness across training runs or the benefit at a fixed training budget.

  2. The search evaluation has unresolved split and metric issues. Appendix A.7 states that "ASearch Test" is used for validation every 50 updates. It also identifies eight exact question-answer duplicates between the reconstructed training and validation splits. This is 2.87% of the holdout. The improvement over matched-retry LoRA is only 1.18 points on this validation score. These observations do not prove that duplication causes the improvement, but they prevent a clean generalization interpretation without the actual filtered splits and checkpoint-selection rule. On the independent BrowseComp Plus benchmark, the corresponding difference is only 0.08 points. In addition, Tables 2 and 19 label this metric token-F1, whereas Appendix A.7 says the default returned score includes repetition, length, and tag penalties. The paper never clearly identifies whether those penalties were removed for the reported results. This ambiguity materially affects the search claim.

Questions for authors
  1. Clarification and validation request: How many independent training runs underlie Tables 1 and 20? Please provide each baseline's actual optimizer settings, checkpoint-selection procedure, and realized generated-token budget. Do the main gains persist across independent training runs?

  2. Clarification request: What precisely replaces the recurrent controller in the matched-MLP ablation? Please specify its inputs, dimensions, output, initialization, and which maps are retained. Also define how the historical IID estimates in Tables 22-24 were obtained and whether those benchmarks were used to select the final configuration.

  3. Clarification and validation request: For Table 2, what are the actual post-filter evaluation counts, train-validation overlaps, and checkpoint-selection rule? Were the reported scores computed from raw token-F1 or the penalized reward? Please report the comparison on a deduplicated holdout and identify the exact scorer and tool configurations used.

2 comments
AuthorsOpenReview ↗

Response to Reviewer tMYs (1/2): Training Controls and Matched MLP

Although we have decided to withdraw this submission, we would still like to address your comments and clarify the points you raised.

Table 1 reports a 6.36-point MathAvg gain for T-Router over LoRA-r16 at a similar parameter count under the same retry rule. We understand that your main concern is whether this improvement is consistent across independent training runs and whether it persists under matched training costs. We clarify the relevant experimental details and limitations below.

Training runs and budgets (Tables 1, 11, 20, and 21). Seeds 42, 43, and 44 are decoding seeds, rather than independent training seeds. Each round generates one answer per question, and the reported SD measures variation across these decoding rounds. Tables 1 and 20 do not report the number of independent training runs behind each row. Thus, these error bars characterize decoding variability and should not be interpreted as evidence of robustness across independent training runs. We agree that this distinction is important when assessing the reproducibility of the reported gains.

For the full-parameter GRPO run that scored 73.79 MathAvg, we used TRL’s GRPOTrainer, eight-GPU FSDP, and BF16. We trained all 8.954B parameters for one epoch on all 7,473 GSM8K questions, kept the final partial batch, and evaluated the final checkpoint at step 3,737. Each full update used two prompts with four completions per prompt. Training used temperature 0.7, top-p 0.95, and a 512-token completion limit. The learning rate was 1e-6, with one warmup step, KL coefficient 0.02, and clipping parameter 0.2. The 1e-4 rate in Table 11 belongs to the configurable adapter recipe.

T-Router and matched-retry LoRA each begin with four completions per prompt. If all rewards are equal, they can generate up to three replacement groups. The number of replacements depends on the model. Removing retries from T-Router changes MathAvg from 83.64 to 62.53 and training cost from 121.23 to 37.92 GPU-hours, indicating that this comparison does not hold compute fixed. Furthermore, applying the same retry rule does not guarantee equal completion or token counts, including discarded attempts. The remaining per-method optimizer settings, checkpoint choices, tuning budgets, and realized generation costs are not established by the reported tables. We therefore agree that the current evidence does not establish a fully compute-matched comparison, and we distinguish this limitation from the parameter-matched comparison reported in Table 1.

The matched MLP (Table 20). The stateless variant sets the recurrent context to zero and skips the slot update and readout. A two-layer MLP with a SiLU activation takes only the current token’s 4,096-dimensional residual. Its 256-dimensional output is added to the source-selection query, without entering the writeback gate. The first weight matrix uses Xavier-uniform initialization; the second matrix and both biases start at zero.

The block compressors, source and layer identities, query/key/value maps, writeback projections, RMS calibration, and signed gate are retained. If these maps are unchanged, the counts in Tables 6 and 21 imply a hidden width of 622. We emphasize that this value is reconstructed from the published counts and implementation, rather than verified from the original run configuration. The MLP variant scores 72.80 with 41,731,978 parameters, compared with 83.64 and 41,730,332 parameters for the recurrent model.

AuthorsOpenReview ↗

Response to Reviewer tMYs (2/2): Historical Estimates and Search Evaluation

Historical IID estimates (Tables 22–24). The dagger marks historical GSM8K and MATH-500 estimates and aggregates derived from them. It does not apply to annual AIME scores or the full T-Router reference. Earlier configuration screening used 100 questions each from GSM8K and MATH-500 and all 30 questions from each AIME year, so these benchmarks were visible during development. However, the screening sample sizes alone do not specify how the daggered estimates were calculated or which scores determined the final configuration. We therefore avoid making stronger claims about the precise extent of potential selection effects than the reported information supports.

Search data and scores (Tables 2 and 19). The “ASearch Test” column reports validation scores, evaluated every 50 updates. Appendix A.7 reconstructs 13,706 training and 279 validation records before filtering. Of the 279 validation records, 8 exactly match a training question and answer. BrowseComp Plus is a separate evaluation with 830 source questions before filtering. T-Router and matched-retry LoRA score 73.98 vs. 72.80 on ASearch validation and 36.98 vs. 36.90 on BrowseComp Plus. We acknowledge that the 0.08-point difference on the independent benchmark is too small to establish a reliable advantage.

The default scorer extracts the last complete <answer> span and takes the highest token-F1 across reference answers. The returned reward multiplies F1 by repetition, length, and answer-tag penalties. Table 2 does not identify whether it aggregates raw F1 or the penalized reward. Both default tools retrieve ten documents per call. ASearch uses e5-base-v2 over Wikipedia with a Qwen3-1.7B summarizer; BrowseComp Plus uses Qwen3-Embedding-8B over its own corpus with Qwen3-8B. These repository defaults provide implementation context, but do not establish the actual scorer and tool settings, checkpoint-selection rule, or post-filter sample counts and overlap for the reported runs.

We appreciate the detailed questions concerning training controls, data construction, and evaluation methodology. They highlight several aspects where additional reporting and experiments would strengthen the evidence and improve reproducibility.

Reviewer_tAaxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Thalamic Router (T-Router), a parameter-efficient reinforcement learning method. It stores block-level changes in a compressed, addressable memory bank and uses a depth-recurrent controller to determine which earlier contributions each receiving layer uses and how strongly they influence it. Experiments on Qwen3.5-9B-Base across four math benchmarks and two search-agent benchmarks report advantages over LoRA-based RL and full-parameter training.

Critical strengths

NO obvious strength.

Critical weaknesses
  1. The evaluation protocol is insufficiently specified, making the results difficult to reproduce. Essential details include the sampling temperature, evaluation metric (e.g., whether avg@kk is used and the value of kk), and maximum generation length.

  2. The evaluation is limited to Qwen3.5-9B-Base on four math and two search-agent benchmarks. At least two of these benchmarks appear close to saturation for this model, limiting its ability to distinguish between methods. Experiments across additional domains, model families, and model sizes would help establish the method’s generality.

  3. The full-parameter RL baseline appears unexpectedly weak, raising concerns about whether it was adequately tuned and trained. The authors should provide its training configuration and learning curves, and verify that the baseline is sufficiently optimized to support a fair comparison.

Questions for authors
  1. The term “thalamic” may be unfamiliar to readers and is unclear when first introduced. The authors should briefly explain its meaning and connection to the proposed routing mechanism at its first occurrence in the introduction.
7 comments
AuthorsOpenReview ↗

Formal Concern Regarding Reviewer tAax's Assessment(1/2)

We respectfully raise concerns regarding Reviewer tAax’s assessment of our paper. The reviewer assigns a score of 1 across presentation, informativeness, soundness, and the overall recommendation, while listing “NO obvious strength.” as the entirety of the strengths section. Although the review accurately summarizes our method and acknowledges the reported improvements, it does not appear to engage with either of the two main-text comparisons that directly evaluate our central contributions.

Table 1 compares methods with similar numbers of trainable parameters under the same retry rule. T-Router achieves 83.64 MathAvg with 41.73M trainable parameters, compared with 77.28 for LoRA with 43.28M parameters, demonstrating consistent improvements across all three mathematical task families. Table 3 further examines the effect of the information stored in the source bank. Under exactly the same parameter budget, block-change memory achieves 83.64 MathAvg, compared with 67.46 for hidden-state memory. These two comparisons provide direct empirical evidence for the proposed method and its architectural design. However, the review does not identify any methodological issue with these comparisons or explain why the corresponding findings do not constitute strengths of the paper. We therefore find the categorical assessment of “NO obvious strength” insufficiently substantiated.

Evaluation protocol. We generated one answer per question in each of three rounds, using seeds 42, 43, and 44, and report the mean and sample standard deviation across rounds. For each round, we first averaged AIME 2024 and 2025 accuracy, then calculated MathAvg by equally weighting GSM8K, MATH-500, and the AIME mean. Section 4.1, Equation 10, and Appendix Tables 10 and 12 define these quantities. All methods used temperature 0.7 and top-p 0.95. The evaluation generation limits were 8,192 tokens for GSM8K (1,319 questions) and MATH-500 (500 questions), and 32,768 tokens for AIME 2024 and 2025 (30 questions each). The 512-token limit was used for training rollouts.

Benchmark saturation and scope. The review suggests that “at least two” benchmarks may be close to saturation, but does not specify which benchmarks or explain how this concern affects the interpretation of our results. Table 1 shows improvements over matched-retry LoRA from 87.93 to 92.73 on MATH-500 and from 48.33 to 60.56 on mean AIME accuracy. In particular, the 12.23-percentage-point improvement on competition mathematics provides relevant evidence that these benchmarks can still distinguish the evaluated methods. We would have appreciated a more specific discussion of why these results might nevertheless be insufficient to support our claims. We evaluated Qwen3.5-9B-Base and also report auxiliary results on MuSR, GPQA-D, IFEval, and WikiText-2 in Appendix C.9, Table 25. Evaluating additional model families would certainly broaden the scope of the study, but this suggestion is distinct from identifying a methodological weakness in the comparisons conducted on the evaluated backbone.

AuthorsOpenReview ↗

Formal Concern Regarding Reviewer tAax's Assessment (2/2)

Full-parameter GRPO baseline. The run producing 73.79 MathAvg used TRL’s GRPOTrainer with eight-GPU FSDP and BF16. It was trained for one epoch on all 7,473 GSM8K training questions, completing 3,737 optimizer steps. We evaluated the final checkpoint. Each full batch contained two prompts with four completions each. The learning rate was 1e-6, with one warmup step, KL coefficient 0.02, and clipping parameter 0.2.

Mean training rewards over four consecutive intervals were 86.87%, 91.77%, 91.72%, and 91.11%. Compared with the frozen backbone, full-parameter GRPO improved MathAvg from 65.33 to 73.79 and mean AIME accuracy from 31.67 to 44.44. The review characterizes this baseline as “unexpectedly weak,” but does not provide a comparable reference result or identify a specific configuration issue that would explain this assessment. We agree that further training details and optimization diagnostics can be useful. However, even if the baseline warrants further investigation, this concern does not directly address the independent comparison between T-Router and LoRA, which shows a 6.36-point MathAvg improvement under the same retry rule. We believe this central comparison merits an independent assessment rather than being subsumed by concerns about a separate baseline.

Meaning of “thalamic.” Introduction lines 47–53 explain the motivation for this terminology. Thalamic circuits regulate communication between cortical areas according to task demands. In T-Router, each receiving layer uses its current state and earlier-layer history to select completed block changes and control their writeback strength. Figure 1 illustrates this correspondence.

Overall, we are concerned that the assessment does not adequately engage with the central empirical evidence presented in the main text. In particular, the claim of “NO obvious strength” is difficult to reconcile with the two direct comparisons in Tables 1 and 3, neither of which is substantively discussed. Similarly, the soundness judgment that the identified concerns “do not appear easy to address” is not accompanied by a specific methodological defect that would render the reported results unreliable or undermine the paper's central conclusions. We respectfully ask that the basis for these categorical judgments be reconsidered in light of the evidence provided in the paper. Our concern is not that the reviewer raised critical questions, but that the severity of the assessment does not appear to be supported by a corresponding technical analysis.

  1. "512-token limit was used for training rollouts." No one will use such a short training rollout for RL.
  2. " We generated one answer per question in each of three rounds, using seeds 42, 43, and 44, and report the mean and sample standard deviation across rounds. F" Typically, we will report avg@32 of AIME.
  3. "7,473 GSM8K training questions, completing 3,737 optimizer steps". No one will train on GSM8K for 3.7K steps (obviously the training batch size is ill-specified).

I believe my assessment fully reflects the quality of this paper. I maintain my score.

Reviewer_o6TeOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper provides T-Router, a set of trainable parameters that improves reasoning capabilities of Large Language Models using Reinforcement Learning. The proposed method introduces a source bank that stores information obtained from residual streams of earlier layers, then uses a routing and depth controller to adaptively select what to pass on for later layers. The information obtained from earlier layers is hence "re-used" to facilitate better re-weighting of earlier-blocks and training. Empirically, the paper demonstrates gains in math tasks (GSM8K, MATH500 and AIME) and search tasks (BrowseCompF1, ASearch) with only a small fraction of additional parameters.

Critical strengths
  • To the best of my knowledge and understanding, learning a routing mechanism that leverages a memory bank of previously completed computations seems to be an adequate approach for improving reasoning.
  • The ablation of the proposed method itself is thorough and well thought-out.
  • The method is parameter-efficient, requiring only a small percentage of additional parameters to achieve gains, especially in AIME math problems. The effectiveness of the method is demonstrated by the results.
Critical weaknesses
  • Based on the related work, it seems Delta Attention Residuals (Luo et al., 2026a) is the closest baseline, but the results neither compare nor demonstrate differences in performance.
  • While the proposed method is effective and obtains gains over traditional LoRA, T-Router is a separate set of parameters that cannot be combined into the network like LoRA.
  • The writing of the paper is very imprecise and disorganized. Take the first paragraph as an example:
    • L35 - "A model already contains useful transformations; an additional opportunity is to learn how their intermediate contributions are organized.": What are transformations and contributions? What does it mean for contributions to be organized?
    • The paper attempts to draw connections with neurobiology and how the thalamus organizes information. However, this connection is loose in the introduction, and is not used to make any prediction as to how such a design would improve the overall design of LLMs.
    • Some matrices and variables can be better stated. In equation (3), A is not clearly defined in the main paper.
  • The paper contains figures that are just repeating the results in other tables, specifically it's unclear what some figures provide that aren't already in other figures and tables. For instance, Figure 3c and Table 2, or Figure 4 and Table 3.
Questions for authors
  1. What is the right interpretation of Table 3? It seems to suggest that the method is brittle, where each component of the T-Router configuration is absolutely essential for the method.
  2. Maybe I missed this, but can the authors provide an estimate of the cost in inference time? That is, how much additional time was needed during inference when T-Router is used vs not used.
2 comments
AuthorsOpenReview ↗

Response to Reviewer o6Te (1/2): Method and Ablation Interpretation

Although we have decided to withdraw this submission, we would still like to address your comments and clarify the points you raised.

Thank you for recognizing the AIME gains and for your questions about Table 3. We agree that the comparison with Delta Attention Residuals and the inference-time cost deserve discussion beyond parameter-count comparisons.

Terminology and relation to Delta Attention Residuals. In line 35, “transformation” refers to the computation performed by a frozen decoder block. Its “contribution” is the difference between the residual leaving and entering that block, measured at the same token position along the actual forward pass (Eq. 2). “Organizing” these contributions means allowing a later layer to select a mixture of earlier block changes and control how strongly it is added to that layer’s input. Every backbone layer still executes in order.

As you note, Delta Attention Residuals is the closest comparison and already attends to sublayer or block changes. The distinction in T-Router is that each completed block change is stored as a compressed record with a learned source identity. A separate set of recurrent slots tracks each token’s history across layers. The receiving layer reads these slots to determine which records to retrieve and the sign and strength of the retrieved mixture’s writeback relative to its own residual. We emphasize that Table 3 evaluates these design choices within T-Router; it does not constitute a direct performance comparison with Delta Attention Residuals.

Interpreting Table 3. The two capacity-controlled rows are intended to isolate design choices that cannot be explained by parameter count alone. At exactly 41,730,332 parameters, storing completed block changes gives 83.64 MathAvg, compared with 67.46 for hidden-state memory. The stateless MLP variant has 1,646 more parameters and scores 72.80. Importantly, it still exceeds the frozen backbone’s 65.33 (Table 1), indicating that removing recurrence reduces the improvement but does not eliminate it.

We believe these results support the importance of stored content and depth history under the reported training recipe. At the same time, we agree that they should not be interpreted as demonstrating that every component is indispensable in every setting or that a fresh training run would necessarily exhibit instability. The no-retry row also changes the generated training data and reduces recorded cost from 121.23 to 37.92 GPU-hours (Table 21), so it should be interpreted separately from the architectural ablations.

AuthorsOpenReview ↗

Response to Reviewer o6Te (2/2): Inference Cost and Presentation

Inference time and LoRA merging. A trained LoRA update can be folded into a fixed weight matrix. In contrast, T-Router must still execute at inference because its source weights and writeback gate depend on the current token and preceding blocks. Appendix A.10 provides a partial operation count: eight compressions and 28 writeback projections add 37.75 million multiply–accumulates per token position, excluding routing and controller work. The source bank, controller slots, and block anchor occupy 31 KiB per position in FP32, excluding temporary buffers. We clarify that these figures represent analytical operation counts and storage requirements, rather than measured prefill or cached-decoding latency.

Biological motivation. The thalamic analogy motivates a testable design hypothesis: if a receiving layer benefits from context-dependent communication, fixing source preferences or removing token-dependent control of writeback strength should reduce accuracy. The corresponding ablations score 64.98 and 63.78 MathAvg, respectively, compared with 83.64 for the full model. These experiments are intended to evaluate the engineering hypothesis underlying our architecture, rather than to establish a comprehensive model of thalamic circuitry.

Notation. In Eq. 3, AhA_h and AmA_m are separate learned, bias-free projections. AhA_h maps the current residual from 4,096 to 256 dimensions; AmA_m maps the mean visible source record from 256 to 256 dimensions (Appendix A.2, Table 6).

Figures. Figure 3(c) presents the paired search differences from Table 2 visually. Panel (d) of Figure 4 shows accuracy against training cost, making the no-retry tradeoff more directly visible than in Table 3 alone.

We appreciate your questions, which help clarify the distinctions between the proposed design, its empirical evaluation, and the limitations of the current analysis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.