- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes T-Router, a compact adaptation module for reinforcement learning around a frozen language model. It stores compressed changes from completed decoder blocks. A controller recurrent over depth conditions attention over these records and a signed writeback gate. RMS calibration sets each intervention's magnitude relative to the receiving residual. All backbone layers still execute in order. On Qwen3.5-9B-Base, the module allocates 41.73M parameters, or 0.466% of the backbone. After GSM8K training, it reports 83.64 MathAvg, compared with 73.79 for full-parameter GRPO and 77.28 for LoRA with matched retries. The paper also presents component ablations, mathematical characterizations of the interface, and separately trained search agents.
Critical strengths
-
The mathematical reasoning results are substantial at a small trainable parameter budget. Table 1 includes a relevant LoRA baseline with approximately the same parameter allocation and the same retry rule. T-Router improves MathAvg by 6.36 points and improves all three task families. Its MATH-500 score increases from 87.93 to 92.73, and mean AIME accuracy increases from 48.33 to 60.56. This is useful evidence for the adaptation interface beyond a comparison against full fine-tuning alone.
-
The design has an explicit and interpretable computational structure. Equations 2, 5, and 7 separate stored content, recurrent control, and relative writeback magnitude. Appendix B correctly distinguishes a local intervention bound from a guarantee about final model behavior. It also identifies the implemented policy gradient as a single-evaluation, sequence-normalized update rather than claiming that clipping provides an active trust region. The capacity-controlled comparisons in Table 20 provide initial evidence that both stored block changes and recurrent context matter.
Critical weaknesses
-
The evidence attributing the gain specifically to the proposed organization of adaptation capacity needs stronger experimental control. Tables 1 and 20 report three decoding seeds, while Appendix A.6 and C.12 do not report independent training runs. Table 11 gives a configurable recipe, but does not specify the complete settings and tuning budgets for each baseline. The exact architecture and inputs of the capacity-matched MLP in Table 20 are also not specified. This makes the central recurrent-control comparison difficult to reproduce. Training exposure matters here: removing retries lowers MathAvg from 83.64 to 62.53, while recorded cost falls from 121.23 to 37.92 GPU-hours (Table 21). Matching the retry rule for LoRA is valuable, but the realized generation counts and costs are still missing. The current evidence supports a parameter-count advantage under the reported recipe. It is less decisive about robustness across training runs or the benefit at a fixed training budget.
-
The search evaluation has unresolved split and metric issues. Appendix A.7 states that "ASearch Test" is used for validation every 50 updates. It also identifies eight exact question-answer duplicates between the reconstructed training and validation splits. This is 2.87% of the holdout. The improvement over matched-retry LoRA is only 1.18 points on this validation score. These observations do not prove that duplication causes the improvement, but they prevent a clean generalization interpretation without the actual filtered splits and checkpoint-selection rule. On the independent BrowseComp Plus benchmark, the corresponding difference is only 0.08 points. In addition, Tables 2 and 19 label this metric token-F1, whereas Appendix A.7 says the default returned score includes repetition, length, and tag penalties. The paper never clearly identifies whether those penalties were removed for the reported results. This ambiguity materially affects the search claim.
Questions for authors
-
Clarification and validation request: How many independent training runs underlie Tables 1 and 20? Please provide each baseline's actual optimizer settings, checkpoint-selection procedure, and realized generated-token budget. Do the main gains persist across independent training runs?
-
Clarification request: What precisely replaces the recurrent controller in the matched-MLP ablation? Please specify its inputs, dimensions, output, initialization, and which maps are retained. Also define how the historical IID estimates in Tables 22-24 were obtained and whether those benchmarks were used to select the final configuration.
-
Clarification and validation request: For Table 2, what are the actual post-filter evaluation counts, train-validation overlaps, and checkpoint-selection rule? Were the reported scores computed from raw token-F1 or the penalized reward? Please report the comparison on a deduplicated holdout and identify the exact scorer and tool configurations used.
2 comments
Response to Reviewer tMYs (2/2): Historical Estimates and Search Evaluation
Historical IID estimates (Tables 22–24). The dagger marks historical GSM8K and MATH-500 estimates and aggregates derived from them. It does not apply to annual AIME scores or the full T-Router reference. Earlier configuration screening used 100 questions each from GSM8K and MATH-500 and all 30 questions from each AIME year, so these benchmarks were visible during development. However, the screening sample sizes alone do not specify how the daggered estimates were calculated or which scores determined the final configuration. We therefore avoid making stronger claims about the precise extent of potential selection effects than the reported information supports.
Search data and scores (Tables 2 and 19). The “ASearch Test” column reports validation scores, evaluated every 50 updates. Appendix A.7 reconstructs 13,706 training and 279 validation records before filtering. Of the 279 validation records, 8 exactly match a training question and answer. BrowseComp Plus is a separate evaluation with 830 source questions before filtering. T-Router and matched-retry LoRA score 73.98 vs. 72.80 on ASearch validation and 36.98 vs. 36.90 on BrowseComp Plus. We acknowledge that the 0.08-point difference on the independent benchmark is too small to establish a reliable advantage.
The default scorer extracts the last complete <answer> span and takes the highest token-F1 across reference answers. The returned reward multiplies F1 by repetition, length, and answer-tag penalties. Table 2 does not identify whether it aggregates raw F1 or the penalized reward. Both default tools retrieve ten documents per call. ASearch uses e5-base-v2 over Wikipedia with a Qwen3-1.7B summarizer; BrowseComp Plus uses Qwen3-Embedding-8B over its own corpus with Qwen3-8B. These repository defaults provide implementation context, but do not establish the actual scorer and tool settings, checkpoint-selection rule, or post-filter sample counts and overlap for the reported runs.
We appreciate the detailed questions concerning training controls, data construction, and evaluation methodology. They highlight several aspects where additional reporting and experiments would strengthen the evidence and improve reproducibility.
Response to Reviewer tMYs (1/2): Training Controls and Matched MLP
Although we have decided to withdraw this submission, we would still like to address your comments and clarify the points you raised.
Table 1 reports a 6.36-point MathAvg gain for T-Router over LoRA-r16 at a similar parameter count under the same retry rule. We understand that your main concern is whether this improvement is consistent across independent training runs and whether it persists under matched training costs. We clarify the relevant experimental details and limitations below.
Training runs and budgets (Tables 1, 11, 20, and 21). Seeds 42, 43, and 44 are decoding seeds, rather than independent training seeds. Each round generates one answer per question, and the reported SD measures variation across these decoding rounds. Tables 1 and 20 do not report the number of independent training runs behind each row. Thus, these error bars characterize decoding variability and should not be interpreted as evidence of robustness across independent training runs. We agree that this distinction is important when assessing the reproducibility of the reported gains.
For the full-parameter GRPO run that scored 73.79 MathAvg, we used TRL’s GRPOTrainer, eight-GPU FSDP, and BF16. We trained all 8.954B parameters for one epoch on all 7,473 GSM8K questions, kept the final partial batch, and evaluated the final checkpoint at step 3,737. Each full update used two prompts with four completions per prompt. Training used temperature 0.7, top-p 0.95, and a 512-token completion limit. The learning rate was 1e-6, with one warmup step, KL coefficient 0.02, and clipping parameter 0.2. The 1e-4 rate in Table 11 belongs to the configurable adapter recipe.
T-Router and matched-retry LoRA each begin with four completions per prompt. If all rewards are equal, they can generate up to three replacement groups. The number of replacements depends on the model. Removing retries from T-Router changes MathAvg from 83.64 to 62.53 and training cost from 121.23 to 37.92 GPU-hours, indicating that this comparison does not hold compute fixed. Furthermore, applying the same retry rule does not guarantee equal completion or token counts, including discarded attempts. The remaining per-method optimizer settings, checkpoint choices, tuning budgets, and realized generation costs are not established by the reported tables. We therefore agree that the current evidence does not establish a fully compute-matched comparison, and we distinguish this limitation from the parameter-matched comparison reported in Table 1.
The matched MLP (Table 20). The stateless variant sets the recurrent context to zero and skips the slot update and readout. A two-layer MLP with a SiLU activation takes only the current token’s 4,096-dimensional residual. Its 256-dimensional output is added to the source-selection query, without entering the writeback gate. The first weight matrix uses Xavier-uniform initialization; the second matrix and both biases start at zero.
The block compressors, source and layer identities, query/key/value maps, writeback projections, RMS calibration, and signed gate are retained. If these maps are unchanged, the counts in Tables 6 and 21 imply a hidden width of 622. We emphasize that this value is reconstructed from the published counts and implementation, rather than verified from the original run configuration. The MLP variant scores 72.80 with 41,731,978 parameters, compared with 83.64 and 41,730,332 parameters for the recurrent model.