acceptodds
Under review as a conference paper at ICLR 2027

LoTR: Logic-of-Thought Routing for Plug-and-Play Reasoning of LLMs

Abstract

Reasoning has become a primary means of improving the problem-solving capabilities of Large Language Models (LLMs), with paradigms prescribing external reasoning strategies such as step-by-step, iterative revision, and search-and-sampling. However, while these paradigms guide how the model reasons, they do not consider whether the model's internal computation pathway is well-matched to the dynamic reasoning logic state under the prescribed strategy, limiting the potential under this strategy. We propose Logic-of-Thought Routing (LoTR), a plug-and-play reasoning module that strengthens existing reasoning paradigms through logic-conditioned internal pathway routing. Concretely, LoTR identifies dynamic logic states from the model's internal information transfer, couples them with attention-head pathway patterns, and uses a lightweight probe to infer the current logic-state mixture and softly route the corresponding pathway. This enables the model to better realize the prescribed reasoning strategy through an internal pathway matched to the current reasoning logic. Across 3 backbones, 10 benchmarks, and 8 reasoning paradigms, LoTR improves matched accuracy by 3.60% on average, reaching 5.37% on Llama with only +0.15% completion tokens and lower latency on 2 backbones. It also outperforms the mean of 4 inference-time intervention baselines on all 3 backbones.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, LLM Reasoning, Inference-Time Reasoning, Logic-State Routing, Internal Computation Pathways, Attention Head Routing, Inference-Time Intervention

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KP5COpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Logic-of-Thought Routing (LoTR), a plug-in controller that dynamically adjusts attention-head contributions while keeping the backbone weights and external reasoning strategy fixed. It clusters sentence-level representations into latent “logic states,” learns state-specific head-gating templates offline, and uses lightweight probes to mix these templates for the next reasoning step. Extensive experiments show the validity of LoTR method.

Critical strengths
  1. From my understanding, LoTR explores a useful direction by adapting attention-head contributions during reasoning. I like that it complements existing reasoning strategies instead of replacing them.

  2. This paper proposes a neat separation of offline controller training and online routing while keeping the backbone frozen.

  3. The evaluation is broad overall.

Critical weaknesses

Generalization

I question whether LoTR's scaling patterns generalize across different inputs and reasoning procedures, given known prompt sensitivities in activation steering [1]. Because sentences sharing a latent state may not benefit from identical head-scaling. Further validation of this scenario shall be appreciated.

Would head-level scaling not affect unrelated functions?

Because LoTR scales entire attention heads, it risks suppressing useful information encoded in polysemantic heads. The current ablation fails to isolate the benefit of state-matching from simply having a flexible scaling mechanism, as it confounds template counts and suppression strength. To prove state-matching is the true driver of performance, please add some further control studies.

The offline objective is not clearly aligned with online next-step control.

There is a structural mismatch between training and application: Eq. (3) optimizes templates for a sentence's current state, but Eq. (4) applies them to condition the next sentence. Please discuss whether this would pose impact on the eventual results.

[1] Tan, Daniel, David Chanin, Aengus Lynch, Brooks Paige, Dimitrios Kanoulas, Adrià Garriga-Alonso, and Robert Kirk. 2024. “Analysing the Generalisation and Reliability of Steering Vectors.” Advances in Neural Information Processing Systems 37. doi:10.52202/079017-4417.

Questions for authors

Please refer to the weakness section.

2 comments
AuthorsOpenReview ↗

Response to KP5C (1/2): Input Adaptation and State Matching

Thank you for your insightful analysis and for recognizing that LoTR complements existing reasoning strategies, separates offline controller training from online routing, and is supported by a broad evaluation. We address input and procedure adaptation Part 1, state matching and effects on other functions Part 2, and how the training objective relates to next-step control Part 3.

Part 1. Input and Procedure Generalization (W1)

Two prefixes can share a dominant state and still receive different head scaling Part 1.1. We then test the same controller under prompt changes and ask whether templates learned without CoT can support CoT Part 1.2.

Part 1.1. Input-Adaptive Scaling

Takeaway. Head scaling adapts to the input even when the dominant state is unchanged.

The same dominant state need not produce identical head-scaling. Eq. (4) uses all state weights at each layer: two prefixes can rank the same state first and still mix the templates differently. We reconstruct 1,019 updates from Llama-3.1-8B and Qwen2.5-14B. Within 14 groups matched by task, procedure, and dominant state at the final layer, the two prefixes in every sampled pair receive different gate values. The fixed-gate experiment in Part 2.1 tests whether the head assignment affects performance.

Part 1.2. Prompt and Procedure Shifts

Takeaway. Prompt changes need no recalibration, and templates learned from other procedures can support CoT.

① Does one controller handle prompt changes? To test prompt sensitivities, we paraphrase instructions and reorder options on 150 questions each from QASC and ScienceQA. We fix the Llama-3.1-8B controller and β. The table compares LoTR with Identity on the same inputs (gains in pp).

Prompt perturbationQASC CoTQASC SRScienceQA CoTScienceQA SR
Equivalent instructions+5.333+6.000+1.333+1.333
Reordered options-0.667+7.333+4.000+0.667

LoTR improves in seven of eight settings, with mean gains of +3.500pp after paraphrasing and +2.833pp after reordering (+3.167pp grouped by original question). The controller needs no recalibration: it reads the new prefix and changes the weights on the same templates.

② Can templates learned without CoT still help on CoT? We exclude CoT traces from compilation and calibrate β only on Plan-and-Solve. On 800 held-out CoT questions across four tasks, the no-CoT controller gains +1.195pp. To see what carries over, we match its clusters to those from the eight-procedure fit on training questions: the fits assign 96.5% (223/231) of held-out CoT readouts to corresponding clusters. Including CoT yields a larger gain; the no-CoT fit shows that other procedures already provide states useful for CoT. Separately, the source controller improves on all six tasks outside its compilation bank, averaged across eight procedures (+2.703pp, Table 1).

Part 2. State Matching and Side Effects (W2)

Correct matching helps at fixed template count and scaling strength Part 2.1. To test the concern about polysemantic heads, we evaluate grammatical judgments with routing active and find no change in overall accuracy Part 2.2.

Part 2.1. Matched-Gate Controls

Takeaway. At the same scaling strength, matching states to the right pathways still improves reasoning.

① State–template matching. To control for template counts and suppression strength, we use the same six templates in a four-task, 1,000-question Self-Refine test on Llama-3.1-8B. For each prefix and layer, we take the gate values from the correct map. We rank the heads using the mismatched map and reassign the original values in that order.

RoutingState–pathway assignmentMacro Δ vs. Identity (pp)
IdentityNo head scaling+0.000
Correct matchCurrent states paired with their original templates+0.904
Matched-gate mismatchAssignment alone changes+0.088

Correct matching leads by 0.816pp, mainly on DROP. With the same gate values, the mismatched map loses most of the observed Self-Refine gain. Choosing which head receives each value helps beyond flexible scaling alone. Correct matching also beats mismatch by +0.600pp on Qwen2.5-14B across 1,000 questions in two tasks, evaluated under two procedures.

② Head layout within a template. The head shuffle starts with the gate values from the correct state–template map and randomly reassigns them to heads. On the same 1,000 questions, shuffling lowers scores by 2.186pp on CoT, 2.654pp on Self-Refine, and 1.015pp on Self-Consistency (1.952pp overall). The learned layout beats shuffling on all four Self-Refine tasks and in all three template fits. On Qwen3-32B it also leads by 3.500pp across two QA tasks. Matching picks the template for the state; the learned layout puts the scaling on the heads where it helps.

AuthorsOpenReview ↗

Response to KP5C (2/2): Polysemantic Heads and Next-Step Control

Part 2.2. Polysemantic Heads

Takeaway. Head scaling leaves overall accuracy on the tested grammar functions unchanged.

Scaling polysemantic heads could affect unrelated functions such as grammar. We test this directly: we take active gates from 12 Llama-3.1-8B reasoning prefixes and apply them when scoring grammatical and ungrammatical continuations. On 4,000 BLiMP minimal pairs, accuracy is 93.80% with and without routing, and LoTR retains 99.23% of correct Identity judgments.

Untargeted functionIdentity (%)LoTR (%)Δpp
Anaphor number agreement97.9097.60-0.300
Determiner agreement96.0095.60-0.400
Subject–verb agreement90.5091.00+0.500
Verb inflection90.8091.00+0.200
Overall: 4,000 pairs93.8093.800.000

Each grammar category changes by at most half a percentage point. LoTR corrects 29 judgments and flips 29 correct judgments to incorrect. The tested grammar accuracy holds up with reasoning gates active. LoTR penalizes departures from Identity when fitting the templates. In 385 recorded Llama CoT updates, about 85% of gates at routed head positions stay at 1 and none are set to zero: routing changes the strength of selected heads without switching them off in these runs.

Direct code generation gives a separate test of reuse without explicit CoT. On HumanEval, Identity passes 104/164 problems, sentence routing 106/164, and 16-token routing 109/164. Reviewer 6Z54, Part 4 gives the full boundary comparison.

Part 3. Next-Step Control (W3)

Takeaway. The current state can guide the next step; training on the next sentence gives no consistent advantage.

① Does training on the next sentence improve the answer? Eq. (3)’s offline objective fits head scales using successful and unsuccessful examples of the current state. Eq. (4) reads the state after the current sentence and applies the scales to the next sentence. To test the structural mismatch, we fit one controller on the current sentence and one on the next sentence of each of the same 574 pairs. Both refit the same five templates under the same budget, leave the sixth template unchanged, and see only the current prefix at inference. The table reports F1 for DROP and accuracy elsewhere (%).

TaskQuestionsIdentityCurrent-targetNext-target
QASC30050.3351.0054.67
ScienceQA30072.0073.3372.67
BoolQ20077.5077.0074.50
DROP20027.3127.9429.17

The current-sentence target gains +0.532pp over Identity; the next-sentence target gains +0.965pp. The next-sentence target leads on QASC and DROP but trails on ScienceQA and BoolQ (mean difference +0.433pp). Training on the current sentence already improves scores when used for next-step routing.

② Why can the current state guide the next step? Across 30,230 adjacent sentence pairs in the compilation traces, final-layer gates change by 0.01155 per head on average in absolute value. The small average change makes the current state a useful starting point for online next-step control. LoTR reads the prefix again at the next sentence boundary and can update the weights as the reasoning changes.

Thank you again for your incisive questions about the routing mechanism, which helped us isolate the benefit of state matching and test how the training objective affects next-step control.

Reviewer_Z3WXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Logic-of-Thought Routing (LoTR), a plug-and-play framework that improves LLM reasoning by dynamically adjusting attention-head pathways according to inferred logic states. It combines offline logic-state discovery and pathway optimization with lightweight online routing. Experiments across three LLM backbones, ten benchmarks, and eight reasoning paradigms demonstrate consistent average performance gains with minimal additional inference overhead.

Critical strengths

1-LoTR introduces logic-conditioned attention-head routing to complement existing reasoning paradigms without modifying model parameters. 2-Experiments across three LLM backbones and eight reasoning paradigms show consistent average improvements with negligible additional token overhead.

Critical weaknesses

1- The identified logic states are derived from hidden-state clustering, but their correspondence to meaningful reasoning processes is not fully established. 2- Improvements vary considerably across tasks and reasoning paradigms, with some negative results (Table 1). The smaller gains on Qwen and Mixtral also raise questions about the robustness and generalizability of the approach (Table 3).

Questions for authors

1-What evidence supports interpreting the clustered hidden representations as meaningful logic states rather than general activation patterns? 2-What explains the negative gains on certain tasks (Table 1) and the smaller improvements on Qwen and Mixtral (Table 3)? Are there identifiable conditions under which LoTR is less effective?

5 comments
AuthorsOpenReview ↗

Response to Z3WX (1/3): State Interpretation and Routing Utility

Thank you for your careful review and for recognizing that LoTR complements existing reasoning paradigms with a frozen backbone, yields average gains across backbones and paradigms, and incurs negligible additional token overhead. We address observable reasoning roles Part 1 and why gains vary and when routing helps Part 2.

Part 1. State Interpretation (W1 / Q1)

We call them logic states because they track how reasoning proceeds and guide which heads LoTR scales. We test whether state matching improves performance Part 1.1, what the scores capture about a derivation Part 1.2, and what the six names describe in real traces Part 1.3.

Part 1.1. Routing Utility

Takeaway. Multi-state routing improves reasoning, and matching states to pathways helps even at the same scaling strengths.

The six-state controller beats an independently compiled one-state controller by 2.87pp across three tasks and four procedures (Table 5). To isolate matching, we keep the six templates and all gate values at each layer and update. The mismatched map changes which heads receive the values. In Self-Refine (four tasks, 1,000 questions), the matched map gains +0.904pp over Identity and the mismatched map gains +0.088pp. See Reviewer KP5C, Part 2.1 for the Qwen check and head-layout control.

Correct matching improves performance by 0.816pp at fixed gate values. We next test what the state scores tell the router about the step it has just read.

Part 1.2. Logical Relations

Takeaway. Routing states distinguish different derivations of the same conclusion.

“Pavel is happy” can follow from “Pavel is red; if red, then happy” or from “Pavel is both happy and red.” We include both premise sets in each prefix and swap which applies to Pavel. The supporting rule changes while the conclusion, tokens, and token counts stay fixed. Truth tables and SAT verify the derivations.

① A deployed state score responds to the rule. We freeze the submitted K=6 basis and probes and select S5 at layer 25 on discovery groups. With new entities, predicates, and wording, that score is higher for modus ponens than for conjunction elimination in 115/120 confirmation groups.

② Together, the state scores distinguish four rules. A linear classifier on the scores from all layers achieves 57.92% (278/480), versus 25% for the current sentence, bag-of-words, and length/position controls.

The concern about general activation patterns is whether the clusters have identifiable reasoning content. The scores LoTR uses for routing distinguish the rule behind a fixed conclusion. The fixed-gate test in Part 1.1 separately shows that matching those states to templates improves performance.

Part 1.3. State Roles

Takeaway. The states distinguish reasoning roles on unseen questions and phrasings.

① What justifies the names? To test their correspondence to meaningful reasoning processes, Qwen3-32B labels 360 Llama excerpts using one rubric, with state IDs hidden and separate discovery and confirmation questions. Each name describes a role enriched in that state.

State and labelExample from a traceConfirmation evidence
S1: Verbal reasoning boundary“2.” precedes the next premiseTransition marker in 26/30; with numbering matched, the next readout is S3 in 257/257
S2: Quantitative setupSets Tammy’s count to 16 plus one third of Derek’sIn GSM8K, planning or modeling in 12/30 vs. 4/30 for S5
S3: Premise and relation analysisCombines premises 1 and 3 to decide whether suspension appliesIn MMLU/FOLIO, states a premise or makes a deduction in 8/20, vs. 1/28 for S1 and 0/20 for S4; same direction within each benchmark
S4: Stating the answer“Answer: True.”States the conclusion in 22/30 vs. 8/150 for the other states; higher on 7/8 matched questions, tied on one
S5: Numerical executionComputes 23×2=46In GSM8K, performs a calculation in 13/30 vs. 2/30 for S2
S6: Arithmetic step boundary“2.” precedes the calculation 5×7=35Transition marker in 27/30; with numbering matched, the next readout is S2 or S5 in 151/152

② Can formatting explain the roles? With the same list marker (e.g., “2.”), length, and procedure, a mathematical expression follows in 1/257 S1 cases and 57/152 S6 cases. The discovery set shows the same difference. On held-out questions and phrasings, a classifier predicts roles at 69.0% from state scores and 65.4% from the actual gates, versus 44.4% from text.

Does the role change the scales LoTR applies? Across 120 arithmetic and entailment questions, we fix the question and sentence position. Changing the role moves the gates more than rephrasing the same role in both domains, by 1.23× on average. LoTR can thus adjust head scaling when the reasoning step changes.

AuthorsOpenReview ↗

Response to Z3WX (2/3): Negative Gains and Backbone Generalization

Part 2. Performance Variation (W2 / Q2)

We first revisit the negative results with the controller fixed and the reasoning procedure varied Part 2.1. We then examine the smaller gains on Qwen and Mixtral and test two new backbones Part 2.2. Finally, we ask what to change when routing has little effect Part 2.3.

Part 2.1. Negative Gains

Takeaway. On the expanded HumanEval set, routing improves Self-Consistency but still lowers Plan-and-Solve scores.

① Do the original negative gains recur? On the original 50 HumanEval problems in Table 1, LoTR solves one fewer problem under Self-Consistency (SC) and one fewer under Plan-and-Solve. We extend both evaluations to all 164 problems, including the original 50. With SC, LoTR passes 73 problems versus 59 for Identity; with Plan-and-Solve it passes 43 versus 46. Both procedures activate routing in every generation, yet the gains differ by 10.37pp. Under SC, 25 solutions switch from failing to passing and 11 switch the other way. Rerunning the official tests reproduces all 656 pass/fail results. On the 114 additional problems alone, SC rises from 38 to 52 and Plan-and-Solve from 26 to 28.

One original failure illustrates how negative gains can arise even when routing is active. The Plan-and-Solve code counts integers containing a digit instead of counting every occurrence of that digit. Plan-and-Solve returns a single implementation, so a missed constraint in that implementation becomes a final-answer error. SC combines several candidates instead. The HumanEval comparison shows that the same controller can help under SC even when the gain under Plan-and-Solve is negative. The call budget stays fixed within each comparison with Identity.

② Does SC also work best on other tasks? We compare CoT, Self-Refine, and SC with the same Llama-3.1-8B controller on four more tasks and 1,000 questions. The table reports gains over Identity (pp).

TaskCoTSelf-RefineSelf-Consistency
QASC+5.000+2.333-1.333
ScienceQA+0.333-0.333-0.667
BoolQ+4.000+2.500+3.000
DROP+9.218+4.705+3.378
Four-task unweighted mean+4.638+2.301+1.095

On QASC, routing helps CoT and Self-Refine but lowers the SC score. In the original Table 1, all eight tasks still improve on average across procedures (overall macro +2.66pp). Use DEV to choose the procedure for the task and call budget; the controller can stay fixed.

Part 2.2. Backbone Generalization

Takeaway. Qwen's absolute gain is close to Llama's, and Mixtral gains more under some workflows.

① Qwen: compare absolute gains. The smaller improvements in Table 3 are relative to Qwen’s higher baseline. Qwen rises from 75.0 to 77.1 (+2.1pp), close to Llama’s +2.7pp (49.5 to 52.2). The gain is smaller as a percentage of Qwen’s starting score. Qwen improves on four of its six original Self-Refine tasks, including +9.0pp on QASC (Appendix K, Table 10).

Mixtral: does the gain depend on the procedure? We keep the controller fixed and compare CoT and SC on the same 800 questions from all four original Table 3 tasks. Each gain is measured against Identity at that procedure’s original call budget.

TaskQuestionsCoT ΔppSC Δpp
CODAH2500.0000.000
DROP250+1.319+1.770
HumanEval50+2.000+4.000
QASC250+0.4000.000
Four-task mean800+0.930+1.443

Mixtral gains more on average under SC with the same controller. On CoT, 478/800 questions receive no update and give the same output as Identity. Under SC, only 410/800 receive no update, giving the controller more chances to affect the answer.

② Do the gains extend to newer and larger backbones? To test robustness and generalizability, we compile a separate controller for Qwen3-32B and Llama-3.1-70B. Both use the same 1,000 questions and three-candidate Self-Consistency budget. The table gives gains over Identity in pp, using F1 for DROP and accuracy elsewhere.

TaskQuestionsQwen3-32B ΔppLlama-3.1-70B Δpp
QASC300+4.000+3.667
ScienceQA300+3.000+0.000
BoolQ200+3.000+1.500
DROP200+0.420+0.441
Four-task unweighted mean1,000+2.605+1.402

The 32B model improves on all four tasks; the 70B model improves on three and ties on one. Across the same four tasks, Qwen3-32B also gains +0.330pp with Vanilla and +0.914pp with Best-of-N. The original backbones also improve on broader six-task panels: Qwen2.5-14B gains +1.808pp with Self-Refine, and Mixtral gains +1.178pp with SC.

One controller can serve different workflows on the same backbone, as the Mixtral comparison shows. For a different backbone, we reuse the compilation procedure to build a new controller. Both the 32B and 70B models benefit at the original call budget.

AuthorsOpenReview ↗

Response to Z3WX (3/3): When Routing Helps

Part 2.3. Routing Conditions

Takeaway. Earlier updates give the same controller a chance to help on short outputs.

① Does the answer finish before LoTR can act? One of the identifiable conditions is an output with no effective routing update. On 1,052 of 1,483 Vanilla questions, no effective update occurs and the output is identical to Identity. In Table 1’s MMLU Vanilla cell, all 245 questions without an update are unchanged. The net loss of one correct answer comes from the five questions with updates. No update explains the unchanged answers; the MMLU drop comes from the questions with active routing.

② If sentence routing updates too late, try token boundaries. On HumanEval without explicit CoT, Identity passes 104/164 problems, sentence routing 106/164, and 16-token routing 109/164. On MBPP, switching from sentence to 16-token boundaries raises the number of problems with an update from 232/500 to 477/500. The same controller can act earlier in code by reading the prefix every 16 tokens. Reviewer 6Z54, Part 4 gives the five settings and compares the token schedules at equal update counts.

③ If updates already occur, calibrate the template mix on DEV. The boundary sets when to update; β sets how strongly the router favors the highest-scoring templates. With one β chosen per controller on DEV, all 15 Llama fits improve on average across four CoT tasks and 800 questions. We can adjust either setting without fitting new states or templates.

Thank you again for your careful review, which helped us clarify what the logic states capture and show how routing gains vary with the task and reasoning procedure.

Reviewer_6Z54OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LoTR, a lightweight, plug-and-play reasoning module for LLMs. It aims to enhance existing reasoning paradigms through logic-conditioned internal path routing without modifying model weights or external reasoning workflows. In particular, LoTR includes:

  1. Offline Logic Modeling: Pooling sentence-level representations from trajectories of various reasoning paradigms to cluster and derive a shared "Logic-state basis."
  2. Online Plug-in Reasoning: During the reasoning process, probes are triggered only at sentence boundaries to infer the current mixture ratio of logical states. Based on this, cached path templates are dynamically combined to softly route the internal attention head pathways for the next sentence generation.
Critical strengths
  1. The paper identifies a blind spot in current reasoning research—the decoupling and mismatch between external strategies and internal execution. By attributing reasoning quality to the alignment between internal paths and dynamic logical states, this perspective offers significant value.

  2. LoTR operates without modifying model weights or altering external reasoning prompts and stopping conditions. Functioning as a pure inference-time controller, it can be seamlessly overlaid onto any paradigm. This modular design holds practical value for real-world deployment.

Critical weaknesses
  1. Although the paper claims to have discovered "logic states," these states are essentially cluster centers obtained via unsupervised PCA and K-means. Apart from post-hoc labeling in the case study, the semantic interpretation of these states remains insufficiently validated. Do these clusters represent human-interpretable logic patterns, or are they merely artifacts of certain statistical regularities within the model? If the states themselves are essentially uninterpretable, "Logic-of-Thought" might be over-claimed.

  2. For unstructured generation tasks where the chain of thought is not obvious, the coarse-grained control may fail or even be detrimental. Although the paper mentions this point in the limitations section, it does not evaluate whether finer-grained routing improves performance on generation tasks without explicit sentence-level reasoning steps, which undermines the claim of the method's generalizability.

  3. LoTR requires separate offline compilation for each backbone model. This process involves collecting trajectories, clustering, training probes, and optimizing templates. The entire pipeline must be repeated for new models or fine-tuned versions. The paper does not quantify the potential benefits of cross-backbone transfer. The so-called "Plug-and-Play" capability is limited to different paradigms within the same model, rather than being universally applicable across models.

  4. Although Figure 6 presents a sensitivity analysis of the probe-logit scale β, the rationale for selecting key hyperparameters, such as K, the soft target temperature τ, and the routing layer range, remains weak. In particular, the selection of K=6 via development set diagnostics implies that the method may depend on the stability of the learned state basis and the distribution of compilation trajectories. The paper also lacks out-of-distribution robustness stress tests.

  5. The models chosen for the experiments do not cover larger-scale or more recent backbone models (mostly reflecting the 2024 model landscape). This raises concerns regarding the timeliness of the experimental results. It is recommended to further verify whether the method can achieve superior performance on open-source models with larger parameter scales (e.g., 32B, 72B) or the latest models.

Questions for authors

See the questions in the Critical Weaknesses. Particularly, the questions regarding hyperparameter robustness and model scale and timeliness should be addressed.

4 comments
AuthorsOpenReview ↗

Response to 6Z54 (1/4): State Count, Temperature, and Layer Range

Thank you for your constructive feedback and for highlighting both the gap between external reasoning strategies and internal execution and the practical value of LoTR’s plug-and-play design. We first address hyperparameter robustness Part 1 and model scale and timeliness Part 2, then discuss what the states represent Part 3, routing granularity without explicit CoT Part 4, and reuse and cost Part 5.

Part 1. Routing Robustness (W4)

On DEV, Figure 6 shows gains at every tested β from 0.35 to 0.65, with the largest at β=0.5. We now test the number and stability of the learned states Part 1.1, the temperature and routing layer range Part 1.2, and robustness to prompt and trajectory shifts Part 1.3.

Part 1.1. State Count and Stability

Takeaway. Routing gains persist near K=6 and across independent fits.

① Why use more than one state? One template fixes the relative scaling of heads within each layer. With several templates, LoTR can change which heads it scales most based on the current prefix. On BoolQ, DROP, and ScienceQA-text across four procedures (Table 5), Identity scores 53.4%, K=1 scores 53.6%, and K=6 scores 56.5%. K=6 beats the independently compiled K=1 by 2.87pp. The fixed-gate control in Part 3.2 tests the head assignment directly.

② Do gains depend on K=6 or on one fit? Algorithm 2 compares candidate state counts using clustering and probe diagnostics and separate DEV runs. K is fixed before evaluation. In the sensitivity test, we fit each K three times at fixed τ=1 and β=0.5. We report the mean gain over Identity (pp) on 800 CoT questions: 200 each from BoolQ, DROP, ScienceQA-text, and QASC. The temperature, layer range, and CoT holdout tests use the same 800 questions.

KSeed 42Seed 43Seed 44Mean
4+2.480+2.018+2.775+2.424
6+5.268+2.973+3.721+3.988
8+3.559+3.423+0.232+2.405

All nine fits at K=4, 6, and 8 improve on Identity, with the highest mean at K=6. At seed 42 and the same τ and β, K=1 falls below Identity (−3.431pp). K=12 and K=16 still improve (+2.745 and +1.243pp) but neither beats K=6 at that seed. Gains persist near K=6; adding more states does not increase them in this comparison.

③ Do the same states reappear when we recluster? To test the stability of the learned state basis, we hold the backbone fixed and recluster ten times, changing the initialization or withholding 20% of questions. Three of the six original clusters retain over 95% of their members in every run. The same activation patterns can therefore be recovered without the original initialization or the full set of questions. Part 1.3 tests what happens when we omit a whole reasoning procedure; Part 3 examines what the states represent.

Part 1.2. Temperature and Layer Range

Takeaway. Gains persist across temperatures with DEV calibration and across routing layer ranges.

① Temperature. The soft target temperature τ sets how much weight the probe’s training target gives to the nearest state versus its neighbors. Eq. (26) builds soft targets from distances between normalized representations and adds a small hard-label term for the nearest state. We test τ=0.5, 1 (the default), and 2. At fixed K=6, we fit three controllers per τ. For each fit, we choose one β from three candidates on 128 DEV questions and hold it fixed across all four evaluation tasks. The table gives mean gains over Identity (pp) on the same 800 questions.

τSeed 42Seed 43Seed 44Mean
0.5+2.971+3.816+4.615+3.801
1+5.268+1.887+1.436+2.864
2+2.522+2.931+1.198+2.217

All nine fits improve with one DEV calibration per controller. At τ=0.5, all three fits select β=0.25, pairing sharper training targets with a softer mix of templates at inference.

② Why start at layer 4? We leave the first three foundation layers untouched to preserve early input encoding (Appendix I). To test the routing layer range, we fit and apply routing from layer 1, 4, or 9 at fixed β=0.5. Starting at layer 1 yields +4.509pp, layer 4 +3.386pp, and layer 9 +3.381pp. Routing still helps when the first eight layers are untouched. The default at layer 4 keeps early encoding unchanged and still improves performance.

AuthorsOpenReview ↗

Response to 6Z54 (2/4): Distribution Shifts and Model Scale

Part 1.3. Input and Trajectory Shifts

Takeaway. Prompt shifts need no recalibration, and the default state count still works when CoT is held out.

① Do prompt changes require retuning? To test out-of-distribution robustness, we paraphrase instructions and reorder answer options on 150 questions each from QASC and ScienceQA. We keep the Llama-3.1-8B controller and β fixed and compare with Identity on the same perturbed inputs (gains in pp).

Prompt perturbationQASC CoTQASC SRScienceQA CoTScienceQA SR
Equivalent instructions+5.333+6.000+1.333+1.333
Reordered options-0.667+7.333+4.000+0.667

LoTR gains +3.500pp under paraphrasing and +2.833pp under reordering (+3.167pp grouped by original question). Neither prompt change requires new templates or calibration. LoTR reads the changed prefix and recombines the templates it already has.

② Does K=6 depend on seeing CoT during compilation? To change the distribution of compilation trajectories, we exclude all CoT traces, keep K=6, and choose β on Plan-and-Solve alone. We then evaluate on CoT: the controller still gains 1.195pp over Identity on 800 questions across four tasks.

To check whether the states learned without CoT are recognizable in CoT traces, we align the no-CoT and eight-procedure clusters using only shared training questions. The no-CoT and eight-procedure fits agree on the cluster assignment for 223/231 held-out CoT readouts (96.5%). Including CoT in compilation gives a larger gain, but states learned from other procedures already let LoTR improve CoT. See Part 5.2 for reuse on new tasks.

Part 2. Larger and Newer Models (W5)

Takeaway. Larger or newer backbones also benefit from LoTR at the original sampling budget.

The submission covers Llama-3.1-8B, Qwen2.5-14B, and Mixtral-8×7B. To test larger-scale or more recent backbone models, we compile a controller for the newer Qwen3-32B and for Llama-3.1-70B, a larger model in the original Llama family. We compare both on the same 1,000 questions with three-candidate Self-Consistency (SC). The table reports gains over Identity in F1 for DROP and accuracy elsewhere.

TaskQuestionsQwen3-32B ΔppLlama-3.1-70B Δpp
QASC300+4.000+3.667
ScienceQA300+3.000+0.000
BoolQ200+3.000+1.500
DROP200+0.420+0.441
Four-task unweighted mean1,000+2.605+1.402

Qwen3-32B improves on all four tasks; Llama-3.1-70B improves on three and ties on one. On the two QA tasks, the 32B model averages +0.944pp with SC across three generation seeds. On the four tasks in the table, the 32B model also gains +0.330pp with Vanilla and +0.914pp with Best-of-N. The same compilation procedure produces useful controllers at 32B and 70B, without increasing the evaluation sampling budget.

AuthorsOpenReview ↗

Response to 6Z54 (3/4): Logic States and Their Role in Routing

Part 3. Logic States (W1)

We call these logic states because their scores capture two aspects of reasoning: how the premises support a conclusion Part 3.1, and whether the model is setting up a problem, carrying out a step, or stating an answer Part 3.2. LoTR learns the states from a frozen model and uses their scores to choose which heads to scale. The names describe behavior observed after fitting.

Part 3.1. Logical Relations

Takeaway. Routing states distinguish different derivations of the same conclusion.

Consider “Pavel is happy.” The model can infer it from “Pavel is red; if red, then happy,” or from “Pavel is both happy and red.” Both premise sets appear in each prefix. Swapping which premises apply to Pavel changes the rule that supports the conclusion while holding the conclusion, tokens, and token counts fixed. Truth tables and SAT verify each derivation.

① A deployed state score responds to the rule. We freeze the submitted K=6 basis and probes. On discovery groups we select S5 at layer 25; on new entities, predicates, and wording, that score is higher for modus ponens than for conjunction elimination in 115/120 confirmation groups.

② Together, the state scores distinguish four rules. A linear classifier on the scores from all layers achieves 57.92% (278/480), versus 25% for the current sentence, bag-of-words, and length/position controls.

This is the evidence for human-interpretable logic patterns: the state scores used for routing distinguish how the same conclusion follows. Geiger et al. (Causal Abstractions of Neural Networks, NeurIPS 2021) likewise use explicit logical relations to test internal representations.

Part 3.2. State Roles and Routing

Takeaway. Routing states reflect the roles of reasoning steps and help select pathways.

① What justifies the names? For the post-hoc labeling, Qwen3-32B labels 360 Llama excerpts using one rubric, with state IDs hidden and separate discovery and confirmation questions. Each name describes a role enriched in that state. The two boundary labels describe the kind of step that follows.

State and labelExample from a traceConfirmation evidence
S1: Verbal reasoning boundary“2.” precedes the next premiseTransition marker in 26/30; with numbering matched, the next readout is S3 in 257/257
S2: Quantitative setupSets Tammy’s count to 16 plus one third of Derek’sIn GSM8K, planning or modeling in 12/30 vs. 4/30 for S5
S3: Premise and relation analysisCombines premises 1 and 3 to decide whether suspension appliesIn MMLU/FOLIO, states a premise or makes a deduction in 8/20, vs. 1/28 for S1 and 0/20 for S4; same direction within each benchmark
S4: Stating the answer“Answer: True.”States the conclusion in 22/30 vs. 8/150 for the other states; higher on 7/8 matched questions, tied on one
S5: Numerical executionComputes 23×2=46In GSM8K, performs a calculation in 13/30 vs. 2/30 for S2
S6: Arithmetic step boundary“2.” precedes the calculation 5×7=35Transition marker in 27/30; with numbering matched, the next readout is S2 or S5 in 151/152

② Can formatting explain the labels? With the same list marker (e.g., “2.”), length, and procedure, a mathematical expression follows in 1/257 S1 cases and 57/152 S6 cases. The difference also appears on discovery questions. On held-out questions and phrasings, a classifier predicts reasoning roles at 69.0% from state scores and 65.4% from the actual gates, versus 44.4% from text.

Changing the role also changes the scales the router applies. On 120 arithmetic and entailment questions, with question and sentence position fixed, changing the role shifts the gates 1.23× as much as rephrasing the same role. The role change moves the gates more in both domains.

③ Does each template affect the applied gates? We recompute 385 recorded CoT updates with the prefixes, state weights, and other templates fixed. For the state with the highest weight at the final layer, we replace its head scales with their mean within each layer. The gates change in all six state groups, showing that each template’s head layout contributes to routing.

Does the head assignment improve answers? In Self-Refine, we keep the gate values fixed and change which heads receive them. The matched map gains +0.904pp over Identity; the mismatched map gains +0.088pp. Separately, the learned head layout beats a shuffle by 1.952pp across three procedures. LoTR uses state scores to choose how to scale heads for the next step; the fixed-gate test shows that the learned assignment improves task performance. Reviewer KP5C, Part 2.1 details both controls and the cross-backbone check.

AuthorsOpenReview ↗

Response to 6Z54 (4/4): Routing Granularity, Reuse, and Cost

Part 4. Routing Granularity (W2)

Takeaway. Finer boundaries can improve code generation without changing the controller.

Sentence ends mark reasoning steps in prose. For unstructured generation such as code, a sentence end may come too late for routing to affect the answer. On HumanEval with Llama-3.1-8B, we change when LoTR reads the prefix while keeping the sentence-trained controller and the update cap fixed.

BoundaryPassed out of 164Pass@1 (%)Δpp
Identity10463.41+0.000
Sentence10664.63+1.220
Code line10664.63+1.220
16 tokens10966.46+3.049
8 tokens10765.24+1.829

Sentence routing passes two more problems than Identity; 16-token routing passes three more than sentence routing. Both the 8- and 16-token schedules average eight updates per problem, yet the 16-token schedule passes two more. Update timing matters even at the same average count.

On MBPP, moving the update boundary from sentences to every 16 tokens raises the number of problems with a routing update from 232/500 to 477/500. Changing only the boundary gives the existing controller more chances to intervene in code.

Part 5. Controller Reuse and Cost (W3 / W4)

For plug-and-play deployment, we test whether the original controller still helps after LoRA (+3.174pp across six variants) Part 5.1 and across tasks and procedures Part 5.2. We then measure compilation and inference separately Part 5.3.

Part 5.1. LoRA Reuse (W3)

Takeaway. The source controller can be reused directly after mild fine-tuning.

To test whether fine-tuned versions need recompilation, we fine-tune Llama-3.1-8B on FOLIO and GSM8K with three LoRA seeds each. We keep the original controller and β and compare with each adapted model’s own Identity baseline on the same 800 questions across four tasks.

Adaptation domain / seedHeld-out domain NLL (before→after)Identity macroLoTR macroΔpp
FOLIO / 429.87→8.4157.9061.04+3.140
FOLIO / 439.87→8.4058.2759.01+0.737
FOLIO / 449.87→8.4558.1860.65+2.472
GSM8K / 421.12→1.0655.5561.85+6.302
GSM8K / 431.12→1.0656.5559.01+2.463
GSM8K / 441.12→1.0656.4760.40+3.929

Held-out NLL falls in all six variants, confirming domain adaptation. The original controller improves all six adapted models (mean +3.174pp). On identical test prefixes, the source and adapted models produce nearly the same gates. The adapted models can reuse the controller without new trajectories or recompilation.

Part 5.2. Scope of Reuse (W3 / W4)

Takeaway. Pathway templates transfer to new tasks and to CoT held out during compilation.

① Reuse within a backbone. The source controller improves on all six tasks left out of compilation when averaged across eight procedures (+1.94 to +3.20pp; task macro +2.703pp, Table 1). When CoT is excluded from compilation and calibration, the resulting controller still gains +1.195pp on 800 CoT questions across four tasks. Templates learned from other tasks or procedures can still help when the same reasoning steps recur. See Part 1.3 for the cluster agreement on held-out CoT.

② Extend to new backbones. For cross-backbone transfer, we reuse the compilation procedure and fit a separate controller to each new backbone’s activations and heads. The 32B and 70B controllers improve both models at the original sampling budget Part 2. The LoRA variants in Part 5.1 instead reuse the original controller directly.

Part 5.3. Deployment Cost (W3)

Takeaway. Compilation costs can be amortized, and online quality improves at nearly the same token budget.

① Online quality and overhead. At fixed call budgets across 64 settings (Table 4), LoTR delivers a 5.37% relative quality gain with 0.15% more completion tokens on average across all calls. In five-task CoT decoding (Figure 5), LoTR's latency is comparable to ITI, ActAdd, and STIR and lower than RISER's.

② One-time preparation and per-update cost. The four controller files total 3.05 MiB for Llama-3.1-8B and 12.65 MiB for Llama-3.1-70B. The table separates offline compilation from online control:

StageMeasured cost
One-time compilation (70B)512 trajectories from 128 questions; parallel H200 shards take 74 and 69 min, then CPU state modeling and probe training. Template fitting: 43 min on one H200.
Online control microbenchmark (8B)A100/BF16, full prefix, no KV reuse. Gates add 5.66% at 128 tokens and 0.55%–0.99% at 512–2,048 tokens; probing and mixing add 6–7 ms per boundary.

The compilation cost is paid once per controller. At each later request, LoTR reuses the cached templates and computes new weights from the current prefix.

Thank you again for your constructive feedback, which prompted us to test the robustness of LoTR’s routing settings, extend the evaluation to larger backbones, and clarify what its logic states represent.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Summary for the Area Chair: Key Concerns, Evidence, and Revisions

Dear Area Chair,

Thank you for overseeing the discussion. We have responded point by point to every weakness and question. As the discussion draws to a close, we summarize the main evidence and revisions below. New manuscript material is marked in blue; the latest recent-model results are identified separately.

ConcernRaised byDirect evidence and what it establishesLocation
Model scale and recency6Z54 W5; Z3WX W2/Q2At matched three-candidate Self-Consistency budgets, Qwen3-32B and Llama-3.1-70B gain +2.605/+1.402 pp on the same 1,000 questions across four tasks. Further tests on Gemma-4-12B-it gain +18 pp on QASC and +8 pp on ScienceQA-text (100 questions each). Internal routing benefits both larger backbones and a recent model family.Table 4; Appendix K.2; 6Z54, Additional Results on Recent Models
Meaningful logic states6Z54 W1; Z3WX W1/Q1With the conclusion, prefix token multiset, and length fixed, a frozen state score shifts in the predicted direction in 115/120 held-out groups for a rule pair selected on separate discovery data. A linear readout of state scores classifies four rules at 57.92%, versus 25% for text and position controls. The deployed states distinguish how premises support the same conclusion.Section 3.4; Appendix J.3
State matching and unrelated functionsKP5C W2; Z3WX W1/Q1With template count and gate values fixed, mismatch reduces the Self-Refine gain from +0.904 to +0.088 pp. Separately, permuting head assignments costs a mean 1.952 pp across three procedures. With gates active, grammar accuracy remains 93.8% on 4,000 BLiMP pairs. Assignment matters beyond scaling strength, while the tested grammar functions are preserved.Table 7; Appendix J.4
Hyperparameters and distribution shifts6Z54 W4; KP5C W1All nine fits at K=4/6/8 improve, as do all tested temperature fits after DEV calibration and all three routing layer ranges. A fixed controller improves in 7/8 settings under prompt changes. Excluding CoT from compilation still yields +1.195 pp on 800 CoT questions. Gains persist across nearby settings, refits, and input or procedure shifts.Section 3.5; Table 12; Appendices I.1, J.5
Controller reuse and deployment cost6Z54 W3The source controller improves all six mild LoRA variants, by +3.174 pp on average, without new trajectories or recalibration. Controller files total 3.05 MiB at 8B and 12.65 MiB at 70B. The original 64-setting evaluation gains 5.37% in relative quality with 0.15% more completion tokens. Reuse supports repeated deployment at nearly the original token budget.Tables 5, 17; Appendices L.2–L.3
Performance variationZ3WX W2/Q2Qwen’s absolute gain (+2.1 pp) is close to Llama’s (+2.7 pp), despite its higher baseline. With the Mixtral controller fixed, CoT/SC gain +0.930/+1.443 pp on the same 800 questions. Expanded HumanEval SC improves 59→73 passes. The response analyzes a missed counting constraint in an original negative result and explains why effectiveness varies with the reasoning procedure.Table 3; Appendix K.1; Z3WX, Parts 2.1–2.2
Generation without explicit CoT6Z54 W2Keeping the controller and update cap fixed, 16-token routing raises direct HumanEval generation from 104 to 109 passes out of 164, versus 106 with sentence routing. Finer update boundaries let the same controller improve code generation without explicit sentence-level reasoning.Table 16; Appendix L.1
Current-state training and next-step controlKP5C W3Matched training on 574 sentence pairs shows that current-sentence targets already improve mean next-step performance; next-sentence targets help on two tasks and hurt on two relative to current targets. The revision explains current-state feedback and directly tests the alternative training target.Appendix J.4

Together, these results strengthen LoTR’s central claim: an existing reasoning procedure can produce better answers by adapting internal execution, with frozen model weights and matched external call budgets. The controls establish the value of state-dependent head assignment, and the transfer tests show that one controller can serve different tasks, procedures, and mild model adaptations.

Reviewers KP5C and Z3WX have confirmed that they maintain their positive ratings. Reviewer 6Z54 noted that most concerns had been addressed and requested further evidence on newer models; the additional Gemma-4 results directly address that remaining concern. We hope this summary is useful for your assessment and welcome any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.