- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Logic-of-Thought Routing (LoTR), a plug-in controller that dynamically adjusts attention-head contributions while keeping the backbone weights and external reasoning strategy fixed. It clusters sentence-level representations into latent “logic states,” learns state-specific head-gating templates offline, and uses lightweight probes to mix these templates for the next reasoning step. Extensive experiments show the validity of LoTR method.
Critical strengths
-
From my understanding, LoTR explores a useful direction by adapting attention-head contributions during reasoning. I like that it complements existing reasoning strategies instead of replacing them.
-
This paper proposes a neat separation of offline controller training and online routing while keeping the backbone frozen.
-
The evaluation is broad overall.
Critical weaknesses
Generalization
I question whether LoTR's scaling patterns generalize across different inputs and reasoning procedures, given known prompt sensitivities in activation steering [1]. Because sentences sharing a latent state may not benefit from identical head-scaling. Further validation of this scenario shall be appreciated.
Would head-level scaling not affect unrelated functions?
Because LoTR scales entire attention heads, it risks suppressing useful information encoded in polysemantic heads. The current ablation fails to isolate the benefit of state-matching from simply having a flexible scaling mechanism, as it confounds template counts and suppression strength. To prove state-matching is the true driver of performance, please add some further control studies.
The offline objective is not clearly aligned with online next-step control.
There is a structural mismatch between training and application: Eq. (3) optimizes templates for a sentence's current state, but Eq. (4) applies them to condition the next sentence. Please discuss whether this would pose impact on the eventual results.
[1] Tan, Daniel, David Chanin, Aengus Lynch, Brooks Paige, Dimitrios Kanoulas, Adrià Garriga-Alonso, and Robert Kirk. 2024. “Analysing the Generalisation and Reliability of Steering Vectors.” Advances in Neural Information Processing Systems 37. doi:10.52202/079017-4417.
Questions for authors
Please refer to the weakness section.
2 comments
Response to KP5C (2/2): Polysemantic Heads and Next-Step Control
Part 2.2. Polysemantic Heads
Takeaway. Head scaling leaves overall accuracy on the tested grammar functions unchanged.
Scaling polysemantic heads could affect unrelated functions such as grammar. We test this directly: we take active gates from 12 Llama-3.1-8B reasoning prefixes and apply them when scoring grammatical and ungrammatical continuations. On 4,000 BLiMP minimal pairs, accuracy is 93.80% with and without routing, and LoTR retains 99.23% of correct Identity judgments.
| Untargeted function | Identity (%) | LoTR (%) | Δpp |
|---|---|---|---|
| Anaphor number agreement | 97.90 | 97.60 | -0.300 |
| Determiner agreement | 96.00 | 95.60 | -0.400 |
| Subject–verb agreement | 90.50 | 91.00 | +0.500 |
| Verb inflection | 90.80 | 91.00 | +0.200 |
| Overall: 4,000 pairs | 93.80 | 93.80 | 0.000 |
Each grammar category changes by at most half a percentage point. LoTR corrects 29 judgments and flips 29 correct judgments to incorrect. The tested grammar accuracy holds up with reasoning gates active. LoTR penalizes departures from Identity when fitting the templates. In 385 recorded Llama CoT updates, about 85% of gates at routed head positions stay at 1 and none are set to zero: routing changes the strength of selected heads without switching them off in these runs.
Direct code generation gives a separate test of reuse without explicit CoT. On HumanEval, Identity passes 104/164 problems, sentence routing 106/164, and 16-token routing 109/164. Reviewer 6Z54, Part 4 gives the full boundary comparison.
Part 3. Next-Step Control (W3)
Takeaway. The current state can guide the next step; training on the next sentence gives no consistent advantage.
① Does training on the next sentence improve the answer? Eq. (3)’s offline objective fits head scales using successful and unsuccessful examples of the current state. Eq. (4) reads the state after the current sentence and applies the scales to the next sentence. To test the structural mismatch, we fit one controller on the current sentence and one on the next sentence of each of the same 574 pairs. Both refit the same five templates under the same budget, leave the sixth template unchanged, and see only the current prefix at inference. The table reports F1 for DROP and accuracy elsewhere (%).
| Task | Questions | Identity | Current-target | Next-target |
|---|---|---|---|---|
| QASC | 300 | 50.33 | 51.00 | 54.67 |
| ScienceQA | 300 | 72.00 | 73.33 | 72.67 |
| BoolQ | 200 | 77.50 | 77.00 | 74.50 |
| DROP | 200 | 27.31 | 27.94 | 29.17 |
The current-sentence target gains +0.532pp over Identity; the next-sentence target gains +0.965pp. The next-sentence target leads on QASC and DROP but trails on ScienceQA and BoolQ (mean difference +0.433pp). Training on the current sentence already improves scores when used for next-step routing.
② Why can the current state guide the next step? Across 30,230 adjacent sentence pairs in the compilation traces, final-layer gates change by 0.01155 per head on average in absolute value. The small average change makes the current state a useful starting point for online next-step control. LoTR reads the prefix again at the next sentence boundary and can update the weights as the reasoning changes.
Thank you again for your incisive questions about the routing mechanism, which helped us isolate the benefit of state matching and test how the training objective affects next-step control.
Response to KP5C (1/2): Input Adaptation and State Matching
Thank you for your insightful analysis and for recognizing that LoTR complements existing reasoning strategies, separates offline controller training from online routing, and is supported by a broad evaluation. We address input and procedure adaptation
Part 1, state matching and effects on other functionsPart 2, and how the training objective relates to next-step controlPart 3.Part 1. Input and Procedure Generalization (W1)
Two prefixes can share a dominant state and still receive different head scaling
Part 1.1. We then test the same controller under prompt changes and ask whether templates learned without CoT can support CoTPart 1.2.Part 1.1. Input-Adaptive Scaling
The same dominant state need not produce identical head-scaling. Eq. (4) uses all state weights at each layer: two prefixes can rank the same state first and still mix the templates differently. We reconstruct 1,019 updates from Llama-3.1-8B and Qwen2.5-14B. Within 14 groups matched by task, procedure, and dominant state at the final layer, the two prefixes in every sampled pair receive different gate values. The fixed-gate experiment in
Part 2.1tests whether the head assignment affects performance.Part 1.2. Prompt and Procedure Shifts
① Does one controller handle prompt changes? To test prompt sensitivities, we paraphrase instructions and reorder options on 150 questions each from QASC and ScienceQA. We fix the Llama-3.1-8B controller and β. The table compares LoTR with Identity on the same inputs (gains in pp).
LoTR improves in seven of eight settings, with mean gains of +3.500pp after paraphrasing and +2.833pp after reordering (+3.167pp grouped by original question). The controller needs no recalibration: it reads the new prefix and changes the weights on the same templates.
② Can templates learned without CoT still help on CoT? We exclude CoT traces from compilation and calibrate β only on Plan-and-Solve. On 800 held-out CoT questions across four tasks, the no-CoT controller gains +1.195pp. To see what carries over, we match its clusters to those from the eight-procedure fit on training questions: the fits assign 96.5% (223/231) of held-out CoT readouts to corresponding clusters. Including CoT yields a larger gain; the no-CoT fit shows that other procedures already provide states useful for CoT. Separately, the source controller improves on all six tasks outside its compilation bank, averaged across eight procedures (+2.703pp, Table 1).
Part 2. State Matching and Side Effects (W2)
Correct matching helps at fixed template count and scaling strength
Part 2.1. To test the concern about polysemantic heads, we evaluate grammatical judgments with routing active and find no change in overall accuracyPart 2.2.Part 2.1. Matched-Gate Controls
① State–template matching. To control for template counts and suppression strength, we use the same six templates in a four-task, 1,000-question Self-Refine test on Llama-3.1-8B. For each prefix and layer, we take the gate values from the correct map. We rank the heads using the mismatched map and reassign the original values in that order.
Correct matching leads by 0.816pp, mainly on DROP. With the same gate values, the mismatched map loses most of the observed Self-Refine gain. Choosing which head receives each value helps beyond flexible scaling alone. Correct matching also beats mismatch by +0.600pp on Qwen2.5-14B across 1,000 questions in two tasks, evaluated under two procedures.
② Head layout within a template. The head shuffle starts with the gate values from the correct state–template map and randomly reassigns them to heads. On the same 1,000 questions, shuffling lowers scores by 2.186pp on CoT, 2.654pp on Self-Refine, and 1.015pp on Self-Consistency (1.952pp overall). The learned layout beats shuffling on all four Self-Refine tasks and in all three template fits. On Qwen3-32B it also leads by 3.500pp across two QA tasks. Matching picks the template for the state; the learned layout puts the scaling on the heads where it helps.