acceptodds
Under review as a conference paper at ICLR 2027

Reorganizing Latent Reasoning in Large Language Models

Abstract

Complex reasoning requires coordinating cognitive functions, such as retrieval, semantic understanding and math calculation, in a task-dependent order. Chain-of-Thought (CoT) improves reasoning in large language models (LLMs) by making intermediate steps explicit, yet leaves the model's latent computation constrained by a fixed layer-wise computational order. In this work, we investigate whether reorganizing latent computation can improve reasoning.We introduce Chain-of-Cognition (CoC), a framework that derives a cognitive ordering from human-verified decompositions of reasoning tasks, localizes internal regions associated with each cognitive function, and dynamically reorganizes computation across these regions to follow that ordering. For example, semantic understanding often precedes math calculation, yet its associated heads can lie deeper; CoC selectively routes semantic representations back to mathematical regions, aligning information flow with the functional dependencies of the task. Across multiple LLMs and diverse reasoning benchmarks, CoC consistently improves reasoning performance without updating model parameters. Further experiments on vision-language models demonstrate its applicability to multimodal reasoning. These results suggest that effective reasoning depends not only on the capabilities available within a model, but also on how they are coordinated and composed during latent computation.

Primary area:
interpretability and explainable AI
Keywords:
reasoning, mechanistic interpretability, cognitive ordering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_34YDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Modular Chain-of-Cognition to reorganize computation within frozen language models using function-guided feedback. It derives a global ordering of cognitive functions from annotated reasoning decompositions and associates each function with selected attention heads. Mismatches between this ordering and the heads' layer locations define feedback connections that expose earlier layers to deeper representations. The resulting routing plan aggregates selected head outputs and recomputes downstream layers during the prefill pass. Experiments on language and vision-language reasoning benchmarks show average accuracy gains over standard inference, with smaller improvements in the multimodal setting and additional prefill cost.

Critical strengths
  1. The idea is interesting. The method turns annotated functional structure into an explicit inference procedure that can be applied without updating the backbone or training a routing policy.
  2. The paper is well-written and easy to follow.
Critical weaknesses
  1. The routing criterion combines correlational head attribution with average first-occurrence positions in human decompositions. Neither establishes that a target component requires the source component's output, or that their layer ordering constitutes a computational error. Moreover, the injected signal enters the shared residual stream and affects all subsequent heads and MLPs. The accuracy gains therefore provide limited evidence for the central interpretation that identifiable cognitive modules have been composed in a more appropriate order, rather than that selected activation feedback improves computation more generally.
  2. Table 6 does not isolate the effect of cognitive ordering as claimed. Permuting the function labels changes the selected source-target connections and can change the number of hops, as the two-hop and three-hop controls demonstrate. These interventions differ in the information injected and the computation performed, in addition to their ordering. Keeping the head sets fixed is insufficient to resolve this confound. A control that matches the routing graph and recomputation budget while changing the execution order would more directly test the proposed mechanism.
Questions for authors
  1. Which routing plan was used for the reported Llama-3.1-8B results? Eq. (8) starts with layer 14 to Input(13), whereas Appendix A.1 describes layer 25 to layer 12 for Retrieval. Could the authors reconcile the source layers and specify the injection sites with a consistent indexing convention?
  2. How are the nonlinear probes fitted and the per-function top-k values selected? Were the probing examples, ordering annotations, and injection-strength selection separated from benchmark evaluation examples, given that CogQA is assembled from reasoning benchmarks?
3 comments
AuthorsOpenReview ↗

Response to Reviewer

Dear Reviewer,

We deeply appreciate the time and effort you have dedicated to reviewing our work. Below, we address the questions you raised.

W. The evidence does not isolate cognitive ordering as the source of the gains. The routing criterion relies on correlational head attribution and human first-occurrence order, neither of which shows that a target requires the source's output; and because injections enter the shared residual stream, the gains may reflect generic activation feedback rather than modules composed in a better order. Table~6 does not resolve this: permuting function labels changes the source--target connections and hop count, so a control that matches the routing graph and recomputation budget while changing only the execution order is needed.

  1. Motivation and Functional Dependencies. We clarify that MoCoC does not assume that correlational head attribution establishes strict functional dependencies, nor that mismatches in representative layer positions necessarily indicate computational errors. Instead, we use human-verified cognitive ordering as an inductive prior to identify potentially beneficial cross-layer feedback pathways.

Importantly, the injected information is intended to influence latent computation (subsequent attention heads and MLPs), enabling the model to explore more effective reasoning pathways by coordinating existing cognitive functions. The case studies in Appendix A.4 further illustrate this effect: MoCoC promotes a more coherent progression from problem understanding to downstream reasoning, whereas mismatched routing can disrupt this functional sequence.

To further examine this motivation, we grouped the 572 CogQA training questions by their number of ordering mismatches (see the table blow). Baseline accuracy (Llama-3.1-8B) decreases from 82.4% (no mismatch) to 74.9% (one) and 73.0% (two or more). The trend also holds across source datasets: datasets with more mismatches are consistently harder, and the mean mismatch count and accuracy are perfectly inversely ranked across the five sources (Spearman ρ = −1.0). Within individual datasets the pattern is less uniform: four of the five sources show lower accuracy on mismatched questions, while ECQA does not. Because the number of mismatches depends on which cognitive functions a task requires, mismatch frequency and task type are intertwined. We therefore regard this as correlational evidence that motivates MoCoC, not as a causal claim.

Table 1. Baseline accuracy (Llama-3.1-8B, LLM-judged) on the 572 CogQA training questions.

(a) By number of cognitive-ordering mismatches.

Mismatches#QAcc.
027982.4
116774.9
≥212673.0

(b) By source dataset, with the mean mismatch count per question and accuracy for questions without (=0) and with (≥1) mismatches.

Source#QMean mism.Acc.Acc. (=0)Acc. (≥1)
GSM8K820.1691.593.180.0
CREAK1170.5882.187.373.9
AQuA920.7080.485.175.6
ECQA1320.8678.068.084.1
e-SNLI1491.2266.469.265.5

Additional empirical validation: To distinguish cognitive-function dependencies from generic cross-layer feedback, we introduce three controls: (i) topology-matched random routing, which preserves the routing graph and computational budget but replaces function-associated heads with equally many random heads from the same layers; (ii) norm-matched routing, which replaces captured signals with random vectors of equal norm; (iii) layer-pair routing, which preserves the number of hops, captures, and injections but randomizes source–target layer pairs.

As shown in following table, MoCoC with cognitive heads achieves the best average accuracy (56.6%), with clear gains on LogiQA (+3.2) and WinoGrande (+2.5). All three controls remove this benefit. Norm-matched injection stays close to the baseline (55.1%, +0.2), and topology-matched routing is slightly below it (54.8%, −0.1). Random layer pairs degrades performance on all three benchmarks (53.3%, −1.6), indicating that arbitrary cross-layer feedback may be harmful. Together, these results show that the improvement depends on both which connections are routed, as determined by the cognitive ordering, and what is routed, namely the signals of the function-associated heads; it is not explained by injecting a perturbation of the same magnitude.

Table 2. Accuracy (%) of Llama-3.1-8B on three reasoning benchmarks.

LogiQAWinoGrandeAQuAAVG
Baseline41.563.459.854.9
Cognitive heads(s=0.3)44.765.959.156.6
Norm-matched41.364.659.455.1
Topology-matched42.262.759.454.8
Layer pairs39.362.458.353.3

Q. Which routing plan was used for the reported Llama-3.1-8B results?

We thank the reviewer for catching the inconsistencies. The routing plan used in all experiments on Llama-3.1-8B is:

14 → Input(12), 25 → Input(23), {24, 25} → Input(13)

The experiments were not affected; only the descriptions were inconsistent. We will correct both in the revision.

Q. How are the nonlinear probes fitted and the per-function top-k values selected? Were the probing examples, ordering annotations, and injection-strength selection separated from benchmark evaluation examples, given that CogQA is assembled from reasoning benchmarks?

Nonlinear Probing and Head Selection. Following [1], we use nonlinear probing to identify attention heads associated with each cognitive function. Head importance is quantified through probe-based attribution, and the top-(k) heads are selected according to their importance scores. The selected heads are localized once offline for each backbone and subsequently reused across downstream reasoning tasks without additional probing or parameter updates. We will provide further details on probe training and the per-function selection of k to improve reproducibility.

""Evaluation Data.** We matched every question in CogQA (Ma et al., 2026) and CogVision (Jiang et al., 2025) against all evaluation sets using 8-gram text overlap and, for CogVision, perceptual image hashing.

CogQA: no overlap. Some CogQA questions match the GSM8K and AQuA training sets (114 and 105, respectively), but none match the GSM8K or AQuA test sets used for evaluation, and none overlap with LogiQA, MMLU, WinoGrande, PIQA or RTE.

CogVision: partial overlap. We found 173 CogVision items that also appear in our MathVista evaluation set and 116 that appear in our MMMU evaluation set, and none from ScienceQA or GQA. After excluding the overlapping questions, MoCoC's macro-average gain over the baseline remains positive on all four VLMs:

ModelFull setsDe-duplicated
Qwen2.5-VL-7B+1.0+0.6
Qwen2.5-VL-3B+0.4+0.2
Gemma-3-4B+0.4+0.7
Gemma-3-12B+1.0+0.9

In both cases, these datasets are used only to derive the cognitive ordering and localize function-associated heads; their answers are never used to update the model or train a routing policy. We will document this overlap in the revision.

[1] Ma, X., Wang, J., Jiang, Y., Erfani, S., Liu, T., & Bailey, J. Cognitive mirrors: Exploring the diverse functional roles of attention heads in llm reasoning. NeurIPS 2025.

Reviewer_34YDOpenReview ↗

Thank you for your rebuttal

Thank you for your detailed rebuttal, which has addressed most of my concerns. Therefore, I will raise my score to 3.

Reviewer_y5emOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MoCoC is a training-free inference-time method. It can derive a global ordering of eight cognitive functions from CogQA decompositions and localizes function-associated attention heads to representative layers. Where a canonically earlier function sits deeper in the network and it routes that function’s head outputs back to the shallower target layer especially during first-token prefill only. It reports macro-average gains of 1.1–3.0 points on four LLMs (seven benchmarks) and 0.4–1.0 on four VLMs, with ablations on capture site, timing, strength and routing order.

Critical strengths

MoCoC is cheap and training-free, because it acts only on prefill, decoding throughput is unchanged, and average gains appear consistently across 8 models and both modalities.

It is quite interesting attempt to turn head-level functional localization into an intervention, with a candid failure analysis: 578 wrong to right vs. 471 right to wrong flips.

Critical weaknesses

The core claim that cognitive ordering drives the gain is not isolated.

  • The order ablation (Table 6) uses only two random permutations with different hop counts. These already reach 66.1–66.6 vs. 67.5 for the canonical order, a gap close to the ~0.7 bootstrap std. Controls with random heads, random layer pairs, or norm-matched injection should need to be considered. The assumption that order across human sub-steps maps to layer depth within a single forward pass is unargued. The localization also looks fragile at 5 of 8 functions tie at layer 14 on Llama-3.1-8B.

The evaluation protocol and consistency are concerning.

  • All design choices (s, injection timing, capture scope) appear to be tuned on the test sets, with no dev split.
  • The routing plan in §3.4 contradicts Appendix A.1.
  • Gemma-12B inner-loop numbers differ between Table 2 and Table 8.
  • The largest gain (+8.0) comes from a near-chance RTE baseline (53.4%).
Questions for authors

Which routing plan and which Gemma-12B numbers are correct?

Were the hyperparameters selected on the test sets? Can you report results with them fixed on a held-out split?

Reshaped outputs are longer and open with “first understand…” steps. How does MoCoC compare to simply prompting the model to restate the question first?

Do CogQA/CogVision overlap with any evaluation benchmark?

3 comments
AuthorsOpenReview ↗

Response to Reviewer

Dear Reviewer,

Thank you very much for your time and effort in reviewing our paper. We have thoughtfully addressed each question you raised and hope our responses alleviate your concerns.

Q. The core claim that cognitive ordering drives the gain is not isolated.

  1. More ablation. To distinguish cognitive-function dependencies from generic cross-layer feedback, we introduce three controls:
  • topology-matched random routing, which preserves the routing graph and computational budget but replaces function-associated heads with equally many random heads from the same layers;
  • norm-matched routing, which replaces captured signals with random vectors of equal norm;
  • layer-pair routing, which preserves the number of hops, captures, and injections but randomizes source–target layer pairs.

As shown in following Table 1, MoCoC with cognitive heads achieves the best average accuracy (56.6%), with clear gains on LogiQA (+3.2) and WinoGrande (+2.5). All three controls remove this benefit. Norm-matched injection stays close to the baseline (55.1%, +0.2), and topology-matched routing is slightly below it (54.8%, −0.1). Random layer pairs degrades performance on all three benchmarks (53.3%, −1.6), indicating that arbitrary cross-layer feedback may be harmful. Together, these results show that the improvement depends on both which connections are routed, as determined by the cognitive ordering, and what is routed, namely the signals of the function-associated heads; it is not explained by injecting a perturbation of the same magnitude.

Table 1. Accuracy (%) of Llama-3.1-8B on three reasoning benchmarks.

LogiQAWinoGrandeAQuAAVG
Baseline41.563.459.854.9
Cognitive heads(s=0.3)44.765.959.156.6
Norm-matched41.364.659.455.1
Topology-matched42.262.759.454.8
Layer pairs39.362.458.353.3
  1. Human reasoning order vs. layer depth. We do not assume that the order of human sub-steps maps directly onto layer depth. Our motivation is weaker: the fixed layer-wise execution order of a Transformer need not align with the functional dependencies a task requires. The ordering derived from human-verified decompositions serves as an inductive prior for proposing cross-layer pathways, not as an exact description of the computational dependencies inside Transformers.

To further examine this motivation, we grouped the 572 CogQA training questions by their number of ordering mismatches (see the table blow). Baseline accuracy (Llama-3.1-8B) decreases from 82.4% (no mismatch) to 74.9% (one) and 73.0% (two or more). The trend also holds across source datasets: datasets with more mismatches are consistently harder, and the mean mismatch count and accuracy are perfectly inversely ranked across the five sources (Spearman ρ = −1.0). Within individual datasets the pattern is less uniform: four of the five sources show lower accuracy on mismatched questions, while ECQA does not. Because the number of mismatches depends on which cognitive functions a task requires, mismatch frequency and task type are intertwined. We therefore regard this as correlational evidence that motivates MoCoC, not as a causal claim.

Table 2. Baseline accuracy (Llama-3.1-8B, LLM-judged) on the 572 CogQA training questions.

(a) By number of cognitive-ordering mismatches.

Mismatches#QAcc.
027982.4
116774.9
≥212673.0

(b) By source dataset, with the mean mismatch count per question and accuracy for questions without (=0) and with (≥1) mismatches.

Source#QMean mism.Acc.Acc. (=0)Acc. (≥1)
GSM8K820.1691.593.180.0
CREAK1170.5882.187.373.9
AQuA920.7080.485.175.6
ECQA1320.8678.068.084.1
e-SNLI1491.2266.469.265.5
  1. Functional localization. Although five functions share representative layer 14 on Llama-3.1-8B, they are associated with largely different attention-head subsets: the pairwise Jaccard overlap of their top-k head sets is only 0.16 on average (0.12–0.24), and 0.24 among their layer-14 heads alone. A representative layer indicates where a function's heads are most concentrated, not that the function is localized exclusively there. The concentration of reasoning-related heads in intermediate layers is also consistent with prior interpretability findings of structured functional specialization across depth [1-2].

[1] Skean, Oscar, et al. "Layer by layer: Uncovering hidden representations in language models." ICML 2025.

[2] Ma, X., Wang, J., Jiang, Y., Erfani, S., Liu, T., & Bailey, J. Cognitive mirrors: Exploring the diverse functional roles of attention heads in llm reasoning. NeurIPS 2025.

Q. The evaluation protocol and consistency are concerning.

Hyperparameter selection and evaluation protocol. The default hyperparameters (injection strength s=0.3, first-token injection, and full top-k capture) were selected on GSM8K. Since this selection used the GSM8K evaluation set, GSM8K should not be regarded as held out. For a few model–benchmark pairs, The following table additionally reported a configuration selected on that benchmark. To remove any dataset-specific tuning, we re-evaluated all models with the single default configuration applied uniformly across the benchmarks:

ModelBaselineMoCoC (uniform config)Δ
Llama-3.1-8B65.367.0+1.7
Llama-3.2-3B55.056.8+1.8
Gemma-3-4B66.068.2+2.2

MoCoC still improves the macro-average on these models, indicating that its gains generalize across tasks rather than depending on dataset-specific optimization.

We thank the reviewer for catching these inconsistencies. The routing plan used in all experiments on Llama-3.1-8B is:

14 → Input(12), 25 → Input(23), {24, 25} → Input(13)

The experiments were not affected; only the descriptions were inconsistent. We will correct both in the revision.

Gemma-3-12B inner-loop results. The numbers in Table 8 are correct (GSM8K 92.9, average 77.9). The value in Table 2 (GSM8K 91.9, average 77.8) is a transcription error.

Q. Reshaped outputs are longer and open with “first understand…” steps. How does MoCoC compare to simply prompting the model to restate the question first?

To address the reviewer's concern, we conducted additional experiments with a Restate-first prompting baseline to examine whether the improvements simply arise from encouraging models to restate the question before reasoning.

As shown in the following table, Restate-first yields inconsistent results, improving performance on some benchmarks while degrading it on others, with a slightly lower average accuracy than the baseline (54.7% vs. 54.9%). In contrast, MoCoC consistently improves performance across all three benchmarks, achieving 57.9% average accuracy and outperforming Restate-first by 3.2 percentage points. These results suggest that MoCoC provides more consistent benefits than simple prompting strategies.

Table 3. Accuracy (%) on three reasoning benchmarks for Llama-3.1-8B.

MethodLogiQAWinoGrandeAQuAAvg.
Baseline41.563.459.854.9
Restate-first42.765.855.554.7
MoCoC44.765.963.057.9

Q. Do CogQA/CogVision overlap with any evaluation benchmark?

We matched every question in CogQA (Ma et al., 2026) and CogVision (Jiang et al., 2025) against all evaluation sets using 8-gram text overlap and, for CogVision, perceptual image hashing.

CogQA: no overlap. Some CogQA questions match the GSM8K and AQuA training sets (114 and 105, respectively), but none match the GSM8K or AQuA test sets used for evaluation, and none overlap with LogiQA, MMLU, WinoGrande, PIQA or RTE.

CogVision: partial overlap. We found 173 CogVision items that also appear in our MathVista evaluation set and 116 that appear in our MMMU evaluation set, and none from ScienceQA or GQA. After excluding the overlapping questions, MoCoC's macro-average gain over the baseline remains positive on all four VLMs:

ModelFull setsDe-duplicated
Qwen2.5-VL-7B+1.0+0.6
Qwen2.5-VL-3B+0.4+0.2
Gemma-3-4B+0.4+0.7
Gemma-3-12B+1.0+0.9

In both cases, these datasets are used only to derive the cognitive ordering and localize function-associated heads; their answers are never used to update the model or train a routing policy. We will document this overlap in the revision.

Thank you for the detailed response and the additional controls experiments. The topology-, norm- and layer-pair-matched experiments have addressed my concerns. I also appreciate the transparent explanation about the GSM8K tuning, the Gemma-3-12B transcription error and the benchmark overlap. These address a substantial part of my concerns.

Before finalising my assessment, could you briefly clarify the following remaining points?

  1. Statistical robustness of the controls: The gaps between the controls and the baseline in Table 1 (+0.2 for norm-matched, −0.1 for topology-matched) are within about 0.7 bootstrap standard deviation which is reported in the paper. And the gain of the cognitive heads comes from LogiQA and WinoGrande (AQuA falls by 0.7). In order to determine the advantage of the cognitive heads exceeds run-to-run noise. Could you report multiple seeds or confidence intervals for these controls, and, if feasible, run them on one further model?

  2. Consistency of reported numbers: MoCoC scores 59.1 on AQuA (average 56.6) in Table 1 but 63.0 (average 57.9) in the restate-first comparison for the same model. Could you clarify which configuration each table uses (if they do have the same configuration, then why does the average vary)?

  3. Routing plan and hyperparameter selection: The plan given in the response (14 → Input(12), 25 → Input(23), {24, 25} → Input(13)) matches neither Eq. (8) nor Appendix A.1. Could you confirm that it is the plan used for all reported results? Also, since s, injection timing and capture scope were selected on GSM8K, could you report results with these fixed on a separate development split?

These clarifications would give me a complete picture for my final evaluation.

Reviewer_MPqhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Modular Chain-of-Cognition (MoCoC), a training-free framework that improves LLM reasoning by reorganizing internal computation according to cognitive-function dependencies. The authors argue that the fixed layer-wise execution order of Transformers may not align with the cognitive ordering required by reasoning tasks. To address this, MoCoC derives a reference cognitive ordering from human-verified reasoning decompositions, identifies function-associated attention heads through nonlinear probing, and routes intermediate representations across layers to better align internal computation with the reference ordering. The method is evaluated across multiple LLMs and vision-language models on diverse reasoning benchmarks, achieving improvements in average accuracy over standard inference. Ablation studies further investigate the effects of source representation selection, injection timing and strength, and cognitive-function ordering.

Critical strengths
  1. The paper introduces an interesting approach to improving LLM reasoning by reorganizing internal computation according to cognitive-function dependencies (Sections 3.1–3.4). Unlike conventional latent reasoning methods that primarily focus on increasing computational depth or modifying execution paths, MoCoC explicitly incorporates cognitive-function information to guide cross-layer routing. This provides a potentially valuable perspective on coordinating existing model capabilities and opens up an interesting direction for future research on latent reasoning.

  2. The paper evaluates MoCoC across multiple LLM architectures, model scales, and reasoning benchmarks (Tables 2–3), including both language-only and multimodal tasks. The results show improvements in average accuracy across all evaluated models, suggesting the potential generalizability of the approach. Furthermore, MoCoC operates without updating model parameters (Section 3.4), making it potentially practical for improving pretrained models at inference time. These aspects demonstrate the potential applicability and practical value of the proposed framework.

Critical weaknesses
  1. The proposed method relies on the assumption that mismatches between the cognitive-function ordering derived from human-annotated reasoning steps and the layer-wise organization of LLMs negatively affect reasoning performance (Sections 1 and 3.1–3.3). However, the paper provides insufficient theoretical or empirical evidence to establish this connection. In particular, the ordering of cognitive functions in human reasoning does not necessarily reflect the functional dependencies within Transformer computations. Moreover, the use of a fixed global cognitive ordering (Section 3.1, Equations 2–3) appears inconsistent with the task-dependent reasoning motivation emphasized in Section 1. These concerns weaken the fundamental justification for the proposed method.

  2. Although MoCoC achieves improvements on several benchmarks (Tables 2–3), the experiments do not sufficiently demonstrate that these gains specifically originate from the proposed cognitive-function-aware routing. In particular, the ordering ablation in Table 6 considers only two random permutations with different routing configurations, making it difficult to isolate the contribution of cognitive ordering from other factors, such as routing topology and additional computation. Furthermore, the relatively modest improvements, particularly on multimodal benchmarks, are not accompanied by statistical significance analysis. Without more rigorous controlled experiments, the results provide limited evidence supporting the paper's central methodological claim.

Questions for authors
  1. The motivation relies on mismatches between the cognitive-function ordering and their corresponding layer locations. Is there empirical evidence that such mismatches actually cause reasoning failures? Can the authors establish a correlation between the degree of mismatch and reasoning performance.
  2. Why should the ordering of cognitive functions in human-annotated reasoning steps correspond to the required computation order within a Transformer forward pass? Moreover, how can a single global ordering capture the task-dependent cognitive dependencies emphasized in Section 1?
  3. How reliable are the function-specific head localization results obtained through nonlinear probing? Are the selected heads and representative functional layers stable across different probing configurations? Additionally, does the required probing process affect the claim that MoCoC is entirely training-free?
  4. Why does an ordering violation defined by representative layer positions necessarily imply a need for backward routing? What theoretical or empirical evidence supports directly aggregating and injecting head outputs across different layers without introducing unintended interference?
  5. How are the computational budgets and hyperparameter tuning procedures controlled across the compared methods? Furthermore, is there any overlap between the data used for cognitive ordering/head localization and the evaluation benchmarks, particularly GSM8K and AQuA?
  6. Although MoCoC improves average accuracy, some individual benchmarks exhibit performance degradation, and the multimodal improvements are relatively small. Can the authors report paired statistical significance tests or confidence intervals for the improvements to establish their reliability?
  7. Although MoCoC improves average accuracy, some individual benchmarks exhibit performance degradation, and the multimodal improvements are relatively small. Can the authors report paired statistical significance tests or confidence intervals for the improvements to establish their reliability?
  8. Why are most ablations restricted to Llama-3.1-8B, and why does the ordering ablation consider only two random permutations with different numbers of routing hops? Would broader evaluations with matched routing budgets and more random permutations provide more reliable conclusions?
  9. Since changing the cognitive ordering also changes the routing directions and target functions, can the current ablation truly isolate the contribution of the proposed cognitive ordering? Would topology-matched random routing and leave-one-function/hop-out ablations provide stronger evidence for the effectiveness of each component?
5 comments
AuthorsOpenReview ↗

Response to Reviewer

Dear Reviewer,

Thank you for your constructive comments and suggestions. Please find our point-by-point responses to your concerns below.

W1, Q2. About the assumption of MoCoC, and fixed global ordering.

  1. Clarifying the motivation. We would like to clarify that MoCoC does not assume that mismatches between the cognitive-function ordering and the layer-wise organization of Transformers necessarily degrade reasoning. Our motivation is weaker: the fixed layer-wise execution order of a Transformer need not align with the functional dependencies a task requires. We therefore investigate whether reorganizing latent computation according to a cognitively informed reference ordering can improve reasoning. The ordering derived from human-verified decompositions serves as an inductive prior for proposing cross-layer pathways, not as an exact description of the computational dependencies inside Transformers.

  2. We also examine whether ordering mismatches are associated with reasoning performance (see our response to Q1). Across source datasets, those with more mismatches are consistently harder (Spearman ρ=−1.0\rho=-1.0 over five datasets), although the association within individual datasets is less uniform.

  3. Global ordering vs. task-dependent reasoning. Cognitive dependencies can vary across tasks. Our global ordering is an aggregated dependency prior estimated from human-verified decompositions, not an optimal ordering for every task. This choice provides a shared, training-free reference structure that requires neither per-task annotations nor a learned routing policy, which keeps MoCoC applicable to new tasks without additional supervision. The task dependence emphasized in Section 1 refers to the gap between a task's functional requirements and the model's fixed layer order; the global ordering is our first, tractable approximation of those requirements. We acknowledge this as a limitation (Section 5) and will discuss task- and input-adaptive orderings as a direction for future work.

W2. Although MoCoC improves performance across benchmarks (Tables 2–3), it remains unclear whether the gains stem specifically from cognitive-function-aware routing. Table 6 includes only two random permutations with differing routing configurations, confounding cognitive ordering with routing topology and computation. Moreover, the modest gains, particularly on multimodal benchmarks, lack statistical significance analysis. More rigorous controls are needed to substantiate the central methodological claim.

Additional empirical validation: To distinguish cognitive-function dependencies from generic cross-layer feedback, we introduce three controls: (i) topology-matched random routing, which preserves the routing graph and computational budget but replaces function-associated heads with equally many random heads from the same layers; (ii) norm-matched routing, which replaces captured signals with random vectors of equal norm; (iii) layer-pair routing, which preserves the number of hops, captures, and injections but randomizes source–target layer pairs.

As shown in Table 1, MoCoC with cognitive heads achieves the best average accuracy (56.6%), with clear gains on LogiQA (+3.2) and WinoGrande (+2.5). All three controls remove this benefit. Norm-matched injection stays close to the baseline (55.1%, +0.2), and topology-matched routing is slightly below it (54.8%, −0.1). Random layer pairs degrades performance on all three benchmarks (53.3%, −1.6), indicating that arbitrary cross-layer feedback may be harmful. Together, these results show that the improvement depends on both which connections are routed, as determined by the cognitive ordering, and what is routed, namely the signals of the function-associated heads; it is not explained by injecting a perturbation of the same magnitude.

Table 1. Accuracy (%) of Llama-3.1-8B on three reasoning benchmarks.

LogiQAWinoGrandeAQuAAVG
Baseline41.563.459.854.9
Cognitive heads(s=0.3)44.765.959.156.6
Norm-matched41.364.659.455.1
Topology-matched42.262.759.454.8
Layer pairs39.362.458.353.3

Q1. Is there evidence that ordering mismatches cause reasoning failures, or that the degree of mismatch correlates with performance?

  1. Role of mismatches. MoCoC does not assume that ordering mismatches cause reasoning failures. Function-associated heads are distributed across many layers, and a representative layer only marks where they are most concentrated, so a mismatch does not mean the model cannot execute a function. Instead, mismatches serve as a heuristic for proposing cross-layer pathways that may better coordinate existing capabilities. MoCoC thus reorganizes internal information flow through training-free, function-guided cross-layer feedback. The case studies in Appendix A.4 illustrate this effect: MoCoC restores a more coherent progression from problem understanding to downstream reasoning.

  2. Mismatch and baseline performance. To further examine this motivation, we grouped the 572 CogQA training questions by their number of ordering mismatches (see the table blow). Baseline accuracy (Llama-3.1-8B) decreases from 82.4% (no mismatch) to 74.9% (one) and 73.0% (two or more). The trend also holds across source datasets: datasets with more mismatches are consistently harder, and the mean mismatch count and accuracy are perfectly inversely ranked across the five sources (Spearman ρ = −1.0). Within individual datasets the pattern is less uniform: four of the five sources show lower accuracy on mismatched questions, while ECQA does not.

Because the number of mismatches depends on which cognitive functions a task requires, mismatch frequency and task type are intertwined. We therefore regard this as correlational evidence that motivates MoCoC, not as a causal claim. The more direct test is interventional: under a matched computational budget, MoCoC's cognitively guided routing outperforms topology-matched random routing, norm-matched injection and random layer pairs (see our response to W2). We will include this analysis and clarify the motivation in the revision.

Table 2. Baseline accuracy (Llama-3.1-8B, LLM-judged) on the 572 CogQA training questions.

(a) By number of cognitive-ordering mismatches.

Mismatches#QAcc.
027982.4
116774.9
≥212673.0

(b) By source dataset, with the mean mismatch count per question and accuracy for questions without (=0) and with (≥1) mismatches.

Source#QMean mism.Acc.Acc. (=0)Acc. (≥1)
GSM8K820.1691.593.180.0
CREAK1170.5882.187.373.9
AQuA920.7080.485.175.6
ECQA1320.8678.068.084.1
e-SNLI1491.2266.469.265.5

Q3. Stability of probe-based head localization, and whether probing conflicts with the training-free claim.

  1. Stability of Functional Head Localization. To assess the reliability of nonlinear probing, we repeated the probing procedure using five different random seeds for each of four model backbones and measured the consistency of attention-head importance rankings using Spearman correlation. As shown in following table, the pairwise correlations range from 0.75 to 0.80 across four different models. These consistently positive correlations indicate that the identified functional importance rankings are relatively stable across different probe initializations, supporting the robustness of our head localization approach.

Table 3. Spearman correlation for different seeds.

ModelMeanMinMax
Llama-3.1-8B0.770.750.79
Llama-3.2-3B0.790.770.80
Qwen3-8B0.770.760.78
Qwen3-4B0.770.760.77
  1. Clarification of the Training-Free Setting. We clarify that functional head localization is performed only once offline for each backbone, independently of downstream evaluation tasks and datasets. The identified functional heads are subsequently reused across different reasoning tasks without additional probing or adaptation. Although offline localization involves training lightweight probes, MoCoC's inference-time routing requires neither backbone parameter updates nor routing-policy training. We will clarify this distinction in the revised manuscript.

Q4. What supports directly aggregating and injecting head outputs across different layers without introducing unintended interference?

Prior theoretical work shows that looped Transformers can implement iterative computations [1] and algorithms [2] further establish that recurrent computation can support multi-step reasoning and simulate latent chain-of-thought. These results motivate extending the effective computational depth of a model through feedback, although they do not establish the optimality of MoCoC's specific routing strategy.

Controlling interference. MoCoC limits interference in three ways: (i) head outputs are projected into the shared hidden space and the injected signal is rescaled to the norm of a single source (Eqs. 9--10); (ii) only function-associated heads are captured, which outperforms injecting the full residual stream (Table 4 in paper ); and (iii) injection is restricted to the prefill pass at a moderate strength, which outperforms every-token injection (Table 4 in paper) and stronger injection (Table 5 in paper). We acknowledge that interference is not fully eliminated: some questions flip from correct to incorrect, although only 14% of these regressions are degenerate outputs (Appendix~A.4).

[1] Giannou et al. (2023). Looped Transformers as Programmable Computers. ICML 2023.

[2] Saunshi et al. (2025). Reasoning with Latent Thoughts: On the Power of Looped Transformers. ICLR 2025.

Q5. How are the computational budgets and hyperparameter tuning procedures controlled across the compared methods? Furthermore, is there any overlap between the data used for cognitive ordering/head localization and the evaluation benchmarks, particularly GSM8K and AQuA?

  1. Computational Budgets and Hyperparameter Settings. We evaluate all methods using the same pretrained backbones and evaluation benchmarks, with baseline configurations following their respective original papers. Table 7 in the Appendix provides an inference-efficiency comparison on Llama-3.1-8B. MoCoC adds modest inference latency without increasing the per-token decoding cost.

  2. Dataset overlap. To check for potential leakage, we matched every CogQA question against the official splits of all seven evaluation benchmarks using 8-gram overlap. A subset of CogQA questions matches the GSM8K and AQuA training sets (114 and 105 questions, respectively), but none match the GSM8K or AQuA test sets used in our evaluation, and none overlap with LogiQA, MMLU, WinoGrande, PIQA or RTE. We therefore find no question-level overlap between CogQA and any evaluation benchmark.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.