acceptodds
Under review as a conference paper at ICLR 2027

CORTEX: Learning to Share and Specialize in Dense Language Models

12 views

Abstract

Large language models are trained on heterogeneous data mixtures, where different knowledge domains require both shared knowledge and specialization. Existing modular approaches typically impose explicit components or discover modules through interpretability analysis after training. In this work, we propose CORTEX, a learning dynamics-inspired framework that learns internal modularization within dense language models. CORTEX partitions trainable matrices into parameter groups and learns module assignments from domain-conditioned gradient and cross-domain gradient similarity. We introduce the selective lesion score and module-domain mutual information to characterize the target-domain lesion effects and alignment, and analyze how module assignment affects the trade-off between assignment bias and update magnitude. Experiments with 160M, Qwen3-8B, and Qwen3-32B backbone models show that CORTEX achieves the highest synthetic-domain exact match and largest average perplexity reduction, while remaining competitive on real-domain evaluations and forming identifiable modules.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Modularization, learning dynamics, interpretability, large language models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_mgkHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This paper proposes CORTEX, a training framework that learns shared and domain-specialized modules within dense language models. It partitions existing weight matrices into parameter groups and learns soft module assignments using domain-conditioned gradient magnitudes and cross-domain gradient alignment. These assignments reweight the domain gradients used to update each parameter group. At inference time, the trained dense model is used directly without domain-dependent routing. Experiments on a synthetic benchmark and real-domain mixtures show improvements over several dense and modular baselines. Module-removal experiments and matched-shuffle controls provide evidence of domain-selective functional organization.

Critical strengths
  1. CORTEX provides a concrete mechanism for learning shared and specialized assignments within the existing dense parameter space. It preserves dense inference and does not require domain labels or an expert router at deployment. This design offers a useful connection between domain-conditioned optimization and internal modular organization.

  2. The paper provides meaningful empirical evidence beyond assignment visualizations. CORTEX improves synthetic-task performance and achieves higher mean downstream scores than the strongest compared baseline in nine of twelve backbone–domain settings. The functional analysis also includes matched-shuffle controls that preserve the magnitude of removed weights.

Critical weaknesses

The incremental value of the learned assignment mechanism over more direct gradient-weighting approaches is not sufficiently established. Operationally, CORTEX modifies the contribution of each domain gradient to each parameter group. Dense and fixed-random assignment baselines provide useful evidence, but they do not establish whether the proposed mechanism offers advantages over informed gradient weighting or conflict-aware optimization using the same domain information. A particularly relevant comparison would compute assignment weights directly from the same utility scores without learning persistent assignment parameters. Such a comparison would help distinguish the benefit of the gradient-derived utilities from that of learning the assignments over time.

Questions for authors
  1. Can the authors compare CORTEX with an informed gradient-weighting baseline that uses the same domain-conditioned gradients and parameter groups? In particular, how does learning persistent assignment parameters compare with directly mapping the current utility scores to assignment weights? Evaluating both predictive performance and domain-selective lesion effects would help establish the additional value of the proposed assignment learning.

  2. How exactly is “CORTEX w/o Shared” implemented in Table 3? Is the model retrained without the shared assignment option, or is the shared component removed after training? Please specify the assignment normalization, backbone parameter count, training budget, and any controls for changes in update scale or perturbation magnitude.

5 comments
AuthorsOpenReview ↗

Response to Reviewer mgkH Part 1

We thank the reviewer for the comments and for recognizing the connection between domain-conditioned optimization and internal modular organization. We explain the assignment-learning mechanism and the shared-module ablation below.

Weakness 1: The incremental value of the learned assignment mechanism over more direct gradient-weighting approaches is not sufficiently established. Operationally, CORTEX modifies the contribution of each domain gradient to each parameter group. Dense and fixed-random assignment baselines provide useful evidence, but they do not establish whether the proposed mechanism offers advantages over informed gradient weighting or conflict-aware optimization using the same domain information. A particularly relevant comparison would compute assignment weights directly from the same utility scores without learning persistent assignment parameters. Such a comparison would help distinguish the benefit of the gradient-derived utilities from that of learning the assignments over time.

Author's Response: We thank the reviewer for the constructive suggestion. Section 5.2 defines Direct-utility and compares it with CORTEX in Table 4 on the six-domain synthetic benchmark with the 160M backbone. Both approaches use the same domain-gradient computation, utility functions, parameter groups, shared and specialized modules, and backbone update rule. Direct-utility calculates

sj,g,m=exp⁡(uj,g,m)∑r=0Mexp⁡(uj,g,r),s_{j,g,m}=\frac{\exp(u_{j,g,m})}{\sum_{r=0}^{M}\exp(u_{j,g,r})},

using softmax with a temperature of 1 on the current utilities at every update. As described in Section 5.2, CORTEX uses the current utilities to update the assignment parameters carried forward from earlier steps. Both approaches use the same training settings.

To evaluate lesions for Direct-utility, we calculate its assignments once on fixed, domain-balanced training data at the selected checkpoint and hold them fixed for all removals.

Table 4: Prediction and lesion effects on the six-domain 160M benchmark.

Target and Other denote mean NLL changes on the paired and other domains after module removal.

ApproachNLL ↓PPL ↓EM (%) ↑Target ΔNLLOther ΔNLLSLS ↑
Random0.40821.504184.830.00970.00860.0011
Direct-utility0.10801.114196.480.15700.01040.1466
CORTEX0.07341.076297.610.19010.00640.1837

Compared with Direct-utility, CORTEX improves EM from 96.48% to 97.61% and SLS from 0.1466 to 0.1837. Its component removal produces a larger mean increase in paired-domain NLL and a smaller increase on the other domains.

Question 1: Can the authors compare CORTEX with an informed gradient-weighting baseline that uses the same domain-conditioned gradients and parameter groups? In particular, how does learning persistent assignment parameters compare with directly mapping the current utility scores to assignment weights? Evaluating both predictive performance and domain-selective lesion effects would help establish the additional value of the proposed assignment learning.

Author's Response: Please see our response to Weakness 1 above for the Direct-utility formula, matched training settings, fixed-assignment lesion protocol, and full prediction and lesion results. The comparison shows that learning persistent assignments improves both prediction and domain-selective lesion effects.

Question 2: How exactly is “CORTEX w/o Shared” implemented in Table 3? Is the model retrained without the shared assignment option, or is the shared component removed after training? Please specify the assignment normalization, backbone parameter count, training budget, and any controls for changes in update scale or perturbation magnitude.

Author's Response: CORTEX w/o Shared is retrained after removing the shared assignment option. For each parameter group, we apply softmax over the specialized modules only, so their assignment weights sum to one. We use the same specialized utilities and assignment-learning procedure. Both variants use the same backbone parameter count, training data, token budget, optimizer steps, optimizer settings, and learning-rate schedule.

Table 3: Effect of the shared module on the synthetic benchmark.

Metric / domainCORTEX w/o SharedCORTEX
PPL ↓1.13021.0762
NLL ↓0.12240.0734
EM (%) ↑95.5697.61
Module–domain MI1.11821.4560
SLS: COPY0.14820.1976
SLS: REVERSE0.16670.1742
SLS: ADD0.16940.1755
SLS: LOOKUP0.13180.1769
SLS: LOOKUP→ADD0.19750.2054
SLS: REVERSE→LOOKUP0.15580.1726

Including the shared assignment improves the average EM from 95.56% to 97.61% and increases SLS in all six domains.

AuthorsOpenReview ↗

Response to Reviewer mgkH Part 2

Author's Response Continued:

This training ablation uses the same optimizer settings and learning-rate schedule in both variants. Each group's update norm follows from the gradient mixture in that variant. Table 17 in Appendix D.5 addresses the removal magnitude separately by matching the norm of the weights removed at fixed trained checkpoints.

Table 17: SLS and paired differences, mean ± SD across training seeds.

BackboneLearned SLSMatched-shuffle SLSPaired difference
160M0.1837 ± 0.01200.0548 ± 0.0100+0.1289 ± 0.0115
Qwen3-8B0.1452 ± 0.01500.0427 ± 0.0120+0.1026 ± 0.0140

We thank the reviewer for his/her constructive comments. We appreciate his/her time for reviewing the paper. We hope that we have addressed the comments in a satisfactory manner.

AuthorsOpenReview ↗

Follow-up to Reviewer mgkH

We thank the reviewer for the careful review and constructive comments.

We would appreciate your feedback on the Direct-utility comparison added in Section 5.2 and Table 4.

Persistent assignment learning. Both approaches use the same utilities, parameter groups, and backbone update rule. CORTEX improves EM from 96.48% to 97.61% and SLS from 0.1466 to 0.1837 on the 160M benchmark. These results support the predictive and functional benefits of learning persistent assignments.

Shared-module ablation. Our response clarifies that CORTEX w/o Shared is retrained with assignments normalized over the specialized modules, using the same backbone and training budget.

We hope that the added results and clarifications address your main concerns and support a more positive assessment of our paper. Thank you again for your time and consideration.

Reviewer_k8W1OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This study proposes a novel algorithm, which organizes transformer blocks into multiple functional modules, each of which is assigned into a specific domain. This algorithm can transform plain transformers into effective mixture of experts (MoE). Notably, CORTEX can convert all modules in the transformer blocks into effective MoE-like functional modules, whereas traditional MoE uses FFNs only.

Critical strengths

The proposed algorithm is novel and can be complementary to MoE. In this study, CORTEX is compared against multiple variants of MoE, and the empirical evaluation suggests that CORTEX can outperform all the baselines.

Critical weaknesses
  • In this study, CORTEX has been tested with a small subset of domains and 3 models. Therefore, it remains unclear, if CORTEX can be scaled up to a bigger set of domains or other models

  • The authors’ explanation on how the parameters are split into groups is vague and insufficient. In lines 193-196, they write, “For Transformer backbones, the projections Πg,j\Pi_{g,j}, are chosen along the functional axes. For feed-forward network (FFN) matrices, a parameter group is a block along the intermediate channel axis”, which explains little. What are transformer backbones? Are they referring to key, query and value matrices? What are functional axes and intermediate axes?

Questions for authors

Could the authors provide the exact cost for training Cortex? The authors do provide the total amount of their budget and training hours, but I would like to know and compare the cost of training CORTEX to the cost of training other algorithms such as “Dense” and “MoE”.

7 comments
AuthorsOpenReview ↗

Response to Reviewer k8W1 Part 1

We thank the reviewer for the comments and for recognizing the contribution of learning functional organization across attention and FFN parameters. We clarify the model settings, parameter groups, and training costs below.

Weakness 1: In this study, CORTEX has been tested with a small subset of domains and 3 models. Therefore, it remains unclear, if CORTEX can be scaled up to a bigger set of domains or other models

Author's Response: Thank you for pointing this out. In Tables 18 and 19 of the revised manuscript, we have included experiments with twelve knowledge domains and the Llama-3.2-1B pretrained backbone, respectively. Both settings compare CORTEX with Dense, Random, MoE, MoM, UpIT, and Self-MoE.

The twelve-domain experiment uses the 160M backbone. All approaches use the same data splits, 15,360 updates with 64 samples per update, and three paired seeds. The results are:

Table 18: Twelve-domain synthetic benchmark with 160M.

ApproachNLL ↓PPL ↓EM (%) ↑PPL reduction over Dense (%) ↑
Dense0.32001.377190.000.00
Random0.38931.475986.50-7.17
MoE0.24521.277992.437.20
MoM0.21321.237793.7710.13
UpIT0.26181.299391.645.65
Self-MoE0.22651.254293.118.93
CORTEX0.16611.180695.0214.27

With twelve domains and twelve specialized modules, CORTEX achieves 95.02% EM and a PPL of 1.1806, compared with 1.3771 for Dense. This is a 14.27% PPL reduction relative to Dense in the same setting.

For the other model family, we use Llama-3.2-1B pretrained/base on the six-domain real-text benchmark. All approaches share the pretrained checkpoint, tokenizer, data splits, and training budget, with 5,000 updates. Table 12 in Appendix D.1 reports the full settings.

Table 19: Six-domain real-text benchmark with Llama-3.2-1B.

ApproachAverage NLL ↓PPL ↓PPL reduction over Dense (%) ↑
Dense2.20009.02500.00
Random2.23709.3648-3.76
MoE2.15758.64914.17
MoM2.14368.53005.49
UpIT2.14488.54075.37
Self-MoE2.13788.48056.03
CORTEX2.09978.16369.54

On Llama-3.2-1B, CORTEX achieves an average NLL of 2.0997 and a PPL of 8.1636, compared with 9.0250 for Dense. This is a 9.54% PPL reduction relative to Dense within this backbone setting. Both result tables appear in Appendix D.5 and are summarized in Section 5.2.

Weakness 2: The authors’ explanation on how the parameters are split into groups is vague and insufficient. In lines 193-196, they write, “For Transformer backbones, the projections Πg,j\Pi_{g,j}, are chosen along the functional axes. For feed-forward network (FFN) matrices, a parameter group is a block along the intermediate channel axis”, which explains little. What are transformer backbones? Are they referring to key, query and value matrices? What are functional axes and intermediate axes?

Author's Response: The Transformer backbone is the dense language model, including its attention and FFN layers. Section 3.1 defines parameter groups through projection operators and describes their functional axes. Appendix D.1 specifies the grouping axes and channel counts for each attention and FFN matrix, summarized below.

MatricesGrouping axisChannels per group
Attention query, key, valueOutput rows64 for 160M/Llama-1B and 128 for Qwen3-8B/32B
Attention outputInput columns64 for 160M/Llama-1B and 128 for Qwen3-8B/32B
FFN input, gate, upIntermediate output rows128
FFN output, downIntermediate input columns128

For matrices stored as output dimension × input dimension, the intermediate channels are the FFN channels between its input-side and output-side projections. For example, a group contains 128 rows in the up projection or 128 columns in the down projection. Each matrix has its own learned assignments. The projection for a group keeps that group's entries and sets the rest to zero. The disjoint groups sum to the original matrix. Any remaining channels form a smaller group. Other parameters remain trainable through the mixture gradient. Tables 9–12 in Appendix D.1 report the backbone settings, grouping axes, and group sizes.

Question 1: Could the authors provide the exact cost for training Cortex? The authors do provide the total amount of their budget and training hours, but I would like to know and compare the cost of training CORTEX to the cost of training other algorithms such as “Dense” and “MoE”.

Author's Response: Figs. 9 and 10 in Appendix D.5 already compare the training step time and peak memory per GPU of CORTEX, Dense, and MoE. We reproduce the corresponding measurements below.

AuthorsOpenReview ↗

Response to Reviewer k8W1 Part 2

Author's Response Continued:

BackboneApproachStep time (ms)Peak memory (GB/GPU)
160MDense503.92
160MMoE657.92
160MCORTEX625.73
Qwen3-8BDense240016.19
Qwen3-8BMoE300045.38
Qwen3-8BCORTEX283237.17
Qwen3-32BDense360018.19
Qwen3-32BMoE468053.66
Qwen3-32BCORTEX424839.42

The 160M, Qwen3-8B, and Qwen3-32B runs use 1, 4, and 16 H100 80GB GPUs, respectively. Within each backbone, step time covers the same token count. For CORTEX, it includes the full domain-gradient collection window and both assignment and backbone updates. In particular, CORTEX uses less step time and memory than MoE on all three backbones.

We thank the reviewer for his/her constructive comments. We appreciate his/her time for reviewing the paper. We hope that we have addressed the comments in a satisfactory manner.

AuthorsOpenReview ↗

Follow-up to Reviewer k8W1

We thank the reviewer for the careful review and constructive comments.

We would appreciate your feedback on the added experiments in our revised manuscript.

More domains and another model family. Tables 18 and 19 report the twelve-domain and Llama-3.2-1B experiments. CORTEX outperforms all six baselines, reducing PPL over Dense by 14.27% and 9.54%, respectively.

Parameter groups and training costs. Our response specifies the grouping axes and channel counts and reproduces the step-time and memory measurements from Figs. 9 and 10.

We hope that the added results and clarifications address your main concerns and support a more positive assessment of our paper. Thank you again for your time and consideration.

Reviewer_1zsNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CORTEX, a framework for learning shared and domain-specialized modules within dense language models without introducing explicit experts or changing the inference architecture. CORTEX partitions model parameters into groups and learns soft module assignments based on domain-conditioned gradient magnitudes and cross-domain gradient similarities. These assignments determine how gradients from different domains contribute to parameter updates. The authors also introduce two interpretability metrics, selective lesion score (SLS) and module-domain mutual information, and provide theoretical analyses of gradient interference, module specialization, and convergence. Experiments on a 160M model and Qwen3-8B/32B show promising improvements over dense and modular baselines on synthetic and real-domain benchmarks.

Critical strengths
  1. Interesting approach to internal modularization. The paper explores a meaningful direction for encouraging parameter sharing and specialization within dense language models. Using domain-conditioned gradient information to guide parameter organization is intuitive and potentially useful, especially since the method preserves the original dense architecture and introduces no inference-time routing overhead.

  2. Extensive empirical evaluation. The authors evaluate CORTEX across three model scales, synthetic and real-world domain mixtures, and multiple baselines. The results show consistent improvements in synthetic exact match and language-modeling performance. The lesion experiments, module-domain mutual information, and matched-shuffle controls also provide useful initial evidence of domain-dependent parameter organization.

Critical weaknesses
  1. The proposed method is essentially domain-conditioned gradient reweighting, and the claimed functional modularization is insufficiently justified.

Although CORTEX is presented as learning shared and specialized modules, its backbone update can be rewritten as

Wj←Wj−η∑g∑kπk(sj,g,0+sj,g,k)Gj,g,k.W_j \leftarrow W_j-\eta\sum_g\sum_k\pi_k(s_{j,g,0}+s_{j,g,k})G_{j,g,k}.

This is equivalent to applying domain-dependent gradient scaling factors to each parameter group. The learned modules are weighted decompositions of the same underlying dense parameters, rather than independently parameterized or executed components. All domains still use the same dense model at inference time. Therefore, it is unclear what distinguishes the proposed modularization from structured gradient reweighting or implicit regularization. The reported improvements may simply result from suppressing interfering gradients or reducing effective update magnitudes.

The experimental evidence does not adequately resolve this concern. The Random baseline has a different assignment distribution and concentration from CORTEX, while the matched-shuffle experiments only evaluate lesion selectivity on already-trained checkpoints. Moreover, the proposed utility functions are not sufficiently ablated, and direct comparisons with gradient optimization approaches such as PCGrad, CAGrad, or simpler domain-wise gradient scaling are missing. Equal training tokens and optimization steps also do not guarantee equivalent computational budgets across substantially different architectures. Finally, the assumption of a fixed one-to-one correspondence between domains and specialized modules is restrictive, and robustness to overlapping or noisy domain labels and alternative parameter-group granularities remains unexplored.

The interpretability evidence is also not fully convincing. Since the optimization objective explicitly encourages alignment between modules and domain labels, the diagonal gradient-share patterns are partly expected. Although the lesion experiments suggest domain-dependent parameter importance, removing weighted parameter components can perturb the dense network in ways unrelated to removing an independently identifiable function. Stronger evidence, such as module transplantation, targeted adaptation, or controlled functional interventions, is needed to substantiate the modularity claim.

  1. The theoretical results provide limited justification for the proposed assignment mechanism, and some interpretations are overstated.

Proposition 1 correctly characterizes the first-order loss change under mixture-gradient updates, but it does not derive or justify the specific utility functions used by CORTEX. The loss change depends on magnitude-weighted gradient inner products and domain probabilities, whereas the proposed utilities combine normalized gradient magnitudes and cosine similarities. The connection between maximizing these utilities and optimizing the actual training objective remains unclear.

Proposition 2 is essentially an algebraic restatement of the SLS definition and offers limited theoretical insight. Moreover, a positive SLS does not necessarily indicate that removing a module harms its target domain: it can remain positive even when removal improves the target-domain loss, provided the improvement is smaller than on other domains. Proposition 3 establishes a standard mutual information upper bound, but maximizing module-domain mutual information favors domain-pure assignments and does not necessarily reflect a desirable balance between sharing and specialization. Thus, higher mutual information should not automatically be interpreted as better modularization.

Theorem 1 contains a persistent assignment-bias term that need not vanish, so it does not generally establish convergence to a stationary point of the original training objective. More importantly, the theorem applies to arbitrary assignment weights and does not demonstrate that the proposed utility-driven optimization improves the convergence bound. The claimed connection between learning module assignments and improved optimization therefore remains largely heuristic.

Questions for authors
  1. What distinguishes CORTEX from structured domain-conditioned gradient reweighting? Can the authors demonstrate that the learned modules provide functionality beyond what can be achieved through direct gradient scaling, such as independent adaptation, module transplantation, or selective functional control?

  2. How are the proposed utility functions theoretically justified by Proposition 1? Why use normalized gradient magnitudes and cosine similarities rather than magnitude-weighted inner products? Could the authors provide ablations separating these components and comparing them with simpler alternatives?

  3. Can the authors compare CORTEX against gradient optimization methods such as PCGrad and CAGrad, as well as random assignments matched for entropy, update magnitude, and effective learning rates? Could the comparisons also account for differences in computational cost rather than matching only tokens and optimizer steps?

  4. Since module-domain correspondence is explicitly encouraged during training, to what extent do the reported mutual information and lesion results provide independent evidence of functional specialization? Could the authors report absolute target-domain lesion effects alongside SLS and conduct more direct functional interventions?

  5. Given that the assignment-bias term in Theorem 1 may remain nonzero, what convergence guarantees can be established for CORTEX? Is there a theoretical connection between optimizing the proposed utility functions and minimizing the convergence bound?

  6. How does CORTEX behave with overlapping or noisy domain labels, different numbers of specialized modules, or different parameter-group granularities? Does the claimed sharing-specialization benefit persist when the predefined domain boundaries are less distinct?

10 comments
AuthorsOpenReview ↗

Response to Reviewer 1zsN Part 1

We thank the reviewer for the detailed comments and for recognizing the value of learning parameter sharing and specialization within dense language models. We respond to the weaknesses and questions below.

Weakness 1: The proposed method is essentially domain-conditioned gradient reweighting, and the claimed functional modularization is insufficiently justified.

Author's Response: We would like to point out that the gradient-scaling form of the backbone update does not imply that the learned modules lack distinct functional roles. In Section 3.1, we define each module as a weighted combination of parameter groups, and in Section 3.2, we use these assignment weights to determine each domain's contribution to the backbone update. In Appendix D.3, we explain that the modules are soft components within the dense model. Using the same model at inference does not rule out distinct functional roles. Table 20 shows that CORTEX outperforms Matched Random with matched groupwise gradient and optimizer-update norms. Table 17 shows that its learned assignments have higher lesion selectivity at matched removal norms.

Table 17: SLS and paired differences, mean ± SD across training seeds.

BackboneLearned SLSMatched-shuffle SLSPaired difference
160M0.1837 ± 0.01200.0548 ± 0.0100+0.1289 ± 0.0115
Qwen3-8B0.1452 ± 0.01500.0427 ± 0.0120+0.1026 ± 0.0140

Section 5.2 defines Direct-utility and compares it with CORTEX in Table 4 on the six-domain synthetic benchmark with the 160M backbone. Both approaches use the same domain-gradient computation, utility functions, parameter groups, shared and specialized modules, and backbone update rule. Direct-utility calculates

sj,g,m=exp⁡(uj,g,m)∑r=0Mexp⁡(uj,g,r),s_{j,g,m}=\frac{\exp(u_{j,g,m})}{\sum_{r=0}^{M}\exp(u_{j,g,r})},

using softmax with a temperature of 1 on the current utilities at every update. As described in Section 5.2, CORTEX uses the current utilities to update the assignment parameters carried forward from earlier steps. Both approaches use the same training settings.

To evaluate lesions for Direct-utility, we calculate its assignments once on fixed, domain-balanced training data at the selected checkpoint and hold them fixed for all removals.

Table 4: Prediction and lesion effects on the six-domain 160M benchmark.

Target and Other denote mean NLL changes on the paired and other domains after module removal.

ApproachNLL ↓PPL ↓EM (%) ↑Target ΔNLLOther ΔNLLSLS ↑
Random0.40821.504184.830.00970.00860.0011
Direct-utility0.10801.114196.480.15700.01040.1466
CORTEX0.07341.076297.610.19010.00640.1837

Compared with Direct-utility, CORTEX improves EM from 96.48% to 97.61% and SLS from 0.1466 to 0.1837. Its component removal produces a larger mean increase in paired-domain NLL and a smaller increase on the other domains.

Matched Random and PCGrad

The backbone update can be written as groupwise gradient scaling. The key question is whether it matters which parameter groups receive each domain’s updates.

We have included PCGrad and Matched Random on the 160M synthetic and Qwen3-8B real-text benchmarks. They use the same initialization, data splits, training tokens, optimizer steps, optimizer settings, and seeds as CORTEX. Tables 13, 14, and 20 in Appendix D.5 report the per-domain results and standard deviations. The per-domain results are reproduced below.

Table 13: Synthetic benchmark EM (%), mean ± SD across ten training seeds.

ApproachCOPYREVERSEADDLOOKUPLOOKUP→ADDREVERSE→LOOKUP
Dense96.82 ± 0.3191.36 ± 0.4488.27 ± 0.5292.74 ± 0.3984.91 ± 0.6186.38 ± 0.57
Random88.54 ± 0.6687.92 ± 0.5880.76 ± 0.7391.18 ± 0.4773.64 ± 0.8586.95 ± 0.60
MoE98.12 ± 0.2595.06 ± 0.3494.38 ± 0.4195.71 ± 0.3387.46 ± 0.5691.64 ± 0.46
MoM98.67 ± 0.2197.84 ± 0.2796.21 ± 0.3396.14 ± 0.3190.82 ± 0.4994.76 ± 0.39
UpIT96.21 ± 0.3693.71 ± 0.4292.04 ± 0.4896.48 ± 0.3286.39 ± 0.5990.43 ± 0.50
Self-MoE97.84 ± 0.2896.52 ± 0.3193.91 ± 0.4395.18 ± 0.3589.34 ± 0.5295.71 ± 0.37
PCGrad98.52 ± 0.2696.21 ± 0.2994.36 ± 0.3896.25 ± 0.3091.31 ± 0.4893.20 ± 0.43
CORTEX99.42 ± 0.1898.83 ± 0.2297.62 ± 0.2998.14 ± 0.2694.76 ± 0.4196.88 ± 0.33
AuthorsOpenReview ↗

Response to Reviewer 1zsN Part 2

Author's Response Continued:

Table 14: Qwen3-8B NLL, mean ± SD across four training seeds.

ApproachGeneralCodeMathBiomedicalLegalReasoning
Dense1.6766 ± 0.01241.7949 ± 0.01412.0047 ± 0.01681.8220 ± 0.01491.9425 ± 0.01611.9683 ± 0.0157
Random1.7997 ± 0.01751.7924 ± 0.01582.0172 ± 0.01941.9534 ± 0.01822.0053 ± 0.01971.9392 ± 0.0171
MoE1.6642 ± 0.01191.7878 ± 0.01361.9262 ± 0.01551.7947 ± 0.01421.9054 ± 0.01531.9133 ± 0.0149
MoM1.6441 ± 0.01121.6578 ± 0.01241.9290 ± 0.01511.7995 ± 0.01381.8714 ± 0.01471.8385 ± 0.0136
UpIT1.6257 ± 0.01081.7658 ± 0.01312.0039 ± 0.01621.7709 ± 0.01341.8171 ± 0.01401.8288 ± 0.0132
Self-MoE1.6684 ± 0.01211.7373 ± 0.01291.9058 ± 0.01481.8208 ± 0.01451.8583 ± 0.01461.7822 ± 0.0128
PCGrad1.6397 ± 0.01061.6745 ± 0.01241.9168 ± 0.01481.7283 ± 0.01341.8363 ± 0.01411.8757 ± 0.0136
CORTEX1.6199 ± 0.00971.6096 ± 0.01151.8694 ± 0.01371.6778 ± 0.01261.7791 ± 0.01311.8258 ± 0.0124

Table 20: Matched Random on the synthetic and real-text benchmarks. Per-domain entries are mean ± SD across ten seeds for 160M and four seeds for Qwen3-8B. Average denotes the mean across domains.

DomainRandomMatched RandomCORTEX
160M EM (%)
COPY88.54 ± 0.6698.12 ± 0.2599.42 ± 0.18
REVERSE87.92 ± 0.5895.10 ± 0.3398.83 ± 0.22
ADD80.76 ± 0.7392.95 ± 0.4197.62 ± 0.29
LOOKUP91.18 ± 0.4795.44 ± 0.3398.14 ± 0.26
LOOKUP→ADD73.64 ± 0.8589.84 ± 0.5194.76 ± 0.41
REVERSE→LOOKUP86.95 ± 0.6091.63 ± 0.4596.88 ± 0.33
Average84.8393.8597.61
Qwen3-8B NLL
General1.7997 ± 0.01751.6483 ± 0.01111.6199 ± 0.0097
Code1.7924 ± 0.01581.7023 ± 0.01281.6096 ± 0.0115
Math2.0172 ± 0.01941.9371 ± 0.01531.8694 ± 0.0137
Biomedical1.9534 ± 0.01821.7499 ± 0.01381.6778 ± 0.0126
Legal2.0053 ± 0.01971.8608 ± 0.01461.7791 ± 0.0131
Reasoning1.9392 ± 0.01711.8971 ± 0.01411.8258 ± 0.0124
Average1.91791.79931.7303

For Matched Random, we sample one permutation of parameter groups within each matrix before training and keep this permutation fixed. At every update, we apply it to the complete assignment vectors from the paired CORTEX run at that step. The assignment weights therefore change during training. This preserves the distribution of assignment entropy and concentration within each matrix while changing which group receives each vector. As specified in Appendix D.1, the original Random baseline keeps its assignment weights fixed.

For each parameter group, we match the norm of the gradient entering the optimizer and the norm of the optimizer update before weight decay, covering all trainable parameters. The ratio of these norms is the effective step size measured here. Weight decay is applied separately according to the original rule.

CORTEX outperforms PCGrad and Matched Random in every domain of both benchmarks.

Utility ablations

Table 5 in Section 5.2 compares three utility variants using the same 160M training configuration. The first two remove the relative magnitude term or the specialized alignment term, respectively, while keeping the original shared utility unchanged. The third uses inner products normalized by the mean gradient energy within each parameter group, preserving the relative influence of domain gradient magnitudes.

Table 5: Utility ablations on 160M: EM (%) averaged across ten training seeds.

Domainw/o Relative magnitudew/o Specialized alignmentInner-product utilityCORTEX
COPY98.0198.6499.0399.42
REVERSE95.2096.5997.7198.83
ADD92.5594.8296.2297.62
LOOKUP95.3496.5297.3398.14
LOOKUP→ADD88.8491.8193.2894.76
REVERSE→LOOKUP92.6393.7395.3196.88
Average93.7695.3596.4897.61

Full CORTEX has the highest EM in all six domains. The average EM falls by 3.85 percentage points when the relative magnitude term is removed, by 2.26 points when the specialized alignment term is removed, and by 1.13 points when the normalized inner-product utility is used.

Functional interventions

Section 4 defines the components removed after training using the trained assignment weights. Each component’s paired domain is specified before we measure the change in loss.

AuthorsOpenReview ↗

Response to Reviewer 1zsN Part 3

Author's Response Continued:

Table 6 in Section 5.2 reports the absolute target-domain lesion effects. At a fixed trained checkpoint, we remove one specialized component and measure the loss changes on its paired domain and the other domains.

Table 6: NLL, module–domain MI, and lesion effects. Target and Other denote mean NLL changes after module removal.

BackboneApproachNLL ↓Module–domain MITarget ΔNLLOther ΔNLL
160MRandom0.40821.38600.00970.0086
160MCORTEX0.07341.45600.19010.0064
Qwen3-8BRandom1.91791.02380.03740.0108
Qwen3-8BCORTEX1.73031.15340.15570.0106
Qwen3-32BRandom1.65711.19960.02980.0097
Qwen3-32BCORTEX1.47861.33310.16570.0059

On 160M, module removal increases the paired-domain NLL by 0.1901 on average, compared with 0.0064 on the other domains. Qwen3-8B and Qwen3-32B also show larger mean increases on the paired domain than on the other domains.

Table 17 in Appendix D.5 controls the removal magnitude at the same checkpoints. We shuffle complete assignment vectors among same-shaped groups within each matrix and preserve their entropy and concentration distributions. For each matrix and module, we match the Frobenius norm of the removed weights.

At matched removal norms, the mean within-seed advantage of the learned assignments is 0.1289 on 160M and 0.1026 on Qwen3-8B.

Section 4 defines MI from domain-gradient shares and SLS from loss changes after component removal. MI measures alignment with the training domains. The removal experiment measures how much the prediction loss changes on each domain.

Computational cost

Appendix D.3 specifies the common training-token and optimizer-step budgets. Figs. 9 and 10 in Appendix D.5 report step time and peak memory for Dense, Random, MoE, MoM, and CORTEX. CORTEX takes 18–24% more time per step than Dense and uses 1.46–2.30 times its peak memory. It takes less time per step than MoE and MoM and uses less memory than MoE.

Domain labels and module settings

Section 3.1 specifies one specialized module per domain and a shared module for common knowledge. This provides each domain with its own learned assignment weights. Section 3.1 also states that the same parameter group can contribute to multiple modules. Thus, the one-to-one pairing does not impose disjoint parameter sets or assume that domains have no shared knowledge. The pairing determines which domain guides each specialized module. The parameter-group assignments are learned.

Appendix E of the original submission identifies overlapping or noisy domain labels, the number of specialized modules, and parameter-group size as limitations.

Table 3 in Section 5.2 shows that including the shared assignment improves prediction and lesion selectivity, including on tasks with shared underlying operations. The added twelve-domain experiment in Table 18 of Appendix D.5 increases both the number of domains and the number of specialized modules from six to twelve. CORTEX outperforms all six baselines in this setting and reduces PPL by 14.27% relative to Dense.

Table 3: Effect of the shared module on the synthetic benchmark.

Metric / domainCORTEX w/o SharedCORTEX
PPL ↓1.13021.0762
NLL ↓0.12240.0734
EM (%) ↑95.5697.61
Module–domain MI1.11821.4560
SLS: COPY0.14820.1976
SLS: REVERSE0.16670.1742
SLS: ADD0.16940.1755
SLS: LOOKUP0.13180.1769
SLS: LOOKUP→ADD0.19750.2054
SLS: REVERSE→LOOKUP0.15580.1726

Table 18: Twelve-domain synthetic benchmark with 160M.

ApproachNLL ↓PPL ↓EM (%) ↑PPL reduction over Dense (%) ↑
Dense0.32001.377190.000.00
Random0.38931.475986.50-7.17
MoE0.24521.277992.437.20
MoM0.21321.237793.7710.13
UpIT0.26181.299391.645.65
Self-MoE0.22651.254293.118.93
CORTEX0.16611.180695.0214.27

Weakness 2: The theoretical results provide limited justification for the proposed assignment mechanism, and some interpretations are overstated.

Author's Response: Proposition 1 shows that the first-order loss change contains a domain's squared gradient norm and its inner products with other domains. Each inner product equals the product of two gradient norms and their cosine similarity. These terms motivate the two signals in our utility design: the domain's relative gradient contribution and its agreement with other domains.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to the Area Chair

We would like to thank the area chair and all reviewers for their careful reading and feedback. We have revised the manuscript to address the reviewers' main concerns. We summarize the key issues each reviewer raised and the corresponding changes in the revised manuscript as follows:

To Reviewer 1zsN:

  • Functional roles of learned modules: We clarify how learned assignments define shared and specialized parameter components within the dense model. Table 6 reports larger mean loss increases on the paired domains after component removal. Table 17 shows higher lesion selectivity for the learned assignments at matched removal norms.
  • Sources of the performance gains: We added PCGrad and Matched Random in Tables 13, 14, and 20. CORTEX outperforms both in all six domains on 160M and Qwen3-8B. Matched Random preserves matrix-level assignment entropy distributions and matches groupwise gradient and optimizer-update norms before weight decay.
  • Justification of the utility design: Table 5 shows that full CORTEX outperforms all three utility ablations. Proposition 4 in Section 4 establishes conditions under which an assignment change lowers both the expected assignment loss and the expected main loss after one backbone step for general domain weights. Appendix C.5 provides the proof.

To Reviewer k8W1:

  • Scalability across domains and models: We added twelve-domain and Llama-3.2-1B experiments in Tables 18 and 19. CORTEX outperforms all six baselines, reducing PPL over Dense by 14.27% and 9.54%, respectively.
  • Definition of parameter groups: We clarify the grouping axes for attention and FFN matrices. Tables 9–12 in Appendix D.1 report the group sizes and backbone settings.
  • Training cost: We point to Figs. 9 and 10 for training step time and peak memory. CORTEX uses less step time and memory than MoE on all three original backbones. The common training protocol matches data exposure and optimizer updates.

To Reviewer mgkH:

  • Benefit of persistent assignment learning: We added Direct-utility, which uses the same utility scores and training settings, in Section 5.2 and Table 4. CORTEX improves EM from 96.48% to 97.61% and SLS from 0.1466 to 0.1837, supporting the predictive and functional benefits of persistent assignment learning.
  • Shared-module ablation protocol: We clarify that CORTEX w/o Shared is retrained with assignments normalized over the specialized modules, using the same backbone and training protocol. Table 3 shows that including the shared assignment improves EM from 95.56% to 97.61% and SLS in all six domains.

Finally, we thank the area chair for carefully reading this response and the reviewers for their constructive comments. We hope that we have addressed their comments in a satisfactory manner.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.