- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes CORTEX, a training framework that learns shared and domain-specialized modules within dense language models. It partitions existing weight matrices into parameter groups and learns soft module assignments using domain-conditioned gradient magnitudes and cross-domain gradient alignment. These assignments reweight the domain gradients used to update each parameter group. At inference time, the trained dense model is used directly without domain-dependent routing. Experiments on a synthetic benchmark and real-domain mixtures show improvements over several dense and modular baselines. Module-removal experiments and matched-shuffle controls provide evidence of domain-selective functional organization.
Critical strengths
-
CORTEX provides a concrete mechanism for learning shared and specialized assignments within the existing dense parameter space. It preserves dense inference and does not require domain labels or an expert router at deployment. This design offers a useful connection between domain-conditioned optimization and internal modular organization.
-
The paper provides meaningful empirical evidence beyond assignment visualizations. CORTEX improves synthetic-task performance and achieves higher mean downstream scores than the strongest compared baseline in nine of twelve backbone–domain settings. The functional analysis also includes matched-shuffle controls that preserve the magnitude of removed weights.
Critical weaknesses
The incremental value of the learned assignment mechanism over more direct gradient-weighting approaches is not sufficiently established. Operationally, CORTEX modifies the contribution of each domain gradient to each parameter group. Dense and fixed-random assignment baselines provide useful evidence, but they do not establish whether the proposed mechanism offers advantages over informed gradient weighting or conflict-aware optimization using the same domain information. A particularly relevant comparison would compute assignment weights directly from the same utility scores without learning persistent assignment parameters. Such a comparison would help distinguish the benefit of the gradient-derived utilities from that of learning the assignments over time.
Questions for authors
-
Can the authors compare CORTEX with an informed gradient-weighting baseline that uses the same domain-conditioned gradients and parameter groups? In particular, how does learning persistent assignment parameters compare with directly mapping the current utility scores to assignment weights? Evaluating both predictive performance and domain-selective lesion effects would help establish the additional value of the proposed assignment learning.
-
How exactly is “CORTEX w/o Shared” implemented in Table 3? Is the model retrained without the shared assignment option, or is the shared component removed after training? Please specify the assignment normalization, backbone parameter count, training budget, and any controls for changes in update scale or perturbation magnitude.
5 comments
Response to Reviewer mgkH Part 2
Author's Response Continued:
This training ablation uses the same optimizer settings and learning-rate schedule in both variants. Each group's update norm follows from the gradient mixture in that variant. Table 17 in Appendix D.5 addresses the removal magnitude separately by matching the norm of the weights removed at fixed trained checkpoints.
Table 17: SLS and paired differences, mean ± SD across training seeds.
| Backbone | Learned SLS | Matched-shuffle SLS | Paired difference |
|---|---|---|---|
| 160M | 0.1837 ± 0.0120 | 0.0548 ± 0.0100 | +0.1289 ± 0.0115 |
| Qwen3-8B | 0.1452 ± 0.0150 | 0.0427 ± 0.0120 | +0.1026 ± 0.0140 |
We thank the reviewer for his/her constructive comments. We appreciate his/her time for reviewing the paper. We hope that we have addressed the comments in a satisfactory manner.
Follow-up to Reviewer mgkH
We thank the reviewer for the careful review and constructive comments.
We would appreciate your feedback on the Direct-utility comparison added in Section 5.2 and Table 4.
Persistent assignment learning. Both approaches use the same utilities, parameter groups, and backbone update rule. CORTEX improves EM from 96.48% to 97.61% and SLS from 0.1466 to 0.1837 on the 160M benchmark. These results support the predictive and functional benefits of learning persistent assignments.
Shared-module ablation. Our response clarifies that CORTEX w/o Shared is retrained with assignments normalized over the specialized modules, using the same backbone and training budget.
We hope that the added results and clarifications address your main concerns and support a more positive assessment of our paper. Thank you again for your time and consideration.
Response to Reviewer mgkH Part 1
We thank the reviewer for the comments and for recognizing the connection between domain-conditioned optimization and internal modular organization. We explain the assignment-learning mechanism and the shared-module ablation below.
Weakness 1: The incremental value of the learned assignment mechanism over more direct gradient-weighting approaches is not sufficiently established. Operationally, CORTEX modifies the contribution of each domain gradient to each parameter group. Dense and fixed-random assignment baselines provide useful evidence, but they do not establish whether the proposed mechanism offers advantages over informed gradient weighting or conflict-aware optimization using the same domain information. A particularly relevant comparison would compute assignment weights directly from the same utility scores without learning persistent assignment parameters. Such a comparison would help distinguish the benefit of the gradient-derived utilities from that of learning the assignments over time.
Author's Response: We thank the reviewer for the constructive suggestion. Section 5.2 defines Direct-utility and compares it with CORTEX in Table 4 on the six-domain synthetic benchmark with the 160M backbone. Both approaches use the same domain-gradient computation, utility functions, parameter groups, shared and specialized modules, and backbone update rule. Direct-utility calculates
sj,g,m=∑r=0Mexp(uj,g,r)exp(uj,g,m),
using softmax with a temperature of 1 on the current utilities at every update. As described in Section 5.2, CORTEX uses the current utilities to update the assignment parameters carried forward from earlier steps. Both approaches use the same training settings.
To evaluate lesions for Direct-utility, we calculate its assignments once on fixed, domain-balanced training data at the selected checkpoint and hold them fixed for all removals.
Table 4: Prediction and lesion effects on the six-domain 160M benchmark.
Target and Other denote mean NLL changes on the paired and other domains after module removal.
Compared with Direct-utility, CORTEX improves EM from 96.48% to 97.61% and SLS from 0.1466 to 0.1837. Its component removal produces a larger mean increase in paired-domain NLL and a smaller increase on the other domains.
Question 1: Can the authors compare CORTEX with an informed gradient-weighting baseline that uses the same domain-conditioned gradients and parameter groups? In particular, how does learning persistent assignment parameters compare with directly mapping the current utility scores to assignment weights? Evaluating both predictive performance and domain-selective lesion effects would help establish the additional value of the proposed assignment learning.
Author's Response: Please see our response to Weakness 1 above for the Direct-utility formula, matched training settings, fixed-assignment lesion protocol, and full prediction and lesion results. The comparison shows that learning persistent assignments improves both prediction and domain-selective lesion effects.
Question 2: How exactly is “CORTEX w/o Shared” implemented in Table 3? Is the model retrained without the shared assignment option, or is the shared component removed after training? Please specify the assignment normalization, backbone parameter count, training budget, and any controls for changes in update scale or perturbation magnitude.
Author's Response: CORTEX w/o Shared is retrained after removing the shared assignment option. For each parameter group, we apply softmax over the specialized modules only, so their assignment weights sum to one. We use the same specialized utilities and assignment-learning procedure. Both variants use the same backbone parameter count, training data, token budget, optimizer steps, optimizer settings, and learning-rate schedule.
Table 3: Effect of the shared module on the synthetic benchmark.
Including the shared assignment improves the average EM from 95.56% to 97.61% and increases SLS in all six domains.