- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper separates training-time expert formation from post-training deployment in a channel-composed linear model. Experiments across five datasets examine how formation rules, deployment strategies, and expert count interact.
Critical strengths
Comparing deployment strategies on the same frozen checkpoint is a useful experimental design that helps separate training effects from post-training composition.
Critical weaknesses
The FixedHalf results need a stronger control. Both experts receive identical task gradients, so their difference is not learned through differentiated task updates. Its gains could therefore reflect initialization and subsequent coefficient fitting rather than task-driven expert differentiation.
Moreover, Appendix E.2 reports that linear probes outperform channel composition by 1.440 points on CIFAR-100 and 0.846 points on Tiny ImageNet, but their weights were not retained. Since both methods deploy a single affine classifier, this comparison directly affects the practical motivation.
Next, the permutation audit has limited explanatory reach. It shows that fitted coefficients depend on their channel correspondence, but testing only MomentBeta does not establish how this sensitivity relates to the formation rule.
Finally, the broader takeaway remains largely descriptive. The study establishes that rankings change across settings, but offers limited guidance on when a deployment strategy should succeed. All results also come from one shallow CNN family with linear expert heads.
Questions for authors
Can you compare FixedHalf with a Dense model initialized at the same effective matrix, , under matched effective optimization? How much of the reported improvement survives this control?
Could you rerun and retain the unrestricted probes under matched fitting and selection protocols? Given the reported vision results, when is channel composition preferable to simply refitting the classifier?
Does the aligned-versus-permuted advantage also appear under FixedHalf and Uniform formation? This would help distinguish a formation-dependent effect from a general consequence of fitting channel-specific coefficients.
Beyond the reported functional-contrast statistic, is there a checkpoint property that can explain or predict whether static deployment helps or hurts?
Response to Reviewer 4YJG
We sincerely thank the reviewer for the specific concerns about FixedHalf optimization, unrestricted classifier fitting, channel assignment, and predictive usefulness. We conducted additional controls in revised Section 5.5 and Appendix I while retaining the original five-dataset results.
1. FixedHalf versus matched Dense (Weakness 1; Q1; Appendix B.1, Appendix I.3, Table 31). We agree that FixedHalf produces identical expert task gradients and cannot establish task-driven expert differentiation. We therefore trained a Dense control matched at the initial effective classifier, with half-scaled classifier gradients and replicated gradient clipping. Across five CIFAR-100 seeds, FixedHalf learned-static minus matched Dense was +0.712 percentage points (95% CI [0.283, 1.141]), whereas neutral-half minus Dense was +0.224 ([−0.316, 0.764]). Crucially, static-minus-neutral within the same FixedHalf checkpoints was +0.488 ([0.273, 0.703]). A separate tied-initialization cohort achieved 51.108% mean accuracy. These results support a contribution from post-training composition, not task-induced specialization or identical optimization trajectories.
2. Retained unrestricted probes (Weakness 2; Q2; Section 5.5; Appendix I.4, Table 32). We refit and retained unrestricted affine probes on frozen MomentBeta representations, selecting the fitted probes using validation data. Relative to recomputed channel-static deployment, probe advantages were +1.440 points on CIFAR-100 (CI [0.967, 1.913]) and +0.992 on Tiny ImageNet ([0.601, 1.383]), both 5/5 positive seeds. Tiny ImageNet uses an independent cohort; its static replay discrepancy of −0.13 to −0.32 points is explicitly reported. Channel composition fits 128 coefficients, versus 12,900/25,800 for the unrestricted probes, although both compile into one affine classifier. We therefore do not claim superior accuracy or fewer deployed parameters for channel composition. Its distinction is constrained fitting while preserving frozen expert weights.
3. Assignment sensitivity across formations (Weakness 3; Q3; Section 5.3, Table 3; Appendix I.2, Table 30). We extended the exact-multiset, post-hoc assignment intervention to Uniform and FixedHalf, without refitting checkpoints. On CIFAR-100, aligned-minus-permuted gains were +0.412/+0.529/+0.908 points for MomentBeta/Uniform/FixedHalf, respectively (5/5 positive seeds each). On FSD50K, all corresponding confidence intervals included zero. Thus, correspondence sensitivity also arises under reference-free formation; it is not evidence that reference-dependent formation information is uniquely necessary.
4. Predictive checkpoint properties (Weakness 4; Q4; Appendix I.4–I.5, Table 33). We investigated a validation-gradient diagnostic for channel versus mean-matched constant-gate deployment. Its directional score agreed with positive held-out advantages in 10/10 evaluated CIFAR-100/Tiny ImageNet checkpoints, with mean advantages +0.302/+0.274 points. However, all evaluated advantages were positive, so this does not validate discrimination of unfavorable deployments or prediction of optimal formation–deployment pairs. Additional 40-run nonlinear/expert-count controls reveal formation-ranking reversals, but do not establish sparse-Transformer generalization.
We sincerely appreciate the reviewer’s constructive questions, which motivated stronger tests of optimization, deployment expressivity, and formation-dependent correspondence.