- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces DEX (Data-Emergent Experts), a modular language modeling architecture that aims to enable selective capability removal without requiring additional training at deployment time. Rather than relying on manually defined domains, DEX clusters the pretraining corpus using semantic embeddings and assigns a feed-forward expert to each cluster. Expert contributions are determined by a fixed, similarity-based router, where the routing weights are computed using a softmax over the similarities between the input embedding and the cluster centroids. Unlike sparse MoE architectures, all experts remain active during inference. To remove a particular capability, the corresponding experts are disabled by setting their gating weights to zero, with an option to renormalize the remaining weights.
The authors evaluate DEX at two model scales, 50M and 500M parameters, across four domains - biology, cybersecurity, nuclear physics, and code. They compare against GRAM and standard MoE variants, and investigate two strategies for selecting experts for removal (nearest-centroid coverage and cumulative gate mass). The experiments examine how forgetting and retention change as more experts are removed, as well as the effect of gate renormalization. The results suggest that DEX can substantially reduce the likelihood assigned to target-domain text while maintaining relatively small changes in unrelated domains under certain configurations. The paper also includes qualitative analyses of the discovered clusters and generation examples following ablation.
Critical strengths
One aspect I particularly appreciate is the attempt to move away from predefined domain partitions when constructing removable experts. This seems like a reasonable direction, especially considering that real-world pretraining corpora rarely have clean domain boundaries, and obtaining reliable domain annotations at scale is not always feasible. Allowing the data itself to determine how the experts are organized could make the approach more flexible than methods that require a fixed set of domains before training.
While unsupervised domain discovery and embedding-based routing are not new ideas, their use in a jointly trained architecture designed for post-training capability ablation is interesting. The fixed router also offers a degree of transparency, since the routing decisions can be traced back to semantic clusters rather than relying entirely on learned gating behavior. I see this as a useful property for a system intended to provide some control over which capabilities remain accessible after deployment.
I also appreciate that the authors do not restrict their evaluation to removing a single expert. The experiments investigate how the removal of additional experts affects both target-domain forgetting and the performance of the remaining model. This is important because stronger ablation does not necessarily imply better selective removal, particularly if it also causes substantial degradation elsewhere.
Some of the results are encouraging. For instance, at 50M parameters, removing a single expert produces a noticeably larger reduction in target-domain log-likelihood than GRAM for biology and nuclear physics, while the reported changes in retained and core-domain performance remain relatively small. At 500M parameters, removing nine experts for biology results in a forgetting change of 1.830 nats, compared with changes of only 0.028 and 0.043 nats on the retain and core sets, respectively (Section 5.3).
The comparison between renormalized and non-renormalized ablation is also useful, as it shows how much of the observed forgetting can be traded for better retention. Together with the two expert-selection strategies, these experiments give a reasonable picture of how the method behaves under different removal budgets. Although I have concerns about what the forgetting metric actually establishes, the observed domain-selective effects are worth investigating.
Critical weaknesses
My main concern is with the way the paper evaluates capability removal. The motivation is centered on eliminating potentially harmful or dual-use capabilities from language models, but the primary forgetting metric is the change in average gold-token log-probability on domain-specific text (Section 4.3). A reduction in this metric shows that the model becomes less likely to predict those sequences after ablation. However, I do not think this is sufficient evidence that the underlying knowledge or capability has actually been removed.
This distinction is especially relevant given how the ablation is implemented. When experts are removed without renormalization, the total active gate mass decreases, effectively reducing the contribution of the feed-forward components. As a result, part of the observed forgetting could simply be due to reduced model capacity or a general disruption of the representations used during generation, rather than the selective removal of domain-specific knowledge.
The results at larger removal budgets illustrate this concern. For biology at 50M parameters, removing 23 out of 46 experts produces a forgetting change of 2.327 nats, but the retain and core losses also increase by 0.936 and 0.511 nats, respectively (Section 5.3). These are fairly substantial changes, and it becomes difficult to interpret the higher forgetting value as evidence of successful capability removal without considering the accompanying degradation.
I had a similar concern when looking at the generation examples in Appendix C. Several of the ablated outputs become repetitive, incoherent, or empty, rather than clearly demonstrating that specific knowledge is no longer available. For example, code generation becomes empty under sufficiently aggressive non-renormalized ablation. I also found some of the chosen examples difficult to connect to the safety motivation. The cybersecurity example concerns bussing blocks and appliance timing mechanisms, while the nuclear example discusses mirror supernovae and gamma rays. Although these may belong to the respective source domains, they do not directly demonstrate the removal of the potentially dangerous capabilities discussed in the introduction.
More importantly, the paper does not examine whether the supposedly removed capabilities can still be elicited through different prompts, related reasoning tasks, or subsequent fine-tuning. There is also no comparison with a model trained without the target-domain data, which would provide a useful reference for what successful removal should look like. This is particularly relevant given the paper's discussion of the limitations of conventional unlearning methods and the possibility of recovering suppressed knowledge.
TBH, I think the experiments provide evidence that DEX can selectively reduce the likelihood of target-domain text, but the stronger claim that it removes the corresponding capabilities is not yet convincingly established. Since this distinction is central to the paper's motivation and intended safety application, I consider it a significant limitation rather than simply a matter of evaluation preference.
Also, I have some reservations about the comparison with GRAM, particularly whether the experimental setup is sufficiently controlled to attribute the observed differences to the proposed architecture. In Section 4.2, DEX is trained on the deduplicated Pile, whereas GRAM is reproduced using its original training setup, which combines FineWeb-Edu with four labeled auxiliary domains. The authors match the parameter counts and training token budgets, which is useful, but the two models are still exposed to different training distributions. In particular, it is not clear whether the relative amounts of biology, cybersecurity, nuclear, and code data are comparable. Differences in the reported forgetting and retention could therefore be influenced by the training data rather than the choice between semantic clustering and predefined domain assignments. Matching parameter counts and token budgets also does not necessarily imply equivalent computational costs, especially considering the additional embedding-based routing in DEX.
The way the ablation budgets are compared raises another concern. In Figure 3, removing one DEX expert is treated as equivalent to removing one GRAM auxiliary module, although these components serve different architectural roles and are not shown to contain comparable numbers of parameters or computational contributions. Interestingly, at 500M parameters, GRAM achieves stronger single-unit forgetting on three of the four domains. DEX only surpasses GRAM after additional experts are removed, but those results are still compared against GRAM's fixed single-module ablation.
I am not convinced that this establishes a clear advantage under comparable removal budgets. It would be more informative to compare the methods at similar levels of retained performance or equivalent fractions of disabled parameters, rather than using the number of removed modules as the primary basis of comparison.
There is also a more fundamental question about how much specialization the discovered experts actually achieve. The paper does not include a random-expert-ablation baseline, an equal-capacity model using a different partitioning strategy, or a direct analysis of whether the knowledge attributed to one expert is absent from the others. These controls seem particularly important because DEX is trained jointly, all experts contribute to every input, and the shared transformer parameters remain active. Although a fixed semantic router may encourage specialization, it does not necessarily mean that the corresponding knowledge is confined to the expert associated with a particular cluster.
Appendix B provides an interesting qualitative analysis of the discovered clusters, but I think its interpretation should be more cautious. Showing that documents close to a centroid have coherent topic labels primarily demonstrates that the embedding model produces semantically meaningful clusters. It does not directly establish that the corresponding expert has learned an independently localized capability. The reported average confidence is also computed using documents selected for their proximity to the centroids, which may not be representative of all documents assigned to each cluster.
I would be careful about how the novelty of data-emergent clustering is framed. Gururangan et al. (2023) already investigated the imperfect correspondence between semantic clusters and predefined domain labels. In my view, the more distinctive contribution of DEX is not the observation that predefined domains fail to align with the natural structure of the data, but the attempt to use this structure for selective capability ablation. However, to support that contribution, the experiments need to establish more directly that the proposed clustering is responsible for the observed selectivity, rather than simply providing another way to partition the model's parameters.
Questions for authors
-
How can we distinguish genuine capability removal from a general reduction in language modeling quality?
-
Although the models are matched in parameter count and training tokens, they appear to use different pretraining distributions. What proportions of the training data correspond to the evaluated domains in each case? It would also help to know how many parameters are disabled by each intervention and whether there are differences in training or inference FLOPs. More importantly, would DEX still show an advantage if the methods were compared at similar retention losses or equivalent fractions of removed parameters, rather than comparing multiple DEX experts against a single GRAM module?
-
Is there direct evidence that the semantic clusters correspond to independently localized capabilities within the trained model? Since the experts are trained jointly and all participate in the forward pass, it would be helpful to understand how much of the observed forgetting is specifically attributable to the proposed semantic specialization (e.g., how would randomly removing the same number of experts compare with removing the experts selected by the proposed strategies?) A quantitative cross-domain specialization analysis or comparison with alternative partitioning methods would help clarify whether the claimed finer-grained capability control actually follows from the data-emergent clustering.
-
Are the prompts used to identify experts separate from the prompts used to evaluate forgetting?