acceptodds
Under review as a conference paper at ICLR 2027

DEX: Data-Emergent Experts for Capability Ablation in Large Language Models

Abstract

Modern language models are trained on vast corpora that inevitably contain harmful or dual-use content, and a capability that helps a legitimate user can just as easily assist a malicious one. A safe deployment therefore benefits from the ability to strip a specific capability from a trained model on demand, without paying to retrain and without degrading everything else the model does. We propose DEX (Data-Emergent Experts), a dense model inspired by the Mixture-of-Experts architecture but with an embedding-based router instead of a learned one. We first cluster the training corpus by each document’s meaning using a pretrained embed- ding model, letting the data’s own structure decide both the categories and which documents belong to each, rather than relying on a predetermined set of topics and fixed assignments. Each cluster is assigned its own expert, and unlike standard sparse routing, all experts stay enabled: every non-ablated expert contributes to each input, weighted by a softmax over the similarity between the input embed- ding and the cluster centroids. Removing a capability is then as simple as zeroing the corresponding experts. We study how to balance capability removal against the retention of knowledge we want to keep, and we find each cluster captures a specific interpretable specialization. DEX forgets far more of the target capability while marginally affecting the rest. Additionally, since knowledge is split into finer clusters, DEX can remove narrower subspaces where prior methods can only drop whole domains.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
capability ablation, modular pretraining, embedding-based routing, dual-use, AI safety, data attribution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_r1nPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces DEX (Data-Emergent Experts), a modular language modeling architecture that aims to enable selective capability removal without requiring additional training at deployment time. Rather than relying on manually defined domains, DEX clusters the pretraining corpus using semantic embeddings and assigns a feed-forward expert to each cluster. Expert contributions are determined by a fixed, similarity-based router, where the routing weights are computed using a softmax over the similarities between the input embedding and the cluster centroids. Unlike sparse MoE architectures, all experts remain active during inference. To remove a particular capability, the corresponding experts are disabled by setting their gating weights to zero, with an option to renormalize the remaining weights.

The authors evaluate DEX at two model scales, 50M and 500M parameters, across four domains - biology, cybersecurity, nuclear physics, and code. They compare against GRAM and standard MoE variants, and investigate two strategies for selecting experts for removal (nearest-centroid coverage and cumulative gate mass). The experiments examine how forgetting and retention change as more experts are removed, as well as the effect of gate renormalization. The results suggest that DEX can substantially reduce the likelihood assigned to target-domain text while maintaining relatively small changes in unrelated domains under certain configurations. The paper also includes qualitative analyses of the discovered clusters and generation examples following ablation.

Critical strengths

One aspect I particularly appreciate is the attempt to move away from predefined domain partitions when constructing removable experts. This seems like a reasonable direction, especially considering that real-world pretraining corpora rarely have clean domain boundaries, and obtaining reliable domain annotations at scale is not always feasible. Allowing the data itself to determine how the experts are organized could make the approach more flexible than methods that require a fixed set of domains before training.

While unsupervised domain discovery and embedding-based routing are not new ideas, their use in a jointly trained architecture designed for post-training capability ablation is interesting. The fixed router also offers a degree of transparency, since the routing decisions can be traced back to semantic clusters rather than relying entirely on learned gating behavior. I see this as a useful property for a system intended to provide some control over which capabilities remain accessible after deployment.

I also appreciate that the authors do not restrict their evaluation to removing a single expert. The experiments investigate how the removal of additional experts affects both target-domain forgetting and the performance of the remaining model. This is important because stronger ablation does not necessarily imply better selective removal, particularly if it also causes substantial degradation elsewhere.

Some of the results are encouraging. For instance, at 50M parameters, removing a single expert produces a noticeably larger reduction in target-domain log-likelihood than GRAM for biology and nuclear physics, while the reported changes in retained and core-domain performance remain relatively small. At 500M parameters, removing nine experts for biology results in a forgetting change of 1.830 nats, compared with changes of only 0.028 and 0.043 nats on the retain and core sets, respectively (Section 5.3).

The comparison between renormalized and non-renormalized ablation is also useful, as it shows how much of the observed forgetting can be traded for better retention. Together with the two expert-selection strategies, these experiments give a reasonable picture of how the method behaves under different removal budgets. Although I have concerns about what the forgetting metric actually establishes, the observed domain-selective effects are worth investigating.

Critical weaknesses

My main concern is with the way the paper evaluates capability removal. The motivation is centered on eliminating potentially harmful or dual-use capabilities from language models, but the primary forgetting metric is the change in average gold-token log-probability on domain-specific text (Section 4.3). A reduction in this metric shows that the model becomes less likely to predict those sequences after ablation. However, I do not think this is sufficient evidence that the underlying knowledge or capability has actually been removed.

This distinction is especially relevant given how the ablation is implemented. When experts are removed without renormalization, the total active gate mass decreases, effectively reducing the contribution of the feed-forward components. As a result, part of the observed forgetting could simply be due to reduced model capacity or a general disruption of the representations used during generation, rather than the selective removal of domain-specific knowledge.

The results at larger removal budgets illustrate this concern. For biology at 50M parameters, removing 23 out of 46 experts produces a forgetting change of 2.327 nats, but the retain and core losses also increase by 0.936 and 0.511 nats, respectively (Section 5.3). These are fairly substantial changes, and it becomes difficult to interpret the higher forgetting value as evidence of successful capability removal without considering the accompanying degradation.

I had a similar concern when looking at the generation examples in Appendix C. Several of the ablated outputs become repetitive, incoherent, or empty, rather than clearly demonstrating that specific knowledge is no longer available. For example, code generation becomes empty under sufficiently aggressive non-renormalized ablation. I also found some of the chosen examples difficult to connect to the safety motivation. The cybersecurity example concerns bussing blocks and appliance timing mechanisms, while the nuclear example discusses mirror supernovae and gamma rays. Although these may belong to the respective source domains, they do not directly demonstrate the removal of the potentially dangerous capabilities discussed in the introduction.

More importantly, the paper does not examine whether the supposedly removed capabilities can still be elicited through different prompts, related reasoning tasks, or subsequent fine-tuning. There is also no comparison with a model trained without the target-domain data, which would provide a useful reference for what successful removal should look like. This is particularly relevant given the paper's discussion of the limitations of conventional unlearning methods and the possibility of recovering suppressed knowledge.

TBH, I think the experiments provide evidence that DEX can selectively reduce the likelihood of target-domain text, but the stronger claim that it removes the corresponding capabilities is not yet convincingly established. Since this distinction is central to the paper's motivation and intended safety application, I consider it a significant limitation rather than simply a matter of evaluation preference.

Also, I have some reservations about the comparison with GRAM, particularly whether the experimental setup is sufficiently controlled to attribute the observed differences to the proposed architecture. In Section 4.2, DEX is trained on the deduplicated Pile, whereas GRAM is reproduced using its original training setup, which combines FineWeb-Edu with four labeled auxiliary domains. The authors match the parameter counts and training token budgets, which is useful, but the two models are still exposed to different training distributions. In particular, it is not clear whether the relative amounts of biology, cybersecurity, nuclear, and code data are comparable. Differences in the reported forgetting and retention could therefore be influenced by the training data rather than the choice between semantic clustering and predefined domain assignments. Matching parameter counts and token budgets also does not necessarily imply equivalent computational costs, especially considering the additional embedding-based routing in DEX.

The way the ablation budgets are compared raises another concern. In Figure 3, removing one DEX expert is treated as equivalent to removing one GRAM auxiliary module, although these components serve different architectural roles and are not shown to contain comparable numbers of parameters or computational contributions. Interestingly, at 500M parameters, GRAM achieves stronger single-unit forgetting on three of the four domains. DEX only surpasses GRAM after additional experts are removed, but those results are still compared against GRAM's fixed single-module ablation.

I am not convinced that this establishes a clear advantage under comparable removal budgets. It would be more informative to compare the methods at similar levels of retained performance or equivalent fractions of disabled parameters, rather than using the number of removed modules as the primary basis of comparison.

There is also a more fundamental question about how much specialization the discovered experts actually achieve. The paper does not include a random-expert-ablation baseline, an equal-capacity model using a different partitioning strategy, or a direct analysis of whether the knowledge attributed to one expert is absent from the others. These controls seem particularly important because DEX is trained jointly, all experts contribute to every input, and the shared transformer parameters remain active. Although a fixed semantic router may encourage specialization, it does not necessarily mean that the corresponding knowledge is confined to the expert associated with a particular cluster.

Appendix B provides an interesting qualitative analysis of the discovered clusters, but I think its interpretation should be more cautious. Showing that documents close to a centroid have coherent topic labels primarily demonstrates that the embedding model produces semantically meaningful clusters. It does not directly establish that the corresponding expert has learned an independently localized capability. The reported average confidence is also computed using documents selected for their proximity to the centroids, which may not be representative of all documents assigned to each cluster.

I would be careful about how the novelty of data-emergent clustering is framed. Gururangan et al. (2023) already investigated the imperfect correspondence between semantic clusters and predefined domain labels. In my view, the more distinctive contribution of DEX is not the observation that predefined domains fail to align with the natural structure of the data, but the attempt to use this structure for selective capability ablation. However, to support that contribution, the experiments need to establish more directly that the proposed clustering is responsible for the observed selectivity, rather than simply providing another way to partition the model's parameters.

Questions for authors
  1. How can we distinguish genuine capability removal from a general reduction in language modeling quality?

  2. Although the models are matched in parameter count and training tokens, they appear to use different pretraining distributions. What proportions of the training data correspond to the evaluated domains in each case? It would also help to know how many parameters are disabled by each intervention and whether there are differences in training or inference FLOPs. More importantly, would DEX still show an advantage if the methods were compared at similar retention losses or equivalent fractions of removed parameters, rather than comparing multiple DEX experts against a single GRAM module?

  3. Is there direct evidence that the semantic clusters correspond to independently localized capabilities within the trained model? Since the experts are trained jointly and all participate in the forward pass, it would be helpful to understand how much of the observed forgetting is specifically attributable to the proposed semantic specialization (e.g., how would randomly removing the same number of experts compare with removing the experts selected by the proposed strategies?) A quantitative cross-domain specialization analysis or comparison with alternative partitioning methods would help clarify whether the claimed finer-grained capability control actually follows from the data-emergent clustering.

  4. Are the prompts used to identify experts separate from the prompts used to evaluate forgetting?

Reviewer_vX2kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DEX uses document clusters to define removable feed-forward experts in a transformer. A pretrained, frozen sentence encoder provides embeddings for clustering and centroid-based routing. The main language model is then trained from scratch on the deduplicated Pile. All non-ablated experts contribute, with one gate shared across layers and token positions. Target-domain probes rank the experts for removal, optionally followed by renormalization of the remaining weights. Experiments use models reported as 50M and 500M parameters, with 46 and 69 experts per layer. Evaluation measures changes in next-token log-probability on biology, cybersecurity, nuclear physics, code, and general text. The proposed benefit is to obtain removable units from the corpus without predefined training-domain labels.

Critical strengths
  1. Experts are defined without predefined domain labels. DEX uses document clustering to define its experts, avoiding the need to assign a predefined capability label to every training document. This is useful when a suitable domain partition is difficult to specify in advance.
  2. Some ablations show good selectivity. At 500M parameters, removing nine experts selected for biology reduces target log-probability by 1.830 nats, while the drops on retained domains and core text are only 0.028 and 0.043 nats. This demonstrates a substantial target-domain effect with limited degradation elsewhere.
  3. The ablation experiments help characterize the removal–retention trade-off. The paper varies the number of removed experts, compares two selection rules, and examines renormalization. These experiments help identify when stronger removal begins to damage retained performance and how renormalization affects that balance.
Critical weaknesses
  1. The claimed finer removal is not directly evaluated. The paper presents finer removal granularity as an advantage of DEX, but the ablation results are reported for four broad domains. Table 2 shows the semantic content of documents near each centroid; it does not measure what the corresponding trained experts contribute. The current evidence therefore leaves open whether DEX can remove one subdomain while preserving a closely related one. For example, an ablation targeting infectious-disease content could be evaluated on held-out infectious-disease and clinical/surgical text. Such a test would connect the discovered clusters to the claimed control over specific capabilities. Increasing the number of removed experts within a broad domain does not answer this question.
  2. The GRAM comparison does not isolate the effect of the method. DEX is trained on The Pile, whereas GRAM uses FineWeb-Edu and labeled domain corpora (Sections 4.1–4.2). Their domain coverage and performance before ablation may therefore differ, making changes from their respective starting points difficult to compare directly. The parameter count removed by each DEX expert and GRAM module is also unspecified. Absolute scores before and after ablation, removal sizes, and comparisons at similar retain/core degradation are needed to judge the reported advantage. Training on a shared data mixture would further strengthen this comparison. The code evaluation raises an additional question: the paper states that GRAM follows its original data recipe, which uses Lisp as the auxiliary code domain (Section 5), but evaluates removal on Python. Unless the reproduction changed that domain, the evaluation would not directly test the capability assigned to the removed module.
  3. The full computational cost is not reported. Section 4.2 describes the comparison as compute-matched based on equal model sizes and training token counts. These quantities do not show whether DEX’s additional document encoding, clustering, and routing are included in the budget. The paper does not state whether the frozen MiniLM encoder is included in the 50M/500M parameter counts or report the preprocessing cost. It also provides no measured inference latency or throughput comparison with GRAM. Please clarify the total and active parameter counts, separate preprocessing from model-training cost, and report inference performance under matched conditions. These details are needed to check the compute-matching claim and assess the practical overhead of the proposed routing scheme.
Questions for authors
  1. Which tokens are used to compute the routing gate? Please specify the encoder input during training, likelihood evaluation, and generation, and identify which tokens are scored. Since the gate is shared across a sequence, including the tokens being predicted could pass future information to the model through the gate. Is the gate computed only from text available at prediction time?
  2. What code domain was used in the GRAM reproduction? Was the auxiliary code domain changed from Lisp to Python? If so, how was Python handled in the core corpus? Please confirm that both methods are evaluated on the same held-out Python set and that the ablated modules target the capability being evaluated.
  3. How were expert-selection and evaluation data separated? Please report the size and sampling procedure of each set, along with any overlap between them or with training documents. Do all models use the same core evaluation set? Which model size and evaluation split are used in Table 1?
Reviewer_qRibOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Data-Emergent Experts (DEX), a way to remove specific capabilities from a model after training. The authors use a frozen sentence encoder and threshold-based clustering to group the data into semantic clusters, each paired with its own feed-forward expert. Every expert contributes to the output, weighted by a similarity-based gate that's shared across all token positions and layers. To remove a capability, they pick out the relevant experts using a few representative probes and zero them. They run experiments at 50M and 500M parameters, compare against GRAM, and look at how removal budget, selection rule, and gate renormalization each affect the results. Language-modeling loss does shift in domain-dependent ways, but that alone doesn't show the method is selectively removing capabilities. That claim needs stronger evidence.

Critical strengths

The paper tackles a real problem: controlling what a model can do without predefined domain labels and without retraining for different deployment setup. Both the architecture and the ablation mechanism are simple, and the way modules get picked for removal is easy to interpret. And the experiments cover the forgetting-vs-retention trade-off at two scales and vary the removal budget, selection rule, and renormalization. I especially appreciated that the authors report results that don’t go their way, such as GRAM winning on single-unit removal in three domains at 500M. That kind of honesty made it much easier for me to judge where the method actually works.

Critical weaknesses
  1. I don't think that the evaluation has isolated the effect of the method itself. DEX is trained on The Pile, while GRAM uses FineWeb-Edu plus a mix of auxiliary domains. So even with matched parameter counts and token budgets, differences in training data, starting domain competence, and how hard each domain is to remove are all tangled together. Removing one module from each model also doesn’t mean you’ve removed the same amount of capacity.

  2. Section 3.2 computes a document embedding once and shares the resulting gate across all token positions, but this paper never said which span gets embedded or how the loss is masked.

  3. I’m not convinced the results show capability removal rather than the target domain just getting worse. Zeroing out some experts doesn’t tell me the knowledge isn’t also sitting in other experts or the shared parameters. It is necessary to have a downstream capability evals and a recovery fine-tuning experiment, since those would be the most direct test. Same for the samples in Appendix C: repetitive or empty continuations don’t look like preserved, meaningful generation to me. On renormalization, my understanding is that it restores the sum of the surviving gate weights, not the original FFN output norm, so I’d want activation-level measurements and a control that removes a matched amount of gate mass. Without those, it's hard to separate expert-specific removal from general degradation.

Questions for authors
  1. Which tokens are embedded during training and likelihood evaluation, and which tokens contribute to the loss? Please specify the prefix/continuation split and loss mask. The all-MiniLM-L6-v2 model card specifies truncation beyond 256 word pieces by default; what truncation, length override, or chunking is actually used for the LM's longer inputs?

  2. Does the advantage persist with identical training data, tokenizer, and evaluation sets, and at matched retention degradation? It's better come with absolute domain losses before and after ablation, architecture dimensions, total and active parameters, and measured computation costs.

  3. Are expert-selection probes and final evaluation probes disjoint, and how large are these sets?

  4. For the assessment-changing evidence: Is there existing downstream or recovery-finetuning evidence that separates capability removal from generation degradation? Controls for removed gate mass and activation scale, and tests with unrelated prefixes or mixed-domain inputs, would directly address this ambiguity. A held-out experiment removing one subcapability while retaining related subcapabilities would test the claimed fine-grained control.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.