acceptodds
Under review as a conference paper at ICLR 2027

AI4AI in unlearning: improving unlearning algorithm autodesign with UDSL

Abstract

Machine unlearning requires removing the influence of selected data while preserving utility on retained data. Existing unlearning methods are typically implemented as monolithic trainers, where choices about what signal to optimize, how to update the model, where to apply the update, and what behavior to protect are intertwined in code. This representation makes it difficult to systematically explore alternative combinations of unlearning mechanisms. We introduce UDSL, a domain-specific language that factorizes an unlearning procedure into four composable dimensions—Signal, Update, Scope, and Protect—and compiles each program onto a shared training stack. This representation enables LLM-guided evolutionary search to explore algorithmic structure rather than implementation details. On TOFU-forget10 with Llama-3.2-1B-Instruct, UDSL-guided search discovers unlearning procedures that outperform several established unlearning methods under a small evaluation budget. Ablations show that the gains stem from exploring new compositions of unlearning mechanisms rather than merely tuning or recombining existing methods. Across forget ratios and datasets, UDSL provides a common representation in which new procedures can be re-searched without redesigning the underlying training infrastructure. These results suggest that, for LLM-guided unlearning design, the representation of the algorithmic search space can be as important as the search procedure itself.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Machine Unlearning, LLM-Guided Evolution, Domain-Specific Language, Programmatic Algorithm Search

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YX5fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces UDSL, a compositional language representing unlearning procedures through Signal, Update, Scope, and Protect. An LLM evolves programs executed through a shared interpreter. Experiments investigate utility–forgetting trade-offs across datasets, forgetting ratios, and model sizes.

Critical strengths
  • Separating algorithm representation from execution is useful, and the shared training stack reduces implementation variability.
  • The paper examines compositionality, transfer, and re-search, and transparently reports settings where transfer fails or GradDiff remains stronger.
Critical weaknesses
  • The primary metric provides limited evidence of successful unlearning. Official TOFU Forget Quality is reportedly near zero for all methods. Although UDSL improves Fitness over GradDiff, its FP, FR, and ES are all worse in Table 1; the gain reflects a different utility–forgetting trade-off.
  • Attribution to representation remains inconclusive. Published SimNPO scores 0.093, whereas its UDSL parent scores approximately 0.434–0.435, leaving only about 0.007–0.008 improvement from search. This encoding gap is unexplained. The Python/OpenEvolve comparator is also restricted to named-trainer configurations, rather than an equally expressive code-search space.
Questions for authors
  • What explains the SimNPO encoding gap, and can the representation comparison use verified equivalent implementations and equally expressive search spaces?
  • Are search selection and final evaluation performed on independent data? Please report complete-search uncertainty and official unlearning metrics.
1 comment

Thanks for the careful reading, and for noting both the value of separating representation from execution and the places where we already report transfer failure or GradDiff remaining stronger. Below we address the metric, attribution, and your two questions.

On Fitness as evidence of successful unlearning. We agree the primary metric gives limited evidence of stronger unlearning. Fitness =MU×(1−FP)= \mathrm{MU}\times(1-\mathrm{FP}) is a search/trade-off proxy. In Table 1, UDSL’s Fitness gain over GradDiff comes with worse FP / FR / ES, i.e., a different utility–forgetting trade-off rather than uniformly better forgetting. Our reruns show the same pattern: GradDiff (γ=1\gamma=1) remains stronger on FP (MU 0.4390.439 / FP 0.0650.065 / Fitness 0.4100.410) than the UDSL discovery run (MU 0.5550.555 / FP 0.2090.209 / Fitness 0.4390.439). Near-zero official Forget Quality across methods further cautions against reading any single scalar as “successful unlearning.” We will present Fitness as a trade-off objective for search, foreground MU–FP Pareto and multi-metric / official TOFU tables, and remove language that equates Fitness gains with stronger unlearning.

On attribution to representation (SimNPO 0.0930.093 vs. parent ≈0.43\approx 0.43, search +0.007+0.007). Your concern is well taken. The large gap is a hyperparameter / labeling issue, not two incompatible losses. Stock SimNPO uses β=4.5\beta=4.5, γ=0.125\gamma=0.125 and yields Fitness ≈0.094\approx 0.094; the same SimNPO loss family under DSL-parent hypers β=1\beta=1, γ=1\gamma=1 recovers ≈0.434\approx 0.434, matching the parent envelope. So the fair shallow reference is DSL-parent (or HPO-tuned) SimNPO, not stock alone. Under the original protocol the parent$\to$UDSL margin is indeed only ≈0.007\approx 0.007–0.0080.008, so most of that Fitness mass is already in the parent encoding—we do not claim every discovery run must leap far past its parent. Our intended claim is complementary: HPO tunes scalars of a fixed program, and UDSL then provides a compositional space to evolve that program further. Matched-budget N=13N=13 HPO with fixed structure left GradDiff at 0.4100.410 and raised NPO / SimNPO to 0.5650.565 / 0.5750.575. Warm-starting UDSL from the SimNPO-HPO winner improved 0.575→0.5800.575\to 0.580; from NPO-HPO the score held (≈0.568\approx 0.568). Freezing the paper depth-3 UDSL tree and sweeping only hypers (N=20N=20) reached 0.5740.574—near SimNPO-HPO but below the warm-started result. We will add these tables and phrase the contribution as HPO as initialization + DSL evolution on top.

On the Python / OpenEvolve comparator. Agreed that restricting code search to named-trainer configurations is not an equally expressive space. Hard-prompted Code-OE that tries to mimic rich compositions is easy to misread as the default strength of unrestricted code search; OE-default stays near GradDiff-level. We will narrow that baseline claim, mark hard-prompt runs as constrained ablations, and emphasize UDSL’s typed compositional interface—while not claiming that unrestricted code search could never write similar losses under the same budget and scaffolding.

Q1 — SimNPO encoding gap; equivalent implementations and expressive search spaces. As above: same loss family, different (β,γ)(\beta,\gamma). For representation comparisons we will (i) verify and report equivalent SimNPO implementations under matched hypers, (ii) use DSL-parent / HPO-tuned SimNPO as the fair shallow reference, and (iii) clarify that named-trainer code search is not equally expressive, with the corresponding claim narrowing.

Q2 — Independent selection vs. final evaluation; search uncertainty; official metrics. In the original paper, search selection and the reported Fitness are on-policy (not independent). For our reruns we kept checkpoints, froze selected candidates before further readout, and will separate search Fitness from a final full-metric / official TOFU evaluation. We will also report official unlearning metrics alongside Fitness. On complete-search uncertainty: the main discovery trajectory uses a fixed seed; we will state this limitation clearly and, where available, report trajectory / replicate variability rather than overstating a single BSF.

In short: Fitness ≠\neq stronger unlearning; the SimNPO gap is explained by (β,γ)(\beta,\gamma); the original parent$\to$UDSL margin is small; and with HPO as a fair start, UDSL can still evolve further (0.575→0.5800.575\to 0.580). We will revise baselines, evaluation independence, and official-metric reporting accordingly.

Reviewer_LUTtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces UDSL, a domain-specific language for automated machine unlearning algorithm design. UDSL decomposes unlearning procedures into four composable components: Signal, Update, Scope, and Protect. Experiments on TOFU show that UDSL search achieves a Fitness of 0.442. The paper further evaluates transferability across forgetting ratios, model scales, and datasets, along with several search-space ablations.

Critical strengths
  • The idea of explicitly representing unlearning algorithms through composable semantic components is interesting and provides a more structured search space for automated algorithm design.

  • The paper includes experiments across multiple datasets and model configurations, as well as ablations investigating the contribution of compositional search.

  • The authors acknowledge limitations in cross-setting transferability and provide examples of the discovered programs, which help illustrate the proposed approach.

Critical weaknesses
  1. The baselines are not systematically tuned under matched search budgets. The hyperparameter-only ablation (0.435) is already close to full UDSL (0.442), making it unclear whether structural search provides meaningful additional gains.

  2. Discovered algorithms transfer poorly across settings. For example, zero-shot transfer from 1B to 3B reduces Fitness from 0.442 to 0.033, raising concerns about setting-specific overfitting.

  3. The choice of 13 evaluations is not justified, and the improvement over the initial candidate pool is marginal. Search-budget scaling experiments are missing.

Questions for authors
  • Can the authors compare UDSL with a setting where the baseline algorithms (e.g., GradDiff, NPO, SimNPO) are fixed, but the LLM searches their hyperparameters under the same evaluation budget? This would help clarify whether the improvements come from the proposed compositional search space or simply from LLM-guided optimization.

  • Why were 13 evaluations chosen? Can the authors provide search curves with larger budgets?

  • How do the authors distinguish generalizable algorithm discovery from setting-specific optimization?

1 comment

Thanks for the careful reading. Below we clarify UnlearnEvo / UDSL with the paper results and new rebuttal runs (Llama-3.2-1B, TOFU forget10; Fitness = MU×(1−FP) unless noted).

On Fitness and “stronger unlearning.” We agree Fitness is not proof of uniformly stronger forgetting: it rewards utility while penalizing FP, so methods can sit in different MU–FP regimes under one scalar. In our reruns GradDiff (γ\gamma=1) is stronger on FP (MU 0.439 / FP 0.065 / Fitness 0.410), while UDSL sits higher on both utility and FP (MU 0.555 / FP 0.209 / Fitness 0.439). A pre-registered γ\gamma grid with matched-forgetting (|Δ\DeltaFP| ≤\le 0.02) yielded few cross-family pairs, confirming different forgetting frontiers. We will present Fitness as a search/trade-off proxy and foreground MU–FP Pareto and multi-metric tables.

On the SimNPO discrepancy (≈0.09 vs ≈0.43). This is hyperparameter labeling, not two losses. Stock SimNPO (β\beta=4.5, γ\gamma =0.125) gives Fitness ≈0.094; the same loss with DSL-parent hypers (β\beta=1, γ\gamma =1) recovers ≈\approx 0.434. We will split stock vs DSL-parent SimNPO.

On search Fitness also being the reported score. Fair point: original reporting is on-policy. For the rebuttal we kept checkpoints, froze selected candidates before further readout, and will separate search Fitness from final full-metric evaluation.

On whether the gains are “just HPO.” We tested this directly. Matched-budget N=13 HPO with fixed structure left GradDiff at 0.410, while NPO / SimNPO reached 0.565 / 0.575—so HPO explains much of a strong Fitness number for shallow trainers. That is not the full story: HPO and UDSL are complementary. HPO tunes scalars of a fixed program; UDSL then evolves that program in a compositional space. Warm-starting from the SimNPO-HPO winner (DeepSeek mutator, N=13) improved 0.575→0.580; from NPO-HPO the score held ($\approx$0.568). Freezing the paper depth-3 UDSL tree and sweeping only hypers (N=20) reached 0.574—near SimNPO-HPO but below the warm-started result. So strong HPO is a fair baseline, and UDSL can still improve from HPO seeds. We will add these tables and phrase the contribution as HPO as initialization + DSL evolution on top.

On N=13. We use N=13 as the paper’s reported UDSL budget so HPO and discovery comparisons are commensurate—not as a uniquely optimal size.

On 1B→3B collapse (0.033). Zero-shot program migration can fail; that is 0.033. Our claim is reusable search representation / re-search at the target scale, not a portable program. We will tighten that wording.

In short: HPO makes shallow trainers strong, and UDSL can still improve from those winners as seeds (0.575→0.580). We will revise claims around an evolvable DSL on top of fair HPO, and clean up SimNPO labeling, Fitness interpretation, evaluation independence, and transfer wording.

Reviewer_i9jJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces UDSL, a domain-specific language that decomposes machine unlearning algorithms into four composable dimensions: Signal, Update, Scope, and Protect. Instead of directly evolving trainer code, the framework allows LLMs to search over structured combinations of unlearning mechanisms. Experiments on TOFU and several other unlearning benchmarks show that UDSL can discover procedures with improved utility-forgetting fitness. Ablations further investigate the contributions of compositional search and individual semantic dimensions.

Critical strengths

I like the idea of making the algorithm representation itself the object of design. Most of nowadays AI4AI approach focus on better search strategies, but this paper trys to argue that exposing meaningful algorithmic components can be as well important. The four-dimensional decomposition in the paper provides a reasonable interface for exploring unlearning mechanisms. I also like that that the experiments go beyond simply reporting a winning algorithm. The comparisons across forgetting ratios and datasets, especially the contrast between direct migration and independent re-search, all these provide useful evidence that the representation can be reused even when individual discovered algorithms do not transfer.

Critical weaknesses

However, as for the methodology side, I’m not convinced that the discovered method actually achieves better unlearning. In Table 1, UDSL improves Fitness over GradDiff, but its forgetting-related metrics are consistently worse: FP, FR and ES all increases. So the improvement mainly comes from higher retained utility. Since the official Forget Quality metric is also reportedly near zero for all methods, I'm very very concerned that the proposed fitness only rewards a favorable numerical trade-off without establishing stronger unlearning. The evidence attributing the improvement to compositional search is actually weaker than it appears. Table 4 reports a SimNPO parent envelope of 0.435, already very close to UDSL's final score, while the published SimNPO achieves only 0.093. This substantial encoding discrepancy is not explained. Moreover, the actual improvement beyond the parent is only 0.007, so this makes me difficult to get how much new algorithmic value the evolutionary search contributes.

Questions for authors

Can you explain the large performance gap between published SimNPO and its UDSL parent? What specific implementation or semantic differences account for this discrepancy, and how does it affect the fairness of the baseline comparison? Would the discovered method still outperform GradDiff under matched forgetting strength, rather than the multiplicative Fitness objective? The code-level comparison mainly searches over named-trainer configurations. What happens if a code-level search baseline is allowed to construct the same mechanism compositions as UDSL under the same evaluation budget?

1 comment

Thanks for the thoughtful assessment, and especially for highlighting the value of treating algorithm representation itself as a design object, as well as the migration vs. re-search evidence. Below we focus on the methodological concerns and the three concrete questions.

Does higher Fitness mean better unlearning? We agree that it does not, and that Table 1 should not be read that way. Fitness =MU×(1−FP)= \mathrm{MU}\times(1-\mathrm{FP}) is a search/trade-off proxy: UDSL’s gain over GradDiff is driven mainly by higher retained utility, while FP / FR / ES are worse. Our reruns make the same point explicit—GradDiff (γ=1\gamma=1) remains stronger on forgetting (MU 0.4390.439 / FP 0.0650.065 / Fitness 0.4100.410) than the UDSL discovery run (MU 0.5550.555 / FP 0.2090.209 / Fitness 0.4390.439). We also ran a pre-registered γ\gamma grid with matched forgetting (∣ΔFP∣≤0.02|\Delta\mathrm{FP}|\le 0.02); cross-family matched pairs were sparse, which itself shows that these methods occupy different forgetting regimes rather than a uniform “stronger unlearning” ranking. Near-zero Forget Quality across methods further cautions against over-interpreting any single scalar. We will revise the manuscript to present Fitness as a trade-off objective for search, foreground MU–FP Pareto / multi-metric tables, and remove language that equates Fitness gains with stronger unlearning.

Published SimNPO (≈0.09\approx 0.09) vs. UDSL parent (≈0.43\approx 0.43). This gap is a hyperparameter / labeling issue, not two incompatible losses. Stock SimNPO uses β=4.5\beta=4.5, γ=0.125\gamma=0.125 and yields Fitness ≈0.094\approx 0.094 (MU 0.5970.597 / FP 0.8430.843). Encoding the same SimNPO loss family under the DSL-parent hypers β=1\beta=1, γ=1\gamma=1 recovers ≈0.434\approx 0.434 (MU 0.5610.561 / FP 0.2260.226)—matching the parent envelope in Table 4. Fairness implication: comparing UDSL only to stock SimNPO would understate the shallow baseline; the right reference is the DSL-parent (or HPO-tuned) SimNPO. We will rename these as stock SimNPO vs. DSL-parent SimNPO and update the baseline discussion accordingly.

How much value does evolutionary search add beyond the parent (+0.007+0.007)? Your reading of Table 4 is fair: under the original protocol the hop from parent to UDSL is small, so most of that Fitness mass is already in the parent encoding. We do not claim that every discovery run must leap far past its parent. Our intended claim is complementary: once a strong program (parent or HPO winner) is available, UDSL provides a compositional space in which that program can keep evolving under a shared interface. As a direct test against “gains are just HPO,” we ran matched-budget N=13N=13 HPO with fixed structure: GradDiff stayed at 0.4100.410, while NPO / SimNPO reached 0.5650.565 / 0.5750.575. Warm-starting UDSL from the SimNPO-HPO winner then improved further (0.575→0.5800.575\to 0.580); from the NPO-HPO winner the score held (≈0.568\approx 0.568). Freezing the paper’s depth-3 UDSL tree and sweeping only hypers (N=20N=20) reached 0.5740.574—near SimNPO-HPO but below the warm-started result. So HPO is a strong and necessary baseline, and DSL evolution can still improve from HPO-optimized seeds rather than merely rediscovering scalar tuning. We will add these tables and phrase the contribution as HPO as initialization + DSL evolution on top, while being explicit that the original parent$\to$UDSL margin was modest.

Q1 — SimNPO gap and baseline fairness. As above: same loss family, different (β,γ)(\beta,\gamma). The fair shallow reference is DSL-parent / HPO-tuned SimNPO, not the stock published configuration alone.

Q2 — Matched forgetting vs. GradDiff. Under ∣ΔFP∣≤0.02|\Delta\mathrm{FP}|\le 0.02 we do not obtain a clean win for UDSL over GradDiff on forgetting-matched Fitness; GradDiff remains preferable when low FP/ES is the priority. We will state this clearly and stop suggesting Fitness dominance as evidence of stronger forgetting.

Q3 — Code-level search with the same compositions / budget. Agreed that searching only named-trainer configs is weaker than UDSL’s compositional space. Hard-prompted Code-OE that tries to mimic rich compositions is easy to misread as the default strength of unrestricted code search; OE-default stays near GradDiff-level. We will narrow the code-baseline claim, mark hard-prompt runs as constrained ablations, and emphasize that UDSL’s contribution is a typed compositional interface—not that every free-form code search is incapable of writing similar losses if given unrestricted scaffolding and the same budget.

In short: we accept that Fitness ≠\neq stronger unlearning and that the parent$\to$UDSL margin in the paper is small; we explain the SimNPO discrepancy via hypers; and we show that with HPO as a fair starting point, UDSL can still evolve further (0.575→0.5800.575\to 0.580), which better matches what we want the representation claim to mean.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.