- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces UDSL, a compositional language representing unlearning procedures through Signal, Update, Scope, and Protect. An LLM evolves programs executed through a shared interpreter. Experiments investigate utility–forgetting trade-offs across datasets, forgetting ratios, and model sizes.
Critical strengths
- Separating algorithm representation from execution is useful, and the shared training stack reduces implementation variability.
- The paper examines compositionality, transfer, and re-search, and transparently reports settings where transfer fails or GradDiff remains stronger.
Critical weaknesses
- The primary metric provides limited evidence of successful unlearning. Official TOFU Forget Quality is reportedly near zero for all methods. Although UDSL improves Fitness over GradDiff, its FP, FR, and ES are all worse in Table 1; the gain reflects a different utility–forgetting trade-off.
- Attribution to representation remains inconclusive. Published SimNPO scores 0.093, whereas its UDSL parent scores approximately 0.434–0.435, leaving only about 0.007–0.008 improvement from search. This encoding gap is unexplained. The Python/OpenEvolve comparator is also restricted to named-trainer configurations, rather than an equally expressive code-search space.
Questions for authors
- What explains the SimNPO encoding gap, and can the representation comparison use verified equivalent implementations and equally expressive search spaces?
- Are search selection and final evaluation performed on independent data? Please report complete-search uncertainty and official unlearning metrics.
Thanks for the careful reading, and for noting both the value of separating representation from execution and the places where we already report transfer failure or GradDiff remaining stronger. Below we address the metric, attribution, and your two questions.
On Fitness as evidence of successful unlearning. We agree the primary metric gives limited evidence of stronger unlearning. Fitness =MU×(1−FP) is a search/trade-off proxy. In Table 1, UDSL’s Fitness gain over GradDiff comes with worse FP / FR / ES, i.e., a different utility–forgetting trade-off rather than uniformly better forgetting. Our reruns show the same pattern: GradDiff (γ=1) remains stronger on FP (MU 0.439 / FP 0.065 / Fitness 0.410) than the UDSL discovery run (MU 0.555 / FP 0.209 / Fitness 0.439). Near-zero official Forget Quality across methods further cautions against reading any single scalar as “successful unlearning.” We will present Fitness as a trade-off objective for search, foreground MU–FP Pareto and multi-metric / official TOFU tables, and remove language that equates Fitness gains with stronger unlearning.
On attribution to representation (SimNPO 0.093 vs. parent ≈0.43, search +0.007). Your concern is well taken. The large gap is a hyperparameter / labeling issue, not two incompatible losses. Stock SimNPO uses β=4.5, γ=0.125 and yields Fitness ≈0.094; the same SimNPO loss family under DSL-parent hypers β=1, γ=1 recovers ≈0.434, matching the parent envelope. So the fair shallow reference is DSL-parent (or HPO-tuned) SimNPO, not stock alone. Under the original protocol the parent$\to$UDSL margin is indeed only ≈0.007–0.008, so most of that Fitness mass is already in the parent encoding—we do not claim every discovery run must leap far past its parent. Our intended claim is complementary: HPO tunes scalars of a fixed program, and UDSL then provides a compositional space to evolve that program further. Matched-budget N=13 HPO with fixed structure left GradDiff at 0.410 and raised NPO / SimNPO to 0.565 / 0.575. Warm-starting UDSL from the SimNPO-HPO winner improved 0.575→0.580; from NPO-HPO the score held (≈0.568). Freezing the paper depth-3 UDSL tree and sweeping only hypers (N=20) reached 0.574—near SimNPO-HPO but below the warm-started result. We will add these tables and phrase the contribution as HPO as initialization + DSL evolution on top.
On the Python / OpenEvolve comparator. Agreed that restricting code search to named-trainer configurations is not an equally expressive space. Hard-prompted Code-OE that tries to mimic rich compositions is easy to misread as the default strength of unrestricted code search; OE-default stays near GradDiff-level. We will narrow that baseline claim, mark hard-prompt runs as constrained ablations, and emphasize UDSL’s typed compositional interface—while not claiming that unrestricted code search could never write similar losses under the same budget and scaffolding.
Q1 — SimNPO encoding gap; equivalent implementations and expressive search spaces. As above: same loss family, different (β,γ). For representation comparisons we will (i) verify and report equivalent SimNPO implementations under matched hypers, (ii) use DSL-parent / HPO-tuned SimNPO as the fair shallow reference, and (iii) clarify that named-trainer code search is not equally expressive, with the corresponding claim narrowing.
Q2 — Independent selection vs. final evaluation; search uncertainty; official metrics. In the original paper, search selection and the reported Fitness are on-policy (not independent). For our reruns we kept checkpoints, froze selected candidates before further readout, and will separate search Fitness from a final full-metric / official TOFU evaluation. We will also report official unlearning metrics alongside Fitness. On complete-search uncertainty: the main discovery trajectory uses a fixed seed; we will state this limitation clearly and, where available, report trajectory / replicate variability rather than overstating a single BSF.
In short: Fitness = stronger unlearning; the SimNPO gap is explained by (β,γ); the original parent$\to$UDSL margin is small; and with HPO as a fair start, UDSL can still evolve further (0.575→0.580). We will revise baselines, evaluation independence, and official-metric reporting accordingly.