acceptodds
Under review as a conference paper at ICLR 2027

Tchebycheff Unlearn: Exploring the Pareto Front in Machine Unlearning

Abstract

Machine unlearning aims to forget target information while keeping the model useful. Applications differ in how much they need to forget and how much utility loss they can accept, necessitating a family of models with different trade-offs instead of a single model. We propose to generate the family by exploring the Pareto front in the forget-retain loss space, treating the forget loss and the retain loss as two objectives. A common approach is linear scalarization, which minimizes a weighted sum of the two losses. However, sweeping the weights cannot reach nonconvex regions of the Pareto front, so some trade-offs remain out of reach. In light of the above limitation of linear scalarization, recent work has also explored adaptive gradient methods from the multi-task learning literature, yet they offer no control over which Pareto-optimal solution they converge to. In this work, we introduce Tchebycheff Unlearn, which uses Tchebycheff scalarization to construct a family of models with different forget–retain trade-offs. We show that varying only the forget reference suffices to represent every attainable Pareto loss pair, with fixed positive task scales and retain reference. We evaluate our approach on the WMDP, TOFU, and MUSE datasets. Our empirical results show that Tchebycheff Unlearn improves both Pareto front coverage and the persistence of intermediate solutions compared with linear scalarization, enabling more comprehensive Pareto front exploration.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, Multi-Objective Optimization, Tchebycheff Scalarization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wehMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies forget–retain trade-offs in machine unlearning and proposes Tchebycheff Unlearn to construct a family of models by exploring the Pareto front. The method uses Tchebycheff scalarization with a forget-reference sweep, supported by theoretical analysis. Experiments on WMDP, TOFU, and MUSE examine empirical Pareto front coverage and the persistence of intermediate solutions compared with linear scalarization.

Critical strengths

1.The research question is meaningful and clearly motivated. Forgetting and retention can conflict, and the preferred balance varies across applications. The focus on providing a family of models with different trade-offs is well motivated.

2.The theoretical analysis is concise and supports the method’s design. The single-coordinate sweep proposition justifies varying only the forget reference and clearly distinguishes finding Pareto-optimal solutions from achieving Pareto front coverage.

3.The experiments are reasonably comprehensive and well documented. They cover multiple datasets, examine both endpoint loss pairs and training trajectories, and provide results across multiple random seeds, detailed configurations, and an explicit discussion of limitations.

Critical weaknesses

1.Behavioral forgetting and retained utility are not directly evaluated. The experiments mainly report surrogate losses, leaving it unclear whether the additional intermediate solutions correspond to better knowledge removal, reduced privacy leakage, or better preservation of model capabilities. Although the authors acknowledge this limitation, a gap remains between the application motivation and the empirical evidence.

2.The added benefit over existing methods is not sufficiently established. The core optimization procedure follows OMD–TCH, but there is no direct comparison with Tchebycheff weight sweeps at a fixed reference point. This makes it difficult to assess the independent value of the reference sweep.

3.The proposed mechanism for sustaining intermediate solutions lacks direct verification. The paper does not report trajectories of the relative weights. It therefore remains possible that the intermediate solutions persist mainly because the forget weight decreases and training shifts toward retention, rather than because of continued feedback between the two objectives.

Questions for authors

1.Could the authors report behavioral forgetting and retained utility metrics to establish whether the additional intermediate loss compromises correspond to useful behavioral trade-offs?

2.Could the authors show trajectories of the relative weights and compare against a two-stage procedure that performs forgetting followed by retention training, to clarify the mechanism behind sustaining intermediate solutions?

2 comments
AuthorsOpenReview ↗

Response to Reviewer wehM

We sincerely thank the reviewer for the careful reading and constructive comments. We respond to the comments below.

W1 & Q1: Behavioral evaluation of forgetting and retained utility

We thank the reviewer for the helpful comment. Tchebycheff Unlearn guarantees better coverage of the forget–retain loss trade-off, but this does not by itself imply better behavioral trade-offs (e.g., WMDP vs. MMLU accuracy, TOFU Forget Quality vs. Model Utility), since surrogate losses and behavioral metrics need not be monotonically related. We acknowledge that this is an important open question, and we will discuss it and clarify the scope of our claims in the next version of this work.

W2: Methodological novelty

We thank the reviewer for the insightful comment. Our main contribution is to identify two limitations of linear scalarization in machine unlearning, and introduce Tchebycheff scalarization as the alternative method to address the limitations:

  • Unstable optimization trajectories: Under weighted-sum objectives, unlearning runs tend to stay near a high-forget-loss regime and then drop abruptly to a low-forget-loss regime. Near the transition weight, runs with the same setting but different seeds end up in separated forget-loss regions (e.g., λ=0.08\lambda=0.08 in Figure 2).
  • Failure to cover non-convex trade-offs: The empirical WMDP front contains unsupported Pareto solutions that no weighted sum can reach, even with a refined weight grid (Figure 3(a), Section 5.1).

Tchebycheff scalarization addresses both limitations: its level sets can reach unsupported solutions, and its adaptive weights help sustain intermediate solutions. Proposition 1 varies only the forget reference and holds for any fixed retain reference, and we view Proposition 1 as a justification for a practical design choice. We will revise the introduction to make this positioning clear in the next version of this work.

W3 & Q2: Mechanism for sustaining intermediate solutions

We thank the reviewer for the helpful comment. We will report the relative-weight trajectories in the next version of this work.

The authors clarified some point yet my other concerns such as W1 remained. I thus keep my rating.

Reviewer_zVvNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to construct a family of unlearned models covering different trade-offs between forgetting and retention, particularly those that linear scalarization may miss on a nonconvex Pareto front. It proposes Tchebycheff Unlearn, which combines Tchebycheff scalarization with an existing online mirror descent solver and explores trade-offs by sweeping the forget reference while fixing the task scales and retain reference. The paper proves that this single-coordinate sweep can represent every attainable Pareto-optimal loss pair under exact optimization. Experiments on WMDP demonstrate broader coverage and sustained intermediate loss compromises, while TOFU shows improved surrogate-loss trade-offs and MUSE yields largely comparable results.

Critical strengths
  1. This paper is well-moitvated by illustrating the limitations of existing Pareto front exploration solutions.

  2. Generally, the proposed Tchebycheff unlearn is technically sound by employing existing tools in multi-task learning.

  3. The WMDP experiments provide informative evidence that reference sweeps recover intermediate surrogate-loss compromises missed by the tested weighted-sum sweeps. Multiple seeds and training trajectories strengthen this observation, while experiments on TOFU and MUSE examine its applicability across different datasets and objectives.

Critical weaknesses
  1. Both Tchebycheff scalarization and the OMD–TCH solver are established methods [1,2], including closely related applications in multi-task learning. The main additions are their application to unlearning and a single-coordinate reference-sweep proposition, which follows relatively directly from the geometry of the two-objective formulation. Thus, the novelty of this paper is rather limited.

  2. The evaluation focuses on RMU/NPO and retention losses rather than behavioral forgetting, privacy leakage, and retained capabilities. Consequently, improved surrogate-loss fronts do not establish a competitive forgetting–utility trade-off on the benchmarks’ substantive evaluation metrics.

  3. The experiments do not directly isolate the advantage of reference sweeping over smooth Tchebycheff scalarization or preference-weight sweeping under a fixed reference. Moreover, the additional WMDP baselines are evaluated on one seed with fixed configurations, including fixed constraint thresholds, which limits conclusions about their relative front coverage.

  4. Section 5.4 states that every tested weighted-sum solution is dominated by a reference-sweep solution, whereas the conclusion describes the TOFU and MUSE outcomes together as broadly similar with potentially slight improvements. Reconciling these descriptions would make the magnitude and scope of the reported advantages clearer.

  5. I suggest the authors rewrite the related work section, especially the first subsection; it mixes the unlearning and MOO methods without clarification, which would confuse the readers.

Reference:

[1] Smooth Tchebycheff Scalarization,ICML 2024

[2] Adaptive Online Mirror Descent for Tchebycheff Scalarization,TMLR 2026

Questions for authors

Please refer to the listed Weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer zVvN

We sincerely thank the reviewer for the careful reading and constructive comments. We respond to the comments below.

W1: Methodological novelty

We thank the reviewer for the insightful comment. Our main contribution is to identify two limitations of linear scalarization in machine unlearning, and introduce Tchebycheff scalarization as the alternative method to address the limitations:

  • Unstable optimization trajectories: Under weighted-sum objectives, unlearning runs tend to stay near a high-forget-loss regime and then drop abruptly to a low-forget-loss regime. Near the transition weight, runs with the same setting but different seeds end up in separated forget-loss regions (e.g., λ=0.08\lambda=0.08 in Figure 2).
  • Failure to cover non-convex trade-offs: The empirical WMDP front contains unsupported Pareto solutions that no weighted sum can reach, even with a refined weight grid (Figure 3(a), Section 5.1).

Tchebycheff scalarization addresses both limitations: its level sets can reach unsupported solutions, and its adaptive weights help sustain intermediate solutions. Proposition 1 varies only the forget reference and holds for any fixed retain reference, and we view Proposition 1 as a justification for a practical design choice. We will revise the introduction to make this positioning clear in the next version of this work.

W2: Behavioral evaluation of forgetting and retained utility

We thank the reviewer for the helpful comment. Tchebycheff Unlearn guarantees better coverage of the forget–retain loss trade-off, but this does not by itself imply better behavioral trade-offs (e.g., WMDP vs. MMLU accuracy, TOFU Forget Quality vs. Model Utility), since surrogate losses and behavioral metrics need not be monotonically related. We acknowledge that this is an important open question, and we will discuss it and clarify the scope of our claims in the next version of this work.

W3: Comparison with smooth Tchebycheff scalarization, preference-weight sweeping, and WMDP baselines

We thank the reviewer for the helpful comment. We will add a discussion of smooth Tchebycheff scalarization, preference-weight sweeping, and WMDP baselines with more seeds in the next version of this work.

W4: Comparison between Tchebycheff Unlearn and linear scalarization on TOFU and MUSE

We thank the reviewer for pointing this out. Section 5.4 refers to that Tchebycheff Unlearn improves over linear scalarization on TOFU, while the MUSE results are largely comparable. We will state it clearly in the conclusion of the next version of this work.

W5: Related work

We thank the reviewer for the suggestion. In the next version of this work, we will reorganize it to separate unlearning objectives, unlearning-specific multi-objective methods, and general multi-task optimization methods.

Reviewer_o3dROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the problem of machine unlearning by reducing the forgetting while preserving the retained-data performance. It proposes Tchebycheff unlearning framework, using Tchebycheff scalarization to construct a family of models with different forgetting--retention tradeoffs. The central proposition establishes that, with fixed positive task scales and a fixed retained reference, varying the forget reference suffices to represent every attainable Pareto loss pair. The experiments involve WMDP, TOFU, and MUSE, with clear theoretical justification and useful empirical observations, but its practical significance remains uncertain.

Critical strengths

S1. Clear problem formulation: the paper distinguishes finding one Pareto solution from covering a range of compromises, making the motivation for a model family convincing.

S2. Clean theoretical result: Proposition 1 shows a simple yet correct construction showing that sweeping single reference coordinate suffices to represent every attainable Pareto loss pair under global optimization.

S3. Reasonable experimental breadth and transparent limitations: the study covers three benchmarks (WMDP, MUSE, TOFU), two objective families, and five seeds per sweep setting; the paper explicitly acknowledges that surrogate-loss improvements do not establish behavioral benefits.

Critical weaknesses

W1. Limited methodological novelty: Prior work [a, b] has established Tchebycheff-based Pareto exploration, and the optimizer is drawn from [a]. The single-coordinate proposition seems a straightforward geometric refinement.

W2. Unverified practice unlearning effectiveness: RMU and NPO loss improvements do not confirm knowledge removal or preserved generation quality. Behavioral evaluation, such as WMDP accuracy compared with MMLU accuracy, is required to show practical benefits.

W3. Limited quantification of coverage gains: The paper illustrates further intermediate trade-offs via loss-space plots, yet it does not provide a numerical summary of the scope or consistency of coverage gains across seeds.

References:

[a] Liu et al., Adaptive Online Mirror Descent for Tchebycheff Scalarization in Multi-Objective Learning, 2026.

[b] Lin et al., Smooth Tchebycheff Scalarization for Multi-Objective Optimization, 2024.

Questions for authors
  1. Proposition 1 yields Pareto representation by adjusting only the forget reference while holding task scales and retain reference constant. How does this outcome diverge from established Tchebycheff reference-point results, and does the single-coordinate constraint represent the primary theoretical advance?

  2. Have the returned models been assessed with behavioral metrics like WMDP accuracy compared to MMLU accuracy or TOFU Forget Quality versus Model Utility? If so, do the extra surrogate-loss compromises align with further useful behavioral trade-offs?

  3. On TOFU, reference-sweep models are said to enhance even the weighted-sum baselines’ own goals. What evidence separates a benefit from scalarization from inadequate optimization of those baselines?

2 comments
AuthorsOpenReview ↗

Response to Reviewer o3dR

We sincerely thank the reviewer for the careful reading and constructive comments. We respond to the comments below.

W1 & Q1: Methodological novelty and the role of Proposition 1

We thank the reviewer for the insightful comment. Our main contribution is to identify two limitations of linear scalarization in machine unlearning, and introduce Tchebycheff scalarization as the alternative method to address the limitations:

  • Unstable optimization trajectories: Under weighted-sum objectives, unlearning runs tend to stay near a high-forget-loss regime and then drop abruptly to a low-forget-loss regime. Near the transition weight, runs with the same setting but different seeds end up in separated forget-loss regions (e.g., λ=0.08\lambda=0.08 in Figure 2).
  • Failure to cover non-convex trade-offs: The empirical WMDP front contains unsupported Pareto solutions that no weighted sum can reach, even with a refined weight grid (Figure 3(a), Section 5.1).

Tchebycheff scalarization addresses both limitations: its level sets can reach unsupported solutions, and its adaptive weights help sustain intermediate solutions. Proposition 1 varies only the forget reference and holds for any fixed retain reference, and we view Proposition 1 as a justification for a practical design choice. We will revise the introduction to make this positioning clear in the next version of this work.

W2 & Q2: Behavioral evaluation of forgetting and retained utility

We thank the reviewer for the helpful comment. Tchebycheff Unlearn guarantees better coverage of the forget–retain loss trade-off, but this does not by itself imply better behavioral trade-offs (e.g., WMDP vs. MMLU accuracy, TOFU Forget Quality vs. Model Utility), since surrogate losses and behavioral metrics need not be monotonically related. We acknowledge that this is an important open question, and we will discuss it and clarify the scope of our claims in the next version of this work.

W3: Quantifying coverage gains

We thank the reviewer for the suggestion. We will add per-seed numerical summaries of coverage across the five seeds in the next version of this work.

Q3: Optimization of the baselines on TOFU

In the TOFU experiments, both linear scalarization and Tchebycheff Unlearn share the same initial model, losses, optimizer, learning rate, training budget, and seeds (Table 1), so the difference comes from the scalarization. We will make this point clear in the next version of this work.

I thank the authors for the clarifications. The response, however, largely acknowledges the primary concerns without explicitly resolving them, and the proposed revisions primarily clarify the positioning of the work. Thus, I will maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.