acceptodds
Under review as a conference paper at ICLR 2027

Gap-Adaptive Per-Objective Regularization against Differential Overoptimization in Multi-Objective Alignment

Abstract

Multi-objective alignment scalarizes heterogeneous rewards into one objective optimized under a shared KL budget; as optimization strengthens, the policy overfits its reward proxies, and the resulting overoptimization is not distributed evenly across objectives. Existing methods either apply one global intervention (a single KL budget or cap) or only steer the direction of optimization, without adapting each objective’s budget to a measured overoptimization gap. We formalize the answer in a minimal model of proxy and gold and name the pattern Differential Overoptimization: damage concentrates on the objective with the largest effective hackability, and each objective’s collapse point follows a closed-form allocation law. The train–held-out gap makes this damage observable during training—a bridge we derive in an extended model, up to one explicitly testable condition. Building on this, we propose Gap-Adaptive Per-Objective Regularization (GAPOR), one line per objective, with provable shrinkage, rebalancing, and bounded-gain closed-loop convergence in its operative regime. Exact optimizer simulations confirm the law, and experiments across multiple LLM families with independent gold evaluation show GAPOR removes the hackable objective’s overoptimization gap while protecting true gold quality, without trading away the other objective. We also analyze the shared-channel geometry our benchmark instantiates and derive a falsifiable collapse criterion whose predicted hump shape the data confirm.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multi-Objective Alignment, Overoptimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hnf1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies differential overoptimization in multi-objective alignment and proposes GAPOR, which adapts per-objective regularization based on training versus held-out score gaps. The authors analyze damage allocation and feedback convergence under simplified models. They further evaluate the proposed method across multiple LLMs and tasks, reporting improvements in independently assessed output quality.

Critical strengths

The paper studies a relevant problem in multi-objective alignment, where overoptimization can cause different degrees of quality degradation across objectives. The proposed GAPOR method uses a simple feedback rule to adjust per-objective regularization. Experiments across multiple models and tasks, together with independent quality evaluations and human assessments, provide empirical support for its effectiveness.

Critical weaknesses
  1. Several inconsistencies make the paper hard to follow. The method relies on temperature scaling to suppress harmful optimization, but Proposition A.22(iii) shows that this need not work on shared channels. The paper also fits the full gold hump using a model that excludes its rising limb. In Table 13, the reference-policy margin is 0.372, although the stated scoring formula gives 0.5.

  2. The paper needs to better justify using the gap between training and held-out scores to set each objective’s regularization strength. It remains unclear whether larger gaps mean greater losses in true quality, or whether gap sizes can be compared across objectives.

Questions for authors
  1. The abstract and contributions both claim “closed-loop convergence.” Does this mean that GAPOR itself converges, and is there a convergence rate? The analysis seems to cover only idealized feedback dynamics. How do these results apply to the actual training algorithm?

  2. Can gap sizes be compared across objectives? Could a smaller gap in helpfulness reflect more damage than a larger gap in harmlessness? If so, what justifies giving stronger regularization to the objective with the larger gap?

Reviewer_77SkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper looks at multi-objective alignment, where several objectives (e.g., helpfulness and conciseness) are trained together under one KL budget. Using a simplified model, it argues that overoptimization damage is not shared equally: it concentrates on the objective that is easiest to game. Based on this, the authors propose GAPOR. During training, GAPOR measures, for each objective, the difference between the model's score on training pairs and on held-out pairs. It then regularizes the objectives with a larger difference more strongly. Experiments on TL;DR, HH-RLHF and BeaverTails with 1.7B-7B models show that GAPOR reduces this difference for helpfulness and improves held-out and judge-based helpfulness over several baselines, mainly in a strong-overoptimization setting (β=0.03\beta = 0.03).

Critical strengths
  • The method is simple and cheap: a one-line change to multi-objective DPO.
  • In the reported setting, gains are consistent across models and tasks, including gold and length-controlled evaluation.
  • Reporting is careful (10 seeds, per-seed values), and the paper is candid about its limitations.
Critical weaknesses
  1. The paper is hard to follow. Throughout the main text, new equations, conditions, and procedures are introduced one after another: the model in Sec. 2.1, Conditions, Protocols P1–P8, and two budget–temperature regimes. The background and intuition needed to understand them are mostly missing. Each new object is stated formally, and why it is needed or what it means is either left out or deferred to the 27-page appendix. As a result, the main text reads as a long list of results rather than an explanation, and it is difficult to understand from the main paper what each result means and why it holds. The same pattern runs through the main text; a few examples follow (not an exhaustive list).
  • Eq. 2-3. The specific reward forms are a cap on the gold direction, a log for the proxy, and a quadratic penalty for gold. They are given with brief citations, but the paper does not explain why these shapes were chosen, what each term means for a real model, or which conclusions depend on them. Remark A.5 shows the exponent of the main law changes if the log is replaced, so this matters.
  • Eq. 5. The "effective hackability" cic_i is simply defined as a normalized wi⋅biw_i \cdot b_i. The paper does not explain why this is the natural quantity to look at before using it.
  • Theorem 2.7. A "memorization direction" mim_i is added to the model in one sentence, and the gap is then split into memorization and miscalibration terms (Eq. 11). The paper does not explain why memorization should be modeled this way or what the held-out response hih_i means intuitively. Condition 2.6, on which the whole bridge rests, is likewise stated formally with no explanation of what it means in practice.
  • Eq. 14. The update rule βi=β(1+ηg^i/s\beta_i = \beta(1 + \eta\hat{g}_i/s) is presented without explaining why it takes this linear form, or how the scale s = 0.3 was chosen.
  1. The theory does not match the experiments. The main guarantees (shrinkage, global convergence) are proven for the reward-model β−1\beta^{-1} regime, while the main experiments use DPO, i.e. the β−2\beta^{-2} regime. There, by the paper's own analysis (Prop. A.18), the loop is stable only below an unknown gain threshold. The RL experiment uses a different actuator (reward scaling) and is small.

  2. The allocation law is verified only within its own model. The quantitative checks (Fig. 2c-d, Table 5) simulate the same toy model. On real models, the main task pairs helpfulness with an exact length rule, so any account predicts all damage falls on helpfulness. The key prediction, that damage ratios among several hackable objectives scale with γ⋅w⋅b\gamma\cdot w\cdot b is not tested.

  3. Non-standard DPO and a favorable β\beta. All DPO-based methods score completions without the prompt and with length normalized log-probs (Remark D.1), and no standard-DPO comparison is given. The headline results use β=0.03\beta = 0.03. At the default β=0.1\beta = 0.1 (Table 18), GAPOR's HV (0.595) is below GAPO and Constrained RLHF (0.625).

Minor

  • Line 1282: the hardware is listed as "A100 (24 GB)", which does not exist (A100s come with 40 or 80 GB).
  • Lines 408–409: two lines of text overlap.
  • Lines 103–104: gi(μ)g_i(\mu) in Eq. 2 and pi(μ)p_i(\mu) in Eq. 3 are not explicitly defined (e.g., as the gold and proxy rewards) where they are introduced.
  • Line 192: "Protocol P2 decides between the two." P2 is not described in the main text and no pointer to the appendix is given, so it is hard to tell what it refers to.
  • Line 201: TL;DR is mentioned here for the first time without a reference; it is only introduced and cited in Sec. 4.1.
  • Table 1 bolds the whole GAPOR row, but Length-regularized DPO has higher held-out conciseness (0.881 vs. 0.868).
Questions for authors
  1. How do DPO-LW and GAPOR compare under standard, prompt-conditioned DPO?
  2. At β=0.1\beta = 0.1, why prefer GAPOR over GAPO or Constrained RLHF?
  3. Can the allocation law be tested with two hackable objectives, e.g. on HH-RLHF or using the whelpw_{\text{help}} sweep in Fig. 5?
2 comments

Dear Reviewer 77Sk,

As the discussion period is nearing its end, we would like to send a gentle reminder regarding our response and the revised manuscript. If your remaining question has been resolved, we would greatly appreciate if you could consider raising your score. Please let us know if anything remains unclear.

Thank you again for your time and thoughtful feedback.

Thanks for the response and the additional experiments. They addressed some of my concerns, though I still have reservations about the theory and how well it is supported by the experiments. I’m keeping my original rating.

Reviewer_CSsvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper models the scalarized template as a budget-allocation problem, and derives a closed-form allocation law. It also derives the bridge up to one explicitly testable condition. The law identifies the per-objective budget as the place to act. GAPOR is proposed by scaling the DPO temperature by the observed gap. The paper gives theoretical results. Experiments are conducted to evaluate the proposed method GAPOR.

Critical strengths
  1. The comparison is broad, and it covers several baselines and ablations.
  2. The paper provides theoretical results for the proposed framework.
Critical weaknesses
  1. The key performance guarantees do not cover the main experiment. Proposition 3.1 analyzes per-objective dynamics derived from the separable model, whereas the main experiment uses joint DPO on a shared channel. Although the paper analyzes convergence on shared channels in Proposition A.12, it does not prove that GAPOR reduces gold damage in this setting. The support for GAPOR here is therefore mainly empirical. Can the performance theory be extended to joint shared-channel DPO?

  2. On page 30 of the paper. it writes "A reference implementation is in length_controlled_eval.py." but no "length controlled_eval.py" file is found. This kind of writing is not professional.

Questions for authors
  1. Can the performance theory be extended to joint shared-channel DPO?
  2. The DPO runs drop the source post and length-normalize the log-probabilities (Remark D.1). Can you report DPO-LW vs GAPOR under standard prompt-conditioned DPO logits, with gold evaluation?
  3. Table 13 reports a held-out margin of 0.372±.0120.372 \pm .012 for the reference policy. With π=πref\pi=\pi_{ref} all log-ratios are zero, so the mean sigmoid score should be exactly 0.5 with no seed variance. How is the margin defined and computed for the reference model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.