- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies differential overoptimization in multi-objective alignment and proposes GAPOR, which adapts per-objective regularization based on training versus held-out score gaps. The authors analyze damage allocation and feedback convergence under simplified models. They further evaluate the proposed method across multiple LLMs and tasks, reporting improvements in independently assessed output quality.
Critical strengths
The paper studies a relevant problem in multi-objective alignment, where overoptimization can cause different degrees of quality degradation across objectives. The proposed GAPOR method uses a simple feedback rule to adjust per-objective regularization. Experiments across multiple models and tasks, together with independent quality evaluations and human assessments, provide empirical support for its effectiveness.
Critical weaknesses
-
Several inconsistencies make the paper hard to follow. The method relies on temperature scaling to suppress harmful optimization, but Proposition A.22(iii) shows that this need not work on shared channels. The paper also fits the full gold hump using a model that excludes its rising limb. In Table 13, the reference-policy margin is 0.372, although the stated scoring formula gives 0.5.
-
The paper needs to better justify using the gap between training and held-out scores to set each objective’s regularization strength. It remains unclear whether larger gaps mean greater losses in true quality, or whether gap sizes can be compared across objectives.
Questions for authors
-
The abstract and contributions both claim “closed-loop convergence.” Does this mean that GAPOR itself converges, and is there a convergence rate? The analysis seems to cover only idealized feedback dynamics. How do these results apply to the actual training algorithm?
-
Can gap sizes be compared across objectives? Could a smaller gap in helpfulness reflect more damage than a larger gap in harmlessness? If so, what justifies giving stronger regularization to the objective with the larger gap?
Dear Reviewer 77Sk,
As the discussion period is nearing its end, we would like to send a gentle reminder regarding our response and the revised manuscript. If your remaining question has been resolved, we would greatly appreciate if you could consider raising your score. Please let us know if anything remains unclear.
Thank you again for your time and thoughtful feedback.