- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of how to select a small portfolio of policies for multi-objective LLM alignment under linear scalarization. Authors introduce PALM to construct a compact portfolio of policies while ensuring the approximation quality across different preferences. It combines preference-space discretization, a lazy search procedure that uses existing policies to assess coverage, and pruning to remove redundant policies.
Theoretical results on approximation quality and portfolio size are provided, with the extension to sparse preferences that makes the size bound polynomial in for a fixed support size.
Experiments show it generally achieves smaller approximation gaps than uniform and random portfolios of the same final size.
Critical strengths
The main strengths are as follows:
-
This paper has clear motivation and aim to addresses an important practical issue in multi-objective alignment. Training a separate policy for every preference can be expensive, while selecting a small set of policies without considering their reward profiles can lead to substantial redundancy. Constructing a compact portfolio that maintains good coverage of the preference space is therefore a practical and useful objective.
-
The theoretical analysis of the approximation quality and portfolio size complements the empirical comparisons.
-
The hybrid grid is a sensible fix for the boundary problem of uniform grids, and the certification idea behind the lazy search is simple and reusable.
Critical weaknesses
However, there are several obvious issues that authors should address:
-
The empirical implementation does not clearly preserve the assumptions used in the theoretical results. There are several places where the experiments depart from what Theorem 3.3 and Lemma 3.6 require. In particular, the theoretical analysis relies on a bound involving , but its empirical estimation appears problematic for the standardized HelpSteer rewards, which can be bounded. Estimating this quantity from a finite sample does not necessarily provide a valid upper bound. It also raises questions about whether the theoretical approximation guarantees hold in the actual experimental setting.
-
The computational advantage over simple baselines appears to be limited when considering the cost of constructing the portfolio.
-
The experiments do not fully establish the claimed coverage guarantees over the continuous preference space. The evaluation uses a finite set of test preferences, they cannot establish that the selected portfolio maintains good approximation quality across the full preference space, which is the central claim of the paper.
Questions for authors
-
Can you compare PALM and the direct Uniform and Random baselines at matched total oracle-call budgets, rather than only at matched final portfolio sizes?
-
How is being estimated in the experiments?
-
For HelpSteer, how is the multiplicative gap being computed at test weights when the oracle is negative or close to zero?
6 comments
Weakness 3
The experiments do not fully establish the claimed coverage guarantees over the continuous preference space. The evaluation uses a finite set of test preferences, they cannot establish that the selected portfolio maintains good approximation quality across the full preference space, which is the central claim of the paper.
Thank you for the comment. Computing the exact worst-case gap requires optimizing a non-convex function over a continuous, multi-dimensional preference space. Each evaluation of also requires a separate fine-tuning run. To the best of our knowledge, no tractable method evaluates this quantity exactly. Evaluating on a finite set of weights is the well-established approach (Zintgraf et al., 2015; Hayes et al., 2022; Alegre et al., 2023). Furthermore, to account for sampling variability, we report bootstrap means and standard deviations over 5,000 resamples of , providing a more robust assessment of empirical performance.
The sampled maximum also has a finite-sample guarantee. In Safety Alignment, Helpful Assistants and RLVR-GSM, is i.i.d. uniform on . By Tempo et al. (1997), with probability at least ,
Here denotes the additive or the multiplicative gap of portfolio at weight , so is the reported worst-case gap.
At , this gives for , for , and for . Thus, with 95% confidence, the gap exceeds the sampled maximum on at most an fraction of the preference space under the uniform measure.
References
- Alegre et al. (2023). Sample-efficient multi-objective learning via generalized policy improvement prioritization. AAMAS (Conference).
- Hayes et al. (2022). A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems (Journal).
- Zintgraf et al. (2015). Quality assessment of MORL algorithms: A utility-based approach. Benelearn.
- Tempo et al. (1997). Probabilistic robustness analysis: Explicit bounds for the minimum number of samples. Systems & Control Letters.
Question 1
Can you compare PALM and the direct Uniform and Random baselines at matched total oracle-call budgets, rather than only at matched final portfolio sizes?
We conducted additional experiments on Safety Alignment and Helpful Assistants, matching the number of oracle calls between PALM and the Uniform and Random baselines. We also include unpruned PALM, allowing a comparison at both equal oracle-call budgets and equal final portfolio sizes.
Safety Alignment
| Method | Oracle calls | Final size | Mult. gap | Add. gap |
|---|---|---|---|---|
| PALM (pruned) portfolio size = 1 | ||||
| PALM (pruned) | 7 | 1 | 0.4448 ± 0.0134 | 0.4386 ± 0.0215 |
| PALM (unpruned) | 7 | 7 | 0.0154 ± 0.0032 | 0.0107 ± 0.0025 |
| Uniform | 7 | 7 | 0.0101 ± 0.0045 | 0.0070 ± 0.0034 |
| Random | 7 | 7 | 0.0871 ± 0.0115 | 0.0866 ± 0.0125 |
| PALM (pruned) portfolio size = 2 | ||||
| PALM (pruned) | 7 | 2 | 0.0538 ± 0.0109 | 0.0533 ± 0.0120 |
| PALM (unpruned) | 7 | 7 | 0.0004 ± 0.0003 | 0.0004 ± 0.0003 |
| Uniform | 7 | 7 | 0.0101 ± 0.0045 | 0.0070 ± 0.0034 |
| Random | 7 | 7 | 0.0871 ± 0.0115 | 0.0866 ± 0.0125 |
| PALM (pruned) portfolio size = 3 | ||||
| PALM (pruned) | 5 | 3 | 0.0158 ± 0.0021 | 0.0110 ± 0.0018 |
| PALM (unpruned) | 5 | 5 | 0.0158 ± 0.0021 | 0.0110 ± 0.0019 |
| Uniform | 5 | 5 | 0.0324 ± 0.0119 | 0.0187 ± 0.0057 |
| Random | 5 | 5 | 0.1039 ± 0.0100 | 0.1031 ± 0.0115 |
| PALM (pruned) portfolio size = 4 | ||||
| PALM (pruned) | 8 | 4 | 0.0004 ± 0.0003 | 0.0004 ± 0.0003 |
| PALM (unpruned) | 8 | 8 | 0.0004 ± 0.0003 | 0.0004 ± 0.0003 |
| Uniform | 8 | 8 | 0.0109 ± 0.0080 | 0.0078 ± 0.0057 |
| Random | 8 | 8 | 0.1851 ± 0.0113 | 0.1824 ± 0.0149 |
Helpful Assistants
| Method | Oracle calls | Final size | Mult. gap | Add. gap |
|---|---|---|---|---|
| PALM (pruned) portfolio size = 3 | ||||
| PALM (pruned) | 9 | 3 | 0.0353 ± 0.0073 | 0.0236 ± 0.0059 |
| PALM (unpruned) | 9 | 9 | 0.0057 ± 0.0028 | 0.0038 ± 0.0020 |
| Uniform | 9 | 9 | 0.0355 ± 0.0132 | 0.0242 ± 0.0093 |
| Random | 9 | 9 | 0.2642 ± 0.0186 | 0.2453 ± 0.0254 |
| PALM (pruned) portfolio size = 4 | ||||
| PALM (pruned) | 8 | 4 | 0.0344 ± 0.0087 | 0.0233 ± 0.0064 |
| PALM (unpruned) | 8 | 8 | 0.0156 ± 0.0036 | 0.0089 ± 0.0018 |
| Uniform | 8 | 8 | 0.0355 ± 0.0132 | 0.0242 ± 0.0093 |
| Random | 8 | 8 | 0.2855 ± 0.0188 | 0.2678 ± 0.0258 |
| PALM (pruned) portfolio size = 5 | ||||
| PALM (pruned) | 14 | 5 | 0.0002 ± 0.0002 | 0.0002 ± 0.0001 |
| PALM (unpruned) | 14 | 14 | 0.0002 ± 0.0002 | 0.0002 ± 0.0001 |
| Uniform | 14 | 14 | 0.0088 ± 0.0038 | 0.0053 ± 0.0021 |
| Random | 14 | 14 | 0.2163 ± 0.0190 | 0.2021 ± 0.0235 |
| PALM (pruned) portfolio size = 6 | ||||
| PALM (pruned) | 14 | 6 | 0.0000 ± 0.0000 | 0.0000 ± 0.0000 |
| PALM (unpruned) | 14 | 14 | 0.0000 ± 0.0000 | 0.0000 ± 0.0000 |
| Uniform | 14 | 14 | 0.0088 ± 0.0038 | 0.0053 ± 0.0021 |
| Random | 14 | 14 | 0.2163 ± 0.0190 | 0.2021 ± 0.0235 |
Unpruned PALM outperforms both baselines in 7 of the 8 configurations at matched oracle-call budgets and portfolio sizes. Moreover, pruned PALM outperforms both baselines in 6 of the 8 configurations despite deploying substantially fewer policies. Overall, these results demonstrate PALM's strong approximation quality under equal training budgets, with further deployment savings through pruning.
Furthermore, we would like to note that the manuscript already includes the Uniform PALM ablation (Figure 4) across all four datasets. This ablation replaces PALM's structured grid with a uniform grid of the same size, while retaining the same lazy-search and pruning procedures and matching the final portfolio size. PALM generally outperforms this ablation, even though the oracle-call budgets are comparable and generally slightly larger for Uniform PALM (See the Appendix for exact oracle numbers). These results provide further evidence of PALM's effectiveness under comparable training and deployment budgets.
Question 2
How is Rmax being estimated in the experiments?
For all experiments except HelpSteer, the reward functions are normalized to , so provides a valid upper bound. For HelpSteer, we estimate using 2,000 calibration responses generated by the base policy (four responses for each of 500 training prompts), as described in Appendix D.6.
Question 3
For HelpSteer, how is the multiplicative gap being computed at test weights when the oracle is negative or close to zero?
All empirical oracle values at the HelpSteer test preferences were strictly positive; the smallest was 0.09508 in our standardized reward units, in both the four-dimensional and nineteen-dimensional evaluations. Thus, we compute the multiplicative gap following the original definition, without denominator flooring.
Weakness 1
Thank you for the comment. The estimate of Rmax does not enter the reported gaps. All reported gaps use actual oracle values on held-out test weights. An error in R^max can therefore change how the portfolio is built, but it cannot bias the reported gaps. Furthermore, we agree that estimating Rmax from finite samples does not necessarily provide a valid theoretical upper bound. However, empirical estimation of unknown global quantities is a recognized practical approach (e.g., Huang et al., 2023, Weng et al., 2018, for Lipschitz constants). We use such estimates to guide the practical implementation, rather than claim that they provide formal certificates. Importantly, PALM can also operate without knowing Rmax or fmax, since Tool 2 of Lemma 3.6 relies only on convexity. Indeed, our RLVR-GSM experiments already use Tool 2 alone. Furthermore, we extend our theoretical analysis to accommodate imperfect policy oracles and provide a formal statement and proof sketch below.
More broadly, the theoretical guarantees are established by our analysis under explicitly stated assumptions, while the experiments assess PALM's practical performance. That is their purpose. Empirical results are not intended to prove worst-case guarantees, nor do practical implementations necessarily need to satisfy every sufficient condition of the theoretical analysis. These are distinct and complementary contributions. As is common in optimization and machine learning, practical parameter choices need not satisfy all sufficient conditions for theoretical guarantees and may yield better empirical performance (e.g., step sizes in gradient descent). We will revise the text to separate the proved guarantees from the empirical gaps more clearly.
References
Weakness 2
We conducted additional experiments matching the number of oracle calls between PALM and the Uniform and Random baselines. The manuscript also includes the Uniform PALM ablation, which uses comparable oracle-call budgets with PALM. These results show that PALM generally outperforms the baselines under comparable training budgets, despite having substantially smaller final portfolio sizes. Please refer to our response to Question 1 below for details.