- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies post-selection certification in continuous-preference multi-objective reinforcement learning, where a user inspects a published trade-off curve before choosing a preference for deployment. The authors propose CERTPREF, which calibrates one-sided conformal bounds for a finite library of frozen anchor policies, applies simultaneous error control, and extends the bounds to continuous preferences using trajectory-level Lipschitz continuity. Deployment executes the anchor policy matched to the displayed certificate. Under exchangeability and Lipschitz assumptions, the method provides a finite-sample marginal lower-bound guarantee for a single realized rollout despite adaptive preference selection. Experiments on multi-objective MuJoCo tasks examine coverage, certificate tightness, and computational cost, with additional analyses of routing, anchor selection, and higher-dimensional preferences.
Critical strengths
- The paper distinguishes fixed-preference guarantees from those needed after inspecting the published curve, and separates realized-rollout coverage from expected-return confidence bounds. CERTPREF combines conformal calibration, simultaneous error control, and Lipschitz extension. Matched policy routing ensures the displayed certificate refers to the policy actually executed.
- The paper clearly states its marginal, single-query guarantee and analyzes how tightness depends on Lipschitz constants, anchor coverage, and preference-space dimension.
- The paper provides extensive empirical diagnostics. Adaptive-selection tests, component ablations, repeated calibration splits, and anchor-budget studies complement the main results. The three-objective audit usefully examines whether certificates remain informative beyond coverage alone.
Critical weaknesses
-
The relative efficiency of the certification construction could be explored further. The paper considers dense-grid Bonferroni and discusses the challenges of applying alternative simultaneous corrections. However, most empirical comparisons are against pointwise bounds or component ablations. Exploring an alternative based on joint prediction regions for vector returns, or discussing its potential trade-offs, would help clarify when the proposed Lipschitz-extension construction is preferable.
-
Practical usefulness beyond two objectives remains an open question. As the authors acknowledge, only about 5.3% of evaluated preferences receive a positive lower bound in the three-objective audit, even with 200 anchors and 4,000 calibration rollouts per anchor. Further analysis of the conditions under which informative certificates are achievable would help clarify the approach’s practical scope.
Questions for authors
In the three-objective audit, what primarily limits certificate informativeness: anchor-policy performance, simultaneous calibration, or the Lipschitz extension penalty? An explanation based on the existing results would help clarify the method’s practical limitations.
3 comments
Joint return bounds on MuJoCo and what limits the bounds on Highway
Your joint-region suggestion changed the empirical conclusion. Our first reply reported that the scalar cone was tighter on HalfCheetah with 10,000-step policies. With one million steps per policy and the same nearest executing policy at each of 401 preferences, the joint rectangle now has a higher grid-mean bound in all 25 MuJoCo libraries. On HalfCheetah the rectangle and cone give 90.03 and 88.40. On Hopper they give 20.43 and 18.65. The rectangle uses 40 paired episodes to fit its shape and 160 to calibrate it. The cone uses all 200 scalar returns. With each method free to execute the policy giving its own highest bound, the rectangle still leads on all five task averages (Tables 1 and 6).
The practical distinction is what was logged. If only the anchor-weighted return is stored, a known bound on within-episode return differences suffices for the cone. If absolute component ranges are also known, the same scalar threshold permits the tighter box projection. If the component returns are stored together, the rectangle was more informative on average in these tasks. The cone was nevertheless higher at 698 of 1,925 off-anchor comparisons on Ant. A weighted return retains the dependence of the two components within an episode. The rectangle may get its two coordinate limits from different episodes.
On Highway, simultaneous calibration produced the larger measured loss. At 200 anchors each policy receives a 0.05% allowance, so its lower bound is the second-lowest of 4,000 returns. At the nearest executing anchor the mean pointwise bound was 0.8527. Simultaneous calibration reduced it by 0.7252, and the distance deduction took another 0.5300, leaving −0.4025 before applying the known zero floor. Holding the same policies and scalar thresholds fixed, projecting onto the known component ranges raised the fraction of 231 preferences with a positive bound from 5.30% to 82.68% and the mean floored bound from 0.0061 to 0.1073 (Table 15). A positive bound can thus still be small. All anchors used the same random-action policy, the third component was zero in this sweep, and successive batches reused action-stream prefixes. These are descriptive comparisons across four seed groups. With a single policy law, fresh exchangeable return vectors could also be used to calibrate one joint region for all preferences. Thus our 200-way allocation measures the chosen construction, rather than an intrinsic cost of this simulator. Evidence for trained policies with three active objectives remains open. The coverage-radius discussion is in §5.2 and Appendix C.7.
We have updated the paper PDF. The supplement includes a one-page summary of the changes by section.
Thank you to the authors for the detailed response and clarifications. I appreciate the effort to address my questions and concerns. I will maintain my score.
Joint Prediction Regions and Three-Objective Certificate Tightness
Thank you for the careful review. The joint-region comparison and the question about three-objective tightness are both helpful ways to sharpen the paper's practical message.
“Exploring an alternative based on joint prediction regions for vector returns, or discussing its potential trade-offs, would help clarify when the proposed Lipschitz-extension construction is preferable.”
A joint return region is a natural alternative, as each rollout already records all reward components together. For each frozen policy πi, calibrate a region Ki for its vector return, then report the lowest weighted return within that region:
bi(w)=infg∈Kiw⊤g,L(w)=maxibi(w).
The policy attaining the maximum is executed. Allocating coverage errors αi across policies with ∑iαi≤α protects the displayed curve when a preference is chosen after inspection.
We compared joint lower rectangles and ellipsoids with the scalar Lipschitz construction on the existing five two-objective tasks, using the same calibration split for every method. The outcome varies by task. With the nearest-anchor policy fixed, on HalfCheetah the scalar construction gives a mean bound of 29.50, compared with 28.16 for a fitted lower rectangle; on Hopper, the rectangle gives 16.58 versus 14.84 for the scalar construction. Choosing a different anchor can also change the result. The advantage of CERTPREF is its direct scalar calibration and simple extension over continuous preferences. Joint regions can be useful when modeling the objectives together yields tighter projected bounds.
“In the three-objective audit, what primarily limits certificate informativeness: anchor-policy performance, simultaneous calibration, or the Lipschitz extension penalty?”
This is an important distinction. Simultaneous calibration causes the larger measured reduction, and the Lipschitz extension contributes another substantial loss. In the existing m=200 audit, the average pointwise anchor lower bound is about 0.853. Requiring simultaneous validity for all 200 anchors lowers it by 0.725, leaving about 0.128. Extending those bounds to the queried preferences subtracts another 0.530, giving an average raw bound of about −0.403.
The negative value is the outcome of the bound calculation. Since the discounted returns in this experiment are nonnegative, the published bound is clipped to zero. At such preferences the guarantee is safe, but zero adds little information. Appendix A.9's 5.3% means that about five in every hundred evaluated preferences still receive a strictly positive lower bound, providing information beyond that known zero floor.
Anchor-policy quality determines the starting lower-tail return; simultaneous calibration and anchor-to-query distance determine how much of it remains in the certificate. This decomposition shows why improving the policies, collecting enough calibration rollouts for the chosen anchor count, and placing anchors closer to likely queries are all useful. Section 5.3 and Appendix C.7 quantify the increasing geometric cost as the number of objectives grows. We will make this interpretation of Appendix A.9 clearer.