acceptodds
Under review as a conference paper at ICLR 2027

Post-Selection-Valid Conformal Guarantees over Continuous Preferences for Multi-Objective Reinforcement Learning

Abstract

Many multi-objective reinforcement learning applications operate in a decision-support mode: a system first reveals a set or curve of trade-offs, and a user then chooses one operating point for execution. Existing guarantees are usually pointwise and protect a preference fixed in advance; they need not remain valid when the preference is selected after the full curve has been revealed. We study this release-then-select problem and propose CERTPREF, which publishes a preference-indexed certified lower-bound curve for the realized normalized scalarized return of the policy that is actually executed. CERTPREF calibrates one-sided conformal bounds at finitely many frozen anchor policies, makes the anchor guarantees simultaneous, extends them to all preferences using a trajectory-level Lipschitz envelope, and executes the same anchor policy that defines the displayed bound. This yields a distribution-free marginal post-selection guarantee for a single deployment query over a continuous preference space. Experiments across multi-objective control benchmarks show that CERTPREF controls post-selection violations below the target level, often conservatively, while retaining informative lower bounds. Pointwise baselines can be tighter, but they are not designed to control selection after the curve is revealed.

Primary area:
reinforcement learning
Keywords:
multi-objective reinforcement learning, conformal prediction, post-selection inference, continuous preferences, uncertainty quantification, decision support, distribution-free guarantees

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qBeMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies post-selection certification in continuous-preference multi-objective reinforcement learning, where a user inspects a published trade-off curve before choosing a preference for deployment. The authors propose CERTPREF, which calibrates one-sided conformal bounds for a finite library of frozen anchor policies, applies simultaneous error control, and extends the bounds to continuous preferences using trajectory-level Lipschitz continuity. Deployment executes the anchor policy matched to the displayed certificate. Under exchangeability and Lipschitz assumptions, the method provides a finite-sample marginal lower-bound guarantee for a single realized rollout despite adaptive preference selection. Experiments on multi-objective MuJoCo tasks examine coverage, certificate tightness, and computational cost, with additional analyses of routing, anchor selection, and higher-dimensional preferences.

Critical strengths
  1. The paper distinguishes fixed-preference guarantees from those needed after inspecting the published curve, and separates realized-rollout coverage from expected-return confidence bounds. CERTPREF combines conformal calibration, simultaneous error control, and Lipschitz extension. Matched policy routing ensures the displayed certificate refers to the policy actually executed.
  2. The paper clearly states its marginal, single-query guarantee and analyzes how tightness depends on Lipschitz constants, anchor coverage, and preference-space dimension.
  3. The paper provides extensive empirical diagnostics. Adaptive-selection tests, component ablations, repeated calibration splits, and anchor-budget studies complement the main results. The three-objective audit usefully examines whether certificates remain informative beyond coverage alone.
Critical weaknesses
  1. The relative efficiency of the certification construction could be explored further. The paper considers dense-grid Bonferroni and discusses the challenges of applying alternative simultaneous corrections. However, most empirical comparisons are against pointwise bounds or component ablations. Exploring an alternative based on joint prediction regions for vector returns, or discussing its potential trade-offs, would help clarify when the proposed Lipschitz-extension construction is preferable.

  2. Practical usefulness beyond two objectives remains an open question. As the authors acknowledge, only about 5.3% of evaluated preferences receive a positive lower bound in the three-objective audit, even with 200 anchors and 4,000 calibration rollouts per anchor. Further analysis of the conditions under which informative certificates are achievable would help clarify the approach’s practical scope.

Questions for authors

In the three-objective audit, what primarily limits certificate informativeness: anchor-policy performance, simultaneous calibration, or the Lipschitz extension penalty? An explanation based on the existing results would help clarify the method’s practical limitations.

3 comments
AuthorsOpenReview ↗

Joint Prediction Regions and Three-Objective Certificate Tightness

Thank you for the careful review. The joint-region comparison and the question about three-objective tightness are both helpful ways to sharpen the paper's practical message.

“Exploring an alternative based on joint prediction regions for vector returns, or discussing its potential trade-offs, would help clarify when the proposed Lipschitz-extension construction is preferable.”

A joint return region is a natural alternative, as each rollout already records all reward components together. For each frozen policy πi\pi_i, calibrate a region KiK_i for its vector return, then report the lowest weighted return within that region:

bi(w)=inf⁡g∈Kiw⊤g,L(w)=max⁡ibi(w).b_i(w)=\inf_{g\in K_i}w^\top g,\qquad L(w)=\max_i b_i(w).

The policy attaining the maximum is executed. Allocating coverage errors αi\alpha_i across policies with ∑iαi≤α\sum_i\alpha_i\leq\alpha protects the displayed curve when a preference is chosen after inspection.

We compared joint lower rectangles and ellipsoids with the scalar Lipschitz construction on the existing five two-objective tasks, using the same calibration split for every method. The outcome varies by task. With the nearest-anchor policy fixed, on HalfCheetah the scalar construction gives a mean bound of 29.50, compared with 28.16 for a fitted lower rectangle; on Hopper, the rectangle gives 16.58 versus 14.84 for the scalar construction. Choosing a different anchor can also change the result. The advantage of CERTPREF is its direct scalar calibration and simple extension over continuous preferences. Joint regions can be useful when modeling the objectives together yields tighter projected bounds.

“In the three-objective audit, what primarily limits certificate informativeness: anchor-policy performance, simultaneous calibration, or the Lipschitz extension penalty?”

This is an important distinction. Simultaneous calibration causes the larger measured reduction, and the Lipschitz extension contributes another substantial loss. In the existing m=200m=200 audit, the average pointwise anchor lower bound is about 0.853. Requiring simultaneous validity for all 200 anchors lowers it by 0.725, leaving about 0.128. Extending those bounds to the queried preferences subtracts another 0.530, giving an average raw bound of about −0.403.

The negative value is the outcome of the bound calculation. Since the discounted returns in this experiment are nonnegative, the published bound is clipped to zero. At such preferences the guarantee is safe, but zero adds little information. Appendix A.9's 5.3% means that about five in every hundred evaluated preferences still receive a strictly positive lower bound, providing information beyond that known zero floor.

Anchor-policy quality determines the starting lower-tail return; simultaneous calibration and anchor-to-query distance determine how much of it remains in the certificate. This decomposition shows why improving the policies, collecting enough calibration rollouts for the chosen anchor count, and placing anchors closer to likely queries are all useful. Section 5.3 and Appendix C.7 quantify the increasing geometric cost as the number of objectives grows. We will make this interpretation of Appendix A.9 clearer.

AuthorsOpenReview ↗

Joint return bounds on MuJoCo and what limits the bounds on Highway

Your joint-region suggestion changed the empirical conclusion. Our first reply reported that the scalar cone was tighter on HalfCheetah with 10,000-step policies. With one million steps per policy and the same nearest executing policy at each of 401 preferences, the joint rectangle now has a higher grid-mean bound in all 25 MuJoCo libraries. On HalfCheetah the rectangle and cone give 90.03 and 88.40. On Hopper they give 20.43 and 18.65. The rectangle uses 40 paired episodes to fit its shape and 160 to calibrate it. The cone uses all 200 scalar returns. With each method free to execute the policy giving its own highest bound, the rectangle still leads on all five task averages (Tables 1 and 6).

The practical distinction is what was logged. If only the anchor-weighted return is stored, a known bound on within-episode return differences suffices for the cone. If absolute component ranges are also known, the same scalar threshold permits the tighter box projection. If the component returns are stored together, the rectangle was more informative on average in these tasks. The cone was nevertheless higher at 698 of 1,925 off-anchor comparisons on Ant. A weighted return retains the dependence of the two components within an episode. The rectangle may get its two coordinate limits from different episodes.

On Highway, simultaneous calibration produced the larger measured loss. At 200 anchors each policy receives a 0.05% allowance, so its lower bound is the second-lowest of 4,000 returns. At the nearest executing anchor the mean pointwise bound was 0.8527. Simultaneous calibration reduced it by 0.7252, and the distance deduction took another 0.5300, leaving −0.4025 before applying the known zero floor. Holding the same policies and scalar thresholds fixed, projecting onto the known component ranges raised the fraction of 231 preferences with a positive bound from 5.30% to 82.68% and the mean floored bound from 0.0061 to 0.1073 (Table 15). A positive bound can thus still be small. All anchors used the same random-action policy, the third component was zero in this sweep, and successive batches reused action-stream prefixes. These are descriptive comparisons across four seed groups. With a single policy law, fresh exchangeable return vectors could also be used to calibrate one joint region for all preferences. Thus our 200-way allocation measures the chosen construction, rather than an intrinsic cost of this simulator. Evidence for trained policies with three active objectives remains open. The coverage-radius discussion is in §5.2 and Appendix C.7.

We have updated the paper PDF. The supplement includes a one-page summary of the changes by section.

Thank you to the authors for the detailed response and clarifications. I appreciate the effort to address my questions and concerns. I will maintain my score.

Reviewer_PPEbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses the challenges in pointwise preference in multi-objective RL, namely that the pointwise guarantee protects only a preference that is fixed in advance and not the preference chosen by a user and that continuous preferences enable a preference to lie between all finite calibration points. Thereby it proposes CERTPREF which provides a distribution‑free guarantee for the routed anchor policy that ensures calibration, simultaneous validity, continuity, and matched execution. The method is evaluated across multiple benchmarks and by looking at multiple performance indicators.

Critical strengths

The paper presents a novel solution (Certperf) for an important problem in MORL preference selection. It defines the motivation and novelty reflected to the state-of-the-art scientific literature, methodology is clearly discussed and experiments and evaluation are comprehensive and sound.

Critical weaknesses

While the paper openly discusses the limitations, the realistic implementation and use of the method especially in cases with larger number of objectives are not discussed. Therefore, authors should add empirical evidence and discussion on Certperf’s practical use to show that it can be useful in non worst case, real settings.

Questions for authors

How realistic are the Lipschitz continuity and simplex coverage assumptions in practical MORL, and do they lead to meaningful bounds as the number of objectives increases?

3 comments
AuthorsOpenReview ↗

Practical Assumptions and Empirical Evidence for CERTPREF

Thank you for the encouraging assessment. Your question about the practical assumptions and their usefulness with more objectives deserves a more concrete answer.

Q1 “How realistic are the Lipschitz continuity and simplex coverage assumptions in practical MORL, and do they lead to meaningful bounds as the number of objectives increases?”

For the linear preferences studied here, the Lipschitz relation follows directly from bounded rollout returns. Once a rollout produces a vector return GG, changing the preference ww only changes the weighted sum w⊤Gw^\top G. Lemma C.2 gives

w⊤G≥w′⊤G−max⁡jGj−min⁡jGj2 ∥w−w′∥1.w^\top G\geq{w'}^\top G-\frac{\max_jG_j-\min_jG_j}{2}\,\|w-w'\|_1.

This holds even when the Pareto front is discrete or the preferred policy changes abruptly. Theorem 5.1 works with a finite library of frozen policies; placing anchors nearer to the queried preferences makes their bounds tighter.

The practical trade-off is visible in the certificate itself: at preference ww, an anchor contributes its calibrated lower-return bound bib_i, reduced by the distance penalty LC∥w−wi∥1L_C\|w-w_i\|_1. Useful bounds therefore depend on strong lower-tail policy returns, adequate calibration data, and good anchor placement. As the number of objectives grows, covering the preference simplex at the same resolution takes more anchors (§5.3, Appendix C.7).

W1 “Therefore, authors should add empirical evidence and discussion on Certperf’s practical use to show that it can be useful in non worst case, real settings.”

We appreciate the emphasis on practical usefulness. Table 1 already reports measured bounds and post-selection violations on five trained two-objective MuJoCo tasks. On Hopper, the observed violation at each method's selected preference is 11.4% for the conformally calibrated Pointwise baseline and 0.3% for CERTPREF, with mean bounds of 16.75 and 15.06. Appendix A.7 (Fig. 7, Table 10) examines less conservative constants: on HalfCheetah, the calibrated Lipschitz constant is 13.14 ± 0.14, compared with the global value 49.7. Appendix A.8 (Fig. 8, Table 11) also evaluates policy libraries obtained from BC, CAPQL, and SAC.

The actual use is simple (Algorithm 1): freeze the policies, calibrate their returns, publish the bound curve, and execute the policy associated with the chosen bound. Section 6.4 reports that most work occurs during training and calibration, with little computation at query time. We will make these practical results easier to find in the revised discussion. The current evaluations are in simulation, and real-world case studies remain future work.

AuthorsOpenReview ↗

What the bounds guarantee and how tight they are in simulation

We have updated the training budget and can now quantify the cost of selection protection. Our earlier reply quoted a Hopper Pointwise violation rate of 11.4%. With million-step policies, each method choosing its own highest displayed bound gave observed rates of 9.72% for Pointwise and 1.04% for CertPref. Averaged over the full preference grid, their displayed lower bounds were 20.45 and 18.73 in clipped, rescaled return units (Table 7). The approximately 1.73-unit difference describes the two full methods, including their respective policy-routing rules. For the curve-only choice, the Hopper rate supplies no evidence of Pointwise excess risk. A different choice using independent reference episodes did show excess risk across six fixed HalfCheetah and Hopper libraries (13.81%, one-sided 99% lower limit 13.40%, CertPref 1.43%).

The continuity requirement concerns reweighting a single rollout, even when the preferred policy changes abruptly. For two components, shifting weight ss changes return by s(G1−G2)s(G_1-G_2). A bound on the component difference therefore gives a checkable slope. Meaningful deployment also requires the selected frozen policy’s next rollout to follow its calibration law. In an Ant example, at weights (0.965, 0.035) the routed policy had been trained at (0.860, 0.140). The anchor lower bound of 84.43 fell to 74.00 after a distance deduction of 10.43. This is a simulated example of a numerical promise for one execution.

Sparse anchors can give a valid but small lower bound. In the three-objective Highway sweep, at 200 anchors with 4,000 calibration episodes each, only 5.30% of the 231 preferences received a strictly positive cone bound. Using the known component ranges with the very same thresholds raised that share to 82.68%, although the mean floored bound was still only 0.1073 (Table 15). Each anchor used the same random-action policy, and the third component was zero throughout this sweep. As successive batches reused action-stream prefixes, the sweep describes informativeness only. It leaves practical deployment with several active objectives and learned policies open (§5.2, Appendix C.7).

We have updated the paper PDF. The supplement includes a one-page summary of the changes by section.

Thank you for replying my comments and the efforts in improving the paper. I will keep my recommendation to accept the paper.

Reviewer_pe6DOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper derives bounds for the realized normalized scalarized return of the policy that is actually executed. The bounds are based on an a number of previously learned policies and on certain assumptions of reproducibility and Lipshitz constants and the distribution of returns per policy. Based on this, the result is considered a distribution-free. Experimental comparisons are made with pointwise baselines which may be tighter but are shown to be less flexible than the results obtained here.

Critical strengths

The paper is quite clearly written and focuses on a rarely considered aspect that may be interesting for the “deployment” of a set of given policies that approximate a Pareto front. It can be shown that bounds for the match of policies and any user-defined preferences can be derived are related to preferences in a (Lipshitz) continuous relation (which may be more or less critical in dependence on the stated assumptions, but one should keep in mind that in certain cases Pareto fronts are actually discrete)

The example in Fig. 10 although not being shown in the main paper helps a lot to clarify the function of the approach.

Critical weaknesses

The term “deployment” is a bit questionable as it would require something like a case study, but the title makes already clear what is meant, although it’s not clear why “a single-query deployment loop.” is it actually a loop?

The results are only highlighting the difference between “doing nothing” and the proposed method, while a more fair approach would be a comparison to any non-certified interpolation or smoothing-out scheme, which, however, would not be so drastic. Also, if the problem of w-violation was critical, one of the policy-interpolation algorithms could produce a manifold of policies at learning time, which again would lack the certification, but would be a good candidate for comparison. Practically, the problem (if examples like half-cheetah are relevant at all) would probably be solved by producing a few more policies in the first place.

Literature review should discuss also paper which produce similar effects already at learning, which can be an advantage, because if the assumption made here are not satisfied, one would anyway go back to the learning phase, to find out more about the problem. It may be pedagogically useful to separate techniques for interpolation from techniques for learning a Pareto-front approximation (be it continuous or pointwise), but it’s not clear who this would be a practical advantage.

Questions for authors

If the scalarized return is normalized, why do we need the γ\gamma in the Lipschitz constant? If the policies are arbitrarily given, γ\gamma may not even be known. On the other hand, the γ\gamma-related constant is a bit trivial, while other values of this constant seem to require more information. Most of the consideration would be possible also with an L2L_2 norm which seem to could give additional insights.

What is the point in using a negative score (C in line 113)? This may seem as a minor variation, but it’s completely arbitrary, distracts from the main points, leads to confusion, and any benefits are not explained.

Can you design an algorithm the uses the present approach to shape also the learning of the policies?

2 comments
AuthorsOpenReview ↗

Practical Comparisons and Clarifications on CERTPREF

Thank you for your thoughtful review.

Pareto fronts

Thank you for pointing out that Pareto fronts may be discrete. The Lipschitz relation used in §4.3 concerns the score of a fixed rollout, C(τ;w)=−w⊤G(τ)C(\tau;w)=-w^\top G(\tau): changing ww only reweights the same realized return vector. This argument places no smoothness requirement on the Pareto front or the mapping from preferences to optimal policies. Lemma C.2 gives the bound used for certificate extension:

C(τ;w)≤C(τ;w′)+max⁡jGj(τ)−min⁡jGj(τ)2 ∥w−w′∥1.C(\tau;w)\le C(\tau;w')+\frac{\max_j G_j(\tau)-\min_j G_j(\tau)}{2}\,\|w-w'\|_1.

The relation still holds when Pareto-optimal policies form a discrete set. Our router may switch policies abruptly as ww changes; Algorithm 1 ensures that the displayed certificate at each preference refers to the same frozen anchor policy that is executed.

Interpolation

We appreciate the suggestion to consider interpolation or smoothing, which can improve the available policies. In the submitted paper, the Pointwise baseline already uses conformal calibration (Fig. 3, Appendix A.1). The comparison examines what happens when a preference is selected after the lower-bound curve has been displayed. Figure 1 and Appendix A.3 (Table 6) evaluate both methods at the same selected preference: on MO-Hopper, the observed violation rates are 11.4% (Pointwise) and 0.0% (CERTPREF), with lower bounds of 23.5 and 20.8. We also examine anchor counts (Fig. 4, Appendix A.2) and frozen BC, CAPQL, and SAC policies (Fig. 8, Appendix A.8). Direct comparisons with learned continuous policy families remain future work.

γ and L₂

We use the standard discounted return G(τ)=∑tγtrtG(\tau)=\sum_t\gamma^t r_t, with each per-step reward component normalized to the interval [0, 1] (§3.1). This retains the usual RL return scale. Consequently, Gj≤1/(1−γ)G_j\le 1/(1-\gamma) and LC=12(1−γ)L_C=\frac{1}{2(1-\gamma)} (Lemmas C.1–C.2). Here γ\gamma specifies how the evaluation rollouts are scored and is fixed before calibration; the supplied policies may have been trained with a different discount. Using (1−γ)G(1-\gamma)G instead would rescale both the return and its bound, giving LC=1/2L_C=1/2.

We chose the global constant because it is available directly from the reward bounds and is worst-case sharp on the simplex. The paper also derives the trajectory-specific constant ℓ(τ)=12(max⁡jGj−min⁡jGj)\ell(\tau)=\tfrac12(\max_jG_j-\min_jG_j) (Lemma C.2) and calibrates it in Theorem B.5, with total failure probability bounded by α+β\alpha+\beta. Appendix A.7 (Fig. 7, Table 10) compares global, calibrated and empirical constants across five tasks: on HalfCheetah, the finite-horizon global constant is 49.7, versus 13.14 ± 0.14 for the calibrated constant. Appendix A.10 (Fig. 10) uses this refinement in the preference-shift experiments.

Appendix C.6 already gives the dual-norm formulation, including ℓ2\ell_2. Our choice of ℓ1\ell_1 yields a sharp dimension-independent constant on the simplex. In our two-objective setting, w(t)=(t,1−t)w(t)=(t,1-t) gives ∥w(t)−w(s)∥1=2 ∥w(t)−w(s)∥2\|w(t)-w(s)\|_1=\sqrt2\,\|w(t)-w(s)\|_2. With their respective sharp constants, both norms therefore give the same extension penalty.

Negative score

We chose C=−w⊤GC=-w^\top G to keep the calibration formulas in the standard split-conformal form, which uses an upper score quantile. Then C≤UC\le U gives exactly the desired return bound w⊤G≥−Uw^\top G\ge -U (§§3.1–3.2), and the published curve is L=−UL=-U. Thank you for pointing out that this choice could be clearer to readers; we will add a short explanation alongside the score definition in §3.1.

Single-query loop

Thank you for flagging this wording. In §3.3, “deployment” refers to using the trained policy after a preference has been selected; §6 evaluates this procedure in simulation. The protocol involves one preference choice followed by one execution. We agree that “loop” suggests repeated interaction, and will replace it with “single-query deployment protocol” and clarify the single-execution setting in §3.3.

Learning

Thank you for the interesting suggestion. One concrete extension is certificate-guided anchor refinement. First, use development rollouts to build a provisional bound curve and estimate the returns of the available policies. Prioritize preferences where the provisional bound falls furthest below the return attainable from the current library. The certificate then indicates what to improve: a large LC∥w−wi∥1L_C\|w-w_i\|_1 penalty suggests training a new anchor at that preference; a weak lower tail at the chosen anchor suggests fine-tuning that policy with a lower-tail return objective (e.g., CVaR). Recompute the provisional curve and repeat. Once training ends, freeze the policy library and use fresh, independent calibration rollouts for Algorithm 1. This would direct learning toward the preference regions that most limit the reported guarantee, while leaving the final post-selection validity argument intact.

AuthorsOpenReview ↗

Larger policy libraries, preference-conditioned policies and the evaluation discount

Your suggestion to train more policies led us to quantify that trade-off. Using successively larger subsets of the same million-step HalfCheetah libraries, increasing from two to sixteen farthest-point anchors raised the grid-mean lower bound from 75.27 to 89.03. Each added policy represents one million environment steps and 200 calibration episodes (Appendix A.2). A larger library improves the available options. Pointwise still calibrates separately at each preference, so increasing the library size alone supplies no guarantee for a preference chosen after viewing the curve.

We also evaluated a preference-conditioned CAPQL network at four fixed anchor inputs and four separately trained SAC policies at an equal total budget of four million environment steps. On Walker2d the mean lower bounds were 24.18 and 24.62, respectively (Table 14). This tests certifying frozen instances from a single network. Your proposed direct comparison would execute the network or an interpolated policy at the user’s exact weights. A comparison of its realized performance with the routed policies remains open. Certifying that execution would additionally require calibration or a justified relation to a calibrated policy.

Our previous reply quoted 11.4% Pointwise violations on Hopper. At the highest point of the Pointwise curve with million-step policies, the observed rate is 9.72%, against 1.04% for CertPref at that same preference (Table 8). The new formal evidence comes from a different rule: on six fixed HalfCheetah and Hopper libraries the preference at which independent reference episodes most often miss the bound gave 13.81% for Pointwise (one-sided 99% lower limit 13.40%) and 1.43% for CertPref (Appendix A.4). The curve-only selector did not establish excess risk on that matched design.

One clarification on normalization: the evaluation discount determines how early and late rewards are weighted, and it can be set even for a supplied policy whose training discount is unknown. Once that discount and the horizon are fixed, dividing the entire discounted sum by H=∑t=0T−1γtH=\sum_{t=0}^{T-1}\gamma^t rescales both the bound and its distance deduction. The sharp slope then becomes 1/21/2 in the simplex ℓ1\ell_1 norm. For two objectives the sharp ℓ2\ell_2 and ℓ1\ell_1 bounds give the same deduction (Appendix C.6). On HalfCheetah, with the new libraries and 450 calibration episodes per policy and two error allowances (4% for return and 6% for slope across the library), the largest calibrated slope averaged 19.79 against the global 49.67 (Table 13). The same 450 episodes supply both ranks.

Sections 3.1–3.3 now explain the minus sign and the single execution. An upper limit on minus the return is a lower limit on the return. The user chooses once, and the system executes the frozen policy supplying that bound. Our training proposal is to build a provisional curve on development episodes, use a large distance deduction to identify gaps in anchor placement and a poor anchor lower tail to identify policies to improve, then freeze the final library and calibrate it on fresh episodes. We have not evaluated that learning procedure.

We have updated the paper PDF. The supplement includes a one-page summary of the changes by section.

Comments on the paper

3 comments
Area_Chair_EYVVOpenReview ↗

Some additional Concerns

Dear Authors,

Thank you for the rebuttals. After reading the paper and the discussion, I have some concerns that the reviewers did not raise. I would appreciate your response during the discussion phase.

1. Necessity of the Lipschitz extension. Since the score C(τ;w)=−w⊤G(τ)C(\tau; w) = -w^\top G(\tau) is linear in ww, a guarantee uniform over all ww for a fixed anchor policy can be obtained without any extension. For example, one can calibrate a sup-score sup⁡w[C(τ;w)−ϕi(w)]\sup_{w} [C(\tau; w) - \phi_i(w)] for a fixed shape ϕi\phi_i, or a joint region for the vector return GG. This still needs only mm Bonferroni terms and has no LC∥w−wi∥1L_C \|w - w_i\|_1 penalty. Your own joint-rectangle result for Reviewer qBeM (Hopper: 16.58 vs. 14.84) supports this. Please compare against such a baseline. Otherwise the contribution seems to be a combination of standard tools: split conformal, a union bound, and a triangle inequality.

2. Evidence that Pointwise fails. With ncal=200n_{\text{cal}} = 200, the nominal miscoverage is about 0.09950.0995. The reported "failures" (0.119 and 0.114) are within about 11–1.51.5 standard errors of this value. Also, viol^max⁡\widehat{\text{viol}}_{\max} is a maximum over 401 noisy estimates, so it will exceed α\alpha even for a calibrated method. An upper confidence bound above α\alpha does not show a violation; a lower confidence bound above α\alpha would. Can you give a proper significance test?

3. Contradictions with Theorem 5.1. In Appendix A.2 (Fig. 4), some CertPref settings with few or poorly placed anchors have a UCB above α\alpha. The text explains this by "poor coverage geometry", but the theorem says validity does not depend on the anchors. Similarly, Fig. 3 shows "No extension" below α\alpha, while the text says it breaks off-anchor. Please clarify.

4. Experimental setup.

  • The anchor policies use only 10410^4 SAC steps (about 20 episodes), so they are almost untrained.
  • With ki=200=ncalk_i = 200 = n_{\text{cal}}, U(wi)U(w_i) is the worst of the 200 calibration rollouts. This explains violations around 1%1\% against a 10%10\% target.
  • Rewards are clipped from below at −5-5, so the bound may not hold for the true return.

5. Manuscript quality. The paper contains leftover pipeline text, for example "FIGURE_ATLAS.md", "Exp4", "reviewer-facing checks", and "strengthened main table". The figures also use "adversarial picking" / "attacker" while the text uses "optimistic-display". Please check the manuscript carefully, in line with your AI use statement.

Best regards, AC

AuthorsOpenReview ↗

When the Lipschitz extension is needed and how we tested Pointwise risk (1/2)

(1) When is the extension needed?

We agree that with paired objective returns, your joint-region and sup-score proposals both protect every preference using one calibration score per episode and one error allowance per policy. Neither requires the global Lipschitz deduction. We implemented both. At each of 401 preferences we held the executing policy fixed across methods (the nearest anchor), then averaged the lower bound over preferences and five libraries per task. Each policy had one million training steps and 200 episodes available before testing (Tables 1 and 6).

TaskScalar coneBox projectionJoint rectangleEllipsoidSup-score
HalfCheetah88.4089.5990.0390.0889.72
Ant68.0968.7968.9769.6070.21
Swimmer49.9150.4751.7051.7051.67
Walker2d31.1631.8432.9332.9332.86
Hopper18.6519.5920.4320.4320.12

The rectangle had a higher grid mean than the cone in all 25 libraries. On Ant the cone reached 98.7% of the rectangle’s grid mean, and was higher at 698/1,925 off-anchor comparisons (36.3%, and 35.9% over 20 additional calibration batches per library). With the cone restricted to the same final 160 episodes, the rectangle still led on all five task averages. This updates the 10,000-step results in our reply to qBeM. The first two methods used all 200 scalar anchor returns. For the three vector methods we used 40 paired returns to fit the shape and 160 to calibrate it. For the sup-score, our chosen shape has a pilot-fitted slope in distance to the anchor. Its offset is calibrated for the whole simplex at once. A shape fixed in advance could dispense with the fitting split.

What if a log retains only the weighted return at the training preference? With known component ranges, box projection finds the lowest return compatible with those ranges and the scalar bound. With only a bound on the within-episode component difference, the cone can be sharp. For example, suppose two components differ by at most 100 and their equal-weight average is at least 75. The pair (25,125) is then possible, so 25 is the best guaranteed lower bound at weight (1,0). Knowing both components also lie in [0,100] raises that lower bound to 50. Our MuJoCo logs contain both coordinates.

The statistical ingredients are standard. Our contribution is to specify what must be linked when a user inspects a MORL display and then acts. We construct an event on which the user’s choice is covered. With probability at least 1−α1-\alpha, one prospective rollout per policy satisfies that policy’s bounds at every preference, simultaneously across policies. Executing the policy that supplies the chosen bound transfers this event to the user’s choice. Theorem 5.1 proves this for the cone under exchangeability, a valid trajectory-level slope bound and fresh matched execution. Section 5.1 gives the same argument for a vector region. This is a marginal guarantee for one new rollout.

(2) Does Pointwise exceed 10% after selection?

The earlier 11.9% and 11.4% were descriptive. Pointwise builds the curve. How we choose from that curve affects the risk. We compared two rules on the same six frozen HalfCheetah and Hopper libraries and 321 interior preferences (Appendix A.4, Table 9).

Preference choicePointwisePointwise 99% lower limitCertPref
Highest bound on the displayed curve9.77%9.35%0.49%
Most misses among 1,000 independent reference episodes per routed policy13.81%13.40%1.43%

Each method selected from its own curve. The reference episodes were held fixed across rounds. Each choice was recorded before its 500 fresh tests. Only the reference-selected Pointwise endpoint was designated for formal testing before these repetitions. The unit of inference is a whole recalibration, selection and test round, with 50 independent rounds per library. If XrX_r is the fraction of misses in round rr, the 99% lower limit is (0.01∏r=1300Xr)1/300(0.01\prod_{r=1}^{300}X_r)^{1/300} (derivation in A.4). For the equally weighted mean risk of these six libraries, the test rejects the 10% null even after the fivefold adjustment (adjusted p≤3.453×10−40p\le3.453\times10^{-40}). The four earlier endpoint results were already known. The reference rule deliberately searches for bounds that look too high. Its risk exceeds 10% on these libraries. For the curve-only choice, excess risk was not established. In a separate 401-point HalfCheetah control the curve-only rule selected the same endpoint in all 300 rounds and had 10.12% misses (95% lower limit 9.71%). Appendix B.0 shows why curve-only selection can still fail. In its analytic example, each fixed choice has 50% risk, while choosing the largest of mm displayed lower bounds gives risk m/(m+1)m/(m+1).

AuthorsOpenReview ↗

Coverage plots, policy training, calibration ranks and reward clipping (2/2)

(3) Why can the upper confidence limit exceed 10%?

Theorem 5.1 averages over calibration and one fresh execution after selection. For each realized display, Fig. 6 (formerly Fig. 4) takes the largest observed miss rate across 401 preferences and constructs a simultaneous upper confidence limit for the largest grid risk. With two farthest-point anchors on HalfCheetah, the five-library mean of the observed maxima was 5.40% and the mean of the upper limits was 10.02%. An upper limit above 10% does not establish undercoverage. The theorem imposes no minimum anchor density. Wider spacing can increase the distance deduction. In the same sweep the mean lower bound rose from 75.27 with two anchors to 89.03 with sixteen.

The “No extension” ablation copies the nearest anchor’s threshold unchanged. Its earlier below-target result was descriptive. In Fig. 5D (formerly Fig. 3), the new HalfCheetah policies show precisely what can go wrong. At the anchor t=0t=0, 18 of 2,500 held-out episodes missed the anchor bound. One grid step away at t=0.0025t=0.0025, all 2,500 missed that same threshold. Deducting 0.248 return units for this weight change gave zero misses on those same episodes. Other constructions that calibrate an entire vector or curve can also handle this change of weights.

(4) Training, rank and clipping.

We replaced the 10,000-step SAC policies in the main comparison. All 400 policies (five tasks, five libraries per task, sixteen policies per library) now have one million environment steps each. The old 10,000-step runs averaged 114 completed episodes per policy across tasks. The new one-million-step runs averaged 10,555. Figure 4 in Appendix A.0 reports training returns and separate deterministic-policy evaluations at six checkpoints. In each of the 25 libraries, the mean raw return at the policies’ own training preferences was higher at one million steps than at initialization.

You are right about the rank. With 200 calibration episodes and a per-policy level of 0.1/160.1/16, the score rank is 200, so the anchor’s lower bound is its smallest calibration return. One fixed anchor has marginal miss probability at most 1/2011/201. Summing those upper bounds gives at most 16/20116/201 (7.96%) for any anchor to miss. The actual selected risk can be much smaller. CertPref’s observed rates at its own highest displayed bound ranged from 0.60% to 1.04% across tasks. These conservative bounds are a real cost of dividing the allowance across sixteen policies. For example, 319 calibration episodes per policy would make the bound the second-lowest return at this allocation.

In the main MuJoCo study the guarantee applies to rewards clipped to [−5,5] and rescaled to [0,1]. On Hopper, the second raw reward fell below −5 on 434,457 of 1,638,985 held-out steps (26.51%), so the main lower bound cannot simply be converted to a raw-return guarantee. We separately calibrated a joint rectangle directly on raw return vectors for three libraries each of HalfCheetah and Hopper. With 200 pilot, 800 calibration and 1,000 test episodes per policy, the nearest-policy grid-mean bounds were 407.58 and 27.46 raw return units, respectively (Appendix A.0). At the highest point of each raw rectangle display, the recorded misses totaled 0/3,000 and 20/3,000, respectively. These are descriptive results from the separate raw-return experiment.

(5) Manuscript corrections.

We checked the revised text and figure labels for the four drafting remnants you identified, including “FIGURE_ATLAS.md” and “reviewer-facing checks.” The two selection rules now have the same names in the paper and figures, “highest displayed bound” and “most reference violations.” Each name states what the selector can observe. The AI-use statement describes assistance with literature retrieval, writing, proofs, code, experimental design and analysis, and preparing and checking the figures and tables. We take responsibility for the manuscript and its results.

We have updated the paper PDF. The supplement includes a one-page summary of the changes by section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.