acceptodds
Withdrawn from ICLR 2027

Pursuing the Pareto Frontier for Interactive Intervention with LLM-based Reward Design

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reward design is challenging when agents must balance multiple deployment objectives. In interactive intervention, a useful reward should sustain task performance across a broad range of expert-assistance budgets, yet existing LLM-based methods rank rewards with a scalar objective, reducing multi-objective reward design to single-point evaluation. We introduce PARSE, a Pareto-aware framework that evaluates each reward by the preference-response set induced by a single preference-conditioned policy. For every candidate reward, one policy is trained across preferences, screened for Interaction Rate span and monotonicity, and evaluated by the hypervolume contribution of its nondominated operating points. Trajectory diagnoses of failed and inefficient episodes then guide reward mutation and crossover, while global exploration maintains search diversity. On PointNav in unseen Habitat scenes, PARSE attains the highest hypervolume among methods with multi-point trade-off sets and reaches Success at a Interaction Rate, outperforming the strongest baseline by percentage points. The selected reward also transfers unchanged to ObjectNav, where it outperforms no-help and random-help baselines.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, reward design, large language models, multi-objective reinforcement learning, Pareto optimization, embodied navigation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GvetOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PARSE, an LLM-guided reward design framework for learning when an agent should request expert assistance. It trains a preference-conditioned interaction policy to balance task success and interaction cost while keeping the navigation policy frozen. PARSE uses GPT-5.5 to generate and iteratively refine reward functions through mutation, crossover, and exploration. Candidate rewards are evaluated across 11 preference settings and selected based on their Pareto hypervolume, interaction-rate coverage, and monotonicity. Experiments on unseen HM3D PointNav scenes show that PARSE achieves the highest hypervolume (0.7581 vs. 0.6715 for random help) and 82.8% success at a 20% interaction rate, outperforming the strongest baseline by 4.9 percentage points. The learned reward also transfers to ObjectNav without modification.

Critical strengths

1. Evaluating a reward by the set of behaviors its preference-conditioned policy induces, with screening for span and monotonicity, fits deployment settings with varying assistance budgets. The collapse of scalar-objective search to extremes (99.98% vs. 0.01% interaction) illustrates the motivation well.

2. Training, ranking and feedback use disjoint MP3D splits, and final evaluation uses unseen HM3D scenes. The appendix provides full prompts, the final reward code and per-method compute budgets.

3. A single policy covers a wide range of interaction rates, with monotone control and smooth interpolation to unseen λ (App. E). The final reward (Eq. 13) is readable and transfers to ObjectNav.

Critical weaknesses
  1. The baselines do not isolate the contribution of the search.

    • The Eureka variants use objectives that degenerate by construction. With a shortest-path expert, maximizing Success means always asking, and the ratio ΔSuccess/IR drives interaction to zero. A fair scalar baseline would rank candidates by the HV of their own preference-response set, which is PARSE's own final selection criterion.
    • The key control is missing: the human reward plus a λ-dependent help cost, trained as a preference-conditioned policy.
    • The human-reward results are taken from Igbinedion & Karaman (2024) rather than rerun with the same frozen policy, expert and episodes.
  2. Statistical evidence is not enough.

    • The search samples the LLM at temperature 1.0 but appears to be run only once for PARSE and for each ablation, so the ablation gaps (0.7581 vs. 0.6782 and 0.7058) may fall within search-to-search variance.
    • HV largely rewards coverage: the human reward scores below random help (0.6089 vs. 0.6715), and baseline coverage depends on the authors' sweep grids.
    • The matched-rate comparison is a single spline-interpolated point (20% interaction). The HULA, MC and random-help baselines use one run each, without confidence intervals.
  3. Limited scope and novelty. Evaluation covers one navigation task family with a perfect planner as the expert. The components (LLM evolutionary reward search, hypervolume-contribution selection, preference-conditioned policies) are established. The search costs 45 training runs of 250k steps plus 96 GPT-5.5 calls, for a 4.9-point gain at one operating point.

Questions for authors

See weaknesses

Reviewer_LE6kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper treats LLM-based reward design as a search over sets of behaviors instead of single scores. Each candidate reward trains one preference-conditioned policy that decides when a navigation agent asks an expert for help, and candidates are ranked by how much they add to the hypervolume of the success-versus-interaction curve. On PointNav with a frozen navigation policy, one reward gives a policy whose help rate can be dialed across a wide range on unseen scenes, with higher hypervolume than four non-LLM ask-for-help baselines. The paper traces its difference from scalar reward search to how candidates are evaluated.

Critical strengths
  • The method adds one knob to Eureka-style reward search: rank each candidate by a set of points across the preference weight instead of one score. Even so, it makes a prediction that scalar search doesn't, namely that one reward and one policy give a trade-off you can dial, and the prediction is borne out on scenes left out of search. The help rate falls steadily from about 73% to 2% as the preference weight goes from 0 to 1 (Table 2), and held-out weights behave the same way. If this generalizes, a deployment budget could change without retraining.
Critical weaknesses
  • Whether set-valued ranking is what makes this work has not been tested. The ablation in Section 5.4 swaps it for two scalars, Success and Success gain per interaction, and both fail by construction: the first rewards always asking and the second rewards never asking. The needed comparison, the same prompt and filter with a sensible scalar, is absent. On top of that, the generation prompt already asks for a help rate that falls monotonically with the weight and covers a wide span, and the screening filter checks the same conditions. Nothing removes the prompt's requirements or the filter alone, so it remains unclear which part produces the curve. Nor does the paper state when ranking by sets should stop outperforming a scalar score, so the authors leave the method's limits unmarked.
  • The comparisons are hard to check. The 4.9-point Success margin in the abstract doesn't name the baseline, and operating points for the other methods appear only in Fig. 2. Section 5.2 states that the authors trained the Human baseline, while the appendix attributes its results to an earlier paper, so it may not be comparable.
Questions for authors
  1. Could you run a scalar-ranking control with the same prompt and filter, such as mean Success over preference weights or Success minus a swept interaction penalty? Runs that drop only the filter, and only the prompt's requirements, would show which piece gives the curve its shape.
  2. Did you train the Human curve yourselves on the same policy and episodes, or copy it? Which baseline is behind the 4.9-point margin?
  3. Random help in Table 1 and the first search iteration on MP3D report the same hypervolume to four digits. Are both values correct?
Reviewer_HUcCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PARSE studies how an autonomous agent should decide when to request expert assistance. Its goal is to support different trade-offs among task success, navigation efficiency, and intervention cost.

An LLM generates reward functions used to train preference-conditioned intervention policies. The framework searches over reward code through mutation, crossover, and trajectory-based feedback. Candidate rewards are evaluated according to the trade-offs their trained policies achieve, including their contribution to the Pareto frontier.

On PointNav, the paper reports 82.8% success at a 20% interaction rate. This particular operating point is obtained through interpolation; nearby directly measured results are provided in the appendix.

Critical strengths
  1. A practical problem formulation. Different users and deployment settings may tolerate different levels of assistance. A preference-conditioned policy makes that trade-off adjustable.
  2. Separation between reward search and final evaluation. Testing on held-out scenarios provides more convincing evidence than evaluating only on scenarios used during reward development.
  3. Several useful empirical experimental setups. The paper includes five seeds, component ablations, and experiments transferring the discovered reward to another task.
Critical weaknesses
  1. The Eureka comparison could be stronger. The scalar-objective variants tend toward nearly constant intervention or almost complete autonomy. A more informative comparison would run reward search across multiple scalar weights under a matched total search budget.
  2. The assistance model is restricted. Experiments use a frozen navigation policy and a shortest-path expert. They do not establish performance with noisy, delayed, inconsistent, or otherwise imperfect human assistance.
Questions for authors

In the ObjectNav experiment, the reward code transfers unchanged, but the intervention policy is trained again. This demonstrates reward transfer rather than zero-shot transfer of the learned policy. So conclusions about transferability can be more precisely framed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.