acceptodds
Under review as a conference paper at ICLR 2027

Pareto-GRPO: Prompt-Local Pareto Credit Assignment for Efficient Search-Augmented Reasoning

Abstract

Group-relative reinforcement learning works well when a verifier supplies a clear scalar reward, but search agents expose a missing interface: trajectories can reach the same answer while using very different amounts of search. Fixed scalarizations resolve such ties by imposing one global exchange rate between correctness and cost, even though the useful amount of search is prompt dependent. We introduce Pareto-GRPO, which keeps the critic-free GRPO update but orders each prompt's rollout group by Pareto dominance over correctness and search efficiency. A bounded rank utility then refines the group-normalized correctness advantage. This ordinal construction is invariant to coordinate-wise monotone reward transforms and admits an exact first-order objective characterization. In a controlled Qwen2.5-7B comparison with identical initialization, data, retrieval, optimization, and validation-only model selection, Pareto-GRPO reaches 0.465 average exact match with 1.78 searches and 0.286 hypervolume. The strongest tuned scalar baseline reaches 0.441, 1.85, and 0.258, respectively; all five seed-wise accuracy differences favor Pareto-GRPO. The advantage persists under enforced search caps, alternative cost definitions, and a three-objective extension. Across seven benchmarks and two model scales, Pareto credit assignment improves accuracy, efficiency, and run-to-run stability together.

Primary area:
reinforcement learning
Keywords:
group-relative policy optimization, multi-objective reinforcement learning, Pareto ranking, credit assignment, search-augmented reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_q7GBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Pareto-GRPO, which incorporates prompt-local Pareto ranking into GRPO to balance answer correctness and search efficiency. Rather than relying on a fixed weighted reward, the method ranks trajectories within each prompt group according to both correctness and search cost. Experiments on seven search-augmented QA benchmarks with Qwen2.5-3B and Qwen2.5-7B show improvements in accuracy, search efficiency, and training stability over several GRPO-based baselines.

Critical strengths
  • Using prompt-local Pareto ranking for credit assignment is intuitive and well motivated. It addresses the limitations of fixed reward weighting while requiring only a relatively minor modification to the standard GRPO framework.
  • In addition to the empirical evaluation, the paper provides theoretical analysis and proofs for the proposed method.
Critical weaknesses
  • The maximum search budget is set to B=4, which seems relatively small for evaluating search efficiency. Allowing only four searches may make it difficult to fully distinguish between efficient and inefficient search strategies, particularly on more complex tasks that require longer search trajectories.
  • Although the evaluation covers seven benchmarks, most of them are relatively easy QA datasets. They may not be sufficiently challenging to demonstrate the effectiveness of the proposed method in more realistic search settings. Additional evaluation on harder benchmarks, such as BrowseComp-Plus and GAIA-Text, would make the conclusions more convincing.
Questions for authors
  • Have the authors considered evaluating larger budgets, such as B=8 or B=12? It would be helpful to know whether the observed improvements remain consistent when longer search trajectories are allowed.
  • Could the authors provide additional results on more challenging search benchmarks, such as BrowseComp-Plus or GAIA-Text? Such experiments would help clarify whether the method generalizes to more complex, real-world search tasks.
Reviewer_7rKUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Pareto-GRPO, which performs Pareto ranking over correctness and search efficiency within each rollout group to refine the group-relative advantage in GRPO. Instead of explicitly scalarizing multiple objectives, the method adds a bounded Pareto rank utility on top of the standard correctness advantage. Experiments are conducted on Qwen2.5-3B and Qwen2.5-7B, demonstrating performance improvements over GRPO.

Critical strengths

Pareto-GRPO is simple and reasonably motivated. The method introduces a search-efficiency signal through only a small modification to GRPO, and shows some performance improvements in the experiments.

Critical weaknesses
  1. A large portion of the paper appears to be AI-generated, and there is still substantial room for improvement in the readability of both the writing and the figures.
  2. The baseline comparisons are not sufficiently comprehensive. The current experiments mainly compare Pareto-GRPO with outcome-only GRPO, but lack direct comparisons with existing methods based on process rewards or step-level credit assignment.
Questions for authors
  1. Could the authors add a baseline that prioritizes correctness and uses search efficiency only as an auxiliary reward to rank trajectories with the same correctness reward? Comparing Pareto-GRPO against such a baseline would help determine whether Pareto ranking itself provides additional benefits.

  2. During actual RL training, how often does Pareto ranking produce an ordering different from the simple “correctness first, efficiency second” ranking described above?

  3. What is the exact definition of valid search used in the paper?

Reviewer_ujLSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies trajectory-level credit assignment for search-augmented language models. When rollouts for the same prompt have identical correctness rewards but different search costs, outcome-only GRPO cannot exploit those efficiency differences, while direct scalarization requires a numerical trade-off between correctness and cost. Pareto-GRPO performs non-dominated sorting over correctness and efficiency within each prompt's rollout group, adds a bounded rank utility to the normalized correctness advantage, and retains a GRPO-style policy update. The paper provides a fixed-behavior first-order objective characterization and ordering-related properties, and reports results on seven QA benchmarks at two model scales. The controlled 7B comparisons, rank ablations, enforced search caps, and alternative-cost experiments provide the main empirical support.

Critical strengths
  1. The method is simple and directly addresses the identified problem. Keeping correctness as the primary training signal while using within-prompt dominance to distinguish otherwise tied trajectories is an understandable design that requires neither additional process labels nor a learned critic. The paper also explicitly acknowledges that a cheap but incorrect trajectory can belong to the first front, rather than presenting Pareto ranking itself as correctness-first ordering. (Sections 3.1–3.2, lines 123–145)

  2. The main text provides relatively thorough empirical support and does not limit the comparison to an outcome-only baseline. In the controlled 7B comparison, EM improves from 0.441 to 0.465 and average searches decrease from 1.850 to 1.780 relative to the strongest validation-selected scalarization baseline, with positive EM differences for all five seeds. The sweeps and high-weight endpoints, rank ablations, and enforced search caps reported in the main text strengthen the evidence for tuning fairness, the contribution of the ordering signal, and gains under budget constraints, respectively. Alternative costs and three objectives further test robustness. (Sections 4.2–4.4, Tables 2–4, 6, and 8–9)

Critical weaknesses

W1. The current ablations do not sufficiently isolate the specific contribution of Pareto fronts over simpler ordinal shaping The central design maps within-prompt Pareto fronts into an auxiliary advantage, but Table 4 primarily compares true ranks with shuffled ranks, rank-only training, and scalarized rewards. True ranks outperforming shuffled ranks establishes that the ordering is informative; it does not distinguish whether the gains require Pareto fronts across correctness classes or merely cost ranking among trajectories with the same correctness. The first-order characterization in Section 3.3 does not directly explain why this particular ordering should outperform simpler alternatives. (Section 3, Table 4, lines 273–280)

W2. Key baselines and evaluation metrics lack sufficiently detailed operational definitions The main text does not sufficiently define several calculation rules that affect the central comparisons: how prompt-local scalarization determines weights for each prompt or rollout group; which rewards GDPO-style normalizes, over what scope, and in what order normalization and weighted combination are performed; and which policy points constitute validation HV and how results are aggregated across tasks and seeds. These rules directly affect baseline implementation, model selection, and result interpretation. Method names, weight-search ranges, and HV coordinates alone do not recover the calculation procedure. (Section 4 protocol, Sections 4.2–4.3, Tables 2 and 5)

W3. The relationship to existing Pareto optimization methods and the specific contribution require clearer positioning Section 3 maps within-prompt Pareto fronts to a bounded auxiliary advantage, but the increment over existing work needs clearer delineation. Li et al.'s ParetoPO (ICML 2026) already performs per-query non-dominated sorting and constructs GRPO advantages for tool use and retrieval-based QA. Zhong et al.'s Panacea (NeurIPS 2024) instead uses preference-conditioned low-rank adaptation for Pareto multi-objective LLM alignment and does not adopt this paper's trajectory-ranking mechanism. This paper retains a normalized correctness backbone and adds a bounded front utility, but should explain what specific problem this design addresses relative to ParetoPO and which results support its additional value. This paper should also clarify the differences in objectives and mechanisms between its trajectory-level credit assignment and Panacea's preference-conditioned optimization.

Related references: [1] Junyi Li et al. Towards Pareto-Optimal Tool-Integrated Agents with Pareto Ranking Policy Optimization. ICML 2026, Proceedings of Machine Learning Research 306:69408–69424. [2] Yifan Zhong et al. Panacea: Pareto Alignment via Preference Adaptation for LLMs. NeurIPS 2024, Advances in Neural Information Processing Systems 37:75522–75558.

Questions for authors
  1. Regarding W1: Has bounded shaping that preserves the correctness advantage and ranks cost only among trajectories with the same correctness been compared? If so, what are the results under matched group size and comparable shaping magnitude? If not, which existing results in the main text distinguish the contribution of Pareto fronts from that of generic cost ranking?

  2. Regarding W2: Please specify the weight-determination rule for prompt-local scalarization, the normalization scope and reward-combination formula for GDPO-style, and the policy-point set and cross-task/cross-seed aggregation procedure for validation HV.

  3. Regarding W3: Relative to ParetoPO, what specific problem is addressed by retaining the correctness backbone and adding a bounded front utility, and which main-text results support that increment? How does trajectory-level credit assignment here differ in objectives and mechanisms from Panacea's preference-conditioned Pareto optimization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.