acceptodds
Under review as a conference paper at ICLR 2027

Policy Optimization with Multiple Verifiers via Pareto Rankings

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become an important approach to improving the reasoning capabilities of large language models. In tasks such as code generation and instruction following, multiple verifiers evaluate whether a response satisfies different requirements. Their outcomes provide detailed feedback about partial success, yet this feedback is commonly aggregated into a scalar reward for policy optimization. Rewarding only complete success discards distinctions among partial solutions, while counts and weighted sums impose comparisons between responses that satisfy different subsets of requirements. We introduce Partial-Order Likelihood Policy Optimization (POLPO), which learns from the comparisons established by verification without first constructing an aggregate reward. Pareto dominance identifies responses that satisfy additional requirements without losing any already satisfied, while leaving responses with conflicting outcomes incomparable. POLPO extends the listwise likelihood underlying Direct Preference Optimization by maximizing the total probability of all complete rankings consistent with these partial orders. This objective provides supervision from verified improvements even before complete solutions are available, without requiring a prescribed ordering of incomparable responses. Experiments on controlled program synthesis and code generation demonstrate improved learning efficiency and higher rates of complete requirement satisfaction compared with scalar-reward training. These results support partial-order likelihood as an effective approach to learning from multiple verifiers.

Primary area:
reinforcement learning
Keywords:
Large Language Models, Reinforcement Learning from Verifiable Rewards, Group Relative Policy Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_B7ejOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Partial-Order Likelihood Policy Optimization (POLPO), a method for learning from multiple binary verifiers in RLVR. Rather than aggregating verifier outcomes into a scalar reward, POLPO constructs Pareto dominance relations among sampled responses and maximizes the probability of rankings consistent with these relations under a Plackett–Luce model parameterized by DPO-style implicit rewards.

Critical strengths

1 The paper provides mathematical derivations and evaluates the approach on controlled program synthesis, code generation, and multi-constraint instruction following. The results are generally positive.

2 The paper addresses an important limitation of scalar reward aggregation in multi-verifier RLVR. The distinction between verified Pareto improvements and unspecified trade-offs is clearly motivated. 3 The partial-order likelihood is well defined, recovers pairwise DPO as a special case, and is accompanied by gradient derivations and an exact dynamic programming implementation.

Critical weaknesses

1 The marginalized Plackett–Luce likelihood for general partial orders is established in prior work, while GraphDPO already combines DPO-style scores with structured preference graphs. The main distinction of POLPO relative to GraphDPO is its joint likelihood over the entire partial order, rather than an aggregation of local graph-based choice losses.

2 The controlled synthesis and coding experiments use a single training seed, and the paper does not report multi-seed variability for the main benchmark results. Several improvements over the strongest competing methods are relatively modest, particularly on external coding benchmarks and MathIF.

Questions for authors
  1. How frequently does POLPO receive informative joint partial-order supervision? Could the authors report the fraction of response groups containing strict dominance relations and characterize the distribution of partial-order structures across datasets and training stages? In particular, how frequently do the sampled groups induce genuinely different objectives under POLPO and GraphDPO?

  2. Can the authors clarify the missing APPS baseline results? Table 7 specifies learning rates for DPO–Pareto and both REINFORCE++ variants on APPS, but Table 2 does not report their performance. Were these experiments completed, and can the results be provided?

  3. Does the advantage over structured preference baselines extend to instruction following? Could the authors provide GraphDPO or DPO–Pareto comparisons on representative multi-instruction-following tasks, or clarify why these direct baselines were excluded?

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors(1/2)

Thank you for your constructive feedback. Please see the Overall Response for an overview of the revisions.

Weakness: What does POLPO contribute beyond existing partial-order likelihoods and GraphDPO?

We agree that the partial-order Plackett–Luce likelihood is established and that GraphDPO combines structured preference graphs with DPO-style scores. POLPO’s contribution combines verifier-derived Pareto supervision with joint likelihood optimization for policy learning.

First, GraphDPO’s reported APPS experiments construct ordered preference layers from scalar test pass rates (Liu et al., 2026, Appendix A.3). POLPO instead constructs a Pareto partial order directly from individual verifier outcomes. Thus, responses with conflicting outcomes remain incomparable even when their pass rates differ.

Second, GraphDPO’s objective is defined for general preference DAGs. We therefore adapt it to the same verifier-derived Pareto graph in our experiments. Under this shared comparison relation, POLPO optimizes the exact joint likelihood of the entire partial order, whereas GraphDPO aggregates local neighborhood losses.

To examine the contribution of the joint objective, we compare POLPO with GraphDPO and DPO–Pareto using the same Pareto relation. On APPS, POLPO improves All-pass / average pass to 25.90% / 42.67%, compared with 21.80% / 37.38% for GraphDPO and 21.70% / 37.57% for DPO–Pareto (Table 2). After AceCode training, POLPO exceeds both baselines on HumanEval, HumanEval+, MBPP, and MBPP+:

MethodHumanEvalHumanEval+MBPPMBPP+
GraphDPO75.6168.9071.1662.17
DPO–Pareto75.6168.9071.1661.90
POLPO78.0572.5674.8764.29

External pass@1 (%) after AceCode training (Table 3).

POLPO also has the highest mean on both MBPP model sizes over three training seeds, exceeding both GRPO variants and the structured baselines shown in the Overall Response (Table 1). These comparisons support the value of modeling Pareto comparisons jointly.

To examine the contribution of Pareto supervision, we added POLPO–Rate and POLPO–All, which retain the partial-order likelihood but use pass-rate and all-pass comparisons. POLPO has a higher mean than both on MBPP. In controlled synthesis at M=60M=60, POLPO reaches 0.9 complete-success probability for all three targets in all five seeds, using approximately 96,000–100,000 programs on average. POLPO–Rate requires approximately 136,000 and 151,000 programs for the first two targets and reaches the threshold in three of five seeds for the third; POLPO–All does not reach the threshold within the budget for any of the three targets (Table 1, Figure 3, and Table 17).

Together, these comparisons support the combination of verifier-derived Pareto ordering and joint likelihood optimization.

Weakness: Are the gains stable across training seeds?

We added three-seed MBPP, IFM5, and MulDimIF results and five-seed controlled-synthesis results. As shown in the Overall Response, POLPO has the highest mean on both MBPP model sizes and the highest overall mean on IFM5 and MulDimIF, including comparisons with GRPO–All and GRPO–Rate (Figure 3 and Tables 1, 4, and 5).

Question: How often is supervision informative in coding and instruction following, and when do the objectives differ? We added signal-availability and objective-structure analyses for both task families (Figures 7–10 and Tables 18–20). For the same group, every all-pass reward difference implies a Pareto comparison, so POLPO can provide a signal whenever GRPO–All does.

In coding, POLPO’s signal shares among unsolved groups at the end of the plotted period are approximately 73% on APPS and 62% on AceCode, versus at most about 52% for GRPO–All in the reported runs (Figure 8). On shared initial IFM5 and MulDimIF rollouts, POLPO matches GRPO–Rate’s signal fractions of 47.7% and 74.8%, and exceeds GRPO–All’s 17.7% and 43.0% (Table 18). Later comparisons use each method’s own rollouts; GRPO–Rate can signal on additional groups with conflicting outcomes.

Among groups containing dominance, POLPO and GraphDPO define different objective functions, beyond a constant factor, on approximately 80–90% of coding groups and 92.8–97.8% of the IFM5 and MulDimIF snapshots (Figure 10 and Table 20). Many differences arise from tied verifier outcomes. The joint-versus-local distinction therefore occurs frequently in sampled groups in both task families.

Question: What are the missing APPS baseline results?

We added the omitted results:

MethodAll-passAverage pass
DPO–Pareto21.7037.57
REINFORCE++–All24.3038.75
REINFORCE++–Rate23.2038.36
GraphDPO21.8037.38
POLPO25.9042.67

APPS Introductory test results (%) after two training epochs (Table 2).

AuthorsOpenReview ↗

Rebuttal by Authors(2/2)

Question 3: Does the advantage over structured baselines extend to instruction following?

We added DPO–Pareto and GraphDPO comparisons on IFM5, MulDimIF, and MathIF.

On IFM5 and MulDimIF, POLPO achieves the highest overall mean over three training seeds:

MethodIFM5MulDimIF
GRPO–All56.4 ± 0.888.3 ± 0.2
GRPO–Rate57.1 ± 1.789.1 ± 1.0
DPO–Pareto61.0 ± 0.689.0 ± 0.4
GraphDPO59.6 ± 2.289.7 ± 0.8
POLPO61.3 ± 0.290.1 ± 0.7

Overall complete-constraint satisfaction (%), mean ± standard deviation over three training seeds (Tables 4 and 5).

The newly added thinking-enabled MathIF results also favor POLPO:

MethodHAccMath AccJoint
GRPO–Rate70.538.830.7
GraphDPO56.430.024.8
DPO–Pareto72.640.231.9
POLPO78.140.233.8

MathIF results (%) with thinking enabled (Table 6). HAcc measures complete instruction adherence; Joint requires both mathematical correctness and complete instruction adherence.

POLPO matches DPO–Pareto’s mathematical accuracy while improving HAcc from 72.6% to 78.1% and Joint from 31.9% to 33.8%. It also exceeds GraphDPO on all three metrics. Together, these results support POLPO’s effectiveness against structured baselines across instruction-following tasks, including tasks that require mathematical correctness.

AuthorsOpenReview ↗

Follow-up on Our Response to Reviewer B7ej

Dear Reviewer B7ej,

Thank you again for your constructive feedback. We have revised the manuscript and responded to your comments, including additional experiments and analyses.

As the discussion period ends on November 18 (AoE), we would greatly appreciate it if you could share your thoughts on whether our response and revisions address your concerns. If any questions remain, we would be happy to provide further clarification.

Thank you for your time and consideration.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal Responses

Dear Reviewer B7eJ,

Thank you very much for your time and effort in reviewing our paper and for your constructive feedback.

We would like to kindly follow up on our previous message. We have provided responses to the concerns raised in your review in our rebuttal.

As the author-reviewer discussion period ends on November 18 (AoE), we would greatly appreciate it if you could review our responses and let us know if any concerns remain. We would be happy to provide further clarification if needed.

Thank you again for your valuable time and consideration.

Reviewer_NQ9UOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses reinforcement learning with verifiable rewards (RLVR) in settings where each generated response is evaluated by multiple binary verifiers such as unit test suites in code generation or multi-constraint checks in instruction following. The authors observe that standard scalar reward aggregations face a fundamental tension, a all-pass reward provides no gradient when all sampled responses in a group are not entirely correct. it proposes POLPO, a policy optimization method for tasks with multiple binary verifiers. It uses Pareto dominance to compare sampled responses and maximizes the likelihood of the resulting partial order. Experiments cover code generation, instruction following, and a controlled FizzBuzz setting. POLPO generally outperforms scalar-reward and local preference baselines on complete requirement satisfaction.

Critical strengths
  • POLPO provides a principled way to learn from partial progress without imposing arbitrary trade-offs between requirements. Its likelihood objective and dynamic programming method are clearly derived.
  • The evaluation includes controlled diagnostics and Pareto-based baselines. POLPO’s gains over DPO-Pareto and GraphDPO support the value of modeling the joint partial order.
  • the paper is good-writing and easy to read.
Critical weaknesses
  • With large KK and N=8N=8, many groups may contain no strict dominance pairs and provide no learning signal. The paper does not report how often this occurs. Larger groups could help, but increase the dynamic program’s cost.
  • the method changes both the reward structure and the training objective relative to GRPO. But during training, the log ratio also depends on the response length and shift from the ref policy. Pareto-based baselines help isolate this effect on MBPP and APPS, but are missing from AceCode and IF results. The ranking coefficients could also be tested within a clipped GRPO-style objective.
Questions for authors
  • Could the authors isolate the partial-order objective from the fixed-reference DPO parameterization, for example by using the ranking coefficients in a clipped GRPO-style update? and how does DPO-Pareto and GraphDPO perform on IFM5 and MulDimIF?
  • how often is RVR_V empty during training, and how runtime scale when NN increase from 8 to 16?
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for your constructive feedback. Please see the Overall Response for an overview of the revisions.

Weakness and question: Can the ranking coefficients be used in a clipped GRPO update?

We added GRPO–Pareto, which uses POLPO’s ranking-gradient coefficients evaluated at equal scores as advantages in a clipped GRPO update. On MBPP, GRPO–Pareto achieves 40.53 ± 0.42% and 66.20 ± 0.35% All-pass for 0.6B and 4B, while POLPO achieves 44.73 ± 0.61% and 67.93 ± 0.83% (Table 1). This supports the value of the partial-order likelihood when both updates use Pareto relations. GRPO–Pareto uses equal-score coefficients rather than the full, dynamically changing POLPO coefficients.

DPO–Pareto and GraphDPO use the same fixed reference and sequence-summed log-probability ratios as POLPO, providing controls for score parameterization when comparing objectives (Section D.2.3).

Weakness and question: Does the advantage over Pareto-based baselines extend across coding and instruction following?

We added the missing DPO–Pareto result on APPS and both DPO–Pareto and GraphDPO on AceCode, IFM5, MulDimIF, and thinking-enabled MathIF.

On APPS, POLPO achieves 25.90% All-pass and 42.67% average pass, compared with 21.80% / 37.38% for GraphDPO and 21.70% / 37.57% for DPO–Pareto (Table 2). After AceCode training, POLPO also outperforms both structured baselines on all four external coding evaluations:

MethodHumanEvalHumanEval+MBPPMBPP+
GraphDPO75.6168.9071.1662.17
DPO–Pareto75.6168.9071.1661.90
POLPO78.0572.5674.8764.29

External pass@1 (%) after AceCode training (Table 3).

On instruction following, POLPO achieves the highest overall mean on both IFM5 and MulDimIF over three training seeds:

MethodIFM5MulDimIF
GRPO–All56.4 ± 0.888.3 ± 0.2
GRPO–Rate57.1 ± 1.789.1 ± 1.0
DPO–Pareto61.0 ± 0.689.0 ± 0.4
GraphDPO59.6 ± 2.289.7 ± 0.8
POLPO61.3 ± 0.290.1 ± 0.7

Overall complete-constraint satisfaction (%), mean ± standard deviation over three training seeds (Tables 4 and 5).

The newly added thinking-enabled MathIF results also favor POLPO:

MethodHAccMath AccJoint
GRPO–Rate70.538.830.7
GraphDPO56.430.024.8
DPO–Pareto72.640.231.9
POLPO78.140.233.8

MathIF results (%) with thinking enabled (Table 6). HAcc measures complete instruction adherence; Joint requires both mathematical correctness and complete instruction adherence.

POLPO matches DPO–Pareto’s mathematical accuracy while improving HAcc from 72.6% to 78.1% and Joint from 31.9% to 33.8%. Together with the coding and other instruction-following results, these comparisons support the value of modeling Pareto comparisons jointly across task families.

Weakness and question: How often is the relation empty relative to GRPO’s signal in coding and instruction following?

We added rollout analyses for both task families (Figures 7–8 and Table 18). On the same group, GRPO–All’s reward signal implies a strict Pareto comparison, so POLPO can signal whenever GRPO–All does. Strict Pareto dominance also implies different pass rates, so GRPO–Rate can signal on every such group and on some additional groups.

In coding, POLPO’s signal shares among unsolved groups at the end of the plotted period are approximately 73% on APPS and 62% on AceCode, versus at most about 52% for GRPO–All in the reported runs (Figure 8). On shared initial IFM5 and MulDimIF rollouts, POLPO matches GRPO–Rate’s signal fractions of 47.7% and 74.8%, and exceeds GRPO–All’s 17.7% and 43.0% (Table 18). GRPO–Rate can have a higher signal fraction later because it also orders conflicting outcomes; later rollouts come from each method’s own policy.

Question: How does runtime scale from N=8N=8 to N=16N=16 in coding and instruction following?

We added per-group coding timings, loss-overhead measurements for both task families, and a direct IFM5 update benchmark (Figure 4 and Tables 7–9). For informative coding groups, likelihood-and-gradient time increases from approximately 2.2–2.6 ms per group at N=8N=8 to 4.1–4.7 ms at N=16N=16. The additional loss time over GRPO is 0.03–0.07% of logged update time at N=8N=8 and 0.04–0.08% at N=16N=16 (Table 7).

For instruction following, the corresponding loss-overhead estimates are 0.05% on IFM5 and 0.12% on MulDimIF (Table 8). A separate direct IFM5 benchmark at N=16N=16 gives a 1.07× update-time ratio to GRPO (Table 9). Thus, measured overhead remains modest in both task families at the tested group sizes.

AuthorsOpenReview ↗

Follow-up on Our Response to Reviewer NQ9U

Dear Reviewer NQ9U,

Thank you again for your constructive feedback. We have revised the manuscript and responded to your comments, including additional experiments and analyses.

As the discussion period ends on November 18 (AoE), we would greatly appreciate it if you could share your thoughts on whether our response and revisions address your concerns. If any questions remain, we would be happy to provide further clarification.

Thank you for your time and consideration.

Reviewer_NQ9UOpenReview ↗

Reply to rebuttal

Thank authors for the detailed rebuttal, my main concerns have been resolved. I will maintain my positive recommendation.

Reviewer_rq2JOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to train language models using feedback from multiple binary verifiers, with the goal of satisfying all requirements simultaneously. The authors argue that all-pass rewards discard useful distinctions among partially successful responses, while pass-rate rewards and weighted sums impose additional preferences between responses that satisfy different subsets of requirements.

The proposed method, Partial-Order Likelihood Policy Optimization (POLPO), constructs a partial order using Pareto dominance. It parameterizes a Plackett–Luce ranking model with DPO-style policy-to-reference log-probability ratios and maximizes the total probability of all complete rankings consistent with the observed partial order. The paper provides gradient derivations, exact dynamic programming procedures, and a decomposition of partial orders that can reduce computation.

The method is evaluated on controlled program synthesis, code generation, multi-constraint instruction following, and MathIF. The main contribution is the application of an established partial-order ranking likelihood to online policy learning from multiple verifiers, rather than a new ranking probability model.

Critical strengths

The objective is well matched to the supervision structure, and the contribution is positioned honestly. POLPO uses only dominance relations supported by the verifiers and does not force incomparable responses to have equal scores. The example in which an unrelated response cancels from the likelihood illustrates a concrete and desirable property. The authors acknowledge prior work on partial-order Plackett–Luce likelihoods and explain the distinction from pairwise DPO and GraphDPO. This makes the contribution both credible and technically well motivated.

The experiments cover several task families and include relevant structured-comparison baselines. The MBPP experiments compare scalar rewards, per-verifier normalization, Pareto-based pairwise DPO, and GraphDPO. This is substantially more informative than comparing only against all-pass GRPO. In Table 1, POLPO improves All-pass accuracy over GraphDPO from 42.8% to 45.4% for the smaller model and from 67.6% to 71.2% for the larger model. The consistent improvements on IFM5 and MulDimIF provide additional evidence that the approach is useful beyond code generation.

Critical weaknesses

The reported improvements lack adequate characterization of training variability and evaluation uncertainty. The coding and controlled synthesis experiments explicitly use a single training seed, and IFM5 also reports one training run per configuration. Several improvements are small in absolute terms. For example, the HumanEval+ improvement from 70.73% to 72.56% corresponds to approximately three additional solved problems out of 164. The MathIF Joint improvement from 19.8% to 21.4% corresponds to approximately seven examples. Without multiple training seeds or paired evaluation analyses, it is difficult to determine how robust these gains are. The results are encouraging, but the evidence does not yet support strong claims of consistent superiority.

The practical applicability and efficiency of the method are insufficiently measured. Groups without strict dominance relations provide no learning signal. This may become a substantial limitation when responses satisfy conflicting subsets of many requirements. Meanwhile, exact likelihood computation remains exponential in the worst case, and the implementation limits components requiring dynamic programming to 16 candidates. The paper does not report the fraction of informative groups during training, the distribution of partial-order structures, or the time and memory overhead of likelihood computation. Consequently, the sample savings demonstrated in the controlled task do not establish practical training-efficiency gains.

Questions for authors

Clarification and supporting evidence: Do the improvements on MBPP, IFM5, and MathIF persist across multiple training seeds? Can you provide paired results on the same evaluation examples to characterize the reliability of the smaller gains?

Supporting evidence: How do the fraction of comparison-free groups, the number of dominance relations per group, and the largest incomparability component change during training? What fraction of total training time is spent computing the exact likelihood?

Contribution attribution: What happens if the same Plackett–Luce likelihood and training configuration are retained, but the Pareto partial order is replaced with ordered groups induced by all-pass or pass-rate rewards? This comparison would help separate the contribution of partial-order supervision from the contribution of changing the optimization objective.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors(1/2)

Thank you for your constructive feedback. Please see the Overall Response for an overview of the revisions.

Weakness and question: Do the benchmark gains persist across training seeds and evaluation examples?

We added three-seed results for MBPP, IFM5, and MulDimIF. On MBPP, POLPO has the highest mean All-pass accuracy at both model sizes:

MethodQwen3-0.6BQwen3-4B
GRPO–All41.47 ± 0.9566.53 ± 0.31
GRPO–Rate42.47 ± 0.6466.60 ± 0.53
DPO–Pareto39.73 ± 2.3066.27 ± 0.76
GraphDPO41.27 ± 1.4265.87 ± 0.99
POLPO44.73 ± 0.6167.93 ± 0.83

All-pass accuracy (%), mean ± standard deviation over three training seeds (Table 1).

POLPO also has the highest overall mean on IFM5 and MulDimIF, exceeding both GRPO variants and the structured baselines shown in the Overall Response (Tables 4 and 5). These results strengthen the evidence across training seeds. A paired analysis on the same evaluation examples is not yet available.

Weakness and question: How often do coding and instruction-following groups provide a signal relative to GRPO?

We added rollout analyses for both task families (Figures 7–8 and Table 18). For the same group of binary verifier outcomes, an all-pass reward difference implies a strict Pareto comparison: a fully successful response dominates any response that fails a requirement. Thus, POLPO provides a comparison-based signal whenever GRPO–All has an all-pass reward difference, and can also learn when no response is fully successful.

In coding, POLPO provides signals for approximately 73% of unsolved APPS groups and 62% of unsolved AceCode groups at the end of the plotted period; GRPO–All’s share is at most about 52% in these runs (Figure 8). On shared initial instruction-following rollouts, POLPO and GRPO–Rate provide signals for 47.7% of IFM5 groups and 74.8% of MulDimIF groups, compared with 17.7% and 43.0% for GRPO–All (Table 18).

GRPO–Rate can signal on additional groups because it orders some responses with conflicting verifier outcomes. Its signal fraction is higher later in the APPS and AceCode runs; POLPO retains verified improvements without prescribing those trade-offs. The later comparisons use each method’s own rollouts.

Question: How do dominance counts and incomparability components affect practicality?

We added analyses of partial-order structures in coding and instruction following, dominance-pair counts during instruction-following training, and largest-component statistics at initial and final policy snapshots (Figures 9 and 11; Tables 19 and 21). Groups with distinct outcomes but no dominance pair account for at most about 1% of coding groups and 1.3% of the instruction-following snapshots.

Among instruction-following groups containing dominance, the mean pair count changes from 42.8 to 36.6 on IFM5 and from 42.7 to 31.4 on MulDimIF. The mean largest-component size changes from 8.9 to 10.7 and from 10.9 to 11.6, respectively. The two component measurements are snapshots of the groups informative at each point.

Together with the coding and instruction-following timings below, these results support the practicality of the observed structures at our experimental group sizes.

Weakness and question: What are the time and memory costs in coding and instruction following?

We added loss-replay measurements for both task families and a direct IFM5 update benchmark (Section C.4, Tables 7–9).

MeasurementCodingInstruction following
Additional loss time over GRPO0.03–0.07% at N=8N=8;0.04–0.08% at N=16N=160.05% on IFM5; 0.12% on MulDimIF
Direct update benchmark—At N=16N=16: POLPO 6.16 s vs. GRPO 5.77 s

Loss differences are compared with logged update times (Tables 7–8). The separate direct benchmark uses IFM5 rollouts and a 0.6B model (Table 9).

Across the reported coding and instruction-following settings, the measured POLPO loss time—including Pareto-relation construction and loss forward/backward computation—is approximately 0.05–0.16% of the logged update time, calculated from Tables 7–8. In the direct N=16N=16 benchmark, the update-time ratio is 1.07× and measured GPU memory is 18.3 GB for POLPO versus 19.5 GB for GRPO. Exact computation remains exponential in the worst case, but the measured cost is modest in both task families.

AuthorsOpenReview ↗

Rebuttal by Authors(2/2)

Question: Does Pareto supervision help when the likelihood objective is held fixed?

We added POLPO–Rate and POLPO–All to both MBPP and controlled synthesis. On MBPP, POLPO–Rate uses the same likelihood and hyperparameters as POLPO but orders responses by pass rate. POLPO improves All-pass from 42.00% to 44.73% for 0.6B and from 67.40% to 67.93% for 4B (Table 1).

In controlled synthesis, adding these two variants expands the comparison from three to five methods. At M=60M=60, POLPO reaches 0.9 complete-success probability for all three targets in all five seeds, using approximately 96,000–100,000 programs on average. POLPO–Rate requires more programs for the first two targets and reaches the threshold in three of five seeds for the third; POLPO–All does not reach it within the budget for these targets (Figure 3 and Table 17). These results support Pareto supervision with the same likelihood objective.

AuthorsOpenReview ↗

Follow-up on Our Response to Reviewer rq2J

Dear Reviewer rq2J,

Thank you again for your constructive feedback. We have revised the manuscript and responded to your comments, including additional experiments and analyses.

As the discussion period ends on November 18 (AoE), we would greatly appreciate it if you could share your thoughts on whether our response and revisions address your concerns. If any questions remain, we would be happy to provide further clarification.

Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response

We thank the reviewers for their constructive feedback. We revised the manuscript and added experiments and analyses that strengthen the evidence for POLPO.

1. Effectiveness across training seeds. We expanded MBPP, IFM5, and MulDimIF to three training seeds. POLPO achieves the highest reported mean in all four settings:

MethodMBPP 0.6BMBPP 4BIFM5MulDimIF
GRPO–All41.47 ± 0.9566.53 ± 0.3156.4 ± 0.888.3 ± 0.2
GRPO–Rate42.47 ± 0.6466.60 ± 0.5357.1 ± 1.789.1 ± 1.0
DPO–Pareto39.73 ± 2.3066.27 ± 0.7661.0 ± 0.689.0 ± 0.4
GraphDPO41.27 ± 1.4265.87 ± 0.9959.6 ± 2.289.7 ± 0.8
POLPO44.73 ± 0.6167.93 ± 0.8361.3 ± 0.290.1 ± 0.7

Results (%), mean ± standard deviation over three training seeds. MBPP reports All-pass accuracy; IFM5 and MulDimIF report overall complete-constraint satisfaction (Tables 1, 4, and 5).

We also expanded controlled synthesis to five training seeds (Figure 3).

2. Contributions of Pareto supervision and the joint likelihood. We added POLPO–Rate and POLPO–All to controlled synthesis, expanding the comparison from three to five methods. Both variants retain the likelihood objective while replacing Pareto comparisons with pass-rate or all-pass comparisons.

In the M=60M=60 setting, POLPO reaches 0.9 complete-success probability for all three targets in all five seeds, using approximately 96,000–100,000 programs on average. POLPO–Rate requires approximately 136,000 and 151,000 programs for the first two targets and reaches the threshold in three of five seeds for the third. POLPO–All does not reach the threshold within the budget for any of the three targets. On MBPP, POLPO also achieves a higher mean All-pass score than both variants at both model sizes. These results support the value of Pareto supervision when the likelihood objective is retained (Figure 3 and Tables 1 and 17).

3. Joint likelihood optimization against structured baselines. We added the missing DPO–Pareto and REINFORCE++ results on APPS, and DPO–Pareto and GraphDPO comparisons on AceCode, IFM5, and MulDimIF.

MethodAPPS All-passAPPS average passHumanEval+ after AceCode
DPO–Pareto21.7037.5768.90
GraphDPO21.8037.3868.90
POLPO25.9042.6772.56

Results (%) from Tables 2 and 3.

POLPO also exceeds both structured baselines on the other three external coding evaluations. Because GraphDPO and DPO–Pareto use the same Pareto relation as POLPO, these comparisons support the value of optimizing the joint likelihood of the entire partial order, beyond using Pareto comparisons alone (Tables 2–5).

4. Learning signal in coding and instruction following. We added rollout analyses for both task families (Figures 7–11 and Tables 18–21). On the same group, an all-pass reward difference always implies a Pareto comparison, so POLPO can learn whenever GRPO–All receives an all-pass reward signal. In coding, POLPO provides signals for approximately 73% of unsolved APPS groups and 62% of unsolved AceCode groups at the end of the plotted period, above GRPO–All’s reported shares. On shared initial IFM5 and MulDimIF rollouts, POLPO matches GRPO–Rate’s signal fractions and exceeds GRPO–All (Figure 8 and Table 18).

5. Computational cost in coding and instruction following. We added loss-replay measurements for both task families and a direct update benchmark. The additional loss time over GRPO is 0.03–0.08% of the logged update time in the coding measurements and 0.03–0.12% in instruction following. In a separate IFM5 update benchmark at N=16N=16, POLPO takes 1.07× the GRPO update time (Tables 7–9).

6. MathIF correction and new thinking-enabled results. An implementation error affected the submission’s thinking-disabled evaluation, and we corrected the reported values. POLPO remains best on Joint. We also added the thinking-enabled evaluation and structured baselines:

MethodThinking disabled: JointThinking enabled: HAccThinking enabled: Math AccThinking enabled: Joint
GRPO–All20.241.927.122.6
GRPO–Rate19.370.538.830.7
GraphDPO–56.430.024.8
DPO–Pareto–72.640.231.9
POLPO21.478.140.233.8

MathIF results (%) on 420 examples (Table 6). HAcc measures complete instruction adherence; Joint requires both mathematical correctness and complete instruction adherence. Dashes indicate results not reported for that mode.

For thinking disabled, the correction changes GRPO–All’s Joint score from 19.8% to 20.2%; POLPO remains at 21.4%. Other evaluation values, including HAcc and SAcc, were also corrected. With thinking enabled, POLPO matches DPO–Pareto’s mathematical accuracy while improving HAcc and Joint. We apologize for the error.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.