acceptodds
Under review as a conference paper at ICLR 2027

EAPO: Entropy-Driven Adaptive Positive-Negative Sample Weighting for Policy Optimization in Open-Ended QA

Abstract

Large Reasoning Models are typically trained via reinforcement learning from verifiable rewards (RLVR). However, existing approaches adopt fixed weights for positive and negative samples, and the conclusions hardly generalize to open-ended question answering (QA). In this paper, we systematically investigate the roles of positive and negative samples in reinforcement learning for open-ended QA. We propose a reward-mean-based strategy for distinguishing positive from negative samples, and observe that negative samples predominantly govern response diversity and the performance upper bound, whereas positive samples primarily determine response quality and convergence stability. Building on these observations, we propose EAPO, an Entropy-driven Adaptive Policy Optimization method that adaptively computes the weighting coefficients of positive samples based on the ratio of the current policy entropy to the initial entropy. During the entropy-decreasing phase, the weight assigned to positive samples is reduced to preserve exploration, whereas during the entropy-increasing phase it is amplified to reinforce stability, thereby mitigating entropy collapse. Experiments on two publicly available open-ended medical QA datasets demonstrate that EAPO consistently and substantially outperforms fixed-weight baselines in both response diversity and stability.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, optimization methods, open-domain QA

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YU54OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EAPO, an entropy-driven policy optimization method that adaptively adjusts positive sample weights based on the ratio of current to initial policy entropy. The method aims to balance exploration and training stability in open-ended QA. Experiments on two medical QA datasets show competitive performance and reduced sensitivity to initial weight settings compared with fixed-weight baselines.

Critical strengths

Simple and intuitive design. EAPO provides a straightforward and easy-to-understand mechanism for adjusting positive sample weights based on policy entropy, making it easy to implement and interpret.

Critical weaknesses
  1. Limited novelty and missing baselines. Asymmetric weighting of positive and negative advantages is not new. The proposed entropy-based adjustment is relatively simple, and comparisons with relevant methods such as SAPO are missing, making its novelty and effectiveness unclear.

  2. Insufficient motivation for medical QA. The paper does not clearly explain why open-ended medical QA requires a specialized approach or how it differs from other domains. The lack of cross-domain evaluation further limits the generalizability of the findings.

Questions for authors
  1. Novelty: How does EAPO fundamentally differ from existing methods such as SAPO that already consider asymmetric weighting of positive and negative advantages?

  2. Baseline comparison:Could the authors provide comparisons with stronger adaptive policy optimization methods, particularly SAPO, to better demonstrate the effectiveness of EAPO?

  3. Medical QA motivation:What makes open-ended medical QA particularly suitable for EAPO? Are there any domain-specific challenges that motivate this design, and does the method generalize to other open-ended tasks?

Reviewer_pRzSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the effects of positive and negative samples in reinforcement learning (GRPO-style) on two open-ended medical QA datasets. and proposed EAPO (Entropy-driven adaptive policy optimization). EAPO adaptively adjusts the positive sample weight using the ratio of current policy entropy to its initial value and keeps the negative sample weight fixed. During the entropy-decreasing phase, the weight assigned to positive samples is reduced to preserve exploration, whereas during the entropy increasing phase it is amplified to reinforce stability. Experiments show that EAPO achieves promising results.

Critical strengths
  • EAPO is a lightweight, and reasonable mechanism
Critical weaknesses
  • limited novelty and without in-depth analysis why this intuitive method would be preferred to established entropy-control methods.
  • results do not show EAPO improves from existing methods meaningfully
Questions for authors
  • Can you compare EAPO with other entropy regularization methods?
  • Can you provide results from multiple independent training runs (>= 3 seeds per method) and report confidence intervals?
Reviewer_vUBPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a method named EAPO for open-ended QA. The main observation is that the weights assigned to positive and negative samples can affect policy entropy differently during training. Based on this, the authors introduce a dynamic weighting strategy to keep the entropy more stable. The paper provides several experiments to support this claim.

Critical strengths
  1. The different roles of positive and negative sample weighting are interesting. The experiments show that they can affect policy entropy in different ways during training, and this observation is useful.
  2. The proposed method uses dynamic weighting to make the entropy trajectory more stable during training.
Critical weaknesses
  1. There is no clear reason why the initial entropy should be used as the anchor, especially in online RL where both the policy and rollout distribution keep changing. It is also unclear why policy entropy should correspond to useful semantic or reasoning diversity.
  2. The empirical benefit of EAPO is weak. The reported results do not show a clear advantage of the proposed method over strong fixed-weight baselines.
Questions for authors
  1. Why is the initial entropy used as the reference level for the controller?
  2. Since the motivation relies heavily on entropy as a measure of exploration, can the authors provide a direct measure of semantic or reasoning diversity and show that it actually tracks the reported entropy changes?
  3. What is the main empirical advantage of the adaptive method beyond the reduced initialization sensitivity reported in three of four comparisons?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.