- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes using Gibbs-Gap as the objective function, transforming the bias caused by policy mismatch into terms in the performance improvement upper bound, thereby avoiding the use of importance weights. The paper conducts experiments on mathematical reasoning tasks and demonstrates that the proposed method outperforms GRPO and AGRO across multiple benchmarks.
Critical strengths
-
The paper uses a Gibbs-Gap-based contrastive loss to avoid importance weights, addressing the instability associated with importance weighting.
-
The paper provides detailed theoretical analyses, explaining why Gibbs-Gap is effective and characterizing its performance guarantees.
-
The experimental results are promising, with the proposed method outperforming the baselines.
Critical weaknesses
-
Some theoretical results may have limited practical significance. For example, the constant $ C_x $ in Equation 5 may not exist, rendering the upper bound meaningless. Moreover, Equation 9 and related results only guarantee that $ J(\pi_\theta) $ is not much lower than $ J_K(\pi_{\mathrm{beh}}) $. Since they only provide upper bounds on the performance gap, they cannot guarantee that $ J(\pi_\theta) $ will actually improve.
-
In Table 6, GGPO with a fixed reference suffers a severe performance collapse. The authors should analyze the reasons behind this behavior.
-
The experiments are not sufficiently comprehensive. They are limited to small models below 7B parameters, the Qwen3 model family, and mathematical reasoning tasks, making it difficult to draw conclusions about the generalizability of the proposed method.
Questions for authors
None