- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces FACTOR-OPD, a multi-reward reinforcement learning framework that addresses token-level credit assignment and gradient interference in large language models. By leveraging verifier-checked factorial reference panels, a frozen teacher provides controlled token contrasts that are converted into bounded, mean-one weights preserving component advantage signs and means. It then coordinates the resulting policy gradients using parameter-space MGDA. Across mathematical reasoning, code generation, and instruction following benchmarks, FACTOR-OPD consistently outperforms existing methods.
Critical strengths
- Novel Token-Credit Allocation Mechanism: this work Introduces verifier-checked factorial panels and bounded, mean-one allocation to precisely distribute feedback within a response while strictly preserving each component advantage's sign and response-wise mean.
- Parameter-Space Gradient Coordination: the proposed method employs parameter-space Multiple-Gradient Descent Algorithm (MGDA) to combine objective-specific policy gradients, accounting for the model's shared-parameter geometry rather than relying on heuristic credit-space weighting.
- Comprehensive Empirical Superiority: this work outperforms strong baselines across six benchmarks spanning mathematical reasoning, code generation, and instruction following over two model scales.
Critical weaknesses
- Insufficient Implementation Details for Hint Guidance: The paper lacks comprehensive implementation details in Appendix B regarding how the constructed hints meaningfully influence and shift student token probabilities during training.
- Unclear Baseline Formulation in Ablations: In Table 3, the baseline configuration (Uniform allocation + Fixed weights) unexpectedly outperforms certain strong baselines from the main evaluation table, making the formulation and relative superiority of these baselines unclear.
Questions for authors
- Question on Hint Influence and Implementation: The paper provides a high-level pipeline for hint construction in Appendix B, but lacks sufficient implementation details. Could the authors provide a concrete example showing how the high and low hints are generated, explain how the framework ensures these hints meaningfully shift student token probabilities during training, and discuss what happens to model performance if the hints have little or no influence?
- Question on Baseline Formulation in Table 3: In Table 3, the ablation baseline (Uniform allocation + Fixed weights) achieves higher scores than certain strong baselines reported in the main evaluation table. Could the authors clarify how this baseline is formulated and why it surpasses those main results?