- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Hint-Guided Diversified Policy Optimization (HDPO), which trains an LLM to propose several candidate solution outlines, select one, and then develop a detailed solution. Training consists of a cold-start SFT stage using strong-teacher trajectories filtered for answer correctness and candidate-selection reliability, followed by GRPO with additional diversity and reliability rewards. Diversity is measured through embedding similarity, while reliability is approximated by the entropy ranking of candidate outlines. The evaluation covers three backbones and nine reasoning benchmarks. The paper reports substantial pass@3 improvements over corresponding GRPO baselines, alongside component ablations, candidate-selection analyses, and experiments with alternative optimization algorithms.
Critical strengths
Substantial and consistent reported improvements. HDPO improves the nine-benchmark average over the corresponding GRPO baseline by 7.27, 8.26, and 7.50 percentage points across the three backbones. Improvements extend beyond mathematics to knowledge-intensive benchmarks, making the empirical results worth investigating.
Critical weaknesses
-
The main comparisons do not clearly isolate the proposed contribution. HDPO receives SFT on 83,279 correctness- and reliability-filtered trajectories generated by a strong teacher before reinforcement learning. It is unclear whether the GRPO baselines receive the same data and initialization. Consequently, the main gains may reflect teacher knowledge, data filtering, additional SFT, or the structured reasoning format, rather than primarily the proposed rewards. The separate reward ablations are useful, but do not replace a control that starts from the same SFT checkpoint and uses only correctness and format rewards under a comparable training budget.
-
The reward design admits shortcuts that are not sufficiently examined. A single candidate is permitted, automatically receives the highest reliability rank, and can receive a group-derived diversity reference. This creates a possible way to obtain auxiliary rewards without meaningful exploration or selection. Moreover, a correct final answer does not establish that the unselected outlines are valid: the model could include unrelated or ineffective outlines to increase embedding diversity while solving through one useful route. Low token entropy also measures confidence rather than correctness and may depend on familiar wording or candidate order. The correlation and selection experiments provide encouraging evidence, but do not resolve these specific concerns.
-
Important baselines and resource controls are missing. Beyond the matched-SFT GRPO control, DRA-GRPO is directly relevant because it incorporates embedding-based semantic diversity into GRPO rewards. Comparing against such a method would help establish the value of rewarding diversity within candidate outlines. The Self-Consistency comparison also needs a clearer evaluation protocol, since majority-vote accuracy and pass@3 assess different outcomes. Finally, the teacher-generation, repeated filtering, SFT, and auxiliary-model costs are not sufficiently quantified to establish an advantage under comparable total resources.
There are also two concrete issues with the supporting analysis. Figure 12 attributes failure to the absence of a complex-number approach, although candidate [3] explicitly proposes complex exponentials and roots of unity. This appears to be a selection or execution failure rather than the stated coverage failure. The inference-latency claims also lack sufficient measurement details: a single generated trajectory still requires sequential decoding, and the reported overhead is substantial in relative terms on some tasks.
Questions for authors
-
Do the GRPO baselines use the same 83,279 filtered trajectories and the same SFT initialization as HDPO? What happens when both methods start from an identical SFT checkpoint, with the baseline retaining only correctness and format rewards?
-
How frequently does HDPO generate only one candidate? Are single-candidate cases included in the reported selection accuracy? Can the authors show that the reward scheme encourages genuine multi-candidate exploration and that the additional outlines are valid, useful alternatives?
-
How sensitive is entropy-based candidate ranking to candidate order and wording? Would rankings remain consistent if the same outlines were evaluated independently under a common context?
-
How does HDPO compare with a diversity-aware baseline such as DRA-GRPO under matched initialization and resources? Please also clarify the evaluation protocol for Self-Consistency and report the full data-construction and training costs.
-
Could the authors provide stronger comparisons with directly related diversity-aware and multi-path reasoning methods, particularly DRA-GRPO and ToTRL, under comparable initialization, training data, and computational budgets? How much of the reported improvement remains over a baseline that uses the same structured SFT checkpoint and propose-select-think format but only correctness and format rewards?