- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes FEST, a few-shot demonstration-guided RLVR framework that combines GRPO with semi-online DPO or negative-advantage GRPO. The main idea is to leverage a small number of expert demonstrations through supervised and on-policy learning signals with adaptive weighting. Experiments on mathematical reasoning benchmarks show improvements over several RLVR baselines using only 128 randomly sampled demonstrations.
Critical strengths
-
The paper addresses a relevant practical problem: improving RLVR with limited expert demonstrations. The reported performance gains with only 128 examples are encouraging.
-
The experimental evaluation includes several competitive baselines, different few-shot subsets, and additional results on a 7B model, providing some evidence of the method's effectiveness.
Critical weaknesses
- Limited novelty and insufficient justification of the proposed design.
The method largely combines existing techniques, including semi-online DPO, joint SFT/RL optimization, and adaptive weighting. The distinction from prior work such as SPIN, HPT, and CHORD is not sufficiently compelling. More importantly, the ablations do not convincingly establish why this particular combination is necessary. Simple alternatives, such as GRPO with standard semi-online DPO or explicitly decaying SFT regularization, are not adequately evaluated under matched settings. Consequently, it is unclear whether the proposed components provide meaningful advantages beyond existing approaches.
- Technical concerns and insufficient empirical evidence.
The claimed automatic decay of the supervision weight is not theoretically guaranteed or convincingly demonstrated. On the experimental side, improvements over strong baselines are relatively modest, and matched multi-seed comparisons are lacking. The method also requires additional online rollouts, but computational efficiency is not sufficiently analyzed.
Questions for authors
-
How does FEST compare with standard semi-online DPO using fixed β or simple decaying SFT regularization under identical training and compute budgets?
-
Can the authors provide direct evidence that the effective supervision weight decreases during training, as claimed?
4 comments
Rebuttal to Reviewer f56z (2/2)
W2 (3). Relatively modest improvement over strong baselines.
-
The performance improvement of our method compared to baselines (+1.8% accuracy, FEST-GRPO vs. RL-G in Tab. 2) is on the same level of prior work compared to their baselines. E.g., ReLIFT (+1.7% accuracy on average compared to strongest baseline in their Tab. 1), HPT (+2.2% accuracy on average compared to strongest baseline in Tab. 2; the learning rates for baselines in their Tab. 3 are too low).
-
Our method not only shows performance improvements both on pass@8 and avg@8, but also is the only method that outperforms pure RL/RL-G with this minimal amount of data. This means that this is the first method that addresses RLVR with few-shot demonstrations to some extent. We also show comparable performance with only 512 expert demonstrations to HPT with 46K expert demonstrations.
W2 (4). Matched multi-seed comparisons.
-
Due to limited computational resource, we focused multi-seed evaluation on FEST-DPO and FEST-GRPO (reported in the original Appendix D.1, Tab. 9) and report single-seed results for other works, following all our baselines, including SRFT (ICLR ‘26), LUFFY (NeurIPS ‘25), ReLIFT (ICLR ‘26), CHORD (ICLR ‘26), HPT and MIFO. We are the first work in this line to report multi-seed results, and the results are recognized by the other two reviewers as part of the critical strength (strength 2 of zWMG; strength 1 of zYQ6). We also conduct more sanity checks to ensure that the benefit persists beyond a specific set of demonstrations and a single training run (Tab. 2, 4, 9, and 10) (recognized in strength 1 of zYQ6).
-
To better answer the reviewer’s question, we conduct two more pure RL-G runs, which is the strongest baseline, and report the updated results here; the result shows that our performance gain over RL-G is consistent. We also add the result into Appendix D.1, Tab. 9:
| Methods | AIME25 | AMC23 | AIME24 | MATH-500 | Olympiad | Minerva | Average |
|---|---|---|---|---|---|---|---|
| FEST-DPO (Avg. of 3 runs) | 13.33+-1.90 | 60+-2.55 | 18.05+-2.51 | 80.32+-0.85 | 46.09+-0.65 | 33.37+-1.14 | 41.85+-0.31 |
| FEST-GRPO (Avg. of 3 runs) | 16.52+-1.73 | 57.60+-1.57 | 16.11+-2.09 | 80.38+-0.66 | 46.97+-0.61 | 34.85+-1.52 | 42.07+-0.40 |
| RL-G (Seed 1) | 14.17 | 57.5 | 15.83 | 78 | 44.49 | 33.36 | 40.55 |
| RL-G (Seed 2) | 15 | 55.63 | 15.42 | 78.95 | 44.75 | 35.29 | 40.84 |
| RL-G (Seed 3) | 14.58 | 53.12 | 14.58 | 77.9 | 43.69 | 34.28 | 39.69 |
| RL-G (Avg. of 3 runs) | 14.58+-0.34 | 55.42+-1.79 | 15.28+-0.52 | 78.28+-0.47 | 44.31+-0.45 | 34.31+-0.78 | 40.36+-0.49 |
W2 (5). Cost of additional online rollouts.
We provided an estimate of the computational cost in Appendix E of the original submission. We additionally report the elapsed time for each step and time breakdown within the step in the updated Appendix D.9. We list these new results below:
1. Elapsed time: We list the mean time of the main methods in our paper for a step here (and also in the updated Appendix E, Tab. 13). Note, to align each method, our implementation of all these methods share identical sampling processes (samples on do not contribute to loss for ReLIFT, HPT and RL); thus, the time difference mainly comes from response length and stall due to long responses.
| Method | Mean time (s/step) |
|---|---|
| FEST-DPO | 793.01 |
| FEST-GRPO | 684.66 |
| ReLIFT | 535.92 |
| ReLIFT-G | 579.23 |
| HPT | 872.68 |
| HPT-G | 702.15 |
| RL | 574.64 |
| RL-G | 643.96 |
2. Time breakdown: For a better estimate of the sampling overhead on , we also provide a time breakdown of the training process. Sampling on and each take about 20% of the total time for a step; training takes another 40% of the time; other work, such as recomputing log probability and preparing the batch takes the remaining 20% of the time. Therefore, the sampling overhead for our method and the “-G” methods are approximately 20% of the total time, which is a modest overhead.
3. Sample-normalized performance. To compare performance of different methods under the same number of examples, we plot pass@1 on the training and test set for different methods. The x-axis is halved for methods that do not use online samples from . The result shows that despite only half the number of gradient updates, our method is still among the best on the test set.
Thank the authors for the rebuttal. It addressed most of my concerns. I have raised the score from 2 to 3.
We sincerely thank the reviewer for the thoughtful feedback on our paper and decision to raise the score; we have added the new experimental results and addressed suggestions from you and the other reviewers in the revised manuscript.
Rebuttal to Reviewer f56z (1/2)
We thank the reviewer for their time and feedback, and for highlighting the importance of our problem and the broad evaluation. Here are our answers:
W1 (1). Limited novelty.
1. The novelty of our work extends beyond the individual methodological components. We study the intersection of two important challenges in the current RLVR literature: the scarcity of high-quality expert reasoning data and the limited exploratory ability of RLVR. To address this joint challenge, we formulate the few-shot demonstration-guided RLVR paradigm, which requires extracting meaningful RL improvements from an exceptionally small expert dataset. We identify the three key challenges in this setting: lack of on-demand data, limited knowledge and overfitting, and accordingly design the first effective solution to this paradigm. Our contributions additionally include extending the unified post-training framework and establishing a rigorous connection between adversarial training and semi-online DPO through the derived loss bound.
Our idea is motivated by a new problem formulation and supported by new insights. Our method not only provides a solution to the novel and important few-shot demonstration-guided RLVR setting as shown in Tabs. 2 and 3, but also yields new insights into how DPO should be adapted for the substantially longer responses encountered in modern reasoning models, particularly through the selection of its hyperparameters (Appendix D.6) and the improvement on its gradient component (in FEST-GRPO; Sec. 3.3 and Appendix D.4). To our best knowledge, these results have not previously been established.
Regarding SPIN: we discuss its major differences to our method in detail in Appendix B.1. Our work is also theoretically a step forward: we prove the bounds in Appendix B.2 for the loss that is only vaguely referred to as an analogy in SPIN, which is a novel result. HPT and CHORD only study different ways of combining SFT and RL; they do not make full use of the few-shot expert data. Also, we list the dataset requirement in Tab. 1, which shows that our method uses much less expert data than HPT and CHORD.
W1 (2) / Q2. Lack of evaluation with standard semi-online DPO or explicitly decaying SFT regularization under matched settings.
We do have a comparison with GRPO + standard semi-online DPO and explicitly decaying SFT regularization. In Appendix D.3, we ablated β with β being (0.1, 0.1, 0.1) and (0.001, 0.001, 0.001) with FEST-DPO. Both runs are GRPO + standard semi-online DPO because all values of β remain fixed, regardless of the rollout correctness. They use the identical sampling and training setting with our FEST-DPO (and FEST-GRPO).
W.r.t. an explicitly decaying SFT regularization, CHORD is exactly the baseline (note we test CHORD-ϕ with token-wise coefficient, as the naive weight CHORD-μ is found to underperform CHORD-ϕ in their paper, as we mentioned in Appendix C.3). Apart from this, “RL+few-shot SFT with our weight” in Sec 4.4.1 provides another comparison with weight choice closer to our method. The training setting for this variant is also identical to FEST as we implement “RL+few-shot SFT with our weight” by applying a zero weight on the on-policy DE loss. In conclusion, the simple alternatives proposed by the reviewers are already adequately evaluated in our original submission.
W2 (1) / Q1. Sign inconsistency in Eq. (13).
Thanks a lot for careful reading. We double checked Eq. (13) and found it to be correct. Note A’i is always negative according to our discussion in Sec. 3.3 and line 1044-1045; therefore, −A’i is positive, and −nM1⋅−A’i is a negative coefficient for logπθ(y+∣x). Minimizing the loss with negative log likelihood corresponds to maximizing log likelihood. We also examine the code and confirm that the implementation matches the intended gradient.
W2 (2) / Q3. Decay of weight is not guaranteed or demonstrated.
The decay of the supervision weight is observable in our reported results. In Appendix D.6, we report the average z value, which is increasing. We also mention that the gradient coefficient is β⋅σ(−z)=1+ezβ (original line 1310; now line 1313). As z keeps increasing and β is a constant, the supervision weight 1+ezβ will decrease. Thanks for pointing this out; we now emphasized this in our revised Sec. 4.4.