acceptodds
Under review as a conference paper at ICLR 2027

Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidance

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has achieved great success in developing Large Language Models (LLMs) with chain-of-thought rollouts. It is used for many tasks such as math and coding. Nevertheless, RLVR struggles with sample efficiency on difficult problems where correct rollouts are hard to generate. Prior works propose to address this issue via demonstration-guided RLVR, i.e., to conduct Supervised FineTuning (SFT) when RL fails. However, SFT often requires a lot of data, which can be expensive to acquire. In this paper, we propose FEST, a FEw-ShoT demonstration-guided RLVR algorithm. It attains compelling results with only demonstrations randomly selected from an SFT dataset. We find that three components are vital for the success: supervised signal, on-policy signal, and decaying weights on the few-shot SFT dataset to prevent overfitting from multiple-epoch training. Across several benchmarks, FEST outperforms baselines in the few-shot setting and even matches the performance of baselines trained with orders of magnitude more SFT data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning with verifiable rewards, few-shot, math

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_f56zOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes FEST, a few-shot demonstration-guided RLVR framework that combines GRPO with semi-online DPO or negative-advantage GRPO. The main idea is to leverage a small number of expert demonstrations through supervised and on-policy learning signals with adaptive weighting. Experiments on mathematical reasoning benchmarks show improvements over several RLVR baselines using only 128 randomly sampled demonstrations.

Critical strengths
  1. The paper addresses a relevant practical problem: improving RLVR with limited expert demonstrations. The reported performance gains with only 128 examples are encouraging.

  2. The experimental evaluation includes several competitive baselines, different few-shot subsets, and additional results on a 7B model, providing some evidence of the method's effectiveness.

Critical weaknesses
  1. Limited novelty and insufficient justification of the proposed design.

The method largely combines existing techniques, including semi-online DPO, joint SFT/RL optimization, and adaptive weighting. The distinction from prior work such as SPIN, HPT, and CHORD is not sufficiently compelling. More importantly, the ablations do not convincingly establish why this particular combination is necessary. Simple alternatives, such as GRPO with standard semi-online DPO or explicitly decaying SFT regularization, are not adequately evaluated under matched settings. Consequently, it is unclear whether the proposed components provide meaningful advantages beyond existing approaches.

  1. Technical concerns and insufficient empirical evidence.

The claimed automatic decay of the supervision weight is not theoretically guaranteed or convincingly demonstrated. On the experimental side, improvements over strong baselines are relatively modest, and matched multi-seed comparisons are lacking. The method also requires additional online rollouts, but computational efficiency is not sufficiently analyzed.

Questions for authors
  1. How does FEST compare with standard semi-online DPO using fixed β or simple decaying SFT regularization under identical training and compute budgets?

  2. Can the authors provide direct evidence that the effective supervision weight decreases during training, as claimed?

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer f56z (1/2)

We thank the reviewer for their time and feedback, and for highlighting the importance of our problem and the broad evaluation. Here are our answers:

W1 (1). Limited novelty.

1. The novelty of our work extends beyond the individual methodological components. We study the intersection of two important challenges in the current RLVR literature: the scarcity of high-quality expert reasoning data and the limited exploratory ability of RLVR. To address this joint challenge, we formulate the few-shot demonstration-guided RLVR paradigm, which requires extracting meaningful RL improvements from an exceptionally small expert dataset. We identify the three key challenges in this setting: lack of on-demand data, limited knowledge and overfitting, and accordingly design the first effective solution to this paradigm. Our contributions additionally include extending the unified post-training framework and establishing a rigorous connection between adversarial training and semi-online DPO through the derived loss bound.

  1. Our idea is motivated by a new problem formulation and supported by new insights. Our method not only provides a solution to the novel and important few-shot demonstration-guided RLVR setting as shown in Tabs. 2 and 3, but also yields new insights into how DPO should be adapted for the substantially longer responses encountered in modern reasoning models, particularly through the selection of its hyperparameters (Appendix D.6) and the improvement on its gradient component (in FEST-GRPO; Sec. 3.3 and Appendix D.4). To our best knowledge, these results have not previously been established.

  2. Regarding SPIN: we discuss its major differences to our method in detail in Appendix B.1. Our work is also theoretically a step forward: we prove the bounds in Appendix B.2 for the loss that is only vaguely referred to as an analogy in SPIN, which is a novel result. HPT and CHORD only study different ways of combining SFT and RL; they do not make full use of the few-shot expert data. Also, we list the dataset requirement in Tab. 1, which shows that our method uses much less expert data than HPT and CHORD.

W1 (2) / Q2. Lack of evaluation with standard semi-online DPO or explicitly decaying SFT regularization under matched settings.

We do have a comparison with GRPO + standard semi-online DPO and explicitly decaying SFT regularization. In Appendix D.3, we ablated β\beta with β\beta being (0.1, 0.1, 0.1) and (0.001, 0.001, 0.001) with FEST-DPO. Both runs are GRPO + standard semi-online DPO because all values of β\beta remain fixed, regardless of the rollout correctness. They use the identical sampling and training setting with our FEST-DPO (and FEST-GRPO).

W.r.t. an explicitly decaying SFT regularization, CHORD is exactly the baseline (note we test CHORD-ϕ\phi with token-wise coefficient, as the naive weight CHORD-μ\mu is found to underperform CHORD-ϕ\phi in their paper, as we mentioned in Appendix C.3). Apart from this, “RL+few-shot SFT with our weight” in Sec 4.4.1 provides another comparison with weight choice closer to our method. The training setting for this variant is also identical to FEST as we implement “RL+few-shot SFT with our weight” by applying a zero weight on the on-policy DED_E loss. In conclusion, the simple alternatives proposed by the reviewers are already adequately evaluated in our original submission.

W2 (1) / Q1. Sign inconsistency in Eq. (13).

Thanks a lot for careful reading. We double checked Eq. (13) and found it to be correct. Note A’iA’_i is always negative according to our discussion in Sec. 3.3 and line 1044-1045; therefore, −A’i-A’_i is positive, and −1nM⋅−A’i-\frac{1}{nM}\cdot -A’_i is a negative coefficient for log⁡πθ(y+∣x)\log\pi_\theta(y^+|x). Minimizing the loss with negative log likelihood corresponds to maximizing log likelihood. We also examine the code and confirm that the implementation matches the intended gradient.

W2 (2) / Q3. Decay of weight is not guaranteed or demonstrated.

The decay of the supervision weight is observable in our reported results. In Appendix D.6, we report the average zz value, which is increasing. We also mention that the gradient coefficient is β⋅σ(−z)=β1+ez\beta\cdot\sigma(-z)=\frac{\beta}{1+e^z} (original line 1310; now line 1313). As zz keeps increasing and β\beta is a constant, the supervision weight β1+ez\frac{\beta}{1+e^z} will decrease. Thanks for pointing this out; we now emphasized this in our revised Sec. 4.4.

AuthorsOpenReview ↗

Rebuttal to Reviewer f56z (2/2)

W2 (3). Relatively modest improvement over strong baselines.

  1. The performance improvement of our method compared to baselines (+1.8% accuracy, FEST-GRPO vs. RL-G in Tab. 2) is on the same level of prior work compared to their baselines. E.g., ReLIFT (+1.7% accuracy on average compared to strongest baseline in their Tab. 1), HPT (+2.2% accuracy on average compared to strongest baseline in Tab. 2; the learning rates for baselines in their Tab. 3 are too low).

  2. Our method not only shows performance improvements both on pass@8 and avg@8, but also is the only method that outperforms pure RL/RL-G with this minimal amount of data. This means that this is the first method that addresses RLVR with few-shot demonstrations to some extent. We also show comparable performance with only 512 expert demonstrations to HPT with 46K expert demonstrations.

W2 (4). Matched multi-seed comparisons.

  1. Due to limited computational resource, we focused multi-seed evaluation on FEST-DPO and FEST-GRPO (reported in the original Appendix D.1, Tab. 9) and report single-seed results for other works, following all our baselines, including SRFT (ICLR ‘26), LUFFY (NeurIPS ‘25), ReLIFT (ICLR ‘26), CHORD (ICLR ‘26), HPT and MIFO. We are the first work in this line to report multi-seed results, and the results are recognized by the other two reviewers as part of the critical strength (strength 2 of zWMG; strength 1 of zYQ6). We also conduct more sanity checks to ensure that the benefit persists beyond a specific set of demonstrations and a single training run (Tab. 2, 4, 9, and 10) (recognized in strength 1 of zYQ6).

  2. To better answer the reviewer’s question, we conduct two more pure RL-G runs, which is the strongest baseline, and report the updated results here; the result shows that our performance gain over RL-G is consistent. We also add the result into Appendix D.1, Tab. 9:

MethodsAIME25AMC23AIME24MATH-500OlympiadMinervaAverage
FEST-DPO (Avg. of 3 runs)13.33+-1.9060+-2.5518.05+-2.5180.32+-0.8546.09+-0.6533.37+-1.1441.85+-0.31
FEST-GRPO (Avg. of 3 runs)16.52+-1.7357.60+-1.5716.11+-2.0980.38+-0.6646.97+-0.6134.85+-1.5242.07+-0.40
RL-G (Seed 1)14.1757.515.837844.4933.3640.55
RL-G (Seed 2)1555.6315.4278.9544.7535.2940.84
RL-G (Seed 3)14.5853.1214.5877.943.6934.2839.69
RL-G (Avg. of 3 runs)14.58+-0.3455.42+-1.7915.28+-0.5278.28+-0.4744.31+-0.4534.31+-0.7840.36+-0.49

W2 (5). Cost of additional online rollouts.

We provided an estimate of the computational cost in Appendix E of the original submission. We additionally report the elapsed time for each step and time breakdown within the step in the updated Appendix D.9. We list these new results below:

1. Elapsed time: We list the mean time of the main methods in our paper for a step here (and also in the updated Appendix E, Tab. 13). Note, to align each method, our implementation of all these methods share identical sampling processes (samples on DED_E do not contribute to loss for ReLIFT, HPT and RL); thus, the time difference mainly comes from response length and stall due to long responses.

MethodMean time (s/step)
FEST-DPO793.01
FEST-GRPO684.66
ReLIFT535.92
ReLIFT-G579.23
HPT872.68
HPT-G702.15
RL574.64
RL-G643.96

2. Time breakdown: For a better estimate of the sampling overhead on DED_E, we also provide a time breakdown of the training process. Sampling on DID_I and DED_E each take about 20% of the total time for a step; training takes another 40% of the time; other work, such as recomputing log probability and preparing the batch takes the remaining 20% of the time. Therefore, the sampling overhead for our method and the “-G” methods are approximately 20% of the total time, which is a modest overhead.

3. Sample-normalized performance. To compare performance of different methods under the same number of examples, we plot pass@1 on the training and test set for different methods. The x-axis is halved for methods that do not use online samples from DED_E. The result shows that despite only half the number of gradient updates, our method is still among the best on the test set.

Thank the authors for the rebuttal. It addressed most of my concerns. I have raised the score from 2 to 3.

We sincerely thank the reviewer for the thoughtful feedback on our paper and decision to raise the score; we have added the new experimental results and addressed suggestions from you and the other reviewers in the revised manuscript.

Reviewer_zYQ6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work analyzes the way in which a small amount of expert trajectories could be leveraged for the training of RLVR systems. FEST includes GRPO on a big dataset with just the final-answer supervision and semi-online DPO, which combines the few expert answers with the model's answers. FEST-GRPO replaces the negative updates on the expert set with a clipped policy objective. Experiments are done on 64 to 512 expert examples, on different random subsets of examples, on three separate training trials, 7B model and transfer to MMLU-Pro.

Critical strengths
  • Benefits of a little amount of expert data are verified through multiple sanity checks. Using 128 expert trajectories sampled at random, FEST-GRPO obtains an average score of 42.38 across six tasks, versus 39.79 by RL, 40.55 by RL-G, and 40.51 by ReLIFT. The appendix provides an average of 42.07 ± 0.40 across three training runs and results using various sets of demonstrations and 7B model. I appreciate these sanity checks because they make it less likely that the benefits come from a specific set of demonstrations or from one training run only (Tables 2, 4, 9, and 10).

  • I find the authors' efforts with regard to the baseline comparisons impressive. The authors tune learning rates not only for pure RL but also for several other baselines. In addition, they provide Avg@8, Pass@8, and MMLU-Pro results. This makes me feel even more confident about the benefits obtained and their generalization outside the 128 expert training questions.

Critical weaknesses
  1. The relative contributions of individual components are not clear. SPIN already makes use of both expert and model-generated answers, and CHORD uses both supervised and on-policy learning. The fixed-β controls in Table 12 keep DPO's adaptive gradient weight adjustment, so they fail to prove that it is required. The no-decay controls in Table 5 make use of the entire expert dataset, and the SPIN baseline in Appendix D.8 does not consider joint RL training on answer-only data. This means that they do not separate out each component with the same few-shot setup.

  2. There should be an equal compute budget for the comparison with RL. FEST uses 128 samples from each dataset per round and generates eight responses for each question, which means that it uses different generation budget than pure RL. RL-G can serve as a control, but sampling budgets and optimizer steps need to be included in the evaluation for each method. In addition, the token generations, runtime, and variance between runs need to be taken into account. There is evidence for the saving in expert demonstrations, but not compute efficiency.

Questions for authors
  • Results from Table 12 are already given using fixed β. Given the same 128 expert trajectories and generation budget, could you use a constant gradient weighting scheme instead of adaptive and investigate sequence-level normalization and token-level clipping independently? It will be helpful to know how much each component is responsible for performance.

  • Could you give separate numbers for sampling rounds, optimizer updates, number of generated tokens, elapsed time and variance across runs of FEST, RL, RL-G, and the best performing baseline hybrid?

  • Section 3.2 already discusses the reason for the inclusion of correct responses in the dispreferred branch and Table 12 evaluates β₃ = 0. In addition to that, could you perform ablation of correct response negative update under the standard configuration while fixing β₁ and β₂, and also for FEST-GRPO?

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer zYQ6 (1/2)

We thank the reviewer for their time and feedback. We thank the reviewer for highlighting that our results are verified through multiple sanity checks, our efforts spent with baseline comparisons, and clear writing. Here are our responses:

W1/Q1. Relative contributions of individual components unclear.

  1. Our work studies the intersection of two important challenges in the current RLVR literature: the scarcity of high-quality expert reasoning data and the limited exploratory ability of RLVR. With only 128 expert demonstrations, our aim is to use the supervised gradient to make reasoning strategies seen in the expert traces more likely under the policy, so that RLVR can sample and reinforce them. The negative on-policy update can further encourage exploration by redistributing probability mass toward other plausible solutions (Remark 3.4). In the updated Fig. 12(a), response length increases early in training, a pattern also observed in HPT and consistent with an early shift in how the model generates its solutions. SPIN and CHORD share individual components with FEST, but neither was developed specifically for this few-shot RLVR setting, and both underperform FEST in our experiments. We also make a step forward theoretically - we extend the unified post-training framework and establish a rigorous connection between adversarial training and semi-online DPO through the derived loss bound.

  2. We agree that DPO retains its margin-dependent gradient weighting even when β\beta is fixed. Beyond the hyperparameter sensitivity analysis, Tab. 12 examines a related question: does it help to choose β\beta based on rollout correctness? In this table, we test β=(0.1,0.1,0.1)\beta=(0.1, 0.1, 0.1) and β=(0.001,0.001,0.001)\beta=(0.001, 0.001, 0.001), i.e., β\beta remains identical and is no longer adaptive to rollout correctness. Both settings underperform our original configuration, in which β\beta depends on rollout correctness. These results also answer the reviewer’s question regarding “SPIN with joint RL training on answer-only data”. With a shared β\beta, FEST trains jointly with the SPIN objective on the expert dataset DED_E and GRPO on the answer-only dataset DID_I.

  3. The no-decay control in Tab. 5, i.e., the “Fixed weight SFT+RL on full dataset” line, serves as an upper bound of the performance of using a constant supervised weight by expanding the gold dataset to include every data in the RL dataset. We select this variant because it sets DE=DID_E=D_I; thus, both the supervised and RL gradient will apply on the entire dataset. It serves as the upper bound for both applying RL on the small DED_E (since DED_E now expands to DID_I) and not applying RL on the small DED_E (since RL gradients still cover DID_I, but now DI=DED_I=D_E). In other words, our experiment favors the ablated baselines.

  4. We are not sure how to conduct token-level clipping with DPO weights: DPO is designed to consider sequence-level log probability ratios, and there is no clipping in the DPO loss term LEL_E of FEST-DPO (see Eq. (3)). We are also not sure what the “sequence-level normalization” should apply to. We’d appreciate more details from the reviewer on this.

AuthorsOpenReview ↗

Rebuttal to Reviewer zYQ6 (2/2)

W2/Q2. Equal compute budget for RL comparison; report separate numbers of sampling and update parameters across methods.

We report the training-related hyperparameters here:

Sampling rounds: FEST, RL-G, HPT-G and ReLIFT-G share the same number of sampling rounds (n=8n=8) on both the gold SFT dataset DED_E and the RL dataset DID_I. RL and other methods also use sampling rounds n=8n=8 but only on DID_I.

Optimizer updates: Identical across all methods. We implement our algorithm such that each minibatch of data from DID_I and DED_E are paired; gradients from both datasets are accumulated before optimizer update.

Number of generated tokens: We update the curves showing the change of number of generated tokens for each response with respect to training steps in Fig. 12 of the paper. The result shows that our method produces a similar amount of tokens than the baselines.

Variance across runs: See Appendix D.1.

Elapsed time: We list the mean time of the main methods in our paper for a step here (also in the updated Appendix E, Tab. 13). Note, to align each method, our implementation of all these methods share identical sampling processes (samples on DED_E do not contribute to loss for ReLIFT, HPT and RL); thus, the time difference mainly comes from response length and stall by long responses.

MethodMean time (s/step)
FEST-DPO793.01
FEST-GRPO684.66
ReLIFT535.92
ReLIFT-G579.23
HPT872.68
HPT-G702.15
RL574.64
RL-G643.96

For a better estimate of the sampling overhead on DED_E, we also provide a time breakdown of the training process. Sampling on DID_I and DED_E each take about 20% of the total time for a step; training takes another 40% of the time; other work, such as recomputing log probability and preparing the batch takes the remaining 20% of the time. Therefore, the sampling overhead for our method and -G methods are approximately 20% of the total time. We add these results in Appendix D.9.

Comparison under the same sampling budget: To further show the performance under the same generation budget, we report the result normalized by the number of RL samples in the updated Fig. 13. The result shows that, on the test set, our method remains among the best, despite only half the number of gradient updates.

Q3. Ablation of correct response with negative update.

Great suggestion. We assume setting β3<0\beta_3<0 will not work as the policy is now effectively unlearning the expert rollout, which does not fit our motivation, i.e., to (carefully with low weight) regularize the reasoning pattern of already correct rollouts towards expert data. To address the reviewer’s question, we are running experiments with β3<0\beta_3<0 to see whether it works, and we will update the results once they become available.

Reviewer_zWMGOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FEST, a few-shot demonstration-guided RLVR method that jointly trains on a large answer-only RL dataset and only 128 reasoning-trace demonstrations. FEST uses a semi-online DPO objective; FEST-GRPO replaces the online part with a negative-advantage GRPO variant. The paper argues that a supervised signal, an on-policy signal, and a decaying demonstration weight are jointly necessary. Experiments on six mathematical/reasoning benchmarks compare the methods with RLVR and demonstration-guided baselines, followed by component, dataset, shot-count, and three-seed analyses.

Critical strengths
  1. The paper does not merely reduce the SFT ratio: it fixes a genuinely tiny, randomly selected 128-example trace set while retaining answer-only RL data. Table 1 makes the intended data-efficiency distinction from prior demonstration-guided approaches concrete. This is consequential because costly reasoning traces are a real bottleneck in RLVR.

2.Tables 2–4, the component analyses, different DED_E sources, shot-count experiments, and the three-seed check in Table 9 collectively support the claim that the method is not solely exploiting one benchmark or one selected demonstration set. The reported separation between Avg@8 and Pass@8 is also informative for this setting.

Critical weaknesses

1.The authors do not provide experiments on other base models to demonstrate the generalizability of their method.

2.The authors do not provide an evaluation of the computational costs associated with their algorithm.

Questions for authors

See weakness

3 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer zWMG

We thank the reviewer for their time, feedback, and for appreciating the importance of our idea and the informative experiments. Below are our answers:

W1. Experiments on other base models to demonstrate the generalizability of the method.

Thanks for pointing this out. Following prior works (HPT and ReLIFT), our experiments focus on two different base models: Qwen2.5-1.5B-Math and Qwen2.5-7B-Math (Appendix D.2). These are also the models chosen by the main baselines in our paper. We hope this already shows generalizability across different base models.

W2. Evaluation of computational cost.

We provided an estimate of the computational cost in Appendix E of the original submission. We additionally report the elapsed time for each step and time breakdown within the step in the updated Appendix D.9. We list these new results below:

1. Elapsed time: We list the mean time of the main methods in our paper for a step here (and also in the updated Appendix E, Tab. 13). Note, to align each method, our implementation of all these methods share identical sampling processes (samples on DED_E do not contribute to loss for ReLIFT, HPT and RL); thus, the time difference mainly comes from response length and stall due to long responses.

MethodMean time (s/step)
FEST-DPO793.01
FEST-GRPO684.66
ReLIFT535.92
ReLIFT-G579.23
HPT872.68
HPT-G702.15
RL574.64
RL-G643.96

2. Time breakdown: For a better estimate of the sampling overhead on DED_E, we also provide a time breakdown of the training process. Sampling on DID_I and DED_E each take about 20% of the total time for a step; training takes another 40% of the time; other work, such as recomputing log probability and preparing the batch takes the remaining 20% of the time. Therefore, the sampling overhead for our method and the “-G” methods are approximately 20% of the total time, which is a modest overhead.

3. Sample-normalized performance. To compare performance of different methods under the same number of examples, we plot pass@1 on the training and test set for different methods. The x-axis is halved for methods that do not use online samples from DED_E. The result shows that despite only half the number of gradient updates, our method is still among the best on the test set.

I have read the answers, Thank authors for their time and computational comsume. I will maintain my score.

We thank the reviewer for maintaining a positive evaluation towards our paper; we have added the new results and addressed suggestions from you and the other reviewers in the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

We thank the AC and the reviewers for their time and feedback. We also thank the reviewers for highlighting the strengths of our paper, including solving a real bottleneck (reviewer zWMG, f56z), being informative, containing broad experiments (all reviewers), and clear presentation (reviewer zYQ6).

As the discussion period is closing, we briefly summarize our response below:

We updated our paper to add the following new experimental evidence:

  1. We report the breakdown time cost within a training step, the response length in tokens produced by each method and the sample-normalized performance curve in Appendix D.9. We also report training time for our method and main baselines in Appendix E (all reviewers).

  2. We update Appendix D.3 to include multiple seed results for RL-G, verifying our performance gain and showing that the gain is statistically significant (reviewer f56z);

  3. We emphasized in our main paper, Sec. 4.4, that the value of zz increases stably (Fig. 6), which is empirical evidence of a decaying supervised weight β1+ez\frac{\beta}{1+e^z}.

We also answer other remaining reviewer questions regarding base models (reviewer zWMG), correctness of equations (reviewer f56z), ablation results (reviewer zYQ6, f56z), and novelty (reviewer f56z). We hope these updates answer the questions of all reviewers, who are now unanimously positive towards acceptance of our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.