- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Exploration-Distillation (ExpDis), a framework for decoupling exploration from optimization in RLVR. ExpDis trains a separate explorer policy with a novelty-augmented reward, filters its trajectories for correctness and quality, distills the retained trajectories into a student, and finally trains the student with standard correctness-only RLVR.
ExpDis consistently outperforms DAPO across three models in both pass@1 and pass@64, with a single-round version already outperforming DAPO trained for substantially more RL steps. Further gains are obtained by distributing a fixed RL budget across multiple parallel explorers and repeated exploration-distillation rounds.
I find the core method elegant, particularly because it separates aggressive exploration from the deployed optimization policy while avoiding direct off-policy RL. However, the paper does not directly establish that the trajectories retained from the explorer are genuinely unlikely under the student before distillation, and some interpretations of the MMLU-Pro degradation could be better supported. Overall, I find the method simple, well motivated, and empirically strong. My overall assessment is weak accept.
Critical strengths
The main strength of the paper is the elegance of the method. I particularly like the way ExpDis separates exploration from optimization: the explorer can be pushed aggressively toward novelty, even at the cost of degrading its own policy, while only filtered successful trajectories are transferred to the student through SFT. The student then returns to its own distribution and consolidates these discoveries with standard on-policy RL. I find this a particularly clever design, as it gets the benefit of aggressive exploration without either corrupting the deployed policy or introducing a direct off-policy RL objective.
The empirical results are also strong and consistent across three models, with improvements in both pass@1 and pass@64. The paper is clearly written and unusually thorough in its ablations, covering model choice, training data, design components, novelty mechanisms, and hyperparameters.
Critical weaknesses
1. The analysis does not directly establish that the retained explorer trajectories expand the student’s support.
The paper evaluates several notions of diversity, including lexical diversity, answer diversity, entropy, and an embedding-based semantic strategy diversity metric. However, these measurements are primarily properties of the resulting generations and do not directly show that the filtered correct trajectories transferred from the explorer were unlikely under the student before distillation.
This distinction seems important because the reported diversity metrics are not uniformly aligned. For example, ExpDis improves the semantic diversity metric in Table 6, while some lexical and answer-diversity measures are comparable to or even lower than DAPO depending on the configuration. This makes it difficult to infer from aggregate diversity statistics alone that the explorer is consistently discovering reasoning trajectories outside the student's existing support.
A more direct analysis would be to evaluate the retained explorer trajectories under the pre-SFT student policy , for example through sequence likelihood, token-level NLL, or perplexity, and compare them against trajectories generated directly by the student. This would provide substantially stronger evidence for the paper's support-expansion interpretation.
I would also be interested in a simpler exploration baseline based on increased stochasticity. Table 3 indicates that training rollouts already use , so a useful comparison would be increasing the temperature to or an entropy-regularized RL baseline. Figure 9 also appears to include a setting; if this corresponds to running the same exploration-distillation pipeline without the novelty reward, it would be helpful to clarify this explicitly and discuss what it reveals about the contribution of novelty itself.
Also, the compute comparisons count RL updates but exclude the SFT stage. Since SFT plays a crucial role in ExpDis, reporting end-to-end training cost including SFT would make the efficiency comparison more complete.
2. The interpretation of the MMLU-Pro degradation could be better supported.
The paper interprets the reduction in MMLU-Pro performance as evidence of knowledge loss under aggressive novelty optimization. I think this interpretation is plausible but stronger than the current evidence supports. The observed degradation is relatively small, and without multiple seeds or confidence intervals it is difficult to distinguish systematic forgetting from optimization variance or broader behavioral changes induced by the novelty objective.
A category-level breakdown would also be informative, particularly given the observation that mathematical categories may improve while aggregate MMLU-Pro performance decreases. More generally, since the paper speculates that degradation may accumulate with stronger or longer novelty optimization, reporting MMLU-Pro performance across different values of or different exploration durations would provide direct evidence for this hypothesis.
Questions for authors
- Section 3 states that applying the novelty bonus to all rollouts rather than only verifier-correct ones leads to training instability. Could the authors characterize this failure mode more concretely? For example, does the explorer collapse toward high-novelty but incorrect behaviors, causing too few trajectories to survive rejection sampling, or does the instability arise through a different mechanism?
- The main empirical evaluation is limited to mathematical reasoning. How well do the authors expect ExpDis to generalize to domains such as code generation, tool use, or long-horizon agentic tasks?
2 comments
Response to Reviewer jmNq (Part 2/2)
(Part 2/2)
Categories (Figures 19 and 20). DAPO (+novelty) is below Base in 13 of 14 MMLU-Pro categories, most in philosophy (−4.4) and history (−3.7), and also in the math category (−1.3). ExpDis is above Base in 12 of 14. MMLU-Redux, GPQA-Diamond and IFEval show the same pattern on all three models. On ZebraLogic, DAPO (+novelty) is equal with Base and ExpDis gains the most:
| Model | Method | MMLU-Pro | MMLU-Redux | GPQA-Diamond | IFEval | ZebraLogic |
|---|---|---|---|---|---|---|
| Qwen3-1.7B | Base (accuracy) | 56.50 | 73.08 | 34.39 | 69.71 | 62.45 |
| DAPO | +0.11 | +0.06 | +0.33 | −0.23 | +0.81 | |
| DAPO (4× steps) | +0.49 | +0.27 | +0.76 | −0.40 | +1.50 | |
| DAPO (+novelty) | −1.91 | −0.89 | −0.95 | −1.77 | +0.01 | |
| ExpDis | +1.59 | +0.88 | +1.96 | +0.32 | +3.41 | |
| Qwen3-4B | Base (accuracy) | 70.40 | 83.44 | 55.08 | 82.00 | 82.08 |
| DAPO | +0.10 | +0.05 | +0.25 | −0.24 | +0.57 | |
| DAPO (4× steps) | +0.40 | +0.22 | +0.55 | −0.43 | +1.02 | |
| DAPO (+novelty) | −1.18 | −0.40 | −0.57 | −0.74 | +0.05 | |
| ExpDis | +1.18 | +0.65 | +1.36 | +0.23 | +2.22 | |
| Ministral-3-3B | Base (accuracy) | 25.50 | 55.38 | 39.02 | 48.76 | 12.13 |
| DAPO | +0.19 | +0.10 | +0.31 | −0.21 | +0.64 | |
| DAPO (4× steps) | +0.33 | +0.18 | +0.55 | −0.43 | +1.13 | |
| DAPO (+novelty) | −0.79 | −0.44 | −0.33 | −0.87 | +0.20 | |
| ExpDis | +1.24 | +0.68 | +1.56 | +0.25 | +2.75 |
Novelty weight (Figures 21 to 23). Change relative to Base when the model is trained with novelty weight λ:
| λ | MMLU-Pro | MMLU-Redux | GPQA-Diamond | IFEval | ZebraLogic |
|---|---|---|---|---|---|
| 0 | +0.11 | +0.06 | +0.33 | −0.23 | +0.81 |
| 0.5 | −1.91 | −0.89 | −0.95 | −1.77 | +0.01 |
| 4 | −6.42 | −5.55 | −1.86 | −8.53 | −6.05 |
| 8 | −9.54 | −8.78 | −2.49 | −13.21 | −10.24 |
| 16 | −13.81 | −13.19 | −3.35 | −19.61 | −15.97 |
| 32 | −19.63 | −19.21 | −4.53 | −28.33 | −23.79 |
Math accuracy for the single model and for the ExpDis student:
| Method | AIME24 | AIME25 | AIME26 | MATH500 | Minerva | Mean |
|---|---|---|---|---|---|---|
| Single model, λ=4 | 46.69 | 34.44 | 35.13 | 69.26 | 27.31 | 42.57 |
| Single model, λ=8 | 44.01 | 32.46 | 33.11 | 65.28 | 25.74 | 40.12 |
| Single model, λ=16 | 39.31 | 29.00 | 29.58 | 58.31 | 22.99 | 35.84 |
| Single model, λ=32 | 32.26 | 23.80 | 24.27 | 47.85 | 18.87 | 29.41 |
| ExpDis student, λ=4 | 51.09 | 37.79 | 40.97 | 74.84 | 29.82 | 46.90 |
| ExpDis student, λ=8 | 50.53 | 37.32 | 39.17 | 74.51 | 29.52 | 46.21 |
| ExpDis student, λ=16 | 50.14 | 37.00 | 37.95 | 74.29 | 29.32 | 45.74 |
| ExpDis student, λ=32 | 49.74 | 36.69 | 37.43 | 73.78 | 29.09 | 45.35 |
At λ=32, the single model is 16.22 points below DAPO on math and 19.63 below Base on MMLU-Pro, while the ExpDis student stays within 0.28 points of DAPO and above Base.
Duration (Figure 24). At λ=0.5 the single model's MMLU-Pro deficit goes from 1.91 points at 300 updates to 3.24 at 600 and 5.05 at 1,200. The ExpDis student stays above Base throughout.
Q1. What is the failure mode when the bonus is applied to all rollouts?
When wrong answers also earn the novelty reward, the explorer is rewarded for being unusual, whether or not it is right (Figure 26). Its responses hit the 32,768-token budget within about 50 updates, so 99% are cut off before giving an answer and correctness falls from 0.73 to 0.01. The filter then finds 259 traces instead of the full 500, almost all from early in training. The explorer drifts toward novel but incorrect behavior, and too few trajectories survive rejection sampling. We have updated our draft to discuss this.
Q2. How well should ExpDis generalize to code, tool use or long-horizon agentic tasks?
In principle, we believe that these domains are just as compatible with ExpDis as math, but we chose math due to its relatively affordable computational costs. We agree that extending our approach to other domains is a valuable objective for future work. Prompted by your feedback, we have updated the discussion section of our draft accordingly.
The revision also extends pass@k to 1,024 samples, where ExpDis remains above the base model at every k on all three models (Figure 3). We also compare ExpDis with R²PO and ICE, two recent exploration methods for RLVR. A single round of ExpDis outperforms both on all three models (Table 1).
Thanks again for your feedback. Motivated by your points, we have run additional experiments and made numerous paper edits, which we feel have improved the quality of our draft. We would greatly appreciate it if you would consider raising your score accordingly. Do you have any other questions we can address?
Response to Reviewer jmNq (Part 1/2)
(Part 1/2)
We thank the reviewer for the detailed and constructive feedback. We address your points below, including additional experiments and paper edits found in our revised draft. Figure, table, and appendix numbers below refer to the revised PDF.
W1a. The analysis does not directly establish that the retained explorer trajectories expand the student's support.
Thanks for the suggestion. We ran the analysis you propose (see Figure 18). We score rollouts under the pre-SFT student, which is the base model in round 1, and compare the NLL of the traces the filter retains with the base model's own samples. The student's own samples average 0.242 nats per token, with a 95th percentile of 0.342.
The last row covers the explorer rollouts that pass the filter (correct, complete and within the length window), which are the traces the student is ultimately trained on. One in four of them is less likely than 95% of the student's own samples, five times the reference rate of 5%. In other words, we observe that our explorers indeed generate unlikely traces under the base model. We have added this experiment to our updated draft.
W1b. Simpler exploration baselines: higher temperature, entropy regularization, and what λ=0 means.
The λ=0 point in Figure 11 is the full ExpDis pipeline (explorer, filter, SFT, student RL) with the novelty term removed. We added the above baselines as well (Figure 17):
Raising the rollout temperature to 1.2 or adding an entropy bonus leaves DAPO at or below its original accuracy. With λ=0, ExpDis performs close to DAPO, so the novelty bonus accounts for most of its gain. Replacing it in the explorer with a higher temperature or an entropy bonus does no better than λ=0, indicating that high temperature and the entropy bonus do not confer exploration benefits on accuracy.
W1c. Compute comparisons exclude SFT.
Thanks for pointing out that our budget matching was confusing in the submitted draft. We want to clarify that runs are wall-clock matched, including the SFT cost and not only the RL cost. Because our filtering scheme selects shorter generations, our models tend to generate marginally shorter traces, which makes up for the small added SFT cost, meaning ExpDis can perform the same number of RL iterations and also the SFT stages in the same time that DAPO performs just the RL iterations. Thanks for this feedback, we have updated our draft to better explain the full training cost and how to compare the runs.
W2. The interpretation of the MMLU-Pro degradation could be better supported (seeds, categories, λ, duration).
Seeds. Change in MMLU-Pro accuracy relative to Base, averaged over three seeds (we will report ten seeds in the camera-ready version and are running them now):