acceptodds
Under review as a conference paper at ICLR 2027

Internalizing Rubric Guidance for Research Agents

Abstract

A research rubric specifies what an answer should cover, but using it only as a scalar reward leaves its instructional content unused. We study how to turn this content into better research agents that act without rubric hints. We introduce rubric-guided self-distillation (RSD): criterion-level feedback selects weak queries, the current policy generates new tool trajectories with targeted rubric guidance, and a reward threshold filters the resulting demonstrations. The agent then learns these trajectories under the original, unhinted prompt in supervised phases interleaved with fresh-policy reinforcement learning. No separate demonstration model is required. On Qwen3-8B, RSD improves over rubric-based GRPO by 14.23, 7.61, and 2.84 score points on ResearchQA, DeepResearch Bench, and DeepResearch Bench II, with paired 95% task-bootstrap confidence intervals excluding zero on all three benchmarks. The method also improves LSB from 26.04 to 28.79. At 14B and 32B, it outperforms both starting models and size-matched GRPO on all four research benchmarks, with RQA gains over GRPO of 15.07 and 14.47 points. Strategy comparisons highlight rubric-guided demonstration learning as a major contributor to the gains, and RSD achieves the highest scores on all four research benchmarks among the evaluated 8B configurations. These results show that rubrics can provide not only rewards but also demonstrations that improve research performance without rubric hints at inference.

Primary area:
reinforcement learning
Keywords:
Large Language Models, Reinforcement Learning, Supervised Fine-Tuning, Self-Distillation, Iterative Self-Improvement, Deep Research

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dgDSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes rubric-guided self-distillation (RSD) to improve model training using rubrics. The method alternates reinforcement learning and SFT. To prepare for SFT, it selects weak queries and adds rubric guidance to the prompts, allowing the same model to generate demonstrations. High-quality demonstrations are selected based on rubric scores, and the rubric hints are removed from the prompts before SFT. This cycle is repeated to improve the model’s performance. Experiments on Qwen3-8B, 14B, and 32B demonstrate performance gains from this training approach.

Critical strengths
  • Although generating a set of task-specific rubrics may require the assistance of an additional, stronger model, these rubrics are reusable, and low-scoring trajectories can be filtered out based on them.
  • Rubric hints are used when generating demonstrations, while the original, unhinted prompts are restored during SFT, and complete tool trajectories are learned. This training pipeline is well-designed and aligns with the ultimate goal of deployment without hints.
  • The experiments cover multiple model sizes and benchmarks.
Critical weaknesses
  • The proposed method combines ideas such as rubric guidance, self-generated demonstrations by the model, filtering high-quality trajectories, removing hints for SFT, and alternating SFT with RL training. Since related work already exists for these basic ideas, the novelty is somewhat incremental. Moreover, the experiments show that SFT-only already captures a large portion of the gains; the clearest contribution is that rubric-guided demonstration learning is effective, while the additional contributions of some other design choices remain unclear.
  • The proposed method relies heavily on the content of the rubrics. In this paper, the rubrics simultaneously control three things: the RL reward, guidance when generating demonstrations, and the filtering of demonstrations. Consequently, the final trained model is highly dependent on how the rubrics are designed.
  • The comparison with baseline methods does not adequately control for the intensity of supervised training. RSD trains for up to eight epochs on the demonstrations, whereas the APTER and ReLIFT adaptations use only one epoch, and the SFT learning rates also differ. The paper does not sufficiently justify these differences in settings, making it difficult to determine whether the performance gains come from the RSD training framework or simply from more extensive supervised training.
Questions for authors
  • Why does RSD train for up to eight epochs on the demonstrations, while the two baseline adaptations (APTER and ReLIFT) use only one epoch, and the SFT learning rates also differ? The paper only explains the motivation for dedicated SFT phases, but I believe this is insufficient to attribute the performance advantage to the RSD training framework rather than to more extensive supervised training.
  • Given that SFT-only already achieves a large portion of the gains, compared with merely learning from demonstrations, how much additional gain do the designs in RSD—such as failure-guided query selection and alternating SFT with RL training—each contribute?
  • The random query selection variant is evaluated at round 19, while RSD-8B is evaluated at round 56. Why are different rounds used for comparison?
4 comments

Thank you for asking us to separate the effects of supervision strength, selection, and scheduling. The completed baseline controls and our clarified interpretation of the ablations now provide a more precise account of what the results establish.

Self-distillation under a matched supervision budget

We make the APTER/ReLIFT comparison consistent with RSD by using self-distillation throughout. Demonstrations are generated by each method's current student policy; GLM-5.3-Flash is used only for independent judging. Both adaptations use eight SFT epochs and a matched 6,105,768 supervised assistant-token exposure budget.

The SFT optimizer settings are also aligned: effective batch size 16, Adam β2=0.98\beta_2=0.98, peak learning rate 10−510^{-5}, and cosine decay with a 10−610^{-6} floor. The timing of consolidation follows each method: RSD consolidates after zero-indexed RL rounds 19 and 39, APTER triggers repair from accumulated criterion failures, and ReLIFT consolidates its 32-demonstration buffer. Matching supervised-token exposure preserves these method-specific triggers rather than forcing identical phase boundaries.

APTER-SD has completed 6,105,768 actual, checkpoint-committed supervised assistant-token exposures:

MethodRQADRBDRB-IILSBGPQA
APTER-SD53.5326.0112.5827.2731.25
RSD-8B (ours)59.35(+5.82)^{(+5.82)}29.56(+3.55)^{(+3.55)}17.45(+4.87)^{(+4.87)}28.79(+1.52)^{(+1.52)}34.25(+3.00)^{(+3.00)}

Notes: SD denotes current-student self-distillation. APTER-SD uses the cleaned 920-query training set, 57 RL rounds, and expanded candidate collection to reach the matched supervised-token exposure. RSD is the reported paper reference; superscripts give RSD minus APTER-SD. All benchmark inputs remain in the denominator. LSB includes one external-search service failure counted as zero; the other four benchmark checks found no infrastructure failures.

The reported RSD reference retains 5.82 RQA points and 4.87 DRB-II points of advantage under this matched exposure. The result supports the importance of demonstration construction alongside supervision quantity: APTER retains its category-level repair trigger, while RSD uses query-specific rollout feedback to guide research and consolidate accepted trajectories. We will add the ReLIFT self-distillation results within 24 hours.

Selection strategy

Random follows the paper's core RSD training hyperparameters and uses uniform sampling from the pending weak-query queue.

MethodRQADRBDRB-IILSBGPQA
Random58.0129.6814.9626.9522.50
RSD-8B (ours)59.35(+1.34)^{(+1.34)}29.56$^{(-0.12)}$17.45(+2.49)^{(+2.49)}28.79(+1.84)^{(+1.84)}34.25(+11.75)^{(+11.75)}

Source and use of the training rubrics

We reuse the training queries and GPT-5-generated rubrics from Deep Research as Rubric for Reinforcement Learning. These rubrics are supplied with the dataset and remain fixed throughout our training. The baseline adaptations reuse the existing rubric text for the retained queries.

Rubric content defines the training objective. Its three uses in RSD target the same task requirements: scoring attempts, guiding a new attempt toward missed requirements, and filtering trajectories for SFT. Holding the existing rubric text fixed allows us to compare how the methods use that information. Rubric-based GRPO uses it for rewards; RSD also uses criterion-level feedback to guide the current student in constructing trajectories that are learned under the original query, with the hints removed. On the external benchmarks, the judge uses each benchmark's own evaluation criteria, and the policy receives no rubric hints.

The new controls align the amount and source of supervision while preserving each method's consolidation rule. Both APTER-SD and ReLIFT-SD complete 6,105,768 supervised assistant-token exposures, eight SFT epochs, and 228 RL updates. ReLIFT-SD uses 13 SFT phases and 176 SFT updates; APTER-SD uses 144 SFT updates. The reported RSD reference is 5.82 RQA points and 4.87 DRB-II points above APTER-SD at this matched exposure.

The shared SFT configuration specifies effective batch size 16, Adam β2=0.98\beta_2=0.98, peak learning rate 10−510^{-5}, cosine scheduling, and a configured minimum learning rate of 10−610^{-6}. Schedule lengths and realized learning-rate trajectories remain method-specific.

MethodRQADRBDRB-IILSBGPQA
APTER-SD53.5326.0112.5827.2731.25
ReLIFT-SD22.427.151.5922.5214.25
RSD-8B (ours)59.35(+5.82)^{(+5.82)}29.56(+3.55)^{(+3.55)}17.45(+4.87)^{(+4.87)}28.79(+1.52)^{(+1.52)}34.25(+3.00)^{(+3.00)}

The supervision audit shows why token exposure and corrective content are distinct. ReLIFT-SD learns 405 demonstrations covering 80 of the 920 training queries. Under the saved judge scores, 280 score no higher than the best preceding student answer, including 160 that score lower. These demonstrations account for 73.2% of the supervised-token exposure. The selection rule asks whether any student attempt satisfies every criterion, while admission requires a new mean rubric score of at least 0.8. An accepted demonstration can therefore score below an answer already available from the student.

This audit concerns the rubric-judged self-distillation adaptation of ReLIFT, whose original formulation verifies external demonstrations. It identifies the quality of the admitted targets as a concrete issue in this setting.

RSD supplies the complete rubric and additionally emphasizes criteria the current student group satisfies less often. Scores from unhinted rollouts determine this extra emphasis; all task criteria remain in the guidance. The current policy retries selected weak queries with that context, and high-reward trajectories accumulate for periodic, multi-epoch SFT. The added guidance is removed from the training input, and assistant actions across the tool interaction are learned under the original query before unhinted RL resumes. The methodological distinction concerns how the full rubric and targeted emphasis shape self-generated supervision; the matched-exposure controls compare the complete training procedures, including their collection and consolidation rules.

If these clarifications and additional results resolve your concerns, we would be grateful if you would consider raising your score.

Thank you for the responses and experiments, which address my concern about differences in the intensity of supervised training and additional gains from alternating SFT with RL training. I would like to raise my rating.

Thank you for your careful review and positive assessment. We are glad that our responses and additional experiments addressed your concerns.

Reviewer_RsXJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

Motivated by the fact that scalar reward is not very expressive compared to hints based on a set of rubrics, they propose a method that while doing GRPO, gathers low-scoring trajectories and fixes them based on some hints, then the model is SFTed on the fixed versions (while leaving out the hints ofc).

Critical strengths
  • The paper is clearly written
  • The solution is for a well-motivated problem
  • The solution works well on the benchmarks
  • The solution fits well within current methods using GRPO, i.e. doesn't add much headroom and is easy to implement.
Critical weaknesses
  • The novelty of the solution is very little, based on a few previous works I remembered, so I asked AI to compare them (confirmed the citations):

RSD's stated difference from STaR, learning under the original prompt (L454–455), is already part of STaR, whose rationalization step removes the hint before fine-tuning (Zelikman et al., NeurIPS 2022). The other stated differences are transferring complete multi-turn tool trajectories (L43–45, L424–428), using self-generated demonstrations, and learning them in SFT phases that alternate with RL (L459–462). All are already in EvolveSearch (Zhang et al., EMNLP 2025, uncited), which alternates GRPO with SFT on the agent's own reward-filtered search rollouts. The ablations reveal that this alternation isn't very effective either: SFT-only gets most of the gain over GRPO, and RSD's margin over it is not significant on DRB or DRB-II (Table 11).

This matches what I remembered from previous work.

Questions for authors
  • Is there any reason that EvolveSearch is not cited in the paper?

  • What do you consider the novelties of this paper over STaR, especially considering that the GRPO+SFT loop doesn't help significantly based on the ablations?

3 comments

Thank you for directing us to EvolveSearch and for asking what RSD adds beyond STaR and alternating SFT/RL. We have completed both research-task comparisons. They support a more specific contribution statement: criterion feedback guides the construction of a research trajectory while the policy is interacting with the tools.

Completed EvolveSearch and STaR-style comparisons

MethodRQADRBDRB-IILSBGPQA
EvolveSearch research adaptation45.9022.5315.1626.1428.00
STaR-style research adaptation, iteration 146.2921.6114.0725.3830.00
STaR-style research adaptation, iteration 246.5323.6513.4426.3726.75
RSD-8B (ours)59.35(+12.82)^{(+12.82)}29.56(+5.91)^{(+5.91)}17.45(+2.29)^{(+2.29)}28.79(+2.42)^{(+2.42)}34.25(+4.25)^{(+4.25)}

The adaptations use Qwen3-8B, shared research tools, and evaluation on the original query without hints. Both RSD and our EvolveSearch adaptation use 57 GRPO rounds in total and two SFT stages. STaR-style training performs two outer iterations of answer-guided generation and eight-epoch SFT. Its reference answers are constructed for training queries, rather than taken from benchmark answers. Each iteration collects data with the preceding policy and trains from the original base model; both iterations are shown without selecting an iteration separately for each benchmark.

The reported RSD reference is higher across the five benchmarks in these comparisons. Its RQA differences are 13.45 points over EvolveSearch and 12.82 points over the second STaR-style iteration. These results compare the complete training procedures.

Contribution and interpretation of the ablations

EvolveSearch establishes iterative learning from search trajectories through SFT and RL; STaR establishes answer-guided rationalization and subsequent learning without the answer hint. RSD builds on these ideas by using the task's evaluation requirements to guide a new research interaction. Reward filtering selects completed trajectories. A criterion hint is available earlier, when the policy chooses search terms, gathers evidence, and decides which conditions or comparisons its answer must explain. Accepted tool actions and answers are then learned under the original query.

The SFT-only ablation retains this criterion-guided demonstration-learning path and reaches 56.49 on RQA and 27.02 on DRB, compared with GRPO's 45.12 and 21.95. Full RSD adds 2.86 RQA points, with a paired 95% task-bootstrap interval of [0.77, 4.92]. The DRB and DRB-II intervals include zero. The scheduling continuations likewise distinguish refreshing targets from repeating a fixed set: on the validated additional benchmarks, Single leads on LSB and Refresh on GPQA. These findings place the main empirical contribution in learning criterion-guided demonstrations, with schedule-dependent additional gains.

Self-distillation under matched supervision

For consistency with RSD, our APTER/ReLIFT comparison uses current-student-generated trajectories, eight SFT epochs, and a common 6,105,768 supervised-token exposure budget, with GLM-5.3-Flash used only for independent judging. APTER-SD has completed this actual, checkpoint-committed exposure.

The SFT optimizer settings are also aligned: effective batch size 16, Adam β2=0.98\beta_2=0.98, peak learning rate 10−510^{-5}, and cosine decay with a 10−610^{-6} floor. The timing of consolidation follows each method: RSD consolidates after zero-indexed RL rounds 19 and 39, APTER triggers repair from accumulated criterion failures, and ReLIFT consolidates its 32-demonstration buffer. Matching supervised-token exposure preserves these method-specific triggers rather than forcing identical phase boundaries.

MethodRQADRBDRB-IILSBGPQA
APTER-SD53.5326.0112.5827.2731.25
RSD-8B (ours)59.35(+5.82)^{(+5.82)}29.56(+3.55)^{(+3.55)}17.45(+4.87)^{(+4.87)}28.79(+1.52)^{(+1.52)}34.25(+3.00)^{(+3.00)}

Notes: SD denotes current-student self-distillation. APTER-SD uses the cleaned 920-query training set, 57 RL rounds, and expanded candidate collection to reach the matched supervised-token exposure. RSD is the reported paper reference; superscripts give RSD minus APTER-SD. All benchmark inputs remain in the denominator. LSB includes one external-search service failure counted as zero; the other four benchmark checks found no infrastructure failures.

The reported RSD reference is 5.82 RQA points and 4.87 DRB-II points higher. Together with the EvolveSearch and STaR-style comparisons, this result supports our focus on how rubric feedback shapes the trajectories the policy learns. We will add the ReLIFT self-distillation results within 24 hours.

Rubric-guided self-distillation (RSD) generates demonstrations with the complete rubric and extra emphasis on the student's weak criteria. The guidance retains all rubric criteria; group-level scores from unhinted rollouts determine which receive additional emphasis. Each retry thus receives the full specification of the task together with feedback targeted to the current policy. The current rollout policy generates the assisted tool trajectory, and a reward threshold selects demonstrations for dedicated, multi-epoch SFT. The added guidance is removed from their input, and the model learns assistant actions, including tool calls and answers, under the original query and preceding interaction history. The updated policy then resumes unhinted GRPO.

The completed ReLIFT-SD control adds a comparison under current-student generation, eight SFT epochs, and 6,105,768 supervised assistant-token exposures. It preserves hardest-question selection and buffer-triggered consolidation from ReLIFT, adapting its verified external supervision to the self-distillation setting. The reported RSD reference exceeds APTER-SD by 3.55 DRB points and 4.87 DRB-II points; the ReLIFT-SD records and their evaluation conditions are included below.

MethodRQADRBDRB-IILSBGPQA
APTER-SD53.5326.0112.5827.2731.25
ReLIFT-SD22.427.151.5922.5214.25
RSD-8B (ours)59.35(+5.82)^{(+5.82)}29.56(+3.55)^{(+3.55)}17.45(+4.87)^{(+4.87)}28.79(+1.52)^{(+1.52)}34.25(+3.00)^{(+3.00)}

The audit identifies what question selection and final-answer filtering leave unspecified. ReLIFT-SD retries selected questions without a criterion-guidance prompt. Under the saved judge scores, 125 of its 405 learned demonstrations exceed the preceding best student attempt, 120 tie it, and 160 score lower. For 94 demonstrations, an original student answer scores at least as highly on every criterion and higher on at least one. These comparisons describe the targets actually supplied to SFT.

The complete rubric supplies all task requirements, and the extra emphasis makes the student's weak criteria more salient. RSD combines these two forms of guidance during trajectory generation, then learns accepted trajectories without the added guidance in the input. Repeating the cycle updates the emphasis as the policy changes. This is the contribution we emphasize alongside the EvolveSearch and STaR-style comparisons in our preceding reply. The matched-supervision controls compare the complete training procedures at the same supervised-token exposure.

If these clarifications and additional results resolve your concerns, we would be grateful if you would consider raising your score.

Dear RsXJ, thank you for pointing us to EvolveSearch and asking what RSD adds beyond STaR. We’ve added comparisons with both methods on the research tasks and clarified how rubric feedback guides trajectory generation. We’d appreciate hearing whether this addresses your concern about the contribution.

Reviewer_svHuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Rubric-Guided Self-Distillation (RSD), a training framework that leverages criterion-level rubric feedback to improve research agents. The method identifies weak queries, generates improved tool-use trajectories with targeted rubric hints, and filters high-scoring demonstrations for supervised fine-tuning under the original prompts. The training process alternates between GRPO and dedicated SFT phases, allowing the updated policy to generate demonstrations for subsequent iterations. Experiments on Qwen3-8B, 14B, and 32B across four research benchmarks show substantial improvements over several baselines.

Critical strengths
  • Practical Training Framework. RSD provides a concrete approach to generating high-quality demonstrations using the current policy and task-specific rubrics. Learning complete multi-turn trajectories, including search actions and answer construction, is practically relevant.

  • Consistent Empirical Improvements. RSD substantially outperforms other baselines across multiple research benchmarks and three model scales.

Critical weaknesses
  • RSD is closely related to RuscaRL[1], as both aim to internalize rubric guidance into the policy. The main methodological distinction appears to be that RuscaRL gradually reduces rubric guidance during RL, whereas RSD alternates unhinted RL with SFT on rubric-guided trajectories. However, the paper discusses RuscaRL in only one sentence, without clearly establishing the novelty of this alternative training paradigm or providing a direct comparison to demonstrate its advantages. This leaves the independent methodological contribution of RSD insufficiently substantiated.

  • Several adapted baselines in Table 1 show little improvement over GRPO, despite incorporating additional supervision. For example, APTER uses 346 teacher demonstrations but achieves almost no gain, while RSD uses 379 demonstrations with substantially more intensive SFT. The paper does not sufficiently explain these weak baseline results or disentangle the effects of the proposed method from differences in demonstration quality and supervised training intensity.

[1]: Zhou, Yang, et al. "Breaking the exploration bottleneck: Rubric-scaffolded reinforcement learning for general llm reasoning." arXiv preprint arXiv:2508.16949 (2025).

Questions for authors
  • Q1: Why do the adapted baselines in Table 1 show little or no improvement over vanilla GRPO on most benchmarks?

  • Q2: Given the strong SFT-only results, why do APTER and ReLIFT, which also incorporate supervised demonstrations, achieve much smaller improvements?

  • Q3: Repeatedly alternating RL and SFT introduces additional implementation complexity. How much performance would be lost by adopting a simpler training procedure with a single SFT stage followed by standard GRPO?

3 comments

Thank you for your questions about baseline strength, demonstration quality, and training complexity. We address them with a matched-supervision comparison and a single-stage continuation experiment.

Matched self-distillation comparison

We align the source and amount of supervision. APTER/ReLIFT generate demonstrations with their own current student policies and use eight SFT epochs and a 6,105,768 supervised assistant-token exposure budget, matching RSD. GLM-5.3-Flash serves only as an independent judge. Shared SFT settings are effective batch size 16, Adam β2=0.98\beta_2=0.98, peak learning rate 10−510^{-5}, and cosine decay with a 10−610^{-6} floor. The timing of consolidation follows each method: RSD consolidates after zero-indexed RL rounds 19 and 39, APTER triggers repair from accumulated criterion failures, and ReLIFT consolidates its 32-demonstration buffer.

APTER-SD has completed this actual, checkpoint-committed exposure:

MethodRQADRBDRB-IILSBGPQA
APTER-SD53.5326.0112.5827.2731.25
RSD-8B (ours)59.35(+5.82)^{(+5.82)}29.56(+3.55)^{(+3.55)}17.45(+4.87)^{(+4.87)}28.79(+1.52)^{(+1.52)}34.25(+3.00)^{(+3.00)}

Notes: SD denotes current-student self-distillation. APTER-SD uses the cleaned 920-query training set, 57 RL rounds, and expanded candidate collection to reach the matched supervised-token exposure.

The reported RSD reference is 5.82 RQA points and 4.87 DRB-II points higher, with corresponding advantages of 3.55 on DRB and 3.00 on GPQA. These results connect the contribution to how the current policy constructs demonstrations under rubric feedback, beyond the amount of SFT supervision. We will add the ReLIFT self-distillation results within 24 hours.

What differs in the demonstrations?

ReLIFT identifies which questions need help; RSD also supplies information that can change how a new solution is generated. In our ReLIFT adaptation, a question is eligible when none of its eight unhinted attempts satisfies all criteria. The generator then retries that question without the criterion-guidance prompt. RSD instead provides the rubric requirements, emphasizing those frequently missed by the current policy. For example, if an answer lists a system's components but misses their coordination, the guidance can direct research toward control strategies and operating conditions.

APTER triggers repair from persistent failures across queries, while RSD uses query-specific rollout feedback. APTER aggregates failures by stable capability categories before requesting repairs. Its repair prompt also contains specific failed rubrics and the full task criteria. RSD uses the criterion profile of a query's rollout group to guide collection and accumulates accepted trajectories for scheduled SFT. The distinction concerns how and when the learning targets are constructed.

RuscaRL gradually reduces rubric scaffolding during RL. RSD learns accepted, complete tool trajectories under the original query in dedicated SFT phases, interleaved with unhinted RL. The SFT-only ablation retains this rubric-guided demonstration-learning path, explaining why it captures much of RSD's benefit.

How much does a simpler schedule retain?

All three continuations start from a shared checkpoint after 20 GRPO rounds and run 40 further GRPO rounds. Single applies 16 SFT epochs to frozen demonstrations D1D_1 once; Repeat uses two eight-epoch stages on D1D_1; Refresh uses new demonstrations D2D_2 in the second stage. The reported RSD reference is also shown:

ContinuationRQADRBDRB-IILSBGPQA
Single SFT → GRPO56.4128.0914.7328.1631.00
Repeat fixed demonstrations55.4126.3913.0727.6125.50
Refresh demonstrations57.6427.3513.9427.2633.50
RSD-8B (ours)59.35(+1.71)^{(+1.71)}29.56(+1.47)^{(+1.47)}17.45(+2.72)^{(+2.72)}28.79(+0.63)^{(+0.63)}34.25(+0.75)^{(+0.75)}

In these recorded results, Single exceeds Repeat by 1.00 RQA points, and Refresh exceeds Repeat by 2.23 points. The reported RSD reference is 2.94 RQA, 1.47 DRB, and 2.72 DRB-II points above Single. A single SFT stage thus retains much of the observed benefit, while refreshing targets improves on repeating fixed demonstrations. Single leads on LSB and Refresh on GPQA. These comparisons distinguish splitting a fixed set of demonstrations from collecting refreshed targets. The paper's full-versus-SFT-only comparison gives +2.86 RQA points, with paired 95% interval [0.77, 4.92]. Our main contribution centers on criterion-guided trajectory learning.

RSD retains an advantage over APTER-SD when supervised-token exposure is matched. In the original table, the APTER and ReLIFT-32 adaptations used one SFT epoch and accumulated 908,539 and 113,260 supervised assistant-token exposures, respectively, compared with RSD's eight epochs and 6,105,768 exposures. APTER's 346 demonstration uses and RSD's 379 therefore do not capture the difference in supervision intensity: exposure also depends on the number of assistant tokens and training passes. This difference motivates the controls below.

The new APTER-SD and ReLIFT-SD controls both complete 6,105,768 actual, checkpoint-committed supervised assistant-token exposures with eight SFT epochs. Demonstrations are generated by each method's current student policy; GLM-5.3-Flash serves as the independent judge. Exposure counts the tokens selected by the assistant loss mask, excluding system, user, and tool-observation tokens. At this matched exposure, the reported RSD reference exceeds APTER-SD by 5.82 RQA points, 3.55 DRB points, and 4.87 DRB-II points. The matched-supervision result strengthens the evidence for the value of how demonstrations are constructed and learned, beyond increasing the amount of SFT alone. The promised ReLIFT-SD records complete the table below.

MethodRQADRBDRB-IILSBGPQA
APTER-SD53.5326.0112.5827.2731.25
ReLIFT-SD22.427.151.5922.5214.25
RSD-8B (ours)59.35(+5.82)^{(+5.82)}29.56(+3.55)^{(+3.55)}17.45(+4.87)^{(+4.87)}28.79(+1.52)^{(+1.52)}34.25(+3.00)^{(+3.00)}

The APTER-SD result addresses supervision quantity; the ReLIFT-SD demonstration audit examines the content of the targets actually learned. The audit examines which demonstrations enter SFT after hard-question selection. ReLIFT-SD selects a question when none of eight student attempts satisfies every rubric criterion, then accepts a new demonstration at a mean rubric score of at least 0.8. Among its 405 learned demonstrations, exact pairing with the source attempts finds 125 higher scores than the preceding best answer, 120 ties, and 160 lower scores. Thus 69.1% score no higher than an answer the student already generated. The demonstrations exceed the group's mean score by 0.20 on average, showing their value for reinforcing stronger responses; their admission does not ensure that they address the remaining errors.

ReLIFT-SD adapts the original method's hardest-question selection and buffer-triggered interleaving to rubric-judged self-distillation, replacing verified external demonstrations.

RSD generates self-distillation targets with the complete rubric and additional emphasis on the student's weak criteria. The guidance retains all task criteria, while scores aggregated over the unhinted rollout group identify which criteria receive extra emphasis. Each retry therefore receives the full task requirements together with feedback targeted to the current policy. The same rollout policy generates a new tool trajectory with this guidance; high-reward trajectories accumulate for dedicated, multi-epoch SFT. RSD removes the added guidance from the input and learns the accepted trajectories' assistant actions, including tool calls and answers, under the original prompt. The updated policy then continues unhinted RL. The key distinction is how the complete rubric and the emphasis on weak criteria jointly guide demonstration construction.

If these clarifications and additional results resolve your concerns, we would be grateful if you would consider raising your score.

Dear reviewer svHu, thank you for asking whether RSD’s gains come from additional supervision and how much a single SFT stage retains. We’ve added a matched-supervision comparison and a single-stage continuation experiment, and we’d appreciate hearing if either point still needs clarification.

Comments on the paper

2 comments

Dear Area Chair and Reviewers,

We have devoted substantial time and effort to preparing a detailed rebuttal and carefully addressing the concerns raised in the reviews. We have also uploaded an updated manuscript PDF for your consideration alongside our rebuttal. We sincerely thank the reviewers who have already provided follow-up feedback and engaged in the discussion.

For review threads that have not yet received follow-up feedback, we are concerned that the discussion may conclude without further engagement with our responses, leaving important clarifications unexamined. We would greatly appreciate hearing whether our rebuttal has resolved the main concerns and, if not, which specific issues still require clarification.

We therefore respectfully ask reviewers who have not yet responded to review our rebuttal and share their feedback before the discussion period closes. If concerns remain, we welcome the opportunity to address them. If our responses have resolved the concerns underlying your initial assessment, we would appreciate an update to your evaluation accordingly.

We would also appreciate the Area Chair’s help in facilitating further discussion where needed. Thank you all for your time and consideration.

AuthorsOpenReview ↗

Summary of the reviews and author responses

Dear Area Chair and Reviewers,

We thank svHu, RsXJ, and dgDS for recognizing the gains across research benchmarks and model sizes, and for asking which parts of RSD account for them. Their questions led us to clarify the contribution and add three sets of comparisons to the revised manuscript.

1. What does RSD add beyond RuscaRL, STaR, and EvolveSearch? RSD gives the current policy the full rubric when it retries a query, emphasizing the criteria it repeatedly missed. The policy can use that feedback while deciding what to search for and which evidence to collect. It then learns the accepted tool calls and answers under the original query, with the hints removed. We clarified how this differs from RuscaRL’s use of rubric guidance within RL, added the EvolveSearch citation, and evaluated EvolveSearch and STaR-style adaptations. On ResearchQA, RSD scored 59.35, compared with 45.90 for EvolveSearch and 46.53 for the second STaR-style iteration. We also clarified that the training rubrics come from the dataset, stay fixed across methods, and are absent from the agent’s evaluation prompts.

2. Could more intensive SFT explain the gains? The original APTER and ReLIFT adaptations used one SFT epoch. We added self-distillation controls matching RSD’s eight epochs, peak SFT learning rate, and 6,105,768 supervised assistant-token exposures. At that exposure, RSD scored 5.82 points higher on ResearchQA, 3.55 on DeepResearch Bench, and 4.87 on DeepResearch Bench II than APTER-SD. We also examined what ReLIFT-SD learned. Of its 405 demonstrations used for SFT, 280 scored at or below the student’s best preceding attempt on the same query. This helps explain why the content of the demonstrations matters alongside their quantity.

3. How much do alternation and query selection contribute? SFT alone captured much of the gain. Full RSD added 2.86 ResearchQA points over SFT alone (paired 95% CI [0.77, 4.92]). The corresponding intervals on the two DeepResearch benchmarks included zero. In continuations from a shared checkpoint, a single SFT stage scored highest on LocalSearchBench, while refreshing demonstrations for a second stage scored highest on GPQA. We also added the requested round-56 Random comparison to replace the earlier round-19 comparison. Random sampled from the same weak-query queue, and its SFT schedule differed from RSD’s, so the results leave the independent contribution of query ordering open.

Reviewer dgDS subsequently confirmed that the additional experiments addressed their concerns about supervision intensity and alternating SFT with RL, and raised their rating. We thank all three reviewers for their time and leave the overall assessment to the Area Chair.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.