- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Agentic-OPD to recover capabilities lost during agentic RL training. Historical checkpoints serve as frozen teachers, while the best-validation checkpoint initializes the student. HCN selects teachers based on their coverage of the student’s remaining failures, and LAD filters supervision to problems where the teacher performs better and to failed student rollouts.
Critical strengths
The evaluation is broad: experiments cover five agentic domains, multiple model sizes, checkpoint-reuse baselines, and component ablations, providing evidence of the method’s applicability.
Critical weaknesses
-
The computational trade-off is unclear. The method requires teacher-pool evaluation, repeated student evaluation, and additional teacher and student rollouts for filtering. Although the appendix discusses these costs, measured runtime or GPU usage and compute-matched comparisons are missing.
-
Tables 1 and 2 contain an unexplained inconsistency. Table 2 specifies Qwen2.5-7B, but its GRPO scores on NQ/HotpotQA are 42.1/34.8, matching the 3B results in Table 1 rather than the 7B results of 43.2/39.2. The authors should clarify whether this reflects a reporting error or different training runs or settings, and ensure that all comparisons use consistent baselines.
Questions for authors
Could the authors report the computational costs and clarify the discrepancy between the GRPO results in Tables 1 and 2, as discussed in the Critical Weaknesses section?
3 comments
We look forward to your response and would greatly appreciate any further comments, suggestions, or guidance you may have.
Thank the authors for their reply. Most of my concerns have been resolved.
Response to Reviewer e3yr: measured compute and Table 1/2 correction
We thank the reviewer for the constructive comments and for pointing out these important issues.
Q1
Using Qwen2.5-7B DeepSearch as an example, the additional overhead mainly comes from three parts:
In total, the integration stage consumes 733,685 environment-interaction trajectories, which is 30.7% of the initial GRPO run, and about 462 GPU hours, which is 85% of GRPO. The average LAD acceptance rate is ρ=0.51, so the oversampling overhead is limited. Only one teacher resides on the GPU during training; peak GPU memory is 101.9 GB, lower than 104.4 GB for GRPO. The 50 checkpoints occupy about 0.76 TB of disk, and deployment requires only the student model, so the inference cost is the same as GRPO.
GRPO has already converged, and continued training yields no meaningful improvement: under the same training-rollout budget (348,160 trajectories, about 136 GRPO steps), the total number of solved questions increases by only 0.8 percentage points, which is within normal fluctuation. Agentic-OPD improves NQ/HotpotQA by 7.2/7.3 points, respectively. We will include the GRPO training curves in the camera-ready version to further demonstrate that continued GRPO training cannot yield improvements.
Q2
Table 2 actually uses Qwen2.5-3B, but its caption incorrectly labels it as Qwen2.5-7B. We will correct this in the revised manuscript and consistently cite the 3B GRPO baseline from Table 1, ensuring that all comparisons use consistent settings.