- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OptScout uses a short RL run on a small model to guide a larger model’s post-training. It converts the source model’s token-level policy shifts into target-model update directions through low-rank gradient probing and cross-split consistency filtering. It then selects a source checkpoint and update magnitude using reward-based evaluation. Experiments show competitive or better performance than direct RL, with substantial end-to-end time savings.
Critical strengths
-
Using small-model RL signals to construct target-specific updates is a sensible approach to reducing post-training cost.
-
The empirical results are promising, supported by runtime measurements, memory analysis, and component ablations.
Critical weaknesses
-
The novelty over existing work needs clearer justification. Policy-shift transfer is already explored in [1], while sparse, sign-consistent updates are related to [2] and [3]. The target-gradient construction is a meaningful difference, but the experiments do not fully separate its contribution and that of checkpoint scouting from the candidate search.
-
Comparisons with compute-matched target-side LoRA-RL or short-run optimization are missing. It is unclear how much of the efficiency gain requires small-model scouting rather than a cheaper target-side training setup. [1] also selects its teacher by source performance, although the paper argues that source performance does not reliably predict transfer quality.
References
[1] Weak-to-Strong Generalization via Direct On-Policy Distillation, 2026.
[2] Editing Models with Task Arithmetic, ICLR 2023.
[3] TIES-Merging: Resolving Interference When Merging Models, NeurIPS 2023.
Questions for authors
-
How are direct-RL checkpoints selected?
-
Are results available for a fixed early source checkpoint with the same magnitude search, to isolate the benefit of checkpoint scouting?
4 comments
3. Additional Ablations: Independent Contributions of Core Components
We conduct two additional controlled ablation studies to examine the independent contributions of Target-gradient Construction, Checkpoint Scouting, and Magnitude Search. Due to time constraints, these experiments are evaluated only on MATH500 and use the same newly rerun source RL trajectory. Since this trajectory differs from that used in the original submission, we focus on the relative contributions of components under identical experimental conditions rather than comparing absolute results with the original submission.
(1) Target-gradient Construction Ablation.
Table 7 of the original submission already compares different update direction construction methods. To further isolate the contribution of Target-gradient Construction, we fix the source checkpoint (step 15) and update magnitude (), disable Checkpoint Scouting and Magnitude Search, and vary only the target-side update direction construction method while keeping all other settings unchanged.
Rebuttal table 13. Controlled ablation of target-side gradient construction on MATH500.
| Direction Construction | MATH500 mean@16 (%) |
|---|---|
| Full Gradient (All Params.) | 79.59 |
| Full Gradient (Eligible Params.) | 83.90 |
| Low-Rank Probe (Eligible Params.) | 84.49 |
| Low-Rank + Magnitude Top-2% | 83.91 |
| Low-Rank + Cross-Split Top-2% | 85.14 |
The results show that restricting updates to attention and MLP projection matrices improves performance, Low-Rank Gradient Probing further enhances the effectiveness of update directions, and Cross-Split Consistency Filtering outperforms magnitude-only sparse selection. Even without Checkpoint Scouting or Magnitude Search, target-side gradient construction and consistency filtering independently improve performance.
(2) Checkpoint Scouting × Magnitude Search.
We further conduct a 2×2 ablation by independently controlling Checkpoint Scouting and Magnitude Search. All experiments use Low-Rank + Cross-Split Top-2% update directions, with other settings held constant. The fixed-checkpoint setting uses the predetermined source RL checkpoint at Step 15, while the search mechanisms follow the original submission.
Rebuttal table 14. Ablation of checkpoint scouting and magnitude search on MATH500.
| Checkpoint Selection | Magnitude Selection | MATH500 mean@16 (%) |
|---|---|---|
| Fixed Checkpoint | Fixed | 85.14 |
| Fixed Checkpoint | Search 8 Magnitudes | 85.94 |
| Checkpoint Scouting | Fixed | 86.39 |
| Checkpoint Scouting | Search 8 Magnitudes | 87.69 |
The results show that Checkpoint Scouting and Magnitude Search each contribute independently, while their combination achieves the best performance, indicating their complementary effects. Together, these two ablation studies further validate the individual effectiveness of all three core components.
Weakness 2. Compute-matched Target-side RL and Teacher Selection
To address the reviewer's concerns regarding the lack of compute-matched target-side RL comparisons and the reliability of source-side teacher selection, we provide additional GRPO and LoRA-GRPO experiments and clarify how OptScout differs from Direct-OPD in checkpoint selection.
1. Compute-matched Target-side RL Comparison
To examine whether OptScout's efficiency advantage merely comes from reducing target-side training costs, we conduct additional experiments with compute-matched GRPO and LoRA-GRPO. Both methods perform RL directly on Qwen3-8B for approximately 7.6 hours, closely matching OptScout's 7.50-hour end-to-end runtime. All methods use the same hardware environment. For LoRA-GRPO, we follow the official verl recommendations and evaluate multiple hyperparameter configurations, selecting the best-performing setting with LoRA rank = 32, alpha = 64, and learning rate = 5 × 10⁻⁵. The remaining training settings are identical to GRPO.
Rebuttal table 15. Comparison with compute-matched target-side RL baselines on Qwen3-8B.
| Method | AIME24 | AIME25 | MATH500 | AMC | OlympiadBench | Avg. | Time |
|---|---|---|---|---|---|---|---|
| Qwen3-8B (Base) | 41.25 | 23.96 | 80.28 | 61.97 | 45.84 | 50.66 | – |
| Compute-matched GRPO | 42.08 | 32.08 | 83.38 | 66.27 | 50.81 | 54.92 | 7.68 h |
| Compute-matched LoRA-GRPO | 42.08 | 31.67 | 83.68 | 68.45 | 50.00 | 55.18 | 7.62 h |
| GRPO (Full-run) | 49.38 | 35.83 | 87.14 | 75.75 | 56.97 | 61.01 | 41.74 h |
| OptScout | 52.92 | 42.71 | 87.65 | 75.68 | 57.90 | 63.37 | 7.50 h |
Under comparable wall-clock budgets, compute-matched GRPO and LoRA-GRPO achieve average accuracies of 54.92% and 55.18%, respectively, substantially below OptScout's 63.37%. This indicates that simply shortening target-side RL training or using LoRA fine-tuning is insufficient to match OptScout's performance.
Furthermore, Figure 3 of the original submission shows that, on both Qwen3-8B and Qwen3-14B, OptScout achieves better MATH500 performance than the final models from full GRPO training while requiring substantially less wall-clock time. Together with the additional experiments, this further supports OptScout's performance advantage under limited computational budgets.
2. Direct-OPD Teacher Selection
We do not argue that source-side performance cannot be used for teacher selection. Rather, we emphasize that improvements in source-model performance do not necessarily translate into better cross-scale transfer effectiveness. As shown in Sec. 3 of the original submission, source-model performance continues to improve during RL training, but target-side transfer effectiveness does not consistently increase. Therefore, selecting checkpoints solely based on source-side performance may fail to identify the most effective optimization signals.
In contrast, OptScout directly evaluates the transfer effectiveness of different checkpoints through lightweight target-side candidate evaluation and jointly selects the best checkpoint–magnitude pair, shifting the selection criterion from source-model performance to actual target-model optimization gains. Moreover, because Direct-OPD relies on iterative target-side optimization, evaluating multiple candidate teachers would require repeated distillation runs, resulting in substantial computational overhead.
Q1. How are Direct-RL Checkpoints Selected?
For the direct target-side RL baselines (GRPO, Dr.GRPO, and DAPO), we adopt a unified validation-based checkpoint selection and early-stopping protocol. The maximum training duration is 300 steps, with evaluation performed every 10 steps on a held-out validation set of 500 questions randomly sampled from MATH (Hendrycks et al., 2021) and DeepScaleR, without overlap with the training data. We select the checkpoint with the highest validation accuracy and stop training early if no improvement is observed over five consecutive evaluations.
All methods use the same validation set, evaluation interval, and selection criteria. The validation set is strictly disjoint from the training and final test sets, and none of the final test benchmarks are used for checkpoint selection.
Q2. Fixed Source Checkpoint vs. Checkpoint Scouting
To further isolate the contribution of Checkpoint Scouting from Magnitude Search, we fix the source RL checkpoints at Steps 10, 20, and 30 and perform magnitude search separately for each checkpoint. We use the same eight candidate magnitudes as full OptScout, while keeping target-side update construction, evaluation, and testing protocols unchanged.
Rebuttal table 16. Fixed source checkpoints with the same magnitude search vs. full OptScout.
| Method | AIME24 | AIME25 | MATH500 | AMC | OlympiadBench | Avg. |
|---|---|---|---|---|---|---|
| Fixed Step 10 | 40.42 | 32.50 | 85.75 | 73.12 | 57.20 | 57.80 |
| Fixed Step 20 | 54.38 | 38.75 | 88.08 | 74.92 | 58.61 | 62.95 |
| Fixed Step 30 | 45.00 | 34.58 | 86.84 | 78.09 | 57.94 | 60.49 |
| Full OptScout | 52.92 | 42.71 | 87.65 | 75.68 | 57.90 | 63.37 |
With the same eight candidate magnitudes, the fixed checkpoints achieve average accuracies of 57.80%, 62.95%, and 60.49%, respectively, demonstrating substantial variation in transfer effectiveness across source RL steps. Although Step 20 performs comparably to full OptScout, its effectiveness cannot be reliably predicted from the source RL training step alone. In contrast, OptScout automatically selects the checkpoint–magnitude pair using target-side reward feedback, achieving the highest five-benchmark average among these settings.
References
[1] Weak-to-Strong Generalization via Direct On-Policy Distillation, 2026.
[2] Editing Models with Task Arithmetic, ICLR 2023.
[3] TIES-Merging: Resolving Interference When Merging Models, NeurIPS 2023.
We thank Reviewer 7LM8 for the detailed and constructive feedback and for recognizing the value of our approach and empirical results. We address each concern below.
Weakness 1. Novelty and Contribution Attribution
To address the reviewer's concerns regarding the novelty of OptScout and the independent contributions of its core components, we clarify its differences from Direct-OPD [1], Task Arithmetic [2], and TIES [3]. We further conduct controlled ablation studies to isolate the contributions of Target-gradient Construction, Checkpoint Scouting, and Magnitude Search.
1. Differences from Direct-OPD [1]
The key differences between Direct-OPD and OptScout lie in checkpoint selection and target-side optimization.
(1) Checkpoint Scouting. Direct-OPD uses a source RL checkpoint at a predetermined training step as the teacher and transfers policy changes through iterative on-policy distillation. Our experiments reveal that cross-scale transfer effectiveness varies substantially across source checkpoints and does not improve monotonically with training steps (Sec. 3). OptScout addresses this issue by evaluating candidate checkpoints through lightweight target-side update construction and jointly selecting the best checkpoint–magnitude pair. In contrast, evaluating multiple candidate teachers within Direct-OPD's iterative distillation framework would require repeatedly running the distillation process, incurring substantial computational overhead.
(2) Target-side Optimization. Direct-OPD optimizes the target model through repeated rollouts and gradient updates. OptScout instead uses Low-Rank Gradient Probing to convert source RL policy shifts into target-specific sparse updates, with Cross-Split Consistency Filtering and forward-only candidate evaluation for update selection, followed by a single sparse parameter update. As reported in Table 2 of the original submission, OptScout achieves a 3.73× speedup over Direct-OPD on Qwen3-8B while improving average accuracy. Table 5 further shows a 25.3% reduction in peak per-GPU memory usage.
To verify the independent contributions of our core components, we conduct additional controlled ablations of Target-gradient Construction, Checkpoint Scouting, and Magnitude Search, as reported in Rebuttal tables 13, 14, and 16.
2. Differences from Task Arithmetic [2] and TIES [3]
(1) Research objectives and cross-scale transfer mechanisms. Task Arithmetic [2] and TIES [3] focus on editing and merging existing model parameter update vectors. In contrast, OptScout addresses cross-scale RL post-training by transferring optimization information accumulated during small-model RL to a larger model. While Task Arithmetic and TIES investigate how to manipulate existing model parameter updates, OptScout investigates how to extract optimization information from small-model RL and reconstruct effective updates in the larger model's parameter space.
(2) Update construction and filtering mechanisms. Task Arithmetic constructs updates through arithmetic operations on task vectors, while TIES employs magnitude-based trimming, sign conflict resolution, and consensus-based merging. In contrast, OptScout selects sparse parameter coordinates based on the consistency of target-model gradient responses across two prompt-disjoint data splits.
We additionally implement a TIES-style parameter aggregation baseline, retaining OptScout's policy-shift signals and Low-Rank Gradient Probing while replacing only the parameter aggregation rule. All other settings remain unchanged.
Rebuttal table 12. Comparison with TIES-style parameter aggregation on MATH500.
The results indicate that, in cross-scale RL transfer, cross-split consistency filtering constructs more effective target-side sparse update directions than TIES-style aggregation.