acceptodds
Under review as a conference paper at ICLR 2027

Train Small, Guide Large: Optimization Scouting for Efficient LLM Post-Training

Abstract

Reinforcement learning with verifiable rewards (RLVR) can substantially improve LLM reasoning, but its computational cost increases rapidly with model scale. This work investigates whether low-cost RL on a smaller source model can replace full RL training for a larger target model. We find that transferable optimization signals already emerge at relatively early stages of source-model RL and can be used to derive effective parameter updates without costly iterative optimization on the target model. Building on these findings, we propose Optimization Scouting (OptScout), a framework for reusing RL-induced optimization signals across model scales. OptScout extracts evolving policy shifts during source-model RL and translates them into stable, target-specific update directions through low-rank gradient probing and cross-split consistency filtering. It then uses forward-only reward evaluation to select the best update direction and magnitude, and stops scouting once further source-model RL no longer improves transfer performance. Finally, the selected update is applied to the target model only once. Experiments across different source-side RL algorithms and target-model settings show that OptScout achieves higher average performance than direct target-side RL on multiple reasoning benchmarks, while providing up to a \(6.32\times\) end-to-end speedup.

Primary area:
optimization
Keywords:
Reinforcement Learning with Verifiable Rewards, LLM Post-Training, Cross-Scale Transfer, Efficient Post-Training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7LM8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OptScout uses a short RL run on a small model to guide a larger model’s post-training. It converts the source model’s token-level policy shifts into target-model update directions through low-rank gradient probing and cross-split consistency filtering. It then selects a source checkpoint and update magnitude using reward-based evaluation. Experiments show competitive or better performance than direct RL, with substantial end-to-end time savings.

Critical strengths
  1. Using small-model RL signals to construct target-specific updates is a sensible approach to reducing post-training cost.

  2. The empirical results are promising, supported by runtime measurements, memory analysis, and component ablations.

Critical weaknesses
  1. The novelty over existing work needs clearer justification. Policy-shift transfer is already explored in [1], while sparse, sign-consistent updates are related to [2] and [3]. The target-gradient construction is a meaningful difference, but the experiments do not fully separate its contribution and that of checkpoint scouting from the candidate search.

  2. Comparisons with compute-matched target-side LoRA-RL or short-run optimization are missing. It is unclear how much of the efficiency gain requires small-model scouting rather than a cheaper target-side training setup. [1] also selects its teacher by source performance, although the paper argues that source performance does not reliably predict transfer quality.

References

[1] Weak-to-Strong Generalization via Direct On-Policy Distillation, 2026.

[2] Editing Models with Task Arithmetic, ICLR 2023.

[3] TIES-Merging: Resolving Interference When Merging Models, NeurIPS 2023.

Questions for authors
  1. How are direct-RL checkpoints selected?

  2. Are results available for a fixed early source checkpoint with the same magnitude search, to isolate the benefit of checkpoint scouting?

4 comments

We thank Reviewer 7LM8 for the detailed and constructive feedback and for recognizing the value of our approach and empirical results. We address each concern below.

Weakness 1. Novelty and Contribution Attribution

To address the reviewer's concerns regarding the novelty of OptScout and the independent contributions of its core components, we clarify its differences from Direct-OPD [1], Task Arithmetic [2], and TIES [3]. We further conduct controlled ablation studies to isolate the contributions of Target-gradient Construction, Checkpoint Scouting, and Magnitude Search.

1. Differences from Direct-OPD [1]

The key differences between Direct-OPD and OptScout lie in checkpoint selection and target-side optimization.

(1) Checkpoint Scouting. Direct-OPD uses a source RL checkpoint at a predetermined training step as the teacher and transfers policy changes through iterative on-policy distillation. Our experiments reveal that cross-scale transfer effectiveness varies substantially across source checkpoints and does not improve monotonically with training steps (Sec. 3). OptScout addresses this issue by evaluating candidate checkpoints through lightweight target-side update construction and jointly selecting the best checkpoint–magnitude pair. In contrast, evaluating multiple candidate teachers within Direct-OPD's iterative distillation framework would require repeatedly running the distillation process, incurring substantial computational overhead.

(2) Target-side Optimization. Direct-OPD optimizes the target model through repeated rollouts and gradient updates. OptScout instead uses Low-Rank Gradient Probing to convert source RL policy shifts into target-specific sparse updates, with Cross-Split Consistency Filtering and forward-only candidate evaluation for update selection, followed by a single sparse parameter update. As reported in Table 2 of the original submission, OptScout achieves a 3.73× speedup over Direct-OPD on Qwen3-8B while improving average accuracy. Table 5 further shows a 25.3% reduction in peak per-GPU memory usage.

To verify the independent contributions of our core components, we conduct additional controlled ablations of Target-gradient Construction, Checkpoint Scouting, and Magnitude Search, as reported in Rebuttal tables 13, 14, and 16.

2. Differences from Task Arithmetic [2] and TIES [3]

(1) Research objectives and cross-scale transfer mechanisms. Task Arithmetic [2] and TIES [3] focus on editing and merging existing model parameter update vectors. In contrast, OptScout addresses cross-scale RL post-training by transferring optimization information accumulated during small-model RL to a larger model. While Task Arithmetic and TIES investigate how to manipulate existing model parameter updates, OptScout investigates how to extract optimization information from small-model RL and reconstruct effective updates in the larger model's parameter space.

(2) Update construction and filtering mechanisms. Task Arithmetic constructs updates through arithmetic operations on task vectors, while TIES employs magnitude-based trimming, sign conflict resolution, and consensus-based merging. In contrast, OptScout selects sparse parameter coordinates based on the consistency of target-model gradient responses across two prompt-disjoint data splits.

We additionally implement a TIES-style parameter aggregation baseline, retaining OptScout's policy-shift signals and Low-Rank Gradient Probing while replacing only the parameter aggregation rule. All other settings remain unchanged.

Rebuttal table 12. Comparison with TIES-style parameter aggregation on MATH500.

MethodMATH500 mean@16 (%)
TIES-style Parameter Aggregation86.29
OptScout87.65

The results indicate that, in cross-scale RL transfer, cross-split consistency filtering constructs more effective target-side sparse update directions than TIES-style aggregation.

3. Additional Ablations: Independent Contributions of Core Components

We conduct two additional controlled ablation studies to examine the independent contributions of Target-gradient Construction, Checkpoint Scouting, and Magnitude Search. Due to time constraints, these experiments are evaluated only on MATH500 and use the same newly rerun source RL trajectory. Since this trajectory differs from that used in the original submission, we focus on the relative contributions of components under identical experimental conditions rather than comparing absolute results with the original submission.

(1) Target-gradient Construction Ablation.

Table 7 of the original submission already compares different update direction construction methods. To further isolate the contribution of Target-gradient Construction, we fix the source checkpoint (step 15) and update magnitude (η=0.005\eta=0.005), disable Checkpoint Scouting and Magnitude Search, and vary only the target-side update direction construction method while keeping all other settings unchanged.

Rebuttal table 13. Controlled ablation of target-side gradient construction on MATH500.

Direction ConstructionMATH500 mean@16 (%)
Full Gradient (All Params.)79.59
Full Gradient (Eligible Params.)83.90
Low-Rank Probe (Eligible Params.)84.49
Low-Rank + Magnitude Top-2%83.91
Low-Rank + Cross-Split Top-2%85.14

The results show that restricting updates to attention and MLP projection matrices improves performance, Low-Rank Gradient Probing further enhances the effectiveness of update directions, and Cross-Split Consistency Filtering outperforms magnitude-only sparse selection. Even without Checkpoint Scouting or Magnitude Search, target-side gradient construction and consistency filtering independently improve performance.

(2) Checkpoint Scouting × Magnitude Search.

We further conduct a 2×2 ablation by independently controlling Checkpoint Scouting and Magnitude Search. All experiments use Low-Rank + Cross-Split Top-2% update directions, with other settings held constant. The fixed-checkpoint setting uses the predetermined source RL checkpoint at Step 15, while the search mechanisms follow the original submission.

Rebuttal table 14. Ablation of checkpoint scouting and magnitude search on MATH500.

Checkpoint SelectionMagnitude SelectionMATH500 mean@16 (%)
Fixed CheckpointFixed η=0.005\eta=0.00585.14
Fixed CheckpointSearch 8 Magnitudes85.94
Checkpoint ScoutingFixed η=0.005\eta=0.00586.39
Checkpoint ScoutingSearch 8 Magnitudes87.69

The results show that Checkpoint Scouting and Magnitude Search each contribute independently, while their combination achieves the best performance, indicating their complementary effects. Together, these two ablation studies further validate the individual effectiveness of all three core components.

Weakness 2. Compute-matched Target-side RL and Teacher Selection

To address the reviewer's concerns regarding the lack of compute-matched target-side RL comparisons and the reliability of source-side teacher selection, we provide additional GRPO and LoRA-GRPO experiments and clarify how OptScout differs from Direct-OPD in checkpoint selection.

1. Compute-matched Target-side RL Comparison

To examine whether OptScout's efficiency advantage merely comes from reducing target-side training costs, we conduct additional experiments with compute-matched GRPO and LoRA-GRPO. Both methods perform RL directly on Qwen3-8B for approximately 7.6 hours, closely matching OptScout's 7.50-hour end-to-end runtime. All methods use the same hardware environment. For LoRA-GRPO, we follow the official verl recommendations and evaluate multiple hyperparameter configurations, selecting the best-performing setting with LoRA rank = 32, alpha = 64, and learning rate = 5 × 10⁻⁵. The remaining training settings are identical to GRPO.

Rebuttal table 15. Comparison with compute-matched target-side RL baselines on Qwen3-8B.

MethodAIME24AIME25MATH500AMCOlympiadBenchAvg.Time
Qwen3-8B (Base)41.2523.9680.2861.9745.8450.66–
Compute-matched GRPO42.0832.0883.3866.2750.8154.927.68 h
Compute-matched LoRA-GRPO42.0831.6783.6868.4550.0055.187.62 h
GRPO (Full-run)49.3835.8387.1475.7556.9761.0141.74 h
OptScout52.9242.7187.6575.6857.9063.377.50 h

Under comparable wall-clock budgets, compute-matched GRPO and LoRA-GRPO achieve average accuracies of 54.92% and 55.18%, respectively, substantially below OptScout's 63.37%. This indicates that simply shortening target-side RL training or using LoRA fine-tuning is insufficient to match OptScout's performance.

Furthermore, Figure 3 of the original submission shows that, on both Qwen3-8B and Qwen3-14B, OptScout achieves better MATH500 performance than the final models from full GRPO training while requiring substantially less wall-clock time. Together with the additional experiments, this further supports OptScout's performance advantage under limited computational budgets.

2. Direct-OPD Teacher Selection

We do not argue that source-side performance cannot be used for teacher selection. Rather, we emphasize that improvements in source-model performance do not necessarily translate into better cross-scale transfer effectiveness. As shown in Sec. 3 of the original submission, source-model performance continues to improve during RL training, but target-side transfer effectiveness does not consistently increase. Therefore, selecting checkpoints solely based on source-side performance may fail to identify the most effective optimization signals.

In contrast, OptScout directly evaluates the transfer effectiveness of different checkpoints through lightweight target-side candidate evaluation and jointly selects the best checkpoint–magnitude pair, shifting the selection criterion from source-model performance to actual target-model optimization gains. Moreover, because Direct-OPD relies on iterative target-side optimization, evaluating multiple candidate teachers would require repeated distillation runs, resulting in substantial computational overhead.

Q1. How are Direct-RL Checkpoints Selected?

For the direct target-side RL baselines (GRPO, Dr.GRPO, and DAPO), we adopt a unified validation-based checkpoint selection and early-stopping protocol. The maximum training duration is 300 steps, with evaluation performed every 10 steps on a held-out validation set of 500 questions randomly sampled from MATH (Hendrycks et al., 2021) and DeepScaleR, without overlap with the training data. We select the checkpoint with the highest validation accuracy and stop training early if no improvement is observed over five consecutive evaluations.

All methods use the same validation set, evaluation interval, and selection criteria. The validation set is strictly disjoint from the training and final test sets, and none of the final test benchmarks are used for checkpoint selection.

Q2. Fixed Source Checkpoint vs. Checkpoint Scouting

To further isolate the contribution of Checkpoint Scouting from Magnitude Search, we fix the source RL checkpoints at Steps 10, 20, and 30 and perform magnitude search separately for each checkpoint. We use the same eight candidate magnitudes as full OptScout, while keeping target-side update construction, evaluation, and testing protocols unchanged.

Rebuttal table 16. Fixed source checkpoints with the same magnitude search vs. full OptScout.

MethodAIME24AIME25MATH500AMCOlympiadBenchAvg.
Fixed Step 1040.4232.5085.7573.1257.2057.80
Fixed Step 2054.3838.7588.0874.9258.6162.95
Fixed Step 3045.0034.5886.8478.0957.9460.49
Full OptScout52.9242.7187.6575.6857.9063.37

With the same eight candidate magnitudes, the fixed checkpoints achieve average accuracies of 57.80%, 62.95%, and 60.49%, respectively, demonstrating substantial variation in transfer effectiveness across source RL steps. Although Step 20 performs comparably to full OptScout, its effectiveness cannot be reliably predicted from the source RL training step alone. In contrast, OptScout automatically selects the checkpoint–magnitude pair using target-side reward feedback, achieving the highest five-benchmark average among these settings.

References

[1] Weak-to-Strong Generalization via Direct On-Policy Distillation, 2026.

[2] Editing Models with Task Arithmetic, ICLR 2023.

[3] TIES-Merging: Resolving Interference When Merging Models, NeurIPS 2023.

Reviewer_ccCfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes OptScout, which replaces RL on a large target model with short RL on a small source model plus a single target update. During source RL (GRPO, DAPO or Dr.GRPO on Qwen3-1.7B), token-level log-probability shifts between the current and initial source policy are measured on target-generated trajectories and used as signed weights in an NLL objective on the target. The target-side gradient is estimated with a fixed rank-8 LoRA probe, filtered to the top-2% coordinates that agree across two prompt-disjoint splits, and the source checkpoint and step size are chosen by forward-only reward on a hard set. On Qwen3-8B, R1-Distill-7B and Qwen3-14B, the paper reports higher average accuracy than direct target-side RL on five math benchmarks, with 3.58–6.32× end-to-end speedup and about 25% lower peak memory.

Critical strengths

S1. A novel and potentially very practical idea. Separating "obtaining optimization information" from "turning it into a parameter update", and doing the second step only once, is a fresh angle on cross-scale transfer. If the gains hold under the checks requested below, the efficiency results are significant: 4.11–6.32× speedup on Qwen3-8B (Table 2, L324–338), and the full pipeline fits on Qwen3-14B where GRPO and Direct-OPD run out of memory (Table 5, L422–428). S2. Ablations that isolate the signal. A random signal gives no gain over the base model (50.23 vs. 50.66, Table 6, L435–437), indicating the improvement depends on the source-RL shifts. Cross-split filtering outperforms magnitude-based top-2% selection (63.37 vs. 60.91, Table 7, L443–451) while shrinking the update direction from 13.89 GB to 1.67 GB (L473–474).

Critical weaknesses

W1. The base-model numbers suggest that the 8,192-token response budget truncates many responses, which may confound the reported gains. Qwen3-14B is reported at 37.71 on AIME24 (Table 3, L349), below Qwen3-8B at 41.25 (Table 2, L329), while it is higher on AIME25 (30.21 vs. 23.96). The Qwen3 technical report lists Qwen3-14B at 79.3 (AIME'24) and 70.4 (AIME'25) in thinking mode, and at 31.7 and 23.3 in non-thinking mode. The numbers in this paper sit between these, and the paper does not state which mode or chat template is used. Training and evaluation both cap responses at 8,192 tokens (Table 9, L777; Table 11, L871), and candidate scoring uses the same cap (Table 10, L829). I therefore suspect that a large fraction of base-model responses are cut off before reaching a final answer. If so, a substantial part of the improvement, for OptScout and possibly for the RL baselines as well, may come from learning to produce shorter responses that fit the budget rather than from better reasoning. This matters for the central claim: a single small update that mainly shifts length or format behavior is a much weaker result than one that transfers reasoning ability.

W2. The evidence for "early emergence", and for some headline comparisons, is within evaluation noise. In Figure 2 (L150–161) and the accompanying text (L162–168), the 8B transfer changes from the Early to the Late stage are -0.55 and +0.07 points on MATH500, roughly the equivalent of three problems and less than one problem out of 500. In Table 1 (L172–176), consecutive checkpoints differ by up to 2.2 points (84.8 to 87.0) with no consistent trend. The paper reports no results over multiple seeds and no error bars or confidence intervals, and on the 30-problem AIME sets a single problem changes the score by 3.3 points. The Dr.GRPO comparison in Table 2 (L334–336: +0.55 on average, with -2.50 on AIME24) therefore cannot support a claim of superiority. Because early emergence is the first listed contribution (L099–101) and motivates the scouting design, it needs statistical support: multiple seeds for both the source RL and the transfer step, confidence intervals (e.g., bootstrap over problems), and significance tests for the stage-wise and method-wise differences.

Questions for authors

Q1. Token-shuffle ablation vs. magnitude selection. Table 6 reports an average of 20.38 for Token-shuffle (L438; text at L457–459), far below the base model (50.66, L435). However, the candidate magnitudes include eta = 0.001 (Table 10, L827), and the best magnitude is selected by reward on D_hard (Section 4.4, L310–316; Table 10, L829–831). An update with eta = 0.001 should leave the model nearly unchanged, so the selection procedure should return something close to the base model. How was 20.38 obtained? Were the same eta grid and the same D_hard selection applied to the ablation variants, or was eta fixed to the value chosen for the full method? If eta = 0.001 already changes the model this much, please report the norm of the parameter change relative to the weights, since that would mean the effective step sizes are much larger than the grid suggests. The answer affects how both the token-shuffle and random-signal ablations (L437–438) should be interpreted.

Q2. Evaluation mode and truncation (related to W1). Were the Qwen3 models run in thinking or non-thinking mode, and with which chat template? Please report the average response length and the truncation rate (the fraction of responses that hit the 8,192-token limit without a final answer) for the base model, each RL baseline, and OptScout on every benchmark. Accuracy restricted to non-truncated responses, or under a longer budget, would also help.

3 comments

We thank Reviewer ccCf for recognizing the novelty and practical value of OptScout. We address each concern below.

Weakness 1 / Q2. Evaluation Mode, Response Truncation, and Reasoning Improvements

To address the reviewer's concerns regarding the Qwen3 evaluation mode, response truncation under the 8192-token generation budget, and whether OptScout's performance gains primarily result from shorter responses, we clarify our evaluation settings and provide additional analyses of response lengths, truncation rates, accuracy on commonly non-truncated responses, and a qualitative reasoning example. Given the limited time available for the rebuttal, we focus our detailed analysis on Qwen3-8B, our primary experimental setting.

1. Evaluation Mode and Truncation Statistics

All evaluations use the native Qwen3 chat template with enable_thinking=True and the same prompt: Please reason step by step, and put your final answer within \boxed{}. All methods use the same tokenizer and answer verifier, with a maximum generation length of 8192 tokens and 16 independent responses per problem. A response is classified as truncated if it reaches the maximum generation length without producing a final answer in the required format.

We report the average response lengths and truncation rates of the base model, direct RL baselines, and corresponding OptScout models across five benchmarks.

Rebuttal table 5. Average response length and truncation rate under the 8192-token budget.

Each entry reports the average response length (tokens) / truncation rate (%).

MethodMax TokensAIME24AIME25MATH500AMCOlympiadBench
Base81927,487 / 40.21%7,741 / 45.21%4,312 / 7.73%6,455 / 25.68%6,453 / 31.94%
GRPO81926,651 / 35.63%7,057 / 42.29%3,192 / 4.65%5,026 / 15.44%5,174 / 22.07%
OptScout-GRPO81926,091 / 22.50%6,532 / 39.38%2,342 / 3.34%4,303 / 12.27%4,493 / 14.36%
Dr.GRPO81926,544 / 35.00%7,008 / 41.88%3,099 / 3.74%4,723 / 11.75%5,115 / 19.99%
OptScout-Dr.GRPO81926,530 / 29.58%6,957 / 38.54%2,723 / 3.18%4,755 / 13.63%4,793 / 16.64%
DAPO81926,418 / 31.04%6,897 / 42.29%2,988 / 3.54%4,813 / 16.11%5,063 / 18.19%
OptScout-DAPO81926,407 / 37.50%6,811 / 39.58%2,559 / 4.24%4,571 / 16.49%4,755 / 16.91%

The results confirm that the 8192-token generation budget introduces truncation effects. To further investigate whether OptScout's performance gains arise solely from reduced truncation, we evaluate all methods on the same subset of non-truncated responses.

2. Accuracy on Common Non-truncated Responses

We identify the common set of responses that are not truncated by any of the compared methods. Specifically, the same retained samples refer to response instances with identical problem IDs and response indices across all methods. A problem is retained if it has at least one common non-truncated response, and accuracy is computed based on these responses.

Rebuttal table 6. Accuracy on the common non-truncated response subset (%).

MethodAIME24AIME25MATH500AMCOlympiadBenchAvg.
Retained Problems17/3013/30471/50064/83467/675–
Base90.0059.3388.6184.0872.3578.87
GRPO98.3368.0092.8894.1279.5586.58
OptScout-GRPO86.1187.3393.4195.0478.6188.10
Dr.GRPO96.1197.3392.5495.8579.1492.19
OptScout-Dr.GRPO97.2298.0093.3995.3980.2592.85
DAPO96.6786.6792.0196.6679.5590.31
OptScout-DAPO97.7896.0093.1597.4680.0092.88

The results show that, even when considering only common non-truncated responses, all three OptScout variants outperform the base model on MATH500, AIME25, AMC, and OlympiadBench. This suggests that OptScout's performance gains also reflect improvements in reasoning ability.

3. Evaluation with a Longer Response Budget

Given the time constraints, we focus this additional evaluation on AIME24 and AIME25, where truncation is particularly pronounced under the 8192-token budget. We increase the maximum response length to 12288 tokens and re-evaluate the same models, keeping all other evaluation settings unchanged.

Rebuttal table 7. Performance on AIME24 and AIME25 with a 12288-token response budget (mean@16, %).

MethodAIME24AIME25
Base Qwen3-8B54.1740.63
GRPO59.1746.67
OptScout-GRPO64.1749.58

Under the longer generation budget, OptScout-GRPO continues to outperform both the base model and direct GRPO, further suggesting that its performance gains reflect improved reasoning ability. We note that this experiment only increases the evaluation-time response budget, while direct GRPO and Source RL training retain the original 8,192-token limit. Due to time constraints, we have not rerun the full RL training procedures with a longer generation budget.

4. Qualitative Analysis of Reasoning Improvements

To qualitatively examine whether OptScout's performance improvements also reflect enhanced reasoning ability, rather than merely changes in response length or truncation behavior, we inspect complete, non-truncated reasoning responses. We observe more accurate handling of mathematical constraints and reasoning steps in several examples.

Consider Problem 12 (index=11) from MATH500:

Let p(x)p(x) be a polynomial of degree 5 such that

p(n)=nn2−1,p(n)=\frac{n}{n^2-1},

for n=2,3,4,…,7n=2,3,4,\ldots,7. Find p(8)p(8).

Let Q(x)=(x2−1)p(x)−xQ(x)=(x^2-1)p(x)-x and R(x)=∏n=27(x−n)R(x)=\prod_{n=2}^{7}(x-n). The base model incorrectly assumes Q(x)=R(x)(x−r)Q(x)=R(x)(x-r), overlooking the leading coefficient, and obtains the incorrect answer 721/9721/9. In contrast, OptScout-GRPO retains the general form Q(x)=R(x)(ax+b)Q(x)=R(x)(ax+b) and uses Q(1)=−1Q(1)=-1 and Q(−1)=1Q(-1)=1 to derive the correct answer 3/563/56.

Both responses are complete and non-truncated, with lengths of 5773 and 7225 tokens for the base model and OptScout-GRPO, respectively. This example illustrates that OptScout's improvements can also manifest as more accurate mathematical reasoning, rather than merely shorter responses.

Weakness 2. Early Emergence and Statistical Reliability

To address the reviewer's concerns regarding insufficient statistical evidence for Early Emergence, potential evaluation noise in transfer performance across checkpoints, and the lack of multiple random seeds and confidence intervals for the main method comparisons, we conduct cross-stage Source RL transfer evaluations across independent seeds, followed by multi-seed method comparisons and statistical analyses of performance differences.

1. Early/Middle/Late Transfer across Independent Source RL Seeds

Early Emergence refers to the emergence of effective cross-scale transferable signals during the early stages of Source RL, rather than requiring Early-stage checkpoints to consistently outperform later checkpoints. We examine its reproducibility using two independent Source RL seeds (20261001 and 20261002), evaluating the cross-scale transfer performance of checkpoints from different training stages.

Rebuttal table 8. Transfer performance across Source RL stages and independent seeds on MATH500 (%).

StageSource RL StepsSeed 20261001Seed 20261002
Early5–2587.9587.56
Middle50–7087.2086.78
Late95–11585.8187.44

We further assess the statistical reliability of stage-wise differences using a problem-level paired bootstrap with 100,000 resamples. The 95% confidence intervals of the paired performance differences are reported below.

Rebuttal Table 9. Paired statistical comparisons across Source RL stages on MATH500.

Source RL SeedComparisonDifference (pp)95% CI
20261001Early − Middle+0.75[−0.20, 1.75]
20261001Early − Late+2.14[1.13, 3.23]
20261002Early − Middle+0.78[−0.30, 1.90]
20261002Early − Late+0.12[−0.95, 1.20]

Across the two independent Source RL seeds, Early-stage checkpoints achieve transfer accuracies of 87.95% and 87.56%, respectively. These results indicate that effective cross-scale transferable signals emerge early in Source RL, while continued source-side optimization does not necessarily lead to further improvements in target-side transfer performance.

2. Multi-seed Evaluation and Method-wise Statistical Analysis

To evaluate the reproducibility of OptScout's performance gains, we conduct two additional independent runs using random seeds 20261113 and 20261114 for direct GRPO, direct Dr.GRPO, OptScout-GRPO, and OptScout-Dr.GRPO. Together with the original results, we report the mean and sample standard deviation across three runs.

Rebuttal table 10. Mean and sample standard deviation across three runs on five benchmarks (%).

MethodAIME24AIME25MATH500AMCOlympiadBenchAvg.
GRPO50.91±2.6439.31±3.1286.65±0.5274.95±0.7957.12±0.2861.79±0.70
OptScout-GRPO53.82±3.1240.35±2.0588.11±0.4075.63±0.1658.17±0.2663.22±0.60
Dr.GRPO53.68±1.9239.38±2.9287.38±0.4076.06±2.1558.21±0.6462.94±0.89
OptScout-Dr.GRPO55.00±2.5342.99±1.1588.08±0.1577.16±1.1558.38±0.4564.32±0.14

Avg. denotes the equally weighted macro-average across the five benchmarks.

The results demonstrate consistent performance gains under both GRPO and Dr.GRPO settings. Compared with their respective direct RL baselines, OptScout-GRPO and OptScout-Dr.GRPO improve the five-benchmark macro-average accuracy by 1.43 and 1.38 percentage points, respectively. The complete per-run results are provided in our response to Reviewer 2cNJ (Rebuttal tables 1–2).

To further quantify uncertainty, we compute Welch 95% confidence intervals for the two method-wise comparisons, obtaining [−0.06, 2.92] and [−0.78, 3.54], respectively. The corresponding exact two-sided unpaired permutation tests both yield p=0.10p=0.10.

Regarding the robustness of the transfer step, we conduct sensitivity analyses on candidate-evaluation samples and probing initialization. Detailed results are provided in our response to Reviewer 2cNJ (Rebuttal tables 3–4).

Q1. Token-shuffle Ablation and Magnitude Selection

To address the reviewer's concerns regarding whether the Token-shuffle ablation result is affected by magnitude selection, whether different signals follow the same search procedure, and how small values of η\eta affect parameter updates, we conduct additional experiments with a unified magnitude search and analyze parameter update norms.

Table 6 in the original paper was designed to compare the effectiveness of different policy-shift signals. Therefore, all signal ablations used the same source checkpoint and a fixed magnitude of η=0.008\eta=0.008 selected by the full OptScout method, without independently searching the magnitude for each signal.

We repeat magnitude selection using the same eight candidate magnitudes as OptScout. The selected magnitudes are η=0.001\eta=0.001 for Token-shuffle and η=0.002\eta=0.002 for Random. Positive-only and Negative-only also follow the same search procedure. The results are reported below.

Rebuttal table 11. Signal ablations with independent magnitude selection on five benchmarks (%).

SignalAIME24AIME25MATH500AMCOlympiadBenchAvg.
Base Qwen3-8B41.2523.9680.2861.9745.8450.66
Random37.7126.0480.0961.3045.9450.22
Token-shuffle31.8820.4278.5958.7344.1146.75
Positive-only55.8334.5886.8576.2056.3661.96
Negative-only50.0036.6787.0375.9857.4261.42
Full Shift52.9242.7187.6575.6857.9063.37

We further analyze the parameter update norms. The L2 norms of the Token-shuffle and Random update directions are 312.840 and 313.044, respectively. At a common reference magnitude of η=0.001\eta=0.001, their corresponding actual update L2 norms are 0.3128 and 0.3130, accounting for approximately 0.0127% of the full model weight L2 norm (2459.622).

The results indicate that, after applying a unified magnitude selection procedure, the overall performance trends remain consistent with those in the original Table 6: Random remains close to the base model, Token-shuffle remains below the base model, and Positive-only, Negative-only, and Full Shift all improve performance, with Full Shift achieving the best five-benchmark average accuracy. This further supports the effectiveness of combining positive and negative policy-shift signals.

Reviewer_2cNJOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OptScout, a framework that leverages RLVR-induced policy shifts from a smaller model to guide efficient post-training of a larger target model. It extracts token-level policy shifts, uses low-rank gradient probing to construct candidate update directions, and applies cross-split consistency filtering and reward-based selection to identify a single parameter update.

Experiments across multiple target models, RL algorithms, and benchmarks show that OptScout achieves competitive or improved performance while reducing computational costs. On Qwen3-8B, it achieves 4.11×–6.32× speedups over direct RL training. Ablation studies further examine the contributions of its key components. Overall, the work presents a promising approach to reducing the cost of LLM post-training.

Critical strengths

Strength 1: A promising approach to reducing the computational cost of LLM post-training.

OptScout converts RL-induced policy shifts from a smaller model into a one-shot parameter update for a larger target model, avoiding the need for full-scale iterative RL optimization on the target. This design directly addresses a key computational bottleneck in LLM post-training and has clear practical value. The authors report 4.11×–6.32× end-to-end speedups on Qwen3-8B, with similar trends observed on other target models. If these gains can be consistently reproduced across random seeds and a broader range of tasks, the approach could have significant practical impact.

Strength 2: Useful empirical analysis of the proposed components.

The authors conduct systematic ablation studies to investigate the contribution of policy-shift signals and target-space update construction. For example, random signals achieve an average score of 50.23, comparable to the base model's 50.66, whereas the full policy-shift signal achieves 63.37. The direction-construction experiments also show that cross-split consistency filtering outperforms selecting coordinates solely by response magnitude. These results provide empirical support for key design choices rather than merely demonstrating improvements in overall performance. Furthermore, evaluations on additional target models and the MBPP code-generation benchmark provide preliminary evidence that the efficiency benefits extend beyond the primary mathematical reasoning setting.

Critical weaknesses

OptScout selects the best update from multiple source checkpoints and candidate magnitudes, but the stability of this selection process remains unclear. Although the authors provide sensitivity analyses for the sparsity ratio and number of probing trajectories, the main results lack multi-seed evaluations and uncertainty estimates. For example, the macro-average improvement over direct Dr.GRPO on Qwen3-8B is only 0.55 percentage points. I encourage the authors to report results across multiple random seeds and assess the sensitivity of the selected updates to probing initialization and candidate-evaluation samples. This would strengthen the evidence for the reproducibility and robustness of the reported gains.

Questions for authors

The paper evaluates candidate updates using a fixed hard-example set and reports sensitivity analyses for the retention ratio and number of probing trajectories. Could the authors provide independent runs across multiple random seeds, along with corresponding variance estimates or confidence intervals? Additionally, how stable are the selected update directions and resulting performance when the low-rank gradient-probing initialization or candidate-evaluation set is changed? These results would help establish the reliability of OptScout's candidate-selection mechanism and the reproducibility of its reported performance gains.

2 comments

We thank Reviewer 2cNJ for recognizing OptScout's effectiveness in reducing LLM post-training costs. We address each concern below.

Weakness. Stability and Reproducibility

To address the reviewer's concerns regarding the reproducibility and robustness of OptScout, we conduct three additional studies: Multi-seed Evaluation, Candidate-Evaluation Sensitivity, and Probing Initialization Sensitivity.

1. Multi-seed Evaluation

To address the concern that the original 0.55 percentage-point improvement of OptScout-Dr.GRPO over direct Dr.GRPO may be affected by randomness, we conduct two additional runs with different random seeds (20261113 and 20261114) for direct GRPO, direct Dr.GRPO, OptScout-GRPO, and OptScout-Dr.GRPO. Each additional direct-RL run repeats target-side training, while each OptScout run independently retrains the source model and performs target-side update construction and candidate selection.

Rebuttal table 1. Results across three runs on five benchmarks.

MethodRun / SeedAIME24AIME25MATH500AMCOlympiadBenchAvg.
GRPOOriginal49.3835.8387.1475.7556.9761.01
GRPO2026111349.3841.8886.7174.9256.9461.97
GRPO2026111453.9640.2186.1074.1757.4462.38
OptScout-GRPOOriginal52.9242.7187.6575.6857.9063.37
OptScout-GRPO2026111357.2939.3888.3375.4558.1863.73
OptScout-GRPO2026111451.2538.9688.3675.7558.4262.55
Dr.GRPOOriginal54.7940.6387.3577.9457.7663.69
Dr.GRPO2026111351.4636.0487.8076.5157.9361.95
Dr.GRPO2026111454.7941.4687.0073.7258.9463.18
OptScout-Dr.GRPOOriginal52.2944.1788.1878.1658.3864.24
OptScout-Dr.GRPO2026111357.2941.8887.9077.4157.9364.48
OptScout-Dr.GRPO2026111455.4242.9288.1575.9058.8264.24

Rebuttal table 2. Mean and sample standard deviation across three runs.

MethodAIME24AIME25MATH500AMCOlympiadBenchAvg.
GRPO50.91±2.6439.31±3.1286.65±0.5274.95±0.7957.12±0.2861.79±0.70
OptScout-GRPO53.82±3.1240.35±2.0588.11±0.4075.63±0.1658.17±0.2663.22±0.60
Dr.GRPO53.68±1.9239.38±2.9287.38±0.4076.06±2.1558.21±0.6462.94±0.89
OptScout-Dr.GRPO55.00±2.5342.99±1.1588.08±0.1577.16±1.1558.38±0.4564.32±0.14

The results show that OptScout consistently improves performance under both GRPO and Dr.GRPO settings. As shown in Tables 1 and 2, all three OptScout-GRPO runs outperform the corresponding direct GRPO runs in five-benchmark macro-average accuracy. The same trend holds for OptScout-Dr.GRPO compared with direct Dr.GRPO. These results indicate that the performance gains are not limited to a single run, providing further empirical evidence for the reproducibility of OptScout.

2. Sensitivity to Candidate-Evaluation Samples

To investigate the sensitivity of candidate selection to evaluation samples, we use four different sampling seeds (20261113, 20261114, 20261115, and 20261116) to independently sample 500-example candidate-evaluation pools from the training candidate pool. Following the original procedure, we evaluate the initial target policy on each pool and use incorrectly answered questions to construct the corresponding DhardD_{\mathrm{hard}}. We then independently rerun the complete OptScout pipeline for each evaluation set, keeping all other settings unchanged. DhardD_{\mathrm{hard}} Accuracy denotes the accuracy of the selected model on the corresponding hard set.

Rebuttal table 3. Sensitivity to candidate-evaluation samples.

Run / Sampling SeedDhardD_{\mathrm{hard}} Accuracy (%)MATH500 mean@16 (%)
Original setting—87.65
2026111344.0487.51
2026111447.3288.38
2026111546.4987.65
2026111642.9687.51
Mean ± Std. (n=4)—87.76±0.42

Across the four additional candidate-evaluation sets, the selected models achieve MATH500 mean@16 scores of 87.51%–88.38%, with a mean and sample standard deviation of 87.76±0.42%, close to the original result of 87.65%. This statistic is calculated over the four additional runs, excluding the original setting. These results indicate that OptScout maintains relatively stable downstream performance across different candidate-evaluation samples.

3. Sensitivity to Probing Initialization

To assess the impact of probing initialization, we independently run the complete OptScout pipeline with four different initialization seeds (20260905, 20260906, 20260907, and 20260908), keeping all other experimental settings unchanged. We use Direction Cosine, Mask Jaccard, and MATH500 mean@16 to measure the similarity of parameter update directions, the overlap of sparse parameter masks, and the final model performance, respectively.

Rebuttal table 4. Sensitivity to probing initialization.

Probing SeedDirection CosineMask JaccardMATH500 mean@16 (%)
Original setting——87.65
202609050.0079840.24547387.74
20260906-0.0000460.24971487.61
202609070.0053160.24853287.91
202609080.0012770.24741387.84
Mean ± Std. (n=4)0.003633±0.0036900.247783±0.00180487.78±0.13

Direction Cosine and Mask Jaccard are computed relative to the original setting. All means and sample standard deviations are calculated over the four additional initializations only (n=4n=4), excluding the original setting.

The results show that different probing initializations affect the specific parameter update directions, while still identifying partially overlapping sparse parameter coordinates. The resulting update vectors exhibit substantial differences in the overall parameter space. Nevertheless, the Jaccard similarity of the Global Top-2% masks remains within 0.245–0.250, corresponding to approximately 39%–40% overlap in selected parameter coordinates, substantially higher than the theoretical expectation for two independently sampled random Top-2% masks (approximately 0.010).

More importantly, the variation in update directions does not translate into substantial variation in downstream performance. Across the four additional initializations, MATH500 mean@16 remains within 87.61%–87.91%, with a mean and sample standard deviation of 87.78±0.13%, close to the original result of 87.65%. These findings indicate that OptScout's robustness lies in its final optimization performance rather than in producing identical update directions across different initializations.

Comments on the paper

1 comment

Dear Area Chairs and Reviewers,

We sincerely thank you for your time and thoughtful reviews. We greatly appreciate the reviewers' recognition of our work as a promising approach to reducing the computational cost of LLM post-training (Reviewer 2cNJ), a novel and potentially very practical idea (Reviewer ccCf), and a sensible approach with promising empirical results (Reviewer 7LM8).

Unlike conventional RL methods that rely on costly iterative optimization on large models, our method efficiently transfers optimization information from small-model RL to larger models, requiring only a single sparse parameter update on the target model to achieve higher average accuracy, with up to a 6.32× end-to-end post-training speedup and approximately 25.3% lower peak GPU memory usage. In our responses, we have carefully addressed all reviewers' concerns, clarified important methodological details, and conducted additional experiments to strengthen our findings:

1. Additional Experiments and Analyses

To further strengthen the empirical evaluation of our method, we conduct additional experiments across multiple random seeds, evaluation settings, and computational budgets.

(1) Multi-seed evaluation: Our method consistently achieves performance improvements across different random seeds and RL algorithms, providing further evidence of its stability and reproducibility. (Reviewers 2cNJ and ccCf)

(2) Sensitivity analyses: Experiments with different candidate-evaluation samples and probing initializations demonstrate the robustness of our method's final performance. (Reviewer 2cNJ)

(3) Reasoning ability analysis: Evaluations on common non-truncated responses and under a longer generation budget provide further evidence that our method effectively improves reasoning ability. (Reviewer ccCf)

(4) Early emergence analysis: Independent source RL runs further demonstrate that effective cross-scale transferable optimization signals can emerge during the early stages of RL training. (Reviewer ccCf)

(5) Compute-matched RL comparison: Our method substantially outperforms compute-matched GRPO and LoRA-GRPO, further demonstrating its computational efficiency advantage. (Reviewer 7LM8)

2. Additional Ablation Studies

To provide deeper insights into our method, we conduct additional controlled ablation studies to examine the effectiveness and complementary contributions of its key components.

(1) Policy-shift signal ablations: Additional ablation experiments further validate the effectiveness of cross-scale transferable optimization information. (Reviewer ccCf)

(2) TIES-style comparison: Additional experiments demonstrate that Cross-Split Consistency Filtering is more effective than TIES-style parameter aggregation. (Reviewer 7LM8)

(3) Independent contributions of core components: Controlled ablations demonstrate the individual and complementary contributions of Target-gradient Construction, Checkpoint Scouting, and Magnitude Search. (Reviewer 7LM8)

3. Clarifications

We further clarify important methodological distinctions, evaluation protocols, and implementation details to facilitate a clearer understanding of our approach.

(1) Evaluation protocol: We provide further details on the reasoning mode, truncation definition, mean@16 evaluation metric, and related implementation settings. (Reviewer ccCf)

(2) Distinctions from existing work: We further clarify the key differences between our method and Direct-OPD, Task Arithmetic, and TIES, particularly in cross-scale optimization transfer and target-specific update construction. (Reviewer 7LM8)

(3) Checkpoint selection: We clarify the checkpoint selection procedures for direct RL baselines and further demonstrate the advantages of Checkpoint Scouting over fixed source-model checkpoints. (Reviewer 7LM8)

We believe these additional experiments, ablation studies, and methodological clarifications substantially strengthen the empirical foundation of our work and provide stronger evidence for its effectiveness, computational efficiency, and robustness.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.