- Rating
- 3(+1)
- Confidence
- 1(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
To solve the high cost and unreliability of computing gradients through long reasoning traces, this paper introduces three gradient proxies: a direct-answer mode (GP-DA), a differentiable continuous-step model (GP-LR), and a hybrid combination (GP-Hybrid). Evaluated across various model lifecycle stages on R1 and Llama models, these proxies cut computational expenses while matching or improving task performance, offering an efficient alternative for gradient-based optimization.
Critical strengths
1 The problem setting is new to me and quite interesting: is it possible to get cheaper proxy gradients for training?
2 There seems to be a significant wall-clock speedup compared to the reference (naive) baseline.
Critical weaknesses
1 All reasoning models are now trained using RLVR, which seems to be missing here. RLVR is also potentially the setting where this method could shine the brightest, as it is the most expensive post-training stage.
2 There are no details on how the checkpoints were chosen on the main results. For example, were all methods given an equal amount of compute and the best checkpoint selected based on a validation set?
3 The datasets and domains need to be expanded, ideally using a more recent base model. This would help demonstrate that the method remains relevant on modern models across diverse settings.
4 There are some inconsistencies in the empirical values. For example:
Table 1 and Table 3 contradict each other. In Table 1, teacher-forcing is 8–38x cheaper than GP-LR. In Table 3, for the same model and the same variable, GP-LR is cheaper than teacher-forcing AT.
The authors should go through these carefully and provide a clear explanation. These discrepancies raise concerns about potential cherry-picking of results.
Questions for authors
None
5 comments
Response to Reviewer y6FF (Part 2/2)
W3. Coverage and newer models (continued)
Safety adversarial training.
| Method | AutoRAN ASR (%) ↓ | Time (% baseline) ↓ |
|---|---|---|
| Teacher-Forcing adversarial training | 66.2 | 100.0 |
| with GP-DA | 65.6 | 21.3 |
| with GP-LR | 63.0 | 56.2 |
| with GP-Hybrid | 60.4 | 76.9 |
Test-time reasoning.
| Method | AMC accuracy (%) ↑ | Time (% baseline) ↓ |
|---|---|---|
| ∇-Reasoner | 78.3 | 100.0 |
| with GP-DA | 77.1 | 8.1 |
| with GP-LR | 83.1 | 34.7 |
| with GP-Hybrid | 81.9 | 41.9 |
Note. Time is reported as a percentage of the corresponding baseline’s time, with each baseline set to 100%.
These results show that the utility of gradient proxies extends beyond the original smaller backbones to the newer, substantially larger Qwen3.8-27B. All three variants improve attack resistance in safety adversarial training, while GP-LR and GP-Hybrid improve test-time reasoning accuracy. Together with R1-Distill-Qwen-1.5B and Llama-3.1-8B-Instruct, these experiments provide evidence across complementary 1.5B, 8B, and 27B scales, supporting the applicability of our approach to newer and larger reasoning models in both training and inference.
W4. Timing consistency and result selection
Timing breakdown. Thank you for identifying this discrepancy. We checked the timing procedure and found an inconsistent accounting boundary in Table 1. We apologize for this confusion.
Starting from a question, obtaining the Teacher-Forcing gradient requires a pipeline consisting of generating a CoT → evaluating the answer loss conditioned on that CoT → backpropagating to the input. In Table 1 of the original manuscript, the Teacher-Forcing and GP-DA entries only reported backward time, whereas GP-LR included its forward computation. The revised Table 1 reports complete gradient-query time, including forward and backward computation (lines 243–245). Below, forward time includes reasoning and answer-loss evaluation; values are seconds per example.
| Setting | Method | Forward | Backward | Total |
|---|---|---|---|---|
| R1 / GSM8K | Teacher-Forcing | 6.320 | 0.089 | 6.409 |
| GP-DA | 0.090 | 0.087 | 0.177 | |
| GP-LR | 0.971 | 2.379 | 3.350 | |
| GP-Hybrid | 1.006 | 2.417 | 3.423 | |
| R1 / Jailbreak | Teacher-Forcing | 5.984 | 0.188 | 6.172 |
| GP-DA | 0.077 | 0.051 | 0.128 | |
| GP-LR | 0.593 | 0.931 | 1.524 | |
| GP-Hybrid | 0.613 | 0.966 | 1.579 |
GP-LR takes 52.3% and 24.7% of Teacher-Forcing's query time on GSM8K and Jailbreak (1.9× and 4.0× speedups). With consistent accounting, GP-LR is faster than Teacher-Forcing in both Table 1 and Table 3 (safety adversarial training). We have corrected Table 1 and updated the accompanying discussion (lines 246–247). Our proposed methods are still more efficient, and the conclusion remains unchanged.
Addressing the concern about cherry-picking. Beyond correcting the timing discrepancy, we make result selection explicit and report repeated runs. In the R1 safety comparison, Teacher Forcing adversarial training and all proxy variants run for six epochs, with checkpoints selected by the lowest STAR-1 validation loss. Prompts are likewise selected on validation data, not test scores. These rules are now specified in Appendix D (Table 6, lines 814–829).
We additionally repeated safety adversarial training and prompt optimization on Llama with three seeds, applying validation-based selection within each run and reporting held-out mean ± standard error across all three runs, rather than selecting the best seed. The baselines are Teacher Forcing adversarial training and GReaTer, respectively:
| Llama task / metric | Baseline | GP-DA | GP-LR | GP-Hybrid |
|---|---|---|---|---|
| Prompt optimization / GSM8K accuracy ↑ | 84.26 ± 0.67 | 84.16 ± 0.59 | 84.79 ± 0.30 | 84.51 ± 0.30 |
| Safety AT / AutoRAN ASR@7 ↓ | 52.81 ± 8.83 | 44.56 ± 5.97 | 42.84 ± 6.43 | 41.25 ± 5.97 |
Prompt optimization remains competitive across seeds, and all proxies achieve lower mean safety ASR. Together, the explicit selection rules and results across all seeds address the concern that the findings depend on choosing favorable test checkpoints or individual runs.
Dear Reviewer,
Thank you again for your time and thoughtful feedback. As the discussion period draws to a close, we would appreciate your thoughts on our responses and additional evidence. In particular, please let us know whether any of your concerns remain unresolved, so that we can address them while the discussion is still open.
Thank you for considering our response.
Response to Reviewer y6FF (Part 1/2)
We thank the reviewer for finding the problem interesting and recognizing the computational savings. We clarify the scope and evaluation protocols and provide additional timing and three-seed results below.
Line and table numbers below refer to the revised manuscript.
W1. Applicability to RLVR
We agree that RLVR is important. Our current method replaces gradients with respect to inputs or intermediate representations; extending it to the policy-parameter gradients used in RLVR requires further investigation. As the policy evolves, a separately prepared proxy such as GP-LR may also require repeated adaptation, whose cost and effect on training stability would need to be evaluated. Improving RLVR efficiency would be highly valuable given its substantial training cost. However, this extension is beyond the scope of the present work, and we leave the investigation of its feasibility to future research.
The practical contribution is to reduce costly gradient queries within both training and inference pipelines. In selective SFT, proxies make supervision selection cheaper; in safety adversarial training, they reduce the cost of constructing input perturbations while retaining the target model’s parameter updates. Prompt optimization and test-time reasoning benefit from cheaper prompt or response updates. Across these applications, shortening the reasoning computation reduces the cost of gradient guidance while preserving the surrounding procedure. These benefits extend across data selection, training, adaptation, and inference. We leave the extension to RLVR to future work, as adapting proxies to an evolving policy and assessing their computational costs require substantial further investigation.
W2. Selection and comparison protocols
We agree that the selection rules should be explicit and have added Selection and evaluation to Appendix D (Table 6, lines 814–829):
For the R1 Safety AT comparison, Teacher Forcing adversarial training and all proxy variants run for six epochs, selecting the checkpoint with the lowest STAR-1 validation loss for held-out evaluation. Prompt selection likewise uses validation performance, not test scores.
We compare methods under the prescribed training or search workloads, rather than equal wall-clock budgets, and measure the resulting runtime. Under these settings, our proxies reduce gradient-computation time, as reported in Tables 2–5.
W3. Coverage and newer models
We would like to clarify the breadth of the current evaluation. The four case studies use gradient guidance for distinct purposes, spanning supervision selection, safety adversarial training, prompt optimization, and test-time reasoning. The benchmarks also extend beyond mathematics. For example, selective SFT includes out-of-domain evaluation on GPQA-Diamond, safety training is evaluated on both attack resistance (AutoRAN) and over-refusal on benign requests (XSTest), and prompt optimization includes five non-mathematical BBH tasks covering linguistic, causal, and commonsense reasoning. Across these settings, our results show that cheaper gradient guidance can retain practical utility across different optimization procedures and reasoning domains.
Evaluation on modern models. We agree with the reviewer that evaluation on “modern models across diverse settings” would strengthen the evidence. Our original model choices followed the scale used by recent, strong gradient-based baselines—GReaTer (ICLR 2025), selective SFT (ICLR 2026), and ∇-Reasoner (ICLR 2026)—whose gradient computations use backbones no larger than 9B. Nevertheless, in response to this suggestion, we extended safety adversarial training and ∇-Reasoner, together with our proxy variants, to the newer, substantially larger Qwen3.8-27B, covering both training and inference. Extending selective SFT and prompt optimization to 27B would exceed our available computational budget during the discussion period.