acceptodds
Under review as a conference paper at ICLR 2027

Computationally Efficient Gradient Proxies for Large Reasoning Models

Abstract

Gradient-based optimization supports applications throughout the development and deployment of large reasoning models (LRMs), but repeatedly obtaining gradients through explicit reasoning traces can be expensive. Inspired by recent advances in efficient reasoning, we introduce computationally efficient gradient proxies using shorter reasoning traces to obtain useful optimization guidance for the original model. We instantiate three proxies by omitting explicit reasoning, reasoning in continuous latent space, or combining these two approaches. We assess the quality and cost of these gradient proxies and evaluate their performance in four representative downstream tasks across stages of model development and deployment, including data preparation, training, task adaptation, and inference. Extensive experiments demonstrate improved performance with lower computational cost in various tasks, supporting the generality of gradient proxies across these distinct optimization problems.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Reasoning Models, Gradient Estimation, Gradient Proxies, Efficient Reasoning, Chain-of-Thought

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_y6FFOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To solve the high cost and unreliability of computing gradients through long reasoning traces, this paper introduces three gradient proxies: a direct-answer mode (GP-DA), a differentiable continuous-step model (GP-LR), and a hybrid combination (GP-Hybrid). Evaluated across various model lifecycle stages on R1 and Llama models, these proxies cut computational expenses while matching or improving task performance, offering an efficient alternative for gradient-based optimization.

Critical strengths

1 The problem setting is new to me and quite interesting: is it possible to get cheaper proxy gradients for training?

2 There seems to be a significant wall-clock speedup compared to the reference (naive) baseline.

Critical weaknesses

1 All reasoning models are now trained using RLVR, which seems to be missing here. RLVR is also potentially the setting where this method could shine the brightest, as it is the most expensive post-training stage.

2 There are no details on how the checkpoints were chosen on the main results. For example, were all methods given an equal amount of compute and the best checkpoint selected based on a validation set?

3 The datasets and domains need to be expanded, ideally using a more recent base model. This would help demonstrate that the method remains relevant on modern models across diverse settings.

4 There are some inconsistencies in the empirical values. For example:

Table 1 and Table 3 contradict each other. In Table 1, teacher-forcing is 8–38x cheaper than GP-LR. In Table 3, for the same model and the same variable, GP-LR is cheaper than teacher-forcing AT.

The authors should go through these carefully and provide a clear explanation. These discrepancies raise concerns about potential cherry-picking of results.

Questions for authors

None

5 comments
AuthorsOpenReview ↗

Response to Reviewer y6FF (Part 1/2)

We thank the reviewer for finding the problem interesting and recognizing the computational savings. We clarify the scope and evaluation protocols and provide additional timing and three-seed results below.

Line and table numbers below refer to the revised manuscript.

W1. Applicability to RLVR

We agree that RLVR is important. Our current method replaces gradients with respect to inputs or intermediate representations; extending it to the policy-parameter gradients used in RLVR requires further investigation. As the policy evolves, a separately prepared proxy such as GP-LR may also require repeated adaptation, whose cost and effect on training stability would need to be evaluated. Improving RLVR efficiency would be highly valuable given its substantial training cost. However, this extension is beyond the scope of the present work, and we leave the investigation of its feasibility to future research.

The practical contribution is to reduce costly gradient queries within both training and inference pipelines. In selective SFT, proxies make supervision selection cheaper; in safety adversarial training, they reduce the cost of constructing input perturbations while retaining the target model’s parameter updates. Prompt optimization and test-time reasoning benefit from cheaper prompt or response updates. Across these applications, shortening the reasoning computation reduces the cost of gradient guidance while preserving the surrounding procedure. These benefits extend across data selection, training, adaptation, and inference. We leave the extension to RLVR to future work, as adapting proxies to an evolving policy and assessing their computational costs require substantial further investigation.

W2. Selection and comparison protocols

We agree that the selection rules should be explicit and have added Selection and evaluation to Appendix D (Table 6, lines 814–829):

TaskSelection and evaluation
Safety ATSelect the checkpoint on validation data, then evaluate on held-out data.
Prompt optimizationOptimize on the support set, select on a separate validation set, and evaluate on the test set; model parameters remain fixed.
Selective SFTCompute attribution on training data and follow the prescribed SFT duration; no validation-based checkpoint search.
Test-time reasoningUse a fixed model and prescribed search configuration. Select responses using the algorithm's reward function; reference answers are used only for final grading.

For the R1 Safety AT comparison, Teacher Forcing adversarial training and all proxy variants run for six epochs, selecting the checkpoint with the lowest STAR-1 validation loss for held-out evaluation. Prompt selection likewise uses validation performance, not test scores.

We compare methods under the prescribed training or search workloads, rather than equal wall-clock budgets, and measure the resulting runtime. Under these settings, our proxies reduce gradient-computation time, as reported in Tables 2–5.

W3. Coverage and newer models

We would like to clarify the breadth of the current evaluation. The four case studies use gradient guidance for distinct purposes, spanning supervision selection, safety adversarial training, prompt optimization, and test-time reasoning. The benchmarks also extend beyond mathematics. For example, selective SFT includes out-of-domain evaluation on GPQA-Diamond, safety training is evaluated on both attack resistance (AutoRAN) and over-refusal on benign requests (XSTest), and prompt optimization includes five non-mathematical BBH tasks covering linguistic, causal, and commonsense reasoning. Across these settings, our results show that cheaper gradient guidance can retain practical utility across different optimization procedures and reasoning domains.

Evaluation on modern models. We agree with the reviewer that evaluation on “modern models across diverse settings” would strengthen the evidence. Our original model choices followed the scale used by recent, strong gradient-based baselines—GReaTer (ICLR 2025), selective SFT (ICLR 2026), and ∇-Reasoner (ICLR 2026)—whose gradient computations use backbones no larger than 9B. Nevertheless, in response to this suggestion, we extended safety adversarial training and ∇-Reasoner, together with our proxy variants, to the newer, substantially larger Qwen3.8-27B, covering both training and inference. Extending selective SFT and prompt optimization to 27B would exceed our available computational budget during the discussion period.

AuthorsOpenReview ↗

Response to Reviewer y6FF (Part 2/2)

W3. Coverage and newer models (continued)

Safety adversarial training.

MethodAutoRAN ASR (%) ↓Time (% baseline) ↓
Teacher-Forcing adversarial training66.2100.0
with GP-DA65.621.3
with GP-LR63.056.2
with GP-Hybrid60.476.9

Test-time reasoning.

MethodAMC accuracy (%) ↑Time (% baseline) ↓
∇-Reasoner78.3100.0
with GP-DA77.18.1
with GP-LR83.134.7
with GP-Hybrid81.941.9

Note. Time is reported as a percentage of the corresponding baseline’s time, with each baseline set to 100%.

These results show that the utility of gradient proxies extends beyond the original smaller backbones to the newer, substantially larger Qwen3.8-27B. All three variants improve attack resistance in safety adversarial training, while GP-LR and GP-Hybrid improve test-time reasoning accuracy. Together with R1-Distill-Qwen-1.5B and Llama-3.1-8B-Instruct, these experiments provide evidence across complementary 1.5B, 8B, and 27B scales, supporting the applicability of our approach to newer and larger reasoning models in both training and inference.

W4. Timing consistency and result selection

Timing breakdown. Thank you for identifying this discrepancy. We checked the timing procedure and found an inconsistent accounting boundary in Table 1. We apologize for this confusion.

Starting from a question, obtaining the Teacher-Forcing gradient requires a pipeline consisting of generating a CoT → evaluating the answer loss conditioned on that CoT → backpropagating to the input. In Table 1 of the original manuscript, the Teacher-Forcing and GP-DA entries only reported backward time, whereas GP-LR included its forward computation. The revised Table 1 reports complete gradient-query time, including forward and backward computation (lines 243–245). Below, forward time includes reasoning and answer-loss evaluation; values are seconds per example.

SettingMethodForwardBackwardTotal
R1 / GSM8KTeacher-Forcing6.3200.0896.409
GP-DA0.0900.0870.177
GP-LR0.9712.3793.350
GP-Hybrid1.0062.4173.423
R1 / JailbreakTeacher-Forcing5.9840.1886.172
GP-DA0.0770.0510.128
GP-LR0.5930.9311.524
GP-Hybrid0.6130.9661.579

GP-LR takes 52.3% and 24.7% of Teacher-Forcing's query time on GSM8K and Jailbreak (1.9× and 4.0× speedups). With consistent accounting, GP-LR is faster than Teacher-Forcing in both Table 1 and Table 3 (safety adversarial training). We have corrected Table 1 and updated the accompanying discussion (lines 246–247). Our proposed methods are still more efficient, and the conclusion remains unchanged.

Addressing the concern about cherry-picking. Beyond correcting the timing discrepancy, we make result selection explicit and report repeated runs. In the R1 safety comparison, Teacher Forcing adversarial training and all proxy variants run for six epochs, with checkpoints selected by the lowest STAR-1 validation loss. Prompts are likewise selected on validation data, not test scores. These rules are now specified in Appendix D (Table 6, lines 814–829).

We additionally repeated safety adversarial training and prompt optimization on Llama with three seeds, applying validation-based selection within each run and reporting held-out mean ± standard error across all three runs, rather than selecting the best seed. The baselines are Teacher Forcing adversarial training and GReaTer, respectively:

Llama task / metricBaselineGP-DAGP-LRGP-Hybrid
Prompt optimization / GSM8K accuracy ↑84.26 ± 0.6784.16 ± 0.5984.79 ± 0.3084.51 ± 0.30
Safety AT / AutoRAN ASR@7 ↓52.81 ± 8.8344.56 ± 5.9742.84 ± 6.4341.25 ± 5.97

Prompt optimization remains competitive across seeds, and all proxies achieve lower mean safety ASR. Together, the explicit selection rules and results across all seeds address the concern that the findings depend on choosing favorable test checkpoints or individual runs.

Dear Reviewer,

Thank you again for your time and thoughtful feedback. As the discussion period draws to a close, we would appreciate your thoughts on our responses and additional evidence. In particular, please let us know whether any of your concerns remain unresolved, so that we can address them while the discussion is still open.

Thank you for considering our response.

Reviewer_ydjHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes cheaper proxies for the gradients of reasoning models. The gradient is taken either through the model’s non-reasoning mode, through a latent reasoning model, or through a convex combination of the two. Experiments on different tasks that rely on gradients show that these proxies cut the time cost substantially while keeping, and sometimes even improving, accuracy.

Critical strengths
  1. The method(s) are simple to implement and do not alter the model structures.
  2. The method(s) show substantial reduction of time consumption compared to the baselines.
Critical weaknesses
  1. GP-DA and GP-LR don't compute the gradient of the target objective (I feel like the authors have misunderstanding about "gradient"). Instead they compute exact gradients of different objectives. A gradient is determined by the map u↦lossu \mapsto \text{loss}. For the target, this map runs through the CoT model's own reasoning trace. GP-DA's map skips the trace entirely, and GP-LR's map runs through the latent trace of a separately distilled model. These are different maps, so their gradients are different quantities, not noisy versions of the target gradient. Sharing weights in GP-DA doesn't change this, since conditioning the answer on no trace already changes the map from uu to the loss. Even if the proxies matched the target's loss values perfectly, which is the stated goal in Section 3.2, their gradients still wouldn't have to match, because matching values doesn't imply matching gradients. In the best case, where the no-thinking mode exactly reproduces the CoT model's answer distribution, GP-DA gives the gradient of the marginal likelihood log⁡pθ(a∗∣u)\log p_\theta(a^* \mid u), which is still a different objective from the target. The paper should state clearly what these quantities are and explain why following them should reduce the target loss.

  2. In Section 3.3, the “high-query reference” is not a ground-truth gradient. It is a sampled estimate of the gradient of a Gaussian-smoothed objective. It therefore depends on the smoothing scale σ\sigma, which is not reported, and it differs from the gradient of the actual objective. Under greedy decoding, the exact gradient of FF, wherever it exists, is simply the Teacher-Forcing gradient. The estimate is also likely very noisy, since K=1024K=1024 samples is small relative to the tens of thousands of embedding coordinates being perturbed. Figure 4 only shows that estimates with different KK agree with each other, not that they agree with the true gradient. As a result, the cosine similarities in Table 1 are hard to interpret. Even an exact gradient would score low against such a noisy reference, so a low cosine may reflect the reference rather than the estimator. Cosine also does not necessarily reflect optimization utility, and Table 1 itself shows this. On GSM8K, Teacher-Forcing has a higher cosine than GP-LR but a lower ΔJ\Delta J .

  3. Table 4, although all the variants are better than GReaTer in time consumption, they don't always improve the task performance. This indicates the existence of the trade-off between task performance and the time consumption when using the GP-DA and GP-LR.

Questions for authors
  1. Instead of "gradient proxies", Isn't it more accurate to call them surrogate descent directions whose usefulness is checked empirically?
5 comments
AuthorsOpenReview ↗

Response to Reviewer ydJH (Part 1/2)

We thank the reviewer for recognizing the simplicity and computational savings of our approach. We appreciate the opportunity to clarify the mathematical interpretation and empirical scope of our contribution.

Line numbers refer to the revised manuscript unless explicitly marked “original submission.”

W1. Proxy gradients and the target objective

Gradient definition. We agree that GP-DA and GP-LR compute gradients of surrogate objectives. The revised Section 3.2 defines a gradient proxy as “the gradient of a cheaper surrogate objective with respect to the original optimization variable u” (lines 151–153). The original submission already states that we “evaluate its agreement with a reference gradient and its utility on the original model” (original lines 149–151); we do not claim that matching loss values guarantees matching gradients.

Theoretical motivation. The relevant question is therefore whether a gradient of a different objective can still provide a useful update for the target model. Demontis et al. [1] (Section 4, Eqs. 16–18) connect surrogate-gradient alignment to attack transferability under a local linear approximation. Applying the same reasoning to a descent step with target loss F and surrogate gradient g gives

\approx R\lVert\nabla F(u)\rVert_2\cos\left(\nabla F(u),g\right).$$ Thus, positive alignment permits target improvement for sufficiently small steps in a smooth region, without requiring identical gradients. Parameter sharing and distillation motivate our choice of proxies; the practical question is whether their gradients provide useful guidance for the original model. We assess this empirically through target-loss reduction and downstream performance. **Direction controls.** To examine the contribution of the proxy direction, we compare GP-LR with norm-matched random directions and GP-LR gradients exchanged across examples. In the shuffled control, each reassigned gradient is rescaled to the recipient's GP-LR gradient norm, with the downstream procedure unchanged: | R1 task / metric | GP-LR | Random | Shuffled | | --- | --- | --- | --- | | Selective SFT / MATH-500 accuracy ↑ | 76.8 | 74.0 | 74.8 | | Safety AT / AutoRAN ASR@7 ↓ | 62.3 | 67.3 | 63.7 | | Prompt optimization / GSM8K accuracy ↑ | 71.6 | 66.8 | 68.2 | | Test-time reasoning / GSM8K accuracy ↑ | 83.4 | 73.3 | 73.5 | GP-LR outperforms both controls in all four settings, supporting useful directional guidance independently of the high-query reference. ## W2. The high-query reference and optimization utility **Interpreting cosine similarity.** Section 3.3 investigates whether substantially cheaper proxy gradients can capture directional information reflected in the high-query reference. We do not treat higher cosine similarity as a guarantee of better optimization performance; practical utility is evaluated through actual target-loss reduction in Table 1 and downstream performance in Section 4. This distinction was already explicit in the original submission: “Agreement with the reference and improvement of the target objective capture different aspects of gradient quality” (original lines 244–245). We agree that smoothing and finite-sample noise can affect cosine similarity, and Figure 4 assesses consistency across query budgets rather than accuracy against the exact target gradient. **Optimization utility.** We agree with the local-derivative statement: Teacher-Forcing gives the gradient assuming that **the greedy trace remains unchanged**. However, this does not describe the effect of an input update that changes the reasoning path. For example, during adversarial input construction in safety adversarial training, the perturbed input may lead the model to generate a different trace. Teacher-Forcing holds the original trace fixed when differentiating and therefore does not capture this path change. We therefore regenerate the original model’s reasoning after each input update and measure the reduction in its unsmoothed loss. On GSM8K, GP-LR yields ΔJ=0.26129, versus 0.17713 for Teacher-Forcing despite lower cosine similarity. The high-query reference achieves the largest mean loss reduction on both datasets, supporting its role as an expensive comparison direction. Our evidence combines these target-loss measurements with downstream performance and the direction controls above. For reproducibility, Appendix B now specifies σ=0.05 (lines 756–757); the reference uses 1,024 perturbation pairs (2,048 target evaluations). ## Reference [1] A. Demontis et al. [Why Do Adversarial Attacks Transfer? Explaining Transferability of Evasion and Poisoning Attacks.](https://www.usenix.org/conference/usenixsecurity19/presentation/demontis) USENIX Security, 2019, pp. 321–338.
AuthorsOpenReview ↗

Response to Reviewer ydJH (Part 2/2)

W3. Performance–cost trade-offs

We agree that the three variants offer different performance–cost trade-offs. Nevertheless, across the experiments in Tables 1–4, our proxies achieve both lower gradient-computation cost and improved utility relative to the corresponding full-CoT baselines in multiple settings. These results validate our central contribution: making gradient-guided optimization of large reasoning models more efficient while retaining or improving its effectiveness. The three variants accommodate different optimization settings and computational budgets, and we encourage choosing accordingly:

GP-Hybrid. We favor Hybrid for settings such as safety adversarial training, where target parameters change. Its DA component follows the updated target and complements the frozen LR proxy, consistent with the safety results and stronger late-training performance in Figure 3(a) (Section 4.3, lines 323–326).

GP-LR. For a fixed target model, LR is a strong default for balancing performance and cost. It achieves the highest average prompt-optimization accuracy on both backbones at 26.9% and 40.0% of GReaTer’s gradient-computation time (Section 4.4, lines 385–389).

GP-DA. DA provides the lowest gradient-computation cost. It suits settings where computation is the main constraint and some accuracy loss is acceptable, as Table 4 illustrates.

Q1. Meaning of “gradient proxy”

We agree that “surrogate directions” makes their role more explicit. The terms “proxy” and “surrogate” are sometimes used interchangeably for auxiliary models in gradient-guided optimization [2–4]. Here, “gradient proxy” denotes the gradient of a cheaper surrogate objective with respect to the original optimization variable, whose utility for the target objective is evaluated empirically. We retain “gradient proxy” in the revision for consistency with the paper’s title and terminology, and are open to adopting the suggested terminology in the final version.

We have made this meaning explicit in Section 3.2 (lines 151–153). We also aligned the surrounding terminology: Section 3.1 describes how decoding approximations affect update directions (lines 137–139), and Section 3.3 evaluates “useful optimization directions” (lines 202–204).

References

[2] S. Yu, S. Ahn, L. Song, and J. Shin. RoMA: Robust Model Adaptation for Offline Model-based Optimization. Advances in Neural Information Processing Systems, 34, 2021.

[3] Y. Su et al. On the Adversarial Risk of Test Time Adaptation: An Investigation into Realistic Test-Time Data Poisoning. ICLR, 2025.

[4] Y. Zhuang et al. Unveiling the Secrets without Data: Can Graph Neural Networks Be Exploited through Data-Free Model Extraction Attacks? USENIX Security, 2024, pp. 5251–5268.

Dear Reviewer,

Thank you again for your time and thoughtful feedback. As the discussion period draws to a close, we would appreciate your thoughts on our responses and additional evidence. In particular, please let us know whether any of your concerns remain unresolved, so that we can address them while the discussion is still open.

Thank you for considering our response.

Reviewer_3wwZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Obtaining gradients through long chains of thought is expensive for large reasoning models (LRMs). This paper proposes gradient proxies: shorter reasoning computations whose gradients guide the optimization of the original CoT model. GP-DA answers directly with no reasoning trace. GP-LR uses CODI to distill explicit CoT into six continuous latent steps. GP-Hybrid linearly mixes the two gradients (λ = 0.5). The proxies are first compared against a zeroth-order finite-difference reference (K = 1024) on cosine similarity, target-loss reduction ΔJ, and runtime. They are then plugged into four downstream methods: segment-selective SFT, safety adversarial training, GReaTer prompt optimization, and ∇-Reasoner test-time reasoning. Experiments use R1-Distill-Qwen-1.5B and Llama-3.1-8B. The paper reports comparable or better accuracy with substantially lower gradient-estimation cost.

Critical strengths

1.A practical idea that plugs into existing pipelines as a replaceable gradient query. 2.Broad evaluation: four tasks across the LLM lifecycle. 3.Table 1 measures actual target-loss reduction, not only cosine similarity, and the ablations (CODI transfer, latent depth, λ) are informative.

Critical weaknesses

1.The efficiency claim rests on timings that reverse without explanation. In Table 1, Teacher-Forcing has similar or higher cosine similarity than GP-LR and is 8–38× faster. In Table 3, GP-LR costs only 44% of Teacher-Forcing AT. No timing breakdown or matched-workload comparison is given. Tables 2–4 also exclude the cost of DA adaptation and CODI distillation. 2.No controls isolate what the proxy direction contributes. Some results suggest the downstream methods are insensitive to gradient quality: GP-DA beats ∇-Reasoner; true-gradient Selective SFT scores below Full-CoT SFT; on Llama, Teacher-Forcing AT is worse than Standard SFT. Norm-matched random, shuffled, or zeroed proxy gradients would settle this. 3.The safety-AT numbers are inconsistent. Page 9 reports final ASR of 64.3% / 74.9% for DA / LR, while Tables 3 and 6 report 61.7% / 62.3%. 4.The proxy input mapping is under-specified. It is unclear how CODI consumes a 5k+ token trace in selective SFT. The reference gradient (K = 1024, m ~ 10⁴–10⁵) is only a rough estimate. All results are single runs on small benchmarks (e.g., 5 questions on AMC23).

Questions for authors

1.What does each timing include? Why does GP-LR's relative cost reverse between Table 1 and Table 3? What is the total cost including proxy preparation? What is the concrete proxy mapping for selective SFT and test-time reasoning? Which setting produces the page-9 ASR values? Can you add random / shuffled / zeroed-gradient controls and multi-seed results?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3wwZ (Part 1/2)

We thank the reviewer for recognizing the practical value of replaceable gradient queries, the breadth of the evaluation, and the use of target-loss reduction. We provide corrected timing, preparation costs, direction controls, and three-seed results below.

Line and table numbers below refer to the revised manuscript.

W1. Timing consistency and preparation cost

Timing breakdown. Thank you for identifying this discrepancy. We checked the timing procedure and found an inconsistent accounting boundary in Table 1. We apologize for this confusion.

Starting from a question, obtaining the Teacher-Forcing gradient requires a pipeline consisting of generating a CoT → evaluating the answer loss conditioned on that CoT → backpropagating to the input. In Table 1 of the original manuscript, the Teacher-Forcing and GP-DA entries only reported backward time, whereas GP-LR included its forward computation. The revised Table 1 reports complete gradient-query time, including forward and backward computation (lines 243–245). Below, forward time includes reasoning and answer-loss evaluation; values are seconds per example.

SettingMethodForwardBackwardTotal
R1 / GSM8KTeacher-Forcing6.3200.0896.409
GP-DA0.0900.0870.177
GP-LR0.9712.3793.350
GP-Hybrid1.0062.4173.423
R1 / JailbreakTeacher-Forcing5.9840.1886.172
GP-DA0.0770.0510.128
GP-LR0.5930.9311.524
GP-Hybrid0.6130.9661.579

GP-LR takes 52.3% and 24.7% of Teacher-Forcing's query time on GSM8K and Jailbreak (1.9× and 4.0× speedups). With consistent accounting, GP-LR is faster than Teacher-Forcing in both Table 1 and Table 3 (safety adversarial training). We have corrected Table 1 and updated the accompanying discussion (lines 246–247). Our proposed methods are still more efficient, and the conclusion remains unchanged.

Preparation cost. We report two costs: preparing the proxy (DA adaptation and/or CODI distillation) and running the downstream task. Each run includes the full preparation cost. Downstream execution includes all task steps, including model training in Safety AT; Tables 2–4 measure only gradient computation. For each task, the baseline’s total running time is set to 100%.

TaskMethodPreparationDownstream executionTotal
Selective SFTGP-DA7.321.128.4
GP-LR23.150.673.7
GP-Hybrid30.463.193.5
Safety ATGP-DA6.750.256.9
GP-LR22.556.779.2
GP-Hybrid29.265.995.1
Prompt optimizationGP-DA11.316.027.3
GP-LR36.029.765.7
GP-Hybrid47.343.090.3

Note. Baselines, in task order: Selective SFT, Teacher Forcing adversarial training, and GReaTer. Total = Preparation + Downstream execution; totals are calculated before rounding.

Across these three tasks, total cost including preparation is reduced by 43.1–72.7% for GP-DA, 20.8–34.3% for GP-LR, and 4.9–9.7% for GP-Hybrid, relative to each baseline's end-to-end downstream time.

Reuse can further amortize preparation. Figure 3(b) already tests CODI training sources across applications: accuracy varies by only 0.6 points in selective SFT and 1.2 points in test-time reasoning, supporting reuse in the evaluated settings.

W2. Contribution of the proxy direction

We agree that comparisons between complete pipelines do not isolate the direction's contribution. We therefore replaced GP-LR guidance with norm-matched random or shuffled directions, retaining the downstream procedure. For the shuffled control, we exchange GP-LR gradients across examples and rescale each reassigned gradient to the recipient's GP-LR gradient norm:

R1 task / metricGP-LRRandomShuffledOriginal task baseline
Selective SFT / MATH-500 accuracy ↑76.874.074.875.0
Safety AT / AutoRAN ASR@7 ↓62.367.363.768.3
Prompt optimization / GSM8K accuracy ↑71.666.868.264.4
Test-time reasoning / GSM8K accuracy ↑83.473.373.572.0

Note. Original task baselines, in row order: Full-CoT SFT, Standard SFT, the unoptimized CoT prompt, and greedy decoding. These are not zero-gradient controls; zero gradients cannot rank segments in selective SFT.

GP-LR outperforms both controls in all four tasks, supporting the value of directional information beyond gradient norm. Together with the computational savings, these results highlight our central contribution: obtaining useful optimization guidance at substantially lower cost.

AuthorsOpenReview ↗

Response to Reviewer 3wwZ (Part 2/2)

W3. The two sets of safety ASR values

The values refer to different checkpoints. For the R1 runs in Figure 3(a), the lowest STAR-1 validation loss selects GP-LR at epoch 3 (ASR 62.3%) and GP-DA at epoch 5 (ASR 61.7%). Their epoch-6 ASRs are 74.9% and 64.3%, respectively. We discuss these final-epoch values to highlight the late-training rebound in ASR; Tables 3 and 7 report the validation-selected checkpoints. Teacher Forcing adversarial training and all proxies run for six epochs with the same selection criterion. The discussion and table captions now distinguish these results (lines 465–467; 351–352, 947–948), and Appendix D specifies the selection protocol (Table 6, lines 814–829; Table 7 was Table 6 in the original submission).

W4. Proxy mapping, reference quality, and reliability

Proxy mapping. Appendix A.2 already defines gradients in the original downstream variable's coordinates (Eq. (3), lines 731–737). We have clarified the task-specific computations:

  • Selective SFT: given the question and annotated trace embeddings, GP-DA and GP-LR compute target-answer gradients in the original embedding coordinates and use them in the same segment-attribution procedure. The reported 5k+ token lengths describe the base model’s ordinary inference outputs, not CODI’s output length. Appendix D.1 details the input mapping and gradient computation (lines 835–842; Algorithm 1, lines 878–879).

  • Test-time reasoning: the experiments in Table 5 replace the reward-gradient query and retain the original likelihood gradient. The reward model receives the question and complete candidate response. Appendix D.4 provides the detailed formulation.

We have also clarified that “Avg. Token Length” measures ordinary model outputs (Table 2/3 captions, lines 331–332, 353–354).

Reference quality. We agree that K=1024 gives a finite-query reference, not an exact gradient. We use 1,024 pairs and 2,048 target evaluations, each regenerating reasoning. Figure 4 measures consistency with a higher-budget estimate.

Our evaluation already distinguishes reference agreement from target improvement (lines 249–250). For Δ J, directions receive the same update radius and the target regenerates its reasoning before loss evaluation. GP-LR improves target loss more than Teacher-Forcing despite slightly lower cosine similarity: 0.26129 vs. 0.17713 on GSM8K, and 0.244 vs. 0.216 on Jailbreak.

Benchmark coverage and repeated runs. We use established benchmarks adopted by the original selective SFT, GReaTer, and ∇-Reasoner studies, covering mathematical, scientific, linguistic, causal, and commonsense reasoning. Our evaluation therefore spans multiple reasoning domains and follows established benchmark choices for these applications. Selective SFT uses all 40 AMC23 questions, rather than five; test-time reasoning uses a separate 83-question AMC set (Appendix D.4). We recognize that small evaluation sets limit conclusions about small performance differences.

We additionally ran three-seed experiments on Llama for safety adversarial training and prompt optimization, assessing variability in both model training and prompt optimization. We report mean ± standard error across all three runs. Checkpoints and prompts are selected using validation data and evaluated on held-out data. The respective baselines are Teacher Forcing adversarial training and GReaTer.

Llama task / metricBaselineGP-DAGP-LRGP-Hybrid
Prompt optimization / GSM8K accuracy ↑84.26 ± 0.6784.16 ± 0.5984.79 ± 0.3084.51 ± 0.30
Safety AT / AutoRAN ASR@7 ↓52.81 ± 8.8344.56 ± 5.9742.84 ± 6.4341.25 ± 5.97

Prompt optimization remains competitive across seeds, and all proxies achieve lower mean safety ASR.

Dear Reviewer,

Thank you again for your time and thoughtful feedback. As the discussion period draws to a close, we would appreciate your thoughts on our responses and additional evidence. In particular, please let us know whether any of your concerns remain unresolved, so that we can address them while the discussion is still open.

Thank you for considering our response.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Reviewers and Area Chair: Contributions, Additional Evidence, and Revisions

We thank the reviewers for their constructive feedback and for revisiting our paper during the discussion.

Core contribution.

Our work shows that a model that reasons through a long chain of thought can be optimized using gradients from a much shorter computation. We propose three ways to obtain these gradients: direct answering (GP-DA), a short latent reasoning trace (GP-LR), and a combination of the two gradients (GP-Hybrid). Each has a distinct strength. GP-DA is the cheapest to compute, GP-LR performs particularly well in fixed-model settings such as prompt optimization and test-time reasoning, and GP-Hybrid is effective in safety adversarial training, where model weights change during training.

We evaluate these proxies across four applications spanning the LLM lifecycle: supervision selection, safety adversarial training, prompt optimization, and test-time reasoning. During the discussion, we extended the evaluation from the original 1.5B and 8B backbones to Qwen3.8-27B. On the same 27B model, different proxies delivered the strongest gains in training and inference. GP-Hybrid reduced the safety attack success rate from 66.2% to 60.4% using 76.9% of the Teacher-Forcing adversarial training baseline’s time. In test-time reasoning, GP-LR raised AMC accuracy from 78.3% to 83.1% using 34.7% of ∇-Reasoner’s time.

Main reviewer concerns and manuscript revisions.

  1. Efficiency under complete timing. We corrected Table 1 to include both forward and backward computation. With the full gradient query timed for every method, GP-LR is faster than Teacher-Forcing on both benchmarks. In the three applications measured end to end, all three proxies still saved time after including the full cost of proxy preparation.

  2. Utility of the proxy direction. We evaluate each direction by measuring how an update changes the original model’s loss. To test whether the direction itself matters, we added norm-matched controls. GP-LR outperformed both random directions and GP-LR gradients exchanged across examples in all four R1 applications.

  3. Clarifications in the revised manuscript. We define a gradient proxy as the gradient of a cheaper objective with respect to the original optimization variable. The appendix now explains how selective SFT and test-time reasoning use the proxy, and how we use validation data to select safety-training checkpoints and optimized prompts before held-out evaluation.

Discussion outcome. All three reviewers indicated they would raise their scores. The corrected timing resolved the main efficiency concern, and the new controls supported the value of the proxy direction. We thank the reviewers for helping us clarify our central claim: gradients from shorter computations can guide optimization of the original reasoning model, and the best choice of proxy depends on the task and compute budget.

Manuscript revisions.

We have clarified the definition and empirical evaluation goal of gradient proxies, corrected Table 1 and its accompanying discussion, specified the high-query reference settings, and added further implementation details for selective SFT and test-time reasoning in the appendix. We have also added selection and evaluation protocols, making explicit how validation data are used to select checkpoints and prompts before held-out evaluation. These revisions make the contribution, computational comparisons, and evaluation procedures clearer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.