acceptodds
Under review as a conference paper at ICLR 2027

SRAO: Coordinating Auxiliary Guidance and Feedback for Efficient Learning

Abstract

Learning systems often combine inexpensive guidance with feedback that is costly to obtain. Their efficiency depends on coordinating how guidance shapes an update with how feedback corrects it. We introduce Shared Residual Audited Optimization (SRAO), which uses corrective feedback both to improve the current update and to inform the use of guidance. For selective gradient acquisition, one reference corrects the estimate and evaluates all available guidance-acquisition rules. The key insight is that the estimation gain from combining guidance and acquisition choices can pay for learning how to combine them. We establish this connection through a sharp variance bound and a cumulative learning guarantee. Scalar and gradient studies explain the mechanism, including about 40% lower estimation error at matched reference cost. On-policy distillation retains similar observed mean accuracy with 35.5% fewer reference-gradient batches. Agentic question answering extends the coordination principle to reflection guidance and outcome feedback, improving mean task quality. Together, these results connect feedback efficiency to the way guidance enters learning updates.

Primary area:
optimization
Keywords:
Stochastic optimization, variance reduction, online learning, knowledge distillation, language model agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UtPJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces Shared Residual Audited Optimization (SRAO), a method for combining inexpensive auxiliary guidance with costly corrective feedback. In selective gradient estimation, SRAO jointly aggregates gradient predictions and feedback-acquisition probabilities, using acquired reference gradients to correct estimates and adapt rule weights. The authors establish a sharp variance inequality and a horizon-independent cumulative excess bound under specified assumptions. Experiments report approximately 40% lower gradient-estimation error at matched reference cost and similar distillation accuracy with fewer reference-gradient batches. The paper also applies the broader guidance feedback coordination principle to agentic question answering, reporting improvements across seven benchmarks.

Critical strengths

The paper establishes an interesting connection between the variance reduction achieved by combining guidance acquisition rules and the cost of adaptively learning their weights. The variance inequality in Theorem 4.1 provides the basis for a horizon-independent cumulative excess bound under the stated assumptions, although the practical benefits of adaptive rule learning are not convincingly established by the empirical results.

Critical weaknesses

W1. The central algorithmic contribution is not convincingly validated experimentally.

One of the key claimed contributions of the paper is the adaptive learning of guidance acquisition rules. However, the experimental results do not convincingly demonstrate an advantage of this adaptive component over simpler fixed-acquisition strategies. In Section 5.1, fixed acquisition achieves higher accuracy with fewer reference batches than learned SRAO (41.48% with 61 batches versus 40.56% with 64 batches). The authors explicitly acknowledge that these results support reference correction without establishing an advantage from learning acquisition. Furthermore, the analysis of distillation histories in Section 4.2 reports positive importance-weighted excess error relative to the best fixed observed rule. Although the theoretical analysis provides some guarantees for adaptive rule learning, the empirical evidence primarily supports conditional aggregation and reference correction, but do not demonstrate the practical benefit of learning acquisition policies.

W2. Clarity of presentation and claimed generality

The paper presents SRAO as a general framework for coordinating auxiliary guidance and corrective feedback. However, its formal algorithm and theoretical guarantees apply specifically to selective acquisition of reference gradients. The agentic QA application uses a substantially different mechanism involving look-ahead optimization, outcome rescoring, and conflict projection. The abstract and introduction use “guidance” and “feedback” very abstractly. A reader cannot tell what SRAO is until Section 3. Figure 1 helps, but Table 3 (which admits the two applications use different operators) is relegated to the appendix. Notation in 4.1 is dense. The paper reads as two projects joined by one very high-level idea.

W3. The experimental evaluation does not isolate the proposed contribution.

The agentic QA improvements are difficult to attribute to SRAO. The compared systems differ in training data and computation. The key coordination ablation simultaneously removes rescoring, conflict projection, and associated training work. The distillation experiments primarily establish reduced reference-gradient computation rather than end-to-end training efficiency, and the main theoretical guarantee concerns estimation error rather than final task performance. Overall, the experiments do not convincingly connect the proposed theoretical contribution to the reported practical improvements.

W4. Limited statistical robustness

The experimental evaluation provides limited evidence of robustness across random seeds. The primary distillation results are averaged over only two seeds, with a third-seed replication reported separately in the Appendix, while the agentic QA results appear to rely on a single fixed-seed configuration.

Questions for authors

Q1. Is there any setting where learned acquisition outperforms the best fixed rule on task accuracy? If not, what is the practical case for Algorithm 1 over fixed-rate cached-gradient correction?

Q2. For the fixed-state MSE study, what is the reduction relative to a single rule at the same pˉ\bar{p} using the zero prediction?

5 comments

Thank you for the detailed review and for recognizing the connection between aggregation gains and the cost of learning. We clarify SRAO's contribution and provide additional task-accuracy comparisons addressing your question about the practical benefit of adaptive rule learning beyond fixed-rate cached-gradient correction. We also distinguish the evidence and theoretical scope of the two applications. The new multi-run fixed/adaptive comparisons appear in revised §5.1 (Table 2 and Fig. 3), with supporting studies in Appendix B.2.


W1/Q1 — What SRAO contributes, and what learning adds beyond fixed acquisition.

SRAO's core contribution is a feedback-efficient framework and a sharp connection between estimation gains and the cost of learning. For selective gradients, SRAO conditionally aggregates residual-corrected rules, preserving the reference mean and expected acquisition cost while removing error from random rule choice. Each acquired reference both corrects the update and evaluates every guidance–acquisition rule. Theorem 4.1 shows how the resulting aggregation gain controls the variance cost of learning their weights, yielding the cumulative guarantee in Eq. (5).

Fixed and adaptive acquisition are configurations of this framework. The submitted §3.1 states, “Fixing all weight on one rule recovers the residual-correction update.” Keeping that rule's query probability constant gives fixed-acquisition SRAO. Its cached prediction still evolves, and fresh references correct its residual. Online rule learning adds adaptation to this corrected-estimator family. The fixed setting's efficiency and the additional benefits of learning therefore evaluate two parts of SRAO's practical contribution.

To examine adaptive rule learning beyond fixed-rate cached-gradient correction, we evaluate two configurations of adaptive SRAO. Adaptive SRAO (guidance + acquisition) learns rule weights that adapt both the guidance mixture and query probability. Adaptive SRAO (acquisition only) learns when to query using the same unscaled, evolving cached-gradient prediction rule as fixed-acquisition SRAO.

Both fixed and adaptive settings provide evidence for the framework's reference efficiency. Adaptive SRAO (guidance + acquisition) attains 40.56% versus 40.46% accuracy with 35.5% fewer references than full feedback (revised Appendix B.2, Table 8). In the new four-run MATH500 comparison, fixed-acquisition SRAO attains 91.65% versus 91.70% with 40% fewer references (Table 2). The following comparisons then test the additional task-accuracy benefit of adaptive rule learning against the paper's fixed p=0.5 and retrospective mean-rate controls.

Within SRAO's framework, learning improves the three-task mean by 1.57 percentage points over fixed acquisition, including a 3.89-point gain on HMMT2025. This additional three-run comparison evaluates adaptive SRAO (guidance + acquisition) against fixed-acquisition SRAO at p=0.5.

The comparison uses three training runs per configuration and evaluates all questions in AIME2024, AIME2025 and HMMT2025 at Avg@12. Task scores are mean ± sample SD. The final score column equally averages the three task means. Reference counts are per-run means.

SRAO configurationAIME2024 (%)AIME2025 (%)HMMT2025 (%)Three-task mean (%)References/run
Fixed-acquisition SRAO, p=0.555.19 ± 1.4339.26 ± 1.1225.18 ± 0.8539.8860.33
Adaptive SRAO (guidance + acquisition)54.81 ± 0.1640.46 ± 1.9729.07 ± 0.5841.4565.33

The observed accuracy gain uses 65.33 rather than 60.33 reference batches per run. Mean training-loop times are 103.70 and 103.07 minutes, respectively. We report the complete task and resource comparison in Appendix B.2.4, Table 19.

Acquisition learning also improves accuracy with the same evolving cached-gradient prediction rule. Adaptive SRAO (acquisition only) gains 1.11 points with 2.17% fewer references against constant mean-rate controls on AIME2025/HMMT2025. Each constant is set to its paired adaptive run's mean post-calibration probability. The controls use fresh query draws. This retrospective comparison uses three newly trained controls and the existing adaptive evaluations.

SRAO configurationAIME2025 (%)HMMT2025 (%)Combined accuracy (%)References/run
Fixed-acquisition SRAO (mean rate)35.8323.6129.72 ± 0.24230
Adaptive SRAO (acquisition only)36.9424.7230.83 ± 0.42225

Evaluation covers all questions at Avg@4 and weights the two tasks equally. Both task means improve by 1.11 points. The three paired combined gains are +1.25, +0.83 and +1.25 points. Revised Fig. 3 and Appendix B.2.2, Table 16 show the paired results and protocol.

W1/Q1 continue— What SRAO contributes, and what learning adds beyond fixed acquisition.

MATH500 provides a complementary four-run comparison of the framework and its adaptive configurations. Adaptive SRAO (guidance + acquisition) gains 1.00% over fixed p=0.5. Adaptive SRAO (acquisition only), which uses the same unscaled, evolving cached-gradient rule as the fixed control, gains 0.75% at nearly identical reference expenditure. All 500 questions are evaluated at Avg@1.

ConfigurationMATH500 accuracy (%)Reference batches/run
Full feedback91.70 ± 0.70100.00
Fixed-acquisition SRAO, p=0.591.65 ± 1.2260.00
Adaptive SRAO (acquisition only)92.40 ± 0.8559.50
Adaptive SRAO (guidance + acquisition)92.65 ± 0.6662.25

The four rows show the reference efficiency of fixed-acquisition SRAO and the additional accuracy gains of its adaptive configurations in the same study. A separate longer-training MATH500 comparison gives 89.53% versus 88.60% for adaptive SRAO (guidance + acquisition) and fixed p=0.5. The adaptive configuration uses 3.05% fewer references, with gains in all three runs (+0.4, +1.4 and +1.0 points).

The learning guarantee connects aggregation gain to the statistical cost of adaptive rule learning. The task comparisons above evaluate trained policies against the named fixed-acquisition controls. Under the finite-library, bounded-residual, positive-query-floor and shared-reference assumptions, Eq. (5) bounds expected cumulative estimation error plus acquisition cost against the best fixed rule identity in expectation on the learner's history. The horizon-independent additive term C measures the learning overhead. The importance-weighted excess discussed in the review describes a realized history. Eq. (5) compares expected objectives and includes allowance C.

Eqs. (6)–(7) separate acquisition error from reference bias and noise. For the stated smooth gradient updates, the learning overhead contributes C/T to the average squared-gradient-norm-plus-cost bound. The task comparisons above evaluate the practical accuracy benefit.


Q2 — What happens relative to zero prediction at the same probability?

At the matched probability 49/80, the fixed-state reconstruction gives MSE changes of +0.64% and +1.48% for Qwen3-0.6B, and +1.88% and +2.40% for Qwen3-1.7B, across two trials per model. Positive values mean higher MSE than zero prediction.

Under the independent-query protocol, for a fixed prediction PP and 0<p<10<p<1, with expectations conditional on the fixed state,

=(p^{-1}-1)(\|P\|^2-2\langle P,\mathbb E[G]\rangle).$$ Thus prediction improves on zero exactly when $2\langle P,\mathbb E[G]\rangle>\|P\|^2$. This condition characterizes the benefit of the prediction itself. Conditional aggregation then removes the additional variance introduced by randomly executing a component rule. Figure 2's **approximately 40% reduction** measures conditional aggregation against random execution of the same component rules at identical expected reference cost (§4.2). We have labeled this comparator explicitly in §4.2 and the Figure 2 caption, and made the prediction-alignment condition explicit in §4.2, with its derivation and zero-prediction comparison in Appendix B.1. --- ## W2 — Concrete presentation and the scope of the applications. We appreciate the presentation suggestions. We have clarified how fixed and adaptive acquisition fit within SRAO in the opening sections. Section 3 gives a concrete cached-gradient example before the formal protocol. For rule predictions $P_a$, query probabilities $p_a$, and weights $r_a$, Eq. (1) gives, when $0<\bar p<1$, $$\bar p=\sum_a r_a p_a,\qquad P_0=\frac{\sum_a r_a(1-p_a)P_a}{1-\bar p},\qquad g=P_0+\frac{A}{\bar p}(G-P_0),\quad A\sim\operatorname{Bernoulli}(\bar p).$$ Here $G$ is the fresh reference and $P_0$ is the prediction conditional on no query. At full acquisition, $g=G$. The same acquired reference supplies residual feedback for every rule. The cached-gradient example introduces the mechanism before the formal protocol, and Eq. (1) specifies how the predictions and query probabilities enter the update. We have added a reading guide to §4.1 that explains rule error, aggregation gain and the two query branches before the theorem. The concrete operators now appear directly in the main text: selective correction in §3.1, Eq. (1), and QA's sequential displacement update in §5.2, Eq. (8). The abstract states the shared coordination principle. The introduction explains how outcome feedback at the auxiliary candidate regulates the accepted update and summarizes the measured benefit of the complete QA coordination stage. The selective-gradient theory analyzes the estimator and rule-learning mechanism. The QA experiments evaluate the look-ahead operator that coordinates auxiliary and outcome displacements.

W3 — Attribution and measured training efficiency.

We have added actual elapsed time alongside accuracy and reference expenditure (§5.1 and Appendix B.2.6). With query-before-generation enabled, the four-run MATH500 comparison gives:

MethodAccuracy (%)Reference batches/runSetup, training and saving (min)
Full feedback91.70 ± 0.70100.00122.69
Adaptive SRAO (acquisition only)92.40 ± 0.8559.5077.96

This is 36.46% less measured training time, with 40.5% fewer reference batches and student generations. Evaluation time is separate. These are per-run means under shared-node conditions.

A separate retained-versus-skipped-generation experiment verifies the implementation change itself. Elapsed time including setup and saving decreases by 30.11% for adaptive SRAO (guidance + acquisition) and 28.81% for fixed-acquisition SRAO, with matching reference counts, recorded training histories and final adapters. These are one-pair measurements per policy. The four-run comparison measures training-policy efficiency, and the paired intervention identifies the time saved by removing unused generation while preserving the recorded training trajectory.

For QA, the results show the benefit of the complete system and of coordinating its two training signals. Revised Table 3 reports F1 46.15% versus 30.16% at 3B and 49.21% versus 34.08% at 7B against Search-R1 under shared retrieval/inference conditions. These system-level comparisons include differences in training data and computation, including our 600 successful SFT trajectories. With both auxiliary and outcome training retained, adding the complete coordination stage raises 3B F1 from 41.21% to 46.15% (+4.94%). The stage rescores the auxiliary candidate and uses the resulting outcome displacement to regulate how auxiliary guidance enters the accepted update. The intervention evaluates rescoring, projection and their associated work together, supporting the practical value of this complete coordination procedure. We have made these attribution levels explicit alongside the results in §5.2, with the full protocol and per-task component results in Appendix B.3.


W4 — Robustness across training runs.

We expanded the original study to five training seeds and evaluated two separate four-run MATH500 cohorts, reported in revised §5.1, Table 2 and Appendix B.2.3 (Table 17). The following table assesses reference efficiency against full feedback, complementing W1's fixed-policy comparisons. Scores are mean ± sample SD within each cohort. References are per-run means.

CohortSRAO configurationFull-feedback accuracy (%)SRAO accuracy (%)References: Full → SRAO
MATH500, four runsAdaptive SRAO (acquisition only)91.70 ± 0.7092.40 ± 0.85100 → 59.50
MATH500, separate smaller-batch cohort, four runsAdaptive SRAO (acquisition only)92.30 ± 0.3892.50 ± 0.78100 → 70.25
Original three-task Avg@12, five runsAdaptive SRAO (guidance + acquisition)40.741 ± 0.83640.759 ± 0.330100 → 59

The five-run extension retains similar observed accuracy with 41% fewer references. W1 additionally reports the three-run mean-rate comparison and the three-run longer-training result. Replication counts refer to training runs within each configuration and cohort, with checkpoint and evaluation reuse identified in the corresponding comparisons. The table reports the mean accuracy and training-seed variation for each cohort. QA retains one learner/data seed pair and configuration selection by the reported seven-task mean. We have disclosed that scope alongside its results in §5.2.

Reviewer_8YhDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Shared Residual Audited Optimization (SRAO), a framework that coordinates inexpensive auxiliary guidance with costly corrective feedback to improve gradient estimation efficiency and reduce reference computation in LLM distillation.

Critical strengths
  1. The paper establishes a relatively strong theoretical foundation.

  2. The problem is important.

Critical weaknesses
  • The paper emphasizes the importance of jointly learning guidance and feedback acquisition. However, the controlled comparison in Section 5.1 shows that fixed acquisition achieves higher accuracy (41.48%) than learned acquisition (40.56%), while also using fewer reference batches (61 vs. 64). What empirical evidence demonstrates that adaptive acquisition provides a meaningful benefit beyond cached-gradient correction and a simple fixed query probability? Could the main efficiency gains be explained almost entirely by the latter?

  • The agentic QA application appears to instantiate a substantially different optimization procedure from the selective-gradient estimator analyzed theoretically. It uses full outcome feedback, look-ahead rescoring, and conflict projection, without the shared-residual importance-weighted estimator or adaptive acquisition mechanism central to Algorithm 1. Beyond the high-level intuition of coordinating guidance and feedback, what concrete mathematical or algorithmic structure is shared between these two applications? What distinguishes the agentic implementation of SRAO from a conventional auxiliary-gradient method with conflict projection?

Questions for authors

The authors should address the points listed in the weakness section.

6 comments

Thank you for recognizing the theoretical foundation and for asking what adaptive acquisition adds beyond cached-gradient correction. We provide comparisons that distinguish these contributions and specify how corrective feedback enters the update in each application. The revised manuscript makes these distinctions explicit in §3.1, §5.1 (Table 2 and Fig. 3), and §5.2 (Eq. (8)).


W1 and corresponding question — Framework efficiency and the additional value of adaptation.

SRAO contributes a feedback-efficient estimator framework and a sharp analysis of how aggregation gains can pay for online learning. Conditional aggregation combines corrected guidance–acquisition rules, and each acquired reference corrects the estimate and evaluates every rule. The resulting estimation gain controls the variance cost of learning their weights (§§3–4).

The fixed strategy in your comparison is a simple SRAO configuration. The submitted §3.1 gives the single-rule residual-correction update. Fixing its query probability gives fixed-acquisition SRAO, with an evolving cache and reference correction. Its efficiency is part of the framework's practical contribution. The new comparisons assess what adaptation adds within this same framework.

To assess what adaptation adds beyond cached-gradient correction, we evaluate two configurations of adaptive SRAO, both using online rule learning. Adaptive SRAO (guidance + acquisition) learns rule weights that adapt both the guidance mixture and query probability. Adaptive SRAO (acquisition only) learns when to query using the same unscaled, evolving cached-gradient prediction rule as fixed-acquisition SRAO. This second configuration directly tests the added value of acquisition learning.

In the four-run MATH500 study, fixed-acquisition SRAO attains 91.65% versus 91.70% for full feedback with 40% fewer references. In the original distillation study, adaptive SRAO (guidance + acquisition) retains similar mean accuracy with 35.5% fewer references (Appendix B.2, Table 8).

Online learning adds further task-accuracy gains within this framework. In the additional three-run comparison, adaptive SRAO (guidance + acquisition) improves the three-task mean from 39.88% to 41.45%, a gain of 1.57 points over fixed-acquisition SRAO (p=0.5). Its HMMT2025 mean rises from 25.18% to 29.07%, a 3.89-point gain. This three-run Avg@12 comparison evaluates joint guidance and acquisition learning. Mean reference use is 65.33 versus 60.33 batches per run. The full task results and resource measurements appear in Appendix B.2.4, Table 19.

Acquisition learning adds accuracy gains with cached-gradient correction retained. The following comparisons use Adaptive SRAO (acquisition only). Both the fixed and adaptive configurations use the same evolving cached-gradient prediction rule and residual correction. These comparisons test the additional value of learning when to query.

Study and fixed-acquisition SRAO controlFixed accuracy (%)Adaptive SRAO (acquisition only), accuracy (%)References: fixed → adaptive
AIME2025/HMMT2025, mean-rate control29.72 ± 0.2430.83 ± 0.42230 → 225
MATH500, paper p=0.591.65 ± 1.2292.40 ± 0.8560.00 → 59.50

The gains are 1.11 and 0.75 points, respectively. The first comparison averages two tasks at Avg@4 over three runs. Both task means improve by 1.11 points, and all three paired combined gains are positive. In the mean-rate comparison, each constant is set retrospectively to its paired adaptive run's mean post-calibration probability. The controls use fresh query draws, and this comparison reuses the adaptive evaluations. The MATH500 comparison evaluates all 500 questions at Avg@1 over four training runs. Scores are mean ± sample SD within each study.

Adaptive SRAO (guidance + acquisition) also reaches 92.65 ± 0.66% in the MATH500 study, compared with 91.65 ± 1.22% for fixed p=0.5. The adaptive configuration uses 62.25 versus 60.00 references per run. A separate longer-training comparison gives 89.53% versus 88.60%, with 3.05% fewer references and positive gains in all three runs. These results assess adaptive SRAO (guidance + acquisition) alongside the acquisition-only comparisons.

Together, these comparisons show reference efficiency for fixed-acquisition SRAO and additional task-accuracy gains for adaptive SRAO. The acquisition-only comparisons retain the same cached-gradient rule, directly addressing the value of learning when to query. The theoretical guarantee controls expected estimation error plus acquisition cost on the learner's history. The task evaluations measure the accuracy gains of the trained policies.

W2 and corresponding question — Shared guidance–feedback principle and concrete update mechanisms.

Both applications use corrective feedback to determine how auxiliary guidance enters the training update. In selective distillation, an acquired reference corrects the estimate and evaluates the available guidance–acquisition rules. In QA, outcome feedback is evaluated at the auxiliary candidate and regulates the guidance retained alongside the outcome displacement. The following operators make this connection concrete:

ApplicationHow feedback is usedUpdate construction and scope
Selective distillationCorrect the estimate and evaluate every guidance–acquisition ruleConditional aggregation of corrected estimates, analyzed by Algorithm 1 and Theorem 4.1 under the stated assumptions
Original Agentic QA coordination studyEvaluate the auxiliary candidate and coordinate its displacement with outcome feedbackLook-ahead rescoring and sequential optimizer-displacement coordination, evaluated empirically

For selective distillation with 0<pˉ<10<\bar p<1,

g=(1−Apˉ)P0+ApˉG,g=\left(1-\frac{A}{\bar p}\right)P_0+\frac{A}{\bar p}G,

where P0P_0 is the prediction conditional on no query and GG is acquired with probability pˉ\bar p. At pˉ=1\bar p=1, the estimator is GG. Every acquired reference also supplies residual errors for all rules. Theorem 4.1 connects the resulting aggregation gain to the learning penalty.

For QA, three auxiliary Adam steps move the parameters from xx to yy. The same trajectories are rescored under the candidate policy at yy, and an outcome-feedback Adam step gives zz. Define df=y−xd_f=y-x and ds=z−yd_s=z-y. The accepted update is

\begin{cases} d_f-\dfrac{\min\{\langle d_f,d_s\rangle,0\}}{\|d_s\|^2}\,d_s, & \|d_s\|^2>10^{-12},\\ d_f, & \|d_s\|^2\le10^{-12}, \end{cases} \qquad x'=x+d_s+w\,d_f^\perp,\quad w\ge0.$$ Outside the numerical guard, $\langle d_f^\perp,d_s\rangle\ge0$: the standard half-space projection retains auxiliary guidance with a nonnegative inner product with the outcome displacement. **The QA mechanism couples candidate-policy rescoring with sequential optimizer updates.** The outcome displacement is computed at the auxiliary candidate $y$, after the auxiliary steps. The accepted update then coordinates the Adam displacements from $x$ to $y$ and from $y$ to $z$. Thus outcome feedback at the candidate policy determines how auxiliary guidance enters the final update. At full acquisition, the selective estimator gives $g=G$. The QA operator combines the outcome displacement with projected auxiliary guidance. These operators implement the shared feedback-guided coordination principle in selective estimation and sequential optimizer updates, respectively. The selective-gradient analysis establishes estimator and rule-learning guarantees. The original QA study evaluates displacement coordination, and the additional QA experiments apply the selective estimator directly to outcome gradients. The abstract presents QA as an extension of the coordination principle. The Introduction presents QA as an empirical application of the optimization mechanisms, and §5.2 specifies candidate rescoring and the coordination of auxiliary and outcome updates. The guarded QA update now appears in the main text (§5.2, Eq. (8)). The QA experiments connect this update mechanism to answer quality. Revised Table 3 reports **46.15% versus 30.16% F1 at 3B and 49.21% versus 34.08% at 7B** against Search-R1. These are system-level comparisons with differing training resources. The reported seven-task mean selects the QA configuration. With both auxiliary and outcome training retained, adding the complete coordination stage raises the selected 3B mean from **41.21% to 46.15% (+4.94%)**. The intervention evaluates rescoring, projection and their associated work together. This gain supports the practical value of using outcome feedback at the auxiliary candidate to coordinate how both training signals enter the accepted update.

Dear Reviewer 8YhD,

Thank you again for asking about the concrete relationship between the selective-gradient estimator and the QA application. We would like to highlight additional experiments in the revised manuscript that directly address this question.

The additional QA experiments apply the selective-gradient estimator itself to outcome optimization. Appendix B.3.2 and Table 25 use Equation (1) to estimate raw PPO outcome gradients through cached-gradient predictions, adaptive reference acquisition and residual correction. Each acquired gradient also evaluates the guidance–acquisition rules. Across four paired RL seeds under the additional QA protocol, adaptive SRAO uses 35.33% fewer reference-gradient recomputations, averaging 97 per run versus 150 for full-reference training, with observed seven-task mean F1 of 22.81% versus 22.21%. These comparisons share a supervised initialization, a reconstructed BM25 corpus and 1,277 evaluation questions across seven tasks.

Appendix B.3.3 compares estimators at the same recorded policy states, using the same fixed rule weights. At matched expected reference cost, conditional aggregation reduces gradient-estimation MSE by 57.45% relative to randomly executing the same rules. This comparison measures the estimation gain from aggregation on QA gradients.

These experiments directly connect QA training to the selective-gradient construction analyzed in the paper. They complement the original study of outcome-guided displacement coordination in Equation (8). Appendix B.3.1 and Table 24 further examine projection with candidate rescoring held fixed. Section 5.2 and the Conclusion now make QA’s role as an empirical application of the stochastic optimization contribution explicit.

We would be grateful to hear whether this additional evidence clarifies the relationship between the two applications, and would be happy to address any remaining questions.

Best regards,
The Authors

Reviewer_NdTGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SRAO tries to reduce gradient computation by reusing cached gradients and occasionally computing a fresh gradient to correct them. It learns how to combine predictions and query rates. The distillation experiment reports similar average accuracy with fewer reference-gradient batches. The paper also gives a theoretical analysis and a separate agent QA experiment.

Critical strengths
  1. The paper provides a theoretical explanation for combining and learning query rules, with explicit assumptions and bounds.
  2. In the 100-step experiment, SRAO reaches a similar average score to full feedback while computing fewer reference gradients.
Critical weaknesses
  1. It is unclear whether learning the eight rules is better than simply using a fixed query probability. Table 11 reports a direct comparison, where the fixed strategy gets a higher score with fewer reference computations.
  2. The query-cost weight is set from four initial squared gradient norms. However, the initial gradient can be much larger than later iterations. The paper also does not show how much the results change when the query-cost weight is changed.
  3. The paper reports fewer reference computations, but does not show how much total training time is saved. Students still generate answers on steps without a reference query.
Questions for authors
  1. Does the learned strategy consistently outperform fixed query probabilities or querying every few steps? Please compare across several seeds with similar reference budgets, and report average accuracy, variation across runs, and training time.
  2. What are the student answers used for on steps without a reference query? Could the method decide whether to query before generating those answers, and skip generation when no query is needed?
5 comments

Thank you for asking whether learning the 8 rules improves on fixed or periodic querying, how the initial query-price calibration affects training, and whether unused student generation can be skipped. We now report accuracy, variation, reference counts and training time across multiple runs, together with 2 price-sensitivity studies and separate measurements of training-policy efficiency and generation-skipping savings. These results appear in revised §5.1 and App. B.2,.


W1/Q1 — Accuracy, reference budgets, variation and training time.

Fixed and adaptive acquisition share SRAO's corrected-gradient framework. Fixed-acquisition SRAO is the constant-rate, single-rule configuration of the corrected-gradient framework, retaining the evolving cache and reference correction. Acquired references both correct the current update and evaluate the available rules. The sharp relation between aggregation gain and learning cost supports learning their weights.

The fixed/periodic schedule comparison evaluates adaptive SRAO (guidance + acquisition), which learns weights over the 8 rules to adapt both the guidance mixture and query probability. To examine acquisition learning with a shared prediction rule, we also evaluate adaptive SRAO (acquisition only). It learns when to query using the same unscaled, evolving cached-gradient prediction rule as fixed-acquisition SRAO.

The framework saves reference computation in both settings. Adaptive SRAO (guidance + acquisition) retains similar observed mean accuracy to full feedback with 35.5% fewer references (App. B.2, Table 8). In the MATH500 comparison below, fixed-acquisition SRAO gives 91.65% versus 91.70% with 40% fewer references. We then compare adaptive SRAO configurations with fixed and periodic schedules to measure the additional value of adaptation.

Adaptive SRAO (guidance + acquisition) adds 1.57 points over fixed p=0.5 on the three-task mean, including 3.89 points on HMMT2025. It also improves the mean over periodic querying by 0.52 points. This additional comparison evaluates guidance and acquisition learning together.

This three-run comparison evaluates all questions in AIME2024, AIME2025 and HMMT2025 at Avg@12. The periodic cached-gradient control queries on five of every 8 steps after calibration. Task scores are mean ± sample SD. The last column equally averages the task means.

PolicyAIME2024 (%)AIME2025 (%)HMMT2025 (%)Three-task mean (%)
Fixed-acquisition SRAO, p=0.555.19 ± 1.4339.26 ± 1.1225.18 ± 0.8539.88
Periodic cached-gradient control54.73 ± 1.4441.02 ± 0.8927.04 ± 0.9840.93
Adaptive SRAO (guidance + acquisition)54.81 ± 0.1640.46 ± 1.9729.07 ± 0.5841.45

Reference expenditure and training-loop time for these same runs are also reported as mean ± sample SD:

PolicyReference batches/runTraining loop (min)
Fixed-acquisition SRAO, p=0.560.33 ± 5.03103.07 ± 1.10
Periodic cached-gradient control64.00 ± 0.00103.50 ± 0.17
Adaptive SRAO (guidance + acquisition)65.33 ± 16.44103.70 ± 3.03

These tables report accuracy, variation and actual resource use for the same training runs. Adaptive SRAO (guidance + acquisition) uses 65.33 references per run, compared with 60.33 for fixed acquisition and 64.00 for periodic querying. Training-loop time includes calibration and generation but excludes setup and saving. W3/Q2 reports separate elapsed-time studies including those stages. The complete schedule comparison appears in App. B.2.4, Table 19.

The mean-rate comparison addresses acquisition learning at comparable reference expenditure. Adaptive SRAO (acquisition only) and the fixed controls use the same evolving cached-gradient rule. On the AIME2025/HMMT2025, adaptive SRAO reaches 30.83 ± 0.42% versus 29.72 ± 0.24%, a 1.11-point gain with 2.17% fewer references (225 versus 230 per run). All three paired combined gains are positive. Each constant rate is set retrospectively to its paired adaptive run's mean post-calibration probability. The controls use fresh query draws, and the comparison reuses the adaptive evaluations.

A four-run MATH500 comparison gives a 0.75-point gain over fixed p=0.5 at nearly identical reference expenditure. The table uses all 500 questions at Avg@1. Accuracy is mean ± sample SD. Reference counts and loop times are per-run means.

PolicyMATH500 accuracy (%)Reference batches/runTraining loop (min)
Full feedback91.70 ± 0.70100.00119.60
Fixed-acquisition SRAO, p=0.591.65 ± 1.2260.0074.73
Adaptive SRAO (acquisition only)92.40 ± 0.8559.5073.40

This table separates the framework's reference savings from the additional accuracy gain of learning acquisition. The generation-skipping comparison under W3/Q2 measures the implementation saving within each policy.

W2 — Initial calibration and query-price sensitivity.

The initial norms calibrate a fixed query price, while newly observed prediction residuals drive adaptation. The first four full-feedback updates set the base price to the mean squared reference-gradient norm. Section 5.1 now explains these two roles. Appendix B.2.5 (Table 20) reports both the price sweeps and the queried-gradient analysis. For a prediction PP, the conditional error–cost objective is

(p−1−1)E∥G−P∥2+κp.\left(p^{-1}-1\right)\mathbb E\|G-P\|^2+\kappa p.

The useful query rate therefore depends on the evolving prediction error as well as the fixed price. Calibration expresses acquisition cost in squared-gradient units, and subsequent rule-weight updates use newly observed prediction residuals. To examine the change in gradient scale, we measured late squared-gradient-norm/initial-price ratios in the three-run study of adaptive SRAO (guidance + acquisition). Their medians range from 0.142 to 2.032 on queried steps, showing that later observed norms can be smaller or larger. The price sweeps below measure how acquisition and task performance respond to changing the calibrated price.

The four-run MATH500 sweep shows that lowering the price increases acquisition, while the default price attains higher observed mean accuracy with fewer references. The columns specify multipliers of the calibrated query price. Acquisition probabilities continue to be learned.

ControllerReferences at 0.25 × base priceReferences at base priceAccuracy: 0.25 × price → base price
Adaptive SRAO (acquisition only)68.0059.5091.70% → 92.40%
Adaptive SRAO (guidance + acquisition)75.7562.2592.30% → 92.65%

Reference counts increase at the lower price in every run for both controllers. A separate three-seed contest sweep of adaptive SRAO (guidance + acquisition) extends the sensitivity comparison to prices below and above the default. Entries are mean ± sample SD.

Price multiplierReferences/runAIME2024 (%)AIME2025 (%)HMMT2025 (%)
0.569.33 ± 4.5154.54 ± 0.7038.61 ± 1.0027.78 ± 0.48
165.33 ± 16.4454.81 ± 0.1640.46 ± 1.9729.07 ± 0.58
249.67 ± 12.6654.17 ± 1.2138.06 ± 1.0029.17 ± 1.47

Raising the multiplier from 0.5 to 2 reduces reference use in every run and by 28.4% on average. These studies quantify the response to price and the accompanying task performance. The initial scale is fixed while residual feedback continues to drive adaptation.


W3/Q2 — Unqueried answers, generation skipping and total time.

Yes—generation can be skipped in this cached-gradient implementation. We have documented the implementation and paired verification in Appendix B.2.6 (Table 21). Unqueried answers previously served logging only. The cached prediction and query decision are available before generation, so an unqueried step can directly apply the no-query prediction P0P_0. Acquired references supply the residual correction and the controller's reference-error feedback. This ordering removes unused generation while preserving the training update.

The retained/skipped pairs preserve reference counts, recorded training histories and saved adapters. Elapsed time below includes setup and saving. These are one-pair measurements per policy, using the same device within each pair and fixed execution order under shared-host conditions.

PolicyStudent generations: retained → skippedReferences/runElapsed time: retained → skipped (min)
Full feedback3,200 → 3,200100118.75 → 116.69
Adaptive SRAO (guidance + acquisition)3,200 → 1,98462109.79 → 76.74
Fixed-acquisition SRAO, p=0.53,200 → 2,08065109.59 → 78.02

The elapsed reductions are 30.11% and 28.81% for adaptive SRAO (guidance + acquisition) and fixed-acquisition SRAO, respectively. Full feedback retains all 3,200 student generations and serves as the unchanged-work control.

Separately, the four-run policy comparison with query-before-generation enabled gives 77.96 versus 122.69 minutes for Adaptive SRAO (acquisition only) versus full feedback, including setup, training and saving: 36.46% less measured time, with MATH500 accuracy 92.40% versus 91.70%. Reference batches and student generations both decrease by 40.5%. The four-run comparison measures the resulting policy's training efficiency. The paired intervention measures the implementation saving within a policy. Both training clocks cover setup, training and saving, with task evaluation timed separately.

Dear Reviewer NdTG,

Thank you again for your time and thoughtful feedback. We have provided point-by-point responses to your concerns and updated the manuscript with corresponding clarifications and additional experimental evidence.

As the discussion period approaches its close, we would be grateful to hear whether these responses address your concerns, and whether any questions remain. We would be happy to provide further clarification during the remaining discussion period.

Thank you for considering our responses and revisions.

Best regards, The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Kindly Requesting for Review of Our Rebuttal

Dear Reviewers and Area Chair,

Thank you again for your time and thoughtful feedback. We would like to follow up on our responses and revised manuscript. For convenience, we summarize the revisions addressing each reviewer’s main concerns:

  • Reviewer NdTG: We added comparisons with fixed and periodic querying across multiple runs, query-price sensitivity studies, and measured training time, including savings from skipping unused student generation (§5.1 and Appendices B.2.4–B.2.6).
  • Reviewer 8YhD: We added acquisition-only comparisons using the same evolving cached-gradient prediction rule to assess the additional benefit of learning when to query (§5.1, Table 2 and Figure 3). We clarified that fixed acquisition is a simple SRAO configuration, and explained the relationship between selective gradient estimation and QA coordination (§§3.1 and 5.2, Eqs. (1) and (8)).
  • Reviewer UtPj: Thank you for confirming that the adaptive-versus-fixed acquisition concern was addressed. To address your remaining concern about QA, we have explicitly foregrounded stochastic optimization and presented QA as its empirical application in the Introduction, §5.2 and Conclusion. We added experiments applying the selective estimator directly to QA training, measuring gradient-estimation error on QA training histories, and isolating projection with candidate rescoring held fixed (§5.2 and Appendices B.3.1–B.3.3, Tables 24–25). These complement the original coordination ablation in Table 3. The expanded distillation replications and requested zero-prediction comparison appear in Appendices B.2.1–B.2.3 and §4.2/Appendix B.1, respectively.

As the discussion period approaches its close, we would particularly appreciate a brief follow-up from Reviewers NdTG and 8YhD on whether our responses address your concerns and whether any questions remain. We also welcome further feedback from Reviewer UtPj and the Area Chair on the additional QA evidence and revised positioning. We would be happy to provide further clarification during the remaining discussion period.

Thank you again for your consideration.

Best regards, The Authors

AuthorsOpenReview ↗

Summary of revision and supporting evidence

Dear Area Chair and Reviewers,

Thank you for the discussion. For convenience, we summarize the contribution and supporting evidence in the revised manuscript.

SRAO’s central contribution connects the estimation gain from combining guidance–acquisition rules to the statistical cost of learning their weights. Each acquired reference both corrects the estimate and evaluates every rule. Fixed acquisition is a simple configuration of this framework; online rule learning provides adaptation.

  • Training efficiency: In the four-run MATH500 study, adaptive SRAO achieves 40.5% fewer reference-gradient batches and 36.46% less measured elapsed time for setup, training and checkpoint saving (Table 2; Appendix B.2.6), with 92.40% versus 91.70% observed mean accuracy.
  • Additional value of acquisition learning of adaptive SRAO over fixed SRAO: Comparisons retaining the same evolving cached-gradient rule and residual correction show gains of 0.75 points on MATH500 and 1.11 points on the AIME2025/HMMT2025 mean at comparable reference expenditure (Table 2; Figure 3).
  • Direct application in QA: The additional QA training experiments use the selective estimator in Equation (1), requiring 35.33% fewer reference-gradient computations, with observed mean F1 of 22.81% versus 22.21% (Table 25). At the same recorded QA policy states, fixed rule weights and expected reference cost, aggregation reduces estimation MSE by 57.45% relative to sampled-rule execution (Appendix B.3.3).

The revised Introduction and Conclusion foreground stochastic optimization, with QA presented as its empirical application. The coordination ablations and their protocols appear in §5.2 and Appendix B.3 (Tables 3 and 24).

Thank you again for your consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.