acceptodds
Under review as a conference paper at ICLR 2027

Learning While Generating in a Single Forward Pass

Abstract

Deployed LLMs constantly receive feedback from users, verifiers, and task outcomes. Test-time adaptation uses this feedback so that each interaction improves the next one. Today, this learning needs a second pass after feedback arrives, which either reruns the model over the response or keeps all of its activations until then. We observe that the model computation in this second pass does not depend on the feedback, so it can be done in the forward pass that generates the response. Based on this observation, we propose **One-Pass Forward-Trace Adaptation (OFTA)**, which learns while generating in a single forward pass. As the model generates each token, OFTA also records how the log-probability of the response changes along a random parameter direction. This design brings three benefits. First, it removes the second pass. When feedback arrives, OFTA turns the record into an update with a few simple operations, without running the adapting model again, using backpropagation, or keeping activations. Second, the record works for immediate and delayed feedback, and one record supports many feedback losses. Third, moving the computation into generation costs nothing in update quality. Theoretically, OFTA gives the same directional derivative as a separate pass after feedback, so it keeps the unbiased estimate of forward-gradient methods. Experiments on Qwen3 series models show that, in the tested settings and counting all extra work, OFTA lowers request time, applies updates almost as soon as feedback arrives, uses much less memory than backpropagation, and learns as well as a separate pass. In simpler terms, OFTA lets a model learn from a response in the same pass that writes it.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Test-Time Adaption, Large Language Models, Continual Learning, One-Pass

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wPHWOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes One-Pass Forward-Trace Adaptation (OFTA), a test-time adaptation method for LLMs that eliminates the second pass. OFTA measures how the log-probability of the response changes along a random parameter direction when LLM generates tokens. Experiments on Qwen3 models demonstrate reduced latency and memory usage while maintaining learning quality.

Critical strengths
  1. This paper addresses a bottleneck in LLM deployment by eliminating the prohibitive computational overhead of re-running models for test-time adaptation. It makes the core motivation both timely and highly impactful.

  2. The manuscript is well-organized with clear logical flow, supported by rigorous and thorough theoretical analysis that grounds the proposed single-pass strategy.

Critical weaknesses
  1. Figure 1 is overly cluttered with excessive text and small fonts, making it difficult to parse. The figure also exhibits noticeable AI-generated styling, so the authors should simplify the layout and better highlight the core contributions.

  2. The experimental validation is insufficient. The online TTA evaluation is limited to simulated binary verifier feedback on the GSM8K dataset and Qwen3. There is a lack of assessment on more complex open-ended tasks or continuous reward signals, which makes it difficult to comprehensively reflect the effectiveness.

  3. The analysis of K greater than 1 is only briefly mentioned in Appendix D.5. This trade-off should be central to the proposed method. In models with millions of parameters, setting K to 1 results in high gradient noise. However, using multiple directions linearly increases generation latency, which contradicts the single-pass low-overhead selling point of the paper.

  4. The theoretical proofs assume infinite-precision arithmetic and fixed sampled tokens. In practice, real LLMs suffer from accumulated errors under low-precision quantization, and stochastic sampling itself is non-differentiable. Furthermore, the theory only guarantees unbiasedness while single-direction estimation causes variance explosion across millions of parameters, making actual convergence unachievable.

Questions for authors

See the weaknesses above.

5 comments

Thank you for the review. We address figure clarity and experiments here, direction count in Comment 2, and theory and numerical evidence in Comment 3.

W1. Figure clarity and presentation.

We have redrawn Figure 1 with less text, a clearer visual hierarchy and consistent visual elements. The top panels directly contrast a separate post-feedback model pass with OFTA's generation-time trace acquisition. The lower panels show the workflow: generate a response and acquire sensitivities in one pass, save the trace, then combine it with feedback to update the parameters. This makes the core contribution explicit: the model-dependent derivative work takes place during generation, so no additional model pass is needed after feedback arrives.

W2. Experimental coverage, open-ended tasks and continuous feedback.

We would first clarify what the original experiments establish. OFTA changes when derivative computation is performed. Model sensitivities are acquired while producing an answer and used once feedback becomes available. The experiments therefore test whether this change preserves the directional update, reduces adaptation cost and supports effective learning. The paper already provides matched derivative/update checks, full-request cost measurements and task-quality results to validate these claims.

The tasks in the main paper are not limited to binary classification or GSM8K. Table 1 covers ten tasks on Qwen3-1.7B and Qwen3-8B, including generative question answering on SQuAD and DROP. We chose these tasks because they are established, widely accepted benchmarks in ZO language-model fine-tuning, used in work such as MeZO and DiZO. They provide a standard basis for assessing learning quality, as well as meaningful comparison with prior methods.

Qwen3's open weights enable BP checks of sensitivities through its KV cache. OFTA's identity uses differentiable statistics and state tangents, not Qwen-specific layers.

Model coverage beyond Qwen3. To directly address your concern about model coverage, we add supervised SQuAD adaptation on OPT-2.7B. All three methods use the same LoRA adaptation space and 8192 updates, with greedy decoding on the full SQuAD validation set.

MethodF1
Base28.2
OFTA59.9
MeZO60.5
FO84.6

OFTA's F1 is close to MeZO's on this second model family, providing direct empirical evidence that its effectiveness is not specific to Qwen3.

These task-quality experiments use known supervision, while the GSM8K study tests adaptation from feedback received after generation. Following your suggestion, we further add SAMSum summarization and SQuAD adaptation driven by graded token-F1 feedback.

Open-ended generation: SAMSum summarization. Summarization requires selecting the important information in a dialogue and expressing it as a free-form answer. OFTA uses reference-summary cross-entropy, with greedy decoding for ROUGE evaluation (K=1K=1, 8192 updates, seed17).

SAMSum: OFTA results and published references

MethodROUGE-1ROUGE-2ROUGE-L
Base31.7411.4823.61
OFTA40.6016.3332.14
MeZO39.7315.9331.17
FO reference47.4622.9039.24

The improvements across all three metrics show that OFTA's updates are useful for open-ended generation, where the desired behavior involves content selection and concision. The following real example makes this change visible.

Real summarization case: the same dialogue before/after adaptation

Instruction: Summarize the following conversation

Dialogue:

Grace: You like mu curly hairstyle or straight?
Andrew: Straight
Grace: Ok I will come with the curly style tomorrow

Reference:

Grace will come with curly hair tomorrow.

Before OFTA / Base:

Grace asks Andrew if he likes curly or straight hair, and Andrew says straight. Grace then agrees to bring her curly hair style the next day.

ROUGE-1/2/L: 18.18 / 6.45 / 18.18

After OFTA / step 8192:

Grace will come with the curly style tomorrow.

ROUGE-1/2/L: 80.00 / 46.15 / 80.00

The base output retells the question-and-answer exchange. After adaptation, the summary focuses on the outcome: Grace's chosen hairstyle for tomorrow. It preserves the important fact while removing the conversational details, illustrating the improvement in content selection.

Continuous feedback: SQuAD token-F1 rewards. In this experiment, the reward is returned only after a response has been generated. A partly correct answer receives partial credit, and that graded score directly drives the parameter update. Thus F1 serves as the learning signal as well as the evaluation measure.

SQuADGreedy F1T1 F1
Base66.1765.39
OFTA K181.0480.48

W3. Direction count, noise and generation cost.

Our experiments identify K=1K=1 as the best practical quality–cost trade-off for our low-overhead online setting. Larger KK adds computation and tangent storage; the updated memory measurements below quantify the storage cost of our optimized implementation. This trade-off explains why we prioritized experiments that verify update consistency, computational savings and effective learning in the main text, while placing the K>1K>1 discussion in the appendix. We now provide the detailed comparison across K=1,2,4,8K=1,2,4,8 below and will include it in the main text.

Single-pass means one autoregressive generation. All K directions share its tokens, primal activations and primal KV cache. Each adds derivative computation and tangent storage that can grow roughly linearly with K. K controls the derivative work within this generation, without generating the full answer K times.

KT1 F1Generation+sensitivity (s)
177.76941.2251
278.55311.5263
479.50532.2230
880.45133.8600

Quality: SQuAD, p=8192, B8, seed29, 4096 updates; T1 averages three decoding repeats. Times are original-backend medians over 18 requests; feedback-time updates take 0.838–1.174 ms. These are separate quality and timing experiments.

KPeak allocated (GiB)
17.8541
28.2740
49.1139
811.0205

Peak allocated: H100, FP32/TF32 off, B8, p8192 (L11 v-projection LoRA-B), prompt cap320, 16 output tokens; includes model residency, direction preparation, generation and scoring, excludes optimizer execution. Same-config native HF inference uses 7.3892 GiB. Across 20 checks against the reference implementation, scores, directional derivatives and Adam update coordinates have zero maximum difference; generation/EOS behavior agrees.

Your concern about high-dimensional noise is important. Averaging independent directions reduces estimator variance by 1/K1/K, while the dimensional penalty remains. The practical question is whether a noisy K1 update can still improve the task. We test this on SAMSum with 8,716,288 trainable parameters across all 196 linear layers (rank8 LoRA A+B):

SAMSumROUGE-1ROUGE-2ROUGE-L
Base31.6211.8124.10
OFTA K141.2617.3433.44

Reference-summary cross-entropy, 4096 updates, seed17, Adam, learning rate 3e-4. These gains directly show useful K1 learning with millions of trainable parameters.

W4. Stochastic sampling, numerical accuracy and learning.

Ideal arithmetic is the reference level for establishing algorithmic correctness, not a requirement for infinite-precision hardware. Such real-valued analysis is common in ZO optimization. MeZO derives unbiasedness as perturbations vanish; MeZO-SVRG and LOZO analyze smooth objectives with bounded stochastic-gradient variance; DiZO also assumes smoothness. MeZO-SVRG separately tests FP32/BF16. OFTA connects its identity to finite-precision implementation through error bounds and numerical checks. Fixed sampled tokens define the differentiated score, without assuming differentiable sampling or deterministic generation.

Stochastic sampling. We differentiate the realized response's score qϕ(x,a)=log⁡πϕ(a∣x)q_\phi(x,a)=\log\pi_\phi(a\mid x) with aa fixed. This is the score-function treatment of discrete samples in stochastic computation graphs, not a derivative through the token draw. For reward-driven learning,

∇ϕEa∼πϕ[r(a)]=Ea∼πϕ[r(a)∇ϕlog⁡πϕ(a)]\nabla_\phi \mathbb{E}_{a \sim \pi_\phi}[r(a)] = \mathbb{E}_{a\sim \pi _ \phi}[r(a)\nabla_\phi\log\pi_\phi(a)]

This uses parameter-independent rewards, the actual sampling/stopping law, locally fixed support and valid interchange of differentiation and expectation. Holding the realized tokens fixed is the correct score derivative; it does not require a deterministic generator or differentiation through the discrete draw. Nor must changed parameters regenerate the same response.

Finite precision. Appendix F.10 bounds how errors in saved statistics, sensitivities, outer loss gradients and their final contraction propagate into the update. This quantifies departures from the identity when stored values are inexact. We compare scalar directional derivatives DvLD_vL with (∇ϕL)⊤v(\nabla_\phi L)^\top v from independent BP, with parameters, realized responses and computation graphs matched (FP32, TF32 off). These are two ways to compute the same scalar in exact arithmetic.

Directional-derivative checkMaximum error
Matched JVP/BP projection; 18 scalars0.000299% relative
Per response; 720 scalars0.009990% relative
Aggregate; 90 scalars0.202639% relative
Trained K1/K4; per response2.095e-5/3.164e-5 absolute
Trained K1/K4; aggregate2.340e-7/3.786e-7 absolute

The K1/K4 labels denote checkpoints after 1024 SQuAD reward-training steps (seed19, p=8192), checked on four train prompts. Each checkpoint is tested with one and four directions. These are numerical errors in directional derivatives, not errors between the random vector g^=(DvL)v\widehat g=(D_vL)v and the full gradient. An accurate DvLD_vL can coexist with high estimator variance. The checked forward-score gap is zero, so sensitivity acquisition preserves these scores.

We also compare generation-time acquisition with matched post-feedback execution over 1024 updates (K1, seed19). Saved replay scores agree, and the saved parameter checkpoints match byte for byte. This also checks the effect on a sequence of updates.

NF4-quantized weights and accumulated update error. To address your quantization concern directly, we retain actual NF4-quantized weight values and execute the model in FP32. In this separate numerical control, each row compares OFTA with an independent BP directional projection on the same model and realized response; both sides use the same K=1K=1 update rule.

SettingMax. relative directional-derivative errorTrainable-parameter relative L2 deviation after 32 updates
FP32 control0.000628%0.000646%
NF4 weight values, FP32 execution0.022962%0.001398%

All 16 NF4 directional-derivative signs agree with BP, and all recorded traces, caches, parameters and Adam states remain finite. The parameter comparison tests error accumulation across repeated Adam updates. Its small final discrepancy supports accurate OFTA sensitivities and closely matched updates with NF4 weight values under FP32 execution.

Optimization beyond unbiasedness. For a fixed, lower-bounded LL-smooth objective, exact directional derivatives and fresh independent standard Gaussian directions at each step, synchronous SGD with averaged directions satisfies

\le F(\phi)-\eta(1-L\eta c_K/2)\|\nabla F(\phi)\|^2, \qquad c_K=1+(p+1)/K.$$ Here $p$ is the trainable dimension. The bound gives expected progress away from stationary points when $0<\eta<2/(Lc_K)$. High variance imposes a stricter step-size condition, rather than ruling out descent. This analysis, together with the graded-feedback gains and million-parameter results above, provides theoretical and empirical evidence for useful learning within the reported settings.
Reviewer_JqK2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes One-Pass Forward-Trace Adaptation (OFTA), a framework for enabling large language models (LLMs) to adapt from feedback without an additional model pass after generation. Experiments on Qwen3 models demonstrate numerical agreement with independent derivative computations, reduced request latency and memory consumption in the evaluated settings, and comparable adaptation performance to post-feedback forward-gradient methods.

Critical strengths
  1. A well-motivated approach to eliminating redundant model computation after feedback. OFTA effectively separates feedback-independent sensitivity computation from feedback-dependent loss evaluation, allowing directional derivatives to be acquired during generation.
  2. Promising efficiency improvements with controlled numerical validation. The experiments demonstrate practical benefits under the evaluated configurations. It also substantially reduces post-feedback update latency while maintaining comparable adaptation performance to post-feedback JVP. These results suggest that incorporating sensitivity acquisition into generation can improve adaptation efficiency.
Critical weaknesses
  1. Limited generalizability of the online adaptation evaluation. The main GSM8K experiment evaluates 100 training-split questions preselected for greedy failure despite having 4–6 correct answers among eight sampled responses (Appendix B.1), potentially overestimating adaptation effectiveness. Moreover, the main online protocol avoids parameter staleness by completing each update before the next interaction. Although Appendix E.3 examines delayed feedback, downstream performance under concurrent asynchronous adaptation remains unverified, limiting the evidence for real-world effectiveness.
  2. Insufficient validation of efficiency advantages under realistic deployment conditions. The main latency comparisons use different execution backends, making it difficult to isolate the benefits of OFTA (Appendix B.2). The matched-backend experiment involves only 8,192 trainable parameters (Appendix D.3), while teacher-forced replay outperforms OFTA on short responses (Appendix D.6). Furthermore, memory evaluations focus on very short outputs. More comprehensive comparisons under matched implementations and realistic workloads are needed to establish the generalizability of the efficiency gains.
  3. Insufficient investigation of the trade-off between gradient variance and adaptation efficiency. OFTA relies on high-variance random directional gradients, with estimation error scaling with the trainable parameter dimension (Section 5). However, the main online experiments use only one direction despite having approximately 8.7 million trainable parameters. Table 1 also shows that backpropagation outperforms OFTA on most tasks despite using substantially fewer updates. Although Appendix D.5 investigates estimator variance, the impact of direction count on actual adaptation performance and computational efficiency remains insufficiently explored.
Questions for authors
  1. How does downstream task performance change when feedback arrives asynchronously and intervening updates introduce nonzero parameter drift?
  2. Can the authors provide more extensive matched-backend comparisons against pre-feedback JVP, post-feedback JVP, and replay-based alternatives under realistic serving workloads? In particular, reporting end-to-end latency, generation latency, throughput, and peak memory across different response lengths, concurrent request loads, and adaptation-space sizes would help establish how much of the efficiency improvement is attributable specifically to one-pass sensitivity acquisition.
  3. How does increasing the number of random directions affect adaptation accuracy, convergence speed, and total computational cost under actual verifier-feedback settings? Can the authors provide accuracy-versus-wall-clock-time comparisons with backpropagation to clarify the practical trade-off between gradient variance and efficiency?
4 comments

Thank you for the review. We address evaluation and efficiency here, and direction count and BP in our second comment.

W1/Q1. Evaluation scope and asynchronous learning.

The purpose of Figure 5 is to show that OFTA supports RL-style training, complementing Table 1's supervised results. OFTA acquires model sensitivities during generation and combines them with the subsequent verifier reward to update the policy. After 50 updates, its corrective accuracy closely matches post-feedback JVP (60% vs. 61%). Together, these experiments support OFTA as a new learning paradigm that accommodates both supervised losses and reward-based objectives.

To answer your staleness question, we add SAMSum with delayed updates. Acquired directional information is delivered after four subsequent batch8 acquisition groups and applied to the current parameters with the current Adam state. Intervening updates therefore introduce real parameter drift.

We use Qwen3-1.7B with K=1 and rank-64 LoRA training.

Published updatesROUGE-1ROUGE-2ROUGE-L
031.623111.813724.1027
51232.195411.854524.5181
102432.187311.887024.4787
204832.753312.180025.1966
409635.876813.616327.4304
819240.212316.054831.8489

Of 8192 publications, 8191 have nonzero drift. Drift is the L2 distance between the selected parameters just before application and at acquisition. For nonzero events, min/median/P95/max = 0.02466/0.05690/0.08396/0.18638, with version gaps 0–4. The quality curve shows that historical sensitivities remain useful for downstream learning despite intervening updates. This is a single-GPU logical-delay experiment; all 768 checkpoint summaries were independently rescored.

W2/Q2. Matched comparisons and serving workloads.

We test pre-/post-feedback JVP and replay across parameter scales, output lengths and simultaneous request loads.

Backend and adaptation size. Fixing the response, parameters, direction, objective, update and backend, OFTA/post-cached JVP match scores, directional scalars and published parameter hashes in all 24 warm/timed pairs.

Trainable parametersOFTA request sPost-cached JVP sTime saved
81927.0466 ± 0.342511.2639 ± 0.571737.44%
8,716,2889.9425 ± 0.408015.4118 ± 0.554735.49%

Qwen3-1.7B/H100 80GB, FP32/TF32 off, 128 output tokens, B=K=1, one-second feedback wait; the larger space uses rank8 LoRA over 196 linear layers. Times are mean ± sample SD of three process means (one warm-up + three timed requests each). This isolates acquisition placement.

Long-response serving workload. We add 8,192-token outputs at p=8,716,288, B=K=1 and immediate feedback, fixing the trajectory, initial parameters, direction, objective and update. TF-JVP uses the original eager full-sequence replay.

Output tokensMethodComplete request (s)Peak allocated (GiB)
8,192OFTA464.90 ± 15.4810.3944
8,192Cached JVP734.91 ± 44.6913.9868
8,192TF-JVP293.10 ± 9.7828.6555

OFTA memory is newly measured over eight retained runs. Times retain the original three-process protocol; requests include generation, acquisition and publication. Memory includes the model.

OFTA saves 36.74% of request time versus matched cached JVP and 63.73% of peak memory versus TF-JVP. In the original 24 GiB H100 allocator-budget control, OFTA completed in 3/3 processes, while TF-JVP exhausted memory during replay in all three.

Pre-feedback JVP and request loads. Controls span 32/128 tokens, both parameter spaces and B1/B4 physical microbatches. Below are three-process means at p=8,716,288 and 32 tokens:

BMethodGen.+prep sRequest sFeedback→publish msSamples/sPeak GiB
1OFTA2.43823.452413.8530.29096.7477
1Pre-cached1.41573.80991393.8030.26326.7477
1Post-cached1.40204.80262400.3380.20866.7477
4OFTA2.57753.58587.9641.11707.3852
4Pre-cached1.46713.97841511.0231.00927.3833
4Post-cached1.42944.98922559.4320.80297.3833

B4 batches four sequences. Gen.+prep is answer delivery, including OFTA's sensitivity work; requests include the one-second feedback event and publication. Throughput averages each request's B/time; memory is peak allocated including the model.

Pre-cached uses the feedback waiting period but still performs a separate traversal. OFTA acquires sensitivities within the original generation. At B4, this cuts request time by 9.87%/28.13% and raises throughput by 10.68%/39.12% versus pre-/post-cached JVP, directly quantifying the one-pass benefit.

W3/Q3. Direction count, noise and generation cost.

Understanding how KK affects quality and cost is important. We gave priority to experiments that establish update consistency, computational savings and learning effectiveness, as these directly validate the proposed one-pass approach. To clarify how additional directions affect this balance, we provide the following comparison of quality, latency and memory for K=1,2,4,8K=1,2,4,8.

Single-pass refers to one original autoregressive generation. All K directions share its generated tokens, primal activations and primal KV cache. Each direction adds derivative computation and tangent storage, which can grow roughly linearly with K, but the full answer need not be generated K times. K therefore controls the derivative work within the same generation, rather than the number of full generations.

KT1 F1Generation+sensitivity (s)
177.76941.2251
278.55311.5263
479.50532.2230
880.45133.8600

Quality: SQuAD, p=8192, B8, seed29, 4096 updates; T1 averages three decoding repeats. Times are original-backend medians over 18 requests; feedback-time updates take 0.838–1.174 ms. These are separate quality and timing experiments.

KH100 peak allocated (GiB)
17.8541
28.2740
49.1139
811.0205

Memory is measured on H100 with Qwen3-1.7B, FP32/TF32 off, B8, p=8192, prompt cap320 and 16 output tokens. Peaks cover generation and scoring, including model residency, excluding the optimizer. The implementation shares primal KV, omits identically zero tangent slots and releases temporary tensors while retaining required historical tangents. Across 20 paired checks, maximum differences in scores, directional derivatives and Adam update coordinates are all zero; generation and EOS behavior match. K1 is only 0.465 GiB above matched native HF inference (7.3892 GiB).

Averaging independent directions reduces estimator variance by 1/K1/K, while the dimensional penalty remains. From K1 to K8, T1 F1 improves by 2.6819 points. K1 remains the lowest-cost tested direction count; larger K offers additional quality at higher compute and storage cost.

Comparison with BP: accuracy versus measured training time.

To answer your request for a BP comparison, we use a separate seed23 experiment on SQuAD. All arms use Qwen3-1.7B, p=8192, batch8, Adam3e-4 and token-F1 feedback returned after generation. BP differentiates the reward-weighted score over all selected coordinates; each arm learns from its own generated responses. Each cell below gives T1 F1 / cumulative training-request seconds. The common Base T1 F1 is 65.3948.

Method1024 updates: F1 / s4096 updates: F1 / s
OFTA K172.7069 / 1163.0179.2348 / 4467.77
OFTA K275.3017 / 1431.1079.9175 / 5554.78
OFTA K475.4793 / 1870.2680.8401 / 7303.51
OFTA K876.8616 / 2592.6881.3269 / 10073.19
Full-gradient BP83.9832 / 1021.7485.6922 / 3991.38

The arms run on separate H100 devices. Times are actual CUDA-synchronized cumulative generation-and-update clocks, excluding initialization, evaluation and checkpoint IO. T1 again averages three decoding repeats.

These results distinguish learning per update from learning per unit time. Averaging more directions reduces estimation variance and can improve early learning per update, but also makes each update more expensive. By 4096 updates, K1 has already improved T1 F1 by 13.84 points over Base. K4 and K8 add 1.61 and 2.09 points over K1, while increasing cumulative training time by 63.47% and 125.46%, respectively. The extra gradient information therefore brings a modest quality gain at a substantially higher time cost.

Together with the latency and memory measurements above, this leads us to select K=1K=1 as the best practical trade-off among the tested OFTA direction counts for our low-overhead online setting. K1 delivers substantial learning gains with the lowest directional compute and storage cost. Larger K remains an option when additional quality justifies that cost.

The BP reference shows the learning benefit of full-gradient information. OFTA's contribution is to acquire directional learning information during the original generation; W2's matched JVP controls isolate the savings from eliminating a separate acquisition traversal.

Thank you for the detailed response and additional experiments. The new results have addressed most of my concerns and substantially strengthened the empirical evaluation.

Regarding W1/Q1, the additional SAMSum experiments provide meaningful evidence that OFTA can support downstream learning under asynchronous feedback with nonzero parameter drift. However, concerns about the representativeness of the original GSM8K evaluation and generalization across independent test samples remain partially unresolved.

Regarding W2/Q2, the expanded matched-backend comparisons, larger adaptation spaces, and additional serving workloads largely address my concerns about the fairness and generalizability of the efficiency evaluation.

Regarding W3/Q3, the direction-count ablations and accuracy-versus-training-time comparisons provide valuable clarification of the trade-off between gradient variance and computational cost. Nevertheless, the BP baseline achieves higher accuracy with less cumulative training time in the reported SQuAD experiment, leaving some concerns about practical learning efficiency.

Overall, I appreciate the authors' thorough efforts. Although some concerns remain, considering the additional evidence and the paper's overall technical contributions, I will maintain my original rating of 3 (Weak Acceptance).

We are pleased that our responses and additional experiments have addressed most of your concerns. Thank you for recognizing the asynchronous learning evidence and the expanded efficiency comparisons.

Regarding W1/Q1, Figure 5 tests learning from rewards received after generation. Its adaptation and evaluation questions are disjoint, but both belong to the selected correction cohort. The result establishes correction within that cohort. We distinguish this from generalization to independently sampled GSM8K questions. The ten-task evaluation and SAMSum experiments provide complementary evidence on other tasks. Delayed SAMSum further demonstrates learning under nonzero parameter drift.

Regarding W3/Q3, BP serves as a full-gradient performance reference. It achieves higher accuracy in less training time in the reported SQuAD setting. However, TTA also involves memory and feedback-time costs. BP must retain backward activations while waiting for feedback or replay the forward computation afterward. OFTA avoids both requirements.

The paper measures these resource benefits in separate TTA workloads. OFTA reduces request time by 35.0–37.1% against BP replay in the response-length study. Its peak memory is 41.9–43.1% lower than BP in the batch16 short-output tests. Together with the SQuAD comparison, these results show the practical trade-off between full-gradient learning quality and TTA resource costs.

We will include the additional tables, experimental results, and these explanations in the camera-ready version. We sincerely appreciate your constructive feedback and positive assessment of our work.

Reviewer_6VvMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper applies ideas from forward-mode auto-differentiation to enable LLM training without needing additional passes or backprop beyond generation. The core idea is that one can approximate the exact full gradient ∇θf(θ)\nabla_\theta f(\theta) via first computing and storing, y=fθ(x,a),z=Jf(x,a)v,v∼N(0,I),y = f_\theta (x, a) , \quad z = J_f (x, a) v , \quad v \sim \mathcal{N}(0, I), Where yy and zz are both Rd\mathbb{R}^d (output dimension), and vv is Rp\mathbb{R}^p (parameter dimension). Jf(x,a)J_f(x,a) is the d×pd \times p Jacobian matrix of fθf_\theta with respect to model parameters θ\theta. Both yy and zz are computed in the forward pass via forward mode AD.

Having the above allows one to get a noisy estimate of the gradient. After the feedback rr receives, the loss can be computed and differentiated w.r.t. just the stored yy, and the full gradient estimate is, g^=(∇yL(y,r)⊤z)v\hat{g} = \big( \nabla_y L(y, r)^\top z \big) v This is unbiased in expectation because, E[g^]=E[∇yL(y,r)⊤(Jf(x,a)v)v]=E[vv⊤]⋅Jf(x,a)⊤∇yL(y,r)=I⋅∇θL(fθ(x,a),r)\mathbb{E}[\hat{g}] = \mathbb{E} [\nabla_y L(y, r)^\top \big( J_f(x,a) v \big) v ] = \mathbb{E}[v v^\top] \cdot J_f(x,a)^\top \nabla_y L(y, r) = I \cdot \nabla_\theta L\big( f_\theta (x, a) , r \big)

The benefit of doing this is faster and lower memory training at the cost of a noisier gradient from sampling vv. They empirically demonstrate their benefit on language modelling tasks where they are able to train with lower runtime and memory than full backprop, and achieves comparable performance than zero-th order methods.

Critical strengths

Promising direction for more efficient LLM training

I find the use of forward-mode autodiff for learning interesting and promising. In my opinion it is very worthwhile to explore alternatives to conventional backprop, and it seems to reduce the cost of training LLMs from feedback.

Specifically thinking about MeZO, OFTA appears to have several interesting properties:

  • Both methods rely on a random direction to estimate the full gradient, but OFTA is essentially “exact” while MeZO further approximates (via finite differences)
  • OFTA requires modestly more peak memory empirically as compared to MeZO in reported experiments
  • OFTA avoids the two additional forward passes required for MeZO, and therefore has much lower runtime compared to MeZO

A useful conceptual contribution

I appreciated the general conceptual contribution of how forward mode AD can be used for LLMs (for instance the model-dependent part can be computed before feedback arrives), which is generally an elegant idea.

A caveat is that I am not involved with zeroth order training or forward mode AD, so I am unaware of the literature landscape at large to know the novelty of this work or if there are more reasonable baselines for OFTA.

Critical weaknesses

Limited understanding of time and memory scaling to long sequences

I think the paper can do a better job with showing how different methods scale with generation length for both runtime and memory. Figure 4 is a very nice step in this direction, although it only reports time. It also only goes up to 256 tokens which is relatively short by modern LLM standards. As far as I know, Figure 3 reports peak memory, but only for 1-4 token outputs.

Would it be possible for the author to show how memory also scales for the different methods? While OFTA may not be the best here, it is nevertheless informative in my opinion to know how the different methods trade-off. Further, would it be possible to scale to longer generations in general? If resource constrained, having a more in depth discussion on if we would expect the current curve to continue extrapolate would be useful, such as through big-oh / asymptotic complexity analysis as a function of sequence length and parameter count.

Questions for authors

1) Comparison of methods. Could the authors provide a concise table comparing BP, MeZO, post-feedback JVP, and OFTA in terms attributes such as: (i) gradient estimator, (ii) number and type of model passes, (iii) time complexity, and (iv) memory complexity? This would help clarify the fundamental tradeoffs between these methods. Please feel free to use different columns if you think other attributes are more representative.

2) Long-sequence scaling. Relating to the weakness above, do the authors expect the runtime and memory of OFTA to behave in longer generations (e.g. 10k+ tokens)? For instance, since OFTA requires storing tangent KV-cache memory, do the authors foresee this as a potential efficiency bottleneck relative to other methods? How much does teacher-forced replay change the relative runtime of different methods at longer lengths?

3) Do you report the full improvement curves over the training set? To my understanding Fig 5 is a specifically chosen set of 100 questions that the base model with greedy decoding cannot solve? While this does demonstrate OFTA’s ability to learn, it feels a bit contrived. Do you have the same time vs. performance improvement on for instance the full train/eval set as a function of time, and potentially with both greedy and non greedy sampling?

4) Are there more relevant baselines? I am unfamiliar with this area. I understand there has been more developments in zeroth-order optimizers since MeZO [e.g. 1, 2]. Can the authors comment on if the recent developments on top of MeZO are or are less relevant as comparisons to OFTA (e.g. if their improvements are complementary to what OFTA is trying to accomplish)? For me personally, this point is less important than the above three, fwiw.

[1] Zhang, Yihua, et al. "Revisiting zeroth-order optimization for memory-efficient llm fine-tuning: A benchmark." arXiv preprint arXiv:2402.11592 (2024). [2] Zhao, Hanzhen, et al. "PaZO: Preconditioned accelerated zeroth-order optimization for fine-tuning LLMs." Advances in Neural Information Processing Systems 38 (2025): 111163-111195.

4 comments

Thank you for your positive assessment and constructive feedback.

Q1. Gradient estimators, passes, computation and memory.

We compare one response plus one update, excluding feedback waiting. Let CC be the arithmetic work of a complete primal response traversal (prefill and all decoding steps), αCα C the incremental work of one tangent lane, and βCβ C one backward traversal. The model fixes the response graph and trainable coordinates; both JVP rows share one primal across KK directions.

MethodGradient estimatorModel passes, including generationTime complexity / leading arithmetic workPeak memory
BP-ReplayExact trainable-coordinate gradientGeneration + replay forward + backward(2+β)C(2+β)CO(P+M+A+p)O(P+M+A+p)
MeZOKK-direction two-point finite differencesGeneration + 2K2K perturbed forwards(1+2K)C(1+2K)CO(P+M+q+K)O(P+M+q+K)
Post-feedback JVPKK-direction forward gradientGeneration + replay with KK tangent lanes(2+Kα)C(2+Kα)CO(P+(1+K)M+Kp)O(P+(1+K)M+Kp)
OFTASame forward gradient from historical tracesJoint generation with KK tangent lanes; no post-feedback model pass(1+Kα)C(1+Kα)CO(P+(1+K)M+Kp)O(P+(1+K)M+Kp)

At fixed precision, PP is model storage, pp trainable coordinates, MM ordinary cached-forward state including KV (excluding weights/directions/backward graphs), AA uncheckpointed backward activations/graph, and q≤pq\le p the largest in-place perturbation block. These memory bounds assume one interaction, plain SGD, blockwise MeZO and materialized JVP directions. Other optimizer/reference/version states are additional.

Parameter operations add O(p)O(p) for BP and O(Kp)O(Kp) for direction methods. A JVP traversal costs (1+Kα)C(1+Kα)C, not CC. WW denotes the leading model-side arithmetic work per interaction. OFTA saves one primal replay, Wpost−WOFTA=CW_{\rm post}-W_{\rm OFTA}=C, while retaining the directional work. Its derivative is at the acquisition parameters; estimator equality requires the same parameter point, objective, directions and computation conventions. Exact directional derivatives still give a random gradient estimate, not a full gradient.

Why K=1K=1 has the lowest leading work under a symmetric model. OFTA is strictly cheaper than all three when α<2α<2 and Kα<1+βKα<1+β. For K=1K=1, assume α=β=γ∈[1,2]α=β=γ\in[1,2] in a model dominated by derivative-active matrix multiplies. Either mode needs one or two forward-sized products, depending on which operands require derivatives. Then

WOFTA=(1+γ)C≤3C=WMeZO≤(2+γ)C=Wpost=WBP−Replay.W_{\rm OFTA}=(1+γ)C\le3C=W_{\rm MeZO}\le(2+γ)C=W_{\rm post}=W_{\rm BP-Replay}.

Both inequalities are strict for 1<γ<21<γ<2. These are cost-model assumptions, not measured coefficients or a wall-clock/time-to-quality ranking. Larger KK adds tangent work; BP's work is independent of KK.

OFTA's feedback work is O(Kp+Km)O(Kp+Km) when output-loss differentiation is O(m)O(m), where mm is statistic dimension. Retained-graph BP costs (1+β)C(1+β)C but holds backward state while waiting; serial multi-direction JVP repeats primal work. OFTA instead retains statistics/traces and seeds after generation, O((K+1)m+K+Rassoc)O((K+1)m+K+R_{\rm assoc}), with RassocR_{\rm assoc} denoting response/feedback metadata.

Weakness / Q2. Long-response runtime and memory.

We compare long responses on H100 80GB, Qwen3-1.7B, all-linear rank8 LoRA A+B (p=8,716,288p=8,716,288), FP32/TF32 off, 204-token prompt, B=K=1B=K=1. Paths share the trajectory, initial parameters, direction, sequence-score objective and update; EOS is suppressed. TF-JVP uses original eager full-sequence replay.

Output tokensPathComplete request (s)Peak allocated (GiB)
2,048OFTA118.94 ± 6.087.5733
2,048Cached JVP181.92 ± 9.028.5447
2,048TF-JVP74.33 ± 2.058.9257
8,192OFTA464.90 ± 15.4810.3944
8,192Cached JVP734.91 ± 44.6913.9868
8,192TF-JVP293.10 ± 9.7828.6555

Timed requests include generation, sensitivities and updates with immediate feedback. Original-backend times are mean ± SD over three independent processes, each with one warm-up and three timed requests. OFTA memory is updated separately and includes the model.

OFTA saves 34.62%/36.74% versus matched cached replay. TF-JVP scores the response in parallel and is faster here, while OFTA uses 63.73% less peak memory at 8,192 tokens.

Original-backend capacity. Requests completed under the same 24 GiB PyTorch allocator budget (three processes).

Output tokensOFTATF-JVP
2,0483/33/3
4,0963/33/3
8,1923/30/3
10,2403/30/3

At 10,240 tokens, OFTA request completed in 657.96 s, with peak allocated memory of 11.3345 GiB.

Scaling. With bytes/element ee, batch BB, layers LL, KV heads hh, head width dhd_h and cache length ScS_c,

MKV=2eBLhdhSc,MKV,OFTA=eBhdhSc(2L+aK).M_{\rm KV}=2eBLhd_hS_c,\quad M_{\rm KV,OFTA}=eBhd_hS_c(2L+aK).

aa counts necessary tangent K/V slots per direction.

Q2 (continued). Cache and parameter scaling.

Full layout a=2La=2L gives (1+K)MKV(1+K)M_{\rm KV}; structural zeros can be omitted. Native 10k primal/tangent KV each occupy 2.23 GiB, excluding weights and live workspace.

Learning results support K=1K=1 as a quality–cost choice. In the paper's batch16 tests (1/4 output tokens), total OFTA memory including tangent KV is close to MeZO (7.0–9.5% higher) and 41.9–43.1% below FO/BP. Long-response OFTA memory is below both cached JVP and TF-JVP. After generation, updates need saved statistics/traces, without KV caches or another model evaluation.

For model/trainable counts N,pN,p and prompt-plus-output length S=Pprompt+nS=P_{\rm prompt}+n, weights/Adam/materialized directions take O(N)/O(p)/O(Kp)O(N)/O(p)/O(Kp) storage. Dense-attention work is C=O(B[SWcore+LdS2])+CvocabC=O(B[SW_{\rm core}+LdS^2])+C_{\rm vocab}. Here WcoreW_{\rm core} is per-token projection/MLP work, dd query width and CvocabC_{\rm vocab} vocabulary work. Fixed-KK tangents preserve these orders but add affected computation. JVP is not merely O(p)O(p); caches grow linearly, attention has quadratic work, and runtime is backend-dependent.

Multi-direction memory. H100/FP32, L11-vB p=8192p=8192, B8, prompt cap320, 16 output tokens; peak allocated includes the model, excludes optimizer execution.

KPeak allocated (GiB)
17.8541
28.2740
49.1139
811.0205

Native HF uses 7.3892 GiB. Directions share weights/primal KV, retaining necessary tangent history. All 20 checks match scores, directional derivatives and Adam updates exactly. Long-request times retain the original backend; new OFTA memory uses 9/8 retained measurements at 2,048/8,192 tokens.

Q3. Full-set performance as a function of adaptation time.

Your understanding of Figure 5 is correct. The 100 evaluation questions were selected because the base model fails under greedy decoding. This is intentional: our goal is to demonstrate that OFTA can learn from post-generation feedback under an RL-style reward objective, rather than to maximize full-benchmark accuracy. Requiring 4–6 correct responses among eight samples provides informative positive and negative rewards, allowing us to test whether OFTA can reinforce correct solutions that the model already generates but fails to select greedily. Adaptation and evaluation questions are disjoint. OFTA improves greedy accuracy on this initially failed cohort from 0% to 60% after 50 updates.

Regarding full-set evaluation, Qwen3-1.7B already achieves 80.97% greedy accuracy on the complete GSM8K test set, leaving less room for improvement than the targeted correction setting. Full-set training would address a different question. Nevertheless, following your suggestion, we additionally evaluate all saved checkpoints on all 1,319 test questions, using both greedy and temperature-1 sampling. OFTA's full-set accuracy changes from 80.97%/80.59% to 81.43%/80.74% (greedy/T1) after 50 updates, with modest, non-monotonic improvements. The complete time–performance curves are reported, with OFTA requiring 31.51 minutes versus 64.33 and 75.71 minutes for post-JVP and MeZO, respectively. These results complement the original targeted correction experiment without changing its training protocol.

UpdatesOFTA timeOFTA G / T1 (%)Post-JVP timePost-JVP G / T1 (%)MeZO timeMeZO G / T1 (%)
00.0080.97 / 80.590.0080.97 / 80.590.0080.97 / 80.59
105.9381.27 / 80.1412.0481.27 / 80.1412.8481.58 / 81.58
2012.7580.67 / 80.5926.0580.67 / 80.6728.7780.74 / 80.82
3018.9581.88 / 81.5038.9281.88 / 81.5843.9081.27 / 81.80
4025.2881.73 / 81.5851.9981.58 / 81.2059.9180.29 / 80.74
5031.5181.43 / 80.7464.3381.58 / 81.2775.7181.27 / 80.89

Q4. Recent ZO methods and the choice of baselines.

We chose baselines to compare how learning information is acquired. MeZO is a standard two-point finite-difference method, using two perturbed loss forwards per direction. Several finite-difference optimizers in Zhang et al.'s benchmark [1] and PaZO's gradient estimator [2] share this loss-query mechanism, while adding momentum, adaptive updates or preconditioning. MeZO's simple form makes the cost of query-based acquisition easy to isolate.

BP supplies a full-gradient quality/resource reference. Matched JVP keeps the estimator and update fixed, directly testing OFTA's acquisition of sensitivities during the original generation. Other ZO optimizers are meaningful additional comparisons; they refine directions and updates, while OFTA changes when sensitivity information is obtained. We will clarify this rationale and the connection to [1,2] in the revision.

Many thanks for the response and new experiments. Overall, I think OFTA is a neat direction to explore for using less memory than backprop , and also not requiring a second pass after feedback.

My questions have been answered and I maintain my positive score. I would encourage the authors to include the new comparison tables and results (especially long-response ones) in the camera-ready.

We are very pleased that our response and additional experiments have answered your questions. We especially appreciate your recognition of OFTA’s memory benefits and its ability to update without another model pass after feedback. We will include the new comparison tables and experimental results in the camera-ready version. And we will give particular attention to the experiments with long responses. Their settings and main findings will be clearly presented. We sincerely appreciate your constructive feedback and positive assessment of our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Discussion

Dear AC and Reviewers,

Thank you very much for your time, thoughtful feedback, and valuable contributions to improving our work. We sincerely appreciate your careful assessment and constructive engagement throughout the discussion. We are encouraged by the recognition of OFTA’s practical motivation, conceptual contribution, theoretical analysis, and efficiency benefits. Your questions prompted substantial additional experiments and analyses that strengthened the empirical evidence and clarified OFTA’s practical trade-offs. To assist the AC in the final assessment, we provide a brief summary of the recognized strengths and reviewer discussions below.

Reviewer wPHW recognized that OFTA addresses an important bottleneck in LLM deployment. They described the motivation as timely and highly impactful and acknowledged the clear organization and rigorous theoretical analysis supporting the single-pass strategy.

Reviewer 6VvM appreciated the elegant conceptual contribution of computing the model-dependent part before feedback arrives. They also highlighted directional derivatives without finite-difference approximation and lower runtime than MeZO with only modest additional peak memory in the reported experiments.

Reviewer JqK2 recognized the well-motivated separation of feedback-independent sensitivity computation from feedback-dependent loss evaluation. They also highlighted controlled numerical validation, substantially reduced post-feedback update latency, and adaptation performance comparable to post-feedback JVP.

These strengths reflect OFTA’s central contribution. OFTA acquires model sensitivities during the original response generation and combines the saved traces with subsequent feedback. Under matched conditions, it preserves the directional update while avoiding another model pass after feedback. It also avoids retaining a backward graph while waiting. This mechanism supports different learning objectives and both immediate and delayed feedback.

In response to the reviewers’ questions, we conducted additional experiments and analyses in three main areas.

  • Broader learning evidence and asynchronous feedback. We expanded the evaluation to open-ended summarization, graded-reward learning, an additional model family, and delayed feedback with nonzero parameter drift. We also clarified the purpose and scope of the original GSM8K correction study. These additions strengthen the evidence across different tasks, objectives, and feedback schedules.

  • Efficiency, scaling, and learning trade-offs. We expanded matched-backend comparisons across adaptation-space sizes, response lengths, and request batches. We added long-response measurements, direction-count ablations, and comparisons with full-gradient BP. Together, these results isolate the benefits of generation-time acquisition and clarify the trade-offs among learning quality, computation, and memory.

  • Theoretical clarification and presentation. We clarified stochastic-sampling semantics, distinguished directional-derivative accuracy from estimator variance, and supplemented the finite-precision evidence. We also redesigned Figure 1 and updated its caption to make OFTA’s workflow easier to follow.

Following the discussion, Reviewer 6VvM confirmed that their questions were answered and maintained a positive assessment. Reviewer JqK2 stated that the additional results addressed most concerns and substantially strengthened the empirical evaluation, while maintaining Weak Acceptance.

We will carefully incorporate the additional results and explanations into the camera-ready version, particularly the long-response comparisons. We sincerely thank all reviewers and the AC for their time, careful assessment, and constructive engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.