- Rating
- 2
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes One-Pass Forward-Trace Adaptation (OFTA), a test-time adaptation method for LLMs that eliminates the second pass. OFTA measures how the log-probability of the response changes along a random parameter direction when LLM generates tokens. Experiments on Qwen3 models demonstrate reduced latency and memory usage while maintaining learning quality.
Critical strengths
-
This paper addresses a bottleneck in LLM deployment by eliminating the prohibitive computational overhead of re-running models for test-time adaptation. It makes the core motivation both timely and highly impactful.
-
The manuscript is well-organized with clear logical flow, supported by rigorous and thorough theoretical analysis that grounds the proposed single-pass strategy.
Critical weaknesses
-
Figure 1 is overly cluttered with excessive text and small fonts, making it difficult to parse. The figure also exhibits noticeable AI-generated styling, so the authors should simplify the layout and better highlight the core contributions.
-
The experimental validation is insufficient. The online TTA evaluation is limited to simulated binary verifier feedback on the GSM8K dataset and Qwen3. There is a lack of assessment on more complex open-ended tasks or continuous reward signals, which makes it difficult to comprehensively reflect the effectiveness.
-
The analysis of K greater than 1 is only briefly mentioned in Appendix D.5. This trade-off should be central to the proposed method. In models with millions of parameters, setting K to 1 results in high gradient noise. However, using multiple directions linearly increases generation latency, which contradicts the single-pass low-overhead selling point of the paper.
-
The theoretical proofs assume infinite-precision arithmetic and fixed sampled tokens. In practice, real LLMs suffer from accumulated errors under low-precision quantization, and stochastic sampling itself is non-differentiable. Furthermore, the theory only guarantees unbiasedness while single-direction estimation causes variance explosion across millions of parameters, making actual convergence unachievable.
Questions for authors
See the weaknesses above.
5 comments
W3. Direction count, noise and generation cost.
Our experiments identify as the best practical quality–cost trade-off for our low-overhead online setting. Larger adds computation and tangent storage; the updated memory measurements below quantify the storage cost of our optimized implementation. This trade-off explains why we prioritized experiments that verify update consistency, computational savings and effective learning in the main text, while placing the discussion in the appendix. We now provide the detailed comparison across below and will include it in the main text.
Single-pass means one autoregressive generation. All K directions share its tokens, primal activations and primal KV cache. Each adds derivative computation and tangent storage that can grow roughly linearly with K. K controls the derivative work within this generation, without generating the full answer K times.
| K | T1 F1 | Generation+sensitivity (s) |
|---|---|---|
| 1 | 77.7694 | 1.2251 |
| 2 | 78.5531 | 1.5263 |
| 4 | 79.5053 | 2.2230 |
| 8 | 80.4513 | 3.8600 |
Quality: SQuAD, p=8192, B8, seed29, 4096 updates; T1 averages three decoding repeats. Times are original-backend medians over 18 requests; feedback-time updates take 0.838–1.174 ms. These are separate quality and timing experiments.
| K | Peak allocated (GiB) |
|---|---|
| 1 | 7.8541 |
| 2 | 8.2740 |
| 4 | 9.1139 |
| 8 | 11.0205 |
Peak allocated: H100, FP32/TF32 off, B8, p8192 (L11 v-projection LoRA-B), prompt cap320, 16 output tokens; includes model residency, direction preparation, generation and scoring, excludes optimizer execution. Same-config native HF inference uses 7.3892 GiB. Across 20 checks against the reference implementation, scores, directional derivatives and Adam update coordinates have zero maximum difference; generation/EOS behavior agrees.
Your concern about high-dimensional noise is important. Averaging independent directions reduces estimator variance by , while the dimensional penalty remains. The practical question is whether a noisy K1 update can still improve the task. We test this on SAMSum with 8,716,288 trainable parameters across all 196 linear layers (rank8 LoRA A+B):
| SAMSum | ROUGE-1 | ROUGE-2 | ROUGE-L |
|---|---|---|---|
| Base | 31.62 | 11.81 | 24.10 |
| OFTA K1 | 41.26 | 17.34 | 33.44 |
Reference-summary cross-entropy, 4096 updates, seed17, Adam, learning rate 3e-4. These gains directly show useful K1 learning with millions of trainable parameters.
W4. Stochastic sampling, numerical accuracy and learning.
Ideal arithmetic is the reference level for establishing algorithmic correctness, not a requirement for infinite-precision hardware. Such real-valued analysis is common in ZO optimization. MeZO derives unbiasedness as perturbations vanish; MeZO-SVRG and LOZO analyze smooth objectives with bounded stochastic-gradient variance; DiZO also assumes smoothness. MeZO-SVRG separately tests FP32/BF16. OFTA connects its identity to finite-precision implementation through error bounds and numerical checks. Fixed sampled tokens define the differentiated score, without assuming differentiable sampling or deterministic generation.
Stochastic sampling. We differentiate the realized response's score with fixed. This is the score-function treatment of discrete samples in stochastic computation graphs, not a derivative through the token draw. For reward-driven learning,
This uses parameter-independent rewards, the actual sampling/stopping law, locally fixed support and valid interchange of differentiation and expectation. Holding the realized tokens fixed is the correct score derivative; it does not require a deterministic generator or differentiation through the discrete draw. Nor must changed parameters regenerate the same response.
Finite precision. Appendix F.10 bounds how errors in saved statistics, sensitivities, outer loss gradients and their final contraction propagate into the update. This quantifies departures from the identity when stored values are inexact. We compare scalar directional derivatives with from independent BP, with parameters, realized responses and computation graphs matched (FP32, TF32 off). These are two ways to compute the same scalar in exact arithmetic.
| Directional-derivative check | Maximum error |
|---|---|
| Matched JVP/BP projection; 18 scalars | 0.000299% relative |
| Per response; 720 scalars | 0.009990% relative |
| Aggregate; 90 scalars | 0.202639% relative |
| Trained K1/K4; per response | 2.095e-5/3.164e-5 absolute |
| Trained K1/K4; aggregate | 2.340e-7/3.786e-7 absolute |
The K1/K4 labels denote checkpoints after 1024 SQuAD reward-training steps (seed19, p=8192), checked on four train prompts. Each checkpoint is tested with one and four directions. These are numerical errors in directional derivatives, not errors between the random vector and the full gradient. An accurate can coexist with high estimator variance. The checked forward-score gap is zero, so sensitivity acquisition preserves these scores.
We also compare generation-time acquisition with matched post-feedback execution over 1024 updates (K1, seed19). Saved replay scores agree, and the saved parameter checkpoints match byte for byte. This also checks the effect on a sequence of updates.
NF4-quantized weights and accumulated update error. To address your quantization concern directly, we retain actual NF4-quantized weight values and execute the model in FP32. In this separate numerical control, each row compares OFTA with an independent BP directional projection on the same model and realized response; both sides use the same update rule.
| Setting | Max. relative directional-derivative error | Trainable-parameter relative L2 deviation after 32 updates |
|---|---|---|
| FP32 control | 0.000628% | 0.000646% |
| NF4 weight values, FP32 execution | 0.022962% | 0.001398% |
All 16 NF4 directional-derivative signs agree with BP, and all recorded traces, caches, parameters and Adam states remain finite. The parameter comparison tests error accumulation across repeated Adam updates. Its small final discrepancy supports accurate OFTA sensitivities and closely matched updates with NF4 weight values under FP32 execution.
Optimization beyond unbiasedness. For a fixed, lower-bounded -smooth objective, exact directional derivatives and fresh independent standard Gaussian directions at each step, synchronous SGD with averaged directions satisfies
\le F(\phi)-\eta(1-L\eta c_K/2)\|\nabla F(\phi)\|^2, \qquad c_K=1+(p+1)/K.$$ Here $p$ is the trainable dimension. The bound gives expected progress away from stationary points when $0<\eta<2/(Lc_K)$. High variance imposes a stricter step-size condition, rather than ruling out descent. This analysis, together with the graded-feedback gains and million-parameter results above, provides theoretical and empirical evidence for useful learning within the reported settings.
Thank you for the review. We address figure clarity and experiments here, direction count in Comment 2, and theory and numerical evidence in Comment 3.
W1. Figure clarity and presentation.
We have redrawn Figure 1 with less text, a clearer visual hierarchy and consistent visual elements. The top panels directly contrast a separate post-feedback model pass with OFTA's generation-time trace acquisition. The lower panels show the workflow: generate a response and acquire sensitivities in one pass, save the trace, then combine it with feedback to update the parameters. This makes the core contribution explicit: the model-dependent derivative work takes place during generation, so no additional model pass is needed after feedback arrives.
W2. Experimental coverage, open-ended tasks and continuous feedback.
We would first clarify what the original experiments establish. OFTA changes when derivative computation is performed. Model sensitivities are acquired while producing an answer and used once feedback becomes available. The experiments therefore test whether this change preserves the directional update, reduces adaptation cost and supports effective learning. The paper already provides matched derivative/update checks, full-request cost measurements and task-quality results to validate these claims.
The tasks in the main paper are not limited to binary classification or GSM8K. Table 1 covers ten tasks on Qwen3-1.7B and Qwen3-8B, including generative question answering on SQuAD and DROP. We chose these tasks because they are established, widely accepted benchmarks in ZO language-model fine-tuning, used in work such as MeZO and DiZO. They provide a standard basis for assessing learning quality, as well as meaningful comparison with prior methods.
Qwen3's open weights enable BP checks of sensitivities through its KV cache. OFTA's identity uses differentiable statistics and state tangents, not Qwen-specific layers.
Model coverage beyond Qwen3. To directly address your concern about model coverage, we add supervised SQuAD adaptation on OPT-2.7B. All three methods use the same LoRA adaptation space and 8192 updates, with greedy decoding on the full SQuAD validation set.
OFTA's F1 is close to MeZO's on this second model family, providing direct empirical evidence that its effectiveness is not specific to Qwen3.
These task-quality experiments use known supervision, while the GSM8K study tests adaptation from feedback received after generation. Following your suggestion, we further add SAMSum summarization and SQuAD adaptation driven by graded token-F1 feedback.
Open-ended generation: SAMSum summarization. Summarization requires selecting the important information in a dialogue and expressing it as a free-form answer. OFTA uses reference-summary cross-entropy, with greedy decoding for ROUGE evaluation (K=1, 8192 updates, seed17).
SAMSum: OFTA results and published references
The improvements across all three metrics show that OFTA's updates are useful for open-ended generation, where the desired behavior involves content selection and concision. The following real example makes this change visible.
Real summarization case: the same dialogue before/after adaptation
Instruction: Summarize the following conversation
Dialogue:
Reference:
Before OFTA / Base:
ROUGE-1/2/L: 18.18 / 6.45 / 18.18
After OFTA / step 8192:
ROUGE-1/2/L: 80.00 / 46.15 / 80.00
The base output retells the question-and-answer exchange. After adaptation, the summary focuses on the outcome: Grace's chosen hairstyle for tomorrow. It preserves the important fact while removing the conversational details, illustrating the improvement in content selection.
Continuous feedback: SQuAD token-F1 rewards. In this experiment, the reward is returned only after a response has been generated. A partly correct answer receives partial credit, and that graded score directly drives the parameter update. Thus F1 serves as the learning signal as well as the evaluation measure.