acceptodds
Under review as a conference paper at ICLR 2027

Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMs

Abstract

Large Language Models (LLMs) employ multi-turn interaction as a fundamental paradigm for completing complex tasks. However, their performance often degrades in extended interactions, as they are typically trained on static, single-turn data, which hinders their ability to adapt to real-time user feedback. To address this limitation, we first propose a new paradigm: Test-Time Policy Adaptation for Multi-Turn Interactions (TPAM), which utilizes user feedback from the ongoing interaction as a reward signal to estimate a latent optimal policy aligned with user preferences, then updates a small subset of parameters to steer the model toward this policy, ultimately enabling efficient in-conversation self-correction. We then introduce Optimum-Referenced One-Step Adaptation (ROSA), a lightweight algorithm that operationalizes TPAM. ROSA guides the model parameters toward a theoretical optimal policy in a single, efficient update step, avoiding costly iterative gradient-based optimization and minimizing computational overhead. We provide a rigorous theoretical analysis guaranteeing that the policy of ROSA converges to the preference of user as the number of interactions increases. Extensive experiments on challenging benchmark demonstrate that ROSA achieves significant improvements in both effectiveness and efficiency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test time, policy optimization, multi-turn policy adaptation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gREbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ROSA, a method that updates an LLM during a conversation using feedback on its answers. It turns the feedback into a target response probability and uses a linear approximation to update the LM head or hidden states. The model is reset between problems. The paper provides a theoretical analysis and reports improvements across various tasks.

Critical strengths
  1. The method learns from negative feedback without needing a corrected answer, which is useful in conversations where users can identify a mistake but cannot provide the solution.

  2. The reported gains are substantial, making the empirical results worth further investigation.

Critical weaknesses
  1. Equations (4) to (6) reduce to a scaled negative gradient update with a direct minimum norm solution. It is unclear what CG adds, and the comparison with ordinary gradient updates does not isolate the benefit of ROSA’s step size.

  2. The convergence claims go beyond the analysis. Theorems 2 and 3 assume exact distribution updates, while Theorem 4 does not bound the approximation errors tightly enough to guarantee progress for the implemented algorithm.

Questions for authors

See weaknesses

9 comments
AuthorsOpenReview ↗

Response to Reviewer gREb (Part1/2)

We sincerely thank the reviewer for the constructive feedback and for recognizing the value of learning from negative feedback without requiring corrected answers, as well as the substantial empirical gains. We address your concerns about the update mechanism, step-size effects, and practical-update guarantees below.

1. Scalar Closed-Form Updates, CG, and Step-Size Effects (W1)

1. The scalar closed-form derivation holds, but addresses a different matching constraint.

For a linearized constraint that matches only the probability of the rejected response, the minimum-norm closed-form solution identified by the reviewer is valid and can be expressed as a scaled, squared-norm-normalized negative-gradient update. We will explicitly discuss this mathematical relationship in the revised manuscript.

Our objective is to approximate the feedback-derived target policy distribution. Equation (3) downweights the rejected response and adjusts the remaining response probabilities through normalization. A single sequence-probability constraint does not explicitly specify how the other output probabilities should change and therefore cannot directly replace the joint distribution-fitting problem used in our implementation.

2. The implementation jointly fits full-vocabulary distributions at multiple token positions.

ROSA concatenates full-vocabulary probability vectors at multiple token positions and jointly fits their feedback-derived targets. Consequently, the Jacobian has multiple rows, and its Gram matrix is generally not rank one.

Let Fk(θ)F_k(\theta) denote the concatenated probability vector, with the current context and response prefixes held fixed during the update, and let Fk⋆F_k^\star denote its target. Define dk=Fk⋆−Fk(θk−1)d_k=F_k^\star-F_k(\theta_{k-1}) and Jk=∂Fk(θ)/∂θ∣θ=θk−1J_k=\left.\partial F_k(\theta)/\partial\theta\right|_{\theta=\theta_{k-1}}. The implemented linearized fitting problem is:

min⁡Δθ∥JkΔθ−dk∥22.\min_{\Delta\theta}\|J_k\Delta\theta-d_k\|_2^2.

We use matrix-free CG to approximately solve the associated normal equations, computing Jk⊤(Jkv)J_k^\top(J_kv) through JVPs and VJPs without explicitly constructing the Jacobian or its Gram matrix. CG therefore handles the joint probability constraints of the implemented distribution-fitting problem.

3. A step-size sensitivity experiment directly examines the reviewer’s concern.

We agree that comparison with ordinary gradient updates alone does not sufficiently isolate step-size effects. We therefore evaluate the reviewer-derived sequence-level closed-form baseline and apply an additional step-size multiplier to both methods:

θk=θk−1+αΔθk.\theta_k=\theta_{k-1}+\alpha\Delta\theta_k.

For the closed-form baseline, Δθk\Delta\theta_k retains the full coefficient derived by the reviewer. For ROSA, it is the update returned by the CG solver. We keep β\beta and all other settings fixed, and evaluate α∈{0.25,0.5,1,2}\alpha\in\{0.25,0.5,1,2\} on MATH with random seed 42. The table reports final cumulative accuracy (%) within ten turns.

ModelMethodα = 0.25α = 0.5α = 1α = 2
Qwen3-0.6BSequence-level closed-form update29.8032.8031.6031.20
Qwen3-0.6BROSA (joint distribution fitting)48.6051.4050.2051.20
Qwen3-8BSequence-level closed-form update58.2057.8057.4056.80
Qwen3-8BROSA (joint distribution fitting)64.8065.2065.8064.60
Qwen3-32BSequence-level closed-form update79.8080.2079.6079.60
Qwen3-32BROSA (joint distribution fitting)85.2087.8087.8086.40

ROSA outperforms the closed-form update in all twelve model–multiplier combinations. Even when comparing each method’s highest accuracy within the tested range, ROSA retains advantages of 18.60, 7.60, and 7.60 percentage points, respectively. Thus, within the evaluated step-size range and seed, the advantage cannot be explained solely by the default step-size choice.

For the complementary effectiveness–cost comparison at the default multiplier, please see our response to Reviewer xdLw, Section 1, point 3 (W1, Q1). That table reports final accuracy, correction uplift, and average update time for standard multi-turn interaction, the sequence-level closed-form update, and ROSA. It quantifies the additional computational cost associated with the joint distribution-fitting update alongside its performance gains.

To avoid ambiguity, we will revise §3.3 to explicitly define the vector output, token-level target construction, and residual, and include the least-squares objective above. We will also discuss the scalar closed-form solution, incorporate the step-size comparison, and clarify the approximation relating practical token-level fitting to the sequence-level target in Equation (3).

AuthorsOpenReview ↗

Response to Reviewer gREb (Part2/2)

2. Exact-Update Analysis and Progress from Practical Updates (W2)

1. The analysis distinguishes ideal distribution updates from practical parameter updates.

Theorems 2 and 3 characterize exact distribution updates. Theorem 4 explicitly incorporates target-fitting and linearization errors when realizing the target through a parameter update.

Its single-step analysis can be summarized as:

ΔDKL≤−C πθk−1(yk∣xk)+ϵk+L2∥Δθk∥22,\Delta D_{\mathrm{KL}}\leq-C\,\pi_{\theta_{k-1}}(y_k\mid x_k)+\epsilon_k+\frac{L}{2}\|\Delta\theta_k\|_2^2,

where C=1−exp⁡(−1/β)C=1-\exp(-1/\beta), the first term captures the ideal negative-feedback improvement, and the remaining terms account for approximation errors. The bound guarantees progress when the ideal improvement exceeds these errors. We will make this condition explicit and align the convergence statements in the abstract and main text with the theorem’s assumptions.

For further discussion of the assumptions underlying this analysis, please see our response to Reviewer qGs4, Section 3, points 1–2 (Q2), which explains the reliable-feedback protocol and the local-smoothness justification for the adapted components, together with the relevant literature. Our response to Reviewer qGs4, Section 1, point 2 (W1) also makes explicit that the single-step comparison evaluates the pre-update and post-update policies under the same current conversational context.

2. Controlled one-step recovery directly evaluates the effect of an additional practical update.

We conducted a controlled experiment using 100 shared failure states per model, selected from mathematical and coding evaluation tasks.

Failure-state construction. We collect states from ROSA’s interaction trajectories whenever a response is judged incorrect by the corresponding evaluator. A state remains eligible even if the problem is successfully solved in a later turn. We randomly select 100 distinct problems with eligible failures and retain one failure state per problem. Selection does not depend on the subsequent correction outcome.

Controlled branching. For each selected state, we preserve the complete conversation history, the incorrect response, the negative feedback, and the model parameters and adaptation state immediately before processing that feedback. Three independent branches start from this identical state:

  • No Additional Update: retain the current model state without applying another parameter update.
  • Sequence-Level Closed-Form Update: apply one update using the reviewer-derived scalar closed-form solution.
  • ROSA: apply one additional joint distribution-fitting update.

All branches receive the same next-turn prompt and use the same decoding configuration and evaluator. Each trial permits exactly one subsequent response. We repeat the evaluation with five decoding seeds, restarting from the same saved failure state for each trial. The table reports next-response recovery rates (%) as mean ± sample standard deviation. Each response is evaluated separately; success is not aggregated by selecting the best response across seeds.

ModelShared Failed Problems (N)No Additional UpdateSequence-Level Closed-Form UpdateROSA
Qwen3-0.6B10029.40 ± 7.9737.20 ± 5.7641.20 ± 3.13
Qwen3-8B10038.20 ± 7.2145.00 ± 5.3152.60 ± 2.97
Qwen3-32B10041.00 ± 7.0349.40 ± 5.1063.80 ± 2.16

ROSA improves next-response recovery over no additional update by 11.80, 14.40, and 22.80 percentage points, respectively. It also exceeds the closed-form update by 4.00, 7.60, and 14.40 percentage points, with lower cross-seed standard deviations.

Because the branches share the same failure states, feedback, and next-turn prompts, this experiment directly shows that an additional ROSA update improves the probability of correcting the error in the next response at failure states encountered during ROSA interactions.

AuthorsOpenReview ↗

Response to Reviewer gREb (Part2/2)

3. Existing experiments provide complementary evidence of stability and sustained effectiveness.

The general-knowledge retention results in Appendix E.6 evaluate Qwen3-0.6B after ten consecutive ROSA updates on unresolved mathematical problems. The average KL divergence from the base model is 0.001, while MMLU accuracy and WikiText perplexity remain essentially unchanged, supporting the stability of repeated practical updates.

The five-seed evaluations in our response to Reviewer xdLw, Section 2 (Q2, Q3) provide additional evidence across complete interaction trajectories. On both MATH-500 and AIME25, ROSA achieves higher final mean accuracy and lower final cross-seed standard deviation than standard multi-turn interaction for all three model sizes. These results complement the controlled one-step experiment by showing consistent gains over repeated full-interaction evaluations.

The extended-horizon evaluation in Appendix E.4 demonstrates continued recovery on unresolved problems over twenty turns. The personalized-preference evaluation in Appendix E.12 further supports effectiveness in conversations with evolving contexts.

The ALOE protocol and its quantitative results are detailed in our response to Reviewer qGs4, Section 1, point 4 (W1). In that existing manuscript experiment, user preferences are progressively revealed during interaction, and ROSA improves both average and final alignment scores, providing complementary evidence beyond verifiable mathematical and coding tasks.

Theorem 4 characterizes the relationship between ideal improvement and approximation errors, while the new shared-state experiment directly measures the corrective effect of an additional implemented update. Together with the existing multi-turn and knowledge-retention results, these findings support the practical effectiveness and stability of ROSA. We will incorporate the new experiment and clarify the correspondence between the theoretical analysis and the implemented solver in the revised manuscript.

Reviewer_xdLwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies test-time policy adaptation for improving the self-correction ability of large language models in multi-turn interactions. The authors propose the Test-Time Policy Adaptation for Multi-Turn Interactions (T²PAM) paradigm and introduce Optimum-Referenced One-Step Adaptation (ROSA), which uses feedback from previously generated responses to update model parameters during inference. Specifically, ROSA constructs a target policy by exponentially reweighting a rejected response under a KL-regularized reward objective, then computes a parameter update through linearized optimization using conjugate gradient (CG). The paper also provides theoretical results on monotonic error reduction and cumulative convergence, together with experiments on mathematical reasoning, general reasoning, code generation, multilingual tasks, and open-ended interaction settings.

Critical strengths

Improving the ability of LLMs to learn from feedback within an ongoing interaction is an important research direction. The proposed method is conceptually simple and does not require additional offline training. The experiments cover multiple models and domains, with substantial improvements in several settings. The paper includes comparisons with test-time compute methods, multi-turn training baselines, and alternative reward mechanisms. Additional experiments investigate extended interaction horizons, larger models, delayed rewards, subjective preferences, and computational efficiency.

Critical weaknesses

In Equation (5), the authors define a single scalar constraint corresponding to the probability of the rejected response. Consequently, the Jacobian JkJ_k is a row vector, and JkTJkJ_k^T J_k in Equation (6) is a rank-one, positive semidefinite matrix. The corresponding linear system is therefore underdetermined, but its minimum-norm solution has a simple closed form: Δθk=dkJkT/∣∣Jk∣∣2\Delta θ_k = d_k J_k^T / ||J_k||². Let p=πθ(yk∣x)p = π_θ(y_k|x) and C=1−exp(−1/β)C = 1 - exp(-1/β). Substituting the practical target in Equation (3) yields:

= -\frac{C(1-p_\theta)}{1-Cp_\theta} \frac{\nabla_\theta\log p_\theta} {\|\nabla_\theta\log p_\theta\|_2^2}$$ This expression shows that, at the level of the stated linearized optimization, ROSA is equivalent to a normalized gradient step that decreases the log-probability of the rejected response. This equivalence is missing in the paper. Based on this observation, this paper should compare ROSA with a squared-norm normalized negative-feedback gradient update. I think this is a critial relation that needs to be discussed in the paper.
Questions for authors
  1. See Major weakness, why the conjugate gradient is necessary to solve the problem if the explicit solution is available?
  2. The paper reports substantial improvements, but uncertainty estimates and variation across random seeds would strengthen the evidence, especially on smaller benchmarks such as AIME25.
  3. Appendix C.3 defines final accuracy AA as the fraction of problems solved within ten turns and correction uplift UU as the fraction of initially failed problems subsequently solved. These quantities imply that first-turn accuracy a1a_1 must satisfy $ A=a_1+(1-a_1)U, $ so $ a_1=\frac{A-U}{1-U}. $ Combining Table 2 with Table 3, first-turn accuracies on MATH-500 for Qwen3-0.6B is 10.00% and 0.60% for baseline and ROSA. According to Algorithm 1, ROSA does not modify model parameters until after the first response has been generated and rejected. Furthermore, Appendix C.6 states that both methods start from the same model and initial prompts. Thus, the two methods should have almost the same first-turn distribution. The observed discrepancy is large enough to require an explanation.
7 comments
AuthorsOpenReview ↗

Response to Reviewer xdLw (Part1/2)

We sincerely thank the reviewer for the thoughtful and constructive feedback, and for recognizing the importance of learning from feedback during interaction, the simplicity of our method without additional offline training, and the breadth of our empirical evaluation. We appreciate your careful analysis of the update formulation and experimental results, and address your concerns below.

1. Scalar Closed-Form Updates versus Joint Distribution Fitting (W1, Q1)

We thank the reviewer for the detailed derivation and the suggestion to include this targeted baseline. We address the mathematical relationship, the implemented update, and the experimental comparison below.

1. The closed-form derivation holds for the scalar constraint.

For the linearized constraint that matches only the probability of the rejected response, the reviewer’s minimum-norm solution is valid. Substituting Equation (3) indeed yields a negative-feedback gradient update normalized by the squared gradient norm. We will explicitly discuss this relationship in the revised manuscript.

Our objective, however, is to approximate the feedback-derived target policy distribution. Equation (3) downweights the rejected response and adjusts the remaining response probabilities through normalization. A single sequence-probability constraint does not explicitly specify how the other output probabilities should change and therefore cannot directly replace the joint distribution-fitting problem used in our implementation.

2. ROSA jointly fits full-vocabulary distributions at multiple token positions.

The implementation concatenates full-vocabulary probability vectors at multiple token positions and jointly fits their feedback-derived targets. Consequently, the Jacobian has multiple rows, and its Gram matrix is generally not rank one.

Let Fk(θ)F_k(\theta) denote this concatenated probability vector, with the current context and response prefixes held fixed during the update, and let Fk⋆F_k^\star denote its target. Defining dk=Fk⋆−Fk(θk−1)d_k=F_k^\star-F_k(\theta_{k-1}) and Jk=∂Fk(θ)/∂θ∣θ=θk−1J_k=\left.\partial F_k(\theta)/\partial\theta\right|_{\theta=\theta_{k-1}}, the implemented linearized fitting problem is:

min⁡Δθ∥JkΔθ−dk∥22.\min_{\Delta\theta}\|J_k\Delta\theta-d_k\|_2^2.

We use matrix-free CG to approximately solve the associated normal equations, computing Jk⊤(Jkv)J_k^\top(J_kv) through JVPs and VJPs without explicitly constructing the Jacobian or its Gram matrix. CG thus handles the joint probability constraints, rather than a single scalar constraint.

AuthorsOpenReview ↗

Response to Reviewer xdLw (Part1/2)

3. Direct comparison with the reviewer-derived closed-form update.

Following the reviewer’s suggestion, we compare the sequence-level closed-form update with ROSA on MATH using Qwen3-0.6B, Qwen3-8B, and Qwen3-32B. All other settings follow the main experiments; only the update target and its corresponding solution differ. The table reports results with random seed 42. Each method is evaluated independently, without sharing first-turn sampled responses. Standard multi-turn and ROSA results are reused from the manuscript.

ModelBenchmarkMethodFinal Accuracy (%) ↑Correction Uplift (%) ↑Avg. Update Time (s) ↓
Qwen3-0.6BMATHStandard multi-turn (no update)25.0017.40—
Qwen3-0.6BMATHSequence-level closed-form update31.6020.406.17
Qwen3-0.6BMATHROSA (joint distribution fitting)50.2048.8713.16
Qwen3-8BMATHStandard multi-turn (no update)55.8023.00—
Qwen3-8BMATHSequence-level closed-form update57.4027.1413.68
Qwen3-8BMATHROSA (joint distribution fitting)65.8040.4228.39
Qwen3-32BMATHStandard multi-turn (no update)76.4035.73—
Qwen3-32BMATHSequence-level closed-form update79.6037.6320.19
Qwen3-32BMATHROSA (joint distribution fitting)87.8045.7931.76

The sequence-level closed-form update improves over standard multi-turn interaction on all three models, demonstrating the practical value of this simple baseline. ROSA further improves final accuracy by 18.60, 8.40, and 8.20 percentage points, respectively, and achieves higher correction uplift in every setting. These results support the benefit of jointly fitting multiple local distributions under the evaluated configuration.

The closed-form update is computationally cheaper: ROSA’s average update time is 2.13×, 2.08×, and 1.57× higher, respectively. The comparison therefore quantifies both the effectiveness gains and the additional update cost of joint distribution fitting. Five-seed aggregate results are provided in our response to Q2, while first-turn sampling variability is addressed separately in Q3.

To further examine whether this advantage depends on the default step size, our response to Reviewer gREb, Section 1, point 3 (W1) reports a sensitivity analysis with additional update multipliers of 0.25, 0.5, 1, and 2 for both methods. ROSA outperforms the closed-form baseline in every tested model–multiplier combination. Comparing the best result of each method within this range, ROSA retains advantages of 18.60, 7.60, and 7.60 percentage points for Qwen3-0.6B, Qwen3-8B, and Qwen3-32B, respectively.

4. Clarifications in the revised manuscript.

To avoid ambiguity, we will revise §3.3, specifically the formulation preceding the CG solver, to explicitly define the vector output, token-level target construction, and residual, and include the least-squares objective above. We will also discuss the scalar closed-form solution and its normalized-gradient interpretation, incorporate this baseline comparison, and clarify the approximation relating practical token-level fitting to the sequence-level target in Equation (3).

AuthorsOpenReview ↗

Response to Reviewer xdLw (Part2/2)

2. Multi-Seed Evaluation and First-Turn Accuracy (Q2, Q3)

We thank the reviewer for suggesting uncertainty estimates and highlighting the first-turn accuracy discrepancy. We repeated the evaluation on MATH-500 and AIME25 using Qwen3-0.6B, Qwen3-8B, and Qwen3-32B with five random seeds: 1, 33, 42, 907, and 11532. All other settings follow the main experiments. The tables below report cumulative accuracy through each indicated turn as mean ± sample standard deviation, in percentage points.

1. Multi-seed results support the consistency of the performance gains.

MATH-500

ModelMethodTurn 1Turn 2Turn 4Turn 6Turn 8Turn 10
Qwen3-0.6BBaseline9.24 ± 3.5713.00 ± 4.0416.00 ± 3.8520.20 ± 3.6624.80 ± 3.5826.80 ± 3.56
Qwen3-0.6BROSA11.80 ± 3.8114.00 ± 3.3617.00 ± 2.8933.36 ± 1.4345.28 ± 1.6150.60 ± 1.54
Qwen3-8BBaseline46.64 ± 4.5050.20 ± 4.4352.60 ± 4.6554.00 ± 4.3358.60 ± 4.7260.40 ± 4.49
Qwen3-8BROSA46.16 ± 4.2953.20 ± 4.1860.80 ± 3.5765.84 ± 1.3371.28 ± 1.6171.68 ± 1.43
Qwen3-32BBaseline65.44 ± 5.0369.80 ± 5.2472.20 ± 5.0974.20 ± 5.2378.40 ± 5.2480.20 ± 5.20
Qwen3-32BROSA63.40 ± 4.7873.20 ± 4.3081.00 ± 3.7186.32 ± 1.5291.00 ± 1.4691.96 ± 1.47

AIME25

ModelMethodTurn 1Turn 2Turn 4Turn 6Turn 8Turn 10
Qwen3-0.6BBaseline7.33 ± 3.6511.33 ± 3.8012.00 ± 4.4712.67 ± 3.6512.67 ± 3.6512.67 ± 3.65
Qwen3-0.6BROSA5.33 ± 3.807.33 ± 3.6512.67 ± 2.7918.67 ± 1.8219.33 ± 1.5319.33 ± 1.53
Qwen3-8BBaseline8.67 ± 4.4711.33 ± 4.4719.33 ± 4.3519.33 ± 4.3519.33 ± 4.3519.33 ± 4.35
Qwen3-8BROSA4.00 ± 4.3512.00 ± 4.4720.67 ± 3.6528.00 ± 1.8232.00 ± 1.8232.66 ± 1.47
Qwen3-32BBaseline16.00 ± 4.9417.33 ± 4.3518.67 ± 4.4719.33 ± 4.3524.67 ± 5.0526.00 ± 4.94
Qwen3-32BROSA18.00 ± 5.0518.67 ± 4.4721.33 ± 2.9829.33 ± 1.4936.00 ± 1.4944.00 ± 1.49

ROSA achieves higher final mean accuracy in all six model–benchmark comparisons. On MATH-500, the improvements are 23.80, 11.28, and 11.76 percentage points, respectively. On AIME25, the smaller benchmark specifically highlighted by the reviewer, the improvements are approximately 6.66, 13.33, and 18.00 percentage points. These repeated evaluations support the observed performance gains and quantify variability that a single run cannot capture.

2. Independent sampling and first-turn variability.

We agree with the metric identity noted by the reviewer:

A=a1+(1−a1)U.A=a_1+(1-a_1)U.

ROSA does not update parameters before generating the first response. Both methods start from the same model and initial prompts, but are evaluated through independent stochastic sampling without sharing first-turn responses. Consequently, identical initial policy distributions do not require identical empirical first-turn accuracies in finite evaluations. Our five-seed results show appreciable first-turn variability before ROSA adaptation, illustrating the sensitivity of individual runs to sampling randomness.

To reduce dependence on individual sampling outcomes, we report means and standard deviations across five seeds. For Qwen3-0.6B on MATH-500, first-turn accuracy is 9.24 ± 3.57% for the baseline and 11.80 ± 3.81% for ROSA. We will explicitly document the independent-sampling protocol and include these repeated-evaluation statistics in the revised manuscript.

Reviewer_qGs4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ROSA, which updates a small part of a language model during a conversation using feedback on its previous response. It constructs a target policy from that feedback and approximates it with a single-parameter update. Experiments across reasoning and code-generation tasks show improved correction of failed answers.

Critical strengths

The method achieves substantial improvements across several models and tasks, and the evaluation contains a good number of tests.

Critical weaknesses

The theoretical guarantees rely on assumptions that are not established for the implemented method. In particular, improving the probability of one rejected response does not guarantee progress toward the user's preferred policy, especially as the conversation context changes.

The actual update needs clearer explanation. As written, the optimization involves a single response probability and admits a simple analytic solution, so the need for conjugate gradient is unclear.

Questions for authors

Does the implementation update a complete response probability or individual token probabilities? If it uses the scalar formulation in the paper, why does it need a conjugate gradient?

What evidence shows that the practical updates satisfy the conditions required by the convergence analysis?

5 comments
AuthorsOpenReview ↗

Response to Reviewer qGs4 (Part1/3)

We sincerely thank the reviewer for the constructive feedback and for recognizing the substantial improvements across multiple models and tasks, as well as the breadth of our evaluation. We address your concerns below and will clarify the theoretical guarantees and practical implementation in the revised manuscript.

1. Distribution-Level Improvement and Evolving Contexts (W1)

1. The guarantee concerns a complete distribution update, not an isolated probability change.

Our analysis does not infer alignment merely from decreasing the probability of one rejected response. Equation (3) defines a complete target distribution: it downweights the rejected response and redistributes probability mass proportionally among the remaining responses through normalization.

This mechanism is consistent with Zhu et al. [1], who show that negative sample reinforcement can suppress incorrect generations and redistribute probability mass toward other plausible candidates, guided by the model’s prior. For our specific ideal update, Theorem 2 establishes a decrease in KL divergence to the user’s target policy under the stated reliable-negative-feedback assumptions.

2. The single-step analysis conditions on the current conversational context.

The context need not remain identical across interaction turns. Let xkx_k denote the conversation history available at turn kk. Theorem 2 compares the policies before and after the update under the same current context xkx_k:

DKL(πuser∗(⋅∣xk)∥πθk(⋅∣xk))−DKL(πuser∗(⋅∣xk)∥πθk−1(⋅∣xk))≤−C πθk−1(yk∣xk),D_{\mathrm{KL}}\left(\pi_{\mathrm{user}}^{*}(\cdot\mid x_k)\Vert\pi_{\theta_k}(\cdot\mid x_k)\right)-D_{\mathrm{KL}}\left(\pi_{\mathrm{user}}^{*}(\cdot\mid x_k)\Vert\pi_{\theta_{k-1}}(\cdot\mid x_k)\right)\leq -C\,\pi_{\theta_{k-1}}(y_k\mid x_k),

where C=1−exp⁡(−1/β)>0C=1-\exp(-1/\beta)>0, under the theorem’s exact-update and reliable-rejection assumptions.

Thus, evolving conversation histories do not invalidate this conditional, single-step result: at each turn, the ideal update reduces the KL error evaluated at that turn’s context. We will make this conditioning explicit to distinguish within-turn policy improvement from changes induced by the evolving context itself.

3. The practical-update analysis explicitly accounts for approximation error.

The implementation jointly fits full-vocabulary probability vectors at multiple token positions to approximate the feedback-derived target. Theorem 4 accounts for the discrepancy between ideal distribution updates and practical parameter updates through approximation-error terms. Its progress guarantee applies when the ideal improvement exceeds these terms; it does not require treating the implemented update as exact. We discuss the supporting evidence in our response to Q2 below.

4. Existing empirical evidence from personalized multi-turn interactions.

The manuscript already evaluates personalized preference adaptation in Appendix E.12, “Adapting to Individual Preferences on ALOE.” This experiment includes 100 human-verified personas and ten-turn conversations in which preferences are progressively revealed. The model does not receive the complete user profile and must adapt through interaction, with ROSA’s adapted parameters reset between personas.

As reported in Appendix E.12, both methods begin with the same first-turn Alignment Level of 2.93. ROSA achieves an average Alignment Level of 4.40, compared with 3.78 for standard multi-turn interaction, and reaches 4.99 at turn 10, compared with 4.70.

These existing results directly address the practical aspect of the reviewer’s concern: the implemented method improves alignment with individual user preferences in conversations with evolving contexts. They complement the conditional theoretical analysis with empirical evidence from personalized interactions.

AuthorsOpenReview ↗

Response to Reviewer qGs4 (Part2/3)

2. Distribution Matching and the Role of CG (W2, Q1)

1. The scalar closed form addresses a different matching constraint.

The analytical solution identified by the reviewer applies to a linearized constraint on the probability of a single rejected response. We agree that this scalar problem admits a minimum-norm closed-form solution. However, satisfying this constraint does not explicitly specify how the remaining output probabilities should change, and therefore cannot directly replace the joint distribution-fitting problem used in ROSA.

2. Our objective and implementation concern distribution matching.

Our objective is to approximate the feedback-derived target policy distribution, rather than merely adjust one response probability. Equation (3) defines a complete target distribution, and the subsequent KL analysis likewise concerns policy distributions.

In the implementation, ROSA jointly fits full-vocabulary probability vectors at multiple token positions. It does not fit only the scalar probability of the complete response or only the probabilities of the generated tokens. Consequently, the corresponding Jacobian has multiple rows, and its Gram matrix is generally not rank one.

3. CG efficiently solves the joint fitting problem.

Let Fk(θ)F_k(\theta) concatenate the full-vocabulary probability vectors at the selected token positions, with the current context and response prefixes held fixed during the update. Let Fk⋆F_k^\star denote the feedback-derived token-level target vector. Define dk=Fk⋆−Fk(θk−1)d_k=F_k^\star-F_k(\theta_{k-1}) and Jk=∂Fk(θ)/∂θ∣θ=θk−1J_k=\left.\partial F_k(\theta)/\partial\theta\right|_{\theta=\theta_{k-1}}. The linearized fitting problem is:

min⁡Δθ∥JkΔθ−dk∥22.\min_{\Delta\theta}\|J_k\Delta\theta-d_k\|_2^2.

Its associated normal equations are:

Jk⊤JkΔθ=Jk⊤dk.J_k^\top J_k\Delta\theta=J_k^\top d_k.

We use matrix-free CG to approximately solve this system, computing products of the form Jk⊤(Jkv)J_k^\top(J_kv) through JVPs and VJPs without explicitly materializing the Jacobian or its Gram matrix. CG therefore handles the joint probability constraints of the implemented distribution-fitting problem, rather than a single scalar constraint.

For empirical comparisons between these two update formulations, please see our response to Reviewer xdLw, Section 1, point 3 (W1, Q1), which reports accuracy, correction uplift, and update time for the sequence-level closed-form baseline and ROSA across three model sizes. Our response to Reviewer gREb, Section 1, point 3 (W1) further evaluates both methods across four step-size multipliers: ROSA achieves higher accuracy in all twelve model–multiplier combinations, supporting the benefit of joint distribution fitting beyond the default step-size choice.

4. Clarification in the revised manuscript.

To avoid ambiguity, we will revise §3.3, specifically the formulation preceding the CG solver, to explicitly define the vector output, token-level target construction, and residual, and present the linearized least-squares objective above. We will clarify how this practical token-level approximation relates to the sequence-level target in Equation (3), without identifying it as an exact realization of that target over the entire sequence space. These additions will make the correspondence between our distribution-matching objective and the implementation explicit.

AuthorsOpenReview ↗

Response to Reviewer qGs4 (Part3/3)

3. Evidence Supporting the Practical-Update Analysis (Q2)

Theorem 4 explicitly accounts for approximation errors in practical parameter updates. Below, we explain the basis for its feedback and local-smoothness assumptions and summarize the existing empirical evidence for update stability.

1. Reliable negative feedback.

In our failure-driven correction protocol for verifiable tasks, an update is triggered by rejection of the current response, and the interaction terminates upon verified success. This requires feedback on the current response rather than a corrected answer or access to the user’s complete preference distribution. Answer checking and executable tests provide a concrete basis for the reliable-negative-feedback assumption in these evaluations. The personalized ALOE evaluation follows its separate ten-turn protocol.

2. Local smoothness along the actual update path.

Theorem 4 requires the log-policy gradient to be Lipschitz continuous locally along the update path, rather than globally throughout the parameter space. Such regularity conditions have precedent in policy-gradient analysis. For example, Bai et al. [2, Assumption 3] explicitly assume Lipschitz continuity of the log-policy gradient. Castin et al. [3] additionally establish Lipschitz bounds for self-attention and masked self-attention on compact input sets, providing related evidence of local regularity in Transformer components.

For ROSA, the more direct justification comes from the adapted components themselves: we update only the LM-head LoRA parameters or the final hidden state. With the context and response prefixes fixed, these components determine output probabilities through smooth mappings and log-softmax. On a compact neighborhood where the second derivatives are continuous and bounded, the log-policy gradient satisfies the required local Lipschitz condition. Our assumption therefore concerns the actual update region, rather than imposing unconditional global smoothness on the entire model.

3. Existing evidence of stability under repeated practical updates.

The general-knowledge retention table in Appendix E.6 directly evaluates Qwen3-0.6B after ten consecutive ROSA updates on unresolved MATH problems. The average KL divergence from the base model is only 0.001, while MMLU accuracy and WikiText perplexity remain essentially unchanged. These results provide empirical support for the stability of repeated updates and the use of a local approximation.

The extended-horizon results in Appendix E.4 further show that ROSA continues to solve previously unresolved problems: cumulative accuracy increases from 50.20% at turn 10 to 64.80% at turn 20, compared with 25.00% to 27.20% for standard multi-turn interaction. The personalized ALOE results in Appendix E.12, discussed in our response to W1 above, provide complementary evidence of effectiveness in evolving conversational contexts.

Additional controlled evidence is provided in our response to Reviewer gREb, Section 2, point 2 (W2). Starting from identical saved failure states encountered during ROSA interactions, we compare no additional update, the sequence-level closed-form update, and one additional ROSA update. Across the three models, ROSA improves next-response recovery over no additional update by 11.80, 14.40, and 22.80 percentage points, respectively. This experiment directly evaluates the corrective effect of the implemented update while holding the starting model state, conversational context, and feedback fixed.

Together, these existing results support the practical stability and effectiveness of the updates. Theorem 4 retains both target-fitting and linearization errors and provides a conditional progress guarantee when the ideal improvement exceeds these errors. We will clarify this relationship in the revised manuscript, distinguishing the empirical support above from direct verification of the sufficient error condition at every update.

References

[1] Xinyu Zhu et al. The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning. NeurIPS, 2025. https://arxiv.org/abs/2506.01347

[2] Qinbo Bai, Washim Uddin Mondal, and Vaneet Aggarwal. Learning General Parameterized Policies for Infinite Horizon Average Reward Constrained MDPs via Primal-Dual Policy Gradient Algorithm. NeurIPS, 2024. https://papers.nips.cc/paper_files/paper/2024/file/c46c759679acea07d7ea92823ea1e290-Paper-Conference.pdf

[3] Valérie Castin, Pierre Ablin, and Gabriel Peyré. How Smooth Is Attention? ICML, 2024. https://proceedings.mlr.press/v235/castin24a.html

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Rebuttal and Summary of Additional Evidence

We thank the Area Chair, Program Chairs, and all three reviewers for their careful reading and constructive discussion. The rebuttal clarified ROSA’s implementation and added targeted evidence addressing the main methodological and empirical concerns:

  • Scalar closed-form solution and the role of CG (qGs4, W2/Q1; xdLw, W1/Q1; gREb, W1). We clarified that the analytic solution applies to a single sequence-probability constraint, whereas ROSA jointly fits full-vocabulary distributions across multiple token positions. We provided the exact target construction and CG settings and will update Sections 3.2–3.3 and Algorithm 1.

  • Closed-form baseline and step-size effects (xdLw, W1/Q1; gREb, W1). ROSA outperforms the reviewer-derived baseline in all twelve combinations of three model sizes and four step-size multipliers. Comparing each method’s best result within the tested range, ROSA retains 18.60, 7.60, and 7.60 percentage-point advantages on MATH. We also report the update-time tradeoff.

  • Effect of an additional practical update (qGs4, Q2; gREb, W2). Using 100 shared failure states per model and five decoding seeds, ROSA improves next-response recovery over no additional update by 11.80, 14.40, and 22.80 percentage points, and over the closed-form update by 4.00, 7.60, and 14.40 points, under identical starting states, contexts, and feedback.

  • Sequence-level improvement and long responses (qGs4, W1/Q2; gREb, W2 follow-up). We added a Qwen3-8B/MATH-500 evaluation against an empirical acceptance distribution over 51 verified correct responses per problem, covering diverse reasoning paths. Mean ΔKL is negative at every reported turn, with 95.6%–100% of updates decreasing KL. Across 2,836 updates, all response-length groups improve; for responses exceeding 1,000 tokens, mean ΔKL is −0.09 nats, with 96.1% of updates decreasing KL. Details appear in “Follow-up Response to Reviewer gREb (Part2/2).”

  • Random-seed robustness and first-turn variability (xdLw, Q2/Q3). Five-seed evaluations across three model sizes show higher final mean accuracy and lower final standard deviation in all six MATH-500/AIME25 comparisons. We also clarified independent first-turn sampling and reported turn-level uncertainty estimates.

  • Evolving contexts and personalized preferences (qGs4, W1). We clarified that each single-step comparison uses the same current context and highlighted the existing Appendix E.12 ALOE results: from identical initial alignment of 2.93, ROSA achieves average alignment 4.40 versus 3.78 and final alignment 4.99 versus 4.70.

  • Theory and implementation (qGs4, W1/Q2; gREb, W2). We distinguished ideal sequence updates from practical token-level fitting and clarified Theorem 4’s approximation-error conditions. The new experiments directly support practical improvement; we will preserve the distinction between this empirical evidence and a direct approximation bound.

Following the discussion, Reviewer xdLw confirmed that their concerns were resolved and maintained a score of 3, while Reviewer gREb stated that their main concerns were largely addressed and raised their score to 3, recognizing the new KL evaluation as convincing empirical evidence.

We will incorporate these experiments, implementation details, and clarifications into the revised manuscript. We hope this summary provides useful context for the final assessment and remain available for further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.