- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ROSA, a method that updates an LLM during a conversation using feedback on its answers. It turns the feedback into a target response probability and uses a linear approximation to update the LM head or hidden states. The model is reset between problems. The paper provides a theoretical analysis and reports improvements across various tasks.
Critical strengths
-
The method learns from negative feedback without needing a corrected answer, which is useful in conversations where users can identify a mistake but cannot provide the solution.
-
The reported gains are substantial, making the empirical results worth further investigation.
Critical weaknesses
-
Equations (4) to (6) reduce to a scaled negative gradient update with a direct minimum norm solution. It is unclear what CG adds, and the comparison with ordinary gradient updates does not isolate the benefit of ROSA’s step size.
-
The convergence claims go beyond the analysis. Theorems 2 and 3 assume exact distribution updates, while Theorem 4 does not bound the approximation errors tightly enough to guarantee progress for the implemented algorithm.
Questions for authors
See weaknesses
9 comments
Response to Reviewer gREb (Part2/2)
2. Exact-Update Analysis and Progress from Practical Updates (W2)
1. The analysis distinguishes ideal distribution updates from practical parameter updates.
Theorems 2 and 3 characterize exact distribution updates. Theorem 4 explicitly incorporates target-fitting and linearization errors when realizing the target through a parameter update.
Its single-step analysis can be summarized as:
where , the first term captures the ideal negative-feedback improvement, and the remaining terms account for approximation errors. The bound guarantees progress when the ideal improvement exceeds these errors. We will make this condition explicit and align the convergence statements in the abstract and main text with the theorem’s assumptions.
For further discussion of the assumptions underlying this analysis, please see our response to Reviewer qGs4, Section 3, points 1–2 (Q2), which explains the reliable-feedback protocol and the local-smoothness justification for the adapted components, together with the relevant literature. Our response to Reviewer qGs4, Section 1, point 2 (W1) also makes explicit that the single-step comparison evaluates the pre-update and post-update policies under the same current conversational context.
2. Controlled one-step recovery directly evaluates the effect of an additional practical update.
We conducted a controlled experiment using 100 shared failure states per model, selected from mathematical and coding evaluation tasks.
Failure-state construction. We collect states from ROSA’s interaction trajectories whenever a response is judged incorrect by the corresponding evaluator. A state remains eligible even if the problem is successfully solved in a later turn. We randomly select 100 distinct problems with eligible failures and retain one failure state per problem. Selection does not depend on the subsequent correction outcome.
Controlled branching. For each selected state, we preserve the complete conversation history, the incorrect response, the negative feedback, and the model parameters and adaptation state immediately before processing that feedback. Three independent branches start from this identical state:
- No Additional Update: retain the current model state without applying another parameter update.
- Sequence-Level Closed-Form Update: apply one update using the reviewer-derived scalar closed-form solution.
- ROSA: apply one additional joint distribution-fitting update.
All branches receive the same next-turn prompt and use the same decoding configuration and evaluator. Each trial permits exactly one subsequent response. We repeat the evaluation with five decoding seeds, restarting from the same saved failure state for each trial. The table reports next-response recovery rates (%) as mean ± sample standard deviation. Each response is evaluated separately; success is not aggregated by selecting the best response across seeds.
| Model | Shared Failed Problems (N) | No Additional Update | Sequence-Level Closed-Form Update | ROSA |
|---|---|---|---|---|
| Qwen3-0.6B | 100 | 29.40 ± 7.97 | 37.20 ± 5.76 | 41.20 ± 3.13 |
| Qwen3-8B | 100 | 38.20 ± 7.21 | 45.00 ± 5.31 | 52.60 ± 2.97 |
| Qwen3-32B | 100 | 41.00 ± 7.03 | 49.40 ± 5.10 | 63.80 ± 2.16 |
ROSA improves next-response recovery over no additional update by 11.80, 14.40, and 22.80 percentage points, respectively. It also exceeds the closed-form update by 4.00, 7.60, and 14.40 percentage points, with lower cross-seed standard deviations.
Because the branches share the same failure states, feedback, and next-turn prompts, this experiment directly shows that an additional ROSA update improves the probability of correcting the error in the next response at failure states encountered during ROSA interactions.
Response to Reviewer gREb (Part2/2)
3. Existing experiments provide complementary evidence of stability and sustained effectiveness.
The general-knowledge retention results in Appendix E.6 evaluate Qwen3-0.6B after ten consecutive ROSA updates on unresolved mathematical problems. The average KL divergence from the base model is 0.001, while MMLU accuracy and WikiText perplexity remain essentially unchanged, supporting the stability of repeated practical updates.
The five-seed evaluations in our response to Reviewer xdLw, Section 2 (Q2, Q3) provide additional evidence across complete interaction trajectories. On both MATH-500 and AIME25, ROSA achieves higher final mean accuracy and lower final cross-seed standard deviation than standard multi-turn interaction for all three model sizes. These results complement the controlled one-step experiment by showing consistent gains over repeated full-interaction evaluations.
The extended-horizon evaluation in Appendix E.4 demonstrates continued recovery on unresolved problems over twenty turns. The personalized-preference evaluation in Appendix E.12 further supports effectiveness in conversations with evolving contexts.
The ALOE protocol and its quantitative results are detailed in our response to Reviewer qGs4, Section 1, point 4 (W1). In that existing manuscript experiment, user preferences are progressively revealed during interaction, and ROSA improves both average and final alignment scores, providing complementary evidence beyond verifiable mathematical and coding tasks.
Theorem 4 characterizes the relationship between ideal improvement and approximation errors, while the new shared-state experiment directly measures the corrective effect of an additional implemented update. Together with the existing multi-turn and knowledge-retention results, these findings support the practical effectiveness and stability of ROSA. We will incorporate the new experiment and clarify the correspondence between the theoretical analysis and the implemented solver in the revised manuscript.
Response to Reviewer gREb (Part1/2)
We sincerely thank the reviewer for the constructive feedback and for recognizing the value of learning from negative feedback without requiring corrected answers, as well as the substantial empirical gains. We address your concerns about the update mechanism, step-size effects, and practical-update guarantees below.
1. Scalar Closed-Form Updates, CG, and Step-Size Effects (W1)
1. The scalar closed-form derivation holds, but addresses a different matching constraint.
For a linearized constraint that matches only the probability of the rejected response, the minimum-norm closed-form solution identified by the reviewer is valid and can be expressed as a scaled, squared-norm-normalized negative-gradient update. We will explicitly discuss this mathematical relationship in the revised manuscript.
Our objective is to approximate the feedback-derived target policy distribution. Equation (3) downweights the rejected response and adjusts the remaining response probabilities through normalization. A single sequence-probability constraint does not explicitly specify how the other output probabilities should change and therefore cannot directly replace the joint distribution-fitting problem used in our implementation.
2. The implementation jointly fits full-vocabulary distributions at multiple token positions.
ROSA concatenates full-vocabulary probability vectors at multiple token positions and jointly fits their feedback-derived targets. Consequently, the Jacobian has multiple rows, and its Gram matrix is generally not rank one.
Let Fk(θ) denote the concatenated probability vector, with the current context and response prefixes held fixed during the update, and let Fk⋆ denote its target. Define dk=Fk⋆−Fk(θk−1) and Jk=∂Fk(θ)/∂θ∣θ=θk−1. The implemented linearized fitting problem is:
minΔθ∥JkΔθ−dk∥22.
We use matrix-free CG to approximately solve the associated normal equations, computing Jk⊤(Jkv) through JVPs and VJPs without explicitly constructing the Jacobian or its Gram matrix. CG therefore handles the joint probability constraints of the implemented distribution-fitting problem.
3. A step-size sensitivity experiment directly examines the reviewer’s concern.
We agree that comparison with ordinary gradient updates alone does not sufficiently isolate step-size effects. We therefore evaluate the reviewer-derived sequence-level closed-form baseline and apply an additional step-size multiplier to both methods:
θk=θk−1+αΔθk.
For the closed-form baseline, Δθk retains the full coefficient derived by the reviewer. For ROSA, it is the update returned by the CG solver. We keep β and all other settings fixed, and evaluate α∈{0.25,0.5,1,2} on MATH with random seed 42. The table reports final cumulative accuracy (%) within ten turns.
ROSA outperforms the closed-form update in all twelve model–multiplier combinations. Even when comparing each method’s highest accuracy within the tested range, ROSA retains advantages of 18.60, 7.60, and 7.60 percentage points, respectively. Thus, within the evaluated step-size range and seed, the advantage cannot be explained solely by the default step-size choice.
For the complementary effectiveness–cost comparison at the default multiplier, please see our response to Reviewer xdLw, Section 1, point 3 (W1, Q1). That table reports final accuracy, correction uplift, and average update time for standard multi-turn interaction, the sequence-level closed-form update, and ROSA. It quantifies the additional computational cost associated with the joint distribution-fitting update alongside its performance gains.
To avoid ambiguity, we will revise §3.3 to explicitly define the vector output, token-level target construction, and residual, and include the least-squares objective above. We will also discuss the scalar closed-form solution, incorporate the step-size comparison, and clarify the approximation relating practical token-level fitting to the sequence-level target in Equation (3).