acceptodds
Under review as a conference paper at ICLR 2027

Where Should LLM Agents Adapt at Test Time? In Context, in Weights, or Both

Abstract

An LLM agent operating repeatedly in the same deployment environment, such as a code repository, a database, or an application suite, can adapt by retaining experience in context, updating its weights, or both. These approaches are usually studied separately, making it difficult to determine their individual contributions and when combining them helps. We compare adaptation through in-context learning (ICL), reinforcement learning with verifiable rewards using GRPO, and self-distillation, individually and in combination, under a common online protocol. Each agent encounters the same task streams, is evaluated on each task before adapting to it, and learns only from its own attempts and environment feedback. We evaluate 14 adaptation configurations and a stateless baseline with Qwen3.5-9B on 109 tasks from four environments drawn from SWE-bench Verified, AppWorld, ScienceWorld, and BIRD-Critic-SQLite, using five task orders per environment. ICL provides the largest standalone gain: its best configuration raises the success rate, averaged across environments, from 28.8% to 49.4%. Adding GRPO to this configuration yields the highest success rate among all configurations at 51.6%. GRPO improves the average success rate under every ICL policy, whereas the effect of self-distillation varies across environments. Further analyses show that GRPO maintains interaction before submission as retained context grows in the database environment, and that adding weight updates to ICL yields larger gains in pass@k at larger k. Finally, GRPO provides larger average gains with an ICL policy that stops admitting new experience once full (ICL-Prefix) than with one that continually replaces the oldest experience (ICL-FIFO). We release a modular framework for evaluating and combining these adaptation mechanisms. Together, our findings provide evidence that context and weight adaptation are complementary and that retaining experience in context does not exhaust the gains available from learning from it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
llms, agents, test-time-adaptation, continual-adaptation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7deXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents an empirical study of how an LLM agent can improve across a stream of tasks in the same environment by retaining experience in context, updating its weights, or combining the two. It asks what weight updates add when an agent already retains its own experience in context, under a common online protocol. It compares raw-trajectory ICL policies, persistent LoRA updates with GRPO or self-distillation (SDPO), and their combinations while scoring each task before adapting to its outcomes. The experiments use Qwen3.5-9B on 109 tasks from four selected environments, with five task orders and 15 configurations including the stateless baseline. The best ICL configuration raises the equal-environment mean success rate from 28.8% to 49.4%; adding GRPO to that policy reaches 51.6%. Further analyses examine database interaction, pass@k on code issues, and the interaction between context retention and training. The authors conclude that context and weights can contribute complementary gains and should be assessed jointly for the particular environment. The paper also describes a modular framework for evaluating these combinations.

Critical strengths
  1. The comparison is controlled and easy to follow.
  2. There is credible evidence that experience in context can leave room for useful weight update, mostly given by the financial database results.
  3. The paper is comprehensive and honest in acknowledging prior work and the claim of contribution.
Critical weaknesses

None

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for the careful reading and positive assessment. We appreciate your recognition of the controlled comparison, the evidence that weight updates can complement retained experience, and our positioning relative to prior work. If any concerns remain, we would be happy to provide further clarification or evidence, and would be grateful if resolving them could lead you to consider a higher score.

Reviewer_q4hfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper systematically compares three test-time adaptation pathways for LLM agents ICL GRPO and SDPO individually and combined under a unified online evaluation protocol. Across 15 configurations on four environment-shared task streams ICL yields the largest standalone gain raising average success rate from 28.8% to 49.4% while adding GRPO reaches the highest at 51.6% demonstrating that context and weight adaptation are complementary rather than substitutive. Further analyses reveal that GRPO sustains necessary environment interaction as context grows improves multi-attempt coverage pass@k and benefits more from an ICL policy that stops admitting new experience whereas SDPO's effect varies across environments.

Critical strengths
  1. The paper goes beyond surface-level metrics to provide in-depth analysis of behavioral changes.
  2. It releases an extensible open-source framework.
Critical weaknesses
  1. The comparison between GRPO and SDPO is not conducted under equal computational budgets. The authors themselves acknowledge that GRPO receives 32 attempts per task with 8 optimizer steps, whereas SDPO receives 48 attempts with 3 optimizer steps. The two methods also differ in the number of rounds (2 vs. 3). Consequently, the comparison reflects differences between two complete procedural pipelines rather than between two objectives at parity in compute, thereby confounding the causal interpretation of the results.
  2. The paper deliberately employs raw trajectory replay without summarization, retrieval, or external judgment. While this design choice helps isolate the intrinsic contribution of the context channel, the most effective ICL methods in practice typically incorporate reflective summarization (e.g., Reflexion) and workflow extraction (e.g., Agent Workflow Memory). Relying solely on raw trajectories may underestimate the ceiling of ICL, thereby biasing the conclusion that weight adaptation still yields incremental gains in favor of the weight channel.
Questions for authors
  1. GRPO and SDPO differ in both compute budget and number of rounds. How can the performance difference be attributed to the methods rather than resource allocation imbalance?
  2. Does relying solely on raw trajectory replay underestimate the ICL ceiling, thereby biasing the conclusion that weight adaptation still contributes gains? Will be re-validated under stronger ICL baselines?
2 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for the careful and constructive feedback. We appreciate your recognition of our analysis of behavioral changes beyond aggregate performance and our extensible open-source framework. Below, we address your remaining concerns. We have revised the paper to clarify these points, with the relevant passages highlighted in blue.

W1. The comparison between GRPO and SDPO is not conducted under equal computational budgets. The authors themselves acknowledge that GRPO receives 32 attempts per task with 8 optimizer steps, whereas SDPO receives 48 attempts with 3 optimizer steps. The two methods also differ in the number of rounds (2 vs. 3). Consequently, the comparison reflects differences between two complete procedural pipelines rather than between two objectives at parity in compute, thereby confounding the causal interpretation of the results.

Q1. GRPO and SDPO differ in both compute budget and number of rounds. How can the performance difference be attributed to the methods rather than resource allocation imbalance?

Thank you for highlighting this limitation. We agree that the comparison between GRPO and SDPO is not compute-matched: GRPO uses four optimization passes over each round’s 16 attempts, whereas our framework’s SDPO implementation supports one optimizer step per round. We therefore gave SDPO three rounds instead of two, allowing it additional fresh on-policy experience and adaptation opportunities. The interaction-budget difference favors SDPO: it receives 48 attempts per task compared with GRPO’s 32, yet GRPO performs better overall. Thus, GRPO’s advantage is not explained by access to more experience, although its larger number of optimizer steps remains a confounding factor. We have clarified this rationale and limitation in Appendix F.3, and will work on supporting multiple optimization steps per rollout batch for SDPO in future releases of our framework.

W2. The paper deliberately employs raw trajectory replay without summarization, retrieval, or external judgment. While this design choice helps isolate the intrinsic contribution of the context channel, the most effective ICL methods in practice typically incorporate reflective summarization (e.g., Reflexion) and workflow extraction (e.g., Agent Workflow Memory). Relying solely on raw trajectories may underestimate the ceiling of ICL, thereby biasing the conclusion that weight adaptation still yields incremental gains in favor of the weight channel.

Q2. Does relying solely on raw trajectory replay underestimate the ICL ceiling, thereby biasing the conclusion that weight adaptation still contributes gains? Will be re-validated under stronger ICL baselines?

Thank you for raising this concern. Raw-trajectory replay is a strong baseline supported by prior evidence. CL-Bench’s large-scale evaluation across frontier models and six domains with shared latent structure finds that naive ICL outperforms dedicated memory architectures on most tasks, including stronger aggregate results than ACE with the same backbone [1]. Our preliminary experiments were also consistent with these findings, motivating our choice of raw replay.

Using raw trajectories also keeps the source of adaptation consistent across mechanisms, the agents’ own interactions and environmental feedback, without introducing additional memory-construction procedures. We study four ICL policies that vary experience admission and retention. We do not claim that our ICL policies reach the ceiling of ICL-only performance, but prior evidence supports raw-trajectory replay as a strong baseline for assessing the additional contribution of weight adaptation. Evaluating dedicated memory systems both with and without weight updates is an interesting direction we leave to future work. We have clarified this motivation in Section 2 and the scope of our conclusions in Appendix A.

We hope these clarifications address your concerns about the adaptation schedules and the strength of our ICL baselines. We would be grateful if you could consider whether they strengthen your overall assessment of the paper.

References

[1] Asawa et al. Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments. 2026. https://arxiv.org/abs/2606.05661

Thank you for author's response. My concerns have been addressed. Considering that I am unfamiliar with this field and that my current rating is already positive, I will maintain my score.

Reviewer_tBZzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper discusses the issue of whether LLM agents' performance gains can benefit from training weights (such as reinforcement learning or self-distillation) plus in-context learning (ICL). The claim in the paper is that a combination of RL plus ICL has not been attempted in the current literature, so the focus of the paper is on evaluating Qwen3.5-9B on four benchmarks (SWE-bench, AppWorld, ScienceWorld, and BIRD-Critic-SQLite) to see what benefits would be gained by updating both weights and contexts. Results show ICL combined with GRPO consistently gives good performance across tasks.

=== post rebuttal ===

Thank you for the response. I still feel my concern for (W1) and (W3) is not well-addressed, because the direction has been already done, and while it is true that the agentic LLm setting is a novel direction to explore, I feel the scope of the experiment is not large enough in terms of evaluating different model families. Thus, I choose to maintain my score. I feel the study would be more interesting if conducted on more model families.

Critical strengths

The paper explains the experimental setup in detail clearly through figures, visualizations, and mathematical formulas. This is a good sign given it is an empirical work, so people can follow its ideas and method.

Critical weaknesses
  1. Discussing the relationship between in-context learning and weight updates in neural networks has been done before (https://www.pnas.org/doi/10.1073/pnas.2510270122). I believe the main difference between the submission and this past work is that the submission focuses on LLM agentic tasks. However, given that LLMs are also neural networks under the hood, the novelty of the work is limited.
  2. The work doesn’t offer any insights as to why ICL + GRPO gives the best results overall. Further insights on why ICL combined with weight updates would strengthen the depth of the work. Intuitively, one would expect agents’ performance to increase when using ICL + weight updates compared with weights-only or context-only methods, since we have more degrees of freedom in the optimization.
  3. Even if point (2) would be hard to find, I still think the paper would be stronger and more convincing if the experiments were conducted on more settings. Currently, the paper evaluates four tasks on a single Qwen model. If the paper’s main contribution stays in its current state, I believe more extensive experiments are needed to demonstrate a point.
Questions for authors

Algorithm 1 involves a pass@16 metric used as a feedback score. I wonder why specifically choose 16 as the value?

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank the reviewer for the careful and constructive feedback. We appreciate the positive assessment of the paper’s clarity and soundness, particularly the detailed experimental setup. Your suggestions motivated additional behavioral analyses of our results and experiments in two new environments. Below, we clarify our contribution relative to prior work, present new evidence on how context and weight adaptation complement each other, and report the additional experimental results. We have revised the paper to address these concerns, with the relevant passages highlighted in blue.

W1. Discussing the relationship between in-context learning and weight updates in neural networks has been done before (Russin et al.). I believe the main difference between the submission and this past work is that the submission focuses on LLM agentic tasks. However, given that LLMs are also neural networks under the hood, the novelty of the work is limited.

Thank you for pointing us to Russin et al.; we have added this reference to our extended related works section and clarified our positioning in Section 2. Our contribution is not the first combination of in-context learning and weight updates. The original submission already discussed approaches combining these channels in the related-work paragraph “Combining context and weights.” Our goal is to understand what each channel contributes and when they complement each other through a controlled empirical study.

We study LLM agents adapting through their own interactions across tasks in a shared environment. In this agentic setting, interaction and learning from experience are essential for discovering latent environment structure that is reusable across tasks. Our work provides a systematic comparison of these adaptation choices, together with behavioral evidence that helps explain their distinct contributions and interactions. This also addresses a gap beyond demonstrating that a particular combination works: understanding what each channel contributes under a common online protocol and how the two can complement each other. We have clarified this positioning in Section 2 of the revision.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

W2. The work doesn’t offer any insights as to why ICL + GRPO gives the best results overall. Further insights on why ICL combined with weight updates would strengthen the depth of the work. Intuitively, one would expect agents’ performance to increase when using ICL + weight updates compared with weights-only or context-only methods, since we have more degrees of freedom in the optimization.

Thank you for this instructive feedback. In response, we conducted further analyses of outcomes, interaction length, and behavior across all four benchmarks, extending the evidence in the original submission. We further examined what context and weight updates each contribute individually, and investigated how they complement each other, providing more insights beyond the additional degrees of freedom available to hybrid mechanisms.

ICL can reuse environment knowledge to act more efficiently. ICL substantially reduces unfinished attempts and shortens interactions across benchmarks (Figure 3). Even successful attempts become shorter (Figure 9). Inspecting execution traces motivated measurements of code lookups on SWE-bench, API lookups on AppWorld, and environment lookups on ScienceWorld. ICL reduces all three, while weight-only mechanisms remain closer to the frozen baseline (Figure 4). These observations are consistent with retained experiences providing knowledge of the environment’s latent structure, reducing the need to rediscover it on each task. Figure 7 provides concrete examples.

Greater efficiency can come at the cost of verification. Although experience can supply reusable environment knowledge, finding information relevant to the current task and checking a proposed solution may still require new interactions. ICL risks extending the shortcut of skipping familiar exploration to these task-specific steps. BIRD illustrates this distinction: ICL increasingly submits SQL without first querying the database, skipping opportunities to inspect relevant data or test its proposed answer (Figure 4). SDPO also reduces querying, without a corresponding improvement in success.

GRPO preserves ICL’s reduced exploration while restoring database interaction. We examine Prefix–All in detail, comparing frozen-weight ICL with its GRPO and SDPO hybrids and the frozen, no-ICL baseline. Adding GRPO improves success while retaining ICL’s reduced code, API, and environment lookups (Figure 5). The most informative change appears on BIRD: querying initially declines as it does with ICL, but then recovers toward the frozen baseline as the stream progresses. ICL and ICL+SDPO instead continue to submit mostly without querying. Higher late-stream querying with GRPO also appears under the other three ICL policies (Figure 10).

The behavioral shift extends to task-specific inspection and verification in other benchmarks. Motivated by the BIRD observation, we measure the fraction of attempts that repeat the same check after a source edit on SWE-bench, inspect final state on AppWorld, and inspect state during melting-point experiments on ScienceWorld. GRPO increases these measures while preserving reduced environment lookups (Figure 6). This distinguishes looking up repository code from rechecking a patch, looking up API documentation from inspecting an operation’s effects, and looking around the environment from inspecting an experiment’s progress. Figure 8 illustrate these differences in execution traces. SDPO increases rechecking on SWE-bench but does not show the same increases in inspection on AppWorld and ScienceWorld.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

Together, these observations support the interpretation that policy optimization can refine how the agent uses retained experience. Retained traces provide environment information and examples of how to act, but a useful procedure or shortcut on one task may be insufficient on another. By optimizing the context-conditioned policy for outcome reward, GRPO can reinforce effective reuse while correcting how that experience is applied. Our evidence is consistent with this interpretation. We have incorporated these analyses and supporting figures into the revised manuscript, splitting the former Section 5 into Section 5, “Individual Contributions of Context and Weights,” and Section 6, “When Context and Weight Updates Complement Each Other,” with detailed analyses and examples in Appendix.

W3. Even if point (2) would be hard to find, I still think the paper would be stronger and more convincing if the experiments were conducted on more settings. Currently, the paper evaluates four tasks on a single Qwen model. If the paper’s main contribution stays in its current state, I believe more extensive experiments are needed to demonstrate a point.

Thank you for encouraging us to broaden the evaluation. In response, we conducted experiments on two additional, larger environments: scikit-learn from SWE-bench Verified, containing 32 tasks, and Toxicology from BIRD-CRITIC, containing 64 tasks. These introduce a different code repository and database, each with shared environment structure that agents can learn through interaction.

We compare the frozen baseline, ICL, GRPO, SDPO, and both hybrids, using Prefix–Successful for ICL. ICL+GRPO achieves the highest success rate among the six mechanisms in both new environments: 62.9% on scikit-learn and 53.1% on Toxicology, compared with frozen baselines of 31.1% and 42.0%, respectively (Figure 12). These additional experiments use seed 4, with the same task order across mechanisms and 16 attempts per task for evaluation.

The channel-gain figure (Figure 13) makes the contributions clearer. ICL is the strongest standalone mechanism in both environments, and adding GRPO improves it further, by 6.4 percentage points on scikit-learn and 4.3 points on Toxicology. The hybrid also exceeds GRPO alone by 9.0 and 6.0 points, respectively. Adding SDPO to ICL instead has mixed effects: it improves success on Toxicology but reduces it on scikit-learn. Thus, the new environments reinforce both central findings: context and weight adaptation can complement each other, but combining channels does not automatically improve performance; the update mechanism matters.

Together with the behavioral analyses described in our response to W2, these experiments broaden the empirical support for our findings and deepen our account of how the channels interact.

Q1. Algorithm 1 involves a pass@16 metric used as a feedback score. I wonder why specifically choose 16 as the value?

Thank you for the question. We sample 16 parallel attempts per task per round, consistently across mechanisms, matching the rollout budget used in SDPO’s test-time adaptation experiments. We therefore report pass@k for budgets up to 16. Pass@16 is an evaluation metric, not the feedback used for adaptation; updates use the individual attempts and their environment rewards and feedback.

We hope the clarified positioning, new behavioral insights, and experiments in two additional environments address your concerns about novelty, depth, and experimental breadth. We would be grateful if you could consider raising your score in light of these additions.

AuthorsOpenReview ↗

Follow-up on our response

Thank you again for your constructive feedback. As today is the final day of the rebuttal period, we would appreciate your thoughts on our responses and revised manuscript. In particular, we added behavioral analyses examining how ICL and GRPO complement each other, experiments in two additional environments, and clarification of our contribution relative to prior work. We would be grateful if you could consider whether these additions address your concerns and support an updated assessment. We would also welcome the opportunity to clarify any remaining questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the AC and All Reviewers

We thank the AC and all three reviewers for their time and constructive feedback. We appreciate the recognition of our clear and controlled experimental setup, behavioral analyses beyond aggregate performance, extensible open-source framework, and careful positioning relative to prior work. The reviewers’ questions motivated additional analyses and experiments that strengthen the paper’s evidence and clarify its contribution.

Our study asks what context and weight updates each contribute when an LLM agent repeatedly operates in a shared environment, and when combining them helps. We compare these channels under a common online protocol, using the same model, task streams, and source of experience and feedback. The central finding is that retained experience provides substantial gains, while policy optimization with outcome rewards can further improve how the agent uses that experience.

We have revised the manuscript to improve its presentation and incorporate the new results, with passages addressing the reviews highlighted in blue. Figure numbers below refer to the revised paper; links open the corresponding anonymous figures. The main additions and clarifications are:

  1. Behavioral evidence explaining how the channels complement each other. In response to the request for insights as to how ICL and GRPO complement each other, we expanded and reorganized the former analysis section into Section 5, “Individual Contributions of Context and Weights,” and Section 6, “When Context and Weight Updates Complement Each Other.” ICL reduces unfinished attempts, interaction length, and environment lookups, consistent with reusing knowledge of latent environment structure (Figure 3, Figure 4). However, it can also skip inspection and verification. Adding GRPO preserves reduced lookups while restoring opportunities for feedback: most clearly, database querying on BIRD initially declines but subsequently recovers (Figure 5), a pattern found across all four ICL policies (Figure 10, Appendix C). Further measurements capture increased rechecking after code edits, final-state inspection, and inspection during experiments (Figure 6) in other benchmarks. These findings support the interpretation that reward optimization can refine how retained experience is applied. Appendix C provides detailed analyses and execution examples.

  2. Experiments in two additional, larger environments. We added scikit-learn from SWE-bench Verified and Toxicology from BIRD-CRITIC, contributing 96 additional tasks and extending the evaluation from four environments to six. ICL is the strongest standalone mechanism in both new environments, and ICL+GRPO achieves the highest success rate among the six tested mechanisms (Figure 12), improving over ICL by 6.4 and 4.3 percentage points, respectively (Figure 13). SDPO’s incremental effect remains environment-dependent, reinforcing that combining channels does not automatically help. Appendix D reports these additional experiments, which use one shared task-order seed.

  3. Clearer positioning and justification of the ICL baseline. Section 2 and Appendix A clarify that our contribution is a controlled empirical study of the channels’ individual contributions and interactions, rather than the first combination of context and weight adaptation. We clarified the motivation for raw-trajectory replay: prior evidence supports it as a strong baseline, and it allows mechanisms to use the same source of experience without introducing separate memory-construction procedures.

We are grateful for the feedback that prompted these improvements. We hope the revised manuscript and our individual responses address the concerns about positioning, explanatory depth, experimental breadth, and comparison scope. We would be happy to provide further clarification or evidence during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.