acceptodds
Under review as a conference paper at ICLR 2027

More Is Not Always Better: Overlapping Gains and Composition Failures in Agent Adaptation

Abstract

Language-model agents can adapt through reflective instructions, retrieved memory, and weight updates. Should combining useful channels improve performance? We study this question with a frozen factorial design, reversed presentation order, and matched token allowances. On 77 held-out coding tasks with Qwen3-8B, 1,386 condition records from 1,232 distinct generations yield no Holm-corrected interaction in either the primary analysis or a grading-replay sensitivity analysis. The primary reflection–memory interaction is percentage points. A task-level audit separates four overlapping-gain cases, where both individual channels and their combination succeed, from three observed combination-damage cases. Adding memory to reflection rescues two tasks and loses two others, leaving the total unchanged. Thus equal scores can hide changes in solved tasks, and a negative interaction estimate alone does not identify destructive interference. An exploratory 270-generation study on reused validation tasks does not establish a benefit from targeted transformation checks. We then test composition decisions on 169 new tasks with 845 generations: a development-fitted, outcome-blind router passes 47 tasks versus 42 for the development-selected fixed combination ( points; Holm ). Reducing memory from three items to one passes 44 versus 42 ( points; Holm ). Evaluation errors remain material: 36 records after replay in the factorial study, 20 in the exploratory study, and 81 in the new-task study. Our evidence distinguishes overlapping benefits from observed composition failures and connects diagnosis to prospective selection tests, while leaving robust antagonism, ordering advantages, and reliable adaptive composition unconfirmed.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Language Model Agents, Multi-Channel Adaptation, Adaptation Composition, Non-Additive Interactions, Adaptive Channel Selection

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ksPYOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks whether combining different methods (reflection, retrieved memory, and LoRA) makes a coding agent better. Its main point is that the same overall score can hide different outcomes: a combination may solve some new tasks while failing others that a single component could solve on its own. The contribution is mainly an evaluation approach, but the selected methods are not sufficiently justified as representatives of instruction-, memory-, and parameter-based adaptation. Each is only one approach within its category, and the paper does not establish whether the observations extend beyond these particular implementations. The results may therefore reflect specific design choices rather than reveal a broader pattern in how adaptation channels interact. Single-generation comparisons in the main experiment, evaluation errors, and frequent generation failures in the follow-up study further limit interpretability. Overall, the study does not provide sufficiently robust or informative evidence to support an empirical contribution, I recommend rejection.

Critical strengths

None

Critical weaknesses

W1 The main point is fairly basic. The same score can hide different solved tasks, and overlapping gains are different from actual losses. The paper offers limited insight beyond that. Each adaptation channel is represented by just one implementation, without testing whether the findings hold under alternative designs. It is therefore unclear how much the results tell us beyond these particular configurations. Overall, I think the contribution is too limited.

W2 The key results are not shown to be repeatable. The main experiment uses just one generation per task and configuration. We therefore do not know whether the reported gains and losses would hold across repeated runs. Evaluation errors make this harder to judge.

Questions for authors

Q1 What principle for choosing the mechanism for each channel, e.g., why reflection, not react, rewoo ?

Q2 For the rescue/loss transitions in Table 2, how many involve conclusive test outcomes in both configurations, generation failures, or unresolved evaluation errors? How does this breakdown change after the grading replay?

Q3 Repeated-seed experiments and comparisons across alternative implementations would strengthen the study

Q4 Can you provide the exact reflection instructions, memory retrieval procedure, and LoRA training setup, including which adaptation examples each channel used? Appendix A indicates that the artifact-construction details are still incomplete.

Reviewer_GqqwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I have a very hard time saying what the paper is about, and I have to admit that I also gave up trying to find out. It seems to evaluate different ways (memory, reflection, weight update) to improve agents and to use a factorial experimental design.

The paper is clearly mostly written by some sort of LLM: It has the property that even if I try to read hard, I simply cannot understand what the text is trying to tell me. Pangram is no poof but also says 100% AI. It is also not properly typeset ("??" in LaTeX) and only has 11 references for a topic that is overflowing with research results.

It clearly has also not been edited: Numbers such as "0.0940239625949737" are still in the text.

But perhaps, I best quote from the paper itself:

"Such checks do not substitute for scientific or source verification. The evaluated language model also generated candidate programs and adaptation materials as described in the experimental protocol. Final human review of the AI-assisted text, code, references, and interpretations remains pending in this working draft."

Critical strengths

I could not review this work because it was AI generated and not readable.

Critical weaknesses

I could not review this work because it was AI generated and not readable.

Questions for authors

I could not review this work because it was AI generated and not readable.

Reviewer_UxLLOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether combining adaptation channels (memory, reflection, weight updates) for a coding agent results in complementary improvements and preserves any original gains. It uses 77 tasks from LiveCodeBench to test this, observing that in some tasks two methods independently solve the same task, or else may cause a previously solved task to fail. None of these interactions are statistically significant. The authors also investigate whether they can identify any deployment rules for memory (how much to use, what to select, etc) but do not find any significant advantage in any of the tested approaches.

Critical strengths
  • The problem studied in this paper is timely and likely of interest to the community
Critical weaknesses
  • Important details required to understand the paper's experiments are obfuscated or not included in the main text, and the writing is dense and often overly abstract, making the paper difficult to follow. For example: (1) The main text doesn't specify which model(s) used in the experiments; I had to search the appendix to find that Qwen3-8B is the only model used.; (2) When reporting statistical analyses, the statistical test used should be clearly mentioned in the same paragraph.; (3) Terminology introduced in the paper is frequently cryptic or tersely and abstractly defined in a way that detracts from readability. For example, "sensitivity" is reported without definition in Table 1, using binary strings to represent experimental conditions makes the paper hard to follow (consider representing conditions with semantically meaningful letters or even words instead), and referring to tasks with names like "abs382_a" also reduces clarity. These are just a few examples; (4) The related work section has many missing references. I recommend revisiting the writing of the paper overall and consider adding components like an example datapoint for LiveCodeBench that illustrates your input data.
  • The experiments performed are not robust enough to provide substantial answers to the research questions posed in the papers. First, only one small model is tested, which limits the generalizability of any results, especially since the statistical power of the experiments seems relatively low. In addition, no statistically significant results are reported, which limits our ability to draw strong conclusions, especially given the limited statistical power.
Questions for authors
  • If you increase your statistical power, do you think your results will change? Why or why not?
  • Do you think that the results from a small model will generalize to larger models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.