- Rating
- 1
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper asks whether combining different methods (reflection, retrieved memory, and LoRA) makes a coding agent better. Its main point is that the same overall score can hide different outcomes: a combination may solve some new tasks while failing others that a single component could solve on its own. The contribution is mainly an evaluation approach, but the selected methods are not sufficiently justified as representatives of instruction-, memory-, and parameter-based adaptation. Each is only one approach within its category, and the paper does not establish whether the observations extend beyond these particular implementations. The results may therefore reflect specific design choices rather than reveal a broader pattern in how adaptation channels interact. Single-generation comparisons in the main experiment, evaluation errors, and frequent generation failures in the follow-up study further limit interpretability. Overall, the study does not provide sufficiently robust or informative evidence to support an empirical contribution, I recommend rejection.
Critical strengths
None
Critical weaknesses
W1 The main point is fairly basic. The same score can hide different solved tasks, and overlapping gains are different from actual losses. The paper offers limited insight beyond that. Each adaptation channel is represented by just one implementation, without testing whether the findings hold under alternative designs. It is therefore unclear how much the results tell us beyond these particular configurations. Overall, I think the contribution is too limited.
W2 The key results are not shown to be repeatable. The main experiment uses just one generation per task and configuration. We therefore do not know whether the reported gains and losses would hold across repeated runs. Evaluation errors make this harder to judge.
Questions for authors
Q1 What principle for choosing the mechanism for each channel, e.g., why reflection, not react, rewoo ?
Q2 For the rescue/loss transitions in Table 2, how many involve conclusive test outcomes in both configurations, generation failures, or unresolved evaluation errors? How does this breakdown change after the grading replay?
Q3 Repeated-seed experiments and comparisons across alternative implementations would strengthen the study
Q4 Can you provide the exact reflection instructions, memory retrieval procedure, and LoRA training setup, including which adaptation examples each channel used? Appendix A indicates that the artifact-construction details are still incomplete.