acceptodds
Under review as a conference paper at ICLR 2027

A Fix Is Not a Rule: Learning When to Reuse Agent Corrections

Abstract

Large language model agents can learn from failures, generate behavioral corrections, and retain them for future tasks. Yet successfully fixing one failure does not determine when a correction should be reused: the same correction may help, have no effect, or cause harm across different tasks. Agents therefore need to learn both how to correct mistakes and the conditions under which a correction is useful. We propose an experience selection method based on correction scope learning. By observing the direct effects of the same correction across task contexts, the method predicts its utility and guides selective reuse. Experiments on LifelongAgentBench and WebShop, together with supplementary experiments using GLM-5, DeepSeek-V4-Flash, and Qwen3.6-27B, show that the same correction can have positive or negative effects across tasks. Further experiments with GLM-5 show that, under the same activation budget, HCAR-Risk improves mean deployment gain by 0.642 per correction over activation-matched random selection and increases the proportion of selected correction–task pairs with positive effects from 51.9% to 87.8%. These findings establish correction scope as a key learning problem in agent experience reuse: agents need to accumulate effective corrections and use direct evidence of their effects to decide when to apply them.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Model Agents, Self-Evolving Agents, Experience Reuse

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ffHXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks a simple question: if an agent learns a fix from one failure, when should it reuse that fix later? The authors show that the same correction often helps on some tasks and hurts on others. Their method, HCAR, runs each correction with and without it on calibration tasks, learns a predictor of its effect, and only applies it where the predicted benefit is high. In a controlled GLM-5 setup, this clearly beats random selection: precision goes from 51.9% to 87.8%, with much lower harm. On the real benchmarks (WebShop and LifelongAgentBench), though, the gains are small and not significant, especially when evidence is limited or the correction is new.

Critical strengths
  1. Most work on self-improving agents focuses on how to generate and store corrections, and assumes that a stored fix can be reused. This paper points out that fixing one failure doesn’t mean the fix will work elsewhere, and treats “when to reuse” as its own problem. As agent memory and skill libraries grow, this question will only matter more, so this research question is deserve to investigate.

2.The evaluation is careful, honest, and valuable (Although it is obvious that AI conducted the experiments). Activation-matched random is a fair baseline. The authors split data before evaluation, fixed their prediction rules before collecting test outcomes, and use BC-level bootstrap CIs. They also report negative results openly: weak gains with limited evidence and no clear transfer to unseen corrections.

Critical weaknesses

I only have one main concern:

The research question is important, and the method is intuitive, simple, and works well in the controlled setting. But it’s hard to see it carrying over to realistic environments. The method relies on a lot of direct evidence per correction: in the GLM-5 study, each BC gets 48 calibration contexts with 3 paired repeats. When evidence is limited (K=10 in WebShop), it doesn’t beat simple baselines like SupportMean. So in practice, the approach seems to depend on running many paired experiments for every single correction. That cost grows with the size of the correction library, which makes it hard to scale. Given this, I think the current experimental scale is too small (30 BCs and 5 synthetic task types in the main study), and the method doesn’t go deep enough. Fitting a separate scope for each correction won’t scale as the number of corrections grows.

What’s really needed is something closer to meta-learning: learning transferable structure across corrections, so that the scope of a new correction can be predicted from only a few observations. The paper’s own cross-correction transfer results (not significant on either LAB or WebShop) suggest that this is exactly the open problem, and the current method doesn’t address it.

Questions for authors

Overall, I think this work is valuable, but it would be better positioned as an exploratory, findings-driven paper than as a methodology paper. The proposed method is quite simple, and the paper’s own comparison (Table 3) shows that plain Ridge and partial pooling perform about the same as HCAR-Risk. This suggests that the gains come from the direct effect evidence rather than from the method itself. On the other hand, the analysis is a useful exploration for the recursive self-improvement (RSI) area. The finding that source-admitted corrections have mixed-sign effects on future tasks is worth sharing. If the authors go in this direction, I’d encourage them to broaden the empirical study (more corrections, more realistic task distributions) to make the findings more convincing.

Reviewer_XMbkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when agents should reuse behavioral corrections learned from previous failures. It distinguishes improvement on the source task from usefulness on future tasks and measures correction effects through paired rollouts with and without the same correction. HCAR predicts these effects using correction–task features and rank-augmented Ridge regression, while HCAR-Risk selects an activation rate using validation precision and gain. Experiments examine effect heterogeneity in LAB, WebShop, and controlled tasks, followed by evaluations of same-correction learning, limited-evidence calibration, and cross-correction transfer.

Critical strengths

The paper addresses a meaningful aspect of experience reuse: successfully fixing one failure does not establish where the correction should be applied again. Measuring direct effects provides a concrete formulation of this problem. The controlled evaluation includes activation-matched comparisons and alternatives trained on the same calibration data. These comparisons help distinguish selection quality from simply applying fewer corrections.The analysis is transparent about the limits of the results. In particular, the paper reports inconclusive improvements in limited-evidence calibration and cross-correction transfer, alongside comparisons showing that simpler predictors perform similarly.

Critical weaknesses
  1. Evidence for practical correction reuse remains limited. The strongest results come from a controlled setting where every test correction has already been evaluated on 48 calibration contexts, with repeated paired rollouts. This provides substantial supervision for each correction. In contrast, the WebShop limited-evidence study and cross-correction transfer experiments do not establish an advantage over their comparators. The cost of collecting direct-effect observations also remains insufficiently characterized, making it difficult to assess the practical benefit of the approach.

  2. The methodological contribution beyond supervised effect prediction is modest. Plain Ridge and partial-pooling models perform comparably to HCAR-Risk, suggesting that the direct-effect data account for much of the benefit. This is a useful empirical finding, but it leaves the contribution of the proposed ranking objective and selection framework relatively limited. The paper would benefit from positioning its contribution more clearly around what is learned about correction reuse.

  3. The controlled setting leaves uncertainty about generalization to less structured tasks. The predictor receives explicit task factors, correction types, and execution-availability features from the constructed environment. Availability features are particularly influential in the ablations. It remains unclear how reliably these inputs can be obtained in ordinary agent environments, where task requirements and correction prerequisites may be less explicit.

Questions for authors
  1. What is the performance–cost trade-off as the number of direct calibration contexts decreases? Identifying when calibration becomes worthwhile would strengthen the practical interpretation.
  2. How would task factors and correction availability be determined for a new task outside the controlled generator?
  3. Could the authors provide a concrete correction-level case study showing calibration effects, a held-out prediction, and the resulting agent behavior? Both a successful reuse decision and an incorrect one would help explain what the predictor learns.
Reviewer_yZgJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies and analyses when agents should reuse corrections learned from earlier failures. The authors propose HCAR, which predicts the benefit of applying a correction using paired evaluations with and without that correction. Experiments show that corrections can help some tasks and harm others. Scope-based selection improves performance in a controlled setting, while results with limited calibration data and unseen corrections remain inconclusive.

Critical strengths

The paper addresses a relevant problem for agents that accumulate experience. Evaluating the same correction across different tasks makes the risks of unrestricted reuse explicit, rather than assuming that success on the source task implies general usefulness. The evaluation includes informative controls. Matching activation budgets separates selection quality from the number of corrections applied, while comparisons using the same calibration data help distinguish the value of direct evidence from the choice of predictor.

Critical weaknesses
  • The demonstrated benefit depends on substantial evidence for corrections already seen during training. The main experiment uses 48 calibration contexts per correction, each with three paired repeats, and evaluates on only 12 shared test contexts. In Table 4, neither limited-evidence WebShop calibration nor cross-correction transfer establishes an advantage over the corresponding baselines. Although acknowledged, this leaves the practical benefit of the approach uncertain when collecting correction-specific evidence is expensive.

  • The contribution of the proposed predictor is limited. In Table 3, plain Ridge matches HCAR-Risk in precision and slightly exceeds its gain, while partial pooling also performs comparably. The results therefore support learning from direct effects more strongly than the particular ranking objective. Since the positive demonstration also uses explicitly constructed task factors and execution conditions, stronger evidence is needed that this approach learns useful correction scope beyond this structured setting.

Questions for authors
  • How does performance change as the number of calibration contexts is reduced within the controlled experiment? This would help separate the effect of limited data from the differences between the controlled and WebShop environments.
  • Could you specify the controlled utility rubric and clarify how the task, mechanism, and availability features would be obtained for naturally occurring tasks? Does the benefit persist when task domains or factor combinations are held out?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.