- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors point out that relying on experience retrieved via similarity-based methods is not always beneficial and can even lead to task failure. To address this, the paper proposes AgentEAES, which frames the utilization of experience as an effect-aware decision problem. This approach leverages lexical and cross-features derived from tasks, experiences, and trajectories to train two logistic regression classifiers—one to predict positive outcomes and the other to predict negative ones. The difference between these predictions serves as an effect score, and an experience is incorporated only if its corresponding score is positive.
Experiments were conducted on the AppWorld and WorkBench benchmarks, utilizing Qwen-3.7-Max and Qwen3-30B-A3B as the underlying task models. The results demonstrate that AgentEAES consistently outperforms random selection as well as retrieval methods based on text or vector similarity, yielding significant improvements across certain task domains.
Critical strengths
-
This paper points out that correlation does not equal utility. Through experiments on AppWorld and WorkBench, it demonstrates that the correlation between similarity and actual performance is weak—or even negative—revealing a frequently overlooked issue of "negative transfer" in experience-augmented agents.
-
Rather than reporting only accuracy, the authors provide a deeper analysis of how factors such as the number of interaction turns, token consumption, the cost of generating training labels, and the size of the training dataset affect the quality of performance predictions. These findings offer valuable insight into the practical costs associated with this approach.
Critical weaknesses
-
Although the classifier is lightweight, each base model requires a large number of additional agent rollouts to generate performance labels; consequently, retraining may be necessary when transitioning to a new model. Performance relies on lexical overlap and predefined cross-patterns, which may struggle to capture synonymous paraphrasing, long-range semantic dependencies, abstract strategy mappings, or cross-lingual transfer.
-
The full model achieves a GAUC of 0.634, which drops by only 0.005 to 0.629 when interaction features are removed; meanwhile, a model using only candidate-based features achieves 0.613. This suggests that the model may primarily be learning which experiences are generally effective, rather than relying heavily on specific task-experience pairings. The paper’s claim that interaction features are "crucial" is an overstatement.
Questions for authors
-
How consistent are the G labels when the same task-experience pair is executed repeatedly? How much noise does a single rollout introduce relative to the true expected reward? Does using the average reward from multiple rollouts significantly improve the performance model?
-
Please provide the proportions of samples with positive, zero, and negative rewards, as well as the class distributions for the two binary classification tasks. Specifically, in the absence of class weighting, does class imbalance affect the prediction results and the abstention thresholds?
-
Generating performance labels for AppWorld requires approximately 1.6 * 10^9 tokens; furthermore, the construction of experiences involves using a GPT-5.4-class model for result extraction, segmentation, and hindsight relabeling. During inference, the total token consumption of AgentEAES on AppWorld is 62%–69% higher than that of the baseline without experiences. The paper suggests that these offline costs can be amortized over subsequent invocations; please specify the number of tasks required to reach a cost break-even point.
-
Table 6 in the appendix reports 441 entries per trial, yet the full development sweep for GAUC utilizes 889 candidates. Could you explain why there is roughly a twofold difference between these figures?
2 comments
Q1: How consistent are the G labels when the same task-experience pair is executed repeatedly? How much noise does a single rollout introduce relative to the true expected reward? Does using the average reward from multiple rollouts significantly improve the performance model?
A3: In Appendix C.1, we provide the detailed success rates over three rollouts. In addition, Appendix B.5 discusses how the number of rollouts affects the model’s estimated gAUC on the AppWorld development set. As shown there, increasing the number of rollout-generated samples consistently improves the estimated gAUC.
| Sample size | Training pairs | Dev help-versus-harm GAUC |
|---|---|---|
| 2,700 | 0.602 | |
| 5,400 | 0.624 | |
| 9,090 | 0.634 | |
| 18,000 | 0.643 | |
| 36,000 | 0.652 | |
| Full sweep | 79,120 | 0.660 |
Q2: Please provide the proportions of samples with positive, zero, and negative rewards, as well as the class distributions for the two binary classification tasks. Specifically, in the absence of class weighting, does class imbalance affect the prediction results and the abstention thresholds?
A4: In Table 31, we report the ratio between samples with positive effects and those with negative effects. Class imbalance does indeed affect model performance. However, it does not affect the choice of the abstention threshold in our method: we consistently use as the criterion for deciding whether to use an experience.
| Distribution | Class | Count | Proportion (%) |
|---|---|---|---|
| Gain sign | Positive () | 2093 | 23.29 |
| Zero () | 4967 | 55.27 | |
| Negative () | 1927 | 21.44 | |
| Positive head () | Positive target | 2093 | 23.29 |
| Negative target | 6894 | 76.71 | |
| Negative head () | Positive target | 1927 | 21.44 |
| Negative target | 7060 | 78.56 |
Q3: Generating performance labels for AppWorld requires approximately 1.6 * 10^9 tokens; furthermore, the construction of experiences involves using a GPT-5.4-class model for result extraction, segmentation, and hindsight relabeling. During inference, the total token consumption of AgentEAES on AppWorld is 62%–69% higher than that of the baseline without experiences. The paper suggests that these offline costs can be amortized over subsequent invocations; please specify the number of tasks required to reach a cost break-even point.
A5: This method incurs two types of cost. The first is the cost of rollouts, and the second is the additional context cost introduced each time an experience is used. The amortization referred to in the paper concerns the rollout cost: the number of sampled rollouts used to construct the training data is controllable, and the LLM cost of generating these samples can be amortized over subsequent deployments. However, compared with not using experience at all, the overall cost is still always higher. The key point is that, as the number of downstream calls increases, the one-time cost of sample construction can be spread across later usage. We will also clarify this more explicitly in the cost analysis section of the paper. In Appendix D. we give detail cost analysis.
Q4: Table 10 in the appendix reports 441 entries per trial, yet the full development sweep for GAUC utilizes 889 candidates. Could you explain why there is roughly a twofold difference between these figures?
A6: 441 is the sum of the separately rounded pool-wise per-trial means (11+58+131+241)(11+58+131+241). 889 denotes the number of unique candidate IDs obtained after merging all three source trials and resolving repeated IDs. Candidate IDs do not encode the trial identifier, so records from different trials may share the same ID; under our parsing scheme, such collisions are resolved with a last-wins rule, which retains the later record. This procedure reflects ID resolution rather than deduplication based on identical text.
Additional clarification. Because the Qwen3-30B-A3B model had already been taken offline before the rebuttal period (see \url{https://www.alibabacloud.com/en/notice/model_studio_notice_of_retirement_for_selected_legacy_mainline_models_79d?_p_lc=1}), we reran all WorkBench experiments with Qwen-3.5-35B-A3B backbone. For the failure-mode analysis, however, we retained the previously sampled results obtained with Qwen3-30B-A3B to avoid incurring substantial additional computational cost.
We thank the reviewer for the helpful comments. Our responses are below.
W1: Although the classifier is lightweight, each base model requires a large number of additional agent rollouts to generate performance labels; consequently, retraining may be necessary when transitioning to a new model.
A1: To test transferability within the Qwen family, we replace the backbone with Qwen3.6-Flash while keeping the same effect-aware model used for Qwen3.5-35B-A3B, without any retraining. Even in this zero-adaptation setting, the method continues to improve over the no-experience baseline, similarity-based retrieval, and CQR, showing that it possesses at least some transferability within the same model family. Due to budget and time constraints, we do not further evaluate transfer to backbones outside the same family in the current study, but we plan to report such cross-family results in the open-source release.
W2: The full model achieves a GAUC of 0.634, which drops by only 0.005 to 0.629 when interaction features are removed; meanwhile, a model using only candidate-based features achieves 0.613. This suggests that the model may primarily be learning which experiences are generally effective, rather than relying heavily on specific task-experience pairings. The paper’s claim that interaction features are "crucial" is an overstatement.
A2: Yes, we agree that the term crucial was not sufficiently precise. We have revised the wording accordingly and further added an ablation study examining the impact of removing cross features on the final performance. We have added a baseline that selects the top-ranked experience using a supervised Candidate-Quality Ranking(CQR) estimator in Table 1 and Table 2 to show the end-to-end performance
AppWorld benchmark
WorkBench benchmark