- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes a experience-based learning that is not only based on the trajectory outcome, but also on the trajectory quality. The method starts from the assumption that what matters in the experience learning is not the final trajectory outcome only, but also the way the agent arrived to this solution. It would allow to benefit from more fine-grained feedback for failed experiences, and distringuish among two successful experiences the more efficient one. In order to implement this, authors propose the Trajectory-Quality-Guided loss (TQG) which combines trajectory outcome (success or failure) with Process Quality indicator, measuring quality of the action validity, length penalty, looping penalty and position of the first detected error.
The method optimizes experience state \Omega = (E, U, N), where E represents the bank of the experiences, U reprents the utility function and N represnts retrieval counts associated with the experiences which minimized the expected TQL loss.
The main features of the method are:
- the way the bank of experiences is constructed: authors make emphasis on the fact that in order to learn to extract useful signal from the experience, it is important to pay attention on the way we select those experiences
- hierarchical abstraction on top of task-specific procedures and task-type-specific decision patterns, as well as raw training trajectories
- the utility function definition: updated with the advantage computed for each task or task type, as difference between baseline TQL loss and Experience-augmented trajectory TQL loss for the fixed task
- the retrieval done for each roll out scores relevant experiences via their semantic similarity combined with their utility
The method is evaluated on 3 benchmarks: ALFWorld, ScienceWorld and WebShop and compared agains other baselines, including MemRL which is pretty close to the proposed method. The evaluation is done in terms of success rate, and efficiency (steps and tokens/episode), and two strong backbone modens (DeepSeek-V4-Flash and Qwen3.5-Flash The reported results demonstrate that TQG-Exp achieves higher success rate, at lower cost compared to the baselines. It also reaches the success in less steps compared to the baseline (which is not necessarily faster, since each step requires more tokens).
Authors perform a number of ablation analysis:
- what is the impact of each component : TQL, trajectory retrieval, hierarchical abstraction for experiences, utility function and raw trjectory retrieval. According to the reported results it seems like Hierarchical abstraction has least impact on the overall method (lowest loss when ablated in presence of other components), even though it is brings something significant when no other components are present, which indicates that the signal it brings is redundant with other components. (eg. I would be curious to see how trajectory-only retrieval behaves at the absence of other components)
Critical strengths
- the motivation makes sense, and the results support the hypothesis
- there is a lot of ablations and details on the prompts, model defitniion, and experience bank creation
Critical weaknesses
There is not many details on the experimental settings, therefore the reproducibility can be quite tricky:
- how exactly the model is trained, the shuffling of the episodes, more details on which tasks are in the training and test subset for different benchmark, to get better idea of the results interpretation Authors mention that they sample 100 test samples from the benchmarks, if it is the case I am also concerned by the "significance of the results" and how comparable they are with the litterature I didn't find any mention of the code and data release in the document, therefore it makes reproducibility a critical issue for this work
Questions for authors
Question:
- in appendix B.1 you mention that for each benchmark you sample 100 test samples, does it mean that the results you report are on 100 samples only?
- what are the exact settings you use for each benchmark? eg. what is the maximum steps allowed? (for more meaningful interpretation of the avg steps metrics reported in the paper)
- I do not fully understand the essence of the method. I understand that there is some sort of learning happens, since authors report results across epochs, and mentione that they sample 100 training samples. Suggestions:
- the same content appears twice in the introduction : lines 91-94 and 101-105; worth rephrasing it for better reading experience
- can you report variance numbers for your experiments (and the baselines) to better assess the significance of the gains
- it would be interesting to have a simple baseline with top-k semantic-relevance retrieval (either experience or trajectory), similar to ExpRag baseline in https://arxiv.org/pdf/2511.20857 or https://arxiv.org/abs/2603.18272 It would provide better understanding of the impact of the utility function that you introduced
9 comments
Official Comment by Authors (part 2)
Dataset Construction. Given the computational cost of repeated agent rollouts, we adopt a resource-constrained experimental setup inspired by [1]. For each benchmark, we construct training and validation sets containing 100 and 30 instances, respectively. For ALFWorld, these instances are sampled according to the original data distribution, while evaluation is conducted on the original Seen and Unseen test sets. For WebShop and ScienceWorld, which contain numerous task categories, sampling directly from their respective original data distributions would leave many categories underrepresented or entirely absent. Drawing on the category-based evaluation settings of [2] and [3], we select the 20 largest task categories in WebShop and the 10 largest in ScienceWorld. The detailed dataset construction procedures are described below.
-
ALFWorld. We perform stratified sampling without replacement from the original ALFWorld training split, allocating integer sampling quotas proportional to the distribution of its six task categories. Using a fixed random seed of 42, we sample 130 task instances and further split them within each category into 100 training and 30 validation instances. The six categories are
pick_and_place_simple,look_at_obj_in_light,pick_clean_then_place_in_recep,pick_heat_then_place_in_recep,pick_cool_then_place_in_recep, andpick_two_obj_and_place, with 22, 8, 19, 13, 15, and 23 training instances and 7, 3, 5, 4, 4, and 7 validation instances, respectively. Both subsets cover all six task categories and contain no overlapping instances. For evaluation, we use the complete officialvalid_seenandvalid_unseensplits, containing 140 and 134 task instances, respectively. -
WebShop. We construct shopping tasks using the WebShop
smallproduct catalog and existing product instructions. For products with valid instructions and attribute information, we enumerate combinations of customizable options and add a maximum price constraint when applicable, yielding 6,910 candidate tasks. After rule-based filtering to remove tasks with obvious attribute conflicts or category mismatches, 3,170 tasks remain. We then select the 20 most frequent query categories based on the frequency of thequeryfield, resulting in a candidate pool of 2,025 tasks. The 20 selected categories are men's loafers & slip-ons, men's t-shirts, throw blankets, men's pants, women's suiting & blazers, men's sleep & lounge, men's shorts, women's activewear, window coverings, women's jumpsuits, rompers & overalls, men's henleys, women's shorts, men's t-shirts & tanks, pantry staples, women's swimsuits & cover ups, women's sandals, women's sweaters, men's casual button-down shirts, women's pants, and men's fashion sneakers. From this candidate pool, we perform balanced sampling without replacement across query categories to construct training, validation, and test sets containing 100, 30, and 100 tasks, respectively. Both the training and test sets cover all 20 categories, with five tasks per category. For the validation set, we sample two tasks from each of the first 10 categories listed above and one task from each of the remaining 10 categories. We use random seed 0 for task generation and seed 20260509 for stratified sampling. The three subsets contain no overlapping goal IDs or identical instruction texts. -
ScienceWorld. We construct our experimental subsets from the ScienceWorld
multi_data/train.parquetdataset, which contains 3,322 task records across 29 task categories. We rank the categories by the number of available records and select the 10 largest categories:inclined-plane-friction-named-surfaces,test-conductivity,test-conductivity-of-unknown-substances,mendelian-genetics-unknown-plant,measure-melting-point-known-substance,find-animal,find-living-thing,find-non-living-thing,find-plant, andmeasure-melting-point-unknown-substance. Using a fixed random seed of 42, we randomly shuffle the records within each category and sample 23 distinct task variations without replacement. For each category, the first 10 variations are assigned to the training set, the next 3 to the validation set, and the remaining 10 to the test set. This yields 100 training, 30 validation, and 100 test tasks, with 10, 3, and 10 tasks per category, respectively. All three subsets cover the same 10 task categories, each accounting for 10% of the tasks within each subset. Moreover, the combinations of task category and variation index (sub_task_name,variation_idx) are disjoint across the training, validation, and test sets. The detailed dataset construction procedures are also provided in Appendix B.1 of the revised manuscript.
Official Comment by Authors (part 3)
Evaluation Size and Comparability. We thank the reviewer for raising this concern and apologize for the ambiguity in our original description. To clarify, the evaluation set sizes differ across benchmarks. For ALFWorld, we evaluate on the complete official valid_seen and valid_unseen splits, containing 140 and 134 tasks, respectively, rather than on 100 sampled test tasks. For WebShop and ScienceWorld, we select the 20 most frequent query categories and the 10 largest task categories, respectively, and construct category-balanced test sets containing 100 tasks each. Importantly, TQG-Exp and all baselines use identical training, validation, and test splits within each benchmark, ensuring that performance comparisons are not affected by differences in task selection.
Code and Data Availability.
To facilitate reproducibility, we have released our implementation, dataset splits, prompt templates, and experimental configurations in an anonymous repository at https://anonymous.4open.science/r/TQG-Exp-1411/.
[1] Jiang, G., et al. XSkill: Continual Learning from Experience and Skills in Multimodal Agents. ICML, 2026.
[2] Gao, P., et al. An Efficient Context-Dependent Memory Framework for LLM-Centric Agents. NAACL Industry Track, 2025.
[3] Yang, W., et al. Coarse-to-Fine Grounded Memory for LLM Agent Planning. EMNLP, 2025.
Q1: in appendix B.1 you mention that for each benchmark you sample 100 test samples, does it mean that the results you report are on 100 samples only?
Reply to Q1: We thank the reviewer for pointing out the ambiguity in our original description in Appendix B.1. We apologize for the confusion caused by our unclear wording, which may have suggested that only 100 test tasks were used for each benchmark.
To clarify, given the computational cost of repeated agent rollouts, we adopt a resource-constrained experimental setup inspired by [1]. For each benchmark, we construct training and validation sets containing 100 and 30 tasks, respectively. However, the evaluation set sizes differ across benchmarks. For ALFWorld, we evaluate on the complete official valid_seen and valid_unseen splits, containing 140 and 134 tasks, respectively, rather than sampling only 100 test tasks.
For WebShop and ScienceWorld, which contain numerous task categories, sampling directly from their respective original data distributions would leave many categories underrepresented or entirely absent. Therefore, drawing on the category-based evaluation settings of [1] and [2], we select the 20 most frequent query categories in WebShop and the 10 largest task categories in ScienceWorld. We then construct category-balanced training and test sets, each containing 100 tasks.
To ensure fair comparisons, TQG-Exp and all baselines use the same training set for training, validation set for selecting the best experience bank, and test set for evaluation within each benchmark. We have revised Appendix B.1 to clarify the evaluation set sizes and provide detailed dataset construction procedures.
[1] Jiang, G., et al. XSkill: Continual Learning from Experience and Skills in Multimodal Agents. ICML, 2026.
[2] Gao, P., et al. An Efficient Context-Dependent Memory Framework for LLM-Centric Agents. NAACL Industry Track, 2025.
[3] Yang, W., et al. Coarse-to-Fine Grounded Memory for LLM Agent Planning. EMNLP, 2025.
Q2: what are the exact settings you use for each benchmark? eg. what is the maximum steps allowed? (for more meaningful interpretation of the avg steps metrics reported in the paper)
Reply to Q2: We thank the reviewer for raising this question. During validation and testing, we set the maximum number of interaction steps to 30 across all three benchmarks (ALFWorld, WebShop, and ScienceWorld), with three rollouts per task. We use temperature 0.4 and top-p 1.0. Experience retrieval returns up to three task-level and one task-type experience, with and . Unlike training, where is used to encourage exploration and prevent retrieval from overly favoring experiences with high historical utility, we set during validation and testing to disable the exploration bonus, as no further experience learning is performed during these stages. For successful trajectory retrieval, we retrieve up to one trajectory on WebShop and two on ALFWorld and ScienceWorld. The experience bank remains fixed during both stages. Avg.Steps is computed over all evaluation trajectories, including both successful and failed ones. We have added the evaluation settings and clarified the definition of Avg.Steps in Appendix B.3 and Section 5.3 of the revised manuscript, respectively.
Official Comment by Authors (part 1)
We thank the reviewer for the constructive feedback. We have substantially expanded the description of our experimental setup and conducted additional experiments and analyses (including repeated evaluations with variance reporting, semantic-only experience retrieval analyses, and a trajectory-only retrieval baseline) to address your concerns regarding reproducibility, evaluation reliability, and the effectiveness of individual components. As the official DeepSeek-V4-Flash model version used in our original experiments is no longer available, we conducted additional experiments using Qwen3.5-Flash.
Reply to W1: Thank you for highlighting these reproducibility concerns. We have expanded the experimental details to clarify the training procedure, episode shuffling, dataset construction, and evaluation settings across all three benchmarks. We also clarify the evaluation sample sizes and ensure that TQG-Exp and all baselines use identical data splits for fair comparisons. We address each concern in detail below.
Training Procedure and Episode Shuffling. TQG-Exp performs parameter-free experience learning by keeping the backbone LLM fixed and updating an external experience bank through environment interactions. In the experiments, the same API-based model is used for action generation and experience reflection. Each benchmark maintains a separate experience bank, initialized as empty. Training runs for 10 epochs over 100 fixed training tasks, with a batch size of 16 tasks and a maximum of 30 interaction steps per trajectory. Each task is assigned eight rollouts, four with experience retrieval and four without. Training rollouts use temperature 1.0 and top-p 1.0, while experience reflection uses temperature 0.0.
After each rollout batch, TQL guides representative-trajectory selection for task-level reflection based on predefined criteria. Generated task experiences are stored in the task-level experience bank and buffered by task type. For each task type, buffered experiences are consolidated into task-type experiences when they cover at least four distinct tasks or their cumulative task-level TQL exceeds 5.0, updating the task-type experience bank. Experience utilities are updated using normalized relative TQL signals against baseline trajectories, with an exponential moving-average coefficient of β=0.5. Retrieval considers up to 20 similarity-ranked candidates from each relevant experience pool and, after filtering according to Eq.13 in the paper, returns up to three task-level experiences and one task-type experience. The relevance–utility trade-off coefficient α and the Upper Confidence Bound (UCB) c are set to 0.7 and 0.05, respectively, across all three benchmarks. During training, we shuffle the training tasks at the beginning of each epoch while keeping the task set fixed. During training, the training task set remains fixed, and the task order is randomly shuffled at the beginning of each epoch. Specifically, for epoch e∈{1,…,10}, we use a PyTorch random generator initialized with seed 42+e−1, ensuring a reproducible task order. We have added more detailed experimental settings in Appendix B.3 of the revised manuscript.