acceptodds
Under review as a conference paper at ICLR 2027

Beyond Outcome Feedback: Trajectory-Quality-Guided Experience-based Learning

Abstract

Experience-based learning enables large language model (LLM) agents to improve over time by reusing knowledge from past executions. However, existing methods typically rely on final task outcomes as feedback for experience learning, overlooking the execution process and limiting their ability to evaluate and optimize experience based on trajectory quality. To address this limitation, we propose TQG-Exp, a trajectory-quality-guided experience-based learning framework that introduces process feedback. Specifically, we introduce Trajectory Quality Loss (TQL), which jointly considers task completion and process quality to provide a learning signal beyond final task outcomes. Based on this signal, we employ paired rollouts with and without retrieved experience to guide experience optimization. Representative trajectories are selected based on their outcomes and trajectory quality to extract task-level and task-type-level experiences. Meanwhile, differences in trajectory quality between the paired groups are used to estimate the contribution of retrieved experience and update its utility. The learned utility is then combined with semantic relevance to guide subsequent experience retrieval. Experiments on ALFWorld, WebShop, and ScienceWorld with two backbone LLMs show that TQG-Exp, outperforms existing experience-based methods, improving Avg.@3 over the strongest baseline by 4.50 and 8.55 percentage points on DeepSeek-V4-Flash and Qwen3.5-Flash, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Experience-based learning, Process feedback, Trajectory Quality Loss, Paired rollouts

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XvGZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a experience-based learning that is not only based on the trajectory outcome, but also on the trajectory quality. The method starts from the assumption that what matters in the experience learning is not the final trajectory outcome only, but also the way the agent arrived to this solution. It would allow to benefit from more fine-grained feedback for failed experiences, and distringuish among two successful experiences the more efficient one. In order to implement this, authors propose the Trajectory-Quality-Guided loss (TQG) which combines trajectory outcome (success or failure) with Process Quality indicator, measuring quality of the action validity, length penalty, looping penalty and position of the first detected error.

The method optimizes experience state \Omega = (E, U, N), where E represents the bank of the experiences, U reprents the utility function and N represnts retrieval counts associated with the experiences which minimized the expected TQL loss.

The main features of the method are:

  • the way the bank of experiences is constructed: authors make emphasis on the fact that in order to learn to extract useful signal from the experience, it is important to pay attention on the way we select those experiences
    • hierarchical abstraction on top of task-specific procedures and task-type-specific decision patterns, as well as raw training trajectories
    • the utility function definition: updated with the advantage computed for each task or task type, as difference between baseline TQL loss and Experience-augmented trajectory TQL loss for the fixed task
  • the retrieval done for each roll out scores relevant experiences via their semantic similarity combined with their utility

The method is evaluated on 3 benchmarks: ALFWorld, ScienceWorld and WebShop and compared agains other baselines, including MemRL which is pretty close to the proposed method. The evaluation is done in terms of success rate, and efficiency (steps and tokens/episode), and two strong backbone modens (DeepSeek-V4-Flash and Qwen3.5-Flash The reported results demonstrate that TQG-Exp achieves higher success rate, at lower cost compared to the baselines. It also reaches the success in less steps compared to the baseline (which is not necessarily faster, since each step requires more tokens).

Authors perform a number of ablation analysis:

  • what is the impact of each component : TQL, trajectory retrieval, hierarchical abstraction for experiences, utility function and raw trjectory retrieval. According to the reported results it seems like Hierarchical abstraction has least impact on the overall method (lowest loss when ablated in presence of other components), even though it is brings something significant when no other components are present, which indicates that the signal it brings is redundant with other components. (eg. I would be curious to see how trajectory-only retrieval behaves at the absence of other components)
Critical strengths
  • the motivation makes sense, and the results support the hypothesis
  • there is a lot of ablations and details on the prompts, model defitniion, and experience bank creation
Critical weaknesses

There is not many details on the experimental settings, therefore the reproducibility can be quite tricky:

  • how exactly the model is trained, the shuffling of the episodes, more details on which tasks are in the training and test subset for different benchmark, to get better idea of the results interpretation Authors mention that they sample 100 test samples from the benchmarks, if it is the case I am also concerned by the "significance of the results" and how comparable they are with the litterature I didn't find any mention of the code and data release in the document, therefore it makes reproducibility a critical issue for this work
Questions for authors

Question:

  • in appendix B.1 you mention that for each benchmark you sample 100 test samples, does it mean that the results you report are on 100 samples only?
  • what are the exact settings you use for each benchmark? eg. what is the maximum steps allowed? (for more meaningful interpretation of the avg steps metrics reported in the paper)
  • I do not fully understand the essence of the method. I understand that there is some sort of learning happens, since authors report results across epochs, and mentione that they sample 100 training samples. Suggestions:
  • the same content appears twice in the introduction : lines 91-94 and 101-105; worth rephrasing it for better reading experience
  • can you report variance numbers for your experiments (and the baselines) to better assess the significance of the gains
  • it would be interesting to have a simple baseline with top-k semantic-relevance retrieval (either experience or trajectory), similar to ExpRag baseline in https://arxiv.org/pdf/2511.20857 or https://arxiv.org/abs/2603.18272 It would provide better understanding of the impact of the utility function that you introduced
9 comments
AuthorsOpenReview ↗

Official Comment by Authors (part 1)

We thank the reviewer for the constructive feedback. We have substantially expanded the description of our experimental setup and conducted additional experiments and analyses (including repeated evaluations with variance reporting, semantic-only experience retrieval analyses, and a trajectory-only retrieval baseline) to address your concerns regarding reproducibility, evaluation reliability, and the effectiveness of individual components. As the official DeepSeek-V4-Flash model version used in our original experiments is no longer available, we conducted additional experiments using Qwen3.5-Flash.

W1: There is not many details on the experimental settings, therefore the reproducibility can be quite tricky: how exactly the model is trained, the shuffling of the episodes, more details on which tasks are in the training and test subset for different benchmark, to get better idea of the results interpretation Authors mention that they sample 100 test samples from the benchmarks, if it is the case I am also concerned by the "significance of the results" and how comparable they are with the litterature I didn't find any mention of the code and data release in the document, therefore it makes reproducibility a critical issue for this work.

Reply to W1: Thank you for highlighting these reproducibility concerns. We have expanded the experimental details to clarify the training procedure, episode shuffling, dataset construction, and evaluation settings across all three benchmarks. We also clarify the evaluation sample sizes and ensure that TQG-Exp and all baselines use identical data splits for fair comparisons. We address each concern in detail below.

Training Procedure and Episode Shuffling. TQG-Exp performs parameter-free experience learning by keeping the backbone LLM fixed and updating an external experience bank through environment interactions. In the experiments, the same API-based model is used for action generation and experience reflection. Each benchmark maintains a separate experience bank, initialized as empty. Training runs for 10 epochs over 100 fixed training tasks, with a batch size of 16 tasks and a maximum of 30 interaction steps per trajectory. Each task is assigned eight rollouts, four with experience retrieval and four without. Training rollouts use temperature 1.0 and top-p 1.0, while experience reflection uses temperature 0.0.

After each rollout batch, TQL guides representative-trajectory selection for task-level reflection based on predefined criteria. Generated task experiences are stored in the task-level experience bank and buffered by task type. For each task type, buffered experiences are consolidated into task-type experiences when they cover at least four distinct tasks or their cumulative task-level TQL exceeds 5.0, updating the task-type experience bank. Experience utilities are updated using normalized relative TQL signals against baseline trajectories, with an exponential moving-average coefficient of β=0.5\beta = 0.5. Retrieval considers up to 20 similarity-ranked candidates from each relevant experience pool and, after filtering according to Eq.13 in the paper, returns up to three task-level experiences and one task-type experience. The relevance–utility trade-off coefficient α\alpha and the Upper Confidence Bound (UCB) cc are set to 0.7 and 0.05, respectively, across all three benchmarks. During training, we shuffle the training tasks at the beginning of each epoch while keeping the task set fixed. During training, the training task set remains fixed, and the task order is randomly shuffled at the beginning of each epoch. Specifically, for epoch e∈{1,…,10}e \in \{1,\ldots,10\}, we use a PyTorch random generator initialized with seed 42+e−142+e-1, ensuring a reproducible task order. We have added more detailed experimental settings in Appendix B.3 of the revised manuscript.

AuthorsOpenReview ↗

Official Comment by Authors (part 2)

Dataset Construction. Given the computational cost of repeated agent rollouts, we adopt a resource-constrained experimental setup inspired by [1]. For each benchmark, we construct training and validation sets containing 100 and 30 instances, respectively. For ALFWorld, these instances are sampled according to the original data distribution, while evaluation is conducted on the original Seen and Unseen test sets. For WebShop and ScienceWorld, which contain numerous task categories, sampling directly from their respective original data distributions would leave many categories underrepresented or entirely absent. Drawing on the category-based evaluation settings of [2] and [3], we select the 20 largest task categories in WebShop and the 10 largest in ScienceWorld. The detailed dataset construction procedures are described below.

  • ALFWorld. We perform stratified sampling without replacement from the original ALFWorld training split, allocating integer sampling quotas proportional to the distribution of its six task categories. Using a fixed random seed of 42, we sample 130 task instances and further split them within each category into 100 training and 30 validation instances. The six categories are pick_and_place_simple, look_at_obj_in_light, pick_clean_then_place_in_recep, pick_heat_then_place_in_recep, pick_cool_then_place_in_recep, and pick_two_obj_and_place, with 22, 8, 19, 13, 15, and 23 training instances and 7, 3, 5, 4, 4, and 7 validation instances, respectively. Both subsets cover all six task categories and contain no overlapping instances. For evaluation, we use the complete official valid_seen and valid_unseen splits, containing 140 and 134 task instances, respectively.

  • WebShop. We construct shopping tasks using the WebShop small product catalog and existing product instructions. For products with valid instructions and attribute information, we enumerate combinations of customizable options and add a maximum price constraint when applicable, yielding 6,910 candidate tasks. After rule-based filtering to remove tasks with obvious attribute conflicts or category mismatches, 3,170 tasks remain. We then select the 20 most frequent query categories based on the frequency of the query field, resulting in a candidate pool of 2,025 tasks. The 20 selected categories are men's loafers & slip-ons, men's t-shirts, throw blankets, men's pants, women's suiting & blazers, men's sleep & lounge, men's shorts, women's activewear, window coverings, women's jumpsuits, rompers & overalls, men's henleys, women's shorts, men's t-shirts & tanks, pantry staples, women's swimsuits & cover ups, women's sandals, women's sweaters, men's casual button-down shirts, women's pants, and men's fashion sneakers. From this candidate pool, we perform balanced sampling without replacement across query categories to construct training, validation, and test sets containing 100, 30, and 100 tasks, respectively. Both the training and test sets cover all 20 categories, with five tasks per category. For the validation set, we sample two tasks from each of the first 10 categories listed above and one task from each of the remaining 10 categories. We use random seed 0 for task generation and seed 20260509 for stratified sampling. The three subsets contain no overlapping goal IDs or identical instruction texts.

  • ScienceWorld. We construct our experimental subsets from the ScienceWorld multi_data/train.parquet dataset, which contains 3,322 task records across 29 task categories. We rank the categories by the number of available records and select the 10 largest categories: inclined-plane-friction-named-surfaces, test-conductivity, test-conductivity-of-unknown-substances, mendelian-genetics-unknown-plant, measure-melting-point-known-substance, find-animal, find-living-thing, find-non-living-thing, find-plant, and measure-melting-point-unknown-substance. Using a fixed random seed of 42, we randomly shuffle the records within each category and sample 23 distinct task variations without replacement. For each category, the first 10 variations are assigned to the training set, the next 3 to the validation set, and the remaining 10 to the test set. This yields 100 training, 30 validation, and 100 test tasks, with 10, 3, and 10 tasks per category, respectively. All three subsets cover the same 10 task categories, each accounting for 10% of the tasks within each subset. Moreover, the combinations of task category and variation index (sub_task_name, variation_idx) are disjoint across the training, validation, and test sets. The detailed dataset construction procedures are also provided in Appendix B.1 of the revised manuscript.

AuthorsOpenReview ↗

Official Comment by Authors (part 3)

Evaluation Size and Comparability. We thank the reviewer for raising this concern and apologize for the ambiguity in our original description. To clarify, the evaluation set sizes differ across benchmarks. For ALFWorld, we evaluate on the complete official valid_seen and valid_unseen splits, containing 140 and 134 tasks, respectively, rather than on 100 sampled test tasks. For WebShop and ScienceWorld, we select the 20 most frequent query categories and the 10 largest task categories, respectively, and construct category-balanced test sets containing 100 tasks each. Importantly, TQG-Exp and all baselines use identical training, validation, and test splits within each benchmark, ensuring that performance comparisons are not affected by differences in task selection.

Code and Data Availability.

To facilitate reproducibility, we have released our implementation, dataset splits, prompt templates, and experimental configurations in an anonymous repository at https://anonymous.4open.science/r/TQG-Exp-1411/.

[1] Jiang, G., et al. XSkill: Continual Learning from Experience and Skills in Multimodal Agents. ICML, 2026.

[2] Gao, P., et al. An Efficient Context-Dependent Memory Framework for LLM-Centric Agents. NAACL Industry Track, 2025.

[3] Yang, W., et al. Coarse-to-Fine Grounded Memory for LLM Agent Planning. EMNLP, 2025.

Q1: in appendix B.1 you mention that for each benchmark you sample 100 test samples, does it mean that the results you report are on 100 samples only?

Reply to Q1: We thank the reviewer for pointing out the ambiguity in our original description in Appendix B.1. We apologize for the confusion caused by our unclear wording, which may have suggested that only 100 test tasks were used for each benchmark.

To clarify, given the computational cost of repeated agent rollouts, we adopt a resource-constrained experimental setup inspired by [1]. For each benchmark, we construct training and validation sets containing 100 and 30 tasks, respectively. However, the evaluation set sizes differ across benchmarks. For ALFWorld, we evaluate on the complete official valid_seen and valid_unseen splits, containing 140 and 134 tasks, respectively, rather than sampling only 100 test tasks.

For WebShop and ScienceWorld, which contain numerous task categories, sampling directly from their respective original data distributions would leave many categories underrepresented or entirely absent. Therefore, drawing on the category-based evaluation settings of [1] and [2], we select the 20 most frequent query categories in WebShop and the 10 largest task categories in ScienceWorld. We then construct category-balanced training and test sets, each containing 100 tasks.

To ensure fair comparisons, TQG-Exp and all baselines use the same training set for training, validation set for selecting the best experience bank, and test set for evaluation within each benchmark. We have revised Appendix B.1 to clarify the evaluation set sizes and provide detailed dataset construction procedures.

[1] Jiang, G., et al. XSkill: Continual Learning from Experience and Skills in Multimodal Agents. ICML, 2026.

[2] Gao, P., et al. An Efficient Context-Dependent Memory Framework for LLM-Centric Agents. NAACL Industry Track, 2025.

[3] Yang, W., et al. Coarse-to-Fine Grounded Memory for LLM Agent Planning. EMNLP, 2025.

Q2: what are the exact settings you use for each benchmark? eg. what is the maximum steps allowed? (for more meaningful interpretation of the avg steps metrics reported in the paper)

Reply to Q2: We thank the reviewer for raising this question. During validation and testing, we set the maximum number of interaction steps to 30 across all three benchmarks (ALFWorld, WebShop, and ScienceWorld), with three rollouts per task. We use temperature 0.4 and top-p 1.0. Experience retrieval returns up to three task-level and one task-type experience, with α=0.7\alpha=0.7 and c=0.00c=0.00. Unlike training, where c=0.05c=0.05 is used to encourage exploration and prevent retrieval from overly favoring experiences with high historical utility, we set c=0.00c=0.00 during validation and testing to disable the exploration bonus, as no further experience learning is performed during these stages. For successful trajectory retrieval, we retrieve up to one trajectory on WebShop and two on ALFWorld and ScienceWorld. The experience bank remains fixed during both stages. Avg.Steps is computed over all evaluation trajectories, including both successful and failed ones. We have added the evaluation settings and clarified the definition of Avg.Steps in Appendix B.3 and Section 5.3 of the revised manuscript, respectively.

Reviewer_rM1ZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TQG-Exp, a trajectory-quality-guided experience-based learning framework. As a main component, this paper proposes Trajectory Quality Loss (TQL), an objective that jointly uses task completion and process quality to provide better feedback signal for large language model (LLM) agents. More specifically, TQL uses paired rollouts with and without retrieved experience to estimate the incremental contribution of retrieved experience.

Critical strengths
  • S1. The proposed method, TQG-Exp seems simple but very effective.

  • S2. This paper provides comprehensive comparison to recent works such as ReasoningBank, MemP, MemRL, and DELTAMEM.

  • S3. This paper demonstrates that TGQ-Exp can considerably improve the performance of LLM agents on the benchmarks such as ALFWorld, WebShop, and ScienceWorld. For example, on ALFWorld, Qwen3.5-Flash with TGC-Exp achieves 96.43% of Pass@3, while Qwen3.5-Flash with MemRL achieves 89.29% of Pass@3.

Critical weaknesses
  • W1. Although TQG-Exp seems simple and effective, it uses some hyper-parameters that may sensitively affect the performance of LLM agents. More specifically, TQG-Exp uses β\beta for experience utility update (Eq. 12) and α\alpha for utility-aware experience retrieval. According to Figure 3, on ALFWorld, Average@3 varies from about 91% to about 96% based on the value of α\alpha.
Questions for authors
  • Q1. Does TQG-Exp perform well with proprietary LLMs such as GPT or Claude?
4 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for the constructive feedback. We have provided additional hyperparameter sensitivity analyses and conducted experiments with a proprietary LLM (GPT-6-Luna) on ALFWorld to further evaluate the robustness and cross-model generalizability of TQG-Exp.

W1: Although TQG-Exp seems simple and effective, it uses some hyper-parameters that may sensitively affect the performance of LLM agents. More specifically, TQG-Exp uses β\beta for experience utility update (Eq. 12) and α\alpha for utility-aware experience retrieval. According to Figure 3, on ALFWorld, Average@3 varies from about 91% to about 96% based on the value of α\alpha.

Reply to W1: We thank the reviewer for raising this important concern. We agree that hyperparameter sensitivity is an important consideration for experience-based learning, and would like to clarify the roles of α\alpha and β\beta.

Regarding α\alpha, it controls the trade-off between semantic relevance and historical experience utility during retrieval. As shown in Fig.3, TQG-Exp maintains relatively stable performance over α∈[0.3,0.7]\alpha \in [0.3, 0.7], with Seen Avg.@3 ranging from 94.76% to 96.19% and Unseen Avg.@3 from 89.30% to 90.80%.

Importantly, the performance drop at α=1.0\alpha = 1.0 should also be interpreted in terms of the underlying retrieval mechanism. At this extreme, historical utility is entirely excluded from experience reranking, effectively disabling the utility-aware component. The resulting performance degradation therefore supports the contribution of learned experience utility, rather than solely indicating hyperparameter sensitivity. Similarly, α=0.0\alpha = 0.0 excludes semantic relevance from reranking and yields slightly lower performance than the best configuration. These observations suggest that semantic relevance and learned utility provide complementary information, while a range of intermediate α\alpha values achieves consistently strong performance.

Regarding β\beta, it serves as the exponential moving-average coefficient in experience utility updates (Eq.12), controlling the balance between historical utility estimates and newly observed trajectory-quality feedback. Smaller values favor smoother updates, whereas larger values emphasize recent feedback. We use β=0.5\beta = 0.5 as a balanced configuration across all benchmarks, without benchmark-specific tuning.

Overall, although the performance varies at the extreme values of α\alpha, TQG-Exp maintains relatively stable performance over a broad intermediate range of α∈[0.3,0.7]\alpha \in [0.3, 0.7]. Meanwhile, β\beta is consistently fixed at 0.5 across all benchmarks without benchmark-specific tuning. These results suggest that TQG-Exp does not require fine-grained hyperparameter tuning to achieve consistently strong performance across different benchmarks.

Q1: Does TQG-Exp perform well with proprietary LLMs such as GPT or Claude?

Table 1: Performance comparison on ALFWorld.

ModelMethodSeen Avg.@3 ↑Seen Pass@3 ↑Unseen Avg.@3 ↑Unseen Pass@3 ↑
GPT-6-LunaNo Experience77.3883.5784.8391.79
MemRL94.7696.4393.7894.78
TQG-Exp97.1497.8694.0394.78

Reply to Q1: We thank the reviewer for this question. To further evaluate the generalizability of TQG-Exp to proprietary LLMs, we conduct additional experiments using GPT-6-Luna on ALFWorld, comparing TQG-Exp with the No Experience baseline and MemRL. As shown in Tab.1, TQG-Exp achieves 97.14%/97.86% on Seen Avg.@3/Pass@3 and 94.03%/94.78% on Unseen Avg.@3/Pass@3. Compared with MemRL, TQG-Exp improves Avg.@3 by 2.38 percentage points on Seen tasks and 0.25 percentage points on Unseen tasks, while achieving a higher Pass@3 on Seen tasks and matching MemRL on Unseen tasks. Overall, these results provide further evidence that TQG-Exp remains effective when applied to proprietary GPT models. We have also added the corresponding experimental details and results to Appendix B.4 of the revised manuscript.

AuthorsOpenReview ↗

Follow-up on Author Response

Dear Reviewer rM1Z,

Thank you again for taking the time to review our submission. We wanted to let you know that we have posted our rebuttal, addressing your questions and concerns. If you have any further questions, comments, or concerns, we would be happy to address them.

As the rebuttal deadline approaches, we would greatly appreciate it if you could read our response and, if our clarifications have addressed your concerns, consider raising your score. Thank you again for your time and consideration.

Sincerely,

The Authors

Thank you for providing thoughtful responses to my comments and questions. My concerns about the sensitivity to hyperparameter choices have been largely addressed. I also appreciate the extra experiments with proprietary LLMs. Therefore, I maintain my initial rating of 3.

Dear Reviewer rM1Z,

Thank you for your thoughtful feedback and for carefully considering our responses. We appreciate your acknowledgment that our additional analyses have largely addressed the concerns regarding hyperparameter sensitivity, as well as your recognition of the experiments with proprietary LLMs. Your feedback has been valuable in helping us improve the manuscript. Thank you again for your time and consideration.

Best regards,

The Authors

Reviewer_b42WOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a trajectory quality metric with process feedback and a trajectory-quality-guided experience bank optimization method, enabling LLMs to leverage more useful experiences for task completion. Experiments on two backbone LLMs and three benchmarks show that TQG-Exp outperforms existing experience-based approaches.

Critical strengths

The key innovation of this work is introducing process-feedback-based reward signals for experience bank optimization, an incremental advance built on existing methods and ideas. Compared with conventional outcome-oriented paradigms, this approach evaluates trajectory quality at a finer granularity with higher accuracy. Meanwhile, failed trajectories can also provide informative feedback signals, greatly expanding the scope of usable historical experience. This method offers valuable insights for experience learning in LLMs.

Critical weaknesses

Comparisons against key baselines are confounded by non-core components, weakening the experimental support for the core claim. Specifically, in the ablation study, the variant without successful trajectory retrieval (ds, 91.43 on seen / 89.80 on unseen) shows no significant difference from the strongest baseline, MemRL (90.00 on seen / 90.03 on unseen). Notably, MemRL only injects general experiences rather than successful trajectory demonstrations. Therefore, I suspect that the performance gain over the baseline mainly stems from successful trajectory demonstrations, instead of incorporating process feedback signals into experience bank optimization. I look forward to a reasonable explanation from the authors for this observation.

Questions for authors
  1. The authors only compare inference-time resource cost with baselines, while the training-time (experience bank optimization) resource cost is not reported. Given the respective methodologies, does TQG-Exp incur substantially higher training overhead than MemRL?
  2. Following Question 1: under the continual learning setting where the experience bank requires continuous updates as tasks proceed, would the inference-cost advantage of TQG-Exp disappear?
  3. Several highly relevant references are missing from the paper, such as AttriMem, SkillMaster, and SEAM.
4 comments
AuthorsOpenReview ↗

Official Comment by Authors (part 1)

We thank the reviewers for the constructive feedback and have conducted additional experiments and analyses, including trajectory-retrieval ablations, training-cost comparisons, and continuous experience-bank updates, while expanding the related-work discussion to address concerns about the source of performance gains, training-time cost, continual learning inference-cost, and the omission of relevant related work. As the official DeepSeek-V4-Flash model version used in our original experiments is no longer available, we conducted additional experiments using Qwen3.5-Flash.

W1: Comparisons against key baselines are confounded by non-core components, weakening the experimental support for the core claim. Specifically, in the ablation study, the variant without successful trajectory retrieval (ds, 91.43 on seen / 89.80 on unseen) shows no significant difference from the strongest baseline, MemRL (90.00 on seen / 90.30 on unseen). Notably, MemRL only injects general experiences rather than successful trajectory demonstrations. Therefore, I suspect that the performance gain over the baseline mainly stems from successful trajectory demonstrations, instead of incorporating process feedback signals into experience bank optimization. I look forward to a reasonable explanation from the authors for this observation.

Table 1: Performance comparison on ScienceWorld with Qwen3.5-Flash.

ModelMethodScienceWorld Avg.@3 ↑ScienceWorld Pass@3 ↑
Qwen3.5-FlashNo Memory42.0055.00
ReasoningBank46.6757.00
MemP58.6773.00
MemRL57.6770.00
DELTAMEM54.6768.00
TQG-Exp w/o Traj. Ret.73.0087.00
TQG-Exp77.6794.00

Reply to W1: We thank the reviewer for raising this important concern. We would first like to clarify the actual implementation of MemRL and the role of successful trajectory retrieval in TQG-Exp.

Under the default configuration of the official MemRL repository, an experience contains not only a general experience summary extracted from the trajectory but also its corresponding original trajectory. Our MemRL baseline strictly follows the default implementation provided in the official repository.

Regarding TQG-Exp, successful-trajectory retrieval is an important part of our method, rather than relying on independently provided demonstrations. Specifically, as described in Appendix A.3, after 10 epochs of experience optimization, we use the experience bank optimized at Epoch 10, together with the successful trajectories collected during that epoch, to guide an additional trajectory-collection stage. The resulting successful trajectories form the trajectory bank used for retrieval during testing, whose quality is shaped by the TQL-guided experience optimization.

Additionally, Fig.3 in the manuscript examines the relevance-utility trade-off coefficient α\alpha. When α=1.0\alpha=1.0, experience retrieval relies solely on semantic similarity, while TQL-guided experience optimization and successful trajectory retrieval remain active. With DeepSeek-V4-Flash, compared with the optimal setting of α=0.7\alpha=0.7, Avg.@3 decreases from 96.19 to 91.43 on Seen tasks and from 90.80 to 86.06 on Unseen tasks. This indicates that historical utility provides additional benefits for experience retrieval beyond semantic relevance, even when successful trajectory demonstrations are available.

With trajectory information available to both methods, TQG-Exp achieves higher Avg.@3 on both Seen and Unseen tasks (96.19/90.80 vs. 90.00/90.30 for MemRL). Furthermore, we conduct an additional experiment on the more heterogeneous ScienceWorld benchmark using Qwen3.5-Flash, with successful trajectory retrieval disabled. As shown in Tab.1, TQG-Exp w/o Traj. Ret. achieves an Avg.@3 of 73.00 and a Pass@3 of 87.00, substantially outperforming MemRL (57.67 and 70.00, respectively). Together with the ALFWorld results, where TQG-Exp without successful trajectory retrieval achieves comparable performance to MemRL (91.43/89.80 vs. 90.00/90.30 on Seen/Unseen tasks), these findings demonstrate that TQG-Exp can achieve comparable or better performance than MemRL even without successful trajectory retrieval, while MemRL retains its default retrieval of trajectory-containing memories.

Overall, these results demonstrate that the performance gains of TQG-Exp do not solely rely on successful trajectory retrieval, but stem from the combined contributions of all its components.

AuthorsOpenReview ↗

Official Comment by Authors (part 2)

Q1: The authors only compare inference-time resource cost with baselines, while the training-time (experience bank optimization) resource cost is not reported. Given the respective methodologies, does TQG-Exp incur substantially higher training overhead than MemRL?

Table 2: Training Token Consumption on ScienceWorld using Qwen3.5-Flash.

BenchmarkMethodTrain Tokens (M) ↓
ScienceWorldMemRL671.23
ScienceWorldTQG-Exp550.47

Reply to Q1: We thank the reviewer for pointing out the missing analysis of training-time resource costs. We therefore conduct an additional training-cost analysis on ScienceWorld using Qwen3.5-Flash. Under the same training data and 10-epoch protocol, we measure the cumulative input and output tokens consumed by all LLM API calls. Notably, the cost of TQG-Exp includes experience bank optimization, followed by additional successful trajectory collection, whereas MemRL’s cost covers its standard experience bank optimization. As shown in Tab.2, TQG-Exp consumes 550.47M tokens, 18.0% fewer than MemRL (671.23M). These results demonstrate that TQG-Exp incurs lower overall training overhead than MemRL. This advantage is attributable to the more concise retrieval contexts used by TQG-Exp during training, which primarily retrieves structured task-level and task-type-level experiences, whereas MemRL retrieves experiences containing both generalized experiences and execution trajectories, resulting in longer retrieval contexts and higher token consumption.

Q2: Following Question 1: under the continual learning setting where the experience bank requires continuous updates as tasks proceed, would the inference-cost advantage of TQG-Exp disappear?

Table 3: Task Performance and Inference Token Efficiency under Continuous Experience-Bank Updates on ALFWorld using Qwen3.5-Flash.

MethodAcc ↑Tok./Ep. ↓Tok./Succ. ↓
MemRL73.00%327.48k448.60k
TQG-Exp75.50%75.22k99.63k

Reply to Q2: We thank the reviewer for this follow-up question. To clarify the setting under which the reported inference-cost advantage is obtained, we first explain our experimental protocol. Specifically, TQG-Exp is trained for 10 epochs. Both the task-level and task-type-level experience banks have a maximum capacity of 300 entries. After training, we freeze the experience bank from Epoch 10. We then perform an additional Frozen Trajectory Collection pass over the training tasks to construct the successful trajectory bank, using the frozen experience bank. Next, we evaluate the experience bank snapshots from Epochs 1–10 on the validation set with this fixed trajectory bank. We select the best-performing experience bank based on validation performance. Finally, we use the selected experience bank and the fixed successful trajectory bank for test evaluation, without further updates. All baseline methods follow the same 10-epoch training and validation-based selection protocol to ensure a consistent evaluation procedure.

To further investigate whether TQG-Exp retains its token-efficiency advantage when the experience bank is continuously updated under the continual learning setting, we conduct a continual learning experiment on ALFWorld using Qwen3.5-Flash. Specifically, we organize the 100 training tasks into a sequential task stream with a batch size of 16. For each batch, the agent retrieves previously accumulated experiences to execute tasks and then updates the experience bank based on the resulting trajectories. This setting uses a single training pass without multi-epoch optimization. For TQG-Exp, we omit the additional trajectory collection stage and generate eight rollouts per task: four experience-guided rollouts and four baseline rollouts. The average accuracy, episode count, and number of successful trajectories are calculated based on the experience-guided group only. For both methods, total token consumption is measured as the cumulative number of tokens used throughout the entire learning process.

As shown in Tab.3, compared with MemRL, TQG-Exp improves accuracy from 73.00% to 75.50%, while reducing Tok./Ep. from 327.48k to 75.22k and Tok./Succ. from 448.60k to 99.63k. These results demonstrate that TQG-Exp maintains a substantial token-efficiency advantage over MemRL even when the experience bank is continuously updated. The main reason for TQG-Exp's token-efficiency advantage is that it retrieves concise, generalized experiences, whereas MemRL retrieves experiences containing both generalized experiences and execution trajectories, resulting in longer retrieval contexts and higher token consumption.

AuthorsOpenReview ↗

Official Comment by Authors (part 3)

Q3: Several highly relevant references are missing from the paper, such as AttriMem, SkillMaster, and SEAM.

Reply to Q3: We thank the reviewer for pointing out these relevant works. We agree that these studies are related to our work and have incorporated them into the Related Work section of the revised manuscript.

Specifically, AttriMem attributes the final question-answering outcome to intermediate memory contents through counterfactual attribution, deriving content-level process rewards that are combined with outcome rewards to optimize the memory construction model. SkillMaster enables agents to autonomously propose, update, or retain skills through trajectory reflection, evaluates the utility of skill modifications using counterfactual execution feedback on related tasks, and employs DualAdv-GRPO to update the agent's policy parameters. SEAM, in contrast, trains a lightweight experience-generation model using downstream execution outcome to provide structured guidance to a frozen large language model (LLM), replacing external experience retrieval with parametric experience generation.

Unlike these approaches that update the parameters of different trainable components, TQG-Exp performs no model parameter updates and instead maintains an explicit hierarchical experience bank. It uses TQL to jointly account for task completion and process quality to guide experience learning, thereby improving downstream task performance. In addition, AttriMem focuses on constructing memories from dialogue histories for downstream question answering, TQG-Exp learns reusable task experiences from agents' interaction trajectories to improve subsequent task execution.

AuthorsOpenReview ↗

Follow-up on Author Response

Dear Reviewer b42W,

Thank you again for taking the time to review our submission. We wanted to let you know that we have posted our rebuttal, addressing your questions and concerns. If you have any further questions, comments, or concerns, we would be happy to address them.

As the rebuttal deadline approaches, we would greatly appreciate it if you could read our response and, if our clarifications have addressed your concerns, consider raising your score. Thank you again for your time and consideration.

Sincerely,

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response Summary

We sincerely thank all reviewers for their time and thoughtful evaluation of our work. We also appreciate their recognition of several strengths of our paper. First, Reviewer b42W highlighted the value of process-level feedback in evaluating trajectory quality more accurately and extracting useful learning signals from failed trajectories. Second, Reviewer rM1Z recognized TQG-Exp as a simple yet effective method, and praised its comprehensive comparisons with recent approaches and strong performance across ALFWorld, WebShop, and ScienceWorld. Finally, Reviewer XvGZ considered the motivation well founded and the experimental results supportive of our hypothesis, while highlighting the extensive ablation studies and implementation details, including prompts, model specifications, and experience bank construction.

In response to the reviewers' concerns, we have conducted additional experiments and analyses and made corresponding revisions to the main manuscript and appendices. Specifically, we have expanded the Related Work section to incorporate relevant studies, revised the Introduction to improve readability. We have also substantially expanded the experimental details, including dataset construction in Appendix B.1, training procedures and episode shuffling in Appendix B.3, and evaluation settings in Appendix B.3, while clarifying the definition of Avg.Steps in Section 5.3. In addition, we have strengthened the empirical evaluation through hyperparameter sensitivity analyses, successful trajectory retrieval ablations, training-cost comparisons, continuous experience-bank updates, experiments with a proprietary LLM, repeated evaluations with variance reporting, and additional semantic-only and trajectory-only retrieval baselines.

Below, we provide point-by-point responses to each reviewer's comments and describe the corresponding revisions and additional results.

AuthorsOpenReview ↗

Request for Feedback on Our Responses

Dear Reviewers,

Thank you again for your valuable feedback and thoughtful evaluations of our work.

We would like to follow up to ask whether our rebuttal and additional experiments have adequately addressed your concerns. If any questions remain, we would be happy to provide further clarification.

If you feel that your concerns have been resolved, we would greatly appreciate it if you could consider revisiting your assessments and ratings accordingly.

Thank you for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.