acceptodds
Under review as a conference paper at ICLR 2027

Trajectory-aware Experience Retrieval with Contrasting Success and Failure Memories for Embodied VLM Agents

Abstract

Vision-language models (VLMs) have improved embodied agents, but errors in perception, planning, and execution often accumulate in unseen environments. Existing memory-based methods do not fully address this limitation: they retrieve experiences using the task instruction and current observation, favoring goal-level similarity over the agent’s actual execution history. Yet the guidance an agent needs depends on that history, as continuing a successful trajectory requires different guidance from recovering after an error. To address these limitations, we propose Trajectory-aware Experience Retrieval for embodied Agents (TERA), a training-free framework that retrieves guidance based on the trajectory executed so far. TERA summarizes each multimodal trajectory, infers failure causes directly from failed executions, and distills situation-conditioned guidance into separate success and failure memory banks. At inference time, TERA queries both banks using a summary of the current trajectory, enabling the planner to contrast guidance on how to proceed with guidance on what to avoid and thereby improve action selection. Across 16 VLM backbones, TERA improves the mean success rate on EB-Manipulation by up to 7.9% and improves Skill Match, Entity Match, and Exact Match on VLABench by up to 13.2%, 12.9%, and 9.4%, respectively. Ablation studies show that both trajectory-aware retrieval and the separation of success- and failure-derived guidance contribute to these gains.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language Models, Embodied Agent, Memory, Experience, Success and Failure

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_56k1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose a training-free framework for retrieving retrieves past embodied experiences to guide an agent on unseen tasks. The novelty lies in separating successful and unsuccessful past trajectories in separate indexes, retrieving from both as context for a new task.

Critical strengths
  • Learning from prior successes and errors at runtime (no training) is an important problem

  • The approach of splitting memories to successes and failures is interesting

  • The paper is overall well written, but somewhat at a high level and missing important details

  • I found the analysis of "Effect of the Number of Retrieved Experiences" enlightening.

Critical weaknesses
  • Why is a separate prompt per task type necessary? Would it be enough to separate between the types but summarize them in the same way (which would be simpler and more efficient)?

  • Some important details of the methods are missing:

    • How is Phi() - the embedding function - computed?
    • Is the inference-time execution situation summary differs than the offline storage one? Which prompt is used, what does it summarize, given that success or failure is unknown yet?
  • "relevant knowledge arrives together with guidance the agent does not need" - this is the underlying assumption in the paper, that this merge is inefficient. But, this has to be backed up by experiments. IIUC, prior work was only compared against on the Manipulation category in EmbodiedBench, but not on VLABench.

  • "replacing their text-environment interfaces with our observation and action interfaces" - I think that using the original text-env is important as a baseline, and to match the performance of the vanilla prior work versions. I would like to see this comparison too, to show that working and visual rep is actually useful beyond the text rep.

  • "we select five tasks from the Complex task set" - it is not clear why these are selected, and why not all the Complex set is evaluated, or even better, all VLABench, to show that there's no regression in easier cases.

  • Similarly, it is not clear why not all the tasks in EmbodiedBench were evaluated.

  • I found the analysis of "Analysis of Base-Agent Failures" not very informative. I would have preferred to see a deeper dive into the unexpected phenomenon in "Effect of the Number of Retrieved Experiences", in which more retrieved memories are not helpful. I would also like to see a similar analysis for other compared methods, e.g. ExpeL, to see if this is prevalent or only unique to TERA. For example, it could be that the currently used embedding function Phi() is not good enough, but an oracle retriever would provide much better results. I would also appreciate an analysis of the generated guidances. What is their quality? How would a prefect summarized guidance impact performance?

Questions for authors
  • See comments above.

  • I miss a running example, especially for trajectory summarizations and situation-conditioned guidance, to better see the outcomes and differences of these steps

2 comments
AuthorsOpenReview ↗

Thank you for your constructive comments.

Dear Reviewer 56k1, thank you for your constructive comments. We address your concerns below.

W1&W2: Does TERA use different prompts for each task type? How do offline and inference-time summaries differ, and how is the embedding function computed?

TERA uses different prompts based on (1) success/failure outcomes and (2) benchmark execution formats, not task types (e.g., pick cube, wipe table; Appendix A.1, Figures 7–10 in the paper).

Success/Failure: Both prompts share factual summarization instructions. Success summaries emphasize effective strategies, while failure summaries additionally infer failure causes for error-avoidance guidance (Sec. 3.1, p.4).

Offline vs. Inference-time Summarization: The success/failure prompts described above are used for offline memory construction (Figures 7–10 in the paper), which summarizes completed trajectories and their outcomes. In contrast, inference-time prompts (Figures 11–12 in the paper) summarize only accumulated observations, reasoning, plans, and actions, without assuming success or failure, to retrieve experiences relevant to the current execution situation.

Benchmark-specific Formats: EB-Manipulation summarizes sequential observations, reasoning, actions, and feedback across replanning steps. VLABench summarizes initial observations, object information, CoT, and predicted actions from single-pass planning (Sec.s 3.1, 4.1). Both produce factual situation summaries for retrieval; only the input formats differ.

Embedding Function (): TERA uses text-embedding-3-small to embed concatenated instructions and situation summaries (Eq. 3, p.5; Appendix A.4, p.21). The same embedding function is used offline and at inference, with cosine similarity for retrieval (Eqs. 4–5, p.5).

W3: Is trajectory-aware retrieval more effective than instruction-only retrieval on VLABench? How does the retrieved guidance differ?

We additionally compared instruction-only and trajectory-aware retrieval using the same VLABench memory bank.

Table R1. VLABench retrieval comparison(%)

MethodGemini-2.5-FlashInternVL3-38B
SkillEntityExactSkillEntityExact
Baseline57.454.626.760.459.423.0
Instruction-only68.064.831.563.262.324.2
TERA69.066.132.864.162.925.2

Instruction-only improves over Baseline, but TERA achieves higher Exact Match (+1.3/+1.0 for Gemini/InternVL3).

In a Book Rearrangement example, instruction-only retrieved guidance from another sorting task involving duplicate books ("verify the number of all objects") and unnecessary manipulation steps. TERA instead retrieved guidance matching the current execution, emphasizing the next book's selection order and sequential pick&place actions, successfully completing the task. This shows that instruction-only retrieval may include guidance unrelated to the current execution history, even when task goals are similar.

W4&W5&W6: Why does TERA focus on visual embodied environments rather than comparisons in text-based environments? Why were EB-Manipulation and VLABench Complex selected?

Our study focuses on memory construction and retrieval in visual embodied environments, rather than reproducing text-only results. Unlike ALFWorld, where scene information is provided as text, embodied VLM agents must recognize objects from images, and perception errors can propagate into planning (Sec. 1, pp.1–2).

We therefore compared memory designs within the same embodied environments: Table 3 (in the paper, p.8) examines experience sources and guidance representations; Figure 4 (in the paper, p.9) compares retrieval queries using identical memories. TERA achieved the highest performance on both backbones, demonstrating that incorporating accumulated execution information into retrieval is more effective than using only instructions or a single observation.

We selected manipulation benchmarks because they allow us to track object-state changes caused by agent actions from a fixed viewpoint and assess progress toward the goal. In robotic manipulation, the scene remains fixed, and changes arise from the agent's actions. Thus, trajectory images reveal what was attempted and its outcome, enabling factual situation summaries to serve as retrieval queries.

  • EB-Manipulation: Low-level 7D control with replanning and feedback; all five capability splits were evaluated.
  • VLABench Complex: Unlike other tasks that end after 1–2 pick/place actions, Complex tasks average approximately seven actions. Short sequences make it difficult to evaluate guidance retrieval based on previous executions, so we focused on long-horizon Complex tasks requiring action ordering and target selection. We evaluated all five Complex tasks (Book Rearrange,Cook Dishes,Hammer Nail and Hang Picture,Chemistry Experiments,Texas Holdem). We clarified the misleading phrase "we select five tasks from the Complex task set" in red in Sec. 4.1 (p.5).

W7: Why does performance decrease as more memories are retrieved? Does the same phenomenon occur in ExpeL, and how do the relevance and contextual consistency of retrieved guidance affect performance?

We agree that oracle guidance could improve performance, but embodied benchmarks lack ground-truth optimal guidance, and multiple strategies may be valid. In our framework, VLMs interpret executed trajectories, summarize the encountered situations, and generate if–then guidance. This process may introduce perceptual errors that affect planning, omit relevant context due to incorrect situation recognition. To investigate why retrieving more memories can degrade performance, we provide additional analyses of how the relevance and contextual consistency of retrieved guidance relate to the performance degradation observed when more memories are retrieved.

Figure 6 (in the paper, p. 9) shows that increasing TERA retrieval from Top-2 to Top-3 reduces performance. To examine whether this phenomenon also occurs in ExpeL, we increased its insight pool by 1.5×. (since increasing TERA's retrieved memories from Top-2 to Top-3 is a 1.5× increase)

Table R2. Effect of additional experiences on EB-Manipulation (%)

MethodGemini-2.5-FlashInternVL3-38B
Baseline34.219.7
ExpeL (Original)37.7221.05
ExpeL (1.5× Experiences)37.2817.11
TERA Top-2 (2S+2F)42.1026.80
TERA Top-3 (3S+3F)39.4723.68

Both methods show performance degradation with increased memory quantity, suggesting that simply adding more experiences does not guarantee better planning. We also added VLABench success/failure guidance examples (Figure 15a) alongside the existing EB-Manipulation example (Figure 15 in the paper). Our findings suggest that retrieved guidance quality, not merely the embedding function, is important: changing query content improves retrieval effectiveness (Figure 4 in the paper), if–then guidance outperforms summaries (Table 3 in the paper), and cross-task-type retrieval still improves performance (+4.0/+4.9 points).

Perception hallucinations by the VLM backbones used in benchmark evaluation (Figure 16, p. 27), insufficient spatial guidance (Figure 19, p. 28), and conflicting rules (Figure 17, p. 27) further illustrate these limitations. Specifically, we analyzed the additional third-ranked memories in TERA using two criteria:

(1) Current relevance: Cosine similarity below 0.80 (bottom 25% of Top-2 memories). Low-relevance rates were 47.5%/52.4% (Gemini/InternVL3) in success-to-failure episodes, versus 24.3%/28.1% in recovered episodes.

(2) Contextual similarity: Similarity of at least 0.90 to either existing memory. Rates were 67.6%/40.4% in recovered episodes, versus 32.1%/28.2% in success-to-failure episodes.

InternVL3-38B: Current relevance showed a larger difference. Low-relevance success memories occurred more often in success-to-failure episodes (63.3% vs. 25.9%). For example, when placing a triangular prism into a navy container, the agent failed to grasp the prism but proceeded to move its empty gripper above the container and release. Despite the failed grasp, the retrieved guidance advised:

  • If placing an object into a container, then move the gripper well inside the container's boundary before releasing.

This guidance addressed object placement rather than the unresolved grasping failure, illustrating how irrelevant guidance can fail to correct the agent's actions.

Gemini-2.5-Flash: Contextual similarity showed a larger gap (67.6% vs. 32.1%). In a shape-sorter task, the agent grasped a lime-colored star but dropped it onto the lid instead of inserting it into the matching hole. The previously retrieved guidance correctly emphasized alignment:

  • If placing an object into a matching hole, then first align the object directly above the opening before releasing.
  • If inserting an object into a tight-fitting hole, then first align the object directly over the opening before lowering it into place._

However, the additional Top-3 memory introduced contextually mismatched guidance:

  • If a pick-and-place action results in no change to the object's state, then attempt the task with a different object.

Rather than addressing the failed insertion, this guidance suggested switching objects, despite the original task remaining incomplete.

We find that performance degradation is associated with additional memories that are less relevant to the current situation or inconsistent with existing guidance, highlighting the importance of accurate situation summarization and contextually consistent, non-conflicting guidance over simply retrieving more memories.

Reviewer_WC1QOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TERA, which summarizes the agent’s execution history and uses this summary to retrieve relevant guidance from past trajectories. It stores guidance from successful and failed trajectories in separate memory banks and provides both to the agent for action planning.

Critical strengths
  1. The method is tested on a surprising number of different VLMs.
Critical weaknesses
  1. The method is quite incremental. Most of its ideas already exist in prior work (for example the trajectory-aware retrieval is very similar to the proposed method in the AutoGuide paper, which is cited by this paper), and the main new idea is simply to store and retrieve successful and failed experiences separately, which sounds like a very specific and naive engineering trick.
  2. The experiments miss the most important comparisons. The paper does not compare against any embodied-agent memory methods or test whether two separate banks are actually better than putting the same memories into one bank.
Questions for authors

Can you provide the experiment results in W2? Otherwise, it's totally insufficient to validate your method.

3 comments
AuthorsOpenReview ↗

Thank you for your constructive comments.

Dear reviewer WC1Q, thank you for your constructive comments. We have carefully considered your concerns and provide detailed responses below.

W1: How does TERA differ from AutoGuide, and what are the benefits of separating success and failure memories?

TERA and AutoGuide both use trajectories for situation-aware guidance but differ in knowledge extraction, retrieval units, and use of success/failure information (Sec. 2, p.3).

First, AutoGuide generates context-aware guidelines by comparing action differences between paired successful and failed trajectories of the same task. TERA instead generates guidance from individual trajectories without requiring matched success–failure pairs (Sec.s 3.1–3.2, pp. 4–5). This enables independent failure analysis despite scarce successes in Embodied AI environments.

Second, AutoGuide identifies the current context from a partial trajectory and retrieves corresponding guidelines. TERA instead summarizes the accumulated multimodal trajectory and retrieves similar past episodes, providing guidance derived from their execution situations (Sec. 3.3, p.5). Thus, retrieval reflects previously attempted actions and their outcomes, rather than only the current task action.

Third, AutoGuide combines success–failure comparisons into unified guidelines, whereas TERA preserves success- and failure-derived knowledge separately and retrieves both at inference time (Sec.s 2–3.3, pp. 3–5). This allows the planner to jointly utilize effective strategies and error-avoidance guidance.

We compared two single-bank configurations with four retrieved experiences: (1) Merged Bank (All), combining all memories, and (2) Merged Bank (Balanced), equalizing success and failure counts before merging.

Table R1. Comparison of input memory configurations on EB-Manipulation

MethodGemini-2.5-FlashInternVL3-38B
Baseline34.219.7
Success-only Top-439.021.9
Failure-only Top-436.025.0
Merged Bank (All)39.019.7
Merged Bank (Balanced)40.421.5
TERA42.126.8

Table R2. Comparison of input memory configurations on VLABench

MethodGeminiInternVL
SkillEntityExactSkillEntityExact
Baseline57.454.626.760.459.423.0
Success-only Top-468.365.132.264.363.227.1
Failure-only Top-467.464.528.762.861.621.4
Merged Bank (All)67.964.730.062.362.321.9
Merged Bank (Balanced)67.863.529.863.764.124.9
TERA69.066.132.864.162.925.2

As shown in Tables R1–R2, TERA outperforms both merged-bank configurations under the same retrieval budget. On EB-Manipulation, merged banks achieve 39.0/40.4% for Gemini-2.5-Flash and 19.7/21.5% for InternVL3-38B (All/Balanced), compared to TERA's 42.1/26.8%. On VLABench, TERA also achieved the highest scores across all metrics with Gemini. For InternVL3, Success-only led in VLABench Exact Match, mostly due to a single task (texas_holdem), while TERA achieved the second-highest overall performance. However, Success-only trailed Failure-only on EB-Manipulation. This supports retrieving balanced success and failure guidance separately over a single source or a merged bank.

W2: Has TERA been sufficiently compared with existing memory-based methods? Is separating success and failure memories more effective than using a single merged memory bank?

Comparisons with existing memory-based methods are already included. We evaluated ExpeL and ReasoningBank using the same EB-Manipulation agent and experience pool (Table 3, p.8; Figure 3, p.7), and compared task recovery with Reflexion on base-agent failures (Figure 5, p.9; Table 11, p.26). Implementation details are provided in Appendix C (pp. 23–24).

Although originally proposed for LLM agents, ExpeL and Reflexion were also evaluated in embodied environments such as ALFWorld [1]. Their core mechanism of storing knowledge from past executions and retrieving it for new tasks is shared with embodied memory methods. We therefore preserved their memory construction and retrieval procedures while adapting only the observation and action interfaces (Sec. 4.1, p.6; Appendix C, pp. 23–24).

Furthermore, Table 3 (p.8) isolates representative embodied memory designs, including experience sources and guidance representations. Our additional experiments (Table R1) extend this comparison to equal-budget single-source and merged-bank settings, showing that balanced retrieval from separate success and failure banks improves VLM performance.


References

[1] Shridhar, et al. "ALFWorld: Aligning Text and Embodied Environments for Interactive Learning." ICLR 2021

[2] Shinn, et al. "Reflexion: Language Agents with Verbal Reinforcement Learning." NeurIPS 2023

[3] Zhao, et al. "ExpeL: LLM Agents Are Experiential Learners." AAAI 2024

[4] Ouyang, et al. "ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory." ICLR 2026

Thanks to the authors for the response. However, it doesn't address my concerns. I didn't question that these tricks can improve the overall performance, but I still feel like it's incremental and more like a specific engineering trick on the given benchmarks ( EB-Manipulation and VLA-Bench are two very similar benchmarks). The comparison is still unfair in my sense. Your method is obviously specially designed for the embodied settings, so you can't only compare it with general LLM-agent methods just because they can be applied in embodied settings or they also evaluated their methods in such settings. You should compare it with prior embodied-memory methods. Overall, I maintain my score and confidence.

AuthorsOpenReview ↗

Thank you for your follow-up comments.

Thank you for your follow-up and for sharing your perspective. We understand your concerns regarding TERA's novelty and comparisons with prior embodied-memory methods. While we agree that broader comparisons would strengthen the evaluation, we would like to clarify the motivation behind our existing comparisons and TERA's contributions.

First, we believe that memory baselines should be evaluated based on their underlying mechanisms, rather than solely on their original application domains. LLM-agent and embodied-agent memory methods are not mutually exclusive methodological categories, as both can extract reusable knowledge from past executions and retrieve it to guide future decisions. Indeed, ExpeL and AutoGuide have also been evaluated in embodied environments such as ALFWorld.

We therefore selected ExpeL and ReasoningBank based on their shared mechanisms of experience-based knowledge extraction and retrieval. By adapting these methods to the same visual embodied environments while maintaining consistent agent backbones, experience pools, and action interfaces, we aimed to compare memory construction and retrieval strategies under controlled conditions. We believe these comparisons remain methodologically meaningful, even though the original methods were developed for different settings.

Second, TERA's contribution extends beyond separating success and failure memories. TERA retrieves guidance conditioned on accumulated multimodal execution history, independently distills situation-conditioned guidance from successful and failed trajectories without requiring paired demonstrations, and provides both sources to the planner. Our additional matched-budget ablations further compare TERA against single-source and merged-memory configurations, including success/failure labels for the merged variants. These results suggest that the improvements are not solely attributable to retrieving more memories.

We acknowledge that comparisons with additional visual embodied-memory methods would provide broader empirical evidence. However, we believe that the breadth of baseline coverage and the methodological validity of the existing comparisons should be considered separately. We will further clarify the motivation and scope of our comparisons, along with TERA's relationship to prior embodied-memory approaches, in the revised paper. Thank you again for your time and consideration.

Reviewer_VK45OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that experience memories for VLM embodied agents are retrieved by instruction and current observation, ignoring what the agent has already tried. The proposed method, TERA, has a VLM summarize each stored multimodal trajectory and, for failures, infer a failure cause; no paired successful run is needed. Each summary is turned into 2–4 if–then rules, and the rules go into separate success and failure banks. At inference, TERA summarizes the trajectory so far, embeds [instruction; summary], retrieves the top-2 items from each bank, and gives the planner both rule groups, labeled by bank. On EB-Manipulation (228 episodes, 13 reported backbones) and five VLABench Complex tasks (16 backbones), it reports gains of up to +7.9 pp mean success and +13.2/+12.9/+9.4 pp Skill/Entity/Exact Match, plus comparisons to adapted ExpeL, ReasoningBank and Reflexion and query/top-k ablations.

Critical strengths
  1. The method is simple, training-free and backbone-agnostic. No paired success/failure trajectories are needed, experiences pool across source models, and the prompts make it easy to reimplement.

  2. The evaluation is broad across backbones. On VLABench all 8 proprietary models improve on all three metrics. On EB-Manipulation, 7 of 8 proprietary models improve.

Critical weaknesses
  1. The separate-bank contrast is not isolated. The success-only and failure-only variants retrieve 2 items, while TERA retrieves 4, so the comparison confounds the contrast with the amount of guidance. There is no merged-pool top-4 ablation, and no run with the same 4 items shown unlabeled. At a matched budget of 2 items, TERA Top-1 (1+1) on Gemini-2.5-Flash is ≈34% : no better than the baseline and below success-only (35.5%). Also, k=2 was chosen on the test set, and the Gemini gain appears only at k=2. The abstract's claim that ablations show the separation contributes is not supported.

  2. Leave-one-out memory over the test set undercuts the "unseen tasks" framing. The success counts per source model in Table 4 and per split in Table 5 exactly equal those models' baseline successes in Table 1. So the memory bank is just the baseline runs on the 228 evaluation episodes, minus the target episode. In Fig. 15, a "put the blue star into the shape sorter" query retrieves another blue-star/shape-sorter episode that differs only in its distractor preamble. VLABench is the same: memory comes from the same 500 instances, restricted to the same task type. This tests reuse of near-duplicate tasks, not the transfer to unseen tasks claimed in the abstract and introduction.

Questions for authors
  1. How does TERA perform when the memory comes from task families or splits disjoint from the evaluation episodes?

  2. How do a single merged bank with top-4 retrieval, and the same 2+2 items presented without success/failure labels, compare with TERA?

  3. On VLABench, which run is the "preceding attempt", how would a plain second attempt or Reflexion do under the same protocol, and how is success defined for memory construction and truncation?

3 comments
AuthorsOpenReview ↗

Thank you for your constructive comments.

Dear reviewer VK45, thank you for your thoughtful and constructive comments. We address your concerns below and will incorporate all feedback in the final version.

W1 & Q2 Is separating success and failure memories effective under the same retrieval budget? How does it compare to a merged memory bank or removing source labels?

The original ablation (Table 3 in the paper, p. 8) retrieved Top-2 memories for Success-only and Failure-only, versus TERA 2S+2F. To control for this budget difference, we added Success-only Top-4, Failure-only Top-4, and Unlabeled (2S+2F), which removes source labels while retaining the same four experiences. We also evaluated two merged banks: All, combining all memories, and Balanced, equalizing success/failure counts before merging. The original banks contain 150 successes/534 failures (EB-Manipulation) and 212 successes/1,215 failures (VLABench; Sec. 4.1, p. 6).

Table R1. EB-Manipulation mean success rate (%)

MethodGemini-2.5-FlashInternVL3-38B
Baseline34.219.7
Success-only Top-235.520.2
Failure-only Top-229.822.4
TERA Top-1 (1S+1F)34.225.9
Success-only Top-439.021.9
Failure-only Top-436.025.0
Merged Bank (All)39.019.7
Merged Bank (Balanced)40.421.5
Unlabeled (2S+2F)37.726.3
Cross-task-type Memory38.224.6
TERA Top-2 (2S+2F)42.126.8

Increasing single-source retrieval from two to four experiences improved both models. Nevertheless, TERA outperformed both Top-4 single-source and merged-bank variants. Removing source labels reduced performance by 4.4 points (Gemini) and 0.5 points (InternVL3). Balancing a merged bank helped relative to merging all memories, but remained below TERA. These results support combining success and failure guidance with explicit source labels and balanced retrieval.

W2 & Q1: Does TERA rely on similar experiences from the same task type, or can it transfer knowledge across different task types?

The original evaluation excludes the target episode from retrieval (Sec. 3.3, p. 5; Sec. 4.1, pp. 5–6; Appendix B, p. 23). Memories may share a task type (pick cube, place shape sorter, stack cubes, wipe table) but come from different object arrangements and scenes. Thus, the Shape Sorter example (Figure 15 in the paper, p. 22) does not reuse the same episode.

To test transfer across task types, we excluded all memories from the evaluated task type (Cross-task-type Memory, Table R1). Despite this restriction, Gemini and InternVL3 exceeded Baseline by 4.0 and 4.9 percentage points, respectively. Relative to TERA, InternVL3 maintained a 25.6% aggregate success rate outside Wipe Table, but fell from 31.3% to 20.8% on Wipe Table. Gemini improved on Wipe Table but fell from 38.3% to 20.0% on Shape Sorter insertion. These results demonstrate that TERA's success- and failure-based guidance improves performance even across different task types, although task-specific procedures still require well-matched guidance.

Q3: How are the preceding attempt and success defined in VLABench? How does TERA compare with Reflexion?

VLABench Complex is non-interactable and lacks intermediate replanning (Sec. 4.1, pp. 5–6; Appendix B, p. 23). We therefore use the preceding attempt's trajectory only to construct the situation summary for retrieval, not as planner context. Failed trajectories are retained until termination; successful trajectories have their final action removed to represent an incomplete state. Final success/failure labels and evaluation scores are excluded from summarization (Appendix B, p. 23).

We define VLABench success as Exact Match = 100%: the entire predicted action sequence matches the reference in skills, entities, and order. All other episodes are treated as failures for memory construction and truncation.

We additionally evaluated Reflexion on VLABench using the same models as in the ablation study.

Table R2. VLABench performance (%)

MethodGeminiInternVL
SkillEntityExactSkillEntityExact
Baseline57.454.626.760.459.423.0
Reflexion68.566.031.266.265.123.8
TERA (Ours)69.066.132.864.162.925.2

For InternVL3, Reflexion improves Skill and Entity Match more, but TERA achieves higher Exact Match (25.2 vs. 23.8), which requires the complete action–entity sequence in the correct order (Sec. 4.1, p. 6). This suggests that while Reflexion helps improve individual action and object selection, TERA's guidance is more effective in organizing them into a complete plan with the correct execution order. Reflexion reflects on a failed trajectory and retries the same instance; TERA instead provides the planner only guidance retrieved from other experiences, without exposing the preceding trajectory (Sec. 4.1, p. 6; Appendix B, p. 23). TERA remains competitive on both models.

Thank you for the detailed reply and the additional experiments. I have some follow up questions.

1. Separate banks at a matched budget (W1/Q2).

Table R1 shows TERA is the best variant on both models, which is encouraging. However, most of the gain appears to come from the retrieval budget: on Gemini-2.5-Flash, every four-item variant reaches 36.0–40.4% against a 34.2% baseline, and TERA is 1.7 points above the balanced merged bank. On InternVL3-38B, removing the labels costs 0.5 points. With 228 episodes and a single run at temperature 0, these gaps correspond to 4 episodes and 1 episode respectively.

  • For the merged-bank variants, is retrieval top-4, and are the items shown with or without success/failure labels?
  • The Gemini gain appears only at k=2. The new ExpeL comparison (Top-2 vs. Top-3) seems to confirm that results are sensitive to this choice.

2. Leave-one-out memory and the “unseen tasks” framing (W2/Q1).

The cross-task-type result is useful: gains of 4.0 and 4.9 points show real transfer. It also shows that a substantial part of the full gain depends on same-type memories, with large per-type drops. To clarify my original point about Fig. 15: I did not suggest the same episode is reused, but that the retrieved memory is a near-duplicate of the query task.

  • Will the abstract and introduction be revised so that claims about unseen tasks/environments reflect the cross-task-type numbers rather than the leave-one-out numbers?
  • Is a comparable disjoint-memory result available for VLABench, where retrieval is restricted to the same task type by design?
AuthorsOpenReview ↗

Thank you for your follow-up comments and for taking the time to review our additional experiments.

For the merged-bank variants, is retrieval top-4, and are the items shown with or without success/failure labels?

We provided explicit success/failure labels to both merged-bank variants, as in TERA. To better understand the relatively small episode-level gains over the compared variants, we further analyzed the composition and overlap of retrieved memories.

For Gemini-2.5-Flash, Merged Bank (All) retrieved an average of 1.75 success and 2.25 failure memories per query. Of these, approximately 1.04 success and 1.04 failure memories overlapped with TERA, yielding 51.99% overlap among the four retrieved memories.

For InternVL3-38B, balancing the merged bank changed the average success/failure retrieval counts from 1.30/2.70 to 1.95/2.05, improving success rate from 19.74% to 21.49%, approaching Success-only Top-4 (21.93%).

Notably, although Failure-only Top-4 performed substantially better than Success-only Top-4 on InternVL3, increasing the proportion of retrieved success memories still improved performance. This suggests that retrieval composition, rather than simply the number of retrieval budget, matters.

The Gemini gain appears only at k=2. The new ExpeL comparison (Top-2 vs. Top-3) seems to confirm that results are sensitive to this choice.

The improvements are not limited to k=2, where k denotes the number of memories retrieved from each bank. Gemini achieved 39.47% at k=3, exceeding its 34.21% baseline by 5.26 pp, although k=1 matched the baseline. InternVL3 consistently exceeded its 19.74% baseline across all tested values of k.

Table R3. Success rates (%) with different numbers of retrieved memories per success/failure bank

MethodGemini-2.5-FlashInternVL3-38B
TERA Top-1 (1S+1F)34.2125.88
TERA Top-2 (2S+2F)42.1026.80
TERA Top-3 (3S+3F)39.4723.68

Based on these results, we empirically selected k=2, which achieved the highest performance on both evaluated backbones, as a common retrieval configuration for our evaluation across 16 VLMs.

Nevertheless, the optimal retrieval budget may vary with backbone capabilities. Future work could explore adaptive memory selection and training strategies that improve VLMs' ability to interpret retrieved experiences.

Will the abstract and introduction be revised so that claims about unseen tasks/environments reflect the cross-task-type numbers rather than the leave-one-out numbers?

Thank you for clarifying your concern regarding Fig. 15. We understand that your concern is not episode reuse, but rather the retrieval of memories from nearly identical tasks.

Since the main results (Tables 1–2) were obtained under a leave-one-out setting using memories from different episodes of the same task type, we will revise "unseen tasks" to "unseen episodes" in the abstract and introduction to accurately reflect the evaluation setting. We will also incorporate the additional cross-task-type results obtained during the rebuttal period and ensure that claims about generalization to unseen tasks and environments are supported by the corresponding results.

Is a comparable disjoint-memory result available for VLABench, where retrieval is restricted to the same task type by design?

We additionally evaluated VLABench under a cross-task-type setting, excluding all memories from the evaluated task type and retrieving only from other task types.

Table R4. Cross-task-type retrieval results (Disjoint) on VLABench (%)

MethodGeminiInternVL
SkillEntityExactSkillEntityExact
Baseline57.454.626.760.459.423.0
Cross-task65.962.929.861.761.423.7
TERA (Ours)69.066.132.864.162.925.2

As shown in Table R4, cross-task-type retrieval improved all three metrics over the baseline for both models, suggesting that guidance from past experiences can transfer across task types to some extent.

However, the gains varied by task. Texas Hold'em, which shares object-selection and movement patterns with other tasks, showed improvements of 11.1 and 6.0 points on Gemini and InternVL3, respectively. In contrast, gains were limited for Hammer Nail and Hang Picture and Chemistry Experiment, which involve task-specific multi-step procedures. We therefore interpret these results as suggesting that guidance for shared procedures may transfer more effectively than guidance for task-specific procedures.

We will incorporate these cross-task-type results into the paper in a subsequent revision and update the abstract and introduction accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.