- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors propose a training-free framework for retrieving retrieves past embodied experiences to guide an agent on unseen tasks. The novelty lies in separating successful and unsuccessful past trajectories in separate indexes, retrieving from both as context for a new task.
Critical strengths
-
Learning from prior successes and errors at runtime (no training) is an important problem
-
The approach of splitting memories to successes and failures is interesting
-
The paper is overall well written, but somewhat at a high level and missing important details
-
I found the analysis of "Effect of the Number of Retrieved Experiences" enlightening.
Critical weaknesses
-
Why is a separate prompt per task type necessary? Would it be enough to separate between the types but summarize them in the same way (which would be simpler and more efficient)?
-
Some important details of the methods are missing:
- How is Phi() - the embedding function - computed?
- Is the inference-time execution situation summary differs than the offline storage one? Which prompt is used, what does it summarize, given that success or failure is unknown yet?
-
"relevant knowledge arrives together with guidance the agent does not need" - this is the underlying assumption in the paper, that this merge is inefficient. But, this has to be backed up by experiments. IIUC, prior work was only compared against on the Manipulation category in EmbodiedBench, but not on VLABench.
-
"replacing their text-environment interfaces with our observation and action interfaces" - I think that using the original text-env is important as a baseline, and to match the performance of the vanilla prior work versions. I would like to see this comparison too, to show that working and visual rep is actually useful beyond the text rep.
-
"we select five tasks from the Complex task set" - it is not clear why these are selected, and why not all the Complex set is evaluated, or even better, all VLABench, to show that there's no regression in easier cases.
-
Similarly, it is not clear why not all the tasks in EmbodiedBench were evaluated.
-
I found the analysis of "Analysis of Base-Agent Failures" not very informative. I would have preferred to see a deeper dive into the unexpected phenomenon in "Effect of the Number of Retrieved Experiences", in which more retrieved memories are not helpful. I would also like to see a similar analysis for other compared methods, e.g. ExpeL, to see if this is prevalent or only unique to TERA. For example, it could be that the currently used embedding function Phi() is not good enough, but an oracle retriever would provide much better results. I would also appreciate an analysis of the generated guidances. What is their quality? How would a prefect summarized guidance impact performance?
Questions for authors
-
See comments above.
-
I miss a running example, especially for trajectory summarizations and situation-conditioned guidance, to better see the outcomes and differences of these steps
2 comments
W7: Why does performance decrease as more memories are retrieved? Does the same phenomenon occur in ExpeL, and how do the relevance and contextual consistency of retrieved guidance affect performance?
We agree that oracle guidance could improve performance, but embodied benchmarks lack ground-truth optimal guidance, and multiple strategies may be valid. In our framework, VLMs interpret executed trajectories, summarize the encountered situations, and generate if–then guidance. This process may introduce perceptual errors that affect planning, omit relevant context due to incorrect situation recognition. To investigate why retrieving more memories can degrade performance, we provide additional analyses of how the relevance and contextual consistency of retrieved guidance relate to the performance degradation observed when more memories are retrieved.
Figure 6 (in the paper, p. 9) shows that increasing TERA retrieval from Top-2 to Top-3 reduces performance. To examine whether this phenomenon also occurs in ExpeL, we increased its insight pool by 1.5×. (since increasing TERA's retrieved memories from Top-2 to Top-3 is a 1.5× increase)
Table R2. Effect of additional experiences on EB-Manipulation (%)
| Method | Gemini-2.5-Flash | InternVL3-38B |
|---|---|---|
| Baseline | 34.2 | 19.7 |
| ExpeL (Original) | 37.72 | 21.05 |
| ExpeL (1.5× Experiences) | 37.28 | 17.11 |
| TERA Top-2 (2S+2F) | 42.10 | 26.80 |
| TERA Top-3 (3S+3F) | 39.47 | 23.68 |
Both methods show performance degradation with increased memory quantity, suggesting that simply adding more experiences does not guarantee better planning. We also added VLABench success/failure guidance examples (Figure 15a) alongside the existing EB-Manipulation example (Figure 15 in the paper). Our findings suggest that retrieved guidance quality, not merely the embedding function, is important: changing query content improves retrieval effectiveness (Figure 4 in the paper), if–then guidance outperforms summaries (Table 3 in the paper), and cross-task-type retrieval still improves performance (+4.0/+4.9 points).
Perception hallucinations by the VLM backbones used in benchmark evaluation (Figure 16, p. 27), insufficient spatial guidance (Figure 19, p. 28), and conflicting rules (Figure 17, p. 27) further illustrate these limitations. Specifically, we analyzed the additional third-ranked memories in TERA using two criteria:
(1) Current relevance: Cosine similarity below 0.80 (bottom 25% of Top-2 memories). Low-relevance rates were 47.5%/52.4% (Gemini/InternVL3) in success-to-failure episodes, versus 24.3%/28.1% in recovered episodes.
(2) Contextual similarity: Similarity of at least 0.90 to either existing memory. Rates were 67.6%/40.4% in recovered episodes, versus 32.1%/28.2% in success-to-failure episodes.
InternVL3-38B: Current relevance showed a larger difference. Low-relevance success memories occurred more often in success-to-failure episodes (63.3% vs. 25.9%). For example, when placing a triangular prism into a navy container, the agent failed to grasp the prism but proceeded to move its empty gripper above the container and release. Despite the failed grasp, the retrieved guidance advised:
- If placing an object into a container, then move the gripper well inside the container's boundary before releasing.
This guidance addressed object placement rather than the unresolved grasping failure, illustrating how irrelevant guidance can fail to correct the agent's actions.
Gemini-2.5-Flash: Contextual similarity showed a larger gap (67.6% vs. 32.1%). In a shape-sorter task, the agent grasped a lime-colored star but dropped it onto the lid instead of inserting it into the matching hole. The previously retrieved guidance correctly emphasized alignment:
- If placing an object into a matching hole, then first align the object directly above the opening before releasing.
- If inserting an object into a tight-fitting hole, then first align the object directly over the opening before lowering it into place._
However, the additional Top-3 memory introduced contextually mismatched guidance:
- If a pick-and-place action results in no change to the object's state, then attempt the task with a different object.
Rather than addressing the failed insertion, this guidance suggested switching objects, despite the original task remaining incomplete.
We find that performance degradation is associated with additional memories that are less relevant to the current situation or inconsistent with existing guidance, highlighting the importance of accurate situation summarization and contextually consistent, non-conflicting guidance over simply retrieving more memories.
Thank you for your constructive comments.
Dear Reviewer 56k1, thank you for your constructive comments. We address your concerns below.
TERA uses different prompts based on (1) success/failure outcomes and (2) benchmark execution formats, not task types (e.g., pick cube, wipe table; Appendix A.1, Figures 7–10 in the paper).
Success/Failure: Both prompts share factual summarization instructions. Success summaries emphasize effective strategies, while failure summaries additionally infer failure causes for error-avoidance guidance (Sec. 3.1, p.4).
Offline vs. Inference-time Summarization: The success/failure prompts described above are used for offline memory construction (Figures 7–10 in the paper), which summarizes completed trajectories and their outcomes. In contrast, inference-time prompts (Figures 11–12 in the paper) summarize only accumulated observations, reasoning, plans, and actions, without assuming success or failure, to retrieve experiences relevant to the current execution situation.
Benchmark-specific Formats: EB-Manipulation summarizes sequential observations, reasoning, actions, and feedback across replanning steps. VLABench summarizes initial observations, object information, CoT, and predicted actions from single-pass planning (Sec.s 3.1, 4.1). Both produce factual situation summaries for retrieval; only the input formats differ.
Embedding Function (): TERA uses
text-embedding-3-smallto embed concatenated instructions and situation summaries (Eq. 3, p.5; Appendix A.4, p.21). The same embedding function is used offline and at inference, with cosine similarity for retrieval (Eqs. 4–5, p.5).We additionally compared instruction-only and trajectory-aware retrieval using the same VLABench memory bank.
Table R1. VLABench retrieval comparison(%)
Instruction-only improves over Baseline, but TERA achieves higher Exact Match (+1.3/+1.0 for Gemini/InternVL3).
In a Book Rearrangement example, instruction-only retrieved guidance from another sorting task involving duplicate books ("verify the number of all objects") and unnecessary manipulation steps. TERA instead retrieved guidance matching the current execution, emphasizing the next book's selection order and sequential pick&place actions, successfully completing the task. This shows that instruction-only retrieval may include guidance unrelated to the current execution history, even when task goals are similar.
Our study focuses on memory construction and retrieval in visual embodied environments, rather than reproducing text-only results. Unlike ALFWorld, where scene information is provided as text, embodied VLM agents must recognize objects from images, and perception errors can propagate into planning (Sec. 1, pp.1–2).
We therefore compared memory designs within the same embodied environments: Table 3 (in the paper, p.8) examines experience sources and guidance representations; Figure 4 (in the paper, p.9) compares retrieval queries using identical memories. TERA achieved the highest performance on both backbones, demonstrating that incorporating accumulated execution information into retrieval is more effective than using only instructions or a single observation.
We selected manipulation benchmarks because they allow us to track object-state changes caused by agent actions from a fixed viewpoint and assess progress toward the goal. In robotic manipulation, the scene remains fixed, and changes arise from the agent's actions. Thus, trajectory images reveal what was attempted and its outcome, enabling factual situation summaries to serve as retrieval queries.