- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ERAct is a retrieval-augmented framework for embodied planning that encodes past experience as step-level rationales. Its Semantic Experience Rationale Retriever (SERR) selects relevant experiences based on the current instruction and preceding rationale. The Rationale-Informed Fusion Network (RIFN) integrates the retrieved rationale features with textual and visual inputs through separate alignment pathways and gated residual fusion. Training proceeds in three stages: learning from expert trajectories, instruction augmentation, and experience-augmented training. On EB-ALFRED, ERAct achieves 65.0% average success, compared with 61.7% for ExpRAG and 63.3% without RIFN. A comparison between rationale-based and action-based experience further suggests that how experience is represented influences its utility for planning.
Critical strengths
-
The rationale-versus-action comparison provides useful evidence about experience representation. Table 3 shows that retrieved rationales improve the rationale-trained model from 60.7% to 65.0%, whereas action-only experience reduces its corresponding model’s performance from 62.7% to 60.7%. This comparison supports the idea that the content and organization of experience matter, rather than retrieval being inherently beneficial.
-
The architecture offers a coherent mechanism for context-dependent experience reuse. SERR conditions retrieval on the instruction and preceding rationale, making experience selection sensitive to inferred task progress. RIFN then uses separate textual and visual pathways to integrate retrieved information with the current situation. This is a reasonable design for addressing mismatches between historical experience and ongoing execution.
-
The evaluation explicitly separates the experience bank from evaluation tasks. Section 4.1 states that training data and stored experiences come exclusively from the ALFRED training split. The authors also re-evaluate RoboAgent without auxiliary action-success feedback and construct ExpRAG using the same backbone, training data, and experience bank. These choices improve the relevance of the comparisons to the intended evaluation setting.
-
The reported results show positive aggregate gains and include several component analyses. ERAct achieves the highest reported average success in Table 1, with 65.0% compared with 61.7% for ExpRAG and 63.0% for RoboAgent. Table 2 additionally examines the dual-path design, selector, and complete fusion module. Although the gains require stronger uncertainty analysis, these experiments provide preliminary evidence that the proposed components are useful in combination.
Critical weaknesses
-
The experiments do not isolate reasoning transfer from richer descriptive information. The rationales include observations, task progress, and action explanations, and are generated with the executed action supplied to the annotator (Appendix A.2). Their advantage over action-only experience could therefore arise from additional state information or action-related cues rather than transferable reasoning specifically. A comparison against state-and-progress descriptions without action justifications, using matched retrieval and input budgets, would better test the central claim.
-
The improvements over controlled alternatives are modest and lack uncertainty estimates. ERAct improves over the same-data “w/o RIFN” variant by only 1.7 percentage points and over ExpRAG by 3.3 points. The manuscript does not report independent-run variation or confidence intervals. Moreover, removing the dual-path design improves Visual and Spatial success from 50% to 54% and from 46% to 52%, respectively. The average gain is encouraging, but its reliability and the claimed visual-grounding benefit remain insufficiently established.
-
The analysis provides limited support for the proposed retrieval and confidence mechanisms. Figure 3 reports correlations of −0.058 and 0.061, both close to zero, despite being used to motivate rationale relevance as a more useful retrieval signal. Its vertical axis is labeled “Task Progress,” while the caption refers to success rate. Qualitative attention maps do not resolve this ambiguity or establish that the confidence gate suppresses unreliable experience. Controlled tests with irrelevant or misleading retrieval, together with measurements of gate responses and task performance, would provide more direct evidence.
-
Important implementation and comparison details remain unclear. Equation 10 introduces G without defining its relationship to S, and the activation in Equation 9 is unspecified. Retrieval K, confidence top-k, temperature, the rationale-generation model, and the construction of Stage 3 trajectories are not clearly documented. The exact retrieval content and context limits for ExpRAG are also needed to assess whether the comparison isolates fusion. In addition, Table 4 reports a Stage 1 average of 44.3%, whereas the six listed category scores average to 36.0%; this discrepancy should be corrected or explained.
Questions for authors
-
Can you compare full rationales with state-and-progress descriptions that omit the action justification, while holding retrieved instances, fusion architecture, and input budgets fixed? This would help distinguish reasoning content from richer descriptive information. How does SERR compare with instruction-only retrieval under the same planner?
-
What exactly is held fixed in the ExpRAG and action-trajectory comparisons: retrieval keys, retrieved instance identities, rationale/action content, training targets, and context lengths? For the “w/o RIFN” variant, are retrieved experiences absent entirely or supplied through another interface?
-
Please provide evaluation counts, run counts, decoding settings, and uncertainty for the main comparisons, particularly the 65.0% versus 63.3% result. Are the improvements consistent across independent training runs? Please also reconcile the dual-path visual-grounding explanation with the higher Visual and Spatial scores obtained without that component.
-
Please define G in Equation 10 and its relationship to S, specify the activation in Equation 9 and the relevant hyperparameters, and explain how the gate is validated under irrelevant retrieval. Which model generates the rationales, how are the Stage 3 trajectories synthesized, and are retrieval matches from the same underlying source trajectory excluded during training?
5 comments
Response to Reviewer 3 (Part 2)
Response to Weakness 3.1: Fig. 3 correlations close to zero.
We thank the reviewer for this comment. Figure 3 originally presents scatter plots of task progress against the retrieval scores, together with the corresponding Spearman's . Since a task is considered successful only when task progress reaches 1, we further partition the scores into six bins and compute the success rate within each bin. We then analyze the relationship between success rate and the corresponding retrieval scores. The Spearman's between success rate and trajectory similarity is -0.87, whereas that between success rate and rationale relevance is 0.348. These results suggest that rationale relevance provides a more useful retrieval signal. We have revised the description and visualization in Figure 3 accordingly.
Response to Weakness 3.2: do not establish that the confidence gate suppresses unreliable experience.
That is an excellent question. In ERAct, the selector filters mismatched reasoning cues, while the confidence mechanism adaptively controls the strength of experience injection. To validate these mechanisms, we construct a controlled degraded-retrieval setting by reducing the experience bank to one-third of its original size and replacing SERR with instruction-only retrieval. The results show that both fine-grained relevance selection and the confidence gate provide additional robustness by suppressing unreliable experience.
| Method | Avg | Base | Common | Complex | Visual | Spatial | Long |
|---|---|---|---|---|---|---|---|
| ERAct | 63.7 | 74 | 76 | 68 | 48 | 48 | 68 |
| w/o Confidence | 63.3 | 78 | 64 | 72 | 52 | 50 | 64 |
| w/o Selector | 61.7 | 78 | 60 | 74 | 52 | 42 | 64 |
These controlled tests have been added to Sec. A.6.
Response to Weakness 4 & Question 4: Important implementation and comparison details remain unclear.
Thank you for your careful attention to detail, specifically:
-
We split Equation 9 to explicitly define the relationship between and (see Sec. 3.3.2), and clarify that the activation function used in Equation 9 is the sigmoid function.
-
The retrieval parameter is set to 3, and the confidence Top- ratio is set to 20%. During inference, we use deterministic decoding with do_sample=False. The rationale-generation model is Qwen2.5-VL-72B-Instruct. Details of the Stage 3 trajectory construction are provided in Sec. A.1.2.
-
ExpRAG and ERAct use the same SERR module and experience bank, with identical retrieval settings. The inputs are not truncated for either method, and the maximum output length is set to 1024 tokens for both.
-
You are correct; Table 4 has a typo in the manuscript, and we have corrected it in the revised paper.
-
Under irrelevant retrieval, the confidence gate adaptively suppresses the strength of experience injection, thereby improving the robustness of RIFN (see Sec. A.6).
-
Retrieval matches originating from the same source trajectory are not excluded during training. Since our experience bank is organized at the step level, with each instance represented as , different steps from the same trajectory correspond to different task progress, reasoning contexts, and decisions, and therefore provide distinct experience information.
Response to Question 1.2: Ablation of SERR
We thank the reviewer for this insightful comment. We have added the ablation studies of SERR in the appendix Sec. A.5. Please see our response for Reviewer 2 Weakness 2.1. The results show that SERR achieves higher average performance than instruction-only retrieval across different experience bank sizes, supporting its effectiveness in retrieving experiences with more relevant underlying reasoning.
Response to Question 2: Details of Variant
We thank the reviewer for this question, which allows us to further clarify the comparisons. Specifically:
-
ExpRAG is kept consistent with ERAct in terms of the retrieval setup.
-
In contrast, the action-trajectory variant removes rationales from the retrieval keys, retrieved instance content: the retrieval key is defined as $ k_i = \pi_e([l_{\tau_i};a_{i-1}]) $ and the retrieved instance is $ m_t = (l_\tau,k_t,a_t) $.
-
For the w/o RIFN variant, retrieved experiences are entirely removed, and the model internalizes the knowledge into its parameters during training.
Thank you for your time and valuable feedback! As the author–reviewer discussion period will end in about one days, we kindly ask you to review our rebuttal and let us know whether it addresses your concerns.
Response to Reviewer 3 (Part 1)
Thank you for your constructive feedback and recognition. Below are our responses, which we hope will address your concerns.
The experience rationales in ERAct contain two complementary types of reasoning content: State Description and Action Reasoning.
We would first like to clarify that the state descriptions contained in retrieved rationales do not provide additional factual information about the current test state. They describe the task progress and environmental state of past ALFRED experiences, whereas evaluation is conducted in different EB-ALFRED environments. Therefore, the utility of both components depends on whether their underlying reasoning patterns can be transferred and re-grounded to the current context.
Following the reviewer’s suggestion, we further add a controlled experiment to disentangle the different components of the rationale. The results show that state-oriented and action-oriented reasoning provide complementary transferable cues, and that the advantage of full rationales cannot be attributed simply to richer descriptive information.
The ablation of experience rationales has been added to Sec. A.4.1.
EB-ALFRED contains 300 evaluation tasks. We additionally conduct three independent training runs for both ERAct and the w/o RIFN variant and report the variation. During inference, we use deterministic greedy decoding with do_sample=False.
These results show that ERAct consistently outperforms the w/o RIFN variant across all three runs.
We thank the reviewer for these valid points.
First, we would like to clarify that the Visual and Spatial subsets mainly evaluate whether the agent can plan actions based on textual descriptions of environmental attributes or spatial relations, such as “pick up a yellow object” and “put the plunger under the sink.” Under these settings, ERAct (w/o dual) can perform better.
In contrast, the visual-grounding capability discussed in our paper refers specifically to identifying whether a target object is present in the current visual observation. To quantitatively evaluate this capability, we conduct two additional analyses:
We separately evaluate tasks in the EB-ALFRED test set where the agent must recognize that the target object is absent from the current observation and subsequently locate an alternative object_2 to recover from action failure. ERAct achieves a success rate of 50%, compared with 30% for ERAct (w/o dual).
We further sample 220 images from FUNTHOR [2] to evaluate the model’s ability to recognize whether a target object is present in the image. ERAct achieves a success rate of 33.64%, whereas ERAct (w/o dual) achieves 21.82%.
These results support that separately modeling textual and visual features helps reduce entangled interference from cues that are semantically relevant but visually mismatched, thereby improving visual grounding.
[2] Fu Z, Zurbrügg R, Qu K, et al. Funfact: building probabilistic functional 3d scene graphs via factor-graph reasoning[J]. arXiv preprint arXiv:2604.03696, 2026.