- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces EmbRACE, a dataset of 3,421 human-controlled egocentric demonstrations containing 48,264 steps in 55 UnrealZoo environments, and a 686-task benchmark in seven held-out environments. The tasks cover Basic navigation, Exploration, Dynamic Spatial-Semantic reference resolution, Multi-stage navigation, Open Door, and Pick & Drop. For each training step, Claude Opus 4.8 generates a rationale from the instruction, observations available before the action, and the executed action; annotators then check grounding, knowledge boundaries, and action consistency. The paper evaluates closed-source and open VLMs in a closed loop, studies SFT on four open backbones, compares Actions only against Actions plus rationale, and evaluates transfer to a physical robot and VLN-CE.
Critical strengths
1.The dataset and benchmark are carefully specified. The paper describes a four-stage collection protocol, replay checks for recorded demonstrations, explicit verification criteria for tasks and rationales, held-out evaluation environments, and success criteria that use final position and door/object state rather than action labels alone. This provides a concrete closed-loop evaluation setting spanning both navigation and interaction.
2.The empirical evaluation is broad and informative. Table 2 evaluates several frontier and open models across all six task types. Table 3 compares Actions only and Actions plus rationale on the same trajectories, reporting success-rate gains on every task type, from 6.9 to 18.3 points. Tables 4–6 further analyze blocked movement, revisits, recovery after failed interactions, physical-robot performance, and transfer to VLN-CE. Together, these experiments provide meaningful evidence that EmbRACE training is practically useful.
Critical weaknesses
- The rationale ablation demonstrates the usefulness of the rationale-augmented training setup, but it does not fully isolate the mechanism responsible for the gain. The rationales are generated by Claude Opus 4.8 after it is given the executed action (Section 3.2 and Appendix B.3). In addition, the Actions-only variant removes rationale text both from the output target and from retained assistant turns (Appendix B.5). Thus, Table 3 changes rationale semantics as well as the amount and form of autoregressively generated history. The current experiment does not determine how much of the gain is specifically due to grounded rationale content rather than additional generated text or recurrent textual state.
- The submitted manuscript does not report several details needed to quantify robustness and facilitate replication, including decoding settings and exact API/model-version details for the evaluated models, per-seed SFT variation, or uncertainty estimates. This is relevant to the physical-robot study, where Table 5 uses five fixed tasks per scene/type and five runs per task, and to Table 6, which uses 100 randomly sampled VLN-CE episodes per split. The results are promising, but the manuscript does not currently quantify their variability or identify the sampled VLN-CE episodes.
Questions for authors
1.Can the authors provide a content-controlled rationale ablation that preserves the output format and approximately matched rationale/history token budget while disrupting rationale semantics? This would help distinguish the contribution of rationale content from that of additional autoregressive text and retained textual state.
2.Please provide the decoding settings, exact model endpoints/versions, number of SFT seeds, and uncertainty estimates where available. For Table 6, please also provide the sampling seed or evaluated episode identifiers.
4 comments
Part two
W2 & Q2: Reproducibility and uncertainty estimates
We provide additional evaluation details, results from two additional training seeds, and uncertainty estimates below.
Table R2 lists the model identifiers/checkpoints, API formats, and decoding settings. Hosted models were accessed through an API gateway, and open models were self-hosted. The hosted model IDs are the identifiers used in our API requests. “API-managed” indicates that our requests omitted the temperature parameter. We used /v1/chat/completions for OpenAI Chat Completions requests and /v1/messages for Anthropic Messages requests.
The results in the main paper use seed 42. We added training runs with seeds 43 and 44 for both action-only and rationale supervision on Qwen3.5-9B. Table R3 reports the mean and standard deviation across all three seeds.
Table R4 reports 95% Wilson confidence intervals for success rates in our VLN-CE evaluation. Table R5 reports the corresponding intervals for each task type in our real-robot evaluation.
To reduce explicit route guidance in the instructions, we prioritized episodes with fewer navigation-related verbs in our VLN-CE evaluation. For R2R, we randomly sampled 100 episodes with at most two such verbs using seed 42. For RxR, we included all 20 episodes with at most two such verbs and randomly sampled another 80 from those with three or four, also using seed 42.
Table R2. Model identifiers, API formats, and decoding settings.
| Model | Model ID / checkpoint | API format | Temperature | Max output tokens |
|---|---|---|---|---|
| GPT-5.5 | gpt-5.5 | OpenAI Chat Completions | API-managed | 2048 |
| Claude Opus 4.8 | claude-opus-4-8 | Anthropic Messages | API-managed | 2048 |
| Gemini 3.1 Pro Preview | gemini-3.1-pro-preview | OpenAI Chat Completions | 1.0 | 2048 |
| Qwen3.5-2B | Qwen/Qwen3.5-2B | OpenAI Chat Completions | 0.2 | 2048 |
| Qwen3.5-9B | Qwen/Qwen3.5-9B | OpenAI Chat Completions | 0.2 | 2048 |
| InternVL3-8B | OpenGVLab/InternVL3-8B | OpenAI Chat Completions | 0.2 | 2048 |
| GLM-4.1V-9B | zai-org/GLM-4.1V-9B-Thinking | OpenAI Chat Completions | 0.2 | 2048 |
Table R3. Qwen3.5-9B success rates (%) across three training seeds, reported as mean ± SD.
| Training target | Basic | Explore | Dynamic | Multi | Open | P&D |
|---|---|---|---|---|---|---|
| Actions only | 73.6 ± 1.3 | 46.2 ± 0.9 | 49.7 ± 0.5 | 41.8 ± 0.5 | 67.3 ± 0.5 | 61.0 ± 0.9 |
| Rationale + Action | 81.0 ± 0.9 | 54.4 ± 0.5 | 66.7 ± 1.5 | 51.9 ± 0.9 | 85.9 ± 1.1 | 70.2 ± 0.8 |
Table R4. 95% Wilson confidence intervals for success rates (%) in our VLN-CE evaluation.
| Model | R2R | RxR |
|---|---|---|
| GPT-5.5 | [20.1, 37.5] | [12.5, 27.8] |
| Claude Opus 4.8 | [21.9, 39.6] | [12.5, 27.8] |
| Gemini 3.1 Pro Preview | [18.4, 35.4] | [10.9, 25.5] |
| Qwen3.5-9B (Origin) | [2.8, 12.5] | [2.2, 11.2] |
| Qwen3.5-9B (SFT) | [23.7, 41.7] | [15.0, 31.1] |
Table R5. 95% Wilson confidence intervals for success rates (%) in our real-robot evaluation.
| Model | Basic | Explore | Dynamic | Multi | P&D |
|---|---|---|---|---|---|
| GPT-5.5 | [77.2, 92.6] | [7.4, 22.8] | [23.7, 44.6] | [12.5, 30.4] | [8.4, 24.4] |
| Claude Opus 4.8 | [82.0, 95.4] | [24.9, 45.9] | [18.0, 37.6] | [18.0, 37.6] | [0.0, 4.9] |
| Gemini 3.1 Pro Preview | [85.3, 97.1] | [12.5, 30.4] | [23.7, 44.6] | [29.7, 51.3] | [7.4, 22.8] |
| Qwen3.5-9B (Origin) | [13.6, 31.9] | [0.0, 4.9] | [0.0, 4.9] | [0.0, 4.9] | [0.0, 4.9] |
| Qwen3.5-9B (SFT) | [95.1, 100.0] | [71.1, 88.5] | [85.3, 97.1] | [77.2, 92.6] | [55.4, 76.3] |
| Qwen3.5-9B (SFT on R2R-CoT) | [77.2, 92.6] | [2.9, 14.7] | [0.0, 4.9] | [8.4, 24.4] | N/A |
Thank you for the detailed response and clarifications. Based on the rebuttal and the original paper, I believe my main concerns have been adequately addressed and the novelty of the work is better supported, so I am willing to raise my score.
Thank you for your careful consideration and for raising your rating to Weak Accept. We are glad that the additional evidence helped resolve your concerns. Your feedback has helped us strengthen the paper, and we will incorporate these results into the revision.
Part one
W1 & Q1: Content-controlled rationale ablations
Thank you for the constructive suggestion. We added a shuffled-rationale control and additional ablations to distinguish the contribution of rationale content from additional generated text and retained textual history (Table R1).
In row 3, we replace each training rationale with one from another trajectory while approximately matching token length, and apply the same replacement to its occurrences in the training history. Visual observations, action labels, and output format remain unchanged. The policy generates text before acting and retains it in history as in row 5. Comparing rows 3 and 5, shuffled rationales substantially reduce success across all six task types. Comparing rows 1 and 3 further shows that training with shuffled rationales substantially degrades performance relative to action-only. A possible explanation is that mismatched rationales introduce misleading textual context for action prediction during training, interfering with learning the correspondence between observations and actions.
Rows 1 and 4 use the same demonstrations, visual observations, and action history, without retaining past rationale text. Row 4 adds supervision and generation of a rationale before the action. It achieves most of the improvement observed in row 5 over action-only, showing the contribution of rationales to action selection beyond retained textual history. Comparing rows 4 and 5 further shows that retaining past rationales provides modest additional gains on Explore, Dynamic, and Multi.
In row 2, rationale supervision is replaced with descriptions of the current view. These descriptions are generated by Claude Opus 4.8, the same model used for rationale annotation, with instructions to keep their length close to that of the rationales. The policy generates the descriptions before acting and retains them in history as in row 5. Comparing rows 2 and 5, rationales substantially outperform descriptive text across all six task types. Comparing rows 1 and 2 further shows that descriptive text reduces performance. A possible explanation is that these descriptions become part of the context for action prediction without explicitly organizing the visual information around the decision to be made, introducing distraction rather than useful guidance.
These comparisons support the value of rationale content beyond simply generating and retaining additional text.
Table R1. Qwen3.5-9B success rates (%) under rationale ablations.