acceptodds
Under review as a conference paper at ICLR 2027

Embodied Reasoning and Action in Complex Environments

Abstract

Vision-language models are increasingly run as embodied agents in a closed loop in which some decisions depend on what the agent saw earlier. Placed in this loop, current models act on the view in front of them, revisit views already searched, lose a target that has left the frame, and switch referents as the viewpoint changes. The demonstrations that embodied models learn from record the view and the action at each step, not the observations the action was based on. We present EmbRACE, a dataset of 3,421 human demonstrations recorded from the egocentric view in photorealistic Unreal Engine environments, indoor and outdoor, on tasks with targets to search for, relational and ordered targets, and door and object interactions. Each of its 48,264 steps carries a rationale that states what the action was based on and is verified against the frames. A benchmark of 686 tasks in 7 held-out environments runs models in the closed loop and scores them on the final position and on the door and object states. Frontier models score highest when the target is in view from the start, lower on all three task types that depend on an earlier observation, and below half when the target has to be searched for or returned to. Fine-tuning on EmbRACE takes four open models of 2B to 9B parameters from 3% to 8% overall to 57% to 68%. Relative to training on the same trajectories without them, the rationales raise success by 7% to 18% across task types and halve the fraction of episodes that return to views already seen. The dataset, the benchmark, and the code will be made publicly available.

Primary area:
datasets and benchmarks
Keywords:
Embodied AI, Vision-Language Models, Embodied Reasoning, Vision-and-Language Navigation, Human Demonstrations, Step-wise Rationales, Closed-loop Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_XbZsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EmbRACE, a dataset of 3,421 human-controlled egocentric demonstrations containing 48,264 steps in 55 UnrealZoo environments, and a 686-task benchmark in seven held-out environments. The tasks cover Basic navigation, Exploration, Dynamic Spatial-Semantic reference resolution, Multi-stage navigation, Open Door, and Pick & Drop. For each training step, Claude Opus 4.8 generates a rationale from the instruction, observations available before the action, and the executed action; annotators then check grounding, knowledge boundaries, and action consistency. The paper evaluates closed-source and open VLMs in a closed loop, studies SFT on four open backbones, compares Actions only against Actions plus rationale, and evaluates transfer to a physical robot and VLN-CE.

Critical strengths

1.The dataset and benchmark are carefully specified. The paper describes a four-stage collection protocol, replay checks for recorded demonstrations, explicit verification criteria for tasks and rationales, held-out evaluation environments, and success criteria that use final position and door/object state rather than action labels alone. This provides a concrete closed-loop evaluation setting spanning both navigation and interaction.

2.The empirical evaluation is broad and informative. Table 2 evaluates several frontier and open models across all six task types. Table 3 compares Actions only and Actions plus rationale on the same trajectories, reporting success-rate gains on every task type, from 6.9 to 18.3 points. Tables 4–6 further analyze blocked movement, revisits, recovery after failed interactions, physical-robot performance, and transfer to VLN-CE. Together, these experiments provide meaningful evidence that EmbRACE training is practically useful.

Critical weaknesses
  1. The rationale ablation demonstrates the usefulness of the rationale-augmented training setup, but it does not fully isolate the mechanism responsible for the gain. The rationales are generated by Claude Opus 4.8 after it is given the executed action (Section 3.2 and Appendix B.3). In addition, the Actions-only variant removes rationale text both from the output target and from retained assistant turns (Appendix B.5). Thus, Table 3 changes rationale semantics as well as the amount and form of autoregressively generated history. The current experiment does not determine how much of the gain is specifically due to grounded rationale content rather than additional generated text or recurrent textual state.
  2. The submitted manuscript does not report several details needed to quantify robustness and facilitate replication, including decoding settings and exact API/model-version details for the evaluated models, per-seed SFT variation, or uncertainty estimates. This is relevant to the physical-robot study, where Table 5 uses five fixed tasks per scene/type and five runs per task, and to Table 6, which uses 100 randomly sampled VLN-CE episodes per split. The results are promising, but the manuscript does not currently quantify their variability or identify the sampled VLN-CE episodes.
Questions for authors

1.Can the authors provide a content-controlled rationale ablation that preserves the output format and approximately matched rationale/history token budget while disrupting rationale semantics? This would help distinguish the contribution of rationale content from that of additional autoregressive text and retained textual state.

2.Please provide the decoding settings, exact model endpoints/versions, number of SFT seeds, and uncertainty estimates where available. For Table 6, please also provide the sampling seed or evaluated episode identifiers.

4 comments

W1 & Q1: Content-controlled rationale ablations

Thank you for the constructive suggestion. We added a shuffled-rationale control and additional ablations to distinguish the contribution of rationale content from additional generated text and retained textual history (Table R1).

In row 3, we replace each training rationale with one from another trajectory while approximately matching token length, and apply the same replacement to its occurrences in the training history. Visual observations, action labels, and output format remain unchanged. The policy generates text before acting and retains it in history as in row 5. Comparing rows 3 and 5, shuffled rationales substantially reduce success across all six task types. Comparing rows 1 and 3 further shows that training with shuffled rationales substantially degrades performance relative to action-only. A possible explanation is that mismatched rationales introduce misleading textual context for action prediction during training, interfering with learning the correspondence between observations and actions.

Rows 1 and 4 use the same demonstrations, visual observations, and action history, without retaining past rationale text. Row 4 adds supervision and generation of a rationale before the action. It achieves most of the improvement observed in row 5 over action-only, showing the contribution of rationales to action selection beyond retained textual history. Comparing rows 4 and 5 further shows that retaining past rationales provides modest additional gains on Explore, Dynamic, and Multi.

In row 2, rationale supervision is replaced with descriptions of the current view. These descriptions are generated by Claude Opus 4.8, the same model used for rationale annotation, with instructions to keep their length close to that of the rationales. The policy generates the descriptions before acting and retains them in history as in row 5. Comparing rows 2 and 5, rationales substantially outperform descriptive text across all six task types. Comparing rows 1 and 2 further shows that descriptive text reduces performance. A possible explanation is that these descriptions become part of the context for action prediction without explicitly organizing the visual information around the decision to be made, introducing distraction rather than useful guidance.

These comparisons support the value of rationale content beyond simply generating and retaining additional text.

Table R1. Qwen3.5-9B success rates (%) under rationale ablations.

Training targetText kept in historyBasicExploreDynamicMultiOpenP&D
ActionN/A73.345.350.042.567.062.1
Descriptive text + Action✓71.640.243.037.764.256.5
Shuffled rationale + Action✓21.611.113.210.415.612.1
Rationale + Action✗80.252.164.950.983.568.5
Rationale + Action✓80.254.767.552.885.369.4

W2 & Q2: Reproducibility and uncertainty estimates

We provide additional evaluation details, results from two additional training seeds, and uncertainty estimates below.

Table R2 lists the model identifiers/checkpoints, API formats, and decoding settings. Hosted models were accessed through an API gateway, and open models were self-hosted. The hosted model IDs are the identifiers used in our API requests. “API-managed” indicates that our requests omitted the temperature parameter. We used /v1/chat/completions for OpenAI Chat Completions requests and /v1/messages for Anthropic Messages requests.

The results in the main paper use seed 42. We added training runs with seeds 43 and 44 for both action-only and rationale supervision on Qwen3.5-9B. Table R3 reports the mean and standard deviation across all three seeds.

Table R4 reports 95% Wilson confidence intervals for success rates in our VLN-CE evaluation. Table R5 reports the corresponding intervals for each task type in our real-robot evaluation.

To reduce explicit route guidance in the instructions, we prioritized episodes with fewer navigation-related verbs in our VLN-CE evaluation. For R2R, we randomly sampled 100 episodes with at most two such verbs using seed 42. For RxR, we included all 20 episodes with at most two such verbs and randomly sampled another 80 from those with three or four, also using seed 42.

Table R2. Model identifiers, API formats, and decoding settings.

ModelModel ID / checkpointAPI formatTemperatureMax output tokens
GPT-5.5gpt-5.5OpenAI Chat CompletionsAPI-managed2048
Claude Opus 4.8claude-opus-4-8Anthropic MessagesAPI-managed2048
Gemini 3.1 Pro Previewgemini-3.1-pro-previewOpenAI Chat Completions1.02048
Qwen3.5-2BQwen/Qwen3.5-2BOpenAI Chat Completions0.22048
Qwen3.5-9BQwen/Qwen3.5-9BOpenAI Chat Completions0.22048
InternVL3-8BOpenGVLab/InternVL3-8BOpenAI Chat Completions0.22048
GLM-4.1V-9Bzai-org/GLM-4.1V-9B-ThinkingOpenAI Chat Completions0.22048

Table R3. Qwen3.5-9B success rates (%) across three training seeds, reported as mean ± SD.

Training targetBasicExploreDynamicMultiOpenP&D
Actions only73.6 ± 1.346.2 ± 0.949.7 ± 0.541.8 ± 0.567.3 ± 0.561.0 ± 0.9
Rationale + Action81.0 ± 0.954.4 ± 0.566.7 ± 1.551.9 ± 0.985.9 ± 1.170.2 ± 0.8

Table R4. 95% Wilson confidence intervals for success rates (%) in our VLN-CE evaluation.

ModelR2RRxR
GPT-5.5[20.1, 37.5][12.5, 27.8]
Claude Opus 4.8[21.9, 39.6][12.5, 27.8]
Gemini 3.1 Pro Preview[18.4, 35.4][10.9, 25.5]
Qwen3.5-9B (Origin)[2.8, 12.5][2.2, 11.2]
Qwen3.5-9B (SFT)[23.7, 41.7][15.0, 31.1]

Table R5. 95% Wilson confidence intervals for success rates (%) in our real-robot evaluation.

ModelBasicExploreDynamicMultiP&D
GPT-5.5[77.2, 92.6][7.4, 22.8][23.7, 44.6][12.5, 30.4][8.4, 24.4]
Claude Opus 4.8[82.0, 95.4][24.9, 45.9][18.0, 37.6][18.0, 37.6][0.0, 4.9]
Gemini 3.1 Pro Preview[85.3, 97.1][12.5, 30.4][23.7, 44.6][29.7, 51.3][7.4, 22.8]
Qwen3.5-9B (Origin)[13.6, 31.9][0.0, 4.9][0.0, 4.9][0.0, 4.9][0.0, 4.9]
Qwen3.5-9B (SFT)[95.1, 100.0][71.1, 88.5][85.3, 97.1][77.2, 92.6][55.4, 76.3]
Qwen3.5-9B (SFT on R2R-CoT)[77.2, 92.6][2.9, 14.7][0.0, 4.9][8.4, 24.4]N/A

Thank you for the detailed response and clarifications. Based on the rebuttal and the original paper, I believe my main concerns have been adequately addressed and the novelty of the work is better supported, so I am willing to raise my score.

Thank you for your careful consideration and for raising your rating to Weak Accept. We are glad that the additional evidence helped resolve your concerns. Your feedback has helped us strengthen the paper, and we will incorporate these results into the revision.

Reviewer_WACDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors introduce EmbRACE, a dataset containing 3,421 human demonstrations with LLM-generated, human-verified rationales. The authors also construct a benchmark of 686 tasks in seven held-out environments. The experiments show substantial improvements after fine-tuning four open models, with additional evaluations on a physical robot and VLN-CE.

Critical strengths
  1. The data collection and verification procedures are carefully designed. The task definitions and evaluation protocol are clearly presented.

  2. The experimental coverage is relatively comprehensive, including an actions-only ablation, behavioral analysis, and transfer experiments. The consistent fine-tuning gains suggest that the dataset is useful.

Critical weaknesses
  1. The interpretation of the rationales needs clarification. They are generated after observing the demonstrated action, rather than recorded from the human demonstrator. Human verification establishes consistency with the observations and action, but does not establish that the explanation captures the demonstrator’s actual reason. Some descriptions of what the dataset records are therefore stronger than the collection procedure supports.

  2. The rationale ablation does not isolate the value of grounded explanations. According to Appendix B.5, Actions-only removes rationales from both the output and retained interaction history. The comparison therefore changes supervision, textual history, and generation together. A current-view-only rationale baseline and length-matched descriptive text would help explain the gains. The rationale-specific experiments are also limited to Qwen3.5-9B.

  3. The generalization evidence has limitations. None of the seven held-out environments is purely indoor, and the task types remain shared with training. The transfer experiments are useful, but uncertainty estimates are missing, particularly for the small VLN-CE evaluation and repeated robot trials.

Questions for authors
  • Does the rationale advantage remain when textual history and generation length are controlled?

  • Can you report results on held-out indoor environments and variability across training runs?

2 comments

W1: Interpretation of the rationales

A rationale need not reproduce the demonstrator’s actual reasoning to provide useful supervision. A demonstrated action can admit more than one observation-grounded justification, and our annotation provides one such justification as a training target. Model-generated rationale supervision is also used in related corpora. ECoT and EMMA-X annotate human-teleoperated Bridge V2 demonstrations through automated annotation pipelines. R2R-CoT-320k uses Qwen2.5-VL-72B, Nav-CoT-110K uses Gemini 2.5 Pro, the TBA-CoT set of OctoNav-Bench uses Qwen-VL and DeepSeek-R1, and Embodied-Reasoner uses GPT-4o.

Our rationales are generated retrospectively from the instruction, observations available up to each step, and demonstrated action. Annotators verify every rationale for visual grounding, knowledge boundaries, and action consistency. The explanation must be supported by the available observations, assert nothing beyond what the demonstrator could have known at that step, and agree with the executed action. For example, knowledge-boundary violations were flagged in 20% of training trajectories on the first pass, triggering regeneration and rechecking (Section 3.2 of the main paper). These checks assess the grounding and consistency of the explanation, without establishing that it captures the demonstrator’s actual reason.

To make this distinction explicit, we will replace “states what the action was based on” with “relates the available observations to the demonstrated action” in the abstract, introduction, and conclusion.


W2 & Q1: Additional rationale ablations

Thank you for the constructive suggestion. We added the requested ablations to better isolate the contribution of rationale content (Table R1).

For Qwen3.5-9B, rows 1 and 3 use the same demonstrations, visual observations, and action history, without retaining past rationale text. Row 3 adds supervision and generation of a rationale before the action. It achieves most of the improvement observed in row 4 over action-only, showing the contribution of rationales to action selection beyond retained textual history. Comparing rows 3 and 4 further shows that retaining past rationales provides modest additional gains on Explore, Dynamic, and Multi.

In row 2, rationale supervision is replaced with descriptions of the current view. These descriptions are generated by Claude Opus 4.8, the same model used for rationale annotation, with instructions to keep their length close to that of the rationales. The policy generates the descriptions before acting and retains them in history as in row 4. Comparing rows 2 and 4, rationales substantially outperform descriptive text across all six task types. Comparing rows 1 and 2 further shows that descriptive text reduces performance. A possible explanation is that these descriptions become part of the context for action prediction without explicitly organizing the visual information around the decision to be made, introducing distraction rather than useful guidance.

These comparisons support the value of rationale content beyond simply generating and retaining additional text. InternVL3-8B and GLM-4.1V-9B show similar trends.

Table R1. Success rate (%) on EmbRACE.

BackboneTraining targetText kept in historyBasicExploreDynamicMultiOpenP&D
Qwen3.5-9BActionN/A73.345.350.042.567.062.1
Qwen3.5-9BDescriptive text + Action✓71.640.243.037.764.256.5
Qwen3.5-9BRationale + Action✗80.252.164.950.983.568.5
Qwen3.5-9BRationale + Action✓80.254.767.552.885.369.4
InternVL3-8BActionN/A65.536.843.936.860.647.6
InternVL3-8BDescriptive text + Action✓64.729.136.831.157.842.7
InternVL3-8BRationale + Action✗75.947.061.442.575.254.8
InternVL3-8BRationale + Action✓76.749.664.045.375.256.5
GLM-4.1V-9BActionN/A63.835.943.942.566.160.5
GLM-4.1V-9BDescriptive text + Action✓62.127.436.835.863.355.6
GLM-4.1V-9BRationale + Action✗76.744.463.251.983.565.3
GLM-4.1V-9BRationale + Action✓75.946.266.753.884.467.7

W3 & Q2: Indoor generalization and uncertainty estimates

Thank you for the constructive suggestion. We added evaluations on held-out indoor environments, repeated fine-tuning with two additional random seeds, and computed uncertainty estimates for the transfer experiments.

Open Door tasks in our evaluation use doors connecting indoor and outdoor spaces. The three additional purely indoor environments do not contain these doors, so we evaluate the remaining five task types, with 30 tasks per type and 150 tasks in total. For Qwen3.5-9B, EmbRACE fine-tuning raises overall success from 8.7% to 68.0%, with improvements across all five task types (Table R2). InternVL3-8B and GLM-4.1V-9B show the same trend.

Our generalization evaluation focuses on unseen environments while retaining the same task types, consistent with the environment-based evaluation used in R2R and RxR.

The results in the main paper use seed 42. We added training runs with seeds 43 and 44 for both action-only and rationale supervision on Qwen3.5-9B. Table R3 reports the mean and standard deviation across all three seeds.

Table R4 reports 95% Wilson confidence intervals for success rates in our VLN-CE evaluation. Table R5 reports the corresponding intervals for each task type in our real-robot evaluation.

Table R2. Success rates (%) in three held-out indoor environments: hangar, qa_Holding_Cells_A, and schoolGymDay.

ModelBasicExploreDynamicMultiP&D
Random0.00.00.00.00.0
Human100.0100.0100.0100.0100.0
GPT-5.573.343.350.046.750.0
Claude Opus 4.873.346.746.753.346.7
Gemini 3.1 Pro Preview66.753.350.056.756.7
Qwen3.5-9B (Origin)16.76.73.310.06.7
InternVL3-8B (Origin)16.73.30.03.30.0
GLM-4.1V-9B (Origin)20.06.70.06.73.3
Qwen3.5-9B (SFT)73.360.066.763.376.7
InternVL3-8B (SFT)70.053.356.750.060.0
GLM-4.1V-9B (SFT)76.760.060.056.770.0

Table R3. Qwen3.5-9B success rates (%) across three training seeds, reported as mean ± SD.

Training targetBasicExploreDynamicMultiOpenP&D
Actions only73.6 ± 1.346.2 ± 0.949.7 ± 0.541.8 ± 0.567.3 ± 0.561.0 ± 0.9
Rationale + Action81.0 ± 0.954.4 ± 0.566.7 ± 1.551.9 ± 0.985.9 ± 1.170.2 ± 0.8

Table R4. 95% Wilson confidence intervals for success rates (%) in our VLN-CE evaluation.

ModelR2RRxR
GPT-5.5[20.1, 37.5][12.5, 27.8]
Claude Opus 4.8[21.9, 39.6][12.5, 27.8]
Gemini 3.1 Pro Preview[18.4, 35.4][10.9, 25.5]
Qwen3.5-9B (Origin)[2.8, 12.5][2.2, 11.2]
Qwen3.5-9B (SFT)[23.7, 41.7][15.0, 31.1]

Table R5. 95% Wilson confidence intervals for success rates (%) in our real-robot evaluation.

ModelBasicExploreDynamicMultiP&D
GPT-5.5[77.2, 92.6][7.4, 22.8][23.7, 44.6][12.5, 30.4][8.4, 24.4]
Claude Opus 4.8[82.0, 95.4][24.9, 45.9][18.0, 37.6][18.0, 37.6][0.0, 4.9]
Gemini 3.1 Pro Preview[85.3, 97.1][12.5, 30.4][23.7, 44.6][29.7, 51.3][7.4, 22.8]
Qwen3.5-9B (Origin)[13.6, 31.9][0.0, 4.9][0.0, 4.9][0.0, 4.9][0.0, 4.9]
Qwen3.5-9B (SFT)[95.1, 100.0][71.1, 88.5][85.3, 97.1][77.2, 92.6][55.4, 76.3]
Qwen3.5-9B (SFT on R2R-CoT)[77.2, 92.6][2.9, 14.7][0.0, 4.9][8.4, 24.4]N/A
Reviewer_xVTpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a benchmark and a dataset to evaluate and train vlm to perform actions such as navigation and pick-and-drop in simulation environment. The finding is that tht frontier models score highest when the target is in view from the start, lower on all three task types that depend on an earlier observation, and below half when the target has to be searched for or returned to. Fine-tuning on the proposed dataset improves results significantly. Training with rationales raise success by 7% to 18% across task. Real hardware experiments are included, demonstrating that the performance gain from finetuning in simulation can transfer to real robots.

Critical strengths
  1. The paper shows that rationale supervision improves downstream closed-loop behavior beyond ordinary action-only behavioral cloning.
  2. Performance gains from training on simulated demonstrations are shown to transfer to real robots and existing navigation benchmarks of real building scans.
  3. The dataset construction is reasonably systematic, and the authors provide extensive evaluations, including rationale ablations, cross-benchmark transfer, and real-robot experiments
Critical weaknesses
  1. The main weakness is limited novelty. As stated in Section 2 and Table 2, the components present in the paper, such as embodied tasks and rationale, are already largely included in existing work. It is not sufficiently clear what new capabilities or scientific questions this combination enables that existing benchmarks cannot address
  2. While the rationale ablation shows meaningful performance improvements, it does not establish that explicit language reasoning is necessary for better action selection. The comparison simultaneously changes the training supervision and inference procedure. This is particularly important given prior findings that generating reasoning at inference time can degrade navigation performance [a] [a] Aux-Think: Exploring Reasoning Strategies for Data-Efficient Vision-Language Navigation. arXiv:2505.11886
  3. The rationales are generated retrospectively, conditioned on the actions already selected by human demonstrators. These explanations are not necessarily faithful representations of the reasoning underlying the demonstrated actions.
  4. The complexity of the evaluated tasks raises questions about whether a large VLM is necessary. In particular, the benchmark primarily involves discrete navigation actions and simplified object interactions. Comparison to a lightweight imitation-learning baseline would substantially strengthen the conclusions. The task construction also appears to limit the extent of exploration and long-term memory required, which weakens the connection to the paper's motivation
Questions for authors
  1. How would an imitation learning baseline perform on the tasks studied in the paper compared to the VLM-based approach?
  2. Is the gain from rationale as a supervision signal or from reasoning in inference?
4 comments

W1: Novelty

Thank you for the careful reading. EmbRACE provides paired action and rationale supervision for navigation and object interaction under changing egocentric observations. Human demonstrators select actions through the agent’s egocentric view. Rationales are generated by a model from the instruction, observations available up to each step, and demonstrated action, then verified by human annotators for visual grounding, knowledge boundaries, and action consistency. These pairs provide explicit supervision for using observation history to decide where to search, pursue targets that have left the frame, and maintain referents as the agent’s movement changes its view.

The construction of the demonstrations determines which decisions receive this supervision. R2R-CoT-320k uses oracle trajectories computed from scene geometry. Nav-CoT-110K’s released R2R/RxR pipeline derives actions from supplied paths, which provide privileged route information beyond the agent’s observation history. OctoNav-Bench constructs trajectories using navigable geometry and sampled endpoints. Such action construction bypasses the need to resolve from observation history where to search or how to continue toward a target that has left the frame. EmbRACE records human choices made from the observations available to the agent and pairs these decisions with stepwise rationales.

Embodied-Reasoner’s object-level navigation actions encapsulate the intervening movement, so movement and turning decisions within each such action are not separately paired with rationales. ECoT and EMMA-X annotate human-teleoperated manipulation under a fixed third-person camera. EmbRACE provides stepwise rationale supervision for individual navigation and interaction actions as the agent’s own motion changes its viewpoint, including decisions about pursuing targets and maintaining referents across views.

The task categories reveal capability gaps beyond reaching a visible target. They distinguish initial target visibility, reference maintenance across viewpoints, and sequential goals alongside interaction tasks. Table 5 in the main paper compares EmbRACE and R2R-CoT training using the same Qwen3.5-9B and training recipe. Across robot scenes, R2R-CoT training achieves 80–100% success on Basic, but only 0–20% on Exploration and 0% on Dynamic Spatial-Semantic. EmbRACE training reaches 80–84% and 80–100% on the latter two types. This comparison supports the value of the corpus as a whole for these decisions. Our response to W2 further examines the contributions of training-time rationale supervision and inference-time rationale use.

Table R1. Comparison of the decisions covered by rationale supervision.

Related workDemonstration settingAdditional supervision provided by EmbRACE
R2R-CoT-320k, Nav-CoT-110K’s released R2R/RxR pipeline, OctoNav-BenchActions derived from scene geometry or supplied pathsHuman choices made from available observations, with rationales relating visual history to search and target pursuit
Embodied-ReasonerObject-level navigation actions encapsulate intervening movementSeparate rationale supervision for movement and turning decisions as observations change
ECoT and EMMA-XHuman-teleoperated manipulation under a fixed third-person cameraNavigation and interaction decisions involving target pursuit and reference maintenance across changes in the agent’s viewpoint

W3: Faithfulness of the rationale

A rationale need not reproduce the demonstrator’s actual reasoning to provide useful supervision. A demonstrated action can admit more than one observation-grounded justification, and our annotation provides one such justification as a training target. Model-generated rationale supervision is also used in related corpora. ECoT and EMMA-X annotate human-teleoperated Bridge V2 demonstrations through automated annotation pipelines. R2R-CoT-320k uses Qwen2.5-VL-72B, Nav-CoT-110K uses Gemini 2.5 Pro, the TBA-CoT set of OctoNav-Bench uses Qwen-VL and DeepSeek-R1, and Embodied-Reasoner uses GPT-4o.

Our rationales are generated retrospectively from the instruction, observations available up to each step, and demonstrated action. Annotators verify every rationale for visual grounding, knowledge boundaries, and action consistency. The explanation must be supported by the available observations, assert nothing beyond what the demonstrator could have known at that step, and agree with the executed action. For example, knowledge-boundary violations were flagged in 20% of training trajectories on the first pass, triggering regeneration and rechecking (Section 3.2 of the main paper).

To make this distinction explicit, we will replace “states what the action was based on” with “relates the available observations to the demonstrated action” in the abstract, introduction, and conclusion.

W2 & Q2: Rationale supervision versus reasoning at inference

To test whether the gain comes from the rationale as a supervision signal or from reasoning at inference, we trained one model in the Aux-Think manner, mixing both training targets over the same trajectories. Each trajectory is used once with a rationale followed by the action and once with the action alone, selected by a control token in the prompt. We then evaluated this single model with the rationale switched on and off.

Table R2. Success rate on the EmbRACE benchmark, with the first two rows from Table 3.

Training targetInference outputBasicExploreDynamicMultiOpenP&D
ActionAction73.345.350.042.567.062.1
Rationale + ActionRationale + Action80.254.767.552.885.369.4
MixtureRationale + Action81.053.867.551.985.370.2
MixtureAction71.647.049.142.567.060.5

Training target is what the model is trained to output at each step, and Mixture uses both targets as described above. Inference output specifies what the model generates at test time. In Table R2, rows 1 and 4 both use action-only inference, while row 4 additionally receives rationale supervision during training. They match to within 1.7 percentage points on every task type. Rows 3 and 4 are one model differing only in whether it writes a rationale at test time, and that gap runs from 6.8 to 18.4 percentage points. In this experiment, training-time rationale supervision alone does not reproduce the gain, which depends mainly on generating rationales during inference.

This runs opposite to the finding reported for Aux-Think (R2R-CoT-320k), where withholding the reasoning at inference was the better strategy. Its authors attribute that result to CoT being trained only on optimal oracle trajectories, while agents at test time enter unfamiliar, off-distribution states. The two corpora differ in how their action demonstrations are collected. R2R-CoT-320k takes its action labels from R2R-CE reference paths computed by shortest-path search, so each action is determined with information beyond the agent’s observations. Each EmbRACE action was chosen by a human demonstrator through the agent’s egocentric view without privileged information, and each rationale is generated from the frames available up to that step and verified by annotators against them. In our setting, the same mixed model performs better when it generates rationales at inference.


W4 & Q1: Comparison with lightweight imitation learning baselines

An episode is a trajectory, not a sequence of independent classifications. Each action changes what the agent sees next, so the inputs a policy faces are the ones its own earlier choices produced. Task success depends on the sequence of decisions that reaches the instructed goal and completes any required interactions. A smaller label space does not by itself make this easier. The two imitation-learning baselines in Table I of NaVid (arXiv:2402.15852) choose among four actions, fewer than ours, and reach 0.00% success for RGB-Seq2Seq and 5.00% for RGB-CMA on VLN-CE R2R Val-Unseen, against 37.4% for a video-based VLM.

To provide the requested comparison, we trained RGB-Seq2Seq and RGB-CMA on the EmbRACE training split, following NaVid’s setup for these baselines. With the same training data and action-only supervision, both lightweight imitation-learning baselines remain substantially below Qwen3.5-9B’s action-only variant across all six task types (Table R3).

Table R3. Success rate (%) on EmbRACE.

ModelBasicExploreDynamicMultiOpenP&D
Random0.00.00.00.00.00.0
RGB-Seq2Seq23.311.714.92.82.811.3
RGB-CMA28.414.515.77.58.315.3
Qwen3.5-9B, actions only73.345.350.042.567.062.1
Qwen3.5-9B, actions and rationale80.254.767.552.885.369.4

I changed the rating to weak acceptance since the rebuttal addressed the concern regarding the effect of reasoning at inference. However, I still find the novelty incremental

Thank you for your time and for updating your rating to weak acceptance. Regarding your question about the scientific novelty, EmbRACE studies how observation-grounded explanations support decisions as an agent’s actions change what it can see, particularly when targets are initially outside the view or leave it during movement.

We organize target search, target identity maintenance, and interaction-state verification into separately evaluated closed-loop tasks, supported by human egocentric demonstrations and verified stepwise explanations. To further investigate these questions beyond success rate, we introduce Revisit, Block, and Recovery, measuring repeated visits, persistent blocked movement, and recovery after failed interactions. Our experiments and additional ablations demonstrate improvements in closed-loop behavior and real-robot experiments also show that the training benefits transfer to physical environments.

Reviewer_3nnSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present EmbRACE, a novel benchmark for vision-driven embodied agents under closed-loop settings. Specifically, EmbRACE includes both a training set and an evaluation set: the former has 3,421 human demonstrations in 55 environments with fine-grained rationale annotations paired at each step, and the latter contains 686 tasks across 7 environments left out of training. The SFT experiments validate the training data's effectiveness and quality in improving agent performance under complex scenarios (e.g., when the object is invisible in initial frames or multiple targets must be traced), and ablation studies with physical robots and other benchmarks further highlight the learned task-level behavior's transferability and generalizability.

Critical strengths
  1. The proposed dataset is large-scale and spans a range of difficulty levels.
  2. The experiments are extensive, and the results strongly support the quality and generality of the training set.
Critical weaknesses

Missing systematic comparison with prior benchmarks. The authors did not clearly state EmbRACE's unique contribution or distinct advantage over existing benchmarks, such as more complex task types or manual annotations in the stepwise rationale. As a result, it is hard to assess EmbRACE's novelty, even though the authors conducted extensive experiments.

Questions for authors

Based on the weaknesses, I hope the authors can respond to the following questions to address my major concerns.

  1. Could you provide a detailed discussion of EmbRACE in relation to existing work, highlighting its unique improvements in dataset scale, difficulty category, and other assessable elements?
3 comments

W1 & Q1: Comparison with prior work

Thank you for the positive assessment and for suggesting a systematic comparison with prior work. EmbRACE provides paired action and rationale supervision for navigation and object interaction under changing egocentric observations. Human demonstrators select actions through the agent’s egocentric view, while the agent’s motion changes the visual information available for subsequent decisions. Each rationale provides an observation-grounded justification for the demonstrated action, offering explicit supervision for using observation history in action selection. The rationales are generated from the instruction, observations available up to the current step, and executed action, then verified by human annotators for visual grounding, knowledge boundaries, and action consistency. Beyond the overview in Table 1 of the main paper, Table R1 in this rebuttal details action construction, viewpoint and action granularity, rationale annotation, and task coverage.

Action construction and privileged information. R2R-CoT-320k derives its action labels from R2R-CE oracle trajectories computed using scene geometry. In Nav-CoT-110K’s released R2R/RxR construction pipeline, actions are derived from supplied paths specifying the positions or viewpoints to visit. These paths provide privileged route information beyond the agent’s observation history. OctoNav-Bench constructs trajectories using the simulator’s navigable geometry and sampled endpoints before adding TBA-CoT annotations. Such action construction bypasses the need to resolve from observation history where to search or how to continue toward a target that has left the frame. EmbRACE instead records human action choices made through the agent’s egocentric view. This provides demonstrations of these decisions based on the observations available to the agent, paired with stepwise rationales that relate visual history to the chosen actions.

Task requirements, action granularity, and viewpoint. Embodied-Reasoner’s object-level navigation actions, such as “Navigate to Drawer,” encapsulate the intervening movement. EmbRACE instead pairs individual movement and turning actions with rationales, providing explicit supervision for these decisions as observations change. ECoT and EMMA-X provide rationale supervision for human-teleoperated manipulation under a fixed third-person camera. In EmbRACE, the agent’s own navigation changes the viewpoint, so the paired supervision also covers decisions about pursuing targets and maintaining referents across changing views.

Task coverage and categories. R2R-CoT-320k covers instruction-following navigation. Nav-CoT-110K covers both instruction-following navigation and object-goal navigation. EmbRACE combines navigation with door and object interactions. OctoNav defines difficulty by the number of composed navigation capabilities. Embodied-Reasoner groups tasks into search, manipulation, transport, and their combinations. EmbRACE distinguishes initial target visibility, reference maintenance across viewpoints, and sequential goals, revealing capability gaps beyond reaching a visible target. In Table 5 of the main paper, R2R-CoT training achieves 80–100% success on Basic across three robot scenes, but only 0–20% on Exploration and 0% on Dynamic Spatial-Semantic. Under the same training setup, EmbRACE training reaches 80–84% and 80–100% on the latter two types.

Rationale annotation and verification. R2R-CoT-320k provides 320k model-generated traces on R2R-CE. Nav-CoT-110K retains 110k CoT examples after automatic filtering. TBA-CoT contains 10k automatically generated rationale–action pairs. EmbRACE provides 48k rationale–action pairs, human-verified at every step.

Table R1. Comparison with rationale-annotated datasets.

DatasetAction constructionViewpoint / action granularityRationale annotationTask coverage
R2R-CoT-320kR2R-CE oracle trajectoriesEgocentric movement and turningModel-generatedInstruction-following navigation
Nav-CoT-110KSupplied R2R/RxR paths in released pipelinePanoramic views in that pipelineModel-generated, automatically filteredInstruction-following and object-goal navigation
OctoNav: TBA-CoTNavigable geometry and endpointsEgocentric movement and turningMulti-stage model annotationNavigation capabilities and their compositions
Embodied-ReasonerGraph-based trajectory synthesis and model self-explorationEgocentric, object-level navigationModel-generatedSearch, manipulation, transport, composite tasks
ECoT / EMMA-XHuman teleoperationFixed third-person, end-effector controlAutomated pipelineManipulation
EmbRACEHuman egocentric controlEgocentric movement, turning, and interactionModel-generated, human-verified at every stepTarget visibility, reference across views, sequential goals, door and object interaction

Thanks for your response. Based on the rebuttal and the original paper, I think EmbRACE's distinct feature is its focus on target visibility, i.e., the object may be invisible at initial observation or be lost during motion planning. This complex scenario is often overlooked by prior benchmarks. The authors should highlight this concept in the revisions. As my primary concerns are well addressed, I am positive about this manuscript and would like to finalize my scoring as a 'Weak Acceptance'.

Thank you for your thoughtful and encouraging feedback. We sincerely appreciate your time and careful consideration. We will revise the manuscript as suggested.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.