- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TRUE, a dataset and benchmark for evaluating embodied control and the evidence consistency of model-generated reasoning. It augments manipulation trajectories, simulated lunar-navigation episodes, and real planetary imagery with structured annotations describing task progress, subgoals, and action consequences. The benchmark separates control performance from reasoning consistency under a fixed deployment interface. The authors also present TRUE-Ref, an OpenVLA-OFT-based reference model incorporating recent action history and reasoning supervision. Experiments evaluate long-horizon manipulation, the contributions of history and reasoning, and reasoning consistency across simulated navigation and planetary imagery.
Critical strengths
-
A useful attempt to evaluate control and evidence-grounded reasoning separately. The paper addresses the limitation that successful task execution does not necessarily imply a correct explanation. Its shared annotation schema spans manipulation, simulated navigation, and planetary imagery, providing a potentially useful resource for studying reasoning across different observation distributions. The fixed deployment interface also prevents improvements from being attributed to additional access to the annotation generator at test time.
-
Controlled ablations provide meaningful evidence of improved long-horizon control. On CALVIN ABC→D, the complete model achieves an average completed chain length of 3.67, compared with 2.66 for OpenVLA-OFT and 3.23 for the history-only variant. The separate history and reasoning ablations, together with the reasoning-coverage study, support the usefulness of the training signals. These results are informative even though they do not independently establish the novelty or specific benefit of the trustworthiness constraints.
Critical weaknesses
-
The incremental contribution beyond existing embodied-reasoning methods is insufficiently established. ECoT [1] already generates synthetic, visually grounded reasoning annotations for VLA training, while Emma-X [2] constructs grounded task-reasoning and spatial-guidance annotations, including trajectory segmentation to reduce hallucination. ERVLA [3] further demonstrates reasoning supervision during training without requiring explicit CoT generation for action prediction at deployment. These precedents cover substantial parts of TRUE’s data-generation and model-training approach. I acknowledge that TRUE uses concise trajectory-level summaries rather than unrestricted CoT, and that a benchmark contribution does not require a new policy architecture. However, the paper should more clearly establish what its annotation constraints and evaluation protocol add beyond these existing approaches. The current ablations compare reasoning coverage and the presence of history, rather than TRUE annotations against alternative grounded-reasoning supervision under matched conditions. Broadening domain coverage through a shared schema is potentially useful, but the additional diagnostic or transfer insight enabled by this combination remains insufficiently demonstrated.
-
The distinctive value of “trustworthy” reasoning is not sufficiently supported by the evaluation. I appreciate that Section 3.2 explicitly defines trustworthiness as evidence-grounded auditability rather than guaranteed correctness. Nevertheless, the reported improvements do not isolate the benefit of these particular constraints from that of additional language supervision. Table 5 compares against OpenVLA and OpenVLA-OFT by decoding their language heads, without a matched baseline trained with alternative reasoning annotations. Consequently, the results establish improved text consistency under the proposed training scheme, but not its advantage over existing grounded-reasoning approaches. Furthermore, the consistency metric penalizes unsupported statements without specifying a corresponding requirement for information coverage. It could therefore favor short or minimally informative outputs. Table 7 includes fragments such as “the basket is” and “terrain behind”; if complete, these do not convincingly demonstrate useful reasoning, and if truncated, the full outputs are needed. Representative evaluations of both factual support and completeness would better establish whether the benchmark measures meaningful auditability rather than simply the absence of explicit errors.
Questions for authors
- What specific benefit does TRUE’s annotation schema provide beyond the grounded-reasoning supervision in [1–3]? Can you provide a controlled comparison that holds the underlying trajectories, available evidence, backbone, action history, and training budget fixed while varying the annotation scheme—for example, ordinary trajectory summaries versus the proposed constrained annotations? This would help distinguish the contribution of the trustworthiness constraints from the general benefit of additional language supervision.
- How does the reasoning evaluation distinguish complete, useful explanations from short but uninformative outputs? Are the examples in Table 7 complete or abbreviated? Please clarify the completeness criteria and scoring protocol, and provide representative full outputs showing a meaningful failure or model distinction that control success alone would not reveal.
References
[1] Michał Zawalski et al. Robotic Control via Embodied Chain-of-Thought Reasoning. arXiv:2407.08693, 2024.
[2] Qi Sun et al. Emma-X: An Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning. ACL, 2025. arXiv:2412.11974.
[3] Nan Sun et al. Revisiting Embodied Chain-of-Thought for Generalizable Robot Manipulation. arXiv:2606.03784, 2026.
Q1: Benefit of TRUE’s annotations and comparison with existing methods
Thank you for suggesting this comparison. We can provide this analysis and are currently working on additional training runs with different annotation schemes, including ordinary trajectory summaries and TRUE’s constrained annotations. We will hold the underlying trajectories, available evidence, backbone, action history, and training budget fixed while varying the annotation scheme. These ablations will assess how the annotations affect control performance and reasoning quality, helping distinguish the contribution of TRUE’s constraints from the general benefit of additional language supervision.
The intended benefit of TRUE’s schema is to make task progress, subgoals, supporting evidence, and observable action consequences explicit across manipulation and planetary settings. Even when an episode ends successfully, intermediate steps may include unsuccessful grasp attempts, temporary setbacks, or corrective actions. Our annotations aim to capture these events when supported by the observations, providing training context about how the task progresses.
Q2: Completeness and usefulness of reasoning outputs
Table 7 presents abbreviated excerpts, and we agree that some are uninformative as displayed. Our intention was to avoid showing only favorable examples, but the shortened text does not allow readers to judge the full explanations. We will provide complete, verbatim outputs covering both stronger and weaker cases, alongside the corresponding observations and recorded control outcomes.
Our current human evaluation primarily considers visual grounding and coherence. However, a short statement can satisfy these criteria while remaining uninformative. We are therefore working on further assessments of task relevance and informativeness and exploring other criteria. In the revised protocol, we will clarify the completeness criteria and how outputs are scored.
The revised assessment will check whether outputs are supported by the observations and provide useful details about task progress and action outcomes. Short fragments that leave out key information will receive a low completeness score, while complete, coherent explanations that cover these details accurately will receive a higher score.
Additional Clarification: Annotation Leakage
We use the annotations as privileged training signals. They describe complete training episodes and may include later observations and action outcomes that are unavailable at an earlier prediction step. These annotations are not provided at inference, when the model uses observations, task instructions, and previously executed actions. We will make this distinction explicit in the revised manuscript.
Missed References
Thank you for pointing out the missing references. We appreciate your suggestions and will include the recommended references in our revised paper.