acceptodds
Under review as a conference paper at ICLR 2027

TRUE: A Trustworthy Reasoning and Unified Embodied Intelligence Benchmark for Multimodal Robotics

Abstract

Embodied vision-language-action (VLA) models are typically evaluated by whether their actions succeed, while the intermediate reasoning that connects perception, task progress, planning, and action remains largely unmeasured. We introduce **TRUE** (**T**rustworthy **R**easoning and **U**nified **E**mbodied Intelligence), a multimodal, multi-task dataset and benchmark for grounded and auditable reasoning in embodied intelligence. TRUE augments embodied trajectories with trustworthy reasoning-and-planning representations that describe task progress, intermediate sub-goals, scene-grounded evidence, and the consequences of recent actions while remaining constrained by the task specification and available observations. TRUE spans three tiers of increasing distance from standard robot learning: tabletop manipulation (CALVIN, LIBERO), CADRE-style lunar multi-rover navigation that we build in NVIDIA Isaac Sim, and real Perseverance Navcam/Hazcam imagery from NASA's Planetary Data System. The benchmark defines complementary two tracks under a fixed deployment interface: *control track* and *reasoning track*: the former measures whether reasoning supervision improves long-horizon action prediction, while the latter measures whether model-generated reasoning remains consistent with observable evidence across domains. We additionally provide a reference embodied model that combines visual observations, language instructions, explicit action history, and trustworthy reasoning within a shared representation. The benchmark reveals that (i) reasoning supervision and action history yield complementary gains in long-horizon control, improving CALVIN ABC→D average length by 38% over OpenVLA-OFT; (ii) gains vanish on near-saturated suites; and (iii) reasoning consistency degrades from simulated navigation (83%) to real planetary imagery (68%), exposing a substantial gap between simulated navigation and real planetary imagery. We release annotations, the Isaac Sim environments, evaluation code, and baselines. TRUE is designed to support joint study of embodied control, interpretability, and cross-task reasoning under a common benchmark.

Primary area:
datasets and benchmarks
Keywords:
Space Robotics, Trustworthy Reasoning Robotics Datasets and Benchmark, Embodied AI for robotics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bdPXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TRUE, a dataset and benchmark for evaluating embodied control and the evidence consistency of model-generated reasoning. It augments manipulation trajectories, simulated lunar-navigation episodes, and real planetary imagery with structured annotations describing task progress, subgoals, and action consequences. The benchmark separates control performance from reasoning consistency under a fixed deployment interface. The authors also present TRUE-Ref, an OpenVLA-OFT-based reference model incorporating recent action history and reasoning supervision. Experiments evaluate long-horizon manipulation, the contributions of history and reasoning, and reasoning consistency across simulated navigation and planetary imagery.

Critical strengths
  1. A useful attempt to evaluate control and evidence-grounded reasoning separately. The paper addresses the limitation that successful task execution does not necessarily imply a correct explanation. Its shared annotation schema spans manipulation, simulated navigation, and planetary imagery, providing a potentially useful resource for studying reasoning across different observation distributions. The fixed deployment interface also prevents improvements from being attributed to additional access to the annotation generator at test time.

  2. Controlled ablations provide meaningful evidence of improved long-horizon control. On CALVIN ABC→D, the complete model achieves an average completed chain length of 3.67, compared with 2.66 for OpenVLA-OFT and 3.23 for the history-only variant. The separate history and reasoning ablations, together with the reasoning-coverage study, support the usefulness of the training signals. These results are informative even though they do not independently establish the novelty or specific benefit of the trustworthiness constraints.

Critical weaknesses
  1. The incremental contribution beyond existing embodied-reasoning methods is insufficiently established. ECoT [1] already generates synthetic, visually grounded reasoning annotations for VLA training, while Emma-X [2] constructs grounded task-reasoning and spatial-guidance annotations, including trajectory segmentation to reduce hallucination. ERVLA [3] further demonstrates reasoning supervision during training without requiring explicit CoT generation for action prediction at deployment. These precedents cover substantial parts of TRUE’s data-generation and model-training approach. I acknowledge that TRUE uses concise trajectory-level summaries rather than unrestricted CoT, and that a benchmark contribution does not require a new policy architecture. However, the paper should more clearly establish what its annotation constraints and evaluation protocol add beyond these existing approaches. The current ablations compare reasoning coverage and the presence of history, rather than TRUE annotations against alternative grounded-reasoning supervision under matched conditions. Broadening domain coverage through a shared schema is potentially useful, but the additional diagnostic or transfer insight enabled by this combination remains insufficiently demonstrated.

  2. The distinctive value of “trustworthy” reasoning is not sufficiently supported by the evaluation. I appreciate that Section 3.2 explicitly defines trustworthiness as evidence-grounded auditability rather than guaranteed correctness. Nevertheless, the reported improvements do not isolate the benefit of these particular constraints from that of additional language supervision. Table 5 compares against OpenVLA and OpenVLA-OFT by decoding their language heads, without a matched baseline trained with alternative reasoning annotations. Consequently, the results establish improved text consistency under the proposed training scheme, but not its advantage over existing grounded-reasoning approaches. Furthermore, the consistency metric penalizes unsupported statements without specifying a corresponding requirement for information coverage. It could therefore favor short or minimally informative outputs. Table 7 includes fragments such as “the basket is” and “terrain behind”; if complete, these do not convincingly demonstrate useful reasoning, and if truncated, the full outputs are needed. Representative evaluations of both factual support and completeness would better establish whether the benchmark measures meaningful auditability rather than simply the absence of explicit errors.

Questions for authors
  1. What specific benefit does TRUE’s annotation schema provide beyond the grounded-reasoning supervision in [1–3]? Can you provide a controlled comparison that holds the underlying trajectories, available evidence, backbone, action history, and training budget fixed while varying the annotation scheme—for example, ordinary trajectory summaries versus the proposed constrained annotations? This would help distinguish the contribution of the trustworthiness constraints from the general benefit of additional language supervision.
  2. How does the reasoning evaluation distinguish complete, useful explanations from short but uninformative outputs? Are the examples in Table 7 complete or abbreviated? Please clarify the completeness criteria and scoring protocol, and provide representative full outputs showing a meaningful failure or model distinction that control success alone would not reveal.

References

[1] Michał Zawalski et al. Robotic Control via Embodied Chain-of-Thought Reasoning. arXiv:2407.08693, 2024.

[2] Qi Sun et al. Emma-X: An Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning. ACL, 2025. arXiv:2412.11974.

[3] Nan Sun et al. Revisiting Embodied Chain-of-Thought for Generalizable Robot Manipulation. arXiv:2606.03784, 2026.

1 comment

Q1: Benefit of TRUE’s annotations and comparison with existing methods

Thank you for suggesting this comparison. We can provide this analysis and are currently working on additional training runs with different annotation schemes, including ordinary trajectory summaries and TRUE’s constrained annotations. We will hold the underlying trajectories, available evidence, backbone, action history, and training budget fixed while varying the annotation scheme. These ablations will assess how the annotations affect control performance and reasoning quality, helping distinguish the contribution of TRUE’s constraints from the general benefit of additional language supervision.

The intended benefit of TRUE’s schema is to make task progress, subgoals, supporting evidence, and observable action consequences explicit across manipulation and planetary settings. Even when an episode ends successfully, intermediate steps may include unsuccessful grasp attempts, temporary setbacks, or corrective actions. Our annotations aim to capture these events when supported by the observations, providing training context about how the task progresses.

Q2: Completeness and usefulness of reasoning outputs

Table 7 presents abbreviated excerpts, and we agree that some are uninformative as displayed. Our intention was to avoid showing only favorable examples, but the shortened text does not allow readers to judge the full explanations. We will provide complete, verbatim outputs covering both stronger and weaker cases, alongside the corresponding observations and recorded control outcomes.

Our current human evaluation primarily considers visual grounding and coherence. However, a short statement can satisfy these criteria while remaining uninformative. We are therefore working on further assessments of task relevance and informativeness and exploring other criteria. In the revised protocol, we will clarify the completeness criteria and how outputs are scored.

The revised assessment will check whether outputs are supported by the observations and provide useful details about task progress and action outcomes. Short fragments that leave out key information will receive a low completeness score, while complete, coherent explanations that cover these details accurately will receive a higher score.

Additional Clarification: Annotation Leakage

We use the annotations as privileged training signals. They describe complete training episodes and may include later observations and action outcomes that are unavailable at an earlier prediction step. These annotations are not provided at inference, when the model uses observations, task instructions, and previously executed actions. We will make this distinction explicit in the revised manuscript.

Missed References

Thank you for pointing out the missing references. We appreciate your suggestions and will include the recommended references in our revised paper.

Reviewer_2RyaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TRUE, a multimodal, multi-task dataset and benchmark for trustworthy reasoning and planning in embodied intelligence. TRUE connects manipulation, simulated lunar navigation, and real planetary observations through a shared reasoning-and-planning schema that makes task progress, sub-goals, scene evidence, and action consequences explicit and auditable. There are two different tracks in TRUE: a control track that evaluates whether reasoning supervision improves embodied action prediction, and a reasoning track that evaluates whether model-generated reasoning remains grounded in observable evidence. A reference model demonstrates how TRUE's reasoning signals and explicit action history can be incorporated into a unified embodied representation, yielding gains on long-horizon CALVIN while exposing a persistent reasoning gap between simulated navigation and real PDS imagery.

Critical strengths
  1. The paper introduces a potentially valuable dataset and benchmark for embodied reasoning and planning. By incorporating manipulation, simulated navigation, and real planetary imagery, the benchmark provides diverse settings for studying embodied reasoning across different tasks and domains.

  2. The paper presents a reference model that integrates reasoning supervision and explicit action history into a unified embodied representation. The experiments demonstrate promising improvements in long-horizon manipulation performance on CALVIN.

Critical weaknesses

1. Insufficient validation of the benchmark's effectiveness and unclear novelty.

The paper claims to provide a unified benchmark for evaluating embodied reasoning and planning across different tasks and domains, but the experiments do not fully substantiate this claim. In particular, the proposed evaluation tracks use different metrics across domains, and the results do not demonstrate whether the benchmark can effectively assess cross-domain reasoning generalization. Furthermore, although the paper compares TRUE with existing robot learning datasets, it lacks a systematic comparison with prior benchmarks specifically designed for embodied reasoning and planning. Consequently, the novelty and advantages of TRUE over existing benchmarks remain unclear.

2. Concerns about reasoning annotation quality.

The reasoning annotations are generated from complete trajectories, including future observations and action consequences, but are subsequently used to supervise predictions at individual time steps. This raises concerns about whether the supervision contains information unavailable at inference time and whether the proposed training protocol adequately prevents information leakage. Furthermore, the annotation quality is assessed using a limited human verification protocol, without sufficient analysis of annotation errors or consistency across domains. These issues raise concerns about the reliability of the reasoning supervision and its contribution to the reported improvements.

3. Insufficient evaluation protocol and baseline coverage.

The reasoning track relies on human evaluation of whether generated statements are supported by observations, but this metric does not adequately capture the correctness, informativeness, or usefulness of reasoning. The reliance on human evaluation also limits the scalability and reproducibility of the benchmark. Moreover, only a small number of closely related VLA models are evaluated, making it difficult to assess whether the benchmark can reliably distinguish different reasoning capabilities across model architectures.

Questions for authors
  1. How does TRUE differ from existing embodied reasoning and question-answering benchmarks, such as ERQA and OpenEQA? What additional reasoning and planning capabilities does TRUE evaluate?

  2. Can the authors provide more comprehensive evaluations involving diverse embodied reasoning models, rather than primarily comparing VLA models designed for robot control?

  3. How does the proposed reasoning consistency metric compare with existing reasoning evaluation protocols? Does it capture reasoning correctness and planning capability beyond visual grounding?

2 comments

Q1. Differences between TRUE and ERQA/OpenEQA

Thank you for this insightful question. TRUE is designed to complement, rather than replace, existing embodied reasoning and question-answering benchmarks such as ERQA and OpenEQA. While ERQA primarily evaluates embodied reasoning through multimodal multiple-choice questions, and OpenEQA assesses environmental understanding through open-ended question answering based on episodic memory or active exploration, TRUE extends embodied reasoning beyond question answering toward trustworthy, action-grounded reasoning, planning, and execution.

Specifically, TRUE introduces the following additional capabilities and evaluation dimensions:

  • Trajectory-level temporal reasoning. TRUE connects visual observations, previously executed actions, and task progress to reason about what has happened and what should follow.

  • Goal-oriented planning. TRUE introduces ordered, visually verifiable intermediate sub-goals that decompose high-level task instructions into meaningful steps grounded in observable evidence.

  • Action-consequence reasoning. TRUE explicitly represents how previous actions change the environment and how these changes constrain subsequent actions and planning.

  • Trustworthy reasoning and auditability. Beyond final-answer correctness, TRUE evaluates whether every generated reasoning and planning statement is supported by observable evidence and consistent with the task specification.

  • Reasoning-to-action effectiveness. TRUE evaluates whether reasoning supervision improves long-horizon robot control through reasoning-coverage experiments and action-history ablations.

  • Cross-task and cross-domain reasoning. TRUE establishes a unified reasoning-and-planning representation across tabletop manipulation (CALVIN and LIBERO), simulated lunar multi-rover navigation (CADRE-style Isaac Sim environments), and real planetary observations (NASA Perseverance PDS imagery).

Importantly, TRUE introduces two complementary evaluation tracks:

  1. Control Track: Evaluates whether trajectory-grounded reasoning supervision improves long-horizon action prediction and task execution.

  2. Reasoning Track: Directly evaluates whether model-generated reasoning and planning remain consistent with observable evidence across different domains.

Our experimental results demonstrate that reasoning supervision and action history improve the average completed task length by 38% over OpenVLA-OFT on CALVIN ABC→D. Furthermore, reasoning consistency decreases from 83% in simulated lunar navigation to 68% on real planetary imagery, revealing a substantial cross-domain grounding challenge.

Therefore, the fundamental distinction is that TRUE moves beyond answer-centric embodied reasoning evaluation toward trajectory-grounded reasoning, planning, and action-centric evaluation, jointly investigating what an embodied agent understands, how it reasons and plans, whether reasoning supervision improves its actions, and whether its reasoning remains trustworthy.

Our original focus was to connect manipulation trajectories, simulated CADRE-style navigation, and real Perseverance observations through structured annotations describing task progress, subgoals, supporting visual evidence, and observable action consequences. Our planetary goal is to study reasoning and planning for rover navigation and multi-agent coordination, including terrain understanding, obstacle assessment, subgoal identification, and mission-level action coordination.

We appreciate the reviewer for highlighting these relevant benchmarks and will incorporate a more comprehensive comparison in our revised paper.

Q2: Broader Evaluation of Embodied Reasoning Models

Thank you for this valuable suggestion. Our initial evaluation primarily focused on VLA models to investigate whether trustworthy reasoning supervision improves embodied reasoning, planning, and long-horizon robot control under a unified deployment interface.

The submitted experiments focused primarily on action prediction and comparisons with VLA models for robot control. We plan to broaden this evaluation in the revision by including additional embodied reasoning comparisons alongside the existing control baselines.

We agree that evaluating a broader range of embodied reasoning models would further strengthen TRUE as a general benchmark. In our revision, we plan to extend the reasoning-track evaluation to include diverse multimodal and embodied reasoning models, such as Gemini Robotics, Qwen3-VL, and other representative vision-language models. These models will be evaluated on temporal reasoning, task-progress understanding, subgoal planning, action-consequence reasoning, and evidence-grounded consistency across manipulation, simulated lunar navigation, and real planetary observations.

This extended evaluation will help distinguish general embodied reasoning capabilities from reasoning capabilities learned through robot-control supervision, further demonstrating TRUE's applicability beyond VLA models.

Reviewer_EUATOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

TRUE proposes a benchmark for grounded reasoning and embodied control across CALVIN, LIBERO, simulated CADRE-style navigation, and Perseverance imagery. It introduces trajectory-level reasoning annotations, control and reasoning evaluation tracks, and TRUE-Ref, an OpenVLA-OFT-based reference model trained with action history and reasoning supervision.

Critical strengths

Evaluating both action success and evidence-grounded reasoning across manipulation, simulated navigation, and real planetary imagery is a potentially useful benchmark direction.

Critical weaknesses
  1. Reliability of experimental results.
    • Table 4 reports HULC's ABCD→D results under ABC→D (average length 3.06, while HULC paper's ABC→D result is 0.67).
    • Table 4's TRUE-Ref (p = 1) average length is inconsistent with the reported per-step success rates (3.26 vs. 3.225).
    • In Table 6, the OpenVLA-OFT results exactly match the published OpenVLA-OFT numbers, without being marked as literature results, although Sec. 5 states that the authors evaluate OpenVLA-OFT by themselves.
  2. Incomplete evaluation and internal inconsistencies.
    • Table 3 and Sec. 4.2 explicitly define 'navigation success' as a control-track metric for CADRE-style navigation, but no corresponding results are reported.
    • Figure 2 includes "Ground Truth — Success: True", although Sec. 3.2–3.3 explicitly state that reasoning annotations must not judge task success and that success labels are not provided to the annotation generator.
    • Table 7 presents incomplete phrases such as "the basket is" as inference-time reasoning outputs, which do not demonstrate the reasoning structure described in Sec. 3.2 and Appendix A.
    • Citation errors: RT-2 is attributed to Brown rather than Brohan; group authors of Octo and Open X-Embodiment are incorrectly rendered as "Team et al." and "Collaboration et al."; SmolVLA appears in Table 6 and Sec. 5 without any citation or corresponding reference entry.
Questions for authors
  1. Why are HULC's ABCD→D results reported under ABC→D in Table 4?
  2. Were the OpenVLA-OFT numbers in Table 6 reproduced independently or taken from the OpenVLA-OFT paper?
  3. How do the authors explain the TRUE-Ref (p = 1) average-length discrepancy, and where are the CADRE navigation control results?
  4. How do Figure 2 and Table 7 conform to the stated annotation and reasoning-output protocols?
1 comment

Q1: HULC results and the CALVIN evaluation split

Thank you for identifying this. When preparing Table 4, we mistakenly included HULC’s published ABCD→D results in the ABC→D comparison. The reported average completed sequence length of 3.06 belongs to ABCD→D. The correct published result for ABC→D is 0.67, as reported in Figure 4 of the original HULC paper (Mees et al., 2022). We will correct the HULC entry in Table 4 to report the published ABC→D results. We will also identify this entry as a result reported in the literature and clarify its training and evaluation split in the table caption.

Q2: Source of the OpenVLA-OFT results

The OpenVLA-OFT results in Table 6 of our submitted version were taken from the original OpenVLA-OFT paper (Kim et al., 2025). We will correct both the table labeling and the experimental descriptions. We have additionally evaluated the authors’ released OpenVLA-OFT checkpoints in our own environment. The published results and our checkpoint-evaluation results are shown separately below. All values are task success rates (%).

Result sourceSpatialObjectGoalLongAverage
Original OpenVLA-OFT paper97.698.497.994.597.1
Our evaluation of released checkpoints97.698.097.895.097.1

Our evaluation obtains the same average success rate, with small differences in the individual suite results. This experiment constitutes an independent evaluation of the released checkpoints. In the revision, we will explicitly distinguish results taken from prior publications from evaluations conducted by us. We will include our checkpoint-evaluation results as a separately labeled entry and revise Section 5 so that the experimental description accurately reflects the source of each reported result.

Q3: Average completed sequence length and CADRE evaluation

Thank you for identifying that in the TRUE-Ref (p = 1) average completed sequence length. The reported per-step success rates imply an average length of 3.225, which rounds to 3.23 at two decimal places. The value of 3.26 in Table 4 was a reporting error. Regarding CADRE-style navigation, Table 5 reports our human evaluation of generated navigation outputs against the corresponding episodes. This evaluation assesses whether the outputs are consistent with the observed scene and episode context.

Q4: Figure 2 and Table 7 protocol clarification

Figure 2: Annotator inputs and appended success metadata. The annotation generator receives the episode images together with the high-level daily plan for planetary examples or the task instruction for manipulation examples. Success labels are not included in these inputs, and the generator is not asked to predict task success. The “Ground Truth — Success: True” field shown in Figure 2 is appended to the full TRUE annotation record after annotation generation. For manipulation examples, this field records the episode’s ground-truth success outcome. It is outcome metadata added during preprocessing, separate from the reasoning text produced by the annotator. For planetary episodes, we set the success field to True during preprocessing. For Isaac Sim, we treated the episodes we created as successful examples when constructing the dataset. For PDS, we grouped images by Martian sol and assumed that the associated planned activity for that sol had been completed. We recognize that the current figure does not clearly distinguish annotator-generated reasoning from appended outcome metadata or explain differences in label provenance across datasets. We will revise Figure 2, its caption, and Sections 3.2–3.3 to clarify these distinctions and the processing order. We will also specify the source of the outcome labels, which fields were included in the original model training, and how these fields are used during evaluation.

#Table 7: Examples of inference-time outputs. Fragments such as “the basket is” do not demonstrate the structured reasoning described in Section 3.2 and Appendix A. We can provide further examples of structured reasoning labels. We will expand Table 7 to show complete, verbatim model generations with their corresponding task instructions and observation context. Any shortened excerpts will be explicitly labeled.

Citation corrections. Noah Brown is a coauthor of RT-2, but our attribution to “Brown et al.” was incorrect. We will correct the citation to “Brohan et al. (2023)” and fix the corresponding bibliographic entry. We will also correct the group-author formatting for Octo and Open X-Embodiment so that their author names are rendered appropriately. Additionally, we will add the missing SmolVLA citation (Shukor et al., 2025) in Table 6 and Section 5, together with its full reference entry. We will check the manuscript for consistency between in-text citations and the bibliography.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.