acceptodds
Under review as a conference paper at ICLR 2027

Spatial Concept Bottleneck for Interpretable Robot Learning

24 views

Abstract

Robot learning models have been proliferating with increasingly stronger results for both generalist and specialist policies. However, true deployment potential of these learned policies is still unknown. As the community closes in on translational research, it is now ever more crucial to develop mechanisms that not only enable interpretability but intervention as well, e.g., through concept bottleneck models. In this work, we present an initial investigation along the lines of an interpretable spatial concept bottleneck that acts as a meaningful intermediate representation between a perception-planning backbone and an action head. Specifically, we define a spatial concept as a geometric relationship between entities corresponding to the task and the robot, which can be computed directly from the scene's geometry, and instantiate it for both navigation and manipulation. We present a series of experiments demonstrating that such a spatial concept bottleneck maintains original performance (without the bottleneck), while serving as an interpretable interface through which humans or VLMs can intervene to correct or steer the robot. Videos and results are available at https://spatial-cbm.github.io.

Primary area:
interpretability and explainable AI
Keywords:
robot learning, interpretability, navigation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uQoZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to make robot policies easier to inspect and correct by having them first predict spatial information that people can understand, then use that information to generate actions. For navigation, this information is a map of distances (a costmap); for manipulation, it is the positions of the object and the placement target relative to the gripper. The authors compare several training approaches on navigation and pick-and-place tasks, and test whether editing this information can correct failures.

Critical strengths
  1. A small amount of spatial information can support good manipulation performance. Using six ground-truth relative-position values, together with the instruction and the robot's own state, achieves an 84.8% success rate, close to the 83.8% achieved by the model that uses images directly. (Section 4.2, Table 3.) This shows that, for the tasks tested, visual input can be compressed into information that people can inspect while retaining good control performance.

  2. Editing this information can change behavior in the intended way. Correcting the predicted positions raises Spatial CBM's success rate from 75.0% to 79.8%. When the target is changed to another identical bowl, the robot completes the task with the new target in 93.5% of episodes. (Section 4.4, Figure 5, Table 3.) This provides direct evidence that the interface can be used to correct errors or change the target.

Critical weaknesses
  1. Two key steps in the navigation method are not explained clearly enough to assess the main results. First, Equation (2) defines the concept as the straight-line distance to the subgoal, but Section 4.1 discusses the shortest-path distance around obstacles and says the model needs to learn that distance. The two distance types lead to substantially different results, yet the paper does not clearly state which one is used to train the main models. Second, Equation (4) describes intervention as changing a target position and recomputing the costmap, but the VLM in Section 4.3 only outputs “left,” “center,” or “right.” The paper does not explain how that direction is turned into an edit of the full costmap. As a result, it is hard to tell exactly what was implemented or how much ground-truth information from the simulator is needed during intervention. This is a gap in the description; whether there is also an implementation issue remains a question for the authors.

  2. There are no real-robot experiments. All evaluations are conducted in simulation, as the authors acknowledge in Section 5. Since the approach relies on spatial information, it remains unclear whether the predicted concepts and interventions would work as well with real-world perception errors and imperfect robot execution. This limits the evidence for practical deployment, which is a main motivation of the paper.

Questions for authors
  1. Does the navigation model learn straight-line distances or shortest-path distances around obstacles? Please clarify what each model in Table 2 uses during training and testing, and explain the relationship between Equation (2) and Section 4.1. If this is only a missing detail in the text, clarifying it would substantially reduce my concern. If different models use different definitions, their performance differences need to be explained accordingly.

  2. How exactly is the costmap edited after the VLM outputs “left,” “center,” or “right”? Please describe the editing rule and identify which steps require simulator ground truth. If the simulator only decides when to intervene, while the edit itself uses information available during deployment, that would strengthen my assessment of the intervention results. Otherwise, the additional ground-truth information required should be made explicit.

3 comments
AuthorsOpenReview ↗

Response to Reviewer uQoZ (1/2)

We thank the reviewer for their suggestions. The gaps are description gaps, and the following should clarify.

Weaknesses:

First, Equation (2) defines the concept as the straight-line distance to the subgoal, but Section 4.1 discusses the shortest-path distance around obstacles and says the model needs to learn that distance. The two distance types lead to substantially different results, yet the paper does not clearly state which one is used to train the main models.

This is an inconsistency in the text only and not a difference between models. An earlier version of our method supervised the concept with geodesic distance. We later switched to the Euclidean distance of Eq. (2), because then the costmap can be computed from 3D points alone and needs no navigation mesh, making our system readily deployable. Section 4.1 still carried sentences from the earlier version, and we have revised it (updated paragraph below).

Results. Table 2 ablates the Spatial CBM layer as defined in Section 3.2. GT-Geodesics clearly establishes that action prediction using the oracle ceiling for true spatial concept values for both training and testing yields nearly perfect results. This implies GT-ED results imply that having the spatial concept as an intermediary is in itself not a performance bottleneck, rather it is the lack of the ability to predict perfect concepts. We also report results for GT-ED which mainly indicate GT-ED results do indicate that geodesic distances provide a better signal than Euclidean distances, and that the perception backend, VGGT, which is trained to learn Euclidean geometry, does would require learning geodesic geometry via fine-tuning to be performant on the navigation tasks. However, we observed that ED concept based supervision performed closely with its true ED counterpart. This is attributed to the use of GT subgoals which are derived from geodesic distances; thus, for our subgoal-conditioned navigation tasks, ED suffices.

We use the shortest-path (geodesic) distance to create the subgoals from the navmesh. Once subgoals are created, we train our models (Spatial CBM, Spatial CBM*, Spatial CAM) on the straight-line (Euclidean) distance to the subgoal, following Equation (2).

There are no real-robot experiments. All evaluations are conducted in simulation, as the authors acknowledge in Section 5. Since the approach relies on spatial information, it remains unclear whether the predicted concepts and interventions would work as well with real-world perception errors and imperfect robot execution. This limits the evidence for practical deployment, which is a main motivation of the paper.

We are currently setting up real world experiments and hope to share preliminary results at the earliest.

AuthorsOpenReview ↗

Response to Reviewer uQoZ (2/2)

Questions:

Please clarify what each model in Table 2 uses during training and testing, and explain the relationship between Equation (2) and Section 4.1.If this is only a missing detail in the text, clarifying it would substantially reduce my concern. If different models use different definitions, their performance differences need to be explained accordingly.

Method (CBM variant)ff trained onff tested onPredictor ggAvg SPLAvg SSPL
GT-Geodesicstrue costmap cctrue costmap ccnone (true depth)88.5988.5993.4093.40
GT-EDtrue costmap cctrue costmap ccnone (true depth)69.6569.6575.3375.33
Direct Action (Standard)VGGT features zzVGGT features zzfrozen VGGT55.5455.5465.2865.28
Spatial CAM (Multitask)VGGT features zzVGGT features zzfine-tuned VGGT, Eq. (2) as auxiliary LC\mathcal{L}_C40.3640.3647.2247.22
Spatial CBM* (Independent)true Eq. (2), ccpredicted Eq. (2), c^\hat{c}fine-tuned VGGT54.3954.3964.0164.01
Spatial CBM (Sequential)predicted Eq. (2), c^\hat{c}predicted Eq. (2), c^\hat{c}fine-tuned VGGT60.3860.3868.5168.51

We only ever use one single distance definition across all non-GT models.

Second, Equation (4) describes intervention as changing a target position and recomputing the costmap, but the VLM in Section 4.3 only outputs “left,” “center,” or “right.” The paper does not explain how that direction is turned into an edit of the full costmap. As a result, it is hard to tell exactly what was implemented or how much ground-truth information from the simulator is needed during intervention. This is a gap in the description; whether there is also an implementation issue remains a question for the authors.

How exactly is the costmap edited after the VLM outputs “left,” “center,” or “right”? Please describe the editing rule and identify which steps require simulator ground truth. If the simulator only decides when to intervene, while the edit itself uses information available during deployment, that would strengthen my assessment of the intervention results. Otherwise, the additional ground-truth information required should be made explicit.

The costmap is the straight-line distance from each pixel’s 3D point to the sub-goal Equation (2). To intervene effectively, we slice the costmap into “left”, “center”, and “right” interpretable (almost) equal width sectors/columns. The intervener picks a column it wants to edit. The new anchor is then set to the lowest-cost predicted pixel in the middle of that sector. The full costmap is then rebuilt as per Equation (4).

how much ground-truth information from the simulator is needed during intervention. If the simulator only decides when to intervene, while the edit itself uses information available during deployment, that would strengthen my assessment of the intervention results. Otherwise, the additional ground-truth information required should be made explicit.

The simulator is only required to decide when to intervene, that too only for the gated runs. An edit is made only when the predicted and true costmap minima fall in different sectors. We designed the edit to only use information which would be available during deployment. We really hope that this clarification strengthens the reviewer's assessment as they indicated and gets reflected through an improved rating.

We have preliminary real-world results on a Unitree Go2 in a home, ran with open loop on recorded data. We record a mapping run and three repeat runs of the same route. We use the frames from these three runs to see where we can intervene.

The setup is the same as Sections 4.1 and 4.3, except the sub-goal comes from A* on the robot's lidar floor map instead of the navigation mesh. The model is the same one from the paper, trained only in simulation.

We gave Claude (Opus 5.5) the Section 4.3 inputs (current view with the predicted costmap, sub-goal frame, destination) on 8 frames. It reasoned for an edit on 3 of them, and each edit moved the costmap minimum where Claude intended. In one, the minimum was on a bed and Claude answered "center", since "the white air purifier from the destination picture stands on open floor at the right edge of the center region". The examples are in Section D of the revised supplementary material.

These qualitative results show that the spatial concept can be predicted and edited on real-world runs, and that an intervener can use it to correct the robot.

Reviewer_F6irOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents Spatial CBM to interpret robot policies. The authors define spatial concepts as the geometric relationship between task-specific egocentric entities, e.g., navigation subgoals or pick/place locations, and robot-centric egocentric entities, e.g., end-effector position. Spatial CBM is implemented in navigation (Habitat) and manipulation (LIBERO) domains. They train a concept predictor from the vision backbone to predict costmaps and pick-and-place locations, and use the interpreted concept for a task predictor. The experiments show that Spatial CBM can achieve competitive performance while allowing intervention at test-time.

Critical strengths
  • The paper is well presented and easy to follow and understand.
  • The test-time intervention is a good demonstration that the learned concepts are useful to generate behavior.
Critical weaknesses
  • The proposed concepts (costmaps and pick-and-place targets) are the geometric target specification rather than the high-level semantic concepts like in the original CBM paper. Intervention of the defined concept is similar to directly modifying the waypoint rather than changing the model’s interpreted scene/task.
  • In the standard CBM, the concept bottleneck is injected in the latent representation of the main backbone. However, in the manipulation setup, the concept predictor is interpreting the vision encoder rather than what the VLM or action expert, leaving what VLA backbone is doing still a blackbox. Furthermore, the no-concept ablation achieves 68.2% success rate, suggesting some memorization of the proprioception is used rather than the predicted concepts.
Questions for authors
  • Why does the manipulation setup inject the bottleneck after the vision encoder? What is the role of the VLM/action expert in this setup?
  • How can we decouple the shortcut of using proprioception from the claim that the 6-token spatial concept captures the scene and how do they know about the tasks since the vision encoder doesn’t have language inputs?
4 comments
AuthorsOpenReview ↗

Response to Reviewer F6ir (1/2)

We thank the reviewer for their suggestions. Most of the questions are about manipulation setup, and the following should clarify. We have run additional experiments and will be adding them to the manuscript. Relevant tables are shared in these comments.

Note: Figure 2 did not show the instruction and state going into gg. This was an oversight and we have now included a revised Figure 2. This may already address several of the concerns below. For completeness, we answer each point below and add new experiments.

Weaknesses:

The proposed concepts (costmaps and pick-and-place targets) are the geometric target specification rather than the high-level semantic concepts like in the original CBM paper.

This is a deliberate choice. The ultimate goal of a concept bottleneck is to allow interpretability and intervenability while preserving model performance. Our proposed geometry-based spatial concepts hold these criteria. Moreover, as the advances in VLMs bridge the gap between semantics and geometry, VLM interpretable/intervenable concepts hold much more value in this era of human-robot collaboration. We position this as a novel take, applicable for the domain of robot tasks.

Intervention of the defined concept is similar to directly modifying the waypoint rather than changing the model’s interpreted scene/task.

We clarify that the concept is not a waypoint. In navigation, the concept is a costmap (Figure 1, Section 3.2.1). The costmap is sent as an input to the controller, which outputs the waypoints. For example, when an intervention happens, the sub-goal point of the scene changes. This requires recomputation of the distances of all pixels of the costmap to be min-max normalized with respect to the new change. This leads to the costmap looking entirely different, as evident from Figure 4. Once the costmap changes, the waypoints change consequently; the controller responds to the new costmap. We never modify the waypoints directly.

It is fair to say that semantically, they are apart. Editing the concept changes where the robot goes, however changing the waypoints is how it goes where. The controller is still in charge of the motion. In manipulation, our concept specifies only the target, and ff must still produce the full grasp-carry-place sequence from it. In the unseen object case, ff results in 67.0% success rate, much higher than the Direct Action's 7.0%.

In the standard CBM, the concept bottleneck is injected in the latent representation of the main backbone. However, in the manipulation setup, the concept predictor is interpreting the vision encoder rather than what the VLM or action expert, leaving what VLA backbone is doing still a blackbox. Furthermore, the no-concept ablation achieves 68.2% success rate, suggesting some memorization of the proprioception is used rather than the predicted concepts.

Like the original CBM, the task predictor (ff) itself is not interpretable. We explain the placement of the bottleneck and ablate the proprioception memorization case in the question below. The 68.2% does not come from proprioception alone. The instruction already tells the policy which bowl to pick through a spatial relation, e.g, “pick up the black bowl between the plate and the ramekin and place it on the plate”, and the robot state tells it where the arm is. Without the images and concept, however, it can’t really see where the bowl is, but the instruction and state are enough to replay the learned trajectories. So the 68.2% shows what instruction and state can achieve, not that the concepts are unused. As per the last row of Table 3 (GT, swapped), by simply swapping the concept bins for source and destination, the robot almost always fails (4.5%). Furthermore, as per Section 4.4 and its corresponding results in the table below, if we use a single-sentence policy (“pick up the black bowl and place it on the plate”), where the instructions do not spatially specify the bowl, the state alone gives 4.5% SR (please see the table below in the next comment).

AuthorsOpenReview ↗

Response to Reviewer F6ir (2/2)

Questions:

Why does the manipulation setup inject the bottleneck after the vision encoder? What is the role of the VLM/action expert in this setup? How can we decouple the shortcut of using proprioception from the claim that the 6-token spatial concept captures the scene …

In our setup, the VLM's language model and the action expert are the task predictor ff. In the existing experiments, we keep the language model frozen, which reads (i) the instruction, (ii) the state, and (iii) the concept tokens. We train the action expert, which outputs the actions.

Another approach is to combine the vision encoder with the VLM’s language model to be the concept predictor g’g’ and keep the action expert as the task predictor f’f’.

We chose the former design over the latter since it matches our navigation experiments closely: in both, the bottleneck sits where the visual inputs are turned into a spatial concept. Here, the concept predictor gg reads the image tokens, the instruction and the state. The task predictor ff reads the concepts, the instruction and the state. So the images reach ff only through the concept. We also checked this choice empirically. When the action expert can see the image tokens, it relies on them and ignores the concepts, so intervening on the concept no longer steers the robot (please see the results below).

In the following table, we use the single-sentence policy from Section 4.4, i.e., the instruction ("pick up the black bowl and place it on the plate") is shared by all ten tasks in place of each task’s instruction. So, language cannot tell the robot which bowl to pick. Every row also reads the state. The concept predictor gg still reads each task’s own instruction, and GT refers to the use of true concepts from the simulator. In the Concepts row, the concept is the only input that tells the robot which bowl to pick.

Action expert readsGT (cc)Spatial CBM* (c^\hat{c})Steering
Concepts85.8±0.485.8 \pm 0.479.8±1.879.8 \pm 1.893.5±2.193.5 \pm 2.1
Images + concepts86.5±0.086.5 \pm 0.080.8±1.180.8 \pm 1.142.8±8.142.8 \pm 8.1
Images61.561.5––
Neither4.54.5––

Keeping the image tokens along with concepts (third last row) gives about the same success, but steering then works only in some tasks, as the images tend to pull the robot back to the bowl it was trained on. With images only (second last row), it recognizes the task from its layout and replays the learned trajectory. Without concepts and images (last row), the single-sentence policy reads only the state and a sentence that does not name the bowl, so it almost always fails.

These results show that removing the image tokens from the action expert enables concept steering, which justifies the choice of bottleneck placement within the pipeline, i.e., after the vision encoder.

… how do they [6 concept tokens] know about the tasks since the vision encoder doesn’t have language inputs?

The vision encoder does not read the task description, but the concept predictor gg does. gg is one cross-attention layer with six trainable query vectors, one for each concept. They attend to both cameras' tokens from the frozen vision encoder, the robot state, and the instruction's word embeddings. The same trained gg gives the predicted concepts in every table.

I thank the authors for the clarification. I understood the concepts are represented as a cost map. While it is easy to learn than semantic concepts, I still find it limited in representing spatial concepts, for example, concepts like "inside" or "behind" are hard to represent. I appreciate the additional results in action expert steering and I think it is a good validation. I will raise my rating, but I hope the authors can discuss the limitations of the chosen concept representations.

Thank you for the discussion and for updating the rating! We agree that relations like "inside" or "behind" aren't directly represented by our costmaps, and we will discuss this in the Limitations.

Reviewer_wU9XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a Spatial Concept Bottleneck Model (Spatial CBM) for interpretable robot learning, introducing task-relevant geometric representations between a perception backbone and an action head. The authors instantiate the framework for navigation using pixel-level costmaps and for manipulation using discretized source and destination positions relative to the gripper. Experiments in Habitat and LIBERO-Spatial show that the bottleneck can largely preserve policy performance when ground-truth concepts are available, while interventions can correct or steer robot behavior. The paper also explores VLM-based interventions for navigation and demonstrates improvements in closed-loop success when interventions are triggered by an oracle.

Critical strengths
  1. The paper addresses an important challenge in robot learning: enabling humans to understand, correct, and steer learned policies at inference time.

  2. The experiments cover two distinct robotic settings and include closed-loop navigation evaluation, multiple bottleneck training variants, and intervention-based correction and steering.

Critical weaknesses
  1. The proposed framework largely adapts conventional concept bottleneck models to robot learning by defining task-specific spatial representations. The navigation costmap and manipulation offsets are intuitive choices, but the general formulation itself appears relatively straightforward. The concept bottleneck models are also not a novel thing (Concept bottleneck models,PMLR 2020).

  2. In Table 3, the direct-action baseline achieves an average success rate of 83.8%, compared with 75.0% for Spatial CBM. While ground-truth concepts recover performance to 84.8%, the predicted-concept bottleneck introduces a noticeable degradation. The intervention experiment recovers only part of this loss, reaching 79.8%. These results suggest that concept prediction remains a substantial limitation rather than establishing that the bottleneck can generally preserve the original policy's performance.

Questions for authors
  1. The intervention experiments use ground-truth simulator states both to construct reference concepts and, in navigation, to determine when intervention should occur. How would the system perform if interventions were triggered solely by the model's uncertainty or another deployable failure-detection mechanism?
  2. Is the navigation concept predictor supervised on Euclidean or geodesic distance? If geodesic, please update Eq. 2 and the Limitations claim about real-robot deployability. If Euclidean, please explain why GT-Geodesics, not GT-ED, is used as the reference oracle.
  3. Could you detail the manipulation concept predictor g more: its architecture, inputs (agent-view, wrist, or both), and loss?
  4. How many seeds and episodes per task underlie Table 2? Please add success rate.
  5. How do Direct Action and Spatial CBM compare under randomized initial object placements? This is where a spatial concept should help most.
3 comments
AuthorsOpenReview ↗

Response to Reviewer wU9X (1/2)

We thank the reviewer for their suggestions. We have run additional experiments and will be adding them to the manuscript.

Weaknesses:

The proposed framework largely adapts conventional concept bottleneck models to robot learning by defining task-specific spatial representations. … The concept bottleneck models are also not a novel thing (Concept bottleneck models,PMLR 2020).

True! We never claimed that the CBM idea was ours. We extended it to spatial concepts for robot learning and intentionally preserved the original CBM paper’s naming and bottleneck ideas to ensure our work is easily comparable. Our proposed spatial concepts differ since our labels come directly from scene geometry instead of human-designed datasets.

In particular, we report closed-loop evaluations where the robot’s actions change its observations, which further changes its actions. This is quite different from the typical open-loop evaluation structure of both the original CBM and follow-up literature where each input/output pair to the model is independently and identically distributed.

The navigation costmap and manipulation offsets are intuitive choices, but the general formulation itself appears relatively straightforward.

The simplicity is deliberate. Eq. 1 is a generic template proposing a 3D geometry relationship ψ\psi between the robot and the environment. We show that Eq. 1 can be used in both navigation (Eq. 2) and manipulation (Eq. 3) tasks; where both tasks are substantially different. In navigation, the same idea leads to a 16×1616 \times 16 costmap and in manipulation it leads to 66 discrete offsets. Moreover, the intervention setting follows the same consistent idea in Eq. 4, without complexity.

In Table 3, the direct-action baseline achieves an average success rate of 83.8%, compared with 75.0% for Spatial CBM. While ground-truth concepts recover performance to 84.8%, the predicted-concept bottleneck introduces a noticeable degradation. The intervention experiment recovers only part of this loss, reaching 79.8%. These results suggest that concept prediction remains a substantial limitation rather than establishing that the bottleneck can generally preserve the original policy's performance.

It would indeed be ideal to win three-way on performance x interpretability x intervenability axes. However, this is rarely the case (Breiman, 2001; Lampert et al., 2009; Branson et al., 2010; Gunning & Aha, 2019; Koh et al., 2020; Mahinpei et al., 2021; Havasi et al., 2022; Espinosa Zarlenga et al., 2022; Almudévar et al., 2026). In fact, the high performance on ground-truth concepts, as noted by the reviewer, establishes that our proposed spatial concept bottleneck is the right approach. The ability to perfectly predict the concepts will only get better in the future, eventually leading to a solution that could win on all three axes.

Questions:

The intervention experiments use ground-truth simulator states both to construct reference concepts and, in navigation, to determine when intervention should occur. How would the system perform if interventions were triggered solely by the model's uncertainty or another deployable failure-detection mechanism?

In the original CBM work, concept bottlenecks were aimed at interpretability and intervenability by human experts. Our VLM experiments were designed as a specific controlled stress-test of the “what” question, i.e., whether an intervener that never sees the true concept can produce useful edits, while the oracle gate fixes the “when” so that the two are not confounded.

Replacing the oracle/human expert with an automatic trigger, solving the when question, is an entire research fractal of its own. For example, in robotics, runtime failure detection is an active area with dedicated methods built on action consistency, out-of-distribution scores, or VLM monitors (Agia et al., 2024; Xu et al., 2025; Duan et al., 2025; Liu et al., 2023; Ren et al., 2023). Moreover, since an incorrect decision can be catastrophic, a deployable trigger’s false-positive rate should ideally be zero. This is beyond the scope of the rebuttal, and we already noted this as a future work in Section 5.

Finally, we note our proposed bottleneck setup provides the needed hooks for triggers. For manipulation, we output a distribution over the bins for each axis, so it may be directly used as a trigger. For navigation, we reported the results for the un-gated VLM experiments where the intervener has to decide when to intervene, without using ground-truth triggers.

AuthorsOpenReview ↗

Response to Reviewer wU9X (2/2)

Is the navigation concept predictor supervised on Euclidean or geodesic distance? If geodesic, please update Eq. 2 and the Limitations claim about real-robot deployability. If Euclidean, please explain why GT-Geodesics, not GT-ED, is used as the reference oracle.

Euclidean. Note that the navigation task is that of learning sub-goal conditioned control (Savinov et al., 2018; Shah et al., 2023; Garg et al., 2025), not end-to-end learnt navigation. The concept is the straight-line distance from each pixel’s 3D point to the sub-goal (Eq. 2). The sub-goals are deliberately generated using geodesic distances in order to discount the effect of sub-goal errors. It can be further noted that Euclidean distance is readily computable from VGGT’s depth maps which ensures real-world deployability.

We set GT-ED as the reference for the proposed concept (Euclidean), since it is the ground-truth straight-line costmap which is readily deployable, while GT-Geodesics is an oracle ceiling for a costmap that requires the simulation navigation mesh.

We will reframe parts of Section 4.1 for clarity.

Could you detail the manipulation concept predictor g more: its architecture, inputs (agent-view, wrist, or both), and loss?

gg is a single cross-attention layer with six learned queries, one per concept token (x, y, z of both the source and the destination). Input: (i) image tokens of both the agent-view and wrist cameras from SmolVLA’s frozen vision encoder, (ii) the robot state and (iii) the frozen word embeddings of the instruction. Each query’s output goes into its own 13-way classifier, trained with cross-entropy (Section 4.2, Figure 2).

How many seeds and episodes per task underlie Table 2? Please add success rate.

Two training seeds. Evaluated on all 126 episodes, containing 32 Imitate, 32 Alt-goal, 31 Shortcut, 31 Reverse navigation tasks (Section 4.1), following exactly the ObjectReact benchmark. Average success rate (%), mean ±\pm std over the two seeds (extension of Table 2):

MethodSuccess rate (%)
GT-Geodesics90.0±1.790.0 \pm 1.7
GT-ED71.0±1.771.0 \pm 1.7
Direct Action57.7±0.157.7 \pm 0.1
Spatial CAM41.4±1.641.4 \pm 1.6
Spatial CBM*60.0±3.460.0 \pm 3.4
Spatial CBM62.5±5.662.5 \pm 5.6

How do Direct Action and Spatial CBM compare under randomized initial object placements? This is where a spatial concept should help most.

Agreed, thank you for the experiment suggestion! We have now run the following experiments.

Setup: the bowl, or, the bowl and the plate which starts 55 - 1010 cm from its demonstration position. With true concepts the bottleneck guides the robot at least as well as Direct Action (bowl: 45.545.5 vs 28.528.5; bowl + plate: 16.816.8 vs 15.315.3). In the case of a single shared instruction (“pick up the black bowl and place it on the plate”), the concept bottleneck is far better (68.068.0 vs 28.528.5 when only the bowl moves). With predicted concepts, Spatial CBM falls below Direct Action (23.523.5 vs 28.528.5 and 3.33.3 vs 15.315.3). The concept predictor is trained on 500500 demonstrations whose layouts vary by about 11 cm per task, which affects generalization to moved objects; however, as in Table 3, we find the limitation to be the predictor and not the bottleneck itself.

MethodBowlBowl + plate
GT45.5±3.545.5 \pm 3.516.8±3.216.8 \pm 3.2
GT, one sentence68.0±1.468.0 \pm 1.425.1±2.825.1 \pm 2.8
Direct Action28.5±2.128.5 \pm 2.115.3±1.115.3 \pm 1.1
Spatial CBM*28.5±3.528.5 \pm 3.57.3±1.17.3 \pm 1.1
Spatial CBM23.5±0.723.5 \pm 0.73.3±0.43.3 \pm 0.4

We will shortly update our manuscript to clarify these details.

References for the cited works as follows. We hope the clarifications and the new experiments address your concerns. If any remain, we will be more than happy to clarify before the discussion period closes.

References:

  • Agia, C., et al. Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progress. CoRL 2024.
  • Almudévar, A., Hernández-Lobato, J. M., and Ortega, A. There Was Never a Bottleneck in Concept Bottleneck Models. ICLR 2026.
  • Branson, S., et al. Visual Recognition with Humans in the Loop. ECCV 2010.
  • Breiman, L. Statistical Modeling: The Two Cultures. Statistical Science, 2001.
  • Duan, J., et al. AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation. ICLR 2025.
  • Espinosa Zarlenga, M., et al. Concept Embedding Models: Beyond the Accuracy-Explainability Trade-Off. NeurIPS 2022.
  • Garg, S., et al. ObjectReact: Learning Object-Relative Control for Visual Navigation. CoRL 2025.
  • Gunning, D. and Aha, D. W. DARPA's Explainable Artificial Intelligence (XAI) Program. AI Magazine, 2019.
  • Havasi, M., Parbhoo, S., and Doshi-Velez, F. Addressing Leakage in Concept Bottleneck Models. NeurIPS 2022.
  • Koh, P. W., et al. Concept Bottleneck Models. ICML 2020.
  • Lampert, C. H., Nickisch, H., and Harmeling, S. Learning to Detect Unseen Object Classes by Between-Class Attribute Transfer. CVPR 2009.
  • Liu, Z., Bahety, A., and Song, S. REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction. CoRL 2023.
  • Mahinpei, A., et al. Promises and Pitfalls of Black-Box Concept Learning Models. arXiv:2106.13314, 2021.
  • Ren, A. Z., et al. Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners. CoRL 2023.
  • Savinov, N., Dosovitskiy, A., and Koltun, V. Semi-Parametric Topological Memory for Navigation. ICLR 2018.
  • Shah, D., et al. GNM: A General Navigation Model to Drive Any Robot. ICRA 2023.
  • Xu, C., et al. Can We Detect Failures Without Failure Data? Uncertainty-Aware Runtime Failure Detection for Imitation Learning Policies. RSS 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.