- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper aims to make robot policies easier to inspect and correct by having them first predict spatial information that people can understand, then use that information to generate actions. For navigation, this information is a map of distances (a costmap); for manipulation, it is the positions of the object and the placement target relative to the gripper. The authors compare several training approaches on navigation and pick-and-place tasks, and test whether editing this information can correct failures.
Critical strengths
-
A small amount of spatial information can support good manipulation performance. Using six ground-truth relative-position values, together with the instruction and the robot's own state, achieves an 84.8% success rate, close to the 83.8% achieved by the model that uses images directly. (Section 4.2, Table 3.) This shows that, for the tasks tested, visual input can be compressed into information that people can inspect while retaining good control performance.
-
Editing this information can change behavior in the intended way. Correcting the predicted positions raises Spatial CBM's success rate from 75.0% to 79.8%. When the target is changed to another identical bowl, the robot completes the task with the new target in 93.5% of episodes. (Section 4.4, Figure 5, Table 3.) This provides direct evidence that the interface can be used to correct errors or change the target.
Critical weaknesses
-
Two key steps in the navigation method are not explained clearly enough to assess the main results. First, Equation (2) defines the concept as the straight-line distance to the subgoal, but Section 4.1 discusses the shortest-path distance around obstacles and says the model needs to learn that distance. The two distance types lead to substantially different results, yet the paper does not clearly state which one is used to train the main models. Second, Equation (4) describes intervention as changing a target position and recomputing the costmap, but the VLM in Section 4.3 only outputs “left,” “center,” or “right.” The paper does not explain how that direction is turned into an edit of the full costmap. As a result, it is hard to tell exactly what was implemented or how much ground-truth information from the simulator is needed during intervention. This is a gap in the description; whether there is also an implementation issue remains a question for the authors.
-
There are no real-robot experiments. All evaluations are conducted in simulation, as the authors acknowledge in Section 5. Since the approach relies on spatial information, it remains unclear whether the predicted concepts and interventions would work as well with real-world perception errors and imperfect robot execution. This limits the evidence for practical deployment, which is a main motivation of the paper.
Questions for authors
-
Does the navigation model learn straight-line distances or shortest-path distances around obstacles? Please clarify what each model in Table 2 uses during training and testing, and explain the relationship between Equation (2) and Section 4.1. If this is only a missing detail in the text, clarifying it would substantially reduce my concern. If different models use different definitions, their performance differences need to be explained accordingly.
-
How exactly is the costmap edited after the VLM outputs “left,” “center,” or “right”? Please describe the editing rule and identify which steps require simulator ground truth. If the simulator only decides when to intervene, while the edit itself uses information available during deployment, that would strengthen my assessment of the intervention results. Otherwise, the additional ground-truth information required should be made explicit.
3 comments
Response to Reviewer uQoZ (2/2)
Questions:
Please clarify what each model in Table 2 uses during training and testing, and explain the relationship between Equation (2) and Section 4.1.If this is only a missing detail in the text, clarifying it would substantially reduce my concern. If different models use different definitions, their performance differences need to be explained accordingly.
| Method (CBM variant) | trained on | tested on | Predictor | Avg SPL | Avg SSPL |
|---|---|---|---|---|---|
| GT-Geodesics | true costmap | true costmap | none (true depth) | ||
| GT-ED | true costmap | true costmap | none (true depth) | ||
| Direct Action (Standard) | VGGT features | VGGT features | frozen VGGT | ||
| Spatial CAM (Multitask) | VGGT features | VGGT features | fine-tuned VGGT, Eq. (2) as auxiliary | ||
| Spatial CBM* (Independent) | true Eq. (2), | predicted Eq. (2), | fine-tuned VGGT | ||
| Spatial CBM (Sequential) | predicted Eq. (2), | predicted Eq. (2), | fine-tuned VGGT |
We only ever use one single distance definition across all non-GT models.
Second, Equation (4) describes intervention as changing a target position and recomputing the costmap, but the VLM in Section 4.3 only outputs “left,” “center,” or “right.” The paper does not explain how that direction is turned into an edit of the full costmap. As a result, it is hard to tell exactly what was implemented or how much ground-truth information from the simulator is needed during intervention. This is a gap in the description; whether there is also an implementation issue remains a question for the authors.
How exactly is the costmap edited after the VLM outputs “left,” “center,” or “right”? Please describe the editing rule and identify which steps require simulator ground truth. If the simulator only decides when to intervene, while the edit itself uses information available during deployment, that would strengthen my assessment of the intervention results. Otherwise, the additional ground-truth information required should be made explicit.
The costmap is the straight-line distance from each pixel’s 3D point to the sub-goal Equation (2). To intervene effectively, we slice the costmap into “left”, “center”, and “right” interpretable (almost) equal width sectors/columns. The intervener picks a column it wants to edit. The new anchor is then set to the lowest-cost predicted pixel in the middle of that sector. The full costmap is then rebuilt as per Equation (4).
how much ground-truth information from the simulator is needed during intervention. If the simulator only decides when to intervene, while the edit itself uses information available during deployment, that would strengthen my assessment of the intervention results. Otherwise, the additional ground-truth information required should be made explicit.
The simulator is only required to decide when to intervene, that too only for the gated runs. An edit is made only when the predicted and true costmap minima fall in different sectors. We designed the edit to only use information which would be available during deployment. We really hope that this clarification strengthens the reviewer's assessment as they indicated and gets reflected through an improved rating.
We have preliminary real-world results on a Unitree Go2 in a home, ran with open loop on recorded data. We record a mapping run and three repeat runs of the same route. We use the frames from these three runs to see where we can intervene.
The setup is the same as Sections 4.1 and 4.3, except the sub-goal comes from A* on the robot's lidar floor map instead of the navigation mesh. The model is the same one from the paper, trained only in simulation.
We gave Claude (Opus 5.5) the Section 4.3 inputs (current view with the predicted costmap, sub-goal frame, destination) on 8 frames. It reasoned for an edit on 3 of them, and each edit moved the costmap minimum where Claude intended. In one, the minimum was on a bed and Claude answered "center", since "the white air purifier from the destination picture stands on open floor at the right edge of the center region". The examples are in Section D of the revised supplementary material.
These qualitative results show that the spatial concept can be predicted and edited on real-world runs, and that an intervener can use it to correct the robot.
Response to Reviewer uQoZ (1/2)
We thank the reviewer for their suggestions. The gaps are description gaps, and the following should clarify.
Weaknesses:
This is an inconsistency in the text only and not a difference between models. An earlier version of our method supervised the concept with geodesic distance. We later switched to the Euclidean distance of Eq. (2), because then the costmap can be computed from 3D points alone and needs no navigation mesh, making our system readily deployable. Section 4.1 still carried sentences from the earlier version, and we have revised it (updated paragraph below).
Results. Table 2 ablates the Spatial CBM layer as defined in Section 3.2. GT-Geodesics clearly establishes that action prediction using the oracle ceiling for true spatial concept values for both training and testing yields nearly perfect results.
This impliesGT-ED results imply that having the spatial concept as an intermediary is in itself not a performance bottleneck, rather it is the lack of the ability to predict perfect concepts.We also report results for GT-ED which mainly indicateGT-ED results do indicate that geodesic distances provide a better signal than Euclidean distances, and that the perception backend, VGGT, which is trained to learn Euclidean geometry,doeswould require learning geodesic geometry via fine-tuning to be performant on the navigation tasks. However, we observed that ED concept based supervision performed closely with its true ED counterpart. This is attributed to the use of GT subgoals which are derived from geodesic distances; thus, for our subgoal-conditioned navigation tasks, ED suffices.We use the shortest-path (geodesic) distance to create the subgoals from the navmesh. Once subgoals are created, we train our models (Spatial CBM, Spatial CBM*, Spatial CAM) on the straight-line (Euclidean) distance to the subgoal, following Equation (2).
We are currently setting up real world experiments and hope to share preliminary results at the earliest.