- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether driving VLAs ground language-specified scene changes (e.g., “the leading vehicle is absent”) in vision, or instead learn an instruction-to-action shortcut. ReScene trains a latent scene editor that masks affected regions in frozen DINOv3 features and predicts them under a factual or counterfactual condition (ADD/DELETE/MOVE/state change), using Qwen3-0.6B with LoRA.
Critical strengths
- A well-motivated problem with a clean, testable design. Whether a VLA’s response to a described situation is visually grounded or a language-to-action shortcut is an important and underexplored question. Routing the condition exclusively through a constructed visual latent, with stop-gradient at the interface, turns this question into an explicit architectural hypothesis that can be tested.
- Practical benefit without inference cost. Counterfactual latent editing acts as a training-time augmentation that improves closed-loop driving over factual replay (+6.41 DS, +4.70 RC, +5.07 SR) while leaving the inference pipeline unchanged. The latent-path intervention indicates that the response model actually uses the constructed features.
Critical weaknesses
-
The central “visually mediated” claim is not directly validated. The constructed latents are checked only by a frozen reader that is also a training loss, and by cycle consistency. Neither shows that the latents actually represent the edited scene. Because the editor sees the language condition, it may simply encode that condition non-visually, which would relocate the shortcut rather than remove it. The authors can re-render edited scenes in CARLA, yet they never compare the constructed latents against DINO features of those re-rendered images, even though this is the most direct test of their claim.
-
Mixed results and limited empirical rigor. The key control, direct conditioning, outperforms ReScene on success rate, QA and joint success, and ReScene has the worst open-loop ADE on observed scenes. These results weaken the claim that visual mediation beats explicit conditioning. The closed-loop gains also come from a single training run without confidence intervals, on a benchmark known for high variance. Absolute performance (DS ≈ 49) remains far below published Bench2Drive methods, and the paper offers no comparison to them.
Questions for authors
See the weakness
Response to Reviewer p8Am: Contribution and Broader Evaluation
Thank you for recognizing the potential of alternative scene representations for studying the relationship between language, perception, and action. We understand your comments on related work and evaluation as asking what RESCENE adds beyond existing methods and whether that value is sufficiently demonstrated. The question we address is not only whether a model produces an appropriate action for a language-described situation, but also whether we can examine the scene representation it uses to produce that response.
Relative to SimLingo’s focus on language–action learning and ReconVLA’s reconstruction of manipulation-relevant regions, RESCENE focuses on constructing alternative scene features that reflect language-specified changes to the same observation. These features are passed to a response model without direct access to the edit condition. Substituting them while holding the response model fixed allows us to measure changes in QA outputs and predicted trajectories. This provides a way to examine how the constructed scene representation influences the response, alongside evaluating whether the output is correct.
In our experiments, Route Completion increased from 61.55% to 67.23%, and Both-action increased from 36.15% to 42.85%, relative to Direct Conditioning. These results come from different checkpoint families and show benefits in driving progress and edited-scene action prediction, respectively. We also confirmed that substituting factual reconstructions for constructed edited features increased action error with the response model held fixed. This provides evidence that the constructed representation is used in producing the response. Both-QA and Joint Success were lower than Direct Conditioning, so we examined the action benefits together with the differences in language performance.
The submitted experiments focused on comparing conditioning pathways within our architecture, with language responses evaluated using our authored QA set. To broaden this evaluation, we are now adding RESCENE’s scene-editing component to the SimLingo model architecture and training and evaluating it on the SimLingo dataset. The purpose is to examine what changes scene editing brings to language and action responses within an external model and dataset. This addresses transfer beyond our original implementation, while broader evaluation across backbone families and model sizes remains a further validation step. Once training and evaluation are complete, we will share the verified results and the corresponding experimental conditions.