acceptodds
Under review as a conference paper at ICLR 2027

RESCENE: Reasoning over Scene Edits for Verifiable Visual-Language-Action Grounding

Abstract

Vision-language-action (VLA) models unify perception, reasoning, and control by mapping visual observations and language instructions to actions. When responding to language-defined situations that are not directly observed, direct language–action supervision alone cannot distinguish visual grounding from an instruction-to-action shortcut. We propose RESCENE, a framework that makes V-L-A alignment through language-specified interventions in visual latent space explicit and testable. RESCENE combines two complementary, shared-weight components: a latent scene editor and a language–action response model. Given observation history, target regions, and an editing condition, the editor masks affected evidence and reconstructs visual features using context, ego motion, and spatial hints. Factual reconstruction anchors these features to observations, while counterfactual construction produces feature-space alternatives for object addition, removal, relocation, and state changes without generating pixels. The response model uses the constructed features as a shared interface for language reasoning and action prediction, without direct access to the edit condition. Response gradients stop at this interface, blocking direct policy-loss updates through latent construction and separating construction from the responses it supports. Reconstruction, language, and action are jointly learned, while normal driving retains a direct observed-feature pathway. On Bench2Drive, RESCENE improves driving score by points and route completion by percentage points over factual replay, with a higher success rate. Implemented in autonomous driving, RESCENE provides an explicit, visually mediated, and testable formulation of V-L-A alignment for language-defined scene changes. Project page: https://resceneiclr2027.github.io/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Visual Grounding, Latent Scene Editing, Counterfactual Reasoning, Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LUgdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether driving VLAs ground language-specified scene changes (e.g., “the leading vehicle is absent”) in vision, or instead learn an instruction-to-action shortcut. ReScene trains a latent scene editor that masks affected regions in frozen DINOv3 features and predicts them under a factual or counterfactual condition (ADD/DELETE/MOVE/state change), using Qwen3-0.6B with LoRA.

Critical strengths
  1. A well-motivated problem with a clean, testable design. Whether a VLA’s response to a described situation is visually grounded or a language-to-action shortcut is an important and underexplored question. Routing the condition exclusively through a constructed visual latent, with stop-gradient at the interface, turns this question into an explicit architectural hypothesis that can be tested.
  2. Practical benefit without inference cost. Counterfactual latent editing acts as a training-time augmentation that improves closed-loop driving over factual replay (+6.41 DS, +4.70 RC, +5.07 SR) while leaving the inference pipeline unchanged. The latent-path intervention indicates that the response model actually uses the constructed features.
Critical weaknesses
  1. The central “visually mediated” claim is not directly validated. The constructed latents are checked only by a frozen reader that is also a training loss, and by cycle consistency. Neither shows that the latents actually represent the edited scene. Because the editor sees the language condition, it may simply encode that condition non-visually, which would relocate the shortcut rather than remove it. The authors can re-render edited scenes in CARLA, yet they never compare the constructed latents against DINO features of those re-rendered images, even though this is the most direct test of their claim.

  2. Mixed results and limited empirical rigor. The key control, direct conditioning, outperforms ReScene on success rate, QA and joint success, and ReScene has the worst open-loop ADE on observed scenes. These results weaken the claim that visual mediation beats explicit conditioning. The closed-loop gains also come from a single training run without confidence intervals, on a benchmark known for high variance. Absolute performance (DS ≈ 49) remains far below published Bench2Drive methods, and the paper offers no comparison to them.

Questions for authors

See the weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer p8Am: Contribution and Broader Evaluation

Thank you for recognizing the potential of alternative scene representations for studying the relationship between language, perception, and action. We understand your comments on related work and evaluation as asking what RESCENE adds beyond existing methods and whether that value is sufficiently demonstrated. The question we address is not only whether a model produces an appropriate action for a language-described situation, but also whether we can examine the scene representation it uses to produce that response.

Relative to SimLingo’s focus on language–action learning and ReconVLA’s reconstruction of manipulation-relevant regions, RESCENE focuses on constructing alternative scene features that reflect language-specified changes to the same observation. These features are passed to a response model without direct access to the edit condition. Substituting them while holding the response model fixed allows us to measure changes in QA outputs and predicted trajectories. This provides a way to examine how the constructed scene representation influences the response, alongside evaluating whether the output is correct.

In our experiments, Route Completion increased from 61.55% to 67.23%, and Both-action increased from 36.15% to 42.85%, relative to Direct Conditioning. These results come from different checkpoint families and show benefits in driving progress and edited-scene action prediction, respectively. We also confirmed that substituting factual reconstructions for constructed edited features increased action error with the response model held fixed. This provides evidence that the constructed representation is used in producing the response. Both-QA and Joint Success were lower than Direct Conditioning, so we examined the action benefits together with the differences in language performance.

The submitted experiments focused on comparing conditioning pathways within our architecture, with language responses evaluated using our authored QA set. To broaden this evaluation, we are now adding RESCENE’s scene-editing component to the SimLingo model architecture and training and evaluating it on the SimLingo dataset. The purpose is to examine what changes scene editing brings to language and action responses within an external model and dataset. This addresses transfer beyond our original implementation, while broader evaluation across backbone families and model sizes remains a further validation step. Once training and evaluation are complete, we will share the verified results and the corresponding experimental conditions.

Reviewer_hR1VOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes RESCENE, a framework for visually mediated VLA alignment through latent scene editing. It constructs language-specified counterfactual scenes in visual feature space and uses the edited representations for language understanding and action prediction. The method combines factual reconstruction, semantic constraints, and cycle consistency to supervise scene editing. Experiments on Bench2Drive evaluate driving performance and responses to counterfactual scene changes.

Critical strengths
  1. The paper addresses a meaningful problem. Distinguishing visually grounded action prediction from direct language-to-action shortcuts is an interesting and relevant research direction.

  2. The latent feature substitution experiment is a useful controlled design for examining how changes in visual representations affect downstream action prediction.

Critical weaknesses
  1. The experimental results do not sufficiently support the central claim. RESCENE improves some action prediction metrics but underperforms direct conditioning on Both-QA and Joint Success. These metrics are directly relevant to the proposed V-L-A alignment, making the claimed advantage of visual mediation difficult to establish.

  2. The counterfactual representations lack direct validation. The method relies on semantic and cycle-consistency constraints, but does not directly demonstrate whether the constructed latent features accurately represent the intended scene changes. Moreover, the direct-conditioning baseline uses different supervision, making it difficult to isolate the contribution of visual mediation itself.

Questions for authors
  1. Could the authors directly evaluate the accuracy of the constructed counterfactual features, for example by comparing them with features extracted from corresponding rendered edited scenes?
  2. Could the authors provide a controlled comparison with matched supervision to better isolate the contribution of visual mediation from the additional semantic and cycle-consistency losses?
2 comments
AuthorsOpenReview ↗

Response to Reviewer LUgd: Visual Grounding and Reliability

Thank you for recognizing the testable design and the absence of additional inference cost during ordinary driving. We understand your concerns to center on whether the architecture leads to the use of visual scene information and how reliably the observed effects can be established. Our aim is to expose the intermediate representation through which a language condition influences the response, allowing its role to be examined through intervention.

In RESCENE, the response model receives constructed scene features without direct access to the edit condition. Stop-gradient blocks direct backpropagation of the response loss through feature construction. Our substitution experiments confirmed that changing these representations affects action predictions. To investigate your concern about nonvisual condition encoding, we also compared the constructed features with DINO features from rendered edited scenes, separately from the constraints used for training. Agreement varied across cases. This comparison examines the visual content of the representation, complementing the substitution experiment’s evidence that the representation is used; it does not by itself rule out nonvisual encoding.

Route Completion increased from 61.55% with Direct Conditioning to 67.23% with RESCENE, and the separate edited-scene evaluation also showed an action benefit. These evaluations use different checkpoint families. Driving success rate, Both-QA, and Joint Success were lower, while observed-scene ADE was higher. Additional QA diagnostics found incorrect responses even when using factual reconstructions or features extracted from actual edited images. These observations motivate examining the interaction between feature construction and response prediction, rather than attributing the language-performance gap to a single component.

We have reanalyzed the driving results using paired route resampling. The Route Completion difference was +5.68 percentage points, with a 95% confidence interval of [+0.63, +10.85]. The intervals for Driving Score and success-rate differences included zero. These intervals describe route variation at fixed checkpoints; variation across training runs remains a separate validation question. The relatively low absolute driving performance also makes comparison with external methods important. We are now training and evaluating an adaptation of scene editing within the SimLingo architecture using its dataset, to examine whether the benefits extend beyond our architecture and authored QA set. We will share the verified results together with the comparison settings once evaluation is complete.

Thank you for the detailed response and additional analyses. The feature substitution experiments provide useful evidence that the constructed representations influence action prediction. However, the feature-level validation remains mixed, and the matched-supervision comparison is still missing. These are important for establishing the paper's central claim about visual mediation. I will therefore keep my rating unchanged at 2.

Reviewer_p8AmOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper targets at addressing visual grounding in vision-language-action models through language-conditioned latent scene editing. It jointly trains an editor and response model using factual reconstruction, semantic constraints, and cycle consistency, while blocking response gradients through edited features. Experiments under Bench2Drive show improved driving and counterfactual action prediction over factual replay, although comparisons with direct conditioning reveal mixed gains in language and action performance.

Critical strengths
  1. The paper addresses a reasonable and potential important issue in VLA learning: correct actions under language-defined scene changes do not necessarily demonstrate visual grounding. The method that the paper introduces, constructing alternative scene representations, could provide a concrete approach to studying how language, perception, and action interact.
Critical weaknesses
  1. The current shape of the paper remains significantly under-prepared. For example, the discussions of the related work is missing. The experimental evaluation primarily compares with two internal variants, without broader comparisons to existing VLA approaches. The paper does not evaluate across multiple backbone families or model sizes, which limits evidence of the method’s generality. The effectiveness of the proposed approach remains unjustified: it cannot fully outperform a simple baseline, direct conditioning. Overall, in my view, the paper requires substantial improvement before it can be taken into the consideration of acceptance.
Questions for authors

Unfortunately, in my view, the paper is significantly below the acceptance standard in its current form, such that mild modifications cannot be sufficient. I would like to suggest a substantial re-preparation of the submission.

2 comments
AuthorsOpenReview ↗

Response to Reviewer hR1V: Feature Validation and Matched Supervision

Thank you for highlighting the importance of distinguishing visual grounding from language-to-action shortcuts and the value of the feature substitution experiment. Your questions address whether the constructed representations reflect the intended scene changes and whether the observed gains can be attributed to visual mediation. Our goal is to make the scene representation connecting a language-specified change to the model’s response directly testable.

To evaluate feature accuracy, we compared constructed features and factual reconstructions against DINO features extracted from corresponding CARLA-rendered edited scenes. In a 12-edit vehicle-and-pedestrian diagnostic, seven edits passed the geometry and repeat-render checks; five were excluded from interpretation because of size, pose, or background-asset mismatches. Over the affected regions, both relative L2 and cosine similarity improved in two interpretable edits—a vehicle deletion and a relocation—and worsened in the other five. The results therefore show correspondence in specific edits, with mixed fidelity across cases. In the vehicle-deletion example, relative L2 decreased from 0.883 to 0.760. With the response model and surrounding feature context fixed, edited-target ADE was 2.127 m with factual reconstruction, 0.416 m with constructed edited features, and 0.376 m with actual edited-image features inserted into the same region. This example links improved feature agreement to an appropriate action change.

The broader substitution evaluation provides complementary evidence of response dependence. Across 1,195 supported scenes from 30 routes, replacing constructed edited features with factual reconstructions increased edited-target ADE by 0.711 m, averaged equally across routes (95% CI: [0.332, 1.199] m), with the response model, question, and navigation held fixed. On the same supported scenes, Both-action increased from 36.15% with Direct Conditioning to 42.85% with RESCENE. Both-QA and Joint Success were lower, at 6.69% versus 14.56% and 3.93% versus 6.03%, respectively. Together, these findings establish that the constructed representations influence action prediction, while identifying differences in how they support action and language responses.

For your matched-supervision question, we are planning a controlled comparison that matches auxiliary losses and training conditions while varying the response input pathway. The current comparison evaluates the methods with their respective training objectives; it does not separately identify the contribution of the pathway and the additional supervision. The planned control addresses this attribution question, complementing the completed feature-accuracy and response-dependence analyses.

Thanks for the responses, while my justification has remained unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.