acceptodds
Under review as a conference paper at ICLR 2027

Evaluating Semantic State Response and Generated Visual Feedback under Distinct Input Contracts

Abstract

Interactive video systems must do more than predict action-conditioned states: their generated observations may become inputs to later decisions. We introduce State–Feedback Separation (SFS), a contract-based reporting schema for factual semantic response (S), no-anchor generated feedback (G), privileged-reference dependence (A), and transfer stress (T), illustrated by a bounded case study and mapping diagnostic. In inspected, scene-disjoint AI2-THOR FloorPlan5 evaluations, a post-hoc ResNet-50 probe reaches source/target accuracy 0.750/1.000, but an action-only rule reproduces all 160 target labels. Two distinct no-anchor RGB routes fail their gates: an object-aware predictor scores 0.375–0.500 with nonpositive gain, and a ResNet-50-guided full-resolution editor scores 0.375–0.500 across five seeds, also with nonpositive gain. Both are weak with factual histories, so these results do not isolate damage from feedback. In a separate anchor diagnostic, a distinct editor scores 200/200 with its factual anchor versus 100/200 with a blank anchor; one anchor-dropout variant preserves the pattern (0/5 seeds meet criterion). Transfer gates fail on Toaster (0/20), StoveKnob (1/20), and Kettle (1/20); oracle-goal inputs raise all tested cells to 1.000, localizing failure to goal inference in this controller pipeline. Both EPIC-KITCHENS routes are negative (0/5). On a separate controlled RGB task, post-hoc calibration yields 0.719 success (gain 0.250 over factual copy) across five frozen decoders, but passes only 1/5 fresh-training seeds. S and G use different systems. The results do not establish a within-checkpoint dissociation, internal causal mechanism, broad cross-architecture or natural-video generality, or a complete interactive world model.

Primary area:
reinforcement learning
Keywords:
feature superposition, sparse autoencoders, representation learning, learning theory, statistical detection, sparse coding, dictionary learning, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xYCfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces State–Feedback Separation (SFS), a four-axis framework for evaluating semantic prediction, generated visual feedback, privileged-reference dependence, and transfer in interactive video systems. Experiments are primarily conducted in AI2-THOR, with additional evaluations on EPIC-KITCHENS and a controlled RGB task. Most results are negative or heavily dependent on privileged inputs.

Critical strengths

The authors transparently report negative results and acknowledge important experimental limitations. The anchor-ablation and controller-invariance analyses offer some useful diagnostics, although their broader scientific significance remains limited.

Critical weaknesses
  1. Insufficient novelty and significance. The proposed SFS framework largely repackages established evaluation dimensions. The authors acknowledge that it is neither a new metric nor a validated benchmark. It is unclear what concrete methodological advancement or genuinely new scientific insight this work provides.
  2. Fundamentally unconvincing experimental evidence. The central semantic–feedback comparison uses different systems, preventing a controlled interpretation. The perfect semantic target accuracy is reproduced by an action-only rule (160/160), while the generative models perform poorly even with factual histories. Consequently, the experiments do not isolate feedback-induced degradation. More concerningly, Section 5.5 shows that replacing generated observations changes none of 960 compared controller decisions under fixed goals, raising serious questions about whether the evaluation meaningfully measures feedback utility.
  3. Weak positive findings. The strongest restoration results rely on privileged factual anchors, with performance dropping from 200/200 to 100/200 upon anchor removal. Transfer experiments largely fail, and the additional calibrated RGB result passes only 1/5 fresh-training seeds. These findings offer little compelling evidence beyond the limitations of the particular experimental pipeline.
  4. Unacceptably weak visual presentation. The figures fall substantially below the presentation standards expected at ICLR. Figure 1 is overcrowded with tiny images, excessive text, and poorly organized information. Figure 2 adds little beyond restating numerical results. More importantly, a paper centered on generated visual feedback provides almost no meaningful qualitative comparisons of generated sequences or their failure modes. This is not merely a cosmetic issue: the lack of informative visual evidence makes the central evaluation difficult to assess. The figures require substantial redesign. Overall, the manuscript resembles an extensive collection of experimental audit records rather than a coherent research contribution. Extensive documentation cannot compensate for limited novelty and unconvincing experimental evidence.
Questions for authors

1.What concrete methodological novelty does SFS offer beyond existing interactive world-model evaluation protocols? 2.Why is there no matched factual-versus-generated history evaluation using the same competent generative model? 3.Given the controller-invariance results, can the authors demonstrate that generated observations meaningfully influence downstream decisions?

Reviewer_jgRcOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose a State-Feedback Separation (SFS) reporting framework. The validation phase primarily focuses on the AI2-THOR task, and includes experiments on EPIC-KITCHENS and controlled RGB calibration. Most results are negative or accompanied by strong qualifiers.

Critical strengths
  1. The authors disclosed scene contamination, inconsistencies in thresholds, and statistical limitations. The applicability boundaries of each conclusion were clearly stated.
  2. The authors ruled out other possible explanations based on anchor bank ablation, action-only auditing, and oracle-goal diagnostics.
Critical weaknesses
  1. The authors' experiments fail to validate their claims. The claim is that performance under one contract cannot imply performance under another, but the authors admit they did not conduct experiments to prove this necessity. It's uncertain whether the same model will exhibit separation under S and G, and the authors acknowledge that S and G use different systems. Therefore, the framework's significance primarily remains at the level of its claims.
  2. The evaluation system is weak, and the conclusions fail to reflect the framework's effectiveness. Feedback impairment was not measured on the G-axis in this paper, and the authors did not actually apply SFS to existing world models or video generation models, failing to demonstrate that SFS can reveal what existing benchmarks have missed.
  3. The manuscript is not well-organized. Numerous exploratory experiments were included in the main text and appendices without screening. Terminology drift, undefined numbering, and internal audit records are mixed into the manuscript, making it difficult for readers to extract the core ideas.
Questions for authors
  1. Can the authors report S and G simultaneously at the same checkpoint?
  2. Can SFS be applied to at least one open-source motion-conditional video world model, and explain what omissions in existing benchmarks it explains?
Reviewer_aZhNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed a State-Feedback Separation model for eval of video models with diffrent inputs: 1. factual semantic state response, 2. no anchor generated visual feedback, 3. privileged anchor reference, 4. transfer stress to unseen object.

The paper is inference on  binary object state changes in AI2-THOR across 10 scenes. The proposed paper shows that it gets 1.0 target accuracy on factual semantic state response. In no anchor generated visual feedback the proposed paper gets 0.375 to 0.5 accuracy. In privileged anchor reference it gets 1.0 accuracy when given a privileged state-1 anchor image. The proposed method also shows offline affine dominance certificate which proves that the controller second action doesn't change when the frame replaced.

Critical strengths
  1. The paper give valid evaluation for interactive video benchmarks. and shows that 1.000 target accuracy is came from the action input. And also shows that anchor conditioned editor is fail when the privileged anchor is blanked out.

  2. They shows that in E2 the controller action score is affine,  which the new generation frame dont change the  decisions.

Critical weaknesses
  1. The paper readability poor and hard to read. for example there are unnecessary  details like "research_closure_consistency_v98 (130/130)". Section 5.7 write about "separate controlled reactive RGB benchmark" without explaining it.

  2. The paper eval on model that trained on 256 or 384 examples which gives poor results. In table 3 the results are  0.463 and 0.256 accuracy even with ground truth factual history.

Questions for authors
  1. Evaluate at least one existing pretrained video world model so that we can observe the degradation on a working model.

  2. explain what is "controlled reactive RGB benchmark" in section 5.7

  3. explain what is task and action space and from where did you choose the floor(step + 0.57).

  4. In section 4 you write about "target-state-gated anchor skip"; please explain it - the exact architecture of this model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.