- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces State–Feedback Separation (SFS), a four-axis framework for evaluating semantic prediction, generated visual feedback, privileged-reference dependence, and transfer in interactive video systems. Experiments are primarily conducted in AI2-THOR, with additional evaluations on EPIC-KITCHENS and a controlled RGB task. Most results are negative or heavily dependent on privileged inputs.
Critical strengths
The authors transparently report negative results and acknowledge important experimental limitations. The anchor-ablation and controller-invariance analyses offer some useful diagnostics, although their broader scientific significance remains limited.
Critical weaknesses
- Insufficient novelty and significance. The proposed SFS framework largely repackages established evaluation dimensions. The authors acknowledge that it is neither a new metric nor a validated benchmark. It is unclear what concrete methodological advancement or genuinely new scientific insight this work provides.
- Fundamentally unconvincing experimental evidence. The central semantic–feedback comparison uses different systems, preventing a controlled interpretation. The perfect semantic target accuracy is reproduced by an action-only rule (160/160), while the generative models perform poorly even with factual histories. Consequently, the experiments do not isolate feedback-induced degradation. More concerningly, Section 5.5 shows that replacing generated observations changes none of 960 compared controller decisions under fixed goals, raising serious questions about whether the evaluation meaningfully measures feedback utility.
- Weak positive findings. The strongest restoration results rely on privileged factual anchors, with performance dropping from 200/200 to 100/200 upon anchor removal. Transfer experiments largely fail, and the additional calibrated RGB result passes only 1/5 fresh-training seeds. These findings offer little compelling evidence beyond the limitations of the particular experimental pipeline.
- Unacceptably weak visual presentation. The figures fall substantially below the presentation standards expected at ICLR. Figure 1 is overcrowded with tiny images, excessive text, and poorly organized information. Figure 2 adds little beyond restating numerical results. More importantly, a paper centered on generated visual feedback provides almost no meaningful qualitative comparisons of generated sequences or their failure modes. This is not merely a cosmetic issue: the lack of informative visual evidence makes the central evaluation difficult to assess. The figures require substantial redesign. Overall, the manuscript resembles an extensive collection of experimental audit records rather than a coherent research contribution. Extensive documentation cannot compensate for limited novelty and unconvincing experimental evidence.
Questions for authors
1.What concrete methodological novelty does SFS offer beyond existing interactive world-model evaluation protocols? 2.Why is there no matched factual-versus-generated history evaluation using the same competent generative model? 3.Given the controller-invariance results, can the authors demonstrate that generated observations meaningfully influence downstream decisions?