- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this work, the authors introduced RISE, which is a system that is able to automatically creates playable visual games, tasks related as well as step-ty-step demonstrations using observations and feedback rather than relying on expensive AI teacher calls at each step. Testing results across various games show that learnt cues from generated tasks helps AI agents to perform better than those using standard methods.
Critical strengths
1: In this work, a core limitation in current demonstration system is identified: relying on internal simulator rules or each step LLM teacher calls to address the limitation with a framework to collect demonstrations only using observations and interaction feedback. 2: The evaluation strategy presented in this work is well-rounded. It incorporates structural checks (i.e., construction audits across 9 games for execution, evidence provenance and export consistency) in parallel with empirical performance tests (i.e., a 16 game study demonstrating improved task execution standard data recipes).
Critical weaknesses
1: Even though the work is quite novel and relative improvements over baselines, the success rate is low, e.g., reaching only 19.32% for the trained 8B model and 17.33% for the 4B model, indicating that these long-horizon visual tasks are still mainly unsolved. All this make the work no practical application. 2: The primary data-recipe evaluation is conducted in this paper still uses an older, 16-game implementation developed before the key evidence-conditioned goal constructor was introduced. 3: The method only works well if you carefully shift training data toward exploration and recovery rather than just basic execution. This indicates the framework relies fairly heavy on manual tuning and setup rather than a plug-and-play solution.
Questions for authors
1: The expert solver uses exact symbol decoding and bounded hypothesis search that are excluded from the visual agent. To what extent is the visual agent learning genuine causal reasoning versus mimicking the specific trajectory patterns produced by the symbolic solver? 2: The main experiment uses an older 16-game setup that was built before the new evidence-based goal constructor was created. Can the authors test the complete, updated version of RISE for this main study?