- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents VIES (Video-Induced Exploration Space), a way to guide the search for programmatic game policies with behavior learned from gameplay videos. It builds on Bilevel Synthesis (Bi-S). In Bi-S, a local search over programs often finds many candidates with the same return, and a behavior target is used to choose among them. The behavior space in Bi-S is made of hand-designed game features. VIES replaces these features with a VQ video tokenizer trained on unlabeled gameplay videos, which describes each program's execution as a short sequence of discrete codes. This gives Bi-S the finite behavior space it needs, and it works with both DSL programs and LLM-written Python programs. In MicroRTS, the trained tokenizer finds programs that beat the opponent in almost all searches and gives the best self-play results. In Can't Stop, both the trained and the untrained tokenizer do better than the hand-designed features.
Critical strengths
-
The idea is simple and fits Bi-S well. A VQ tokenizer gives the finite, factorized space the CMAB needs, and it works with both DSL mutation and LLM-written programs. On CombatMap, where the handcrafted features are all zero, both VIES variants still beat No guide, which is a good case for replacing hand-made features.
-
The experiments are extensive and carefully designed. All conditions share the same search code, self-play results come with confidence intervals, and the opponent bot is kept out of the tokenizer data. The paper also reports where VIES loses.
Critical weaknesses
-
It is unclear whether the gain comes from learning or from the discrete abstraction. An untrained tokenizer already gets much of the gain in MicroRTS and all of it in Can't Stop. There is no comparison with a simple non-learned summary, such as unit counts on the same grid, or with the features before quantization. Appendix A.8 also suggests that the acceptance rule is not very sensitive to similarity, so part of the LLM gap may come from accepting more ties.
-
The MicroRTS results compare the methods only against one opponent and against each other. Best-response uses a single opponent, and success appears to mean winning one deterministic match, so the small differences from Handcrafted are not conclusive. Self-play win rates are measured only among the four conditions, so they say little about how strong the programs are in absolute terms.
Questions for authors
- Is Raw video computed on the same input as VIES, with repeated frames removed and 128 frames kept? Appendix A.3 says its padding differs from the tokenizer preprocessing. If the input is different, part of its poor result may come from the preprocessing rather than from the lack of a tokenizer.
- Which map do the LLM runs use? I could only infer workers 24x24 from Section 5.2.2. Do the found programs also win when playing as player 1?
2 comments
Thank you for clarifying many of my questions, and I would keep my positive score.
Response to Reviewer PZ81
Comment: It is unclear whether the gain comes from learning or from the discrete abstraction. An untrained tokenizer already gets much of the gain in MicroRTS and all of it in Can’t Stop. There is no comparison with a simple non-learned summary, such as unit counts on the same grid, or with the features before quantization.
Answer: Our comparison between Trained VIES and Untrained VIES isolates the effect of representation learning, since both conditions use the same architecture and quantization mechanism. The improvement of Trained VIES over Untrained VIES in MicroRTS demonstrates that learning contributes beyond discretization. We also compare against the handcrafted behavioural representations used in previous work, which include unit counts on the same grid, showing that VIES can outperform manually specified features. Finally, the bilevel search algorithm requires a discrete representation, as it treats the features as arms in a bandit problem, which makes it unclear how to use the features before quantization.
Comment: Appendix A.8 also suggests that the acceptance rule is not very sensitive to similarity, so part of the LLM gap may come from accepting more ties.
Answer: Thank you for giving us the chance to clarify this. We know that similarity plays an important role due to the large gap between VIES and No Guide. They both use Simulated Annealing, but the former uses similarity from our learned abstractions to break ties. VIES accepts more ties from the win-draw-lose metric, but it does so in a principled way, by choosing carefully which ties to accept based on the similarity metric.
Comment: The MicroRTS results compare the methods only against one opponent and against each other.
Answer: This is an important point, which we clarify in our revised version. The system using the handcrafted features was shown to defeat many other agents from the MicroRTS codebase (cf. Table 1 of Aleixo and Lelis, 2023). In our work, we show that our system with learned features can defeat this agent that was shown to defeat many others. This allowed us to focus on the research question of whether features learned from videos can be used to guide the process of synthesizing programmatic policies with a direct comparison to the system of Aleixo and Lelis. We added a paragraph, “Indirect Comparisons”, in Section 5.1 of the revised version to clarify this.
Question: Is Raw video computed on the same input as VIES, with repeated frames removed and 128 frames kept?
Answer: Good point about the pre-processing of the videos with the raw video baseline. We reran the raw video baseline with the same pre-processing used with VIES and have updated the plots in Figures 2 and 3. The results are qualitatively similar to our original submission, with raw video performing worse than the other systems we evaluated.
Question: Which map do the LLM runs use? I could only infer workers 24x24 from Section 5.2.2. Do the found programs also win when playing as player 1?
Answer: We used the Workers 24x24 map on the LLM-based experiments, and we had the synthesized programs play as player 0. We have not evaluated the programs as player 1, but we would expect a performance drop. In MicroRTS, a player's position on the map could influence the strategy required to win. Our choice of player 0 was arbitrary, and we expect the agent to perform well as player 1 if given the chance to learn to play in that position. We added a sentence to the "LLM-based synthesis" paragraph in Section 5.1 clarifying the map used and the position played.