- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents one new framework which defines one bi-level optimization over the VLM foundation model as agent and the RL policy of games, such that the VLM agent uses TrajQL query to locate the failure stage of the policy and make useful corrections which can lead high reward demonstrations for train the RL policy, and the VLM agent can use all intervention results to finetune its query ablility.
Critical strengths
-
The problem is interesting and the proposed algorithm is very intuitive: to use foundation model to find where is the problem and how to solve it, so the RL policy can receive good off policy trajectory to train instead of random explorations based on the policy itself.
-
The performance is superior than baselines.
Critical weaknesses
The idea is nice, but scalability is one big challenge. The paper's six games are all pixel games with very finite discrete action spaces. These are jobs that one VLM can give reasonable and effective interventions. However, for more challenging game, for instance, if the action is continuous, select the action itself is very challenging since the foundation model needs to understand more physical laws and mapping between the real state input to the policy and physical movements. Also, the SQL style query information can not be directly transfer to other games. Another limitation is from the paper the computation time by involving the entire training algorithm is 20 times more than original Rainbow, and the paper does not give one training wall-time axis for the evaluation.
Questions for authors
-
Can you provide one practical example for one different type of the game besides these pixels games? For instance, for the game such as bipedalwalker, how VLM in the pipeline can work to help the policy cross the stump?
-
I want to know fair comparison such as under the same training time, how your method is better than traditional RL with its random exploration without prior knowledge across baselines.
2 comments
Thank You for Your Follow-up and Consideration
Thank you very much for considering our additional experiments and clarifications, and for taking them into the discussion with the other reviewers. For convenience, the two results most relevant to your concerns are: (i) on continuous control, TrajTeach applied to BipedalWalkerHardcore with a SAC student improves stump crossing from 41% to 78% (Table 19), and the stump-crossing example is now in the main text (Figure 4(g)); (ii) at equal training time, Rainbow and Rainbow + RND remain below TrajTeach on Frostbite and Seaquest while using 4.6–4.8× more environment interactions (Figure 7(c), Table 11). We would be happy to clarify anything further that may help the discussion. Thank you again for your time and constructive feedback.
Thanks for the additional experiments and clarifications. I will further discuss with other reviewers to determine my final score.