acceptodds
Under review as a conference paper at ICLR 2027

Learning to Teach, Learning to Play: Dual Reinforcement Learning with Active Trajectory Inquiry

Abstract

Foundation models can guide reinforcement learning (RL) agents, but useful guidance depends on understanding how a situation arose and where it can still be changed. In dynamic games, the information needed for this judgment may be scattered across a trajectory and become apparent only during investigation, making it difficult to specify a sufficient context in advance. We introduce TrajTeach, a dual reinforcement learning framework that trains a foundation-model teacher to investigate and improve an RL student's behavior. We develop TrajQL, a structured trajectory-query language through which the teacher locates relevant events, state changes, and objects, follows up on its findings, and decides whether, where, and how to intervene. Return gains over the student's continuation train the teacher's inquiry and intervention policy, while the resulting experience trains the student. Trajectory analyses across six games illustrate how inquiry connects observed problems to actionable corrections, including tracing delayed failures to earlier decisions, recovering missing task prerequisites, and revising action timing. Only the student is deployed, preserving its original inference cost.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Vision-Language Models, Active Information Acquisition, Trajectory Reasoning, Teacher–Student Learning, Game Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HK5uOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents one new framework which defines one bi-level optimization over the VLM foundation model as agent and the RL policy of games, such that the VLM agent uses TrajQL query to locate the failure stage of the policy and make useful corrections which can lead high reward demonstrations for train the RL policy, and the VLM agent can use all intervention results to finetune its query ablility.

Critical strengths
  1. The problem is interesting and the proposed algorithm is very intuitive: to use foundation model to find where is the problem and how to solve it, so the RL policy can receive good off policy trajectory to train instead of random explorations based on the policy itself.

  2. The performance is superior than baselines.

Critical weaknesses

The idea is nice, but scalability is one big challenge. The paper's six games are all pixel games with very finite discrete action spaces. These are jobs that one VLM can give reasonable and effective interventions. However, for more challenging game, for instance, if the action is continuous, select the action itself is very challenging since the foundation model needs to understand more physical laws and mapping between the real state input to the policy and physical movements. Also, the SQL style query information can not be directly transfer to other games. Another limitation is from the paper the computation time by involving the entire training algorithm is 20 times more than original Rainbow, and the paper does not give one training wall-time axis for the evaluation.

Questions for authors
  1. Can you provide one practical example for one different type of the game besides these pixels games? For instance, for the game such as bipedalwalker, how VLM in the pipeline can work to help the policy cross the stump?

  2. I want to know fair comparison such as under the same training time, how your method is better than traditional RL with its random exploration without prior knowledge across baselines.

2 comments

Thanks for the additional experiments and clarifications. I will further discuss with other reviewers to determine my final score.

AuthorsOpenReview ↗

Thank You for Your Follow-up and Consideration

Thank you very much for considering our additional experiments and clarifications, and for taking them into the discussion with the other reviewers. For convenience, the two results most relevant to your concerns are: (i) on continuous control, TrajTeach applied to BipedalWalkerHardcore with a SAC student improves stump crossing from 41% to 78% (Table 19), and the stump-crossing example is now in the main text (Figure 4(g)); (ii) at equal training time, Rainbow and Rainbow + RND remain below TrajTeach on Frostbite and Seaquest while using 4.6–4.8× more environment interactions (Figure 7(c), Table 11). We would be happy to clarify anything further that may help the discussion. Thank you again for your time and constructive feedback.

Reviewer_r26fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors propose a dual RL setup in which we have a vision-language model as a teacher and a small deep RL agent we are trying to learn in the games of Atari, mario, BabyAI etc. The agent is trying to distill some of the exploration knowledge from the VLM into it to help it become a strong agent. The VLM is not static and also learned. Interestingly, here the teacher can look through the agent's recorded games as if it were footage and use a small SQL-like language (which they call TrajQL) to pick up and suggest moments as interventions, and from those pivotal intervention points. Interestingly, here the teacher can look through the agent's recorded games and use a small SQL-like language which they call TrajQL to pick up and suggest pivotal moments as interventions with suggestion on what to do instead. Then we have a comparison of reward with and without this suggestion from this pivotal point. The teacher is trained with GRPO too with improvement on student as the reward. The authors show clear gains over Rainbow and other LLLM based baselines

Critical strengths
  • The paper has a well written motivation that instead of handing the teacher a given fixed context, it can query the past and then suggest interventions.
  • Using SQL-like TrajQL to let the agent search the past of the agent's game state is fun and clever.
  • The reward design is clean and well done.
  • Rolling the env from those chosen moments into two branches, with and without suggestions for 256 steps with matching seeds and averaged over 3 trials gives confidence in experimental setup
  • worked examples for the river raid case makes the whole loop easy to follow and build intuition
Critical weaknesses
  • In this work, the teacher has a lot of privileged access to the decoded game state and memory and can rewind the simulator, and it's unclear how much of this gain comes from this privileged inquiry vs from this priviledged access
  • It's further unclear whether the baselines get the same privileged access ie its not clear if other LLM teacher baselines also get this rewind decoder state and controller access.
  • The method uses much more compute around 17x GPU hours as compared to plain Rainbow with no compute-matched baseline.
  • Toy game aside, it is unclear what this teaches us about training stronger agents in realistic settings & environments and why not just use a strong off-the-shelf model.
Questions for authors
  • How does a strong off-the-shelf open VLM used without GRPO training compare with this trained teacher? Basically, is the GRPO cost worth it?
  • In Table 3 Rainbow when continued for 10 million steps reaches a 1958 score on Frostbite, but from scratch it reaches about 5K in Figure 3. Can you explain this gap?
Reviewer_WtHuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces TrajTeach, a dual-RL framework in which a foundation-model teacher actively queries an RL student’s recorded trajectories, selects an earlier actionable state, and intervenes with a short correction. The teacher is optimized using paired return gains against the student’s continuation; the student learns from resulting transitions and is deployed without the teacher. The idea is compelling, and results across six game environments show strong learning improvements over Rainbow, RND, adapted teacher baselines, and a fixed-context variant.

Critical strengths

The problem is well motivated: useful guidance often depends on tracing an observed failure to an earlier decision, which static trajectory summaries cannot reliably expose.

TrajQL provides a reasonably concrete interface for active temporal investigation, with conditional retrieval, temporal references, and multiple modalities. The paired evaluation protocol is a sensible way to assign teacher credit relative to the student policy that would otherwise act from the same restored state.

The paper includes useful ablations separating active retrieval, content-conditioned retrieval, intervention timing, and teacher learning. Evaluation appropriately measures the student with the teacher disabled, supporting the claim that improvements are distilled into the student. The authors report nontrivial resource accounting rather than presenting teacher assistance as free.

Critical weaknesses

Generality is substantially less clear than the framing suggests. Each environment requires a game-specific state adapter, event schema, restoration mechanism, eligibility logic, and sometimes specialized local controllers or simulator search. The gains may rely heavily on this engineering rather than on a generally transferable “foundation-model teacher.”

The action interface is a major confound. In Frostbite, Seaquest, and BabyAI, the teacher can invoke feedback-driven or search-based controllers. These capabilities can make an intervention much stronger than a language-model-selected short action sequence, and the comparison to prior teacher-guidance baselines may not isolate the contribution of active inquiry.

The computational overhead is high: the Frostbite configuration uses an 8B VLM, multi-turn querying, 16 proposals per context, three paired trials, state restoration, and roughly 62 GPU-hours per seed versus 3.6 for Rainbow. Equalizing simulator calls does not establish comparable practical training cost. The main plots show means and intervals, but the paper provides limited numerical uncertainty reporting for the headline cross-game comparisons. With five training seeds and complex nonstationary joint training, stronger per-task statistical reporting would improve confidence.

The paired teacher reward is based on a finite continuation horizon. For delayed-effect settings such as River Raid, a locally helpful correction could be mis-scored if its longer-term costs or benefits occur outside that horizon.

Questions for authors
  1. How much of the improvement remains when every method is restricted to primitive actions only, with no game-specific controller, simulator planning, or feedback-based execution tool?

  2. What is required to port TrajTeach to a new environment? Please quantify per-environment engineering effort and test whether a state schema, query grammar, and action interface developed for one game transfer meaningfully to another.

1 comment
AuthorsOpenReview ↗

Gentle Follow-up on Our Response

Dear Reviewer,

Thank you again for your detailed review. We wanted to gently follow up on our response and the revised PDF, in which all changes are marked in blue. In brief, regarding your questions: with controllers disabled for every method, TrajTeach still outperforms all baselines (Q1; Figure 7(b), Table 10); porting a new game required only a 230-line adapter and no change to TrajQL (Q2; Tables 17–18); and baselines given TrajTeach's GPU-hours or wall-clock time remain below it (W3; Figure 7(c), Table 11). We would greatly appreciate it if you could let us know whether these address your concerns, or whether any questions remain that we can clarify. Thank you for your time.

Comments on the paper

1 comment

Dear reviewers,

The authors have responded to your initial reviews in a timely fashion. Kindly go through and acknowledge these responses and follow up with any questions or comments you may have. Specifically, you may highlight issues that have been addressed by the author responses, as well as those that require further analysis/clarifications.

Thanks,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.