- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
LatentGym is a controllable testbed for studying how LLM agents learn across related tasks. Its seven text-game environments use explicit latent variables to define structure shared by tasks in a sequence. The paper uses agent switching to diagnose the usefulness of collected experience and an agent’s ability to act on inherited experience. It also compares sequence-level cross-task RL with single-task RL. Across the main training environments, cross-task RL improves cumulative reward, with additional analyses of held-out latents, held-out environments, matched episode budgets, and the removal of shared structure.
Critical strengths
- Useful controlled setting for a difficult question. Explicit shared latents make it possible to vary what information can transfer between tasks and to test whether performance depends on that structure. This gives the benchmark more diagnostic value than task sequences whose relationship is only implicit.
- Substantial empirical validation within the proposed setting. The gains appear across two model families and are supported by held-out-latent evaluation, a matched task-instance and episode-budget comparison, and a shared-latent ablation (Tables 2, 4, 21, and 24). These controls make the central result—that sequence-level training helps in these environments—reasonably convincing.
Critical weaknesses
- The evidence for general transfer remains limited. The environments are synthetic games, and the held-out-environment study covers only four of them. The OOD-2 improvement is modest for Qwen3-8B and largely absent for Gemma3-12B outside Number Guessing (Tables 5 and 16). This matters because transfer beyond the training environments is an important part of the paper’s motivation; the conclusions should be scoped accordingly.
- Agent switching is informative but does not cleanly isolate collection from utilization. Post-switch reward also depends on the continuation agent’s task-solving ability, subsequent exploration, and how the inherited history is presented, as the paper acknowledges. The diagnostic supports a relative comparison, but a stronger causal interpretation of the two gains is not established.
Questions for authors
- How do the authors interpret the different OOD-2 outcomes for Qwen3-8B and Gemma3-12B? Which claims about adaptation to unseen environments do these results support?
- Can the authors provide a control that holds the inherited history fixed while varying the continuation agent, or otherwise quantify the influence of general task-solving ability and history presentation on the switching results? If not, would they narrow the interpretation of the reported collection and utilization gains?
3 comments
Response Part 2
W2 and Q2: Interpretation of agent switching
We are grateful for this close reading of agent switching. As we note in Section 4 and Appendix B.5, it supports a relative comparison rather than a clean causal decomposition. Below, we clarify our interpretation and add two controls. Agent switching shows where one agent falls short of another, in the experience it collects or in how it uses inherited experience, and how this changes with the switch point K; on Number Guessing, for example, the collection advantage of Cross-Task RL (CT) over single-task RL (ST) grows with K (Table 5). As the reviewer notes, both gains partly reflect task-solving ability: collection gains through the first agent, since a target is revealed only when its task is solved, and utilization gains through the continuation agent. Part of this ability is itself a product of adaptation, since an agent that identifies the latent also solves later tasks more often.
-
Fixed inherited history. Replaying the same recorded histories to every continuation agent changes both gains by at most about 2 percentage points, within standard errors (first table), so the gains are robust to the choice of inherited history.
-
History-free baseline. We evaluate each continuation agent on tasks with and without the history collected by Cross-Task RL (second table). This history raises post-switch reward for both continuation agents at every switch point, and its relative benefit is larger at later switch points (from 7% at K=2 to 34% at K=8 for Cross-Task RL, and from 6% to 23–26% for single-task RL). Cross-Task RL therefore collects important information that continuation agents can use. Note that the increases in the second table, both in percentage and in absolute value, are relative to each agent's own reward without inherited history and do not by themselves isolate each agent's ability to use the history, since the two agents start from different baselines.
Table R1. ExploreGain and ExploitGain for reference C, with the original vs. fixed history.
| K | C | Explore (orig.) | Explore (fixed) | Exploit (orig.) | Exploit (fixed) |
|---|---|---|---|---|---|
| 2 | CT | −0.61±0.31 | −0.49±0.53 | 27.96±0.45 | 27.35±0.50 |
| ST | 6.74±1.41 | 6.67±1.10 | 37.42±1.53 | 36.52±1.44 | |
| 4 | CT | 0.15±0.41 | 0.29±0.86 | 14.49±0.48 | 15.30±0.80 |
| ST | 22.35±1.05 | 21.68±1.30 | 39.87±1.06 | 39.89±1.43 | |
| 6 | CT | 5.04±0.54 | 5.33±1.07 | 12.50±0.49 | 12.11±0.43 |
| ST | 26.08±1.60 | 26.73±1.54 | 35.02±1.46 | 34.88±0.96 | |
| 8 | CT | 9.79±0.86 | 8.80±0.35 | 12.29±1.03 | 13.58±0.81 |
| ST | 28.58±2.94 | 26.80±2.71 | 31.45±2.43 | 32.29±1.74 |
Table R2. Return of the continuation agent with no inherited history vs. with history collected by CT.
| K | Continuation agent | No history | CT history | Increase |
|---|---|---|---|---|
| 2 | CT | 5.339±0.012 | 5.728±0.025 | +0.389 |
| ST | 4.259±0.028 | 4.498±0.006 | +0.239 | |
| 4 | CT | 3.863±0.013 | 4.395±0.021 | +0.532 |
| ST | 3.273±0.010 | 3.812±0.020 | +0.539 | |
| 6 | CT | 2.443±0.006 | 2.984±0.009 | +0.541 |
| ST | 2.115±0.008 | 2.662±0.014 | +0.547 | |
| 8 | CT | 1.124±0.005 | 1.504±0.006 | +0.380 |
| ST | 1.074±0.004 | 1.325±0.008 | +0.251 |
Thank you for the response and additional controls. The distinction between held-out latents and unseen environments clarifies the limits of transfer. The fixed-history and history-free comparisons also strengthen the switching analysis, while retaining its interpretation as a relative diagnostic rather than a causal decomposition. Please include these new results and the narrower transfer claims in the revised manuscript. I will maintain my positive score.
Response Part 1
We are grateful for the reviewer's careful reading and positive assessment of our work, and are glad that the reviewer finds explicit shared latents a useful controlled setting and appreciates our extensive empirical validation. We refer the reviewer to the general response for the contributions and broader significance of our work. The reviewer's questions also prompted an in-depth analysis of transfer and two new controls for agent switching, which we believe have made our conclusions better grounded. Below, we address each of the reviewer's points.
We thank the reviewer for this important question. Our held-out evaluations are designed to test whether the adaptation learned through Cross-Task RL transfers beyond its training conditions, which LatentGym makes possible because what should transfer is explicit and controllable. Prompted by the reviewer's question, we used the same property to analyze the trajectories of the trained models in depth, revealing whether and how they act on the shared structure. This analysis gives a clearer understanding of when and why transfer holds. Accordingly, we have revised the introduction and conclusion to distinguish transfer to held-out latents from transfer to held-out environments, rather than stating that gains "mostly extend" to held-out environments. We believe this materially improves both the exposition and the rigor of the paper. The results support the following claims:
Held-out latents (OOD-1). We evaluate on one held-out latent in each of the six training environments. Cross-Task RL (CT) transfers to these latents for both models: it outperforms the base model by 58% (Qwen3-8B) and 24% (Gemma3-12B) on cumulative reward on average, and single-task RL (ST) by 28% and 12% (Tables 4 and 15).
Held-out environments (OOD-2). We use a leave-one-out protocol over four environments, training on three and testing on the fourth. Transfer here is limited and model-dependent: Qwen3-8B improves cumulative reward by 11% on average and final-task reward on all four environments (Table 5), whereas Gemma3-12B improves only on Number Guessing (Table 16). To understand these outcomes, we conducted a new trajectory analysis of the trained models and a control that trains Cross-Task RL on re-dealt sequences with no shared latent.
We interpret the results by distinguishing two capabilities that transfer requires: solving the individual tasks, and identifying and exploiting the shared latent.
On held-out latents, the environment is unchanged, so task-solving carries over directly, and whether adaptation also transfers depends on how close the latent is to those seen in training, that is, on whether the strategy learned on them still fits. In Number Guessing, a single recurring target (set of 1, evaluated on a wider range with 20 attempts) is close to the training latents, and Cross-Task RL outperforms single-task RL for both models. On range 100, where targets fall in a fixed window but rarely repeat, the latent is further from training, and the outcome depends on the strategy each model learned. In Secretary, the held-out latent (best is last) lacks the cue the learned strategy relies on, so Cross-Task RL rarely applies it (see our responses to Reviewers h3Dj, Q2, and wmWR, W1, for details).
On held-out environments, both the task and the latent are new, and transfer depends on how closely the new environment resembles those seen in training. Here, the gains come largely from task-solving: on the word games, Qwen3-8B’s Cross-Task RL already leads the base model on the first task, before any cross-task evidence is available (see our response to Reviewer h3Dj, Q2; the new leave-one-out results are in our response to Reviewer wmWR, W1).
The difference between the two models has three sources. First, they learn different strategies: Qwen3-8B learns to act on remembered targets, which also helps on related latents (it opens 95% of later range 100 tasks inside the hidden window, vs. 19% for Gemma3-12B), whereas Gemma3-12B learns to reuse its own earlier searches, which helps only when the exact target recurs (see our response to Reviewer h3Dj). Second, multi-environment training improves on single-environment Cross-Task RL by 20% on average for Qwen3-8B but not for Gemma3-12B (Section 4), suggesting that Gemma3-12B learns less that is shared across environments. Third, Gemma3-12B's base model is already stronger on most held-out environments (Tables 5 and 16), leaving less room for Cross-Task RL to add.
Our results therefore support transfer to held-out latents within trained environments, to the extent that they are close to the latents seen in training, while transfer to unseen environments is more modest. We expect it to improve as training covers a larger and more diverse set of environments and latents, an important direction that LatentGym is designed to study.