acceptodds
Under review as a conference paper at ICLR 2027

LatentGym: A Testbed for Cross-task Experiential Learning with Contollable Latent Structure

Abstract

We envision continually learning agentic systems that become more useful over time: as they encounter sequences of related tasks, they should infer the hidden structure shared across those tasks and use it to improve future decisions. This cross-task experiential learning capability is pivotal in domains such as personalization and interactive assistance, but existing training/evaluation frameworks do not provide shared, controllable latent structures and cannot measure whether or why agents improve. We introduce LatentGym: a controllable suite in which each environment is organized around a ground-truth latent variable governing the structure across tasks. Our construction yields agent-switching diagnostics that compare experience collection (how useful an agent's collected history is for subsequent performance) and experience utilization (how effectively an agent acts given an inherited history). We demonstrate our suite on empirical studies addressing three questions: how and why frontier models fail to adapt across related tasks; whether post-training on related task sequences improves cross-task adaptation, and where those gains come from; and how design choices such as inter-task feedback shape training dynamics and generalization. Together, these results establish a controlled foundation for studying how LLM agents learn from experience across tasks, and for designing agents that adapt more reliably in personalized, and interactive settings.

Primary area:
datasets and benchmarks
Keywords:
Continual learning, Meta RL, In-context Learning, LLM Agents, LLM post training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9NsyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LatentGym is a controllable testbed for studying how LLM agents learn across related tasks. Its seven text-game environments use explicit latent variables to define structure shared by tasks in a sequence. The paper uses agent switching to diagnose the usefulness of collected experience and an agent’s ability to act on inherited experience. It also compares sequence-level cross-task RL with single-task RL. Across the main training environments, cross-task RL improves cumulative reward, with additional analyses of held-out latents, held-out environments, matched episode budgets, and the removal of shared structure.

Critical strengths
  1. Useful controlled setting for a difficult question. Explicit shared latents make it possible to vary what information can transfer between tasks and to test whether performance depends on that structure. This gives the benchmark more diagnostic value than task sequences whose relationship is only implicit.
  2. Substantial empirical validation within the proposed setting. The gains appear across two model families and are supported by held-out-latent evaluation, a matched task-instance and episode-budget comparison, and a shared-latent ablation (Tables 2, 4, 21, and 24). These controls make the central result—that sequence-level training helps in these environments—reasonably convincing.
Critical weaknesses
  1. The evidence for general transfer remains limited. The environments are synthetic games, and the held-out-environment study covers only four of them. The OOD-2 improvement is modest for Qwen3-8B and largely absent for Gemma3-12B outside Number Guessing (Tables 5 and 16). This matters because transfer beyond the training environments is an important part of the paper’s motivation; the conclusions should be scoped accordingly.
  2. Agent switching is informative but does not cleanly isolate collection from utilization. Post-switch reward also depends on the continuation agent’s task-solving ability, subsequent exploration, and how the inherited history is presented, as the paper acknowledges. The diagnostic supports a relative comparison, but a stronger causal interpretation of the two gains is not established.
Questions for authors
  1. How do the authors interpret the different OOD-2 outcomes for Qwen3-8B and Gemma3-12B? Which claims about adaptation to unseen environments do these results support?
  2. Can the authors provide a control that holds the inherited history fixed while varying the continuation agent, or otherwise quantify the influence of general task-solving ability and history presentation on the switching results? If not, would they narrow the interpretation of the reported collection and utilization gains?
3 comments
AuthorsOpenReview ↗

Response Part 1

We are grateful for the reviewer's careful reading and positive assessment of our work, and are glad that the reviewer finds explicit shared latents a useful controlled setting and appreciates our extensive empirical validation. We refer the reviewer to the general response for the contributions and broader significance of our work. The reviewer's questions also prompted an in-depth analysis of transfer and two new controls for agent switching, which we believe have made our conclusions better grounded. Below, we address each of the reviewer's points.

W1 and Q1: Evidence for general transfer and OOD-2 outcomes

We thank the reviewer for this important question. Our held-out evaluations are designed to test whether the adaptation learned through Cross-Task RL transfers beyond its training conditions, which LatentGym makes possible because what should transfer is explicit and controllable. Prompted by the reviewer's question, we used the same property to analyze the trajectories of the trained models in depth, revealing whether and how they act on the shared structure. This analysis gives a clearer understanding of when and why transfer holds. Accordingly, we have revised the introduction and conclusion to distinguish transfer to held-out latents from transfer to held-out environments, rather than stating that gains "mostly extend" to held-out environments. We believe this materially improves both the exposition and the rigor of the paper. The results support the following claims:

  • Held-out latents (OOD-1). We evaluate on one held-out latent in each of the six training environments. Cross-Task RL (CT) transfers to these latents for both models: it outperforms the base model by 58% (Qwen3-8B) and 24% (Gemma3-12B) on cumulative reward on average, and single-task RL (ST) by 28% and 12% (Tables 4 and 15).

  • Held-out environments (OOD-2). We use a leave-one-out protocol over four environments, training on three and testing on the fourth. Transfer here is limited and model-dependent: Qwen3-8B improves cumulative reward by 11% on average and final-task reward on all four environments (Table 5), whereas Gemma3-12B improves only on Number Guessing (Table 16). To understand these outcomes, we conducted a new trajectory analysis of the trained models and a control that trains Cross-Task RL on re-dealt sequences with no shared latent.

We interpret the results by distinguishing two capabilities that transfer requires: solving the individual tasks, and identifying and exploiting the shared latent.

  • On held-out latents, the environment is unchanged, so task-solving carries over directly, and whether adaptation also transfers depends on how close the latent is to those seen in training, that is, on whether the strategy learned on them still fits. In Number Guessing, a single recurring target (set of 1, evaluated on a wider range with 20 attempts) is close to the training latents, and Cross-Task RL outperforms single-task RL for both models. On range 100, where targets fall in a fixed window but rarely repeat, the latent is further from training, and the outcome depends on the strategy each model learned. In Secretary, the held-out latent (best is last) lacks the cue the learned strategy relies on, so Cross-Task RL rarely applies it (see our responses to Reviewers h3Dj, Q2, and wmWR, W1, for details).

  • On held-out environments, both the task and the latent are new, and transfer depends on how closely the new environment resembles those seen in training. Here, the gains come largely from task-solving: on the word games, Qwen3-8B’s Cross-Task RL already leads the base model on the first task, before any cross-task evidence is available (see our response to Reviewer h3Dj, Q2; the new leave-one-out results are in our response to Reviewer wmWR, W1).

The difference between the two models has three sources. First, they learn different strategies: Qwen3-8B learns to act on remembered targets, which also helps on related latents (it opens 95% of later range 100 tasks inside the hidden window, vs. 19% for Gemma3-12B), whereas Gemma3-12B learns to reuse its own earlier searches, which helps only when the exact target recurs (see our response to Reviewer h3Dj). Second, multi-environment training improves on single-environment Cross-Task RL by 20% on average for Qwen3-8B but not for Gemma3-12B (Section 4), suggesting that Gemma3-12B learns less that is shared across environments. Third, Gemma3-12B's base model is already stronger on most held-out environments (Tables 5 and 16), leaving less room for Cross-Task RL to add.

Our results therefore support transfer to held-out latents within trained environments, to the extent that they are close to the latents seen in training, while transfer to unseen environments is more modest. We expect it to improve as training covers a larger and more diverse set of environments and latents, an important direction that LatentGym is designed to study.

AuthorsOpenReview ↗

Response Part 2

W2 and Q2: Interpretation of agent switching

We are grateful for this close reading of agent switching. As we note in Section 4 and Appendix B.5, it supports a relative comparison rather than a clean causal decomposition. Below, we clarify our interpretation and add two controls. Agent switching shows where one agent falls short of another, in the experience it collects or in how it uses inherited experience, and how this changes with the switch point K; on Number Guessing, for example, the collection advantage of Cross-Task RL (CT) over single-task RL (ST) grows with K (Table 5). As the reviewer notes, both gains partly reflect task-solving ability: collection gains through the first agent, since a target is revealed only when its task is solved, and utilization gains through the continuation agent. Part of this ability is itself a product of adaptation, since an agent that identifies the latent also solves later tasks more often.

  • Fixed inherited history. Replaying the same recorded histories to every continuation agent changes both gains by at most about 2 percentage points, within standard errors (first table), so the gains are robust to the choice of inherited history.

  • History-free baseline. We evaluate each continuation agent on tasks K+1,…,NK+1, \dots, N with and without the history collected by Cross-Task RL (second table). This history raises post-switch reward for both continuation agents at every switch point, and its relative benefit is larger at later switch points (from 7% at K=2 to 34% at K=8 for Cross-Task RL, and from 6% to 23–26% for single-task RL). Cross-Task RL therefore collects important information that continuation agents can use. Note that the increases in the second table, both in percentage and in absolute value, are relative to each agent's own reward without inherited history and do not by themselves isolate each agent's ability to use the history, since the two agents start from different baselines.

Table R1. ExploreGain and ExploitGain for reference C, with the original vs. fixed history.

KCExplore (orig.)Explore (fixed)Exploit (orig.)Exploit (fixed)
2CT−0.61±0.31−0.49±0.5327.96±0.4527.35±0.50
ST6.74±1.416.67±1.1037.42±1.5336.52±1.44
4CT0.15±0.410.29±0.8614.49±0.4815.30±0.80
ST22.35±1.0521.68±1.3039.87±1.0639.89±1.43
6CT5.04±0.545.33±1.0712.50±0.4912.11±0.43
ST26.08±1.6026.73±1.5435.02±1.4634.88±0.96
8CT9.79±0.868.80±0.3512.29±1.0313.58±0.81
ST28.58±2.9426.80±2.7131.45±2.4332.29±1.74

Table R2. Return of the continuation agent with no inherited history vs. with history collected by CT.

KContinuation agentNo historyCT historyIncrease
2CT5.339±0.0125.728±0.025+0.389
ST4.259±0.0284.498±0.006+0.239
4CT3.863±0.0134.395±0.021+0.532
ST3.273±0.0103.812±0.020+0.539
6CT2.443±0.0062.984±0.009+0.541
ST2.115±0.0082.662±0.014+0.547
8CT1.124±0.0051.504±0.006+0.380
ST1.074±0.0041.325±0.008+0.251

Thank you for the response and additional controls. The distinction between held-out latents and unseen environments clarifies the limits of transfer. The fixed-history and history-free comparisons also strengthen the switching analysis, while retaining its interpretation as a relative diagnostic rather than a causal decomposition. Please include these new results and the narrower transfer claims in the revised manuscript. I will maintain my positive score.

Reviewer_wmWROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • summary: The paper introduces LatentGym, a set of seven text-game environments (built on TextArena) where a sequence of tasks shares a hidden latent that the experimenter controls.
  • contribution: Each environment can vary the latent, the prompt (no / some / full information about the latent), the feedback between tasks and the horizon. It plugs into SkyRL for RL training.
  • contribution: An agent-switching diagnostic separates how useful an agent's collected history is from how well an agent uses an inherited history.
  • validation: Experiments show that frontier models often fail to adapt across tasks, and that GRPO over full task sequences (Cross-Task RL) on Qwen3-8B and Gemma3-12B beats single-task RL, with partial OOD transfer.
Critical strengths
  1. The controllable latent makes clean controls possible, and the paper uses them.
  • In number guessing, the trained models beat the history-blind bound (0.87 vs 0.82, Appendix A.2).
  • Re-dealing tasks so that no sequence shares a latent drops final reward from 0.64 to 0.34 (Table 24).
  • This is the main reason for my positive score: the suite works as a diagnostic tool.
  1. The framework is modular and released, and the reporting is careful (equal-budget baseline, extra training seeds, per-latent breakdowns in Tables 17 to 23).
Critical weaknesses
  1. The cross-task vs single-task RL comparison mixes two things: using the shared latent, and coping with a long multi-task context.
  • Single-task RL is trained with N = 1 but evaluated with 10 tasks in context, and its gain is negative almost everywhere (e.g. −57% on Wordle).
  • Table 24 shows that the shared latent explains the gain in Number Guessing, Mastermind and Secretary, but not in Hangman (6.4 vs 6.2), Word Ladder (9.4 vs 9.3) or Wordle (6.1 vs 6.1).
  • The missing control is Cross-Task RL trained on N = 10 re-dealt sequences without a shared latent.
  1. There is limited evidence that the models learn a general adaptation skill.
  • There are only three training latents per environment. Held-out-environment gains are small for Qwen3-8B (about 11%) and mostly absent for Gemma3-12B (Table 16).
  • The frontier-model study is thin: 5 trajectories per cell, failure rates quantified for one setting only (Table 13), and Gemini 2.5 Flash run on only four of seven environments (Table 12).
  • There are no oracle or history-blind reference points per environment, so the difference to the maximum is unclear, and several environments seem to be near the top.
Questions for authors

Clarifications:

  1. How does Cross-Task RL trained on re-dealt N = 10 sequences (no shared latent) compare, in distribution and on held-out environments?
  2. How do the trained models do with the no-info prompt? All RL runs seem to use the "some info" hint.
  3. Is the base Qwen3-8B run in thinking mode, and does the 256-token response cap (Table 8) cut off its reasoning?
  4. In agent switching, does the inherited history include the first agent's reasoning text?

Suggestions (not affecting my score):

  • Elaborate on Tajwar et al. (2025, "Training a Generally Curious Agent") and Krishnamurthy et al. (2024), and make the difference to the cross-episode meta-RL of Jiang et al. (2026) and Lin et al. (2026) explicit.
3 comments
AuthorsOpenReview ↗

Response Part 1

We thank the reviewer for the thorough and constructive review, and are glad that the reviewer finds the controllable latent useful for clean controls and the suite effective as a diagnostic tool, and appreciates our modular framework and careful reporting. We are particularly grateful for the reviewer's precise and actionable suggestions, which inspired a new control (Cross-Task RL trained on re-dealt sequences without a shared latent), an evaluation of the trained models with the no-info prompt, and a more explicit comparison with prior work on exploration and cross-episode meta-RL; we believe these have made the paper considerably stronger. Below, we address each of the reviewer's points, and we have incorporated the relevant parts of this discussion into the revised manuscript.

W1 and Q1: Separating use of the shared latent from coping with a long context

We thank the reviewer for this excellent suggestion. Cross-Task RL trained on re-dealt sequences is exactly the right control to separate use of the shared latent from coping with a long multi-task context, and its results clearly show that Cross-Task RL learns to exploit the shared latent. We have run this control, comparing Cross-Task RL trained on re-dealt N=10 sequences with no shared latent ("re-dealt") with Cross-Task RL (CT) and single-task RL (ST); full results are in the tables below.

ModelLatentAttemptsCT CumCT Gain %CT FinalRe-dealt CumRe-dealt Gain %Re-dealt FinalST CumST Gain %ST Final
Qwen3-8BIn-dist. (3-latent avg.)105.73 ± 0.04+82.6 ± 5.70.64 ± 0.015.15 ± 0.05+0.6 ± 1.70.51 ± 0.013.06 ± 0.03−19.7 ± 1.20.27 ± 0.00
range_100106.16 ± 0.08+47.3 ± 14.90.63 ± 0.036.98 ± 0.03−2.2 ± 1.50.73 ± 0.013.65 ± 0.04−6.7 ± 3.30.35 ± 0.01
range_1000207.16 ± 0.03+0.6 ± 1.00.70 ± 0.016.82 ± 0.03−2.6 ± 1.70.67 ± 0.015.29 ± 0.07−3.7 ± 1.30.54 ± 0.01
set_of_1209.47 ± 0.02+41.5 ± 1.50.98 ± 0.006.86 ± 0.07+0.7 ± 0.10.69 ± 0.015.63 ± 0.14+2.7 ± 0.60.57 ± 0.01
Gemma3-12BIn-dist. (3-latent avg.)105.98 ± 0.04+43.2 ± 1.50.64 ± 0.014.92 ± 0.05+2.4 ± 1.60.47 ± 0.013.75 ± 0.03−48.0 ± 2.60.26 ± 0.01
range_100105.95 ± 0.05−11.9 ± 1.70.54 ± 0.016.42 ± 0.04−8.8 ± 3.70.60 ± 0.025.28 ± 0.06−47.3 ± 4.50.35 ± 0.03
range_1000205.99 ± 0.03−25.6 ± 3.30.54 ± 0.026.44 ± 0.01−18.3 ± 1.40.60 ± 0.015.49 ± 0.05−43.5 ± 1.50.40 ± 0.01
set_of_1208.49 ± 0.02+20.3 ± 0.80.87 ± 0.007.42 ± 0.04+2.9 ± 0.70.75 ± 0.007.80 ± 0.09+30.1 ± 1.40.87 ± 0.01

All models are trained with 10 attempts. The range_1000 and set_of_1 latents use a 1–10000 range and are evaluated with 20 attempts; set_of_1 has a single recurring target, and range_1000 has targets in a fixed 1000-number window.

  • In distribution, training on long multi-task contexts without a shared latent recovers part of the gap to single-task RL (cumulative reward of 5.15 vs. 3.06 for Qwen3-8B and 4.92 vs. 3.75 for Gemma3-12B), consistent with the reviewer's point. However, the re-dealt model does not improve across tasks (Gain of +0.6% and +2.4%, vs. +82.6% and +43.2% for Cross-Task RL) and reaches a lower final-task reward (0.51 vs. 0.64 and 0.47 vs. 0.64), so the improvement within a sequence comes from the shared latent. The trajectories confirm this: the re-dealt model restarts its search on every task and is no faster when a target recurs, whereas Cross-Task RL needs 3.4–4.9 guesses on repeated targets instead of about 9.3 for Qwen3-8B (see our trajectory analysis in the response to Reviewer h3Dj).

  • On held-out latents, the outcome depends on how close the latent is to those seen in training. A single recurring target (set of 1) resembles the training latents, and Cross-Task RL clearly outperforms the re-dealt model (9.47 vs. 6.86 and 8.49 vs. 7.42). The range latents are further from training: their targets rarely repeat, and knowing them only narrows the search. Cross-Task RL nonetheless carries over its strategy of recalling earlier targets, which helps relative to single-task RL but is not enough to outperform the re-dealt model, which is ahead on range 100 for both models and on range 1000 for Gemma3-12B. We expect training on a more diverse set of latents to teach Cross-Task RL to adapt its strategy to the latent at hand.

AuthorsOpenReview ↗

Response part 2

  • On held-out environments, Cross-Task RL is ahead of the re-dealt model on Word Ladder and Hangman and comparable on Number Guessing. A single policy can learn adaptation strategies for several environments at once: the four-environment model opens 77% of later Number Guessing tasks with a recurring target. But this strategy does not yet transfer: when Number Guessing is held out, the model opens every task with 500 and is no faster on repeated targets. Its advantage on held-out environments therefore comes from task-solving rather than adaptation, consistent with our response to W2. Learning strategies that transfer to entirely new environments likely requires training on many more environments, and LatentGym makes this question measurable.

Qwen3-8B - Held-out Environment (OOD-2), leave-one-out

Held-out environmentCT CumCT Gain %CT FinalRe-dealt CumRe-dealt Gain %Re-dealt Final
Number Guessing3.23 ± 0.04−29.0 ± 1.40.25 ± 0.013.18 ± 0.06−24.1 ± 3.60.25 ± 0.01
Hangman5.61 ± 0.06+2.4 ± 0.40.57 ± 0.015.45 ± 0.03+4.2 ± 0.60.56 ± 0.00
Word Ladder4.89 ± 0.06−1.5 ± 2.00.50 ± 0.014.65 ± 0.05−1.7 ± 3.80.48 ± 0.02

W2: Evidence for a general adaptation skill

We are grateful for these points, as they helped us state the scope of our claims more precisely, and we address each in turn below.

Training latents and held-out environments. Our results distinguish two levels of transfer. Cross-Task RL transfers to held-out latents for both models (+58% and +24% over the base model). Transfer to held-out environments is more limited (+11% on Qwen3-8B, and largely absent for Gemma3-12B), as we would naturally expect: the test environment is never seen during training, in any latent, so both the task and its latent structure are new. As discussed in W1, these gains come largely from task-solving; we discuss the differences between the models in our response to Reviewer 9Nsy. We agree that these results do not yet establish a general adaptation skill across environments, and expect transfer to improve with more diverse training latents and environments.

Frontier-model study. As noted in Section 3, the frontier-model study is intended to illustrate the failures, rather than benchmarking frontier models. We limited its scale because querying these models is costly. We evaluated Gemini 2.5 Flash on four environments because these already exhibit all three failure modes for this model: neglect and breakdown in Number Guessing (Table 13), and miscalibration in Word Ladder (Figure 6). Table 13’s latent-use statistics are specific to Number Guessing, where the latent can be read off a single opening guess; elsewhere, Table 12 reports the same failure modes through Gain and final-task reward.

Reference points. Exact upper bounds are generally difficult to obtain for these environments, since they require optimal play given the latent; we provide them where they can be derived. For Number Guessing, Appendix A.2 gives a history-blind bound of 0.82 and a latent-aware expected reward of 0.97. For the other environments, we provide empirical lower references without the shared latent: the re-dealt evaluation (Table 24), which reports each trained model on the same tasks re-dealt across sequences, and Cross-Task RL trained on re-dealt sequences (see our response to W1). We also flag environments close to the maximum reward, such as Word Ladder (final-task reward of 0.94–0.96), where little room remains for the shared latent to help.

AuthorsOpenReview ↗

Response Part 3

Q2: Trained models under the no-info prompt

We are grateful for this question, as it tests whether Cross-Task RL's adaptation relies on the hint in the some-info prompt. We evaluated the trained Qwen3-8B models with the no-info prompt, itself a shift from the some-info training prompt. Cross-Task RL (CT) still improves across tasks without the hint in Number Guessing, retains its performance in Wordle, and remains well ahead of single-task RL (ST) in every environment shown below, so its adaptation does not depend on being told that a shared structure exists, although its gain in Number Guessing is much smaller, so the hint contributes there.

Env.PromptCT Cum.CT Gain (%)CT FinalST Cum.ST Gain (%)ST Final
Num. guesssome-info5.73 ± 0.04+82.6 ± 5.70.64 ± 0.013.06 ± 0.03−19.7 ± 1.20.27 ± 0.00
Num. guessno-info5.01 ± 0.05+20.7 ± 3.70.52 ± 0.012.85 ± 0.05−9.0 ± 2.10.24 ± 0.01
Mastermindsome-info7.21 ± 0.02+6.6 ± 0.90.73 ± 0.015.17 ± 0.04−19.7 ± 0.90.50 ± 0.01
Mastermindno-info7.14 ± 0.03+0.8 ± 1.10.70 ± 0.015.27 ± 0.02−12.7 ± 1.30.51 ± 0.01
Wordlesome-info6.15 ± 0.01+13.1 ± 1.50.62 ± 0.013.12 ± 0.01−56.9 ± 0.90.26 ± 0.01
Wordleno-info6.15 ± 0.02+17.3 ± 1.30.62 ± 0.003.40 ± 0.02−55.6 ± 0.30.27 ± 0.00

Q3 and Q4: Thinking mode, the 256-token cap, and reasoning in inherited histories

We appreciate these questions, which concern important details of our setup. Qwen3-8B, including the base model, was run with thinking mode disabled for both training and evaluation; we have added this to Appendix B. We disabled thinking because a cross-task rollout keeps all ten multi-turn tasks in context, and the accumulated reasoning could exceed Qwen3-8B’s native 32,768-token context and substantially slow RL training. The 256-token cap is rarely reached (0%, 0.61%, and 0.08% of responses for the base, single-task RL, and Cross-Task RL models). Since neither model produces separate reasoning traces (Gemma3-12B has no thinking mode), the inherited history in agent switching consists of the first agent’s full responses, including any brief reasoning in them, and the environment’s feedback.

Related work

We are thankful for these pointers and have incorporated the discussion in the revised version. Krishnamurthy et al. (2024) and Tajwar et al. (2025) study exploration and information gathering within a single task, evaluating and training LLMs for it respectively, whereas LatentGym studies adaptation across distinct tasks that share a latent. Jiang et al. (2026) and Lin et al. (2026) use cross-episode meta-RL but hold the task fixed across episodes (Lin et al. resample only the initial state), so the agent learns from repeated attempts at the same task; in LatentGym, each task is new and only an explicitly specified latent is shared, which lets the experimenter control what should transfer and remove the shared latent as a control (Appendix C.6).

Reviewer_h3DjOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces LatentGym, a testbed of seven text games for studying whether LLM agents can discover shared patterns across tasks and use past experience to improve later decisions. Tasks within each sequence share a hidden rule, and agents adapt through accumulated interaction history without weight updates. The framework supports configurable prompts and feedback, sequence-level GRPO training with cumulative rewards, and agent-switching diagnostics for comparing experience collection and utilization. Experiments identify adaptation failures in commercial models and show that cross-task training improves cumulative performance in two open-source models, with transfer varying across held-out patterns and environments.

Critical strengths

S1. Clear motivation. The paper addresses whether LLM agents can learn from related tasks and improve their decisions through experience.

S2. Controlled and reusable framework. The testbed combines configurable environments, evaluation, and training, making it useful for studying and developing adaptation methods.

S3. Broad experimental coverage. The study evaluates multiple model families, task types, and generalization settings, supported by ablations and training controls.

S4. Informative empirical findings. The results characterize both adaptation failures in existing models and improvements from cross-task training.

Critical weaknesses

W1. Limited methodological novelty. Shared latent structure, in-context adaptation, and cumulative-reward optimization are established components of meta-RL. The main contribution is the integrated testbed and diagnostics. The paper should clarify which controlled comparisons or insights these enable beyond existing benchmarks and recent language-agent frameworks.

W2. Limited coverage of long-horizon adaptation. The main experiments use ten-task sequences of simple text games with a fixed shared regularity. These settings do not establish whether agents can discover patterns over longer, more complex interactions, where evidence may be sparse, delayed, or mixed with irrelevant information.

W3. Adaptation is not consistently isolated from task-solving ability. In several settings, removing the shared latent barely changes performance (for example, Table 24). Although some environments provide stronger evidence of history use, the results leave unclear how consistently benchmark scores reflect cross-task adaptation. Each configuration would benefit from validation that its performance depends on the shared structure.

Questions for authors
  1. What behavioral changes does Cross-Task RL induce? Does it help models identify shared regularities, act on them more reliably, or both? Qualitative analysis of trained models would help clarify this.

  2. What explains the differences in adaptation and transfer across models and environments? Are there common features of the settings where cross-task training succeeds or fails?

  3. How can the agent-switching diagnostics guide improvements to an agent? An example connecting a diagnosed bottleneck to a suitable intervention would make their practical value clearer.

  4. What are the main open research questions that LatentGym makes possible to study? Beyond adding more latent variants, it would be helpful to outline how the current findings motivate concrete hypotheses and follow-up experiments. This would clarify the testbed’s value for future research.

5 comments
AuthorsOpenReview ↗

Response Part 1

We thank the reviewer for the careful and detailed review, and are glad that the reviewer finds LatentGym a controlled and reusable framework with broad experimental coverage and informative findings. We are particularly grateful for the reviewer's thoughtful questions, which inspired a trajectory analysis of how Cross-Task RL changes agent behavior and a more thorough discussion of the limitations of our work and the research directions it opens up; we believe these have substantially strengthened the paper. Below, we address each of the reviewer's points, and we have incorporated the relevant parts of this discussion into the revised manuscript.

W1: Methodological novelty and what LatentGym enables

We appreciate the reviewer raising this point. We agree that shared latent structure, in-context adaptation, and cumulative-reward optimization are established components of meta-RL. However, classical meta-RL benchmarks with shared latent structure, such as XLand (Open-Ended Learning Team et al., 2021), Alchemy (Wang et al., 2021), and XLand-MiniGrid (Nikulin et al., 2023), are non-linguistic and train small policies from scratch. Conversely, in existing LLM-agent benchmarks and recent cross-episode meta-RL frameworks (Jiang et al., 2026; Lin et al., 2026), what should be learned across a sequence remains implicit. For example, in SWE-Bench-CL, linked tasks indicate a possible opportunity for reuse without specifying what should transfer (Appendix A.11). LatentGym makes this structure explicit and controllable for pretrained LLM agents operating in language, with additional axes of difficulty that can be varied independently (Table 1).

This allows us to investigate not only whether LLM agents improve across tasks, but why. We can test whether agents notice the shared structure but fail to act on it, and how revealing the latent in the prompt changes their behavior (Section 3). We can also construct controls that remove the shared latent while keeping the tasks fixed to test whether gains actually rely on the latent (Appendix C.6), and run controlled RL studies that vary the training latents, environments, and feedback independently (Section 4). Following the reviewer's suggestion, we have added this comparison to the related work section and clarified in the introduction the controlled comparisons and insights that LatentGym enables. We thank the reviewer for inspiring this discussion, which has helped us position our contributions more clearly.

W2: Long-horizon adaptation

We agree that this is an important limitation of our current evaluation: our experiments show that agents can learn to adapt across short sequences of simple games, but not whether they can discover regularities over longer, more complex interactions. We have included a careful discussion of this limitation.

However, the choice to keep the environments and latents simple is deliberate. The simple settings highlight failures in cross-task adaptation more than within-task difficulty. Surprisingly, even in these minimal settings, frontier models can fail to adapt: without an explicit cue, no evaluated model opens a later Number Guessing task with one of the recurring targets (Table 13). Our choice of a short horizon is also driven by compute: context length grows with the number of tasks, and our runs already required approximately 11,000 GPU-hours (Appendix B.6). That said, the testbed itself is not limited to this setting: its horizon, latent, and feedback components can be configured to create longer sequences and sparser, noisier, or nonstationary evidence. Studying adaptation in these settings is an important direction for future research that LatentGym directly enables.

As a step to study longer-horizon adaptation, we evaluated the Number Guessing models trained on N=10 sequences on N=20 sequences. Adaptation holds at twice the training horizon: Cross-Task RL (CT) retains a positive Gain for both models, and its lead over single-task RL (ST) in cumulative reward grows (from +87% to +90% for Qwen3-8B, and from +60% to +79% for Gemma3-12B).

ModelMethodN=10 CumN=10 Gain %N=10 FinalN=20 CumN=20 Gain %N=20 Final
Qwen3-8BCT5.73±0.04+82.6±5.70.64±0.0111.91±0.05+65.8±2.60.61±0.01
ST3.06±0.03−19.7±1.20.27±0.006.27±0.08−7.4±2.80.30±0.01
Gemma3-12BCT5.98±0.04+43.2±1.50.64±0.0112.18±0.09+46.9±1.90.63±0.00
ST3.75±0.03−48.0±2.60.26±0.016.80±0.12−43.2±2.70.29±0.01
AuthorsOpenReview ↗

Response Part 2

W3 and Q1: Whether and how trained models use the shared latent.

We appreciate the thoughtful suggestion, which motivated the two new analyses below. The shared-latent ablation (Table 24) shows that the extent to which performance depends on the shared structure varies across environments: removing the latent clearly lowers cumulative and final-task reward in Number Guessing, Mastermind, and Secretary for both models, and in Wordle for Gemma3-12B, whereas Word Ladder is close to saturation (final-task reward of 0.94–0.96) and the effect in Hangman is small. We have revised the text to reflect this environment-dependent picture.

Following the reviewer's suggestion, we validate the use of shared structure more directly in two ways:

1. Trajectory analysis. We examine Number Guessing trajectories (set of 2 and set of 3). For Qwen3-8B, the base and single-task RL (ST) models restart binary search from 500 on every task, whereas Cross-Task RL (CT) opens later tasks with a recurring target, often solving them in a single guess. For Gemma3-12B, the base and ST models often identify the recurring targets, naming at least two of them in about 40% of sequences, but act on them unreliably, failing 76–87% of the time when they open with a previous target that has changed. CT instead reuses its own earlier searches, stopping early on 46–62% of repeated targets, compared with at most 6% for the other models. To quantify this, we compare the number of guesses on a target's first appearance in a sequence with that on its repeats; a model that ignores the history needs as many guesses on both.

ModelSet of 3: Turns₁→Turns₁₀First→repeatRepeats solvedSet of 2: Turns₁→Turns₁₀First→repeatRepeats solved
Qwen3-8B base9.5→9.69.5→9.539%9.2→9.49.4→9.345%
Qwen3-8B ST9.2→9.39.3→9.155%9.0→9.29.3→9.151%
Qwen3-8B CT9.5→5.99.4→4.978%9.2→3.39.3→3.490%
Gemma3-12B base9.5→8.49.5→8.164%9.2→6.99.4→7.370%
Gemma3-12B ST9.1→8.79.2→8.353%8.9→7.49.1→7.662%
Gemma3-12B CT9.2→6.39.0→6.792%8.9→5.89.0→6.195%

CT therefore helps models both identify and act on the shared structure, with the balance depending on the base model.

2. Re-dealt control. We have also trained CT on re-dealt sequences with no shared latent (see our response to Reviewer wmWR). This model never uses the shared targets: it opens every later task with 500 and needs as many guesses on the last task as on the first (Turns₁₀ of 8.5–8.8, vs. 3.3–5.9 for CT), indicating that CT's improvement across tasks comes from the shared structure rather than from long-context training. We have added both analyses, together with this environment-dependent picture of the shared-latent ablation, to the revised manuscript.

Q2: What explains differences across models and environments

We thank the reviewer for prompting us to discuss the results in more depth and illustrate the deeper diagnostics that LatentGym supports. Three factors explain the differences.

Gains within the training distribution. Gains of Cross-Task RL (CT) over single-task RL (ST) are largest where knowing the latent sharply reduces within-task search, such as Number Guessing (+87% and +60%), and Secretary for Gemma3-12B (+130%) (Table 2). Gains are smaller where the main difficulty lies within each task, or where performance is already near the maximum, as in Word Ladder (final-task reward of 0.94–0.96; Table 24). Adaptation also reaches lower performance when the latent is harder to identify: in Number Guessing, CT’s final-task reward for Qwen3-8B is 0.39 when targets fall in a shifting range (dynamic range), compared with 0.87 when two targets recur (set of 2).

Transfer to held-out latents. The within-task problem is unchanged, so improvements in task-solving carry over, but whether the model can also exploit the new latent depends on whether the strategy it learned in training still fits. In Number Guessing, a single recurring target (set of 1, evaluated on a wider range with 20 attempts) resembles the training latents and transfers for both models. On range 100, where targets fall in a fixed 100-number window, the outcome depends on the learned strategy: Qwen3-8B's CT opens later tasks with a remembered target, which places 95% of its later opening guesses inside the window (vs. 12% for the base and single-task models) and raises its success rate from 52% to 75% across the sequence, whereas Gemma3-12B's CT reuses its own earlier searches, which does not help when targets do not repeat. Similarly, in Secretary, the learned strategy relies on a within-task cue that the held-out latent (best is last) lacks, so CT rarely applies it there (1–7% of tasks, vs. 83–100% under its closest training latent).

AuthorsOpenReview ↗

Response Part 3

Transfer to held-out environments. Both the task and the latent are new, and the more modest gains here come largely from task-solving (Tables 5 and 16). For Qwen3-8B, the largest gains are on Word Ladder (+18%) and Hangman (+17%), each of which has the other in its training mix, whereas Secretary (an optimal stopping problem), unlike any training environment, shows the smallest. On the word games, CT already leads the base model on the first task, before any cross-task evidence is available, indicating improved task-solving rather than latent use. These gains depend on how much room the base model leaves: Gemma3-12B’s base model already solves the word games better than Qwen3-8B’s (7.1 vs. 4.8 on Hangman, 6.2 vs. 4.2 on Word Ladder), consistent with its small gains on held-out environments.

Q3 and Q4: Using the diagnostics to guide improvements, and open research questions

We are grateful for these two helpful suggestions. Agent switching indicates which intervention to pursue by showing whether one agent falls short of another in collecting experience or in using it. For example, single-task RL (ST) falls short of Cross-Task RL (CT) mainly in utilization on Number Guessing (Table 5), suggesting interventions such as summarizing evidence across tasks, but mainly in collection on Hangman (Appendix C.7), suggesting incentives for exploration; the diagnostics can then measure whether an intervention closes the gap. Since agent switching is comparative, it identifies relative rather than absolute bottlenecks, and we have not yet tested these interventions; doing so is a natural next step that LatentGym directly supports. More broadly, we have added a discussion of open research questions to the revised manuscript, which we excerpt here:

Open questions. Our findings motivate several testable hypotheses.

(i) Training objectives and feedback. Under the cumulative-reward objective of Cross-Task RL, exploration that lowers reward on early tasks must pay off later in the sequence. Objectives that weight later tasks more heavily, such as final-task reward, may therefore induce more early exploration, which agent switching can test by measuring collection and utilization separately. Similarly, since training with standard feedback transfers more robustly than training with full feedback (Appendix C.8), curricula that gradually reduce feedback may improve transfer further.

(ii) Learned strategies and transfer. Cross-Task RL induces different strategies in the two base models: Qwen3-8B learns to open with a remembered target, whereas Gemma3-12B learns to reuse its earlier searches, and this difference determines transfer to held-out latents such as range 100. On held-out environments, transfer comes largely from improved task-solving rather than latent use. What determines which strategy RL discovers remains open; we hypothesize that training on more diverse latents and environments will yield strategies that transfer more broadly, which our leave-one-out protocol can test directly.

(iii) Harder evidence. By extending the horizon and making evidence sparser, noisier, or nonstationary, the testbed can identify where adaptation breaks down, and whether failures arise from collecting evidence or from using it.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their positive and constructive feedback on our paper. We are glad to see that the reviewers appreciate the diagnostic value of explicit, controllable latents and the clean controls they enable (h3Dj, wmWR, 9Nsy), and recognize the value of our modular, reusable framework (h3Dj, wmWR) and the breadth of our empirical validation (h3Dj, wmWR, 9Nsy). We summarize our key contributions below.

We study whether, and why, LLM agents improve across sequences of related tasks. Our main contribution is a testbed in which the structure shared across tasks is explicit and controlled by the experimenter. Shared latent structure and in-context adaptation are established ideas in meta-RL; what has been missing for LLM agents is a setting in which this structure is known, so that failures to adapt can be interpreted and competing methods can be compared.

  1. Controllable latent structure. Each environment is organized around a ground-truth latent, with within-task dynamics, prompt information, inter-task feedback, and horizon as independently composable components. This allows within-task, latent-identification, and cross-task difficulty to be varied separately (Table 1). This enables interpretable diagnostics, principled controls, and controlled RL studies that are unavailable when shared structure is implicit.

  2. Diagnostics for how agents adapt. The agent-switching diagnostic compares experience collection (how useful an agent's collected history is) with experience utilization (how effectively an agent acts on an inherited history), revealing environment-dependent differences that cumulative reward alone does not show (Section 4; Appendix C.7).

  3. Empirical findings. Frontier models exhibit three interpretable failure modes: adaptation neglect, breakdown, and miscalibration (Section 3). Cross-Task RL improves cumulative reward over single-task RL by 43% (Qwen3-8B) and 55% (Gemma3-12B) on average in distribution (Table 2). These gains persist under multi-environment training and on held-out latents, while gains on held-out environments are more modest (Tables 3–5 and 14–16). Training with standard rather than full feedback yields more robust transfer (Section 4).

Summary of revisions. In response to the reviewers, we have added:

  1. A control that trains Cross-Task RL on re-dealt N=10 sequences with no shared latent, separating gains from using the shared latent from gains from training on long multi-task contexts (wmWR, h3Dj).

  2. A trajectory analysis of the base, single-task, and Cross-Task RL models, showing whether and how each acts on information revealed in earlier tasks (h3Dj, wmWR).

  3. An analysis of when adaptation transfers beyond the training distribution that separates improved task-solving from identifying and exploiting a new latent (h3Dj, wmWR, 9Nsy).

  4. Additional revisions: We have added two additional controls for the agent-switching diagnostic, whose gains we now present as relative comparisons (9Nsy); a discussion of open research questions enabled by LatentGym (h3Dj); and expanded related work (wmWR).

Dear reviewers,

The author-reviewer discussion period closes on November 18 (AoE). Please engage in response periods so that we have our issues discussed thoroughly.

AC

AuthorsOpenReview ↗

Final Response from the Authors

We sincerely thank the reviewers for their thoughtful and constructive feedback, and the reviewers and area chair for their engagement throughout the discussion.

We have uploaded a revised manuscript, with all changes marked in blue, which incorporates the additions summarized in our general response. The reviewers’ suggestions have made the paper considerably stronger: they inspired new controls, analyses, and discussions that make our conclusions better grounded, and helped us state the scope of our contributions more precisely.

We are grateful for the time and care the reviewers and area chair have devoted to the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.