acceptodds
Under review as a conference paper at ICLR 2027

Selecting What to Transmit: Cross-Generational In-Context Reinforcement Learning

Abstract

In-context reinforcement learning promises that a trained network can continue to improve on a new instance by reading its own attempts, without updating its parameters. Prior work seeks better inference performance while maintaining efficiency, typically by accumulating more information in the context window. This is a trade-off: more information in the context window raises the computational cost of inference. Inspired by the intergenerational accumulation of knowledge in human societies, we move from within-generation to cross-generational inference. To this end, we propose CGIRL, which partitions inference into successive generations. Within each generation, the policy reads a fixed short window. Across generations, MAP-Elites, an evolutionary algorithm, selects the transmitted experience from candidates archived by their return dynamics, rather than transmitting the full history or a single incumbent. On the travelling salesperson problem and a sequence-recall task, experiments show that CGIRL achieves state-of-the-art performance and efficiency.

Primary area:
reinforcement learning
Keywords:
in-context reinforcement learning, cross-generational inference, evolutionary algorithm, combinatorial optimization

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_AUvsOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose CGIRL, a RL training setup that combines MAP-Elites with cross-generation inference. Specifically, they consider cases where a system is able to keep some limited amount of past context over the course of inference, and train a model to effectively use this at inference time. At inference they run multiple samples in parallel and then use MAP-Elites to store and mutate promising context candidates. Experiments show that using MAP-Elites in this way for transfer works best relative to simpler context transfer methods, and other key aspects of the method (what to transmit, how many parallel agents to run, etc.) are also ablated.

Critical strengths
  1. The core idea as proposed is fairly reasonable and makes sense, being to use MAP-Elites in the cross-generational inference setting to ensure diversity, and providing a method for effectively training a model to be able to be used in this way. The results, while only on two fairly toy tasks, do seem to indicate that CGIRL improves over other methods.
  2. Many core aspects of CGIRL are ablated, such as transfer operator and context size.
Critical weaknesses
  1. There is not much analysis in this work as to why CGIRL works better than other methods, beyond that MAP-Elites 'better transmit(s) experience that improved within a generation' (lines 377-417). It would really benefit the paper to back up these claims with some analysis; for instance, some numbers or visualisation of what the diverse context is actually capturing, and how this has led to better performance. As it is, the paper makes a reasonable case that the algorithm works, but provides little understanding into why.
  2. I think the work also misses many core details: it would be useful to know golden/optimal TSP scores (or results in the fully-observed setting) to know how big of a gap remains; it would be useful to ablate the fitness functions (or at least the chosen weighting and the descriptor setup; and hyperparameter details are missing, which would be especially useful given reruns of the same setting seem to have non-trivial noise differences (For TSP-80, the default CGIRL setup seems to get 6.82 in Table 1, 7.86 in Table 3, and 6.76 in Table 4). A gap of 1 point is larger than some of the gains over baselines, so it would be useful to clarify this, and provide stderr numbers / confidence intervals for the evaluations if possible.

nit: I found the writing of this work a bit dense and terminology-laden in ways that reduced understanding. For example, in lines 247-256, the phrase "quantises the two halves onto a B × B grid using the task-inherent score upper bound s_max" does not explain the actual method well - but the following sentences do! B is introduced early but not defined until the final sentence of the paragraph, and the grid notation is similarly explained at the end, not the start of the paragraph. Another example is how samples are mutated, which is 'hidden' in single sentences inside Appendix B and difficult to find at a glance (the sentences saying "Search applies..."). These descriptions themselves are also somewhat short. In general, there is a pattern of dense language combined with undefined (or underdefined) terms that makes this work hard to read.

nit: typo: line 237, "equation equation 1"

Questions for authors
  1. How were the values in equation 8 (weights of the fitness values) decided?
  2. What is the variation between evaluation reruns for TSP and Memory? Why does CGIRL Table 3 TSP 80's score differ so much to tables 1 and 4?
  3. What sort of information ends up being captured in the memory? What sort of diversity does the method capture?
  4. What is the motivation for the descriptor function? Could you explain it further?
Reviewer_P3FXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper splits test-time inference into generations. Each generation reads a fixed window, and only an "experience" c is passed on. c is chosen by an evolutionary algorithm keyed by early/late score descriptors. Elites come from a "weak-early, strong-late" region R. The policy is trained with decaying demonstration visibility. Evaluations are on TSP and Memory, which are engineered sequence/search tasks.

Critical strengths

None

Critical weaknesses
  1. The TSP numbers look completely wrong. The mean optimal tour for random uniform 50-city instances in the unit square is about 5.69 (or 7.76 for 100 cities). CGIRL reports 5.40 on TSP50, 6.36 on TSP70 and 6.82 on TSP80, which would be below the optimum on average.

  2. In general the experimental setup is littered with confounders, undertesting and inconsistencies. (i) CGIRL differs from Direct not only in archiving but in mutating c (random segment reversals, which are local-search moves on tours, or Gaussian noise on hidden states) and re-evaluating with the policy. Direct has neither of these modifications. (ii) The efficiency claim (Table 2) compares Direct with a long-window DT. It never reports CGIRL's own cost, including archive search, re-evaluation and 5 parallel agents. (iii) Table 3 gives TSP80 = 7.86 versus 6.82 in Table 1 and 6.76 in Table 4, and Memory = 12.17 versus 11.11. These should be identical, and 7.86 would be worse than DT, so these results are entirely uninterpretable/inconsistent.

Questions for authors
  1. How is 5.40 below the known TSP optimum possible?
  2. Please report Direct + the same mutation/re-evaluation step, and the number of policy forward passes per method.
  3. Can the authors reconcile all of the experimental numbers being reported?
  4. Can the authors strengthen the benchmark beyond TSP and a synthetic recall task?
Reviewer_bxgmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the growing computational cost of expanding context windows in in-context RL. The authors propose Cross-Generational In-Context RL (CGIRL) to mitigate this issue by grouping inference into generations and selectively transferring experience from one generation to the next. MAP-Elites is used to maintain an archive of candidate experiences and select those exhibiting improvements across trials. Experiments on TSP and sequence-recall (Memory) tasks show that CGIRL outperforms baselines such as within-generation context accumulation, retrieval, and direct experience transfer.

Critical strengths
  1. Good motivation and practical significance. The paper addresses the growing computational cost of expanding context windows in in-context RL and proposes a reasonable approach to maintaining a fixed context window through selective experience transfer.
  2. Promising empirical improvements. CGIRL outperforms the evaluated baselines across TSP and sequence-recall tasks. The comparison with direct experience transfer also supports the benefit of the proposed MAP-Elites-based transfer mechanism.
Critical weaknesses
  1. Somewhat limited methodological contribution. The proposed CGIRL combines existing techniques, including cross-generational experience transfer and MAP-Elites. The main contribution appears to be the application of MAP-Elites with a heuristically defined experience-selection criterion.
  2. Insufficient component ablations and efficiency evaluation. Although the paper provides comparisons across transfer operators, channels, and generation configurations, it does not isolate the contributions of MAP-Elites mutation, individual fitness components, or the return-dynamics-based selection criterion. Additionally, the inference-time comparison evaluates Direct rather than CGIRL, leaving the computational overhead of the complete proposed method unclear.
Questions for authors

Can the authors clarify to what extent the improvements over Direct arise from the proposed experience-selection mechanism, rather than the additional evolutionary search?

Reviewer_z6ECOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript proposes CGIRL, a cross-generational in-context reinforcement learning framework that improves test-time performance without updating model parameters. Unlike conventional in-context RL methods that accumulate historical trajectories in the context window, CGIRL divides inference into multiple generations with a fixed context size. MAP-Elites is introduced to select experiences for transmission based on within-generation return dynamics, rather than simply transferring the best trajectory. Experiments on TSP and a sequence-recall task demonstrate improved performance compared with existing context construction and transfer strategies.

Critical strengths
  1. This paper provides an interesting perspective on in-context reinforcement learning by introducing cross-generational experience transfer. Maintaining a fixed context window while selecting useful experiences across generations is meaningful for improving test-time inference efficiency.
  2. The experimental results are promising. Compared with several baseline methods, CGIRL achieves better performance on both TSP and Memory tasks. The evaluation on TSP70/80 using a model trained only on TSP50 also demonstrates the generalization ability of the proposed framework.
Critical weaknesses
  1. Although the paper claims improved inference efficiency, the computational comparison in Table 2 only evaluates Direct against the long-context DT baseline, rather than the complete CGIRL framework. The additional costs of maintaining the MAP-Elites archive, mutating experiences, and re-evaluating candidates are not considered. Therefore, it remains unclear whether CGIRL achieves better performance under the same computational budget.

  2. The contribution of MAP-Elites is not sufficiently justified. Although the comparison demonstrates the advantage of the proposed transfer strategy, it does not isolate whether the improvement comes from selecting experiences based on return dynamics, maintaining diverse candidates, or the additional evolutionary search. More evidence is needed to support the proposed experience selection mechanism.

Questions for authors
  • The authors should clarify how the additional computational costs of MAP-Elites are included in the inference budget. In particular, Table 2 reports the inference time of Direct rather than CGIRL. Since CGIRL requires additional policy evaluations during evolutionary search, comparing the same number of trials may not represent the same computational cost.

  • The authors should clarify whether the performance improvement comes from the proposed weak-early, strong-late experience selection or from the additional search and diversity preservation. The comparisonalone is insufficient to demonstrate the effectiveness of selecting experiences according to their within-generation return dynamics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.