- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose CGIRL, a RL training setup that combines MAP-Elites with cross-generation inference. Specifically, they consider cases where a system is able to keep some limited amount of past context over the course of inference, and train a model to effectively use this at inference time. At inference they run multiple samples in parallel and then use MAP-Elites to store and mutate promising context candidates. Experiments show that using MAP-Elites in this way for transfer works best relative to simpler context transfer methods, and other key aspects of the method (what to transmit, how many parallel agents to run, etc.) are also ablated.
Critical strengths
- The core idea as proposed is fairly reasonable and makes sense, being to use MAP-Elites in the cross-generational inference setting to ensure diversity, and providing a method for effectively training a model to be able to be used in this way. The results, while only on two fairly toy tasks, do seem to indicate that CGIRL improves over other methods.
- Many core aspects of CGIRL are ablated, such as transfer operator and context size.
Critical weaknesses
- There is not much analysis in this work as to why CGIRL works better than other methods, beyond that MAP-Elites 'better transmit(s) experience that improved within a generation' (lines 377-417). It would really benefit the paper to back up these claims with some analysis; for instance, some numbers or visualisation of what the diverse context is actually capturing, and how this has led to better performance. As it is, the paper makes a reasonable case that the algorithm works, but provides little understanding into why.
- I think the work also misses many core details: it would be useful to know golden/optimal TSP scores (or results in the fully-observed setting) to know how big of a gap remains; it would be useful to ablate the fitness functions (or at least the chosen weighting and the descriptor setup; and hyperparameter details are missing, which would be especially useful given reruns of the same setting seem to have non-trivial noise differences (For TSP-80, the default CGIRL setup seems to get 6.82 in Table 1, 7.86 in Table 3, and 6.76 in Table 4). A gap of 1 point is larger than some of the gains over baselines, so it would be useful to clarify this, and provide stderr numbers / confidence intervals for the evaluations if possible.
nit: I found the writing of this work a bit dense and terminology-laden in ways that reduced understanding. For example, in lines 247-256, the phrase "quantises the two halves onto a B × B grid using the task-inherent score upper bound s_max" does not explain the actual method well - but the following sentences do! B is introduced early but not defined until the final sentence of the paragraph, and the grid notation is similarly explained at the end, not the start of the paragraph. Another example is how samples are mutated, which is 'hidden' in single sentences inside Appendix B and difficult to find at a glance (the sentences saying "Search applies..."). These descriptions themselves are also somewhat short. In general, there is a pattern of dense language combined with undefined (or underdefined) terms that makes this work hard to read.
nit: typo: line 237, "equation equation 1"
Questions for authors
- How were the values in equation 8 (weights of the fitness values) decided?
- What is the variation between evaluation reruns for TSP and Memory? Why does CGIRL Table 3 TSP 80's score differ so much to tables 1 and 4?
- What sort of information ends up being captured in the memory? What sort of diversity does the method capture?
- What is the motivation for the descriptor function? Could you explain it further?