- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors argue that in a self-evolving skill library, an LLM evaluator's preference for task-irrelevant attributes (e.g., formatting) can be amplified by recursive loops into a permanent shift in library composition. To address this, the paper proposes CRR (Causal Replicator–Retrieval), which models three mechanisms separately and uses them to mitigate the shift on three synthetic tasks.
Critical strengths
None
Critical weaknesses
The paper does not demonstrate that the problem CRR aims to solve arises in practice, and simulated dynamics and highly synthetic tasks alone can hardly establish it. Specifically, the main evidence (e.g., Table 1) comes from a simulator that implements CRR's own dynamics, and the bias in the sandbox is injected in a very naive way (e.g., a "VERIFIED" header in the code comments). For these reasons, I lean toward clear rejection.
The method relies on many design choices (e.g., which preference bias the evaluator has) and hyperparameters (e.g., ϵ_q, the stability margin m, and the fixation threshold) that are unknown in advance, even during deployment. In addition, many design choices are not well motivated (e.g., why a two-dimensional mean-field state with logistic and softmax forms is sufficient). This leaves me doubtful about the method's applicability to real skill libraries.
Note: The authors do not need to respond to the weaknesses above separately; answers to the questions below are sufficient.
Questions for authors
- Can the authors provide a less synthetic example that supports their main claim, namely that an LLM evaluator's preference for task-irrelevant attributes (e.g., formatting) can be amplified by recursive loops into a permanent shift?
- Could the authors describe in detail how the design choices (including hyperparameters) would be applied to a real skill library? For example, when z is unknown or twin matching is unavailable, how can the mechanism parameters be identified?
- As noted in the Weaknesses, how were these complex design choices motivated? This concern extends beyond the example given there; other choices also appear arbitrary (e.g., the δ + ηh_t interaction form, the exposure update based on p_{t+1}, and updating the controller every five generations). Were any alternatives compared?