- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies an agent that retrieves from a growing store and appends its own answers. It measures a copying response as a function of retrieved contamination, then uses this to model how errors persist. The experiments show sharply different outcomes across seeds and suggest that grounding early can help more than spending the same budget later.
Critical strengths
-
The seed-level results are more informative than a single mean contamination score. I like the extended runs and the matched-budget timing experiments: they make the feedback problem tangible.
-
The authors report failed predictions and interventions rather than only successful examples. That is useful evidence about where the proposed model stops working.
Critical weaknesses
-
The sampling model does not match the stated process. Section 3 samples without replacement yet Definition 1 uses a binomial distribution; the exact distribution is hypergeometric. This matters early: with nine false records out of ten and eight draws, the chance of retrieving the true record is 0.8, not 1 - 0.9^8, approximately 0.57. I also could not find a derivation of the claimed critical threshold gamma = 1/k from the stated drift equation. Finally, the finite-store bound (n-1)/n approaches one as the store grows, so it does not by itself establish a nonzero gap from full contamination in the infinite-tenure limit. These points affect the paper's central explanation.
-
Some interpretations go further than the experiments support. Keeping a true record in the store does not establish that the model can still use it reliably. Likewise, pass@8 exceeding pass@1 shows that repeated sampling can recover a correct answer; it does not on its own distinguish retained capability from other sources of stochastic success. I would narrow the claims about capture and preserved capability unless there is a more direct test.
Questions for authors
-
Clarification: does the simulator sample with or without replacement? With the correct sampling law, which drift predictions and intervention conclusions remain unchanged, and what assumptions yield gamma = 1/k?
-
Clarification: is the infinite-tenure claim only about never reaching exactly one at a finite time, or about a limiting contamination fraction bounded away from one?
-
Please also reconcile Section 4.3's claim that source marking lowers capture and gamma with Table 2's statement that those two predictions failed.
-
Why are there some anonymous citations in the reference, e.g.:
Anonymous. The capability convergence hypothesis: Capability from access structure, not scale. arXiv preprint, 2026a. author list and repository withheld for double-blind review; a concurrent manuscript of the same research programme, arXiv identifier withheld.
Anonymous. Position: Preregistration Protects the Criterion, Not the Comparison: One Programme Audited Against Its Own Laws Seven Times. concurrent manuscript; author list and repository withheld for double-blind review, 2026b.
Anonymous. The Infinite-Rerun Limit: Computing what a criterion can decide, before it is frozen: a resolution, a rule, and the flip rate a band buys. concurrent manuscript; author list and repository withheld for double-blind review, 2026c.