- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper asks what evidence is needed to distinguish improved task artifacts from improved recursive self-improvement mechanisms. It defines a finite-horizon witness requiring target gain (G), a matched install-versus-retain effect (I), and higher competence of a self-produced successor (R), with evidence of installation and continuation along one lineage. It derives statistical and identifiability conditions and retrospectively inspects public releases and 17 benchmark families. The paper does not run a prospective RSI verification experiment.
Critical strengths
The distinction between artifact improvement and improvement of the updater is important, and the requirement that the successor actually be installed and run is well motivated. The paper correctly notes that under direct continuation R_g = I_{g+1}; these are not independent pieces of evidence.
The public-record audit is unusually careful about what its numbers can establish. For example, recovering source metadata for 5,596 of 5,597 AutoResearch attempts repairs a documentation gap without being mistaken for proof of a causal installation effect. The analysis also avoids treating dependent transitions or absent evidence as negative RSI outcomes.
Critical weaknesses
-
The proposed verification standard is never exercised on a complete observed lineage. Section 3 explicitly estimates none of G, I, or R for the inspected systems. The analytical C_k construction in Appendix C.2 establishes logical feasibility, but does not test whether complete-state forks, independently scored successors, or physical install/continuation checks can be implemented reliably. Without even a small controlled end-to-end demonstration—including nonpassing cases, abstentions, and cost—the paper's main operational claim remains a specification rather than a validated method.
-
The theoretical contribution needs a sharper boundary. Proposition 1 is conditional Hoeffding with a union bound; Proposition 2 is a telescoping argument; Theorem 1 derives nonidentification from a model class explicitly closed under changes to omitted facts; and Theorem 2 combines a lower confidence sequence with a separately assumed false-pass bound. These statements appear coherent, but do not themselves establish a new verification algorithm. More fundamentally, R compares the next updater's artifact-production competence, not its ability to design further updaters. The deliberately programmed C_k sequence satisfies the witness, so the formal claim is narrower than a demonstration of acquired self-improvement capability. The authors acknowledge this choice, but its consequences for the claimed contribution should be made explicit.
-
The 17-family comparison is too coding-sensitive to carry substantial empirical weight. The two blinded recodings agree on 171/187 resolved cells, yet agree with the original on only 92/193 and 102/190. Both assign zero explicit future-replay codes where the original assigned seven. Without independent adjudication, Figure 4 is useful evidence of interpretation instability, not a reliable assessment of benchmark coverage. The authors largely recognize this limitation; the issue is that this leaves little direct empirical support for the proposed standard.
Questions for authors
- Can the authors provide one executed two-stage lineage with the actual matched install/retain forks, independently confirmed G/I/R, and verified successor installation, including the result when a required link is absent?
- Which mathematical guarantee is genuinely new relative to existing controlled-update and anytime-valid confirmation frameworks, beyond the choice of conjunction and estimand?
- Would human-adjudicated coding materially change the conclusions drawn from Figure 4?