- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces RecurLens, an executable auditing framework for evaluating recursive self-improvement in fixed-weight language-model agents. The central motivation is that improvements in downstream task performance do not necessarily imply improvements in an agent's ability to generate more effective future updates. The framework distinguishes five evidence dimensions: local update effects (G0), realized method inheritance (G1), descendant task effectiveness (G2), future-update utility under a common recipient (G3), and persistence across repeated handoffs (G4). It evaluates these dimensions using paired Frozen and Recursive lineages with three scheduled update opportunities. The experimental environment involves generating executable test plans to detect transaction and cache consistency faults. The authors evaluate multiple language models and introduce controlled experiments to distinguish task-policy improvements from changes in the improvement method itself. The results illustrate several important attribution issues. For example, one model achieves a positive Recursive-versus-Frozen endpoint contrast while performing worse than its initial state and without changing its improvement method. In another experiment, restoring the initial task policy substantially reduces the observed downstream benefit of a supplied reference method. Overall, the paper contributes a structured methodology for auditing recursive improvement claims. Its primary contribution lies in clarifying what different experimental observations can and cannot establish, rather than demonstrating successful recursive self-improvement.
Critical strengths
S1. A well-motivated decomposition of recursive self-improvement evidence. The paper addresses an important methodological problem: improvements in final task performance are often insufficient to establish that an agent has become better at improving itself. The G0–G4 framework provides a useful decomposition of local behavioral changes, method inheritance, descendant performance, downstream utility, and persistence. This distinction is valuable for interpreting self-improving agent experiments. S2. Careful attention to realized method inheritance and operational validity. A notable strength is the explicit distinction between a proposed method, a legally installed method, and a method actually consumed by a successor. The framework also distinguishes invalid outputs, missing observations, dependent skips, and valid negative outcomes. These distinctions help prevent misleading interpretations of agent improvement trajectories. S3. Useful controlled experiments exposing task-policy confounding. The common-recipient evaluation and task-policy restoration experiment provide an informative demonstration of attribution difficulties. In Table 6, the average reference-minus-base contrast decreases from 77.72 to 21.98 TES points when the task policy written by the first update is removed. This is an important illustration of why downstream performance differences cannot automatically be attributed to improved update-generation ability. S4. Transparent reporting of experimental limitations. The authors explicitly document incomplete observations, failed output-format qualification, response-provenance issues, and the distinction between exploratory and confirmatory evidence. The extensive appendices make the limitations of individual experiments relatively clear and reduce the risk of overstating the findings.
Critical weaknesses
W1. The end-to-end empirical demonstration is incomplete, especially for G3/G4. The paper’s negative-result framing is valuable: it explicitly shows that a favorable Recursive-versus-Frozen contrast need not imply a better improvement method. For example, Sol’s Recursive branch scores 5.38 TES points above Frozen but 17.09 points below its own T0 state without changing its method. This is a legitimate diagnostic finding, not a defect by itself. The weakness is that the natural lineages do not provide a fully archived, evaluable end-to-end demonstration of the complete G0–G4 chain. Several models have no changed-method inheritance; models with observed handoffs do not supply convincing downstream method-utility evidence; and the only archived Scout lineage satisfying the two-handoff G4 eligibility rule has no evaluable later utility. The later author-checked batch reports seven of 24 natural lineages with two substantive handoffs and five with evaluable utility, but its raw records are not attached to the replayable archive. Consequently, G3 is illustrated primarily through controlled experiments and incomplete natural-lineage diagnostics, whereas G4 remains without a confirmed, independently reproducible natural persistence result. This does not invalidate the audit framework, but it limits the empirical demonstration of its full capabilities. At least one fully reproducible lineage—even an unsuccessful one—showing proposal, legal installation, successor consumption, downstream task evaluation, and the G3/G4 readouts would substantially strengthen the paper. W2. Limited statistical power and uncertainty in the key controlled comparisons. The task-policy restoration experiment in Table 6 is particularly interesting, but it includes only four independent task packages. The authors report package-bootstrap confidence intervals and exact tests. However, the task-policy-restored contrast has a wide 95% confidence interval of [0, 43.96], and the exact tests have limited power with only four independent packages (p = 0.125 for the full contrast and p = 0.5 for the restored contrast). Moreover, the restored contrast is positive in only two packages. Consequently, the evidence is insufficient to establish a stable improvement-method effect. Larger independent experimental units and stronger uncertainty analyses are needed to determine whether the remaining contrast is robust. W3. Incomplete reproducibility of several important results. The manuscript acknowledges that the raw responses, scoring files, and intervention records underlying the later controlled experiments in Tables 7 and 8 are not included in the replayable archive. These experiments report potentially important findings concerning method-content sensitivity, feedback, transfer, and candidate selection. Without the corresponding records, these results cannot be independently reconstructed from the current submission. The authors state that the complete benchmark implementation, exact scoring configuration, and experimental artifacts will be released after review. While this release plan is understandable, the current submission does not yet permit independent reconstruction of the reported experiments. W4. Restricted evaluation domain and insufficient independently verifiable held-out evaluation. The main benchmark focuses on executable transaction and cache fault-detection tasks, primarily involving two fault mechanisms. Although this environment supports controlled analysis, it is unclear whether the proposed framework and its empirical findings generalize to broader agent tasks, such as coding, planning, or research workflows. The authors also acknowledge that the main evaluation panel was reused during development. This limits the strength of generalization claims and makes an independent held-out evaluation particularly important. Although the later experiments include new targets and task variations, their underlying records are not available for independent verification. Consequently, the submission still lacks a sufficiently documented held-out evaluation to support strong generalization claims. W5. The practical incremental value of the full framework remains insufficiently demonstrated. The manuscript does not yet demonstrate how each additional evidence dimension changes conclusions or improves method selection relative to simpler protocols. The archived candidate-selection experiment selects the same method under all three rules, so it provides no evidence of incremental selection value. The later positive selection contrast lacks replayable raw records, and the manuscript itself notes that a complete cumulative-readout ablation—holding one lineage record fixed while selectively revealing endpoints, handoffs, common-recipient utility, and coverage—was not run. While the framework demonstrates diagnostic value in identifying attribution failures, the additional predictive or decision-making value of G3 and G4 beyond simpler task-performance and handoff-based assessments remains insufficiently established. A component-level ablation or a controlled selection experiment with replayable records would materially strengthen the contribution. As written, the framework is well motivated and carefully qualified, but its practical incremental value over existing task-versus-improver evaluations remains insufficiently established. This is a substantive but addressable concern that does not necessarily require changing the core protocol.
Questions for authors
Q1. Demonstrating the complete audit. Could the authors provide at least one fully reproducible natural lineage showing the complete audit trace—raw updates, installation decisions, successor inputs, G0–G3 readouts, and the G4 eligibility/evaluability status? If the lineage is G4-eligible, the later utility readout would be especially useful. Even an unsuccessful lineage would help demonstrate operational completeness. Q2. Statistical robustness. Can the authors extend the task-policy restoration experiment beyond four independent task packages? In particular, how stable is the remaining 21.98-point contrast under additional independently generated packages, and how sensitive is it to the choice of recipient and task family? Q3. Attribution of improvement-method effects. In the common-recipient experiment, OUTER can modify the task policy before downstream evaluation. Beyond restoring the initial task policy, what additional interventions would be necessary to isolate changes in future-update ability from other effects of the method text or generated intermediate states? Q4. Incremental contribution over prior work. Could the authors clarify which specific conclusions RecurLens enables that could not be obtained from existing task-versus-improver evaluations, such as Promptbreeder, STOP, or harness-updating studies? A controlled comparison or component-level ablation would be particularly helpful. Q5. Reproducibility of later experiments. Will the authors provide an anonymized, policy-compliant minimal reproduction package during review—or, if that is not permitted, a precise plan and timeline for releasing the raw responses, scoring files, intervention records, and exact configurations for Tables 7 and 8? Q6. Generalization and held-out evaluation. How would the authors expect the framework to behave on tasks outside transaction and cache fault detection? Can the authors provide a genuinely held-out evaluation with independently generated tasks and fault mechanisms? Q7. Reliability of repeated handoffs. Given the limited number of evaluable natural G4 trajectories, what additional experimental design or sample size would be required to make statistically meaningful claims about persistence across successive improvement steps?
3 comments
Response to Reviewer dpWr part II
W2+Q2+Q3
Reviewer Concern (W2+Q2+Q3)
The four-package Table 6 comparison does not establish a robust Method effect. Can expanded recipient and task-family tests, together with causal interventions, distinguish Method content from Policy mediation, intermediate states, and stochastic variation?
Response
We completed the requested larger-sample robustness evaluation. The original four-package comparison was extended to 32 independently assigned SOURCE packages under newly frozen matched contracts. For the Qwen 27B recipient, 28 valid policy-restored pairs averaged +0.256 TES (95% CI [−3.01, 3.52]). A separate 8B recipient yielded 12 valid pairs, all with observed zero differences; the other assigned packages were not silently counted as zeros. We retained separate M1/M2 and A/B task-family analyses; they do not establish a positive residual invariant to task family. The historical +21.98 TES residual was not confirmed, and we do not represent the new contracts as an exact recreation of the historical deployment.
Our attribution study includes Method restoration, Policy restoration, combined rollback, actual post-OUTER state checks, same-Method regeneration, and length-matched controls. G3 is a matched-recipient downstream contrast, so Policy restoration estimates a restricted effect after blocking a possible Method-to-Policy-to-utility path. The current ten-lineage supplement supports its saved natural contrasts, but does not by itself identify a pure Method-content causal effect; that question requires the separate intervention studies.
In a separate preregistered 48-SOURCE M3 controlled-content experiment, 46 valid pairs yielded +27.34 percentage points (95% CI [14.57, 40.10], adjusted p = 0.000122; 14 positive, 32 zero, 0 negative). Policy-restored differences were zero, consistent with mediation by generated Task Policy. We therefore have evidence of controlled downstream content sensitivity and measured stochastic variation, while keeping the unconfirmed historical direct residual and natural isolated Method effect distinct.
W3+Q5
Reviewer Concern (W3+Q5)
Tables 7–8 lack independently replayable raw and intervention records. Can the authors supply a compliant reviewer package or a concrete release plan?
Response
The current review-stage supplement provides the ten-lineage structured artifacts, including the complete Scout 06 operational trace, together with the TES scoring code and offline replay dependencies. All 680 saved TASK scores passed arithmetic checks and scoring-engine re-execution. These are two checks of the same evaluation set, not 1,360 independent evaluations. Offline replay recomputes the saved utility measurements; it does not regenerate model outputs or independently re-adjudicate qualification. The larger Table 6 matched validations and other controlled studies are separate from this ten-lineage supplement. The supplied materials support offline recomputation of the included natural-lineage measurements.
We provide partial reproducibility materials for review and will open-source the remaining retained, redistributable materials upon acceptance. The planned release includes controlled-study code and data, available execution artifacts, configurations, and a claim-to-artifact index. The new Table 6 experiments address the requested expanded matched validation; they are not exact reconstructions of historical deployment conditions. We do not claim that historical Tables 6–8 and Table 24 have been fully reconstructed. The current supplement is available through the anonymous link below.
Reviewer package: https://anonymous.4open.science/r/recurlens-supplement-2750/.
Response to Reviewer dpWr part III
W4+Q6
Reviewer Concern (W4+Q6)
The evaluation relies on a narrow, partly reused fault-detection panel. Can genuinely held-out tasks and new mechanisms, together with cross-domain tests, support broader applicability?
Response
We added a separately constructed M3 fault mechanism and a preregistered evaluation cohort of 48 independently assigned SOURCE packages. Of these, 46 yielded valid controlled comparisons, with a mean content-intervention effect of +27.34 percentage points (95% CI [14.57, 40.10], adjusted p = 0.000122). This is held-out confirmation beyond the original M1/M2 development mechanism and panel, while remaining within executable fault detection.
We also implemented the evaluation contract on HumanEval and MBPP, obtaining eight valid controlled comparisons in each setting. These runs test protocol execution on coding tasks but do not confirm positive cross-domain Method utility. G0–G4 can be transferred when a task supplies an auditable update artifact, legal successor input, and an independently executable utility measure; for coding this is test-based success. Planning and research tasks would require their own predeclared scoring contracts. We therefore make a scoped claim of held-out confirmation within fault detection and initial cross-domain implementation, not established generalization of natural method gains.
W5+Q4
Reviewer Concern (W5+Q4)
The incremental value of G3/G4 over simpler task-and-improver evaluations and existing self-improvement studies is unclear. Can a cumulative ablation or controlled comparison identify what additional conclusions RecurLens supports?
Response
We ran the requested cumulative-readout ablation on 180 diagnostic questions nested within 12 independent task packages, with a fixed candidate pool. Handoff-aware evaluation correctly resolved 72/180, a task-and-improver evaluator 108/180, and a stronger task-and-improver comparator 120/180. Full RecurLens resolved 132/180. The additional 12 verified conditional findings concern whether a downstream contrast survives Task Policy restoration. They required 96 restoration-related TASK evaluations and 296,874 tokens. We report this acquisition cost alongside coverage.
We also gave the strong comparator the same restoration evidence. It then resolved 132/180, matching Full RecurLens. Thus, the supported increment is access to verified intervention-conditioned information, not a superior decision rule on identical evidence. Neither improved Method selection nor lower decision regret was established. The archived candidate-selection experiment also selected the same Method under its three rules, and the later historical positive selection claim lacks complete replayable raw. We do not use either result to assert a selection gain. This matched-information control makes the source of the 12 additional findings explicit.
Promptbreeder (Fernando et al., 2024) [1], STOP (Zelikman et al., 2024) [2], and harness-updating studies already examine forms of self-improvement; we do not characterize them as endpoint-only evaluations. RecurLens instead standardizes the evidentiary status of method changes. It requires distinct records for proposed updates, legal installations, successor consumption, matched future-update utility, and qualified persistence. The natural Scout follow-up illustrates a conclusion unavailable from handoff counts alone: the two included eligible two-handoff chains had zero measured G4 utility. We therefore claim a concrete audit and evidence-acquisition contribution, with its measured cost and scope, rather than performance superiority over earlier self-improvement systems or equal-information evaluators.
References
[1] Fernando, C., Banarse, D. S., Michalewski, H., Osindero, S., and Rocktäschel, T. (2024). Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution. Proceedings of the 41st International Conference on Machine Learning (ICML), PMLR 235, 13481–13544. https://proceedings.mlr.press/v235/fernando24a.html
[2] Zelikman, E., Lorch, E., Mackey, L., and Kalai, A. T. (2024). Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation. Conference on Language Models (COLM). https://proceedings.colm.cc/paper_files/paper/2024/hash/52eb24c316c78cc144dfa54f1f4bb4ec-Abstract-Conference.html
Response to Reviewer part dpWr I
Summary
We thank the reviewer for the constructive assessment and for suggesting concrete tests of incremental value, held-out generalization, and G4 reliability. We address the requested end-to-end trace, expanded comparisons, causal controls, independent evaluation, incremental-value ablation, and G4 sampling plan. The current ten-lineage supplement includes a complete structured Scout
06operational trace and two measured native G4 null outcomes under the primary rules. Expanded Table 6 validations do not confirm the historical residual; the separate 48-SOURCE M3 study supports controlled downstream responsiveness. The 180-question study reports 12 additional intervention-conditioned findings, while an equal-information strong comparator closes the coverage gap. The available experiments establish audit applicability and bounded diagnostic value, not population-level persistence or a selection advantage. We distinguish these separate studies from the current package and reserve population G4 claims for adequately powered evaluations.W1+Q1+Q7
Reviewer Concern (W1+Q1+Q7)
Response
The current supplement contains ten natural lineages: two Scout, five Luna, and three Terra, evaluated under one frozen G0–G4 protocol. Scout
UNIFIED_V3_NATURAL_06provides a complete structured G0–G4 operational trace. The archived states and messages connect Method revisions and legal installations to actual successor inputs, same-SOURCE G0 comparisons, fixed-transfer G2 evaluations, native G3 readouts, and the in-window G4 follow-up. We checked that installed states continue into the next generation and that the preceding Method appears in the successor input. Scout10also meets the frozen primary operational criteria. Independent assessments disagree on the substantive content of some edits, so operational completeness is not treated as independently agreed evidence of a better update algorithm.Scout
06and10meet the saved primary consecutive-handoff G4 qualification rules and each has an evaluable native G4 effect of 0 TES. The other eight lineages are not counted as zero-effect G4 observations. All ten have evaluable G3 contrasts: five Luna and two Scout readouts are 0 TES, while Terra03,04, and09yield +6.07, +0.48, and +13.60 TES (mean +6.72 TES). Across these ten lineages, the mean is +2.01 TES, with a Student-t 95% interval of [−1.20, 5.23] (9 degrees of freedom). These statistics describe the ten natural lineages evaluated here. The Terra values are downstream contrasts, not isolated causal Method-content gains. This trace demonstrates operational completeness without assuming successful RSI.For a future population-level G4 claim, the unit must be an independently enrolled qualified, evaluable lineage, with ineligible attempts and missing follow-up accounted for separately. Under a prespecified utility bound of [−100, 100] TES, Hoeffding's inequality gives 738 independent evaluable lineages as a conservative sufficient sample size for a 95% mean interval with 10 TES half-width. This is a precision target, not a universal minimum. The current observations establish persistence evaluability, not population equivalence to zero. The complete structured Scout
06operational trace and its scoring code are included in the current supplement.