acceptodds
Withdrawn from ICLR 2027

RecurLens: Tracing the Evidence for Recursive Self-Improvement

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

When an agent improves after revising itself, the final task score does not reveal which part of the revision worked. The new method may never reach a successor, and a stronger descendant may still be worse at producing its next update. We introduce , an executable audit for fixed-weight agents that records proposed and installed policies, follows their use across three update opportunities, and tests inherited methods in a common recipient. The audit exposes separations that an endpoint score hides. Sol's Recursive descendant scores 5.38 points above its Frozen counterpart but 17.09 below its own initial state, with no method change. A supplied method reaches 136 successors in a later handoff condition, while failed qualification checks limit the utility inference. In four independent packages, its full downstream contrast falls from 77.72 to 21.98 points when the task policy written by the first update is removed. Thus task gains, actual inheritance, and later improvement utility require separate evidence; the recorded runs establish some links while leaving others unresolved. We will release the benchmark implementation and reproducibility materials after review.

Primary area:
datasets and benchmarks
Keywords:
recursive self-improvement, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dpWrOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RecurLens, an executable auditing framework for evaluating recursive self-improvement in fixed-weight language-model agents. The central motivation is that improvements in downstream task performance do not necessarily imply improvements in an agent's ability to generate more effective future updates. The framework distinguishes five evidence dimensions: local update effects (G0), realized method inheritance (G1), descendant task effectiveness (G2), future-update utility under a common recipient (G3), and persistence across repeated handoffs (G4). It evaluates these dimensions using paired Frozen and Recursive lineages with three scheduled update opportunities. The experimental environment involves generating executable test plans to detect transaction and cache consistency faults. The authors evaluate multiple language models and introduce controlled experiments to distinguish task-policy improvements from changes in the improvement method itself. The results illustrate several important attribution issues. For example, one model achieves a positive Recursive-versus-Frozen endpoint contrast while performing worse than its initial state and without changing its improvement method. In another experiment, restoring the initial task policy substantially reduces the observed downstream benefit of a supplied reference method. Overall, the paper contributes a structured methodology for auditing recursive improvement claims. Its primary contribution lies in clarifying what different experimental observations can and cannot establish, rather than demonstrating successful recursive self-improvement.

Critical strengths

S1. A well-motivated decomposition of recursive self-improvement evidence. The paper addresses an important methodological problem: improvements in final task performance are often insufficient to establish that an agent has become better at improving itself. The G0–G4 framework provides a useful decomposition of local behavioral changes, method inheritance, descendant performance, downstream utility, and persistence. This distinction is valuable for interpreting self-improving agent experiments. S2. Careful attention to realized method inheritance and operational validity. A notable strength is the explicit distinction between a proposed method, a legally installed method, and a method actually consumed by a successor. The framework also distinguishes invalid outputs, missing observations, dependent skips, and valid negative outcomes. These distinctions help prevent misleading interpretations of agent improvement trajectories. S3. Useful controlled experiments exposing task-policy confounding. The common-recipient evaluation and task-policy restoration experiment provide an informative demonstration of attribution difficulties. In Table 6, the average reference-minus-base contrast decreases from 77.72 to 21.98 TES points when the task policy written by the first update is removed. This is an important illustration of why downstream performance differences cannot automatically be attributed to improved update-generation ability. S4. Transparent reporting of experimental limitations. The authors explicitly document incomplete observations, failed output-format qualification, response-provenance issues, and the distinction between exploratory and confirmatory evidence. The extensive appendices make the limitations of individual experiments relatively clear and reduce the risk of overstating the findings.

Critical weaknesses

W1. The end-to-end empirical demonstration is incomplete, especially for G3/G4. The paper’s negative-result framing is valuable: it explicitly shows that a favorable Recursive-versus-Frozen contrast need not imply a better improvement method. For example, Sol’s Recursive branch scores 5.38 TES points above Frozen but 17.09 points below its own T0 state without changing its method. This is a legitimate diagnostic finding, not a defect by itself. The weakness is that the natural lineages do not provide a fully archived, evaluable end-to-end demonstration of the complete G0–G4 chain. Several models have no changed-method inheritance; models with observed handoffs do not supply convincing downstream method-utility evidence; and the only archived Scout lineage satisfying the two-handoff G4 eligibility rule has no evaluable later utility. The later author-checked batch reports seven of 24 natural lineages with two substantive handoffs and five with evaluable utility, but its raw records are not attached to the replayable archive. Consequently, G3 is illustrated primarily through controlled experiments and incomplete natural-lineage diagnostics, whereas G4 remains without a confirmed, independently reproducible natural persistence result. This does not invalidate the audit framework, but it limits the empirical demonstration of its full capabilities. At least one fully reproducible lineage—even an unsuccessful one—showing proposal, legal installation, successor consumption, downstream task evaluation, and the G3/G4 readouts would substantially strengthen the paper. W2. Limited statistical power and uncertainty in the key controlled comparisons. The task-policy restoration experiment in Table 6 is particularly interesting, but it includes only four independent task packages. The authors report package-bootstrap confidence intervals and exact tests. However, the task-policy-restored contrast has a wide 95% confidence interval of [0, 43.96], and the exact tests have limited power with only four independent packages (p = 0.125 for the full contrast and p = 0.5 for the restored contrast). Moreover, the restored contrast is positive in only two packages. Consequently, the evidence is insufficient to establish a stable improvement-method effect. Larger independent experimental units and stronger uncertainty analyses are needed to determine whether the remaining contrast is robust. W3. Incomplete reproducibility of several important results. The manuscript acknowledges that the raw responses, scoring files, and intervention records underlying the later controlled experiments in Tables 7 and 8 are not included in the replayable archive. These experiments report potentially important findings concerning method-content sensitivity, feedback, transfer, and candidate selection. Without the corresponding records, these results cannot be independently reconstructed from the current submission. The authors state that the complete benchmark implementation, exact scoring configuration, and experimental artifacts will be released after review. While this release plan is understandable, the current submission does not yet permit independent reconstruction of the reported experiments. W4. Restricted evaluation domain and insufficient independently verifiable held-out evaluation. The main benchmark focuses on executable transaction and cache fault-detection tasks, primarily involving two fault mechanisms. Although this environment supports controlled analysis, it is unclear whether the proposed framework and its empirical findings generalize to broader agent tasks, such as coding, planning, or research workflows. The authors also acknowledge that the main evaluation panel was reused during development. This limits the strength of generalization claims and makes an independent held-out evaluation particularly important. Although the later experiments include new targets and task variations, their underlying records are not available for independent verification. Consequently, the submission still lacks a sufficiently documented held-out evaluation to support strong generalization claims. W5. The practical incremental value of the full framework remains insufficiently demonstrated. The manuscript does not yet demonstrate how each additional evidence dimension changes conclusions or improves method selection relative to simpler protocols. The archived candidate-selection experiment selects the same method under all three rules, so it provides no evidence of incremental selection value. The later positive selection contrast lacks replayable raw records, and the manuscript itself notes that a complete cumulative-readout ablation—holding one lineage record fixed while selectively revealing endpoints, handoffs, common-recipient utility, and coverage—was not run. While the framework demonstrates diagnostic value in identifying attribution failures, the additional predictive or decision-making value of G3 and G4 beyond simpler task-performance and handoff-based assessments remains insufficiently established. A component-level ablation or a controlled selection experiment with replayable records would materially strengthen the contribution. As written, the framework is well motivated and carefully qualified, but its practical incremental value over existing task-versus-improver evaluations remains insufficiently established. This is a substantive but addressable concern that does not necessarily require changing the core protocol.

Questions for authors

Q1. Demonstrating the complete audit. Could the authors provide at least one fully reproducible natural lineage showing the complete audit trace—raw updates, installation decisions, successor inputs, G0–G3 readouts, and the G4 eligibility/evaluability status? If the lineage is G4-eligible, the later utility readout would be especially useful. Even an unsuccessful lineage would help demonstrate operational completeness. Q2. Statistical robustness. Can the authors extend the task-policy restoration experiment beyond four independent task packages? In particular, how stable is the remaining 21.98-point contrast under additional independently generated packages, and how sensitive is it to the choice of recipient and task family? Q3. Attribution of improvement-method effects. In the common-recipient experiment, OUTER can modify the task policy before downstream evaluation. Beyond restoring the initial task policy, what additional interventions would be necessary to isolate changes in future-update ability from other effects of the method text or generated intermediate states? Q4. Incremental contribution over prior work. Could the authors clarify which specific conclusions RecurLens enables that could not be obtained from existing task-versus-improver evaluations, such as Promptbreeder, STOP, or harness-updating studies? A controlled comparison or component-level ablation would be particularly helpful. Q5. Reproducibility of later experiments. Will the authors provide an anonymized, policy-compliant minimal reproduction package during review—or, if that is not permitted, a precise plan and timeline for releasing the raw responses, scoring files, intervention records, and exact configurations for Tables 7 and 8? Q6. Generalization and held-out evaluation. How would the authors expect the framework to behave on tasks outside transaction and cache fault detection? Can the authors provide a genuinely held-out evaluation with independently generated tasks and fault mechanisms? Q7. Reliability of repeated handoffs. Given the limited number of evaluable natural G4 trajectories, what additional experimental design or sample size would be required to make statistically meaningful claims about persistence across successive improvement steps?

3 comments
AuthorsOpenReview ↗

Response to Reviewer part dpWr I

Summary

We thank the reviewer for the constructive assessment and for suggesting concrete tests of incremental value, held-out generalization, and G4 reliability. We address the requested end-to-end trace, expanded comparisons, causal controls, independent evaluation, incremental-value ablation, and G4 sampling plan. The current ten-lineage supplement includes a complete structured Scout 06 operational trace and two measured native G4 null outcomes under the primary rules. Expanded Table 6 validations do not confirm the historical residual; the separate 48-SOURCE M3 study supports controlled downstream responsiveness. The 180-question study reports 12 additional intervention-conditioned findings, while an equal-information strong comparator closes the coverage gap. The available experiments establish audit applicability and bounded diagnostic value, not population-level persistence or a selection advantage. We distinguish these separate studies from the current package and reserve population G4 claims for adequately powered evaluations.

W1+Q1+Q7

Reviewer Concern (W1+Q1+Q7)

The G0–G4 audit lacks a fully reproducible natural demonstration, particularly a valid G4 follow-up. Can the authors provide a complete lineage, even with null utility, and specify a design and sample size for statistically reliable persistence estimates?

Response

The current supplement contains ten natural lineages: two Scout, five Luna, and three Terra, evaluated under one frozen G0–G4 protocol. Scout UNIFIED_V3_NATURAL_06 provides a complete structured G0–G4 operational trace. The archived states and messages connect Method revisions and legal installations to actual successor inputs, same-SOURCE G0 comparisons, fixed-transfer G2 evaluations, native G3 readouts, and the in-window G4 follow-up. We checked that installed states continue into the next generation and that the preceding Method appears in the successor input. Scout 10 also meets the frozen primary operational criteria. Independent assessments disagree on the substantive content of some edits, so operational completeness is not treated as independently agreed evidence of a better update algorithm.

Scout 06 and 10 meet the saved primary consecutive-handoff G4 qualification rules and each has an evaluable native G4 effect of 0 TES. The other eight lineages are not counted as zero-effect G4 observations. All ten have evaluable G3 contrasts: five Luna and two Scout readouts are 0 TES, while Terra 03, 04, and 09 yield +6.07, +0.48, and +13.60 TES (mean +6.72 TES). Across these ten lineages, the mean is +2.01 TES, with a Student-t 95% interval of [−1.20, 5.23] (9 degrees of freedom). These statistics describe the ten natural lineages evaluated here. The Terra values are downstream contrasts, not isolated causal Method-content gains. This trace demonstrates operational completeness without assuming successful RSI.

For a future population-level G4 claim, the unit must be an independently enrolled qualified, evaluable lineage, with ineligible attempts and missing follow-up accounted for separately. Under a prespecified utility bound of [−100, 100] TES, Hoeffding's inequality gives 738 independent evaluable lineages as a conservative sufficient sample size for a 95% mean interval with 10 TES half-width. This is a precision target, not a universal minimum. The current observations establish persistence evaluability, not population equivalence to zero. The complete structured Scout 06 operational trace and its scoring code are included in the current supplement.

AuthorsOpenReview ↗

Response to Reviewer dpWr part II

W2+Q2+Q3

Reviewer Concern (W2+Q2+Q3)

The four-package Table 6 comparison does not establish a robust Method effect. Can expanded recipient and task-family tests, together with causal interventions, distinguish Method content from Policy mediation, intermediate states, and stochastic variation?

Response

We completed the requested larger-sample robustness evaluation. The original four-package comparison was extended to 32 independently assigned SOURCE packages under newly frozen matched contracts. For the Qwen 27B recipient, 28 valid policy-restored pairs averaged +0.256 TES (95% CI [−3.01, 3.52]). A separate 8B recipient yielded 12 valid pairs, all with observed zero differences; the other assigned packages were not silently counted as zeros. We retained separate M1/M2 and A/B task-family analyses; they do not establish a positive residual invariant to task family. The historical +21.98 TES residual was not confirmed, and we do not represent the new contracts as an exact recreation of the historical deployment.

Our attribution study includes Method restoration, Policy restoration, combined rollback, actual post-OUTER state checks, same-Method regeneration, and length-matched controls. G3 is a matched-recipient downstream contrast, so Policy restoration estimates a restricted effect after blocking a possible Method-to-Policy-to-utility path. The current ten-lineage supplement supports its saved natural contrasts, but does not by itself identify a pure Method-content causal effect; that question requires the separate intervention studies.

In a separate preregistered 48-SOURCE M3 controlled-content experiment, 46 valid pairs yielded +27.34 percentage points (95% CI [14.57, 40.10], adjusted p = 0.000122; 14 positive, 32 zero, 0 negative). Policy-restored differences were zero, consistent with mediation by generated Task Policy. We therefore have evidence of controlled downstream content sensitivity and measured stochastic variation, while keeping the unconfirmed historical direct residual and natural isolated Method effect distinct.

W3+Q5

Reviewer Concern (W3+Q5)

Tables 7–8 lack independently replayable raw and intervention records. Can the authors supply a compliant reviewer package or a concrete release plan?

Response

The current review-stage supplement provides the ten-lineage structured artifacts, including the complete Scout 06 operational trace, together with the TES scoring code and offline replay dependencies. All 680 saved TASK scores passed arithmetic checks and scoring-engine re-execution. These are two checks of the same evaluation set, not 1,360 independent evaluations. Offline replay recomputes the saved utility measurements; it does not regenerate model outputs or independently re-adjudicate qualification. The larger Table 6 matched validations and other controlled studies are separate from this ten-lineage supplement. The supplied materials support offline recomputation of the included natural-lineage measurements.

We provide partial reproducibility materials for review and will open-source the remaining retained, redistributable materials upon acceptance. The planned release includes controlled-study code and data, available execution artifacts, configurations, and a claim-to-artifact index. The new Table 6 experiments address the requested expanded matched validation; they are not exact reconstructions of historical deployment conditions. We do not claim that historical Tables 6–8 and Table 24 have been fully reconstructed. The current supplement is available through the anonymous link below.

Reviewer package: https://anonymous.4open.science/r/recurlens-supplement-2750/.

AuthorsOpenReview ↗

Response to Reviewer dpWr part III

W4+Q6

Reviewer Concern (W4+Q6)

The evaluation relies on a narrow, partly reused fault-detection panel. Can genuinely held-out tasks and new mechanisms, together with cross-domain tests, support broader applicability?

Response

We added a separately constructed M3 fault mechanism and a preregistered evaluation cohort of 48 independently assigned SOURCE packages. Of these, 46 yielded valid controlled comparisons, with a mean content-intervention effect of +27.34 percentage points (95% CI [14.57, 40.10], adjusted p = 0.000122). This is held-out confirmation beyond the original M1/M2 development mechanism and panel, while remaining within executable fault detection.

We also implemented the evaluation contract on HumanEval and MBPP, obtaining eight valid controlled comparisons in each setting. These runs test protocol execution on coding tasks but do not confirm positive cross-domain Method utility. G0–G4 can be transferred when a task supplies an auditable update artifact, legal successor input, and an independently executable utility measure; for coding this is test-based success. Planning and research tasks would require their own predeclared scoring contracts. We therefore make a scoped claim of held-out confirmation within fault detection and initial cross-domain implementation, not established generalization of natural method gains.

W5+Q4

Reviewer Concern (W5+Q4)

The incremental value of G3/G4 over simpler task-and-improver evaluations and existing self-improvement studies is unclear. Can a cumulative ablation or controlled comparison identify what additional conclusions RecurLens supports?

Response

We ran the requested cumulative-readout ablation on 180 diagnostic questions nested within 12 independent task packages, with a fixed candidate pool. Handoff-aware evaluation correctly resolved 72/180, a task-and-improver evaluator 108/180, and a stronger task-and-improver comparator 120/180. Full RecurLens resolved 132/180. The additional 12 verified conditional findings concern whether a downstream contrast survives Task Policy restoration. They required 96 restoration-related TASK evaluations and 296,874 tokens. We report this acquisition cost alongside coverage.

We also gave the strong comparator the same restoration evidence. It then resolved 132/180, matching Full RecurLens. Thus, the supported increment is access to verified intervention-conditioned information, not a superior decision rule on identical evidence. Neither improved Method selection nor lower decision regret was established. The archived candidate-selection experiment also selected the same Method under its three rules, and the later historical positive selection claim lacks complete replayable raw. We do not use either result to assert a selection gain. This matched-information control makes the source of the 12 additional findings explicit.

Promptbreeder (Fernando et al., 2024) [1], STOP (Zelikman et al., 2024) [2], and harness-updating studies already examine forms of self-improvement; we do not characterize them as endpoint-only evaluations. RecurLens instead standardizes the evidentiary status of method changes. It requires distinct records for proposed updates, legal installations, successor consumption, matched future-update utility, and qualified persistence. The natural Scout follow-up illustrates a conclusion unavailable from handoff counts alone: the two included eligible two-handoff chains had zero measured G4 utility. We therefore claim a concrete audit and evidence-acquisition contribution, with its measured cost and scope, rather than performance superiority over earlier self-improvement systems or equal-information evaluators.


References

[1] Fernando, C., Banarse, D. S., Michalewski, H., Osindero, S., and Rocktäschel, T. (2024). Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution. Proceedings of the 41st International Conference on Machine Learning (ICML), PMLR 235, 13481–13544. https://proceedings.mlr.press/v235/fernando24a.html

[2] Zelikman, E., Lorch, E., Mackey, L., and Kalai, A. T. (2024). Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation. Conference on Language Models (COLM). https://proceedings.colm.cc/paper_files/paper/2024/hash/52eb24c316c78cc144dfa54f1f4bb4ec-Abstract-Conference.html

Reviewer_5ULYOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

RecurLens studies how to evaluate recursive self-improvement without treating a better final task score as sufficient evidence. It separates five checks: local update effects (G0), realized method handoff (G1), descendant task performance (G2), later utility in a common recipient (G3), and persistence across repeated handoffs (G4). The evaluation uses a cache-fault test-writing task, Frozen and Recursive lineages, and several supplied-method controls.

The main results are diagnostic rather than demonstrations of recursive self-improvement. Sol’s Recursive endpoint is 5.38 TES points above Frozen but 17.09 below its initial state, with no method change. A repaired interface delivers a reference method to 136 of 144 successors, but missing outcomes and qualification failures still block a confirmatory utility estimate. In Table 6, restoring the initial task policy reduces the mean reference-minus-base contrast from 77.72 to 21.98 across four packages. I therefore read the paper as a careful auditing framework whose natural-run validation is still incomplete.

Critical strengths
  1. The paper separates evidence stages that are often conflated. The G0–G4 definitions distinguish a proposed update, an installed method, actual successor use, task performance, and later utility (p. 4, Secs. 2.3–2.5; pp. 12–14, App. A). This matters because a favorable endpoint alone cannot identify which part of a revision produced the result.
  2. The diagnostic examples are useful and interpreted cautiously. Sol’s contrasting endpoint comparisons and the task-policy clamp in Table 6 show how the audit can change an interpretation (pp. 5–7). The paper is also careful about missing outcomes, reused responses, and provenance instead of turning unresolved cases into success or failure claims (pp. 18–19, App. E). These are the main reasons I see value in the framework despite my recommendation.
Critical weaknesses
  • The natural runs do not validate the most distinctive G3/G4 measurements end to end. Luna’s G3 bound crosses zero, Terra’s positive readout lacks verified independent provenance, and Scout’s partial observed contrast is zero (p. 25, Tables 17–18). The only archived G4-eligible Scout lineage has no evaluable later utility, and App. G.1 says there is no source-complete successful natural lineage (pp. 8, 21, and 25). The controlled positive results do not close this gap: Table 6 has four independent packages, and the later contrasts in Tables 7–8 and 24 lack raw records and intervals. The issue is not simply the lack of a positive RSI result; the key measurements remain unavailable or exploratory.
  • I could not reconstruct the central results from the supplied materials. The manuscript package does not include the experimental release (p. 10). Construction manifests, exact TES aggregation weights, complete natural traces, and raw records for the later manually checked contrasts are unavailable (pp. 14, 17–18, 21, and 29). This is consequential for a paper whose main contribution is an evidence audit, because readers cannot independently check the scoring and intervention records behind its main examples.
Questions for authors
  • Which natural run provides the strongest end-to-end validation of G3/G4? Can you provide its complete raw trace and clarify whether any natural run has a verified nonzero G3 effect or evaluable G4 result under one frozen condition?
  • Can you release the exact TES weights and scorer, task and SOURCE manifests, prompts and schemas, configurations, and raw response, scoring, and intervention records needed to reconstruct Tables 6–8 and one complete natural lineage?
  • Would the residual METHOD_CLAMP contrast in Table 6 remain in a prespecified comparison with more independent packages while holding deployment and output contract fixed?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 5ULY part I

Summary

We thank the reviewer for emphasizing independent reproducibility and the statistical robustness of the G3/G4 evaluations. We respond to the concerns about incomplete natural G3/G4 evaluation, the four-package METHOD_CLAMP contrast, and reproducibility. The current ten natural lineages provide a complete structured Scout 06 trace, two complete operational certificates, ten evaluable G3 readouts, and two qualified native G4 measurements. Expanded matched Table 6 validations do not confirm the historical +21.98 TES residual; the independent 48-SOURCE M3 study supports a different controlled Method-content contrast. The natural-audit and controlled-calibration claims are supported by distinct experiments and are not conflated. We keep historical results, the current natural-lineage package, and the additional controlled studies separate.

W1+Q1

Reviewer Concern (W1+Q1)

Natural G3/G4 evidence remains incomplete, including Scout's G4 follow-up, Luna's uncertain utility, and Terra's provenance. Can the authors provide a complete, auditable natural lineage with qualified nonzero G3 or evaluable G4 measurements under one frozen protocol?

Response

The current supplement contains ten natural lineages: two Scout, five Luna, and three Terra, evaluated under one frozen G0–G4 protocol. Scout UNIFIED_V3_NATURAL_06 provides a complete structured G0–G4 operational trace. The archived states and messages connect Method revisions and legal installations to actual successor inputs, same-SOURCE G0 comparisons, fixed-transfer G2 evaluations, native G3 readouts, and the in-window G4 follow-up. We checked that installed states continue into the next generation and that the preceding Method appears in the successor input. Scout 10 also meets the frozen primary operational criteria. Independent assessments disagree on the substantive content of some edits, so operational completeness is not treated as independently agreed evidence of a better update algorithm.

Scout 06 and 10 meet the saved primary consecutive-handoff G4 qualification rules and each has an evaluable native G4 effect of 0 TES. The other eight lineages are not counted as zero-effect G4 observations. All ten have evaluable G3 contrasts: five Luna and two Scout readouts are 0 TES, while Terra 03, 04, and 09 yield +6.07, +0.48, and +13.60 TES (mean +6.72 TES). Across these ten lineages, the mean is +2.01 TES, with a Student-t 95% interval of [−1.20, 5.23] (9 degrees of freedom). These statistics describe the ten natural lineages evaluated here. The Terra values are downstream contrasts, not isolated causal Method-content gains. This trace demonstrates operational completeness without assuming successful RSI.

The original absence of a complete evaluable natural audit is therefore addressed at the operational level. The small-sample controlled comparisons and the historical raw-record concerns are handled separately below; the new cohort is not presented as a reconstruction of the old tables.

Q3

Reviewer Concern (Q3)

Does the original +21.98 TES METHOD_CLAMP residual persist in a prespecified comparison with more independent packages under fixed deployment and output contracts?

Response

We completed the requested expanded matched validation. We used larger, newly frozen cohorts to extend the four-package Table 6 comparison. For GPT-5.6-Sol, 32 valid SOURCE-level comparisons gave a mean policy-restored residual of +0.47 TES (95% CI [−2.22, 3.16]), compared with a full downstream contrast of +3.52 TES. A separate Qwen 27B matched revalidation yielded +0.256 TES across 28 valid comparisons (95% CI [−3.01, 3.52]). The original four-package +21.98 TES residual was evaluated directly. The expanded comparisons did not confirm a stable positive residual. Deployment and output contracts were fixed within the new cohorts; these experiments are matched revalidations, not exact historical-environment replications.

A different preregistered content-control experiment provides an important complementary result: in the 48-SOURCE M3 confirmation, 46 valid pairs yielded a mean downstream contrast of +27.34 percentage points (95% CI [14.57, 40.10], multiplicity-adjusted p = 0.000122). Policy restoration eliminated that measured difference, consistent with Policy-mediated content responsiveness. This confirms sensitivity to the tested controlled intervention, not persistence of the old policy-restored TES residual. We have kept these estimands separate in the revised interpretation.

AuthorsOpenReview ↗

Response to Reviewer 5ULY part II

W2+Q2

Reviewer Concern (W2+Q2)

The central results, especially Tables 6–8, cannot be independently reconstructed from the submission. Can the authors release exact scoring code, SOURCE/TASK manifests, raw responses, intervention configurations, and one complete natural lineage?

Response

The current review-stage supplement provides the ten-lineage structured artifacts, including the complete Scout 06 operational trace, together with the TES scoring code and offline replay dependencies. All 680 saved TASK scores passed arithmetic checks and scoring-engine re-execution. These are two checks of the same evaluation set, not 1,360 independent evaluations. Offline replay recomputes the saved utility measurements; it does not regenerate model outputs or independently re-adjudicate qualification. The larger Table 6 matched validations and other controlled studies are separate from this ten-lineage supplement. The supplied materials support offline recomputation of the included natural-lineage measurements.

We provide partial reproducibility materials for review and will open-source the remaining retained, redistributable materials upon acceptance. The planned release includes controlled-study code and data, available execution artifacts, configurations, and a claim-to-artifact index. The new Table 6 experiments address the requested expanded matched validation; they are not exact reconstructions of historical deployment conditions. We do not claim that historical Tables 6–8 and Table 24 have been fully reconstructed. The current supplement is available through the anonymous link below.

Reviewer package: https://anonymous.4open.science/r/recurlens-supplement-2750/.

Reviewer_9uNgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RecurLens, an evaluation framework for auditing recursive self-improvement in fixed-weight agents. Rather than treating an improvement in final task performance as evidence of successful recursive improvement, RecurLens decomposes the process into five stages: local update effects (G0), realized method handoff to a successor (G1), descendant task performance (G2), downstream utility of the inherited method in a matched recipient (G3), and persistence across repeated handoffs (G4).

The framework is evaluated on an executable cache-fault testing task using Frozen and Recursive lineages, together with controlled reference-method experiments. The experiments illustrate several important failure modes of endpoint-only evaluation. For example, one Recursive lineage outperforms its Frozen counterpart by 5.38 TES points while remaining 17.09 points below its own initial performance and without changing its improvement method. The authors also show that a large downstream reference-method contrast decreases substantially when the task policy produced by the first update is removed, indicating that apparent improvement-method gains can partly arise through task-policy adaptation.

Overall, I view the paper primarily as a methodological and benchmarking contribution rather than evidence that the evaluated systems exhibit successful recursive self-improvement. The framework provides a useful vocabulary and instrumentation for determining exactly where an improvement chain succeeds or fails, although the current empirical validation leaves the most distinctive G3/G4 portions of the framework incompletely demonstrated.

Critical strengths

The paper makes a consequential distinction between task improvement and improvement of the mechanism that produces subsequent updates. RecurLens traces proposed updates through installation, successor consumption, downstream task behavior, and later update utility rather than inferring recursive improvement from an endpoint score. This addresses a genuine attribution problem in evaluating self-modifying agents.

Critical weaknesses

The most distinctive parts of the framework, G3 and especially G4, are not convincingly validated end to end on natural recursive trajectories. The manuscript states that the available materials contain no source-complete successful natural lineage. Scout reaches the two-handoff G4 entry condition but has no evaluable later utility, Luna's later-utility bound crosses zero, and Terra's positive historical contrast cannot be treated as independently verified because of provenance issues. Thus, the paper provides a strong argument for why these measurements should exist, but substantially weaker evidence that they can yet provide reliable measurements in realistic natural runs.

Questions for authors
  1. What is the strongest natural lineage in the current experiments for demonstrating RecurLens end to end? Could the authors provide a complete trace through G0-G3, and G4 if available, including the proposed and installed method texts, successor inputs, qualification decisions, and downstream evaluations? Importantly, this need not be a successful recursive-improvement result. A complete negative or null lineage would still substantially strengthen the validation of the audit.

  2. How should I interpret the empirical validation of G3 independently of task-policy effects given the four-package METHOD_CLAMP result? The residual mean contrast is substantially smaller than the unclamped contrast and is positive in only two of four packages. Would the authors consider the present experiments sufficient evidence that G3 reliably distinguishes method-content improvements from ordinary stochastic variation, or is G3 currently best viewed as a proposed diagnostic whose statistical calibration remains preliminary?

  3. Can the authors make the experimental artifacts required to reproduce at least one complete natural lineage and the principal G3 controls available during the review period? In particular, the exact TES aggregation weights/scorer, task and SOURCE manifests, prompts and schemas, condition configurations, raw responses, installation/intervention records, and provenance information would materially affect my confidence in the empirical conclusions.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9uNg Part I

Summary

We thank the reviewer for highlighting the need for complete natural G3/G4 evidence and careful causal interpretation.The current supplement contains ten natural lineages, including a complete structured Scout 06 G0–G4 operational trace, two complete operational certificates under the primary rules, and two evaluable native G4 follow-ups. We have completed expanded matched validations addressing Table 6; they do not confirm the historical +21.98 TES policy-restored residual. A separate 48-SOURCE M3 study supports controlled downstream sensitivity to Method-content interventions. The new evidence addresses operational validation while preserving the distinction between causal attribution and statistical certainty. We distinguish operational eligibility, causal effects, and statistical precision, and separate the current natural-lineage supplement from the additional controlled studies.

W1+Q1

Reviewer Concern (W1+Q1)

Natural G3/G4 validation remains incomplete across Scout, Luna, and Terra. Can the authors provide one fully traceable natural G0–G4 lineage, including Method installation, successor consumption, qualification, and downstream measurements, even if the result is null?

Response

The current supplement contains ten natural lineages: two Scout, five Luna, and three Terra, evaluated under one frozen G0–G4 protocol. Scout UNIFIED_V3_NATURAL_06 provides a complete structured G0–G4 operational trace. The archived states and messages connect Method revisions and legal installations to actual successor inputs, same-SOURCE G0 comparisons, fixed-transfer G2 evaluations, native G3 readouts, and the in-window G4 follow-up. We checked that installed states continue into the next generation and that the preceding Method appears in the successor input. Scout 10 also meets the frozen primary operational criteria. Independent assessments disagree on the substantive content of some edits, so operational completeness is not treated as independently agreed evidence of a better update algorithm.

Scout 06 and 10 meet the saved primary consecutive-handoff G4 qualification rules and each has an evaluable native G4 effect of 0 TES. The other eight lineages are not counted as zero-effect G4 observations. All ten have evaluable G3 contrasts: five Luna and two Scout readouts are 0 TES, while Terra 03, 04, and 09 yield +6.07, +0.48, and +13.60 TES (mean +6.72 TES). Across these ten lineages, the mean is +2.01 TES, with a Student-t 95% interval of [−1.20, 5.23] (9 degrees of freedom). These statistics describe the ten natural lineages evaluated here. The Terra values are downstream contrasts, not isolated causal Method-content gains. This trace demonstrates operational completeness without assuming successful RSI.

Q2

Reviewer Concern (Q2)

The four-package METHOD_CLAMP result raises uncertainty about G3's ability to distinguish Method-content effects from Task Policy mediation and stochastic variation. How has the measurement been calibrated beyond this limited comparison?

Response

G3 estimates a downstream contrast under a matched recipient, not an automatically isolated direct effect of Method text. OUTER may change the intermediate Task Policy, so Policy restoration tests a narrower contrast and can also remove a genuine mediation pathway. We have now measured the original residual question with more independent units. In a frozen 32-SOURCE matched revalidation, the 27B recipient yielded +0.256 TES over 28 valid restored comparisons (95% CI [−3.01, 3.52]). A separate GPT-5.6-Sol cohort yielded +0.47 TES over 32 valid comparisons (95% CI [−2.22, 3.16]). Neither supports a stable residual comparable to the original +21.98 TES; the new deployment contracts are not claimed to reproduce the unidentified historical configuration exactly. The expanded comparisons do not establish a robust policy-restored residual.

We also added Method/Policy and double-restoration interventions, same-Method regeneration, and length-matched controls. In a separately preregistered 48-SOURCE M3 content test, 46 pairs were valid: 14 positive, 32 zero, none negative. The mean controlled downstream difference was +27.34 percentage points (95% CI [14.57, 40.10], multiplicity-adjusted p = 0.000122). The corresponding Policy-restored differences were zero. Hence the tested content intervention changes downstream behavior, plausibly through generated Policy, while the original direct residual remains unconfirmed. This is calibration of specific controlled contrasts, not a claim that natural self-generated Methods have a statistically established direct benefit.

AuthorsOpenReview ↗

Response to Reviewer 9uNg Part II

Q3

Reviewer Concern (Q3)

Can the authors provide reviewer-accessible raw records, scoring code, configurations, and provenance sufficient to reproduce a complete natural lineage and the principal G3 controls?

Response

The current review-stage supplement provides the ten-lineage structured artifacts, including the complete Scout 06 operational trace, together with the TES scoring code and offline replay dependencies. All 680 saved TASK scores passed arithmetic checks and scoring-engine re-execution. These are two checks of the same evaluation set, not 1,360 independent evaluations. Offline replay recomputes the saved utility measurements; it does not regenerate model outputs or independently re-adjudicate qualification. The larger Table 6 matched validations and other controlled studies are separate from this ten-lineage supplement. The supplied materials support offline recomputation of the included natural-lineage measurements.

We provide partial reproducibility materials for review and will open-source the remaining retained, redistributable materials upon acceptance. The planned release includes controlled-study code and data, available execution artifacts, configurations, and a claim-to-artifact index. The new Table 6 experiments address the requested expanded matched validation; they are not exact reconstructions of historical deployment conditions. We do not claim that historical Tables 6–8 and Table 24 have been fully reconstructed. The current supplement is available through the anonymous link below.

Reviewer package: https://anonymous.4open.science/r/recurlens-supplement-2750/.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.