- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies harness-level recursive self-improvement with a frozen foundation model. The authors argue that the value of a persistent self-edit may not become apparent immediately. For example, a newly introduced verifier may incur additional short-term cost but prevent failures on later tasks, while its diagnostics may also support a subsequent modification to the planner or router. Therefore, recording only the immediate probe outcome after an edit may fail to capture its long-term value.
To address this issue, the paper proposes SelfLineage. The method records actual persistent edits and subsequent outcomes as immutable nodes, and retrospectively adds two types of relations when supported by sufficient provenance: whether a later task actually reused an earlier edit, and whether a later self-edit depended on diagnostics, interfaces, or representations produced by an earlier edit. The shared admission gate accepts edits whose immediate probe gain satisfies , thereby allowing short-term neutral or slightly costly infrastructure changes to enter the current system. The central comparison is between Raw-History and SelfLineage: at the same decision checkpoint, both receive exactly the same edit, outcome, and provenance nodes, and differ only in whether SelfLineage explicitly presents verified lineage relations. The authors also propose Shuffled-Link, truthful Temporal, fixed/adaptive continuations, and Live/Frozen-Link controls, and use Audited Descendant Advantage (ADA) to evaluate the downstream value of edits selected under different history representations.
I believe the paper addresses an important problem, and the core matched-fact comparison is valuable. However, the current manuscript does not adequately clarify the status of its experimental results or several key implementation details. Some tables present numerical values as formal results, while other values are explicitly described as working ranges or examples, and the conclusion refers to the overall experiment as a proposed study. It is therefore difficult to determine which effectiveness claims have been supported by completed experiments. In addition, although the paper provides conceptual definitions of Contains, Verified, rollback, and supersession, the supplied materials do not explain the concrete implementation of the lineage verifier or clearly specify how to handle an accepted edit that produces no verifiable long-term benefit, or a component that is modified again before the consequences of the previous edit become observable. These omissions limit reproducibility and make the long-term behavior of the method difficult to assess. Overall, I lean toward rejection, while recognizing that the research question and the matched-fact experimental design are worth further development.
Critical strengths
The research question is practically relevant. Modifications to routers, verifiers, recovery mechanisms, context managers, and skill libraries can persistently affect future tasks, and their value may only emerge after repeated reuse. Enabling a self-editing system to use such delayed information is more difficult, and potentially more meaningful, than one-shot prompt optimization.
The authors explicitly acknowledge several important limitations. The paper states that lineage edges represent observed reuse or dependence rather than full causal credit; that degradation under Shuffled-Link may simply indicate that misinformation is harmful; and that independently evolving online trajectories may diverge after selecting different actions. These qualifications make the conceptual argument more careful.
Critical weaknesses
- The manuscript does not clearly distinguish measured results, study scope, and illustrative numbers, making the central effectiveness claims difficult to verify
This is my primary concern. Table 1 reports, in the language of formal results, improvements of SelfLineage over Raw-History on Tools, Skills, and Agents—for example, success-rate gains of 8, 6, and 9 percentage points, respectively. Table 3 likewise reports ADA values for the full system and several ablations. However, the manuscript does not provide the number of independent runs, random seeds, variances, confidence intervals, or significance analyses for these values. Costs for the new arms required by Table 1 are also explicitly stated as unavailable, so the AUC values and method rankings cannot support conclusions about efficiency.
At the same time, Table 2 explicitly states that the bracketed numbers are working ranges rather than confidence intervals, and explains that a zero immediate difference together with a positive delayed difference merely illustrates the research hypothesis rather than an observed rank reversal. The numbers in Figure 2 and Table 4 are also explicitly described as examples, while the conclusion refers to the overall experiment as a proposed study. Consequently, it is unclear whether Tables 1 and 3 contain final experimental averages, pilot results, expected values, or illustrative numbers similar to those elsewhere in the paper.
This ambiguity directly affects the scientific conclusions. If Tables 1 and 3 report measured results, the authors should clearly state their status and provide the run scale, uncertainty estimates, missingness information, and supporting artifacts. If the numbers are planned or illustrative, they should not be presented as evidence of the method’s effectiveness. In its current form, the manuscript reads more like a carefully designed experimental proposal than a complete empirical study.
In addition, the Reproducibility Statement says that the appendices provide details about EvoHarnessBench isolation, checkpoint audits, statistical analysis, and cost accounting, but I could not find these appendices in the supplied 12-page PDF.
- Verified lineage relations are central to the method, but their construction and the associated version lifecycle lack a reproducible specification
The effectiveness of SelfLineage depends on whether Contains and Verified can reliably identify the two relation types. For runtime reuse, the paper requires that downstream computation actually use the content, interface, or artifact produced by an earlier edit, rather than merely loading or reading it. For evolution dependency, the paper requires traceable use of a diagnostic, tool, interface, or representation produced by an earlier edit when constructing a later self-edit. These requirements are conceptually reasonable, but the main paper does not provide sufficient operational detail.
For example, the paper does not specify which events are recorded for different types of harness components, how it determines that a read artifact actually affected downstream computation, or how copying, rewriting, indirect invocation, and multiple coexisting versions are handled. It also does not state whether evolution dependency is determined by a deterministic trace, another model, or human auditing. The precision, recall, audit consistency, and computational cost of relation verification are not reported.
This omission is especially important when the same component is modified repeatedly. The paper states only that lifecycle events distinguish deployment, rollback, and supersession, and that runtime-reuse relations must satisfy version containment. It does not define how supersession is detected, whether an earlier edit remains observable after being overwritten by a later edit before its benefits are observed, whether superseded nodes remain eligible for retrieval, or whether a later outcome should be linked to the earlier edit, the later edit, or multiple edits. Although the paper states that one outcome may be referenced by multiple edits, it does not provide concrete edge-construction or conflict-resolution rules for multi-version, multi-ancestor cases. The core process therefore cannot be reproduced from the current manuscript alone.
- There is a gap between permissive admission and subsequent lifecycle management, and the paper does not define an exit mechanism for edits with no observed long-term benefit
The paper uses a shared gate requiring , deliberately allowing edits with zero or slightly negative immediate benefit to be persistently deployed. This design is necessary for observing delayed benefits and is therefore central to the research question. However, the gate can only exclude edits that are clearly harmful in the short term; it cannot determine whether an edit has long-term value.
The manuscript does not provide a corresponding downstream management rule. It does not state whether an accepted edit should be rolled back, retired, downweighted, or excluded from retrieval after some number of opportunities without verified reuse or dependency. Nor does it impose a constraint on cumulative performance degradation or system complexity. The lineage window, with as a sensitivity analysis, is only an eligibility window for constructing lineage relations; it is not an automatic exit rule, and the absence of an edge within this window is not recorded as zero benefit. The authors also explicitly note that deduplication does not bound history growth.
Questions for authors
Please clarify whether Tables 1 and 3 report completed empirical measurements. Some values in Table 2, Figure 2, and Table 4 are explicitly labeled as working ranges or examples, while the conclusion refers to the overall experiment as a proposed study, making the status of the reported results difficult to determine. If Tables 1 and 3 contain final results, please provide the number of independent runs, random seeds, uncertainty estimates, and supporting experimental artifacts. If they contain illustrative or preliminary results, please label them explicitly in the tables.
Where are the appendices and registration materials referenced in the Reproducibility Statement? Please provide the dataset/stream revision, model backend, tokenizer, full prompts, retriever weights, gate configuration, checkpoint-sampling procedure, and statistical code.
Please provide executable definitions of Contains and Verified. Beyond the abstract node fields in Equation (4), what concrete trace schemas and decision procedures are used for runtime reuse and evolution dependency? Please describe the blinded trace-audit procedure, link precision, missing coverage, and verification cost. How does the system determine version containment under copying, rewriting, indirect invocation, multiple coexisting versions, and supersession? If the same component is modified again before the downstream consequences of an earlier edit appear, how are subsequent observations linked across the different versions?
For an edit that passes the epsilon gate but subsequently produces no verified long-term benefit, is there a rule for rollback, retirement, downweighting, or exclusion from retrieval? If not, how are growth in deployed components, version chains, and immutable history controlled? Please report the long-term survival, supersession, rollback, and no-link rates of accepted edits.