- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies support inflation in trajectory-driven skill evolution, where trajectories derived from one upstream observation are counted as independent evidence for a skill update. LCA-RLOO uses a recorded lineage graph to down-weight trajectories with shared roots. It requires support to survive the removal of each root family, and it gates admission on independent canary tasks. On a synthetic benchmark with Qwen2.5-7B-Instruct, it lowers false skill admissions from 31.8% to 12.9% relative to raw voting, while true admission and clean utility stay close to the raw-voting baseline.
Critical strengths
The paper identifies a meaningful failure mode in self-evolving agents, where derived copies of one observation inflate support for a skill update, and proposes an auditable admission procedure. The dose-response experiments and case studies demonstrate the problem clearly.
Critical weaknesses
The evaluation relies on a synthetic benchmark in which shared ancestry is non-transferable by construction, so the results largely confirm the method's own assumption. The comparisons omit the closest prior work, and most baselines are simple author-defined rules. It therefore remains unclear whether the method offers a practical advantage in realistic settings.
Questions for authors
- Is there evidence, outside the benchmark's construction, that shared ancestry predicts harmful updates? For example, results on a real skill-evolution pipeline under an existing poisoning attack.
- How does the method compare with Bara (2026), copy-detection truth discovery, and the baselines listed in the protocol but not reported?
- Which thresholds produced the main results, given that the appendix text and the sensitivity table report different selected values?
2 comments
Compared with the three adaptive system-level baselines, LCA-RLOO has lower FSAR and ASR under the matched protocol. Its TSAR is $ 90.1% $, compared with $ 91.0% $ for native Trace2Skill, $ 90.3% $ for compute-matched self-consistency, and $ 72.4% $ for write-rate-matched random admission. Its clean utility is $ 77.3% $, compared with $ 78.4% $, $ 78.0% $, and $ 74.8% $, respectively. LCA-RLOO has the highest OOD utility at $ 73.8% $.
The no-evolution control is non-adaptive and has a TSAR of $ 0.0% $. We will clarify this distinction, define the asterisk in the revised caption, include the complete results in the manuscript, and limit the corresponding claims to the matched protocol evaluated here.
Q3. Thresholds used for the main results
Appendix B.3 and Table 8 report inconsistent configurations. This is a reporting error. The main results were produced using
$ (\kappa,\theta,\delta)=\mathbf{(3.0,0.12,0.01)}. $
The value $ (4,0.20,0.02) $ in Appendix B.3 is incorrect. The reported development-search grid is also incorrect because it omits the selected value $ \theta=0.12 $.
We will recover the executed grid and selected configuration from the saved experiment configuration and logs, report the configuration used for each table, and include the frozen configuration in the released artifact.
Response to Reviewer kNyV
We thank the reviewer for the careful and constructive feedback.
Q1. Evidence outside the benchmark construction
LineageSkillBench evaluates whether multiple derivations from one observational origin should receive the same evidential weight as agreement across independent origins when semantics, candidate content, trajectory count, labels, and compute are controlled. The controlled benchmark does not establish that shared ancestry universally predicts harmful updates outside this setting.
Held-out tasks and canary executions provide separate behavioral measurements, although the benchmark remains synthetic and structurally controlled. We will describe this evaluation as mechanism validation and stress testing and limit the conclusion to the evaluated settings.
Real-pipeline validation on Trace2Skill and SpreadsheetBench-Verified. We evaluated LCA-RLOO in a real-pipeline stress test using Trace2Skill on SpreadsheetBench-Verified. Trace2Skill evolves reusable spreadsheet skills from execution trajectories, and SpreadsheetBench-Verified provides spreadsheet-manipulation tasks with programmatic evaluation. We created controlled same-root duplication by paraphrasing or relaying selected evolution trajectories while preserving their recorded observational roots. We then applied LCA-RLOO at Trace2Skill’s candidate-skill admission stage.
Across the completed settings, FSAR decreases from approximately 29% to 12–14%, ASR decreases from approximately 36–38% to 16–19%, and clean utility decreases by approximately 1–2 percentage points. We will report this evaluation as a real-pipeline stress test. The revision and artifact will specify the exact number of candidate updates and seeds, duplication rate, SpreadsheetBench-Verified split, matched candidate/verifier protocol, and the exact mean and uncertainty for every metric. The current ranges support transfer of the mechanism to this pipeline but do not provide a comprehensive deployment evaluation.
Q2. Comparison with Bara, copy-detection truth discovery, and omitted baselines
Bara motivates the principle that duplicated agents or derivations should not multiply evidence. Effective sample size and leave-one-cluster-out analysis are also established components. Our contribution operationalizes these ideas for persistent skill admission through:
We will revise the novelty statement to reflect this scope.
A direct numerical comparison with Bara’s conceptual implementation is undefined because that implementation does not specify candidate generation, independent canary evaluation, or a protocol for persistent-skill admission and downstream utility. We will identify Root Deduplication and Root-Capped Mean as operational baselines for Bara’s non-multiplication principle. LCA-RLOO additionally includes a root-family stability test, an independent behavioral gate, and a persistent admit/defer/reject/rollback protocol.
Copy-detection and semantic truth-discovery methods identify duplicated or correlated content through observed similarity. Our controlled intervention evaluates a case in which independently rooted evidence is semantically similar and descendants of one root are heavily rewritten. In Table 2, semantic methods deteriorate as descendant similarity decreases, whereas lineage-aware methods preserve the ancestry distinction. Lineage and semantic similarity provide complementary signals.
We evaluated native Trace2Skill aggregation, compute-matched self-consistency, write-rate-matched random admission, and no evolution under the same candidate-update, verifier, and canary protocol: