- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper try to study endogenous feedback amplification in self-evolving LLM Agents, exploring whether initial skill behavioral biases are recursively reinforced via the “skill-execution-skill update” closed loop and separating this effect from static retention, repeated evidence exposure, and random drift. We propose Paired Counterfactual Skill Dynamics (PCSD), a causal evaluation framework with four paired experimental setups: closed-loop, open-loop, replay, and frozen baselines. Despite its rigorous counterfactual design, the work overclaims on stability, hysteresis and causal mediation, and relies on synthetic canary tasks with limited external validity. Focusing contributions on finite-horizon causal effects would strengthen its argument.
Critical strengths
S1. The research question is well-formulated. The paper targets endogenous feedback amplification in self-evolving Agents and seeks to disentangle static retention of initial skill drift from trajectory-driven progressive reinforcement. The research objective is concrete.
S2. The controlled experimental design is sound. Four arms — Closed-loop, Open-loop, Replay, and Frozen — are deployed to examine the effects of feedback closure, fixed evidence sources, repeated evidence exposure, and static retention, respectively. Compared with a single evolving trajectory, this design facilitates separation of contributions from distinct factors.
S3. Experimental variables are systematically controlled. All arms share identical initial perturbations, task sequences, environment snapshots, and update budgets, mitigating confounding effects from task heterogeneity and computational resource disparities.
Critical weaknesses
W1: (Section 3.3,Lines 294–298; Section 4.4,Lines 428–439) The definition of Local Loop Gain is insufficient to support conclusions on stability. The local empirical loop gain yields . A value indicates local contraction, whereas indicates local amplification along the measured coordinate. Per the definition provided in the paper, , which gives . If both responses are evaluated using the same finite-difference pair, the term in this factorization cancels out directly. This implies that the product alone does not demonstrate independent identification of the behavioral response and evolution response mechanisms. For signed scalar perturbations, the contraction condition should generally be , rather than the simple condition . Since is merely a scalar observation of the high-dimensional skill state, one cannot infer the dynamical stability of the full system from this quantity. The authors' conclusions regarding loop gain, local amplification, and the stability boundary require more rigorous definitions and derivations.
W2: The causal identification scope of the Closed–Open comparison suffers from overclaiming (Page 2, Lines 91–99; Page 6, Lines 305–315). This comparison can estimate the aggregate causal effect of the updated skill on subsequent evidence generation. However, as evolution proceeds, the skill states, trajectory distributions, historical evidence, and their interactions with the updater diverge between the two groups. Consequently, the final Closed–Open contrast cannot further disentangle the independent contributions of individual intermediate mechanisms. The authors’ claim that this design can ``isolate the causal pathway'' may exceed the identification bounds supported by the experiment. A more precise statement would be that the method identifies the aggregate effect of the closed-loop feedback mechanism, rather than enabling independent causal identification of its internal mediating pathways.
W3: In Section 4.3 (Lines 410–414), the paper claims that the independently evaluated skill-state metric mediates 61% (95% CI: 45%–75%) of the branching effect. However, it does not adequately specify the precise definition and measurement of this mediator, the estimation procedure for causal mediation effects, nor the underlying identification assumptions. Given the dynamic dependencies and potential time-varying confounding among skill states, trajectory evidence, and subsequent updates, statistical association alone is insufficient to sustain a definitive causal mediation interpretation. Accordingly, the 61% estimate lacks sufficient methodological grounding; its reproducibility and causal interpretation are questionable, constituting a potential overclaim.
W4: The paper’s Replay comparison only matches task domain, trajectory count, trajectory length, non-Canary outcomes, and marginal Canary frequencies (Page 5, Lines 253–261; Page 16, Lines 816–822), yet it fails to adequately control for the semantic content, joint distribution, temporal ordering, and historical dependence of evidence. Since the LLM Evolver updates skills based on full natural-language trajectories rather than merely Canary occurrence frequencies, the failure of the Replay setup to reproduce closed-loop effects is insufficient to rule out alternative explanations such as differences in evidence structure or temporal sequence. The existing experiments only demonstrate that evidence exposure matched on these limited statistical properties cannot replicate the closed-loop effect, and they do not rigorously establish that the effect arises entirely from the endogenous feedback mechanism.
Questions for authors
My primary concerns are W1–W3.
Concern about changes to the submission record after review
I reviewed the 41-page manuscript “Causal Identification of Endogenous Feedback Amplification in Self-Evolving Agent Skills.” When I checked the forum on November 18, I found that the submission had been withdrawn, the title had been shortened to “Causal Identification of,” and the abstract consisted of just three words: “Large language model.” The PDF now contains a single page stating “The content is current not available.”
All four reviews remain public, including questions about how the reported experimental results were obtained. Readers following those reviews can no longer examine the manuscript through the forum’s PDF link.
After noticing these changes, I then checked all 18 withdrawn submissions I found that list the same two co-authors. In 15 of the 18, the PDF contained only the same one-sentence notice. The other three still linked to full-length manuscripts. I ask the Program Chairs to preserve the version histories and investigate when these files changed, who made the changes, and whether this pattern violated conference policy.