acceptodds
Under review as a conference paper at ICLR 2027

A Risk-Bounded Gate for Self-Improving Agents: Selecting Modifications as a Mean–Variance Knapsack

Abstract

Self-improving agent systems accumulate modifications—memory modules, reflection steps, judges, caches, extra restarts—each promising a gain on a benchmark. Recent re-evaluations show that stacking them often increases run-to-run variance and that reported gains can reverse under a shuffled task order. We treat modification selection as a mean–variance knapsack: given measured mean effects, pairwise interactions, a covariance of effects across runs and a phase-resolved cost vector, choose the subset that maximises expected gain subject to a cost budget and a variance ceiling. Mean-only selection is the unconstrained corner of this problem and is blind to correlated noise. We give an exact solver for pools of up to sixteen modifications, a variance-penalised greedy and a diversity-seeded local search; on synthetic instances with correlated noise the last recovers the exact optimum on 10 of 10 instances (greedy 4, genetic algorithm 6), but pays off over enumeration only from M=16. We lock a measurement protocol with held-out validation on a disjoint task stream: a two-agent knapsack testbed with a computable optimum, twelve pre-registered prompt add-ons, and a frozen LLM packer. The live packer run is pre-registered and pending; every number reported here comes from running the same fitting and selection pipeline end-to-end on a synthetic ground truth built from our own model. There, mean-only selection inflates true held-out variance 3–20×, the inflation grows with the correlation of add-on noise, and five held-out runs suffice to detect it; but the pre-registered ceiling at the base noise level keeps only a quarter of the gain, a kill we report. An exploratory, not pre-registered, sweep finds that a ceiling of 3σ̂₀ recovers median full gain at ρ=0.8 only, at about half the mean-only variance. We also show why: with K runs the estimated covariance has rank K−1, so a tight ceiling can be met along directions the data never saw, and we pre-specify the shrinkage diagnostic that exposes this. The frozen harness, pool, prompts, analysis script and protocol are released so that the live run can be performed without changing anything.

Primary area:
foundation or frontier models, including LLMs
Keywords:
self-improving agents, LLM-based agents, agent architecture search, mean-variance knapsack, evaluation variance, held-out validation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_yy4QOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper frames the problem of self-improving agents as selecting a subset of candidate modifications under a cost budget and a variance constraint, formulated as a quadratic mean–variance knapsack problem. Because all live LLM-agent experiments failed or were excluded after a post-lock code audit, the submission reads like an automated pre-registration audit report and is presented by the authors solely as a 'synthetic failure analysis of measurement assumptions' on a 12-variable Gaussian generator.

Critical strengths

I could not found any strenghts apart the voluntiees of the author to public a negative result not hiding neither creating false results on their research.

Critical weaknesses

The paper is essentially unreadable. I tried to follow it, but starting from the introduction the motivation itself is not clear, stated minimally, because the results and content itself of the paper as not validate motivation that could explain them. A part from the poor formalization of the problem, i found not acceptable the structure of the experimental part, as a list of ai generated comments, that implicitly pointed out to files, present in the supplementary material, that underlines some auto-audit to the methodology that actually failed.

Questions for authors

I have no question, I invite the author to complete their work before submitting again.

Reviewer_eCWNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the selection of a fixed pool of agent modifications through a mean–variance knapsack with pairwise mean effects and a cost budget. Its principal contribution is a synthetic failure analysis of the proposed measurement and selection procedure. It distinguishes the covariance of independently measured singleton contrasts from the covariance of modification effects within a combined execution, and shows why feasibility under the fitted surrogate does not establish control of total outcome variance.

The experiments comprise a fixed-coefficient solver comparison on ten synthetic instances and a noisy estimation-and-selection study on fifteen Gaussian instances. In the latter, the primary gate retains a median of 26% of the mean-only gain, and none of its selected portfolios meets the joint variance and gain-retention target. Live-agent and ALFWorld evaluations remain unexecuted protocols.

Critical strengths

A useful distinction between the statistical target and the optimisation constraint. Proposition 3.1 clearly explains the covariance induced by subtracting a shared base measurement from independently executed singleton measurements. Together with the base–effect covariance term in Eq. (2), this identifies an important reason why a fitted quadratic constraint may fail to represent total outcome variability. The distinction between measurement validity, solver feasibility, and held-out reliability is valuable.

Transparent reporting of negative findings and evidence boundaries. The manuscript reports the failed joint criterion, separates solver recovery from computational work, and documents the post-lock implementation corrections. It also clearly identifies the ceiling sweep as exploratory and the live-agent protocols as unevaluated. These choices make the scope of the reported evidence substantially easier to assess.

Critical weaknesses
  1. ∑k=1vk+m2v0>v0.\sum_{k=1}v_k+m^2v_0>v_0. At the primary population ceiling, σmax⁡2=v0\sigma_{\max}^2=v_0, every nonempty portfolio is therefore infeasible. Thus, even with arbitrarily accurate estimation of its own covariance target, this gate must select the empty portfolio. The reported nonempty selections arise through finite-sample deviations from that target.

Separately, the paper does not establish whether the joint true-VI/retention target is achievable by any portfolio on these instances. This is acknowledged in Section 6 and Figure 6, but remains consequential: without an oracle using the true total variance, the experiment cannot quantify how much useful performance is lost because of the audited measurement rule. Given twelve modifications, analytic ground-truth quantities, and exact enumeration, this comparison is feasible within the existing benchmark and would materially strengthen the failure analysis.

  1. The broader significance of the demonstrated failure remains limited. The optimisation ingredients and covariance-conditioning facts are explicitly classical. The main experiment studies a gate whose covariance target is already analytically misaligned under the chosen independent-execution design. The manuscript does not establish that this particular measurement rule underlies existing agent-selection pipelines, while the total-variance alternative in Appendix F remains unevaluated. Consequently, the current evidence offers a useful caution about this concrete construction, but limited additional understanding of when reliable modification selection is possible or how this failure affects established agent practice.

These concerns lead me to weak rejection. A more informative decomposition of the existing synthetic results could materially improve my assessment.

Questions for authors

Clarification of the population behaviour: Do you agree that Proposition 3.1, together with the primary ceiling σmax⁡2=v0\sigma_{\max}^2=v_0, makes every nonempty portfolio infeasible in the population limit? If so, how should the finite-sample selections and the reported 26% median retention be interpreted relative to this necessarily empty population gate?

Analysis of the existing benchmark: For each of the fifteen instances, what is the maximum attainable true gain under the same cost budget and the constraint VItrue≤1\mathrm{VI}_{\mathrm{true}}\leq1? Relative to the same mean-only comparator used in Table 4, how many instances permit at least 90% retention? This oracle comparison would distinguish an unattainable operating target from avoidable loss caused by the measurement procedure.

Scope clarification: Is the singleton-contrast covariance rule analysed here used by an existing agent-selection or composition procedure, or was it introduced for this study? Clarifying its connection to existing practice would help establish the significance of the failure analysis.

Furthermore, the abstract in the manuscript differs from the one on OpenReview.

Reviewer_wXY3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigate the problem of selecting modifications for self-improving agents by formulating it as a mean-variance knapsack problem. The paper proposes a gate that selects from a pool of candidate modification under a cost budget and a variance constraint to manage the risk of accumulating noise. The core contribution is a sysnthetic failure analysis demonstrating that subtracting a shared base score from independently executed singleton scores does not identify the correlated modification effects within a joint execution. The paper relies entirely on synthetic data generated from a Gaussian model and explicitly states that no valid LLM agent results are reported.

Critical strengths
  • The formulation of the agent modification selection is mathematically interesting.
Critical weaknesses
  • As noted by the paper's abstract, there's a complete lack of real world validation. all experiments are done with synthetic data. Making this paper more like an mathematical exercise rather than a research paper.
  • Even the proposed method seems to fail on its own proposed synthetic data, the propsoed risk bounded gate select a portfolio that failed the joint target of no variance inflation.
  • The data generator is also overly simplified.
Questions for authors
  • Why not get some real world empirical evidences for this work?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.