acceptodds
Under review as a conference paper at ICLR 2027

Feedback Alignment in Self-Improving Agents

Abstract

Self-improving agents often use the same evaluation signal both to select updates and to report progress. When that signal captures only part of the intended objective, a rising score does not necessarily mean better task performance: the agent may instead be getting better at satisfying the evaluator. We study this failure mode in scaffold-level self-improvement, where a fixed language model repeatedly revises the executable program used to solve future tasks. We introduce a controlled protocol that separates visible feedback used for scaffold selection from held-out target evaluation used only for reporting. Across two-voice counterpoint repair and cross-database SQL repair, scaffold evolution can raise the visible score while lowering the independently measured target. Ordinary best-of-five selection also reduces the counterpoint target, but less than the full evolution procedure under the same call cap. Retained scaffolds further encode evaluator-specific prompting, checking, and selection rules. These results indicate that a rising optimization score is insufficient evidence of improvement; in the tested comparisons, feedback alignment predicts the direction of target change.

Primary area:
interpretability and explainable AI
Keywords:
self-improving agents, LLM agents, reward hacking, feedback alignment, scaffold evolution, counterpoint

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zpMROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studied whether agents that improve their own harness become better at solving tasks or merely better at satisfying their evaluator. With model weights fixed, it separates feedback used to select scaffold revisions from target performance measured on held out tasks, comparing aligned evaluators with incomplete proxies in counterpoint music repair and SQL repair. Across generations, proxy feedback increases visible scores while reducing target performance.

Critical strengths

The paper show that evaluator choice can change the direction of harness improvement, which is informative to the current self-improvement studies. Within each domain, the starting harness, model, search procedure, and resource limits remain fixed, while selection uses either aligned feedback or an incomplete proxy. Independent target evaluation reveals substantial degradation under proxy feedback in both music and SQL repair.

The study also has complementary controls and program audits, which makes it more solid. Fixed solver best of 5 selection produces less target degradation than harness evolution under the same inference call cap, and withholding diagnostic reports reproduces the negative direction. Source code audits further document evaluator specific prompting, scoring, and selection procedures in retained scaffolds.

Critical weaknesses

The conceptual novelty is kinda limited. The central finding that optimizing a misaligned proxy can degrade true performance is already well established in reward hacking literature. Extending this observation to harness evolution provides a new setting, but the paper offers limited additional mechanistic insights or solutions.

Also, the experiment scope is narrow. Both counterpoint and SQL repair use synthetic tasks with deliberately incomplete proxy evaluators, making the observed failures somewhat predictable. The small evaluation sets and lack of realistic agent benchmarks further limit the generalizability of the conclusions.

And attribution to scaffold evolution does not seem to be sufficiently established, as the comparison with best of 5 selection lacks a direct statistical test and does not control for outer loop computation.

Questions for authors

See weaknesses

Reviewer_CxDgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how feedback affects iterative improvement of an agent’s executable solver scaffold while keeping the language model and outer update procedure fixed. It compares feedback based on the intended target with incomplete proxies in two settings: counterpoint repair and SQL repair. The main contributions are a protocol separating search feedback from held-out target evaluation, a comparison with fixed-solver best-of-five selection, and an audit of persistent changes to the generated programs.

Across the reported proxy conditions, visible feedback improves while final target performance falls below the single-shot baseline. Counterpoint experiments also show deterioration over generations, including a rerun that removes scores and error reports from the proposal prompt. SQL experiments provide a second domain with the same direction of effect. The fixed-solver comparison suggests that the full evolution procedure can produce a larger target loss than instance-level selection under the same inference call cap. The program audit documents retained changes to prompting, checking, and candidate selection. Together, these results provide a controlled demonstration that improving an evaluator score need not improve the intended task.

Critical strengths
  1. The paper separates progress on the evaluator from progress on a clearly defined task target. Counterpoint rule satisfaction and SQL execution provide concrete evaluation criteria, and held-out instances are excluded from scaffold selection. In the primary counterpoint proxy condition, visible feedback increases by 0.0980.098, while the final target effect relative to B0B_0 is −0.292-0.292, with an unadjusted confidence interval of [−0.346,−0.237][-0.346,-0.237] and a Holm-adjusted pp-value of 0.0070.007. This is substantial evidence of a practically important failure within the constructed setting. The distinction between formal rule satisfaction and broader musical quality also makes the scope of the target reasonably clear.

  2. Several complementary checks strengthen the empirical observation. Removing scores and error reports from the counterpoint proposal prompt still produces a negative target effect of −0.171-0.171, with a Holm-adjusted pp-value of 0.0370.037. The SQL style-proxy condition has negative target effects in all three seeds. The fixed-solver control uses five calls per instance, whereas the evolved counterpoint proxy scaffold averages 4.474.47, making the inference budget comparison informative. Finally, the source audit shows that selected programs retain changes to prompting, checking, and selection. These checks support the claim that feedback can shape persistent procedures that perform poorly under an independent target evaluation.

Critical weaknesses
  1. The reported evaluator conditions leave the significance and scope of the contribution uncertain. The main counterpoint comparison places exact rule feedback against a symbolic proxy that ranks clean artifacts above violated ones in only 57.5%57.5\% of calibration pairs. This establishes a valid failure case, but the proxy already has poor target ordering before scaffold evolution. The more informative question is how evolution behaves when feedback initially tracks the target reasonably well. The learned ridge proxy has a reported Spearman correlation of 0.8240.824, and its local-search failure is interesting, but its scaffold-evolution results are not reported. Appendix A describes a family of 15 comparisons, including noisy graded feedback, the ridge proxy, a frozen language-model judge, and prompt-only conditions; only four counterpoint evolution rows are presented. These missing outcomes matter because they could substantially change the apparent generality of the result and clarify the contribution beyond existing work on feedback-driven reward hacking, particularly Pan et al. (2024).

  2. The evidence does not yet establish how much of the additional harm should be attributed to repeated scaffold evolution. Section 3.4 describes both arms as ranking the same candidate programs, but it is unclear how candidates remain shared after the retained scaffolds diverge. Feedback can also enter program execution and proposal generation, so the exact intervention needs to be specified before interpreting it as a change only to candidate ranking. The fixed-solver comparison is useful, but the reported ratio Γ=4.17\Gamma=4.17 is descriptive: it has no uncertainty estimate and does not isolate rewriting from the additional search used to find programs. Moreover, the first generation accounts for 86%86\% of the counterpoint trajectory loss and 111%111\% of the SQL loss, followed by partial recovery in SQL. These observations support a harmful full procedure, but provide weaker evidence that successive rounds progressively amplify the failure. The authors acknowledge several of these limits; nevertheless, the Section 5.2 heading and corresponding abstract claim should reflect the descriptive nature of the comparison.

Questions for authors
  1. Clarification of the intervention. Section 3.4 says that the two arms rank the same candidate programs. Are identical candidate pools generated once and shared across arms, or does each arm generate candidates from its own retained scaffold after the first generation? Please specify how proposal histories, scorer access, and diagnostic information differ between conditions. Is the incumbent scaffold eligible for retention, and what happens when all proposed revisions fail validation? These details determine whether the experiment isolates candidate ranking or evaluates a broader change in feedback configuration.

  2. Clarification of the unreported conditions. Could you report the outcomes for all 15 comparisons in the stated correction family, including the prompt-only, noisy graded-feedback, ridge-proxy, and frozen-judge conditions? Please include seed counts, target effects, and unadjusted and adjusted significance results, and identify the relevant scorer definitions and weights. The ridge condition is especially consequential because its reported calibration correlation is substantially stronger than the symbolic proxy’s ordering accuracy. Its behavior during scaffold evolution would help establish whether the main result extends to initially informative feedback.

  3. Clarification of baselines and comparative uncertainty. ΔVevol\Delta V_{\mathrm{evol}} is measured relative to the starting scaffold, whereas EtargetE_{\mathrm{target}} is measured relative to B0B_0. Please report the corresponding start-to-final target change and the direct aligned-versus-proxy and evolved-versus-fixed comparisons, with uncertainty that reflects the experimental sampling structure. For primary counterpoint, the reported effects imply an evolved-minus-fixed point estimate of −0.222-0.222; how uncertain is this contrast across runs and instances? If a direct comparison is unavailable, please qualify the claim that scaffold evolution exceeds fixed-solver selection as a statement about the observed point estimates.

  4. Clarification of the contribution relative to closely related work. Please cite and discuss Alexander Pan, Erik Jones, Meena Jagadeesan, and Jacob Steinhardt, “Feedback Loops With Language Models Drive In-Context Reward Hacking” (ICML 2024). That paper studies reward hacking through both output refinement and policy refinement, making it closely relevant to the present motivation. Please also discuss the objective-hacking example in the already cited Darwin Gödel Machine paper. Two relevant coding benchmarks are Bingchen Zhao, Dhruv Srikanth, Yuxiang Wu, and Zhengyao Jiang, “SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents” (2026), and Jonathan Gabor, Jayson Lynch, and Jonathan Rosenfeld, “EvilGenie: A Reward Hacking Benchmark” (2025). What do the present feedback intervention, fixed-solver comparison, and persistent-program analysis establish beyond these results? The important issue is the distinction in contribution, rather than citation coverage alone.

  5. Clarification of reproducibility and the audit population. Section 5.1 and Table 5 describe the primary split as preregistered. Could you provide the registration record, its timing, and the decisions it fixed? Please map F1 and F4 explicitly to the reported evaluation variants, introduce B1, B3, and B4 in the main text, and specify the exact model version and execution dates behind the name deepseek-flash. Finally, what is the denominator for the three proxy scaffolds that name exposed proxy components in Table 7? This is needed to interpret how frequently that behavior appeared in the audited proxy condition.

Reviewer_WK44OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies evaluator misspecification in scaffold-level self-improvement: a fixed language model revises an executable solver, while a visible evaluator selects revisions and a separately measured target assesses performance. Experiments use counterpoint repair and synthetic SQL repair, contrasting target-aligned feedback with incomplete feature-based proxies. Proxy-guided evolution increases proxy scores but produces worse target performance relative to a single-shot baseline. A counterpoint best-of-five control exhibits less degradation, and source-code audits document evaluator-specific changes in retained programs.

Critical strengths
  1. Useful separation of optimization and evaluation. Holding out target instances and preventing their use in scaffold selection is an appropriate experimental design. The distinction between data independence and scoring-function independence is explicitly acknowledged.

  2. Substantial observed degradation. The primary counterpoint proxy condition reports a target effect of −0.292, with a confidence interval well below zero and a Holm-adjusted p=0.007p=0.007. The withheld-diagnostics rerun and SQL experiment provide additional evidence that the phenomenon is not confined to one configuration.

Critical weaknesses

1. The proxies make the central result relatively unsurprising

The symbolic music proxy ranks clean artifacts above violated counterparts only 57.5% of the time, with an average separation of 0.004. Before any scaffold evolution, a simple local optimizer already increases this proxy while substantially degrading the target. The SQL proxy rewards formatting and style without checking execution.

These experiments establish that persistent programs can optimize clearly inadequate evaluators. They provide much weaker evidence about failure under evaluators that initially constitute credible measures of task success. The learned ridge proxy, with reported Spearman correlation 0.824, would be a more informative evolution condition, but its evolution results are not presented despite being included in the correction family.

A stronger contribution would characterize degradation as evaluator quality or optimization pressure varies, including reasonably predictive learned evaluators and realistic judge-based feedback.

2. The primary endpoint does not directly measure target deterioration during evolution

Equation 5 compares visible-score change against the starting scaffold but target performance against a different system, B0B_0. The authors explicitly acknowledge this distinction, but it still weakens the central directional-agreement analysis.

A positive visible change and a negative target effect relative to B0B_0 do not, by themselves, establish that evolution reduced target performance relative to its own starting scaffold. Figure 2 supplies descriptive evidence for counterpoint, but the paper should report matched start-to-final target changes, absolute starting/final scores, and uncertainty for every primary condition. The common-baseline endpoint can remain as a separate practical comparison.

3. The controls do not isolate the scaffold-specific contribution

The fixed best-of-five control is useful, but it differs from evolution in prompts, procedures, and accumulated optimization effort. Consequently, the 4.17 magnitude ratio is not strong evidence that persistent executable rewriting introduces a distinct amplification mechanism.

This matters especially because the first generation accounts for 86% of the counterpoint loss and 111% of the SQL loss. Much of the phenomenon may therefore arise from an initial evaluator-conditioned rewrite rather than cumulative adaptation.

Important missing comparisons include:

  • A single rewrite without subsequent evolution.
  • Prompt-only optimization under comparable search effort.
  • Fixed, evaluator-aware candidate generation rather than the initial solver alone.
  • Ablations of scorer access, internal selection, and fallback behavior.
  • An incumbent-preserving retention rule. Equations 1–2 do not explicitly include the current scaffold among eligible candidates, potentially forcing replacement even when revisions are worse.

Prompt-only conditions apparently exist, but their results are omitted. These are particularly relevant baselines for the paper’s claimed scaffold-level contribution.

4. Statistical uncertainty does not adequately capture evolution-run variability

Averaging across seeds before instance-level testing avoids counting seed-instance pairs as independent observations. However, the resulting tests and instance bootstrap primarily characterize uncertainty across test instances after averaging the observed runs; they do not fully describe uncertainty over new stochastic evolution runs.

This distinction is consequential for SQL, where proxy effects range from −0.260 to −0.015 across only three seeds. Music has just 16 held-out instances, and the withheld rerun uses two seeds. Please report all run-level effects and hierarchical uncertainty over both runs and instances.

SQL template-level uncertainty is appropriately disclosed, but splitting across templates, schemas, and data seeds needs clearer specification. “Cross-database” should not imply held-out structural generalization unless the partition supports it.

Questions for authors
  1. The symbolic proxy prefers the clean artifact on only 57.5% of calibration pairs, with a mean separation of 0.004, and a model-free local search already raises it while lowering rule satisfaction from 0.856 to 0.605. Appendix A places the ridge proxy (Spearman ρ=0.824\rho=0.824) and a frozen language-model judge in the family of 15 confirmatory tests, but Table 2 reports only graded rules and the symbolic proxy. What are ΔVevol\Delta V_{\mathrm{evol}}, EtargetE_{\mathrm{target}}, and the seed-level effects for the ridge-proxy and judge-based evolution runs? If those runs were not executed, why are they included in the Holm family?

  2. Equation 5 measures visible ascent from hstarth_{\mathrm{start}} to hfinalh_{\mathrm{final}}, but the target effect from hfinalh_{\mathrm{final}} to the single-shot baseline B0B_0. For every primary and withheld condition, what are the absolute DtestD_{\mathrm{test}} target scores of hstarth_{\mathrm{start}}, hfinalh_{\mathrm{final}}, and B0B_0, and the paired start-to-final target change with a confidence interval? In the proxy conditions, does the final scaffold fall below its own starting scaffold?

  3. The first generation accounts for 86% of the counterpoint target loss and 111% of the SQL loss, while Equations 1–2 select only among newly proposed revisions. What is the held-out target effect of one evaluator-conditioned rewrite with no further generations, of the three prespecified prompt-only conditions, and of a rule that retains hgh_g unless a revision improves VV? After these comparisons, how much of the reported gap versus best-of-five remains?

  4. The three SQL style-proxy seed effects are −0.260-0.260, −0.083-0.083, and −0.015-0.015, while the reported tests average seeds before testing instances. Please report every run-level target effect and a hierarchical interval over both evolution runs and test instances. For SQL, which of templates, schemas, and data seeds are disjoint between search and held-out data, and which comparison supports generalization beyond the schemas used during selection?

  5. Section 3.4 says the proxy arm ranks “the same candidate programs,” but each arm’s retained scaffold determines the next proposals. Are the candidate pools shared, or only the proposal procedure? One of the three style-feedback SQL scaffolds also calls an execution report, although the proxy arm is described as unable to call the target evaluator. What does that report expose, and did any proxy scaffold observe execution outcomes during search? Please also include the proxy and rule weights, style-feature definitions, proposal prompts, sampling settings, the exact model version, and the outcomes of all 15 prespecified comparisons.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.