acceptodds
Under review as a conference paper at ICLR 2027

Learning What Not to Learn: Target-Conditioned Update Eligibility for Self-Improving Agents

Abstract

Self-improving large language model (LLM) agents distill interaction feedback into persistent memory to enhance future capabilities, but local task success alone does not warrant persistent reuse. We formulate target-conditioned update eligibility to assess whether observable source evidence supports admitting a feedback span under specified persistence and reuse contracts. Controlled interventions demonstrate contract dependence under construction-defined labels and show admission changes learner states, while downstream utility varies across reuse settings. To examine current evaluation paradigms, we develop a Span-PRM that passes construction-defined development tests but fails confirmation against fresh human judgments. It admits 0 of 300 generated reflections at the frozen threshold under a cross-task rule contract and achieves 37.5% ranking agreement with human majorities on a separate challenge set. A post hoc diagnostic reveals a fixed contract ordering satisfies all 120 human-majority comparisons without processing feedback or evidence, demonstrating success on this ranking target alone cannot establish evidence-sensitive judgment. The frozen model fails all three prespecified tests in a prospectively frozen counterfactual audit against construction-defined references. Our findings highlight vulnerabilities in self-improving agent evaluations, showing local metric optimization alone does not establish evidence-based reasoning and that construction fit, reference-aligned input sensitivity, human agreement, and downstream utility must be validated separately.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model agents, learning from feedback, agent memory, empirical evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wZ2COpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies one problem in self-improving agents: should self-improving agents distill interaction feedback into persistent memory to enhance future capabilities from one interaction? The main idea is whether feedback is worth to learn or not does not depend only on whether the feedback itself is correct, but also on the evidence supporting that feedback, and how agent plan to use it in the future. The contribution of this paper is twofold: (1) it provides the formulation of target-conditioned update eligibility, (2) it systematically evaluates why current evaluation methods and feedback models are unable to provide which experience should be put into persistent memory.

Critical strengths
  1. The problem definition and research perspective are meaningful. The paper proposes target-conditioned update eligibility. It emphasizes that whether feedback should be included in long-term learning should not depend only on content and its evidence, but also on the future target and reuse scope. This perspective is beyond traditional feedback quality filtering. It provides the self-improving agents' long-term memory management and selective update with a more definite framework.
  2. The experimental design is meaningful, and the negative results are valuable. This paper combines controlled experiments, human annotation, and counterfactual testing. Thus, it reveals the problem that the model performs well on the synthetic dataset but cannot generalize to real human judgment. Especially, the fixed contract order can satisfy all the human preference comparisons, revealing shortcuts of current evaluation metrics.
Critical weaknesses
  1. Although Span-PRM achieves near-perfect performance in the construction set, it fails in the human evaluations. In the Natural dataset, it rejects all 300 feedback items (At the frozen threshold, it admits none of the 300 reflections, yielding zero observed ineligible admissions and zero eligible retention). In the Challenge dataset, the pair-ranking accuracy is only 37.5% (Span-PRM obtains flip ranking 0.375 and pair-binary 0.000, failing the prespecified 0.80/0.60 criteria). And it fails to pass the counterfactual test. Also, in the online-memory evaluation, this method is not obviously outperforming Write-All. These results indicate that even this paper proposes a critial problem. However, it cannot prove that the proposed method can resolve this problem well.
  2. This paper highly relies on the human-constructed eligibility labels. And it does not pass the individually validated real feedback judgment. Especially in the Challenge dataset, 62 constructed labels disagree with human judgments. And 60 of them are from not well-defined non-operative ledger contract. Hence, it is hard to differentiate whether the performance gap is due to a lack of evidence understanding ability or the evaluation criteria itself. Hence, it is hard to believe the main conclusion of this paper.
Questions for authors
  1. In the Challenge dataset, 60 of the 62 data points where the labels are from the majority-vote human judgments involved "non-operative ledger contracts", which is a category lacking a clear definition. Could the authors clarify whether the primary evaluation conclusions for Span-PRM still hold if these ambiguous cases are excluded? This would help determine whether the model’s poor performance is from a genuine lack of reasoning capability or from flaws in the evaluation criteria themselves.
  2. The paper reports gains for Span-PRM in controlled experiments; however, in independent human evaluations, the frozen Span-PRM failed to demonstrate an effective capability for feedback filtering. Could the authors clarify the relationship between the models and training configurations used in these two sets of experiments? Furthermore, to what extent do the gains observed in the controlled experiments support the effectiveness of the proposed "target-conditioned admission" method in real-world settings?
  3. The reference labels for ContractFlip and EvidenceFlip are still generated via manually defined rules and have not been independently validated by humans. How do the authors confirm that these rules reliably reflect "evidence-sensitive eligibility"? Given that previous efforts have shown significant differences between constructed labels and human judgments, to what extent should failures in counterfactual tests be interpreted as a deficiency in the model’s reasoning capabilities, rather than as an inconsistency between the constructed rules and the actual criteria used for human judgment?
12 comments
AuthorsOpenReview ↗

Response to Reviewer wZ2C (Part 1/5)

We thank the reviewer for the insightful and valuable comments. We respond to each comment as follows and sincerely hope that our rebuttal could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to your comments and improving our submission.

New evidence on human agreement and downstream utility. Span-PRM-H adapts the original checkpoints using human supervision; it retains 70/78 eligible Challenge items with 4/222 ineligible admissions, improving retention over per-contract calibration by 24.4 percentage points (95% CI [14.3, 35.3]). In the new coverage-matched 7B training study, its selection reaches 42.917% success, exceeding matched random selection by 10.833 percentage points and Success-only by 8.750 percentage points, with both adjusted 97.5% paired intervals above zero (Comments 4 and 6).

Comment 1: “Could the authors clarify whether the primary evaluation conclusions for Span-PRM still hold if these ambiguous cases are excluded? This would help determine whether the model’s poor performance is from a genuine lack of reasoning capability or from flaws in the evaluation criteria themselves.”

Response: We appreciate the reviewer’s request for a ledger-exclusion analysis. It yields a specific diagnosis: the ranking failures are confined to comparisons involving the historical ledger, while an absolute-admission failure remains after those comparisons are removed. Excluding all ledger rows raises ranking from 37.5% to 100% on the 42 retained comparisons, but frozen-threshold retention remains 0/78. The following analysis uses unchanged predictions, human-majority labels, and thresholds:

Challenge subsetRowsGroupsValid pairsRanking, % [95% CI]Eligible retained
Full set30010012037.5 [30.6, 44.3]0/78
Exclude all ledger rows20010042100.0 [100.0, 100.0]0/78
Exclude the 60 disputed ledger rows2401006272.6 [65.2, 81.1]0/78
Exclude the 60 groups containing disputed ledger rows120404057.5 [50.0, 66.7]0/20

Disputed ledger rows are defined by construction–majority disagreement, without model scores. Row exclusion retains the other contracts; group exclusion removes all three. Valid pairs compare eligible and ineligible contracts within a group, with half credit for ties. The 10,000 group-bootstrap replicates keep retained contracts together and condition on the frozen majority reference. We report these as post-hoc intervals; the 100% endpoint describes the retained comparisons and their resamples.

Pair-binary accuracy is zero under every exclusion for the original candidate and both comparison baselines.

The calibration controls quantify the extent to which threshold adjustment improves admission. Per-contract calibration restores retention to 51/78 eligible items with 6/222 ineligible admissions; Span-PRM-H reaches 70/78 with 4/222, a further retention improvement of 24.4 percentage points over the tested recalibration (Comment 6). Excluding all ledger rows from the adapted evaluation gives 70/78 retained with 4/122 ineligible admissions. Thus the exclusion identifies where ranking failures occur, and the adaptation improves absolute admission beyond ledger decisions and the tested threshold adjustment.

Separately, q1>q0>q2q_1 > q_0 > q_2 satisfies every retained comparison in all subsets. The ranking target therefore admits a contract-only solution. This property of the evaluation target motivates measuring admission errors alongside pair ranking. The historical diagnostics and exclusion analysis are reported in Section 5.5 and Appendix A.

AuthorsOpenReview ↗

Response to Reviewer wZ2C (Part 2/5)

Comment 2: “Especially in the Challenge dataset, 62 constructed labels disagree with human judgments. And 60 of them are from not well-defined non-operative ledger contract.”

Response: We thank the reviewer for highlighting the historical ledger ambiguity. Appendix A.2 already specifies non-operative use: it permits relevant observations or explicitly unverified hypotheses for later investigation, requires uncertainty preservation, prohibits direct action control, and excludes stale observations. The ambiguity concerns the write operation: whether to store feedback verbatim, rewrite it for the target, or add an uncertainty qualification.

In Case B (Appendix L), annotator A accepts a supported workflow step as non-operative evidence; B/C reject its directive form. These rationales reflect different interpretations of the writing operation. With 60 of the 62 disagreements concentrated in ledger cases, the evaluation must distinguish contract interpretation from model prediction errors. The exclusion analysis in Comment 1 quantifies how these cases affect the evaluation.

We preserve the original majority labels and report the historical and prospective contracts separately. The prospective contracts explicitly specify write transformations and action authority.

Comment 3: “Could the authors clarify the relationship between the models and training configurations used in the two sets of experiments?”

Response: We appreciate the opportunity to clarify model identity. The controlled routers and fresh-confirmation candidate use distinct training configurations and checkpoints (Appendices F and A.1):

SettingModel and configurationEvaluation role
Loop-adapted controlled state/reuse studiesEarlier DeBERTa-v3-base router; 4 training epochs; 5,634 training rows (Appendix F)Admission consequences within specified learners and reuse mechanisms
Fresh human confirmationSeparately developed DeBERTa-v3-base candidate; 560/120/120 training/validation/locked-development groups, each with 3 contracts; 5 epochs; mean probabilities across seeds 13, 17, and 23 (Appendix A.1)Agreement with fresh human references at frozen thresholds
Prospective counterfactual auditSame frozen candidate as human confirmation; separately specified contracts; no additional training (Appendix G)Sensitivity to controlled contract/evidence changes under construction-defined references
Human-supervised extension (new)Original checkpoints adapted using human classification/ranking and construction replay; 3 epochs, learning rate 1e-5, seeds 13, 17, and 23 (Appendix H)Retrospective grouped evaluation with separate training, calibration, and test folds
Full-budget 7B selection study (new)Qwen2.5-7B-Instruct with QLoRA; frozen 14B writer; four arms, seeds 13, 17, and 23; 40M non-padding input tokens per run with complete action coverage; coverage-matched groups share 104 sources (Appendix K)Prespecified paired utility comparisons on 80 new task-disjoint cases

Appendix F also reports a template-disjoint router trained on 5,598 rows; Section 5.1 uses a matched-architecture target-conditioning intervention. The fresh candidate covers the full serialized context in chunks; its qq-masked control retains the architecture and objectives. The same-data retrained base is target-agnostic and single-window. Original confirmation models and thresholds were frozen before annotation, with the hard evidence gate disabled. Span-PRM-H and its qq-masked control receive the same subsequent human supervision. Separate selector fits using all 600 human-labeled rows serve downstream data selection only and never enter human held-out metrics. The full-budget study keeps the original all-label Span-PRM-H selector fixed; it does not substitute a Natural follow-up model. Appendices H and K detail human adaptation and the coverage-matched parameter-update study, respectively.

AuthorsOpenReview ↗

Response to Reviewer wZ2C (Part 3/5)

Comment 4: “Also, in the online-memory evaluation, this method is not obviously outperforming Write-All.”

Related question: “Furthermore, to what extent do the gains observed in the controlled experiments support the effectiveness of the proposed "target-conditioned admission" method in real-world settings?”

Response: We appreciate the reviewer’s focus on practical effects. We now verify downstream gains through actual 7B parameter updates, with source coverage and training budgets controlled. Section 5.3 separately reports a +0.583 reward gain over No-Memory in controlled memory-required chains (95% CI [0.438, 0.729]) and a −0.004 difference from Write-All in generated-feedback online memory (95% CI [−0.063, 0.050]). These evaluate the respective memory mechanisms; the new training experiment tests the parameter-update contract directly.

New Section 5.8 and Appendix K reuse 216 successful demonstrations from 104 source tasks, generated by the same frozen Qwen2.5-14B-Instruct writer. Random-Coverage and Span-PRM-H-Coverage select one demonstration from each of the same 104 source tasks. Success-only uses the full pool; the fixed stratified Random-50% subset remains a secondary control. All four arms train the same 7B student with QLoRA, 40M non-padding input tokens per run, and seeds 13, 17, and 23. Every run traverses its complete selected action set. The official solo-agent/dummy-user evaluation uses the same 80 new held-out tasks, tools, prompts, decoding, and scoring for all models, without writer feedback or added memory.

Training dataSeed 13Seed 17Seed 23Mean success
No-update (evaluated once)2/80——2.500%
Success-only30/8027/8025/8034.167%
Random-50%34/8028/8011/8030.417%
Random-Coverage24/8027/8026/8032.083%
Span-PRM-H-Coverage32/8039/8032/8042.917%

The two prespecified gains are +10.833 percentage points over Random-Coverage and +8.750 percentage points over Success-only. Bonferroni-adjusted 97.5% paired CIs are [5.417, 16.667] and [2.500, 15.000], respectively. Every training seed favors Span-PRM-H-Coverage in both primary comparisons. The 10,000 task-bootstrap resamples are stratified by family and preserve all three seed outcomes within each task; intervals condition on these fitted models.

This directly supports the learned selection policy’s practical value under the tested parameter-update contract. Source success is shared by every demonstration; the principal quality comparison also matches all 104 source tasks and the input budget. Actual supervision is 90,892–92,946 action tokens per full-budget run. Appendix K provides the complete protocol, budget comparisons, and separate development results.

Reviewer_R7rROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work examines what observable evidence suffices to justify incorporating feedback into a learner’s persistent state. The authors propose "target-conditioned update eligibility", a framework for evaluating whether observable evidence supports the integration of a given piece of feedback, subject to specific requirements regarding persistence and reusability. The findings highlight a vulnerability in how self-improving agents are evaluated: optimizing for local metrics alone is insufficient to establish evidence-based reasoning. Instead, criteria such as construction fit, reference-aligned input sensitivity, human agreement, and downstream utility must each be verified independently.

Critical strengths

This work focuses on how to determine whether a feedback is worth incorporating into a persistent state, which is a topic that has not been fully discussed and aligned in the RSI field. The problem itself has a certain degree of novelty.

Critical weaknesses

It remains somewhat questionable whether dimensions such as reference-aligned input sensitivity, human agreement, and downstream utility—which are currently the focus—are sufficient to characterize the long-term performance of self-improving agents.

Questions for authors

The research topic is both novel and significant. Are the dimensions used—reference-aligned input sensitivity, human agreement, and downstream utility—sufficiently comprehensive for characterizing the long-term performance of self-improving agents?

1 comment
AuthorsOpenReview ↗

Response to Reviewer R7rR

We thank the reviewer for the insightful and valuable comments. We respond to each comment as follows and sincerely hope that our rebuttal could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to your comments and improving our submission.

Comment 1: The research topic is both novel and significant. Are the dimensions used—reference-aligned input sensitivity, human agreement, and downstream utility—sufficiently comprehensive for characterizing the long-term performance of self-improving agents?

Response: We appreciate the reviewer's recognition of our research and the opportunity to clarify the evaluation design. The three dimensions assess target-conditioned feedback admission and its consequences for persistent learning. The dimensions separately test whether decisions track the specified evidence, whether they agree with human judgments, and whether the resulting updates improve subsequent task performance. Long-term agent evaluation extends these checks across repeated updates and evolving task distributions.

  1. Scope of our study. We investigate whether observable evidence justifies admitting feedback under a specified persistence and reuse contract (Section 3.1, Eq. (1)). This decision determines which feedback can influence subsequent learner-state transitions (Section 3.2, Eq. (3)). Our controlled experiments in Section 5.2 demonstrate that admission changes actual learning updates and subsequent state reuse, confirming its consequential role in self-improvement.

  2. Complementary evaluation dimensions. The three dimensions examine distinct aspects:

    1. Reference-aligned input sensitivity tests whether decisions respond appropriately to controlled changes in contracts and decisive evidence (Section 5.6).

    2. Human agreement evaluates alignment with independent human judgments. Section 5.4 tests whether construction-defined development performance transfers to fresh human references. New Section 5.7 evaluates the human-supervised extension, Span-PRM-H, on retrospectively held-out templates: 70/78 eligible Challenge items are retained with 4/222 ineligible admissions. These measurements assess admission agreement under the specified contracts.

    3. Downstream utility measures realized benefits under the intended update mechanism. Controlled memory-required tasks yield 0.883 reward versus 0.300 for No-Memory (Section 5.3). New Section 5.8 extends utility evaluation to actual Qwen2.5-7B-Instruct parameter updates with matched source coverage and complete action-data traversal. Span-PRM-H selection reaches 42.917% success, exceeding coverage-matched random selection and Success-only in both prespecified paired comparisons.

  3. Why the checks are separate. A fixed contract ordering satisfies all 120 human-majority Challenge comparisons without examining feedback or source evidence (Section 5.5). This diagnostic identifies a contract-only solution to the ranking target. Evaluating input sensitivity, human agreement, and downstream utility separately distinguishes these different aspects of performance.

  4. Relationship to long-term evaluation. Longitudinal studies would apply these checks across repeated updates while measuring cumulative state changes, memory interactions, retrieval reliability, and performance under changing task distributions. Our framework specifies the admission and utility questions to examine within such a study.

The revised Introduction, Discussion, and Limitations make this scope explicit: the dimensions provide complementary checks of feedback admission and its consequences, with longitudinal evaluation addressing their behavior over sustained agent operation.

We humbly hope our response has addressed your concerns. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work.

Reviewer_EtwoOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates what evidence is required to justify incorporating interaction-derived feedback into the persistent memory of self-improving agents. The authors argue that local task success does not necessarily justify persistent reuse, and existing benchmarks may reward shortcut heuristics rather than genuine evidence-based judgments. To address this, they introduce target-conditioned update eligibility, which evaluates candidate feedback against an explicit update contract specifying its destination, persistence, and scope of future reuse. Based on this framework, the authors develop a contract-conditioned Span-PRM to determine whether feedback should be retained, discarded, or assigned to a particular memory tier. Experimental results show that contract conditioning can influence memory updates and reduce contamination in controlled settings. However, the proposed model fails to generalize to fresh human evaluations, admitting none of the 300 generated reflections, achieving only 37.5% ranking agreement with human judgments, and failing prospective counterfactual audits.

Critical strengths
  1. Interesting and timely research problem. The paper addresses an important challenge in recursive self-improving (RSI) agents. As agents increasingly learn from their interactions and update persistent memory, determining what information should be retained and reused becomes critical for improving long-term performance and preventing memory contamination.
Critical weaknesses
  1. Limited actionable insights and practical contributions. The paper’s main conclusions are relatively intuitive and abstract, limiting their value to the research community. In particular, the findings largely summarize observations from controlled benchmark settings rather than providing deeper insights into the underlying mechanisms of feedback eligibility or memory contamination. Moreover, the proposed approach fails to generalize to fresh human evaluations, raising concerns about its practical effectiveness. It remains unclear what concrete lessons researchers can draw from these findings to design more reliable self-improving agents.
  2. Poor clarity and organization. The paper is difficult to follow due to unclear terminology, insufficiently motivated concepts, and confusing organization. Several central concepts, such as target-conditioned update eligibility, are introduced without sufficiently clear operational definitions or concrete examples that distinguish them from existing approaches. Furthermore, the connections between the proposed conceptual framework, model implementation, and experimental evaluation are not always clear. The authors should substantially improve the presentation, provide more intuitive explanations of key concepts, and reorganize the paper to establish a clearer narrative.
Questions for authors
  1. Beyond the observations reported in the controlled experiments, what are the main actionable insights that can guide future research on self-improving agents? In particular, what do the negative generalization results reveal about the limitations of existing feedback-selection mechanisms, and how could these findings inform the design of more reliable memory update strategies?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Etwo (Part 1/4)

We thank the reviewer for the insightful and valuable comments. We respond to each comment as follows and sincerely hope that our rebuttal could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to your comments and improving our submission.

New evidence on human agreement and agent training. Span-PRM-H adapts the original Span-PRM checkpoints using human supervision and retains 70/78 eligible Challenge items with 4/222 ineligible admissions, improving retention over per-contract calibration by 24.4 percentage points (paired 95% CI [14.3, 35.3]). With matched source coverage and 40M-token student training, its selection reaches 42.917% task success, improving over matched random selection by 10.833 percentage points and Success-only by 8.750 percentage points; both adjusted 97.5% paired intervals exclude zero (Comments 3–4).

Comment 1: “Limited actionable insights and practical contributions. The paper’s main conclusions are relatively intuitive and abstract, limiting their value to the research community.”

Response: We appreciate the opportunity to clarify the concrete contributions of our work. A fixed contract ordering (q1>q0>q2q_1 > q_0 > q_2) satisfies all 120 historical Challenge human-majority comparisons without examining feedback or source evidence (Section 5.5). This identifies a defect in the evaluation target: contract preference alone can satisfy a metric intended to assess evidence-sensitive judgment.

The transferable contribution is a check for whether contract identity alone can satisfy a ranking evaluation. Appendix A.6 proves that a fixed order satisfies every strict comparison exactly when the contract-preference graph is acyclic. Benchmark designers can apply this check before attributing high ranking to evidence use, then use balanced preferences and controlled evidence changes to test the shortcut (Section 5.6). The new adaptation and training experiments further test which improvements translate into better admission or task performance (Comments 3–4).

Comment 2: “In particular, the findings largely summarize observations from controlled benchmark settings rather than providing deeper insights into the underlying mechanisms of feedback eligibility or memory contamination.”

Response: We appreciate the reviewer's emphasis on understanding how feedback admission affects learning. Our controlled studies intervene on two distinct components of the learning loop:

  1. Contract dependence (Section 5.1). We hold feedback (zz) and source evidence (cc) fixed while varying only the update contract (qq), using a matched architecture. Under construction-defined labels, target conditioning achieves median OOD balanced accuracy of 0.984 versus 0.684 for target masking across seeds 13, 17, and 23, on unseen contract paraphrases. This establishes that the intended reuse contract materially affects the model’s admission decisions in the controlled setting.

  2. Effects on optimization (Section 5.2). We change feedback routing over the same 720-trace pool while keeping the learner and training protocol fixed. Write-All admits 525 contaminated spans, with a trajectory-mean gradient cosine of −0.018 relative to the clean-oracle update. The Span-PRM router in this protocol admits no contaminated spans and achieves a cosine of 1.000 (Figure 3). This directly demonstrates that admission affects the direction of actual learner updates, rather than merely determining which text is stored.

Together, these controlled interventions establish two consequential links: the update contract affects admission decisions, and admission decisions affect subsequent optimization.

AuthorsOpenReview ↗

Response to Reviewer Etwo (Part 2/4)

Comment 3: “Moreover, the proposed approach fails to generalize to fresh human evaluations, raising concerns about its practical effectiveness.”

Related question: “In particular, what do the negative generalization results reveal about the limitations of existing feedback-selection mechanisms, and how could these findings inform the design of more reliable memory update strategies?”

Response: We appreciate the opportunity to address this concern with new experimental evidence. The added experiment demonstrates improved human-reference admission on held-out Challenge templates, beyond the tested calibration baselines. Span-PRM-H retains DeBERTa-v3-base and continues training from the original checkpoints using human-majority classification, within-feedback ranking, and low-weight construction-data replay (new Section 5.7; Appendix H).

Each of five folds holds out one Challenge template for testing and one for calibration; the other three provide training data. Contracts of the same feedback remain together. Natural is grouped by source task and linked trajectory. Predictions are averaged across seeds 13, 17, and 23 under fixed hyperparameters. This is retrospective grouped evaluation using 600 previously annotated rows, with training, calibration, and test groups kept separate in every fold.

Results on the complete Challenge set are:

MethodAUPRCEligible retainedIneligible admitted
Original frozen Span-PRM0.30510/7860/222
+ global threshold calibration0.30510/780/222
+ per-contract threshold calibration0.305151/786/222
Contract-only0.551560/7840/222
Human-adapted qq-masked control0.355546/7844/222
Span-PRM-H0.995770/784/222

The original baseline retains its historical threshold. All new thresholds use calibration folds only, maximizing retention subject to an empirical ineligible-admission rate of at most 0.10; the table reports the resulting test outcomes. Relative to per-contract calibration, Span-PRM-H improves retention by 24.4 percentage points (paired 95% CI [14.3, 35.3]). Intervals use 10,000 group-bootstrap replicates conditional on the fitted models, folds, and thresholds.

Both Span-PRM-H and contract-only rank every pair correctly, yet the latter admits 40 ineligible rows; the equally supervised qq-masked control retains fewer positives and admits more negatives. Within-contract AUPRC is 0.9759 for persistent cross-task use and 1.0000 for local use. These controls support improved conditional discrimination on Challenge beyond the tested calibration and contract-only alternatives.

Persistent cross-task use within Challenge retains 10/18 positives with 2/82 ineligible admissions. The adaptation establishes a concrete improvement in template-held-out Challenge admission, measured through both eligible retention and ineligible admission.

Natural: learning from annotation disagreement (Appendix J). A separate joint majority-label and vote-fraction probe reaches AUPRC 0.1448, approximately 3.7 times that of Span-PRM-H (0.0390). Natural has 12/300 majority-eligible items (4%): all 12 receive 2/3 eligible votes, while 45 majority-negative items receive one eligible vote. Binary majority labels omit these differences in agreement, motivating vote fractions as auxiliary supervision. The probe freezes pretrained DeBERTa-v3-base and uses shared linear weights with separate biases for the two targets; the matched frozen-feature hard-label probe reaches AUPRC 0.1023. With Natural-only retention-maximizing calibration, the joint probe retains 2/12 eligible items with 4/288 ineligible admissions; a separately specified negative-only rule yields 4/12 with 27/288. These retrospective exploratory evaluations keep training, calibration, and test groups separate and preserve the majority test labels. Appendix J reports the matched probes, calibration variants, and paired uncertainty.

AuthorsOpenReview ↗

Response to Reviewer Etwo (Part 3/4)

Comment 4: “It remains unclear what concrete lessons researchers can draw from these findings to design more reliable self-improving agents.”

Related question: “Beyond the observations reported in the controlled experiments, what are the main actionable insights that can guide future research on self-improving agents?”

Response: We thank the reviewer for requesting concrete guidance. The revised experiments support three steps, each tied to an explicit check:

  1. Specify the write operation and reuse authority. Sixty of the 62 construction–majority disagreements concern the historical ledger (Section 5.5). This motivates specifying what is written, how uncertainty is preserved, and whether the record may guide actions. Our prospective contracts make these operations explicit; the parameter-update study specifies the writer, student parameters, and intended reuse.

  2. Adapt using contract-matched human judgments and test against simple alternatives. Section 5.7 implements this step. Calibration, contract-only, and equally supervised qq-masked controls show that its Challenge improvement exceeds these alternatives. Reporting eligible retention alongside ineligible admission separates useful selection from ranking shortcuts and rejection of everything.

  3. Preserve source-task coverage and test the resulting parameter updates. New Section 5.8 uses a common pool of 216 successful demonstrations from 104 source tasks, generated by one frozen 14B writer. Random-Coverage and Span-PRM-H-Coverage each select one demonstration per source task; Success-only uses all 216. Each 7B QLoRA training arm is run with seeds 13, 17, and 23; every run receives 40M non-padding input tokens and visits every selected action example. We evaluate every model on the same 80 new held-out tasks, without writer feedback or added memory.

Training dataMean task success
Success-only34.167%
Random-Coverage32.083%
Span-PRM-H-Coverage42.917%

The prespecified gains are +10.833 percentage points over Random-Coverage and +8.750 percentage points over Success-only, with Bonferroni-adjusted 97.5% paired CIs [5.417, 16.667] and [2.500, 15.000], respectively. All three training seeds favor Span-PRM-H-Coverage in both comparisons. The 10,000 bootstrap resamples are stratified by task family and retain all three seed outcomes within each task.

This supplies a tested design choice: select within source tasks to preserve coverage, then validate the selected demonstrations through the intended parameter update. The two coverage-matched groups share exactly 104 source tasks, and all demonstrations already succeed on their source task. The comparison therefore tests the added value of the learned selection policy under matched coverage and input budgets. Appendix K records the complete schedules, budget comparisons, and 90,892–92,946 supervised action tokens per full-budget run.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Updates & Reviewer Discussion for the Area Chair

Dear Area Chair,

We sincerely thank you and all reviewers for their constructive feedback and for recognizing the core strengths of our work, particularly the importance and novelty of the research question, the formulation of target-conditioned update eligibility, and the diagnostic value of combining controlled experiments, human judgments, and counterfactual tests. Building on this feedback, we conducted additional experiments on human-supervised adaptation and 7B agent fine-tuning and revised the manuscript to clarify the framework and strengthen its practical implications. New material in the Introduction and Sections 5.7–5.8 is highlighted in blue for ease of review.

To assist with your assessment, we summarize the key updates and their connections to the reviewer concerns below:

  • Human-Supervised Adaptation (Etwo, wZ2C): Added Span-PRM-H, a human-supervised adaptation of the original Span-PRM. In retrospective template-held-out Challenge evaluation, it retains 70/78 eligible items with 4/222 ineligible admissions, exceeding contract-specific calibration by 24.4 percentage points (paired 95% CI [14.3, 35.3]). (Section 5.7; Appendix H.)
  • Downstream Training Benefits (Etwo, wZ2C): Added coverage-matched 7B agent fine-tuning with 104 source tasks and 40M input tokens per run. Across 80 held-out tasks and three seeds, Span-PRM-H-Coverage achieves 42.917% success, exceeding matched random selection by 10.833 percentage points and Success-only by 8.750 percentage points. Both adjusted 97.5% paired confidence intervals exclude zero. (Section 5.8; Appendix K.)
  • Ledger Ambiguity and Ranking Diagnostics (wZ2C): Added four exclusion analyses. Removing historical ledger rows raises ranking agreement from 37.5% overall to 100% on the 42 remaining comparisons, separating ledger-related ranking discrepancies from absolute-admission errors. Formalized a check for whether contract ordering alone can satisfy a ranking benchmark. (Section 5.5; Appendix A.6.)
  • Counterfactual Validity and Evidence Use (wZ2C): Clarified write transformations, action authority, and reference semantics. Under the fixed prospective references, full-input EvidenceFlip ranking falls 0.096 below shuffled-context controls, providing a threshold-independent diagnostic that uses no historical ledger labels. (Section 5.6; Appendix G.)
  • Framework Clarity and Practical Guidance (Etwo, wZ2C): Expanded the worked example, clarified the connections among eligibility, admission, and learner updates, and added a unified model-configuration table. Specified actionable checks for contract shortcuts, calibration, admission quality, and downstream utility. (Sections 1, 3–4, and 6; Appendix F, Table 14.)
  • Long-Term Evaluation Scope (R7rR): Clarified the distinct roles of construction fit, reference-aligned input sensitivity, human agreement, and downstream utility, and outlined longitudinal evaluation of repeated updates, memory interactions, retrieval errors, and distribution shifts. (Sections 6–7.)

Current discussion status: As of this summary, we have not yet received follow-up comments from any of the three reviewers. We remain available to answer further questions and clarify the additional evidence.

We respectfully hope that these additional experiments, diagnostic analyses, and manuscript revisions will help inform your final assessment, particularly regarding the concerns addressed in our rebuttal for which reviewer follow-up is still pending.

Thank you again for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.