- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies one problem in self-improving agents: should self-improving agents distill interaction feedback into persistent memory to enhance future capabilities from one interaction? The main idea is whether feedback is worth to learn or not does not depend only on whether the feedback itself is correct, but also on the evidence supporting that feedback, and how agent plan to use it in the future. The contribution of this paper is twofold: (1) it provides the formulation of target-conditioned update eligibility, (2) it systematically evaluates why current evaluation methods and feedback models are unable to provide which experience should be put into persistent memory.
Critical strengths
- The problem definition and research perspective are meaningful. The paper proposes target-conditioned update eligibility. It emphasizes that whether feedback should be included in long-term learning should not depend only on content and its evidence, but also on the future target and reuse scope. This perspective is beyond traditional feedback quality filtering. It provides the self-improving agents' long-term memory management and selective update with a more definite framework.
- The experimental design is meaningful, and the negative results are valuable. This paper combines controlled experiments, human annotation, and counterfactual testing. Thus, it reveals the problem that the model performs well on the synthetic dataset but cannot generalize to real human judgment. Especially, the fixed contract order can satisfy all the human preference comparisons, revealing shortcuts of current evaluation metrics.
Critical weaknesses
- Although Span-PRM achieves near-perfect performance in the construction set, it fails in the human evaluations. In the Natural dataset, it rejects all 300 feedback items (At the frozen threshold, it admits none of the 300 reflections, yielding zero observed ineligible admissions and zero eligible retention). In the Challenge dataset, the pair-ranking accuracy is only 37.5% (Span-PRM obtains flip ranking 0.375 and pair-binary 0.000, failing the prespecified 0.80/0.60 criteria). And it fails to pass the counterfactual test. Also, in the online-memory evaluation, this method is not obviously outperforming Write-All. These results indicate that even this paper proposes a critial problem. However, it cannot prove that the proposed method can resolve this problem well.
- This paper highly relies on the human-constructed eligibility labels. And it does not pass the individually validated real feedback judgment. Especially in the Challenge dataset, 62 constructed labels disagree with human judgments. And 60 of them are from not well-defined non-operative ledger contract. Hence, it is hard to differentiate whether the performance gap is due to a lack of evidence understanding ability or the evaluation criteria itself. Hence, it is hard to believe the main conclusion of this paper.
Questions for authors
- In the Challenge dataset, 60 of the 62 data points where the labels are from the majority-vote human judgments involved "non-operative ledger contracts", which is a category lacking a clear definition. Could the authors clarify whether the primary evaluation conclusions for Span-PRM still hold if these ambiguous cases are excluded? This would help determine whether the model’s poor performance is from a genuine lack of reasoning capability or from flaws in the evaluation criteria themselves.
- The paper reports gains for Span-PRM in controlled experiments; however, in independent human evaluations, the frozen Span-PRM failed to demonstrate an effective capability for feedback filtering. Could the authors clarify the relationship between the models and training configurations used in these two sets of experiments? Furthermore, to what extent do the gains observed in the controlled experiments support the effectiveness of the proposed "target-conditioned admission" method in real-world settings?
- The reference labels for ContractFlip and EvidenceFlip are still generated via manually defined rules and have not been independently validated by humans. How do the authors confirm that these rules reliably reflect "evidence-sensitive eligibility"? Given that previous efforts have shown significant differences between constructed labels and human judgments, to what extent should failures in counterfactual tests be interpreted as a deficiency in the model’s reasoning capabilities, rather than as an inconsistency between the constructed rules and the actual criteria used for human judgment?
12 comments
Response to Reviewer wZ2C (Part 2/5)
Comment 2: “Especially in the Challenge dataset, 62 constructed labels disagree with human judgments. And 60 of them are from not well-defined non-operative ledger contract.”
Response: We thank the reviewer for highlighting the historical ledger ambiguity. Appendix A.2 already specifies non-operative use: it permits relevant observations or explicitly unverified hypotheses for later investigation, requires uncertainty preservation, prohibits direct action control, and excludes stale observations. The ambiguity concerns the write operation: whether to store feedback verbatim, rewrite it for the target, or add an uncertainty qualification.
In Case B (Appendix L), annotator A accepts a supported workflow step as non-operative evidence; B/C reject its directive form. These rationales reflect different interpretations of the writing operation. With 60 of the 62 disagreements concentrated in ledger cases, the evaluation must distinguish contract interpretation from model prediction errors. The exclusion analysis in Comment 1 quantifies how these cases affect the evaluation.
We preserve the original majority labels and report the historical and prospective contracts separately. The prospective contracts explicitly specify write transformations and action authority.
Comment 3: “Could the authors clarify the relationship between the models and training configurations used in the two sets of experiments?”
Response: We appreciate the opportunity to clarify model identity. The controlled routers and fresh-confirmation candidate use distinct training configurations and checkpoints (Appendices F and A.1):
| Setting | Model and configuration | Evaluation role |
|---|---|---|
| Loop-adapted controlled state/reuse studies | Earlier DeBERTa-v3-base router; 4 training epochs; 5,634 training rows (Appendix F) | Admission consequences within specified learners and reuse mechanisms |
| Fresh human confirmation | Separately developed DeBERTa-v3-base candidate; 560/120/120 training/validation/locked-development groups, each with 3 contracts; 5 epochs; mean probabilities across seeds 13, 17, and 23 (Appendix A.1) | Agreement with fresh human references at frozen thresholds |
| Prospective counterfactual audit | Same frozen candidate as human confirmation; separately specified contracts; no additional training (Appendix G) | Sensitivity to controlled contract/evidence changes under construction-defined references |
| Human-supervised extension (new) | Original checkpoints adapted using human classification/ranking and construction replay; 3 epochs, learning rate 1e-5, seeds 13, 17, and 23 (Appendix H) | Retrospective grouped evaluation with separate training, calibration, and test folds |
| Full-budget 7B selection study (new) | Qwen2.5-7B-Instruct with QLoRA; frozen 14B writer; four arms, seeds 13, 17, and 23; 40M non-padding input tokens per run with complete action coverage; coverage-matched groups share 104 sources (Appendix K) | Prespecified paired utility comparisons on 80 new task-disjoint cases |
Appendix F also reports a template-disjoint router trained on 5,598 rows; Section 5.1 uses a matched-architecture target-conditioning intervention. The fresh candidate covers the full serialized context in chunks; its -masked control retains the architecture and objectives. The same-data retrained base is target-agnostic and single-window. Original confirmation models and thresholds were frozen before annotation, with the hard evidence gate disabled. Span-PRM-H and its -masked control receive the same subsequent human supervision. Separate selector fits using all 600 human-labeled rows serve downstream data selection only and never enter human held-out metrics. The full-budget study keeps the original all-label Span-PRM-H selector fixed; it does not substitute a Natural follow-up model. Appendices H and K detail human adaptation and the coverage-matched parameter-update study, respectively.
Response to Reviewer wZ2C (Part 3/5)
Comment 4: “Also, in the online-memory evaluation, this method is not obviously outperforming Write-All.”
Related question: “Furthermore, to what extent do the gains observed in the controlled experiments support the effectiveness of the proposed "target-conditioned admission" method in real-world settings?”
Response: We appreciate the reviewer’s focus on practical effects. We now verify downstream gains through actual 7B parameter updates, with source coverage and training budgets controlled. Section 5.3 separately reports a +0.583 reward gain over No-Memory in controlled memory-required chains (95% CI [0.438, 0.729]) and a −0.004 difference from Write-All in generated-feedback online memory (95% CI [−0.063, 0.050]). These evaluate the respective memory mechanisms; the new training experiment tests the parameter-update contract directly.
New Section 5.8 and Appendix K reuse 216 successful demonstrations from 104 source tasks, generated by the same frozen Qwen2.5-14B-Instruct writer. Random-Coverage and Span-PRM-H-Coverage select one demonstration from each of the same 104 source tasks. Success-only uses the full pool; the fixed stratified Random-50% subset remains a secondary control. All four arms train the same 7B student with QLoRA, 40M non-padding input tokens per run, and seeds 13, 17, and 23. Every run traverses its complete selected action set. The official solo-agent/dummy-user evaluation uses the same 80 new held-out tasks, tools, prompts, decoding, and scoring for all models, without writer feedback or added memory.
| Training data | Seed 13 | Seed 17 | Seed 23 | Mean success |
|---|---|---|---|---|
| No-update (evaluated once) | 2/80 | — | — | 2.500% |
| Success-only | 30/80 | 27/80 | 25/80 | 34.167% |
| Random-50% | 34/80 | 28/80 | 11/80 | 30.417% |
| Random-Coverage | 24/80 | 27/80 | 26/80 | 32.083% |
| Span-PRM-H-Coverage | 32/80 | 39/80 | 32/80 | 42.917% |
The two prespecified gains are +10.833 percentage points over Random-Coverage and +8.750 percentage points over Success-only. Bonferroni-adjusted 97.5% paired CIs are [5.417, 16.667] and [2.500, 15.000], respectively. Every training seed favors Span-PRM-H-Coverage in both primary comparisons. The 10,000 task-bootstrap resamples are stratified by family and preserve all three seed outcomes within each task; intervals condition on these fitted models.
This directly supports the learned selection policy’s practical value under the tested parameter-update contract. Source success is shared by every demonstration; the principal quality comparison also matches all 104 source tasks and the input budget. Actual supervision is 90,892–92,946 action tokens per full-budget run. Appendix K provides the complete protocol, budget comparisons, and separate development results.
Response to Reviewer wZ2C (Part 1/5)
We thank the reviewer for the insightful and valuable comments. We respond to each comment as follows and sincerely hope that our rebuttal could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to your comments and improving our submission.
New evidence on human agreement and downstream utility. Span-PRM-H adapts the original checkpoints using human supervision; it retains 70/78 eligible Challenge items with 4/222 ineligible admissions, improving retention over per-contract calibration by 24.4 percentage points (95% CI [14.3, 35.3]). In the new coverage-matched 7B training study, its selection reaches 42.917% success, exceeding matched random selection by 10.833 percentage points and Success-only by 8.750 percentage points, with both adjusted 97.5% paired intervals above zero (Comments 4 and 6).
Response: We appreciate the reviewer’s request for a ledger-exclusion analysis. It yields a specific diagnosis: the ranking failures are confined to comparisons involving the historical ledger, while an absolute-admission failure remains after those comparisons are removed. Excluding all ledger rows raises ranking from 37.5% to 100% on the 42 retained comparisons, but frozen-threshold retention remains 0/78. The following analysis uses unchanged predictions, human-majority labels, and thresholds:
Disputed ledger rows are defined by construction–majority disagreement, without model scores. Row exclusion retains the other contracts; group exclusion removes all three. Valid pairs compare eligible and ineligible contracts within a group, with half credit for ties. The 10,000 group-bootstrap replicates keep retained contracts together and condition on the frozen majority reference. We report these as post-hoc intervals; the 100% endpoint describes the retained comparisons and their resamples.
Pair-binary accuracy is zero under every exclusion for the original candidate and both comparison baselines.
The calibration controls quantify the extent to which threshold adjustment improves admission. Per-contract calibration restores retention to 51/78 eligible items with 6/222 ineligible admissions; Span-PRM-H reaches 70/78 with 4/222, a further retention improvement of 24.4 percentage points over the tested recalibration (Comment 6). Excluding all ledger rows from the adapted evaluation gives 70/78 retained with 4/122 ineligible admissions. Thus the exclusion identifies where ranking failures occur, and the adaptation improves absolute admission beyond ledger decisions and the tested threshold adjustment.
Separately, q1>q0>q2 satisfies every retained comparison in all subsets. The ranking target therefore admits a contract-only solution. This property of the evaluation target motivates measuring admission errors alongside pair ranking. The historical diagnostics and exclusion analysis are reported in Section 5.5 and Appendix A.