- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces “revision sufficiency,” asking whether experience retained for an original objective remains useful after the success criterion changes. The authors compare a trajectory archive with a compact strategy constructed from the same experience for the original objective. In a synthetic tool-use environment, the original objective checks two directly updated variables, whereas the revised objective additionally requires a variable governed by a hidden ordering constraint. On 507 complete paired seeds, the compact strategy performs better before revision but substantially worse afterward. A separate control shows that constructing the strategy for the revised objective under the same length cap improves revised-task performance. The paper also presents execution audits, additional controls, and a proposed evidence-dependency framework called LEDGER.
Critical strengths
-
The paper raises a relevant concern for agents that convert experience into reusable strategies: improvement on the current objective does not establish usefulness under future requirements.
-
The primary comparison uses the same source experience and matches planning and execution conditions within each stage. The revised-objective builder control is useful because it distinguishes the construction objective from the length cap. Additional controls holding the instance and budget fixed strengthen the interpretation of the behavioral result.
-
The authors transparently report protocol limitations, unsuccessful diagnostics, and negative results. Sensitivity analyses show that missing outcomes and the undeclared turn guard cannot fully explain the direction of the post-revision difference. A fixed-representation second-reader experiment also reproduces the ranking reversal, although with a smaller effect.
Critical weaknesses
-
Limited novelty. The central finding is a relatively predictable limitation of objective-specific lossy compression: information unnecessary for the original objective may become necessary later. The paper does not sufficiently establish what “revision sufficiency” adds beyond this observation.
-
Narrow empirical scope. The main evidence comes from one synthetic task family, one objective expansion, and a very small archive. The second-reader experiment reproduces the reversal, but realistic workloads, cross-structure transfer, and repeated revisions remain unestablished. Hundreds of seeds improve within-task precision rather than external validity.
-
Unresolved mechanism. Text-matching audits cannot distinguish information loss from utilization difficulty. Evidence add-back does not establish behavioral recovery, archive interventions do not localize the effect, and the gradient diagnostic fails qualification. The paper establishes the behavioral gap more clearly than its cause.
-
Unvalidated practical recommendations. LEDGER remains a design proposal, while scope stamps and archive pointers are untested. The small archive also leaves the practical tradeoff between compression, evidence retention, and retrieval cost unresolved. Table 1 partly emphasizes corrected evaluations, whereas the main experiment tests objective expansion.
-
Protocol and statistical limitations. The undeclared turn guard affects the two paths unequally; sensitivity bounds protect the direction but not the effect size under the intended budget. The preregistered gates do not directly establish ranking reversal, which instead relies on separate paired contrasts.
Questions for authors
-
What specific conceptual or empirical finding distinguishes revision sufficiency from the expected limitations of representations constructed for an earlier objective?
-
Does the main result persist when unsuccessful combination returns an explicit error, the preparation state is observable, or different variables become relevant after revision?
-
How does the archive compare with compact representations explicitly designed to preserve operation rules, failure conditions, and evaluation scope without knowing the specific future objective?
-
What evidence supports applying the findings to repeated strategy updates in realistic agent systems, beyond the isolated handoff evaluated here?
-
Can the authors provide anonymous, timestamped registration and amendment records and clearly separate registered analyses from exploratory ones?