acceptodds
Under review as a conference paper at ICLR 2027

When Can a Skill Library Trust Its Execution Records?

Abstract

A skill library may use a successful execution record to predict an update's effect, decide whether to deploy it, or justify replacing the full skill with shorter text. These are different judgments. On sixteen known QA questions, records of the same proposal at other receivers reduce squared F1-effect risk by relative to equal-budget records of other proposals at the same receivers. That information raises the value of a fixed retrospective gate by F1 points over its matched control, yet the gate's own value relative to rejecting every update is estimated at points ( interval ). Receiver-only and pooled histories also attain lower absolute prediction risk than same-proposal history. In a separate crossed study, three local trials beat pooled records in two settings and lose in two others as the balance of local execution noise and nonlocal error reverses. Finally, performance substitution—replacing a full skill with one section while maintaining its mean score within a prespecified margin—changes answer distributions and source outcomes under GLM. Execution records do transfer useful information, but prediction, deployment, and retention of particular successes require distinct comparisons.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LLM agents, textual self-evolution, agent skills, skill transfer, lifelong learning, agent evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_K5vHOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper tries to answer the question: When Can an AI Skill Library Trust Its Past Results? They split the topic into three (1) Predict i.e how well a new instruction will work in a new setting. (2) Deployment i.e. to decide to deploy an update. and (3) Retention/Compress i.e. shortening a long instruction. They show that: For Predict: Past results give a good idea, but do not guarantee success. For Deployment: Local test are useful when the AI system has high output variation, but past history is better otherwise. For retention/compression: shortening instructions changes the answers and some fixed mistakes may come back.

Critical strengths

A good modular framework: separating prediction, deployment, and retention/compression. Theoretical Grounding: which is nice to have - but to be honest I did not proof check. Some degree of integrity: reporting some minor negative results (in particular a result worse than baseline) Multiple AI models were tested.

Critical weaknesses

Paper is difficult to read - at least if not in the nichc (?) domain. Experimentation is weak: all based on Short QA Benchmark with everything associated (and even pointed out by the authors in the annex): for instance high sensitivity to phrasing. Some conclusion are a bit to strong: for instance authors claim that deployment gates in general fail to create value. But the reality is that they tested a "naive" zero-threshold deployment gate and this failed to yield positive value.

Questions for authors

None

2 comments
AuthorsOpenReview ↗

Reviewer K5vH

We can see why the gate result reads as a broader claim about deployment than the experiment can support. We evaluated one fixed rule: install an update when its predicted effect, averaged over the sixteen questions, is positive. The resulting value is -0.591 F1 points relative to rejecting every update, with a descriptive 95% interval [-1.425, 0.042]. This leaves the payoff of that rule unresolved. It cannot tell us whether deployment gates are useful in general.

What the comparison does show is that changing the history used by this rule improves its value by +0.324 points [0.013, 0.717] relative to matched other-proposal history (Section 3; Appendix C). A record can therefore help this particular decision without yet making it a demonstrably beneficial one. We should have kept that qualification close to the deployment claim throughout the introduction, rather than leaving the reader to recover it from the detailed results.

Your concern about short-answer QA also affects how broadly these findings should be read. We used this setting because it lets us hold the question and installed texts fixed while changing which records a predictor can consult. Matching the donor receivers and record count then gives a controlled test of proposal-associated information. The sixteen evaluation questions are held out from optimizer training and development, but their records at other text combinations are visible to the predictors. The finding consequently concerns cross-receiver prediction on known questions. It does not establish reuse on new tasks or long-horizon agent workflows.

For the extraction study, phrasing is part of the difficulty you identify. Two answers may differ after normalization even when the difference has little practical consequence. Section 6 therefore compares cross-text disagreement with the variability from rerunning the same text, and Appendix M.4 checks sensitivity across four scoring interfaces. These controls tell us more than a single-run score comparison, but they still do not give the factual-error rate among changed answers. The 128-question HotpotQA extraction panel supports a finding about mean-score replacement and answer changes within that setting.

The presentation would be easier to follow if we introduced the decisions through a concrete example. Suppose a library adds an evidence-checking instruction to a receiving document. Its history might help forecast the score change; the library still has to decide whether installing it is worthwhile; and shortening it raises the further question of which particular answers survive. We will use this example to explain the three targets before introducing the notation, define the receiving document and its comparator at first use, and give each main study a short explanation of the question it answers. We will also state the fixed-rule scope beside the gate result and describe extraction outcomes as lost or recovered exact matches, keeping their factual interpretation separate.

Thank you for your comments. Unfortunately, I stick to my rating.

Reviewer_6K5FOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether execution records from previous skill updates can reliably guide skill reuse in LLM agents. Specifically, the authors did some experiments to examine whether past results can predict an update's performance in new contexts, whether local testing is more useful than historical records, and whether shortening a skill preserves its behavior. The results suggest that historical records provide some predictive information, but this does not necessarily translate into better deployment decisions. They also find that local testing is not always preferable.

Critical strengths

None

Critical weaknesses

1.There is an obvious gap between claimed research motivation and the actual evaluation experiments. The title suggests that the paper proposes to address whether or not skill library can trust the successful execution records. However, in the core experimental part (Section 3), only 1 update out of 144 candidates was accepted, which make it less convincing that this paper has real practical application value (in the results in Table 1, the simpler Receiver-only and Pooled predictor even has lower prediction risk than the Same-proposal).

2.The method novelty is very limited, and the mathematical analysis is trivial. For example, the so-called decomposition in Section 2 (V(U)-V(N) = [V(P)-V(N)]+[V(U)-V(P)]) is very simple, and the theoretical analysis in Section 4 is just comparing the local estimation noise and nonlocal prediction error.

Questions for authors

None

2 comments
AuthorsOpenReview ↗

Reviewer 6K5F

The gap you point out between the motivation and the proposal queue is real. Only one of the 144 proposals passed optimizer-development selection, and Section 3 includes all of them. Keeping the rejected candidates lets us study what their records predict at other receivers, but this queue gives limited evidence about a library of selected successful skills. The opening should make that distinction clear instead of allowing the successful-skill motivation to stand in for the population actually tested.

Your observation about receiver-only and pooled prediction is important for the same reason. Our matched comparison was designed to ask whether proposal identity carries information when the two donor receivers and two-record budget stay fixed. Using the target proposal's records reduces squared F1-effect risk by 0.00797 [0.00131, 0.01797] compared with other proposals' records at those receivers. All target proposal-receiver executions are excluded, while the question is known through other combinations. Receiver-only and pooled histories nevertheless attain lower absolute risks (Table 1). The matched result establishes information under those access rules; it does not make same-proposal history the best available predictor, nor show that adding it to a receiver-only predictor would help.

This also explains why the deployment comparison needs its own evaluation. In the retrospective replay, the same-proposal gate improves over its matched counterpart by +0.324 F1 points, yet its own value is -0.591 [-1.425, 0.042]. We have not established a positive practical payoff for that rule. Its 106 accepted placements out of 432 (Appendix C, Table 6) refer to a later decision at receiving documents, whereas 1/144 refers to optimizer-development selection. These are different stages, and the later acceptance rate does not repair the representativeness problem you identify.

On the mathematical contribution, Equation (2) is indeed an accounting identity. Adding and subtracting the parent score is useful here because it makes explicit what an update contributed beyond the instruction it inherited. Likewise, the local-history calculation in Equations (3)-(4) applies the usual squared-error bias-variance comparison, as Appendix A states. The empirical question is whether paying for a few local observations actually reduces prediction error relative to the available history.

The fully crossed study gives a concrete answer in four execution configurations. With three local observations, pooled-minus-local risk is +0.01655 for Flash with thinking off and -0.00993 with thinking high; Pro shows the same reversal, and all four individual 95% intervals exclude zero (Table 2). The decomposition explains the observed change: the high configurations have more local sampling noise and less pooled prediction error. This is where the calculation helps interpret the evidence. It does not identify a causal effect of thinking or evaluate a cost-optimal acquisition policy.

The extraction study asks a related question about what an aggregate evaluation can justify. Under GLM, the section meets the prespecified +/-1 F1-point replacement criterion while changing the normalized-answer distribution beyond same-text variability (Section 6, Figure 3). Even here, the full skill's improvement over the starting prompt remains unresolved, so the result concerns replacing its score and changing its recorded outcomes. We will revise the opening and contribution statement around these controlled comparisons, describe the main queue as candidate updates on known questions, and make the explanatory role of the elementary identities explicit.

Thanks for your response, which actually acknowledges the weaknesses I listed in my review. Currently I think the paper does not reach the bar of ICLR, so I decide to keep my score.

Reviewer_bW4QOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies what execution records can justify when textual skills are reused or shortened. It distinguishes predicting an update's effect at a receiving document, deciding whether to install it, and retaining particular successes after extracting a shorter section. Matched comparisons find proposal-associated predictive information across receivers on known questions, but do not establish a positive payoff for the primary deployment gate. A separate crossed study shows that the relative value of local trials and pooled history changes across execution configurations. Finally, repeated HotpotQA evaluations show that an extracted Answer Format section can preserve GLM's average F1 while changing answer distributions and exact-match outcomes.

Critical strengths
  • The paper makes a useful distinction between predictive information and deployment value. Matching donor receivers and record counts while excluding all target-pair executions provides a controlled test of proposal-associated information. Reporting that receiver-only and pooled histories achieve lower absolute risk, and that the primary gate has no established positive value, makes the interpretation substantially more credible.

  • The repeated-execution controls strengthen the outcome analysis. Comparing cross-text disagreement against same-text variability addresses whether changing the instruction has an effect beyond ordinary execution noise. Re-executing the full skill on historically selected gain and loss sets also addresses the selection artifact that would arise from comparing fresh section outputs only against previously successful or failed responses. These controls are consequential to the claim that the observed changes reflect the intervention.

Critical weaknesses
  • The main prediction study uses only 16 known questions and 143 development-rejected proposals out of 144. I am not convinced that this setting sufficiently represents reusing successful skills on new tasks.

  • The extraction analysis does not establish how often lost exact matches reflect factual errors rather than formatting changes. This makes the practical significance of the retention result unclear.

Questions for authors
  • Why were 143 of 144 proposals rejected, and were most rejections ties or performance losses?

  • Does the matched predictive advantage reproduce on an independent question panel?

  • What fraction of lost and recovered exact matches corresponds to changes in factual correctness?

2 comments
AuthorsOpenReview ↗

Reviewer bW4Q

The high rejection rate does limit how well the main queue represents a library of successful skills. The 143 rejections occurred during optimizer-development selection: twelve source blocks each generated twelve proposals, using eight training and four development questions per proposal. We retained all 144 candidates for the cross-receiver analysis (Section 3; Appendix F). We cannot yet give you a verified count of how many development rejections were ties and how many were score losses. The evaluation scores cannot supply that missing breakdown, because they measure a different stage on different questions. We will state that gap explicitly and describe the queue as candidate updates rather than selected successful skills.

For your question about an independent panel, no replication of the matched prediction-risk contrast on new questions has been completed. The sixteen questions were held out from optimizer training and development, but each predictor reads the target question's records at other text combinations. This is why the existing result is a test of cross-receiver prediction on known questions.

We did examine whether records from three question categories could support decisions on the fourth. In that internal check, each gate is formed on twelve questions and evaluated on the other four. The same-minus-matched F1 decision-value difference is -0.0438 points, with a descriptive 95% interval [-0.5504, 0.5506] (Appendix D). That result leaves the cross-question decision contrast unresolved. It also answers a different question from the original prediction-risk comparison, using the same sixteen questions, so it cannot serve as the independent replication you ask for. New question records would be needed to repeat the matched prediction test; evaluating deployment would additionally require freezing the gate before assessing those questions.

Your last question matters because an exact-match loss need not be a factual regression. A shorter answer can stop matching the reference while preserving its meaning, and an instruction change can also alter the entity or relation being answered. We inspected fifteen selected response triples together with their evidence to understand such cases (Appendix P, pp. 44-45). For example, the Robert Hay case changes from the oldest part of the castle to the castle itself, whereas other examples involve answer boundaries or wording. These cases help interpret possible changes, but they were purposefully selected and were not a blinded prevalence study. We therefore do not have a validated estimate of the factual-correctness fraction among lost or recovered exact matches. The semantic labels in Appendix R come from a different natural-repair experiment and cannot answer that question for the extraction panel.

The repeated-execution controls address a separate ambiguity: whether apparent losses and recoveries simply reflect rerunning responses selected for their past scores. We reran both the full skill and the section on the frozen historical gain and loss sets. The resulting section-minus-full EM changes are -9.79 and +10.00 points. Their fixed-input execution bounds exclude zero, while the wider question-by-block intervals include it (Appendix M). This supports exact-match changes on the fixed diagnostic inputs without establishing their prevalence across questions or their factual meaning. Similarly, the GLM result in Figure 3 establishes mean-score replacement alongside changed normalized-answer distributions; the distribution test does not label those changes as factual errors.

We will make these distinctions explicit in the main text: what the candidate queue represents, which question information the predictors have seen, and what the extraction outcomes actually measure. In particular, we will use "lost/recovered exact matches" when discussing retention and state that the factual-correctness fraction remains unmeasured. Estimating it would require a systematic sample with an explicit denominator and independently checked, evidence-based labels.

Reviewer_bW4QOpenReview ↗

Response by Reviewer

Thank you for the candid response. I appreciate that it does not overstate what the existing analyses show. However, none of my questions is answered with new evidence:

  • The tie/loss breakdown of the 143 development rejections is not reported. This matters for the deployment results, because the negative always-accept (−2.656) and same-proposal gate (−0.591) values may reflect a queue of mostly non-improving candidates rather than the reuse of successful skills.
  • The lack of an independent question panel is acceptable, since the paper states its known-question scope.
  • The fraction of lost or recovered exact matches that reflect a change in factual correctness remains unmeasured.

The proposed revisions narrow the claims to candidate updates and exact-match retention. This fixes the overclaiming, but it also confirms the gap between the motivating scenario (reusing successful updates) and what is tested. Both the tie/loss count and a systematic labeling of the GLM lost (44) and recovered (41) exact-match cells seem obtainable from existing records. I would reconsider my score if they were provided. For now, I keep my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.