- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper tries to answer the question: When Can an AI Skill Library Trust Its Past Results? They split the topic into three (1) Predict i.e how well a new instruction will work in a new setting. (2) Deployment i.e. to decide to deploy an update. and (3) Retention/Compress i.e. shortening a long instruction. They show that: For Predict: Past results give a good idea, but do not guarantee success. For Deployment: Local test are useful when the AI system has high output variation, but past history is better otherwise. For retention/compression: shortening instructions changes the answers and some fixed mistakes may come back.
Critical strengths
A good modular framework: separating prediction, deployment, and retention/compression. Theoretical Grounding: which is nice to have - but to be honest I did not proof check. Some degree of integrity: reporting some minor negative results (in particular a result worse than baseline) Multiple AI models were tested.
Critical weaknesses
Paper is difficult to read - at least if not in the nichc (?) domain. Experimentation is weak: all based on Short QA Benchmark with everything associated (and even pointed out by the authors in the annex): for instance high sensitivity to phrasing. Some conclusion are a bit to strong: for instance authors claim that deployment gates in general fail to create value. But the reality is that they tested a "naive" zero-threshold deployment gate and this failed to yield positive value.
Questions for authors
None
2 comments
Thank you for your comments. Unfortunately, I stick to my rating.
Reviewer K5vH
We can see why the gate result reads as a broader claim about deployment than the experiment can support. We evaluated one fixed rule: install an update when its predicted effect, averaged over the sixteen questions, is positive. The resulting value is -0.591 F1 points relative to rejecting every update, with a descriptive 95% interval [-1.425, 0.042]. This leaves the payoff of that rule unresolved. It cannot tell us whether deployment gates are useful in general.
What the comparison does show is that changing the history used by this rule improves its value by +0.324 points [0.013, 0.717] relative to matched other-proposal history (Section 3; Appendix C). A record can therefore help this particular decision without yet making it a demonstrably beneficial one. We should have kept that qualification close to the deployment claim throughout the introduction, rather than leaving the reader to recover it from the detailed results.
Your concern about short-answer QA also affects how broadly these findings should be read. We used this setting because it lets us hold the question and installed texts fixed while changing which records a predictor can consult. Matching the donor receivers and record count then gives a controlled test of proposal-associated information. The sixteen evaluation questions are held out from optimizer training and development, but their records at other text combinations are visible to the predictors. The finding consequently concerns cross-receiver prediction on known questions. It does not establish reuse on new tasks or long-horizon agent workflows.
For the extraction study, phrasing is part of the difficulty you identify. Two answers may differ after normalization even when the difference has little practical consequence. Section 6 therefore compares cross-text disagreement with the variability from rerunning the same text, and Appendix M.4 checks sensitivity across four scoring interfaces. These controls tell us more than a single-run score comparison, but they still do not give the factual-error rate among changed answers. The 128-question HotpotQA extraction panel supports a finding about mean-score replacement and answer changes within that setting.
The presentation would be easier to follow if we introduced the decisions through a concrete example. Suppose a library adds an evidence-checking instruction to a receiving document. Its history might help forecast the score change; the library still has to decide whether installing it is worthwhile; and shortening it raises the further question of which particular answers survive. We will use this example to explain the three targets before introducing the notation, define the receiving document and its comparator at first use, and give each main study a short explanation of the question it answers. We will also state the fixed-rule scope beside the gate result and describe extraction outcomes as lost or recovered exact matches, keeping their factual interpretation separate.