- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper develops a theoretical framework for composing natural-language skill libraries around a frozen, tool-using LLM. It measures performance against a model-realizable, plan-relative optimum, separates within-invocation decisions from cross-invocation artifact propagation, and proposes a decomposition of performance loss into retrieval, selection, interference, tool, and helper-input terms. Its central propagation result combines a sensitivity-weighted sum over DAG paths with an additive rupture-probability budget, yielding contractive, linear, and amplifying regimes. Additional results concern library size, authoring sample complexity, safety under library edits, and dangerous capability combinations. These components are assembled into an agent certificate, with checkpointing and other proposed algorithms intended to improve it.
Critical strengths
- The framework distinguishes several sources of failure that require different interventions. Defining a frozen-model, plan-relative target avoids attributing every failure to skill authoring, while the two-layer model distinguishes tool execution from the effects of intermediate artifacts. This is consequential because it gives a potentially useful vocabulary for diagnosing which failures a library edit or checkpoint could actually address (Sections 3-5).
- The propagation model explicitly accommodates discontinuous failures and heterogeneous dependencies. Per-edge sensitivities and a separate rupture budget are more suitable for schema failures, refusals, and artifact corruption than an exclusively smooth model. Conditional on valid constants, Theorem 1 provides a transparent way to reason about branching pipelines and checkpoint placement. This is the most promising technical contribution.
Critical weaknesses
- The supplementary proofs do not establish the stated compositional safety guarantee. Theorem 4 still reasons that each ruptured skill adds at most c capabilities, although c is defined as a per-action budget. One skill can add different capabilities across several actions, so the required number of distinct ruptured skills need not be ceil(k/c). In supplementary Theorem 9, p_bar = max_{i != j} P[R_i | R_j] is a pairwise quantity, but the chain-rule proof needs bounds conditional on all preceding ruptures. A concrete counterexample uses independent fair bits X and Y, with R1 = {X = 1}, R2 = {Y = 1}, and R3 = {X = Y}. Every pairwise conditional is 1/2, yet P[R1 ∩ R2 ∩ R3] = 1/4 > (1/2)^3. Proposition 16 adds a Markov restriction, but does not repair the unrestricted Theorem 9. These are failures in the assumptions-to-guarantee connection, consequential to the final safety certificate (supplement pp. 5 and 13).
- The expanded experiments support rupture modeling more strongly than conservative certification. I credit the Lipschitz-only comparison, 28/30 perturbation-seed agreement, bootstrap analysis, and three public-library evaluations; these checks are present and relevant. The unresolved issue is whether measured constants upper-bound downstream return loss on new executions. Proposition 5 assumes a uniform proxy-to-return Lipschitz bound, while the live calibration uses fitted sensitivities and caps per-stage ratios at 3 (Appendix G.2); flooring them at 1 does not generally make this downward cap conservative. Lemma 2's proof also needs the gap terms to bound the supremum over downstream continuations uniformly. Appendix G.3 reports robust checkpointed cell verdicts in only 13/30 bootstrap intervals, and 11/15 displayed live bounds saturate at 1. Moreover, J_realizable - J(P) ≤ 1 - accuracy is an upper surrogate for the left-hand side, not evidence that the proposed certificate upper-bounds it. Public-library tests broaden the LLM-conditioning channel, with third-party tools unexecuted. Thus, the additions establish useful mechanisms and broader sources, but do not yet show that non-vacuous performance certificates and checkpoint decisions are reliably conservative.
- Writing quality: frequent defensive statements and excessive emphasis on methodological boundaries. The paper repeatedly reassures the reader that its formulation is non-vacuous, no loss term is omitted, and the results go beyond restating an identity. Examples include “so the split is not vacuous” (Section 3), “not dropped” (Remark 1), and “rather than restating an identity” (Figure 2). Similar assurances that every term is controllable or measurable recur in the abstract and Sections 2, 8, and 11. The cumulative effect is an overly defensive, preemptive-rebuttal style: repeated reassurance draws attention to unresolved qualifications and can create the impression of protesting too much. This weakens the clarity and economy of the exposition. This is the consequence of the author's excessive reliance on GPT writing, and the author should optimize these writing sections to make their paper statements more persuasive, thereby improving the quality of their writing.
Questions for authors
- Is the capability budget c enforced over an entire skill invocation? How should Theorem 4 change if it is per action? For supplementary Theorem 9, is the intended bound over all rupture histories rather than only pairwise conditionals, or is a verified Markov assumption required? Please reconcile the stated guarantee with the counterexample above.
- What establishes a uniform proxy-to-return constant in Proposition 5 and the downstream-continuation bound required by Lemma 2? How are fitted slopes, rupture probabilities, and the LCAP = 3 ratio cap converted into conservative held-out bounds with a stated confidence level?
- Among non-vacuous held-out pipeline bounds, what coverage and useful deploy/reject decisions are obtained when verifier recall and false positives are measured? At comparable verifier cost, how does certificate-guided checkpoint placement compare with uniform placement? The supplied Lipschitz-only comparison already addresses the need for a rupture term.
Response to comment from reviewer 63h9
Respected reviewer 63h9,
Thank you for such a technically detailed review, each issue you raised was useful for us and the counterexample in particular was very helpful.
(1) About Theorem 4 and the capability budget. You are right, the proof uses "each ruptured skill adds ≤c capabilities", while c was defined per action. Revised definition would be c is the maximum number of undeclared capabilities exercisable by one ruptured invocation, enforced by a per-invocation sandbox. With only a per-action budget ca and at most A actions per invocation, the theorem holds verbatim with c:=min(∣C∣,A⋅ca). The order ⌈k/c⌉ is then in this effective c.
(2) About Theorem 9. Your counterexample is correct, pairwise conditionals do not support the chain rule. The fix is to state Theorem 9 with the history conditional (we also mentioned this in general comment as well, kindly check it once)
pˉ=maxσmaxBPr[Rσ⋂σ′∈BRσ′],
which is the definition in main-text Theorem 4 and the one its proof uses. Theorem 4 in the main paper is therefore unaffected, only the appendix restatement was wrong. In your example Pr[R3∣R1,R2]=1, so pˉ=1 and the bound is 1≥1/4. A pairwise quantity suffices only under an explicit 1-Markov rupture dependence (Prop. 16), which we now state as an assumption to be checked, not a default.
(3) About Lemma 2 (uniformity over continuations). We have revised the proof, by the triangle inequality for dA along the idealization ladder,
εj≤∑rungssupκ∈KjJκ(a(r))−Jκ(a(r+1)).
We define gj with these sup rungs, they coincide with the Lemma 1 rungs when Kj is a single continuation. Because Σℓ is finite, Kj is finite, and the authoring bound holds uniformly over Kj at an additive log∣Kj∣ cost.
(4) About Proposition 5 and conservativeness of the live constants. We agree on both counts.
(i) The uniform proxy-to-return constant is an assumption we cannot measure. The live certificates are certificates for drift in the proxy metric, and we now state this.
(ii) The LCAP=3 cap and the use of a pooled median sensitivity were not conservative. Recomputing the original Llama benign pipeline per stage moves its certificate from 0.26 to 0.54, which flips it to REJECT. The new protocol removes both: per-stage envelopes are fitted along the actual trajectory with no cap, and the rupture probability carries a one-sided Clopper–Pearson bound.
(5) About the held out coverage with measured verifier recall/false positives. The checkpoint is now a real node applied to both branches: a gpt-oss-120b verifier that, on NO, calls a repair step. Because repair also runs on the clean branch, its false positives enter the measured loss rather than being assumed away. Measured: the verifier fires on 3.7% (clean branch) / 4.5% (perturbed branch) of calls for gpt-oss-20b and on 0.0% / 1.2% for Qwen, with median contraction 1.0 (245 invocations each, calibration split). Held-out results (general-response table): all four cells hold, path-wise coverage is 0.81–0.96 against guaranteed levels of 0.48–0.84, and held-out violation rates are below Prup in every cell. The one non-trivial certificate (0.73) is sound but about 3.4$\times$ loose.
(6) One more thing we would like to address is about the certificate guided vs uniform placement at equal cost.
Also we agree on the writing part. The revision will remove "so the split is not vacuous", "not dropped", "rather than restating an identity", and the repeated "every term controllable or measurable", replaces them with direct statements, and collects the proved-vs-assumed boundary into one paragraph in Section 3.
Regards, Authors of #13289