- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
ASSAY measures how including individual instructions affects an agent's task success under randomized skill masks. The resulting task-indexed scores guide offline library restructuring and task-dependent filtering at inference. For a new task, the method combines scores from similar development tasks rather than observing the new task's reward. The evaluation covers seven models on AppWorld and τ-bench retail, with detailed controls on GPT-5.1/AppWorld.
Critical strengths
The paper identifies a useful limitation of accumulated skill libraries: a generally relevant instruction can still impair execution on a particular task. Measuring outcomes provides a concrete basis for deciding which instructions to retain. The estimator has a clear interpretation under the specified mask distribution, and the paper distinguishes it from both Shapley attribution and full-library leave-one-out effects.
The matched controls are persuasive. On GPT-5.1/AppWorld test_normal, ASSAY reaches 77.4% versus 70.8% for ACE with the same five templates. On the same restructured library and with matched removal counts, random masking reaches 70.0%, global-score masking 73.2%, and task-conditioned masking 77.4% (Appendix F). These comparisons support a contribution beyond supplying templates or merely shortening the prompt. Fresh masks and three resampled development sets provide further evidence that the result is not peculiar to one attribution run. I also appreciate the explicit reporting of noisy cell estimates and unchanged τ-bench results in two model settings.
Critical weaknesses
-
The contribution of library repair is not separated from the contribution of selection. ACE with the same templates reaches 70.8% (Table 5), while the restructured library without masking reaches 69.6% (Table 6). This does not show that repair is harmful: restructuring may make subsequent selection more effective. But the comparison needed to establish that interaction is missing—task-conditioned masking applied directly to the original ACE-plus-template library. Figure 1 demonstrates such masking for one task, which reinforces the relevance of this control without answering it at dataset level. The current evidence supports the selection component more clearly than it supports the full measurement-and-repair pipeline.
-
The uncertainty of the key task-conditioned versus global-score comparison is not directly reported. Table 7 differs by seven successes out of 168, but the paired disagreement counts are not given. The McNemar test in Appendix I.1 compares two ASSAY runs and answers a different question. The development-set repetitions are reassuring; paired uncertainty for the existing matched-control outcomes would show how strongly the main result supports task-dependent selection on this evaluation set.
Questions for authors
-
What is the result of applying the same task-conditioned selection procedure to ACE plus the same templates, without split/retire/merge? Comparing this with the full pipeline would clarify whether repair improves the final decisions.
-
Is there evidence that negatively scored skills also hurt when evaluated against a background closer to deployment, with protected entries always present? A limited check on selected skills would help connect the measured effect to the removal decision.
-
Could you report the paired discordant counts or a paired confidence interval for ASSAY versus global-score masking, and for ASSAY versus ACE plus templates? Those comparisons would be more informative for the main claims than another test of agreement between ASSAY runs.
3 comments
I appreciate the authors’ response and their answers to my questions. However, I still believe my initial overall rating reflects my assessment of the paper, and therefore I would like to keep it.
Clarification on Remaining Concerns
Thank you very much for following up and for considering our response! We would appreciate any further guidance on concerns that remain after the additional experiments. We have added the requested original-library selection control, paired uncertainty analyses, and deployment-background interventions. In particular, the new control shows a +5.36 pp benefit from repair with selection held fixed (95% CI: [1.19, 9.52]). Could you help us understand where you find the evidence still insufficient? This would help us focus our remaining revisions. Thank you again for your time and feedback!
To Reviewer BqPx: Additional Evidence on Repair, Selection, and Paired Outcomes
Thank you for identifying the missing control needed to establish the contribution of repair! We have completed this experiment and the paired analyses you requested.
1. Repair improves upon direct selection on the original library.
We apply the same attribution-guided selection procedure directly to ACE with identical templates, preserving the original attribution rows, selection hyperparameters, and fallback rule.
Full ASSAY solves nine additional tasks relative to direct selection: 12 ASSAY-only versus three direct-selection-only successes. The improvement is +5.36 percentage points, with a scenario-clustered 95% confidence interval of [+1.19, +9.52].
This fills the missing experimental condition: with the selection procedure held fixed, restructuring improves the resulting pipeline. We distinguish this incremental contribution from the repair–selection interaction, whose estimated effect is positive but whose interval includes zero.
2. Paired uncertainty for the principal comparisons.
These intervals resample complete scenarios, retaining paired outcomes and dependencies among task variants. They directly quantify uncertainty for the requested comparisons, rather than agreement between two ASSAY runs.
3. Negative scores predict harmful effects under deployment backgrounds.
We performed single-skill inclusion/removal interventions on the same 40 eligible tasks, keeping all other deployed instructions fixed and protected entries present. Removing strongly negative, near-threshold, and nonnegative skills yields net benefits of +25, 0, and −15 percentage points, respectively.
The within-task negative-versus-nonnegative contrast is +40 points [95% scenario-clustered CI: +12.20, +69.05]. Because each task contributes all three interventions, this comparison controls for differences in task composition. The result applies to the audited population satisfying all three score-stratum requirements.
Together, these experiments supply the missing evidence for repair’s incremental contribution and directly connect attribution predictions to execution consequences under deployment conditions. Thank you again!