- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ConScribe adapts existing agent skills by generating residual updates rather than replacing the affected implementation or writing a complete target skill. Execution checks guide further repair and eventual reconstruction. On 24 Mind2Web transfers, initial repair improves element-selection accuracy. The paper also reports lower authoring and online adaptation costs, and controlled experiments examining source retention and when reconstruction becomes cheaper than repair.
Critical strengths
The problem is useful, and the distinction between retaining an old skill and actually building new behavior on it is worth studying. The aligned Mind2Web comparison is the strongest evidence: both methods receive the source information, retain a source path, and have comparable qualification opportunities.
Critical weaknesses
1. The contribution beyond a practical patch-generation strategy needs sharper evidence. In the main text-skill implementation, ConScribe asks the model to write only a target-conditioned update and appends it to the source skill. The dual-path baseline instead generates a self-contained target skill. This establishes a useful comparison between two output formats, but leaves open whether an ordinary source-aware patch editor would obtain the same benefits without the proposed composition framework.
A source-ablation test would help identify which retained instructions actually contribute to the accuracy gain. Appendix C.2 correctly identifies this as a counterfactual question, but does not answer it experimentally.
2. The strongest results concern narrow tasks rather than general web-skill maintenance. The main Mind2Web experiment selects an element from 50 candidates given the operation, value, and action history. This is a legitimate component task, not an end-to-end navigation evaluation. The larger controlled studies use deterministic simulators, deterministic browser proposals, or linear scorers. Aligned SkillRevise contains only two lookup-rule changes.
3. The value of the repair controller is not isolated from the value of shorter updates. The aligned comparison provides evidence for residual authoring under a shared adaptation protocol. It does not separately establish the benefit of qualification-driven widening and fallback over simpler policies such as one-shot repair or immediate reconstruction.
4. Experimental reliability and result presentation need improvement. Online wall-time savings cover 22/24 transfers, but total-token savings cover only 12/24 transfers in four website clusters. Excluding unknown usage is appropriate, but without a breakdown of missingness by method or a sensitivity analysis, the token-saving estimate may not represent the full study. The main experiment also uses one model configuration without repeated generation runs. Its cluster intervals capture variation across sampled websites, not variability across repeated authoring and execution. Repeating a representative subset would make the conclusions more convincing.
The reporting also needs correction and clearer presentation. Section 5.4 states that REAL adaptation cost and edit fraction decrease by 43.2%, whereas Table 5 and the per-run figures support approximately 8.3% token savings and 11.0% time savings. And many tables are also poorly organized and presented.
Questions for authors
- Does ConScribe outperform a source-aware patch editor with the same evidence, checks, and budget? How much of its target gain disappears when inherited instructions are removed at execution?
- How does repair behave when old selectors remain executable but become semantically wrong, or after several successive environment changes?
- Do prospective controller comparisons retain the cost advantage when all qualification attempts are charged symmetrically?
7 comments
W2 & Q2: Narrow tasks, semantically wrong selectors, and successive changes.
We agree that element selection is a component task rather than end-to-end navigation, and we will state this as a limitation. We chose it because it isolates the grounding step, where website changes most directly invalidate a skill, from planning errors that would otherwise confound the comparison.
When an old selector remains executable but becomes semantically wrong, a lookup-failure activation does not fire, because the old match succeeds. As discussed in Section 4.2 and Appendix C.2, recovery then relies on the task verifier: its failure triggers further edits with a condition that distinguishes the target state, or reconstruction. In the text hosts, the website identifier provides this distinguishing observation, so the residual can condition its target rules on the target website and override inherited rules that are wrong there. We have not quantified this case separately and will list it as a limitation.
The workflow supports successive changes by design (Eq. 8): the adapted skill becomes the inherited skill for the next transition. We have not yet evaluated long sequences of changes; as noted in Section 6, we leave this to future work.
W4: Token coverage, repeated runs, and the 43.2% figure.
Token coverage. Of the 270 recorded attempts, the 14 without reported token usage are exactly the attempts that returned no successful response (e.g., timeouts); 5 belong to dual path and 9 to ConScribe. Because a complete-cost estimate requires both target transitions in a source block to be fully measured, total-token savings cover 12 of 24 transfers. As a conservative sensitivity check, we impute each missing attempt with the mean per-attempt usage of its own method:
| Online total tokens | Saving vs. dual path (%) | Transfers covered |
|---|---|---|
| Complete records only (reported) | 27.25 | 12/24 |
| Missing attempts imputed per method | 17.43 | 24/24 |
| B1 author total tokens (for reference) | 33.89 | 22/24 |
Repeated runs. We agree that the cluster intervals capture variation across websites, not across repeated generation. We therefore ran the full aligned study three times in total with independent generation:
| Run | Gain of ConScribe over dual path (pp) |
|---|---|
| Original run | 9.90 |
| Repeat 1 | 10.42 |
| Repeat 2 | 8.33 |
| Mean ± SD | 9.55 ± 1.09 |
The gain remains positive in every run.
The 43.2% figure. We thank the reviewer for catching this error. The 43.2% figure refers to the reduction in edit fraction only. On REAL, adaptation cost decreases by 8.3% in total tokens and 11.0% in wall time (Table 5). We will correct Section 5.4 accordingly and reorganize the tables for readability.
We thank the reviewer again for the careful reading and the concrete suggestions.
Thank you for the detailed rebuttal and the additional experiments.
Two questions remain. The Chromium comparison shows an advantage over locator replacement, while the Mind2Web comparison uses a complete target-skill baseline. It is still useful to show a source-aware patch editor evaluated with the same evidence, checks, and budget. The improvement from the first residual to the final candidate shows that later stages can help, but it remains unclear whether qualification-driven escalation is more cost-effective than simpler policies under the same budget and cost accounting.
Given the stronger evidence, I will raise my score to 2.
We thank the reviewer for the constructive review and for recognizing the aligned Mind2Web comparison as the strongest evidence. Unless stated otherwise, numbers are on the 24 aligned Mind2Web transfers (Section 5.2; Appendix A.2).
W1 & Q1: Source ablation and source-aware patch editors.
Removing the inherited source instructions at execution while keeping the residual lowers target accuracy by 5.20 percentage points (pp). About half of ConScribe's gain over static reuse (+9.63 pp) therefore requires the inherited instructions; the gain does not come from the additional instructions alone.
In the text host, ConScribe's residual is itself a source-aware patch, but a constrained one: it may add or revise target rules, yet every inherited component is retained and the new behavior is attached through an activation condition (Eqs. 6–7). An unconstrained patch editor may instead delete or overwrite inherited content. The effect of this constraint is what our Chromium comparison isolates (Table 3, 48 transitions), with the same failure feedback and the same proposals for both editors:
The Mind2Web results point the same way: a method that rewrites the skill in full retains only 73.44% of the source rules (judged by an independent model), and does not improve over the unchanged skill. The contribution is thus not the patch format itself, but the composition constraint combined with qualification and escalation, applied uniformly across hosts.
W3 & Q3: Value of the controller and symmetric charging.
The aligned study already contains both simpler policies the reviewer mentions. One-shot repair corresponds to stopping at B1, which reaches 75.78%; the full controller, which widens and falls back when qualification fails, reaches 77.60%; immediate reconstruction corresponds to scratch, which reaches 67.71%. Sixteen of the 24 transfers stop at B1, so the controller adds stages only where qualification shows they are needed.
Online costs are charged symmetrically: for both methods, they include every call along the adaptation path, including diagnosis, authoring, qualification, and failed attempts (Appendix A.2). Savings of ConScribe relative to dual path:
In the staged Mind2Web diagnostic (Table 21), the controller is compared with a scratch reference that is charged for authoring only, without qualification calls. This favors scratch, so the reported ratios of 1.661× and 2.376× are upper bounds on the controller's relative cost. We will state this explicitly.