acceptodds
Under review as a conference paper at ICLR 2027

ConScribe: Repairing Learned Skills with Residual Updates for Changing Web Environments

Abstract

Environment changes can leave learned skills partly useful and partly invalid. Adapting them requires deciding how new behavior should build on their inherited implementation. We introduce ConScribe, a skill-maintenance framework that generates residual updates and composes them with retained components, including those within the edited scope. Execution checks guide acceptance, wider repair, and reconstruction. On 24 public Mind2Web transfers, initial repair raises element-selection accuracy from 67.97% under static reuse to 75.78%. The last candidate outperforms information-aligned, source-preserving dual path by 9.90 percentage points. Author output decreases by 37.57% on 22 measured transfers, and online wall time decreases by 28.83% on 22 complete-record transfers. A controlled comparison across 48 Chromium transitions holds target AUC at 0.867 while source pass rises from 33.3% under local replacement to 100% under residual composition. Fixed-decomposition experiments also identify conditions where reconstruction costs less than repair. Together, these results show that the form of a skill update matters for target recovery, source retention, and adaptation cost, establishing the value of building target behavior on an inherited skill.

Primary area:
foundation or frontier models, including LLMs
Keywords:
web agents, skill learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MB7qOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ConScribe adapts existing agent skills by generating residual updates rather than replacing the affected implementation or writing a complete target skill. Execution checks guide further repair and eventual reconstruction. On 24 Mind2Web transfers, initial repair improves element-selection accuracy. The paper also reports lower authoring and online adaptation costs, and controlled experiments examining source retention and when reconstruction becomes cheaper than repair.

Critical strengths

The problem is useful, and the distinction between retaining an old skill and actually building new behavior on it is worth studying. The aligned Mind2Web comparison is the strongest evidence: both methods receive the source information, retain a source path, and have comparable qualification opportunities.

Critical weaknesses

1. The contribution beyond a practical patch-generation strategy needs sharper evidence. In the main text-skill implementation, ConScribe asks the model to write only a target-conditioned update and appends it to the source skill. The dual-path baseline instead generates a self-contained target skill. This establishes a useful comparison between two output formats, but leaves open whether an ordinary source-aware patch editor would obtain the same benefits without the proposed composition framework.

A source-ablation test would help identify which retained instructions actually contribute to the accuracy gain. Appendix C.2 correctly identifies this as a counterfactual question, but does not answer it experimentally.

2. The strongest results concern narrow tasks rather than general web-skill maintenance. The main Mind2Web experiment selects an element from 50 candidates given the operation, value, and action history. This is a legitimate component task, not an end-to-end navigation evaluation. The larger controlled studies use deterministic simulators, deterministic browser proposals, or linear scorers. Aligned SkillRevise contains only two lookup-rule changes.

3. The value of the repair controller is not isolated from the value of shorter updates. The aligned comparison provides evidence for residual authoring under a shared adaptation protocol. It does not separately establish the benefit of qualification-driven widening and fallback over simpler policies such as one-shot repair or immediate reconstruction.

4. Experimental reliability and result presentation need improvement. Online wall-time savings cover 22/24 transfers, but total-token savings cover only 12/24 transfers in four website clusters. Excluding unknown usage is appropriate, but without a breakdown of missingness by method or a sensitivity analysis, the token-saving estimate may not represent the full study. The main experiment also uses one model configuration without repeated generation runs. Its cluster intervals capture variation across sampled websites, not variability across repeated authoring and execution. Repeating a representative subset would make the conclusions more convincing.

The reporting also needs correction and clearer presentation. Section 5.4 states that REAL adaptation cost and edit fraction decrease by 43.2%, whereas Table 5 and the per-run figures support approximately 8.3% token savings and 11.0% time savings. And many tables are also poorly organized and presented.

Questions for authors
  1. Does ConScribe outperform a source-aware patch editor with the same evidence, checks, and budget? How much of its target gain disappears when inherited instructions are removed at execution?
  2. How does repair behave when old selectors remain executable but become semantically wrong, or after several successive environment changes?
  3. Do prospective controller comparisons retain the cost advantage when all qualification attempts are charged symmetrically?
7 comments

We thank the reviewer for the constructive review and for recognizing the aligned Mind2Web comparison as the strongest evidence. Unless stated otherwise, numbers are on the 24 aligned Mind2Web transfers (Section 5.2; Appendix A.2).

W1 & Q1: Source ablation and source-aware patch editors.

MethodTarget acc. (%)
Static reuse (source skill unchanged)67.97
Scratch (same target evidence, no source skill)67.71
Dual path (complete target skill)67.71
ConScribe77.60
ConScribe without inherited instructions at execution (source ablation)72.40

Removing the inherited source instructions at execution while keeping the residual lowers target accuracy by 5.20 percentage points (pp). About half of ConScribe's gain over static reuse (+9.63 pp) therefore requires the inherited instructions; the gain does not come from the additional instructions alone.

In the text host, ConScribe's residual is itself a source-aware patch, but a constrained one: it may add or revise target rules, yet every inherited component is retained and the new behavior is attached through an activation condition (Eqs. 6–7). An unconstrained patch editor may instead delete or overwrite inherited content. The effect of this constraint is what our Chromium comparison isolates (Table 3, 48 transitions), with the same failure feedback and the same proposals for both editors:

ChromiumTarget AUCSource pass (%)
Local replacement (edits the inherited locator)0.86733.3
Residual composition (ConScribe)0.867100.0

The Mind2Web results point the same way: a method that rewrites the skill in full retains only 73.44% of the source rules (judged by an independent model), and does not improve over the unchanged skill. The contribution is thus not the patch format itself, but the composition constraint combined with qualification and escalation, applied uniformly across hosts.

W3 & Q3: Value of the controller and symmetric charging.

The aligned study already contains both simpler policies the reviewer mentions. One-shot repair corresponds to stopping at B1, which reaches 75.78%; the full controller, which widens and falls back when qualification fails, reaches 77.60%; immediate reconstruction corresponds to scratch, which reaches 67.71%. Sixteen of the 24 transfers stop at B1, so the controller adds stages only where qualification shows they are needed.

Online costs are charged symmetrically: for both methods, they include every call along the adaptation path, including diagnosis, authoring, qualification, and failed attempts (Appendix A.2). Savings of ConScribe relative to dual path:

Cost measureSaving (%)95% CITransfers covered
Online wall time28.83[25.35, 32.55]22/24
Online attempts23.94[20.92, 27.72]24/24
B1 author output tokens37.57[33.00, 41.00]22/24

In the staged Mind2Web diagnostic (Table 21), the controller is compared with a scratch reference that is charged for authoring only, without qualification calls. This favors scratch, so the reported ratios of 1.661× and 2.376× are upper bounds on the controller's relative cost. We will state this explicitly.

W2 & Q2: Narrow tasks, semantically wrong selectors, and successive changes.

We agree that element selection is a component task rather than end-to-end navigation, and we will state this as a limitation. We chose it because it isolates the grounding step, where website changes most directly invalidate a skill, from planning errors that would otherwise confound the comparison.

When an old selector remains executable but becomes semantically wrong, a lookup-failure activation does not fire, because the old match succeeds. As discussed in Section 4.2 and Appendix C.2, recovery then relies on the task verifier: its failure triggers further edits with a condition that distinguishes the target state, or reconstruction. In the text hosts, the website identifier provides this distinguishing observation, so the residual can condition its target rules on the target website and override inherited rules that are wrong there. We have not quantified this case separately and will list it as a limitation.

The workflow supports successive changes by design (Eq. 8): the adapted skill becomes the inherited skill for the next transition. We have not yet evaluated long sequences of changes; as noted in Section 6, we leave this to future work.

W4: Token coverage, repeated runs, and the 43.2% figure.

Token coverage. Of the 270 recorded attempts, the 14 without reported token usage are exactly the attempts that returned no successful response (e.g., timeouts); 5 belong to dual path and 9 to ConScribe. Because a complete-cost estimate requires both target transitions in a source block to be fully measured, total-token savings cover 12 of 24 transfers. As a conservative sensitivity check, we impute each missing attempt with the mean per-attempt usage of its own method:

Online total tokensSaving vs. dual path (%)Transfers covered
Complete records only (reported)27.2512/24
Missing attempts imputed per method17.4324/24
B1 author total tokens (for reference)33.8922/24

Repeated runs. We agree that the cluster intervals capture variation across websites, not across repeated generation. We therefore ran the full aligned study three times in total with independent generation:

RunGain of ConScribe over dual path (pp)
Original run9.90
Repeat 110.42
Repeat 28.33
Mean ± SD9.55 ± 1.09

The gain remains positive in every run.

The 43.2% figure. We thank the reviewer for catching this error. The 43.2% figure refers to the reduction in edit fraction only. On REAL, adaptation cost decreases by 8.3% in total tokens and 11.0% in wall time (Table 5). We will correct Section 5.4 accordingly and reorganize the tables for readability.

We thank the reviewer again for the careful reading and the concrete suggestions.

Thank you for the detailed rebuttal and the additional experiments.

Two questions remain. The Chromium comparison shows an advantage over locator replacement, while the Mind2Web comparison uses a complete target-skill baseline. It is still useful to show a source-aware patch editor evaluated with the same evidence, checks, and budget. The improvement from the first residual to the final candidate shows that later stages can help, but it remains unclear whether qualification-driven escalation is more cost-effective than simpler policies under the same budget and cost accounting.

Given the stronger evidence, I will raise my score to 2.

Reviewer_RZXrOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ConScribe, a framework for adapting learned agent skills to changing environments through residual updates. It preserves inherited skill components, generates targeted repairs, and uses execution feedback to guide further revision or reconstruction. Experiments across Mind2Web, Chromium, and other settings evaluate target recovery, source retention, and adaptation efficiency. ConScribe improves element-selection accuracy by 9.90 percentage points over a dual-path baseline on Mind2Web, preserves source behavior in controlled Chromium experiments, and reduces adaptation costs in several settings.

Critical strengths

1. Meaningful problem and a unified skill adaptation framework.

The paper addresses the practical problem of maintaining learned skills under environmental changes. ConScribe proposes a framework that preserves inherited skill components, generates residual updates, and uses execution feedback to guide repair or reconstruction. Considering both target recovery and source retention is a useful perspective on skill maintenance (Sections 3–4).

2. Controlled experiments and analysis of adaptation tradeoffs.

The paper includes controlled experiments that help isolate the effects of residual composition. In Chromium, local replacement and residual repair use matched proposals and achieve the same target AUC (0.867), while differing in source retention (33.3% vs. 100%, Table 3). The analysis of repair versus reconstruction also shows that the cost balance depends on retained skill size: at k=4, repair costs more than reconstruction for n=6 (44.0 vs. 39.5 interactions) but less for n=10 (76.0 vs. 81.5) (Section 5.5, Table 8).

Critical weaknesses

1. Limited evidence for generalizable skill repair.

The paper proposes a broad skill-maintenance framework, but the evaluation relies on small, simplified, and heterogeneous studies. The main Mind2Web results evaluate offline element selection on 24 transfers rather than end-to-end execution. Chromium uses a deterministic proposer across only two task families, while the larger controlled experiments rely on synthetic environment changes. SkillEvolBench (Lei et al., 2026) is evaluated on only 12 of its 180 tasks, omitting its frozen deployment tests for context shift and composition, which are particularly relevant to skill generalization. The end-to-end REAL evaluation covers only 10 transfers, with no improvement in target success over dual path (7/10 for both).

More importantly, REAL uses repeated target rollouts with verifier-guided revisions but does not clearly describe independent held-out target evaluation (Tables 6 and 16; Appendices A.2 and B.2). If the same tasks are used for adaptation and evaluation, the results reflect task-specific recovery rather than reusable skill repair. Adaptation also relies on labeled target examples (Mind2Web) or task-verifier feedback (Chromium and REAL), raising questions about the supervision available in deployment. Results in other settings are mixed: target quality decreases on PolySkill (−2.1 pp) and SkillEvolBench (−1.32 points), while staged Mind2Web task transfer underperforms scratch (0.417 vs. 0.500, Table 21). The main model-backed findings are not systematically replicated across models or independent LLM runs. Overall, the experiments provide insufficient evidence that repaired skills generalize to new, complex tasks.

2. Unclear benefits of persistent residual updates over simpler alternatives.

The paper does not sufficiently isolate the contribution of residual composition. The actual repairs vary substantially across hosts, from locator fallbacks in Chromium to target-conditioned Markdown instructions in Mind2Web (Appendix B.2). It remains unclear what reusable knowledge is actually repaired, particularly in REAL, and whether target behavior depends on inherited components rather than simply benefiting from additional instructions (Appendix C.2).

More importantly, the paper lacks a matched runtime adaptation baseline, where an agent uses the inherited skill as guidance and recovers from failures without persistent updates. In Chromium, the same deterministic locator proposer could potentially be invoked at runtime. The practical value of source retention is also unclear: in Mind2Web, both methods preserve the original source path through website-based dispatch and achieve identical source accuracy (59.38%). The main aligned Mind2Web comparison does not report a target scratch baseline, while dual path (67.71%) performs slightly worse than static reuse (67.97%), raising questions about the strength of the reported improvement. The main Mind2Web and REAL results also lack matched no-skill or runtime-retry baselines. Without these comparisons or repeated-use cost analysis, it remains unclear when persistent residual repair offers meaningful advantages over simpler alternatives. Related localized-repair work, such as ContractSkill (Lu et al., 2026), is also not discussed.

Questions for authors
  1. REAL evaluation: Are the target tasks used for verifier-guided repair also used to report final target success, or is there an independent held-out evaluation? What information from the verifier is available during adaptation?

  2. Evaluation scale and generalization: Can the authors provide a substantially larger and more representative end-to-end evaluation, covering diverse tasks and environment changes? Why was only a 12-task subset of SkillEvolBench evaluated, excluding its frozen deployment tests for context shift and skill composition?

  3. Runtime adaptation baselines: How does ConScribe compare against (a) direct no-skill execution and (b) an agent that uses the inherited skill as guidance and recovers from failures at runtime without persistent updates (e.g., runtime locator fallback in Chromium or runtime retry with matched attempts and feedback in REAL)? Under matched information and execution budgets, when does persistent repair become beneficial?

  4. Value of residual repair: Since the source skill can be retained separately, what advantage does residual repair offer over a target skill built from scratch with the same target evidence? Scratch matches or exceeds the residual controller in staged Mind2Web (Table 21), and the main Mind2Web and REAL comparisons report no scratch arm. Do residual-repaired skills generalize better to unseen tasks or reduce cumulative cost under matched budgets?

9 comments

We thank the reviewer for the detailed and constructive review. We group our responses by concern.

Q1 & W1: REAL protocol and held-out evaluation.

Each REAL transition (Garg et al., 2025) adapts the skill on twelve target tasks, with at most three target rollouts of twelve steps each. During adaptation, the verifier returns only a binary pass/fail outcome. After adaptation ends, the final skill is executed once more on these tasks in fresh episodes, without further revision, and this execution determines the reported target success; a transition counts as a success only if all twelve tasks pass. This tests whether the stored skill itself works, not whether the agent eventually recovered during adaptation. We will make this protocol explicit in Appendix B.2.

REAL (10 transitions)Target successJoint (target and source) successAdaptation tokensAdaptation time
Dual path7/101/10referencereference
ConScribe7/102/10−8.3%−11.0%

Since the evaluation tasks are the same as the adaptation tasks, we agree that REAL does not test generalization to new tasks, and we will state this explicitly. Our evidence for generalization comes from the aligned Mind2Web study, where support, qualification, and blind test actions are drawn from disjoint task groups (Appendix A.2), and from the Chromium study, where held-out tests run on a separate server after adaptation (Appendix A.1).

Q3(a) & Q4: No-skill and scratch baselines; W2: do gains depend on inherited components?

On the 24 aligned Mind2Web transfers (Section 5.2), with all methods sharing the same model, prompt template, and candidate sets:

MethodTarget acc. (%)
No skill10.16
No skill + 8 target support examples14.84
Static reuse (source skill unchanged)67.97
Scratch (same target evidence and qualification)67.71
Dual path67.71
ConScribe77.60
ConScribe without inherited instructions at execution (source ablation)72.40

The inherited skill is the main source of target accuracy, and building on it outperforms building a target skill from scratch with the same evidence by about 9.9 percentage points (pp). The source ablation answers W2: removing the inherited source instructions at execution while keeping the residual lowers accuracy by 5.20 pp, so about half of ConScribe's gain over static reuse (+9.63 pp) requires the inherited instructions. The residual and the inherited skill are complementary; the gain does not come from the additional instructions alone. In the staged Mind2Web diagnostic (Table 21), the controller does not beat scratch on task transfer; we report this as a limitation and narrow our claims accordingly

Q3(b): Runtime recovery without persistent updates.

The key difference is cost over repeated use. Runtime recovery repeats the failure-and-search cost on every execution, whereas a persistent residual pays the adaptation cost once, and every later execution runs the repaired skill directly. Persistent repair therefore pays off once its one-time adaptation cost is amortized over reuses, and its advantage grows with the number of reuses.

W2: Value of source retention.

In aligned Mind2Web, we deliberately give both methods the same source path so that the comparison isolates target recovery; identical source accuracy (59.38%) is a property of this design, not a finding. The value of retention is tested in Chromium (Table 3, 48 transitions):

ChromiumTarget AUCSource pass (%)
Static reuse0.333100.0
Scratch0.47533.3
Local replacement0.86733.3
Residual repair (ConScribe)0.867100.0

At the same target AUC, residual repair keeps source behavior that local replacement loses (0% vs. 100% on the 32 changed transitions).

Q2 & W1: Evaluation scope, end-to-end execution, and SkillEvolBench.

Isolating update form requires holding the source skill, target evidence, and qualification fixed across methods, so we evaluate in settings where these can be controlled precisely. Mind2Web element selection is a component task, but it isolates the grounding step, where website changes most directly invalidate a skill, from planning errors that would otherwise confound the comparison. On REAL, target success ties with dual path (7/10 each); the differences are in joint success and cost (see the REAL table in the previous comment), so we do not claim a target-quality gain on REAL. The 12 SkillEvolBench tasks (Lei et al., 2026) are two verifier-scored tasks from each of six task families, used as a component study of authoring cost (Appendix A.2); its frozen deployment tests for context shift and composition evaluate reuse beyond a single adaptation step, a natural extension of this work. We will state the scope of each evaluation explicitly and present broader end-to-end evaluation as future work.

W1: Supervision, mixed results, and replication.

In every comparison, both methods receive identical supervision (labeled target examples or verifier feedback), so the comparisons are fair; we agree that this supervision is an assumption of the setting and will state it explicitly. The negative results on PolySkill, SkillEvolBench, and staged Mind2Web are reported in the main text. We will limit the target-recovery claim to the aligned Mind2Web comparison, presenting lower authoring cost as the finding that holds across hosts and source retention as the finding of the controlled Chromium comparison. For replication, we ran the full aligned study three times in total with independent generation:

RunGain of ConScribe over dual path (pp)
Original run9.90
Repeat 110.42
Repeat 28.33
Mean ± SD9.55 ± 1.09

W2: ContractSkill.

We thank the reviewer for the pointer. ContractSkill (Lu et al., 2026) compiles a draft web-agent skill into an executable artifact with explicit preconditions, steps, postconditions, and recovery rules, and improves it through deterministic verification, fault localization, and minimal local repair. ConScribe shares the goal of localized repair but differs in two ways. First, the setting: ContractSkill refines a newly generated skill in the environment it targets, whereas ConScribe adapts an already validated skill after its environment changes, where the skill should keep working in the source environment. Second, the form of the update: rather than editing the faulty part, ConScribe keeps the inherited component, appends a residual with an activation condition, and checks source retention, which is exactly the difference isolated in the Chromium table above. The two are complementary: ContractSkill's explicit step structure could serve as the segmentation that ConScribe's host interface requires. We will add this discussion to the related work.

Reference: Lu et al. (2026). ContractSkill: Repairable Contract-Based Skills for Multimodal Web Agents. arXiv:2603.20340.

Thank you for the additional experiments and detailed response. The scratch comparison, source ablation, and repeated runs address some of the earlier concerns. However, the following concerns remain:

  1. Generalization (W1, Q1): The authors confirm that REAL uses the same tasks for adaptation and evaluation. Evaluating the same tasks in fresh episodes does not establish generalization to unseen tasks. The main Mind2Web results evaluate offline element selection rather than end-to-end agent execution. The paper still lacks convincing evidence that residual-repaired skills generalize to new, complex tasks.

  2. Evaluation scale (W1, Q2): The main Mind2Web study covers 24 transfers, the end-to-end REAL study only 10 transitions, Chromium uses a deterministic proposer across two task families, SkillRevise uses two synthetic lookup shifts, and SkillEvolBench is evaluated on just 12 tasks. The authors acknowledge these limitations and defer broader evaluation to future work, but the existing experiments remain too small and simplified to support the proposed general framework.

  3. Methodological contribution (W2): The main Mind2Web implementation appends LLM-generated instructions to an inherited skill, while Chromium adds locator fallbacks. The source ablation shows that inherited instructions contribute to target accuracy, but it remains unclear what generalizable procedural knowledge is being repaired or what broader mechanism the framework establishes beyond these relatively simple modifications.

  4. Practical advantage (W2, Q3): The authors argue that persistent repair amortizes its cost over repeated executions, but provide no matched runtime-recovery baseline or break-even analysis. The practical advantage over simpler alternatives therefore remains unsubstantiated.

New questions:

  1. The new no-skill accuracy (10.16%) is surprisingly low for GPT-5.5 selecting among 50 candidates given the operation, value, and action history, only modestly above the 2% random baseline. Adding eight labeled target examples in context raises it only to 14.84%, whereas a skill that the same model distilled from eight labeled examples on a different website reaches 67.97% without modification. Since distillation adds no new information and the source examples are less relevant to the target site, this 53-point gap suggests the arms may differ in task description, candidate representation, or output-format instructions rather than in site-specific knowledge. Could the authors share the execution-time prompts for each arm, confirm whether they are identical apart from the skill text, clarify how the target examples are provided, and report whether answer-parsing failures affect these results?

  2. Could the authors provide representative Mind2Web source skills and generated residuals to illustrate what information is actually reused or repaired?

  3. Could the authors clarify how the 9–39 calls per REAL transition in Table 16 are counted and how they relate to the stated protocol of 12 target tasks with up to three rollouts of twelve steps each?

Since my main concerns have not been addressed, I maintain my rating.

Reviewer_pjvPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies what to do with a skill an agent has learned (a browser action sequence, a text rule file, a library entry) when its environment changes and part of the skill stops working. It contrasts several update forms: reuse the skill unchanged; edit it in place; keep it and write a complete new skill for the target (the paper calls this the dual path); keep it and append an increment with an activation condition (the paper calls this a residual); or rebuild from scratch. The thesis is that the appended-increment form recovers target behavior better, retains source behavior, and costs less than the alternatives.

The method is a workflow around that form. A host exposes four operations (segment, execute, edit, scratch). The workflow executes the inherited skill on the target, localizes the failing unit, asks the host to propose an increment, composes it so that everything outside the edited scope is unchanged and the inherited components inside it are kept, checks the result on a few target tasks and optionally on source tasks, and widens the scope or rebuilds when checks fail. In the browser host, the increment is an alternative locator placed behind the original. In the text hosts the increment is a block of rules appended to the source text, so localization and scope reduce to how much evidence the author sees.

The evidence is spread over six studies, and every transition in them is constructed by the authors: pairs of different websites drawn from Mind2Web records and cast as a transfer, author-written pages with injected locator renames, a simulator with seven mutation families, and a synthetic lookup-code shift. The quality claim rests on one study: 24 cross-website Mind2Web transfers, where the appended-increment method reaches 77.60% element-selection accuracy against 67.71% for a method that sees the same source skill and target examples but must write a complete target skill. The unchanged source skill scores 67.97%, so the comparator ends where it started. A second Mind2Web protocol with a model-backed author finds the increment method below scratch at equal evidence and more expensive over the full path; the other hosts tie or come out slightly lower on quality, with consistent savings in authored text. A Chromium study with a rule-based proposer shows that keeping the original locator preserves source behavior where in-place replacement does not, and a cost crossover between repair and rebuild measured at retrospectively selected checkpoints. The result that holds across hosts is that authoring an increment costs less than authoring a full skill.

Critical strengths
  1. The proposed problem of updating skills in dynamic environments is realistic, and treating the form of a skill update as the variable under study is a useful move. This paper lays out the options (reuse, in-place edit, a separate complete target skill, an appended increment with an activation condition, rebuild), the four quantities an update affects (target recovery, source retention, lifecycle cost, edit magnitude), and the four host operations an adaptation layer needs (segment, execute, edit, scratch), and it realizes the same contract in four representations. The browser realization is concrete and inspectable.

  2. The experiment details are well included, and adverse results and accounting details are reported in the main text. Section 5.4 reports the PolySkill target decrease and the lower SkillEvolBench verifier score; Section 5.5 reports that the staged controller costs 1.661 to 2.376 times scratch authoring and that task-transfer quality falls below reference.

Critical weaknesses
  1. The headline comparison does not control or verify that the complete-generation method realizes the behavior it stands for. The Dual Path method is meant to author a complete target skill from the source skill and target evidence, but whether its skills use the source rules is not reported, and the qualification does not show it.

  2. The constructed transitions are not verified to realize the problem the paper studies: neither that the inherited skill remains partly valid on them nor that their changes correspond to observed ones. Every transition is author-made: pairs of different Mind2Web websites, author-written pages with injected locator renames repaired by a proposer that ranks alternatives by lexical overlap, a sealed simulator with seven transition families, and two lookup-code shifts. The premise that a change leaves the skill "partly useful and partly invalid" is measured on none of them, with no no-skill accuracy being reported.

  3. The claim that update form determines target recovery is supported by one of the six comparisons with model-backed edits. Aligned Mind2Web is the one. The remaining positive results come from the deterministic proposer, the simulator, and the linear scorer on constructed transitions, where repair starts from a mostly correct skill and the comparison is made at fixed budgets or integrated over them. The abstract states that update form matters for target recovery, and Section 6 cites only the aligned study.

  4. The reported repair saving is not the cost of the policy that was run, and that cost is not reported. The 23.7% interaction saving is computed at a retrospectively selected checkpoint. Figure 3's caption calls it retrospective, but the main text describes it as the cost "through the first held-out target pass" without saying that the held-out tests did not guide the run. This is concerning because contribution 3 is stated as an empirical account of when repair is cheaper than reconstruction.

Questions for authors
  1. What target accuracy does the model reach on the 24 transfers with no skill, and with the eight target support examples but no skill? Did the dual-path skills reproduce the source rules, and how was this checked?
  2. Which observed changes to real websites do the seven transition families and the Chromium locator renames correspond to, and on what basis were their proportions chosen? On the Mind2Web pairs, how much of the inherited rule file is valid on the target site before adaptation?
  3. What are the per-cell interaction costs of Appendix Table 8 when each run stops at its calibration success rather than at the retrospectively selected checkpoint?
5 comments

We thank the reviewer for the careful reading and for recognizing the value of treating update form as the variable of study. Unless stated otherwise, all numbers below are on the 24 aligned Mind2Web transfers (Section 5.2; Appendix A.2), built from original Mind2Web records (Deng et al., 2023).

W1 & Q1(b): Is dual path a faithful baseline?

The aligned protocol is designed so that dual path and ConScribe differ only in the form of the update. Both receive the same source skill, target support examples, and environment-rule requirements, and both receive the same additional evidence when widening or reconstruction is triggered. Both retain the original source path and use the same website-identifier dispatch, so their source accuracy is identical by construction (59.38%). Both face the same target and source qualification criteria and terminate at B1/B2/B3 on 16/7/1 transitions. The only difference is the output: dual path writes a complete target skill, while ConScribe writes a residual appended to the source skill.

Both methods also receive the same author instruction (Appendix C.1, Listing 3), which supplies the source skill as common prior information and asks the author to "infer and preserve reusable environment-conditioned rules supported by the evidence." To check whether dual-path skills actually reproduce the source rules, we segmented each source skill into individual rules and asked an independent judge model (DeepSeek-Flash, 0920 version), distinct from the GPT-5.5 author model, whether each rule has a semantically equivalent counterpart in the corresponding dual-path skill. On average, 73.44% of source rules are retained (95% CI [68.80, 77.61]). Dual path therefore carries over most of the source knowledge, yet regenerating a complete skill yields no target gain over the unchanged skill, whereas appending a residual does (table below). We will add this analysis to the revision.

Q1(a): No-skill baselines.

MethodTarget acc. (%)
No skill10.16
No skill + 8 target support examples14.84
Static reuse (source skill unchanged)67.97
Dual path (complete target skill)67.71
Scratch (same target evidence, no source skill)67.71
ConScribe, first update (B1)75.78
ConScribe, last candidate77.60

Both no-skill baselines use the same prompt template, top-50 candidate sets, and GPT-5.5 model as all other methods, with only the skill text removed. Static reuse exceeds them by 57.81 and 53.13 percentage points (pp), showing that the inherited skill remains largely useful on the target. ConScribe's further gain over static reuse (+7.81 pp at B1) shows that the skill is also partly invalid. Together, these two gaps provide direct evidence for the "partly useful, partly invalid" premise on the Mind2Web transfers.

W2 & Q2: Constructed transitions.

The 24 aligned Mind2Web transfers are pairs of real, functionally matched websites from the original Mind2Web records; they involve no injected changes. The seven simulator families (Appendix A.1) model common categories of real interface change:

  • no-impact changes: cosmetic or layout updates that leave the skill's steps intact;
  • surface-token changes: relabeled buttons or fields;
  • path rebinding: a feature moved to another menu or route;
  • control substitution: e.g., a dropdown replaced by an autocomplete field;
  • new required steps: e.g., a consent dialog or a new mandatory field;
  • workflow reordering: the same steps in a different order;
  • capability deletion: a retired feature, where the correct outcome is rejection.

The Chromium transitions change the accessible names of controls (e.g., "Save Contact" → "Create Contact"; Appendix E.1), which corresponds to routine UI copy changes. The family proportions come from a balanced factorial design (12 seeds × 3 skill lengths × 7 families × 2 transfer types, i.e., 72 transitions per family). They are not calibrated to real-world change frequencies, and we will state this explicitly as a limitation.

W4 & Q3: Correction to the cost description.

We thank the reviewer for scrutinizing this point, which led us to find an error in our description, not in the reported numbers. The 23.7% saving and the per-cell costs in Table 8 are the costs of the executed runs. Each run stops as soon as a calibration execution succeeds and its task verifier passes, or when the budget is exhausted, and the reported cost counts all calibration episodes, source probes, and page actions up to that point. Held-out tests are run only after adaptation, on a separate server created after the calibration server is closed, and never guide adaptation or stopping (Appendix A.1).

The word "retrospective" in the captions of Figure 3 and Table 8, and the corresponding description in the appendix, are therefore incorrect. Recomputing the costs from the saved trajectories under the calibration stopping rule reproduces the reported values (46.17 vs. 60.50 interactions on average for repair vs. scratch). Regarding Q3, the per-cell costs in Table 8 are thus already those under calibration stopping. We will correct these descriptions in the revision.

W3: Scope of claims.

We agree. We will revise the abstract, Contribution 2, and Section 6 to state that (i) the target-recovery gain is established in the aligned Mind2Web comparison; (ii) lower authoring cost holds across all hosts; and (iii) source retention at matched target recovery is shown in the controlled Chromium comparison (Table 3).

We thank the reviewer again for these constructive comments, which will substantially sharpen the paper.

Dear Reviewer pjvP, thank you again for your time and review. We hope our responses have addressed your main concerns: whether dual path actually uses the source rules, whether the constructed transitions reflect the problem studied, the scope of the claim on update form, and how the repair cost was measured. We would be happy to clarify anything further, and if our responses have resolved these concerns, we would appreciate it if you could take them into account in your final assessment.

Comments on the paper

1 comment

Hi reviewers, as we approach the rebuttal deadline, please review the authors' responses and participate in the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.