acceptodds
Withdrawn from ICLR 2027

TrajSmith: Reusable Web Tools from Trajectories through Exploration and Verification

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Web agents repeatedly plan browser procedures that earlier trajectories already contain. This repetition increases inference cost and exposes smaller models to errors over long action sequences. Converting trajectories into reusable tools can reduce these demands, but a working source trajectory does not establish when the resulting tool will apply on a changing website. We present TrajSmith, a trajectory-based method for synthesizing web tools with explicit reuse conditions. A strong model actively explores the structure around a successful or failed trajectory, inspects related controls and input options, and compiles the observations into atomic and composite tools. An applicability stage challenges candidate tools with changed arguments, boundary cases, neighboring intents, and empty-result queries. Structural checks and an expected result type screen the returned evidence, and a Tool-Use Contract records argument domains, use conditions, and rejection rules. At deployment, a weak model reads the contract to select a tool and fill its arguments; contract checks determine whether the proposal proceeds to execution or fallback. On a 159-task Online-Mind2Web subset selected for tool coverage, the better of two paired runs raises Qwen3-4B success from 18.2% to 30.8% while using fewer recorded agent-loop steps. On WebArena's 763 single-domain tasks, success increases from 19.8% to 24.5%. Additional shared-budget comparisons show gains on source-overlapping tasks for GPT-4o-mini and Qwen3-VL-4B. These results support reuse of recorded procedures across repeated runs and several weak-model configurations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Web agents, trajectory reuse, web tool synthesis, strong-to-weak transfer, browser automation, Tool-Use Contracts

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Cp2XOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

TrajSmith uses strong models to probe the webpage structures via the recorded web-agent trajectories and turn them into atomic and composite tools. It then tests and validates each tool with unseen argument values, out-of-scope intents, boundary values and empty-result queries, and records the outcome. At deployment, a weak model reads the contracts to pick a tool and fill its arguments, and a programmatic gate decides whether to run the tool or fall back to ordinary browsing. The method is tested on two benchmarks. On a 159-task Online-Mind2Web subset selected for tool coverage, the better of two paired runs raises Qwen3-4B success from 18.2% to 30.8% while using fewer recorded agent-loop steps. On WebArena’s 763 single-domain tasks, success increases from 19.8% to 24.5%.

Critical strengths

A clear way for a small model to use tools built by a large model. Each tool comes with a short description, which the paper calls a contract. It tells the small model when the tool applies; which argument values the tool has been tested with; which similar-looking requests it should refuse; and what a correct result looks like. The paper gives enough detail to rebuild the system.

Critical weaknesses
  • The evaluation does not test the paper's central claim that tools remain reusable beyond the instance they were built from. 155 of the 159 tasks from the Online-Mind2Web are used for both evaluation and tool synthesis (Sec. 4.1, Table 12). So it's more about testing the weak model's tool-call capability, not the tool's reusability. To my understanding, reusable tools mean that the agent can use a set of the same tools with different parameter values to solve similar tasks, rather than the tasks from which the tools are derived. Also, according to Table 10, 787 of the 896 tools are in the paper’s fixed tier. The paper describes these as parameterless tools that receive only “lighter replay checks”, which hints at a potential overfitting on the environment or the tasks used to construct the tools.

  • The paper claims two technical contributions. The first is exploring the page around each recorded run to build better tools. The second is testing each tool and writing down when it should and should not be used (the contract). The paper’s own experiments do not show that either one helps.

    • Removing the new parts does not hurt. The authors rebuilt the library with no exploration and no testing. It solved exactly as many tasks as the full system: 18 of 58 (Table 18).

    • Removing or loosening several other parts of TrajSmith's design changes the solved tasks mostly by only one or two, and 2 out of 5 changes in TrajSmith's design can actually hurt performance according to Table 5.

    • The authors themselves write that the contributions of exploration and verification "remain unresolved" (Line 422–423).

    • When a tool is used, and the task still fails, we don't know why. The cause could be any of three things: 1) the small model picked the wrong tool; 2) the small model filled in the wrong parameter values; 3) the tool itself was broken.

  • The comparisons with other methods are not fair or are missing.

    • ContractSkill is the only other method compared. Its skills were built by the small Qwen3-4B model while solving each task (Appendix C.11, Line 1224-1225). TrajSmith's tools were built in advance by Claude Opus 4.6, from the same tasks (Appendix C.11, Line 1225-1226). So the 6-vs-18 result in Table 4 does not support the claim.

    • WALT and SkillWeaver (as cited in the paper) are very related earlier method, but it is not compared. The argument for not using these two methods is "archived WALT and SkillWeaver records use different task sets or incomplete protocols" (Line 412-413). I do not understand the explanation nor see it as convincing.

Questions for authors

Please see the critical weaknesses section. Those are also my questions.

Reviewer_dUxCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TrajSmith, a method for extracting reusable, site-specific tools from web-agent trajectories through structural exploration and applicability verification. Experiments on Online-Mind2Web and WebArena show that these tools improve task success rates for weaker models.

Critical strengths

The idea of extracting reusable tools from web-agent trajectories is practically motivated and could reduce repeated planning for weaker agents.

Critical weaknesses
  1. The performance gains from replacing low-level browser actions with reusable tools are unsurprising. Comparisons are limited to adapted ContractSkill, while relevant approaches such as WALT and SkillWeaver, both discussed in related work, are not evaluated under matched conditions. This leaves TrajSmith’s advantage over existing tool-learning methods unclear.

  2. Generalization is insufficiently evaluated. Of the 159 main evaluation tasks, 155 supplied tool-construction trajectories, with no separately reported evaluation on held-out tasks from the same websites. The experiments also emphasize weaker deployment models, leaving the tools’ value for more capable agents insufficiently established.

Questions for authors
  1. For atomic and composite tools, could you report the typical number of primitive browser actions and internal model calls involved in each invocation? Could you also compare an atomic-only library with the full library under matched online budgets, to clarify how tool granularity and composition affect task success and efficiency?

  2. Given that 155 of the 159 main evaluation tasks contributed tool-construction trajectories, can the model generalize to new tasks on the same website using tools learned from other tasks?

  3. Could you add experiments with other similar baselines, such as WALT and SkillWeaver, under matched settings? Additionally, since ContractSkill uses Qwen3-4B for skill generation and repair while TrajSmith uses a library constructed by Claude Opus 4.6, could you evaluate a TrajSmith library constructed by Qwen3-4B itself?

  4. The paper reports results for models such as GPT-4.1 and Qwen3-4B. Could you further validate the usefulness of the tools with more capable models?

Reviewer_WZVGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces TrajSmith, a method that uses trajectories to synthesize web tools that can be reused. The problem being tackled here is that solving recurring tasks involves repeated planning - even though prior runs already contain the steps to solve - and this leads to more token consumption and possibilities for errors. Stronger models are better at not making the errors, but if this "experience" isn't saved, the model needs to make the same decisions over and over again. Instead, the paper tackles the question of how to save and leverage this experience - and further, how a weaker model can use experience. The paper proposes a strong-to-weak reuse, exploring the trajectories to synthesize reusable procedures AND dictate when it's use is appropriate. The tools are bundled with a usage contract, conditions for use, argument domains and expected results - defining a strong interface for reusability.

The tools themselves are synthesized by analyzing the structures of prior trajectories, the actual DOM elements (or accessibility trees) on the web pages, and create semantic locators to turn the trajectory into an arsenal of tools, which may be created from learnings across multiple trajectories. This may involve replaying of the trajectories, analyzing the DOM to convert UI labels into actual URL or script parameters, and inferring result types. The tools themselves are then implemented: A trajectory containing many reusable actions effectively becomes a family of tools. First, atomic tools are created for an action provided by a UI control, and can be parameterized over the objects and values we observe. Then, a composite tool chains these atoms to produce an e2e task.

A stable implementation is chosen before testing these tools for applicability by varying the arguments, probing nearby elements, checking the expected return types, etc. This is done by a challenger that generates multiple classes of tests that decide whether a tool should remain a candidate or be admitted into the active state. Admission depends on local capability - basically "does this tool do what it says it is supposed to", not on whether the task itself succeeded or not. This process effectively ensures that the agent has access to a validated set of tools with well defined contracts that have rejection rules for invalid arguments/use cases.

The weak model then uses these tools and the contract can be used to validate the usage before execution. During usage, the tool is first matched to the task, and if the tool is appropriate for the task, using a contract based retrieval system, filtering the tool library by the lifecycle, risk permissions, etc., then ranking the remaining tools, and finally provides the model with tools and their corresponding parameter domains. The model fills the parameters if there candidate tools are found, otherwise the model abstains from using the tool and interacts with the browser directly.

The paper also goes into more detail regarding the usage contract, defining three things - (1) matching fields - an argument schema with a "domain" of the slots, description for retrieval and expected result. (2) governance - defines the what seems to be effectively metadata about the tool - the lifecycle stage, risk, generalization tier, and (3) composition - which specify the tool's role and the state/capability it provides or consumes. For a multi-slot tool, the parameters are verified independently (no joint verification).

Evaluation is done on Online-Mind2Web and WebArena with using Qwen 3 (4B) and reports improvement in success rates on both test suites. Results on Mind2Web using Qwen 3 4B instruct show improvement in success rate and decrease in number steps over an agent with no tools, and on WebArena, shows a net success rate improvement on all 5 categories.

Additionally, shared budget experiments are run - capping the number of LLM calls/tool calls. The results on Online-Mind2Web are reported compared against an agent with no tools, and shows improvements on both gpt-4o-mini and qwen-3-vl 4B.

Finally the authors also compare against an adapted version of ContractSkill and show that the method improves over ContractSkill as well.

The paper does include ablations that check the effect of contract retrieval, addition of candidate tools, removal of tool generated from failed trajectories and conclude that retrieval has the most observed effect here. Weak agent failure is also investigated, where of 53 failures without tools, 29 have associated tools.

Critical strengths

The general idea of using a strong model to build tools for weaker models by generalizing trajectory structures into repeated patterns is a good mechanism - having contracts that store the type of arguments, expected results, rejection rules also provides human reviewabilty. The fact that this was tested on two separate web datasets is another point in favor of the evaluation rigor/completeness. Additionally, I appreciate the ablation studies that were already done, the amount of detail in the appendix around the attribution of the differences between runs.

Critical weaknesses
  • Because the selection of the tasks were coverage based, this makes it harder to see how easily this transfers to brand new tasks. Of course, if the scope was purely to improve on recurring tasks, this makes sense, but it is equally plausible that there are tools that can be built from one recurring task that is useful for another. Additionally, this would make it more useful as a result for people building these tools - minimizing the number of tools they would have to create.

  • Success rate increase is correlated with the tools, but not entirely sure if the tools are actually used - in appendix C.13, we see that the model solves 9/32 tasks without tools, 16 out of 32 with TrajSmith, but 6 out of 7 of those gains did not use the tool evidence. So how much of this is just run to run variance? C.13 explicitly reports two runs. It would be good to get repeated runs and variance on the success rate to rule out noise from the generative process. Though both pairs improve over the baseline, and while the magnitude is hard to dismiss, the mechanism is hard to understand, especially since only 1 of the 7 cases involved using the tool evidence.

  • The verification and applicability checks seem to be doing a lot less of the heavy lifting compared to the actual construction of the tools themselves. the ablations especially show that there is very little difference (1 task) between admitting/not admitting candidate tools. Removing the semantic intent check seems to have increased the number of successful tasks in table 5 - the current experiments are not really showing how much the verification step is helping or hurting here, plus the number of tasks being evaluated here are potentially too few.

Questions for authors
  1. How generalizable are the tools that are being constructed? Suggestion: hold out an unseen dataset of tasks, see how performance improves based on tools created from OTHER tasks.

  2. Can we get more details around what semantic verification includes? What are we removing when we run "w/o semantic verification"?

  3. Can we see a breakdown of how often the successes are coming from actual tool use beyond the evidence in C.13? Would be good to get those numbers on both datasets.

  4. How can we better measure the benefit coming from verification/applicability checks? At present, there doesn't seem to be strong evidence either way.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.