- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
TrajSmith uses strong models to probe the webpage structures via the recorded web-agent trajectories and turn them into atomic and composite tools. It then tests and validates each tool with unseen argument values, out-of-scope intents, boundary values and empty-result queries, and records the outcome. At deployment, a weak model reads the contracts to pick a tool and fill its arguments, and a programmatic gate decides whether to run the tool or fall back to ordinary browsing. The method is tested on two benchmarks. On a 159-task Online-Mind2Web subset selected for tool coverage, the better of two paired runs raises Qwen3-4B success from 18.2% to 30.8% while using fewer recorded agent-loop steps. On WebArena’s 763 single-domain tasks, success increases from 19.8% to 24.5%.
Critical strengths
A clear way for a small model to use tools built by a large model. Each tool comes with a short description, which the paper calls a contract. It tells the small model when the tool applies; which argument values the tool has been tested with; which similar-looking requests it should refuse; and what a correct result looks like. The paper gives enough detail to rebuild the system.
Critical weaknesses
-
The evaluation does not test the paper's central claim that tools remain reusable beyond the instance they were built from. 155 of the 159 tasks from the Online-Mind2Web are used for both evaluation and tool synthesis (Sec. 4.1, Table 12). So it's more about testing the weak model's tool-call capability, not the tool's reusability. To my understanding, reusable tools mean that the agent can use a set of the same tools with different parameter values to solve similar tasks, rather than the tasks from which the tools are derived. Also, according to Table 10, 787 of the 896 tools are in the paper’s fixed tier. The paper describes these as parameterless tools that receive only “lighter replay checks”, which hints at a potential overfitting on the environment or the tasks used to construct the tools.
-
The paper claims two technical contributions. The first is exploring the page around each recorded run to build better tools. The second is testing each tool and writing down when it should and should not be used (the contract). The paper’s own experiments do not show that either one helps.
-
Removing the new parts does not hurt. The authors rebuilt the library with no exploration and no testing. It solved exactly as many tasks as the full system: 18 of 58 (Table 18).
-
Removing or loosening several other parts of TrajSmith's design changes the solved tasks mostly by only one or two, and 2 out of 5 changes in TrajSmith's design can actually hurt performance according to Table 5.
-
The authors themselves write that the contributions of exploration and verification "remain unresolved" (Line 422–423).
-
When a tool is used, and the task still fails, we don't know why. The cause could be any of three things: 1) the small model picked the wrong tool; 2) the small model filled in the wrong parameter values; 3) the tool itself was broken.
-
-
The comparisons with other methods are not fair or are missing.
-
ContractSkill is the only other method compared. Its skills were built by the small Qwen3-4B model while solving each task (Appendix C.11, Line 1224-1225). TrajSmith's tools were built in advance by Claude Opus 4.6, from the same tasks (Appendix C.11, Line 1225-1226). So the 6-vs-18 result in Table 4 does not support the claim.
-
WALT and SkillWeaver (as cited in the paper) are very related earlier method, but it is not compared. The argument for not using these two methods is "archived WALT and SkillWeaver records use different task sets or incomplete protocols" (Line 412-413). I do not understand the explanation nor see it as convincing.
-
Questions for authors
Please see the critical weaknesses section. Those are also my questions.