- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper is hard to read, but based on my understanding, it proposes a new subset of the Opera dataset curated for web interactions, a new metric (SIR) for identifying useful datapoints, and focuses on anlayzing mind2web and opera (another web benchmark), and report several metrics that are completely unintelligible. To parse this, here Claude’s summary:
IntentTwin shows that offline web-behavior benchmarks can be gamed through how their candidate sets are built. Benchmarks insert the logged action among 15 distractors drawn from training actions. When the true action never appeared in training (about a third of OPeRA test rows), it is the only unfamiliar candidate, so "pick the unseen one" gets those rows right without modeling behavior. The authors quantify this with a support-identifiability rate (1.0 on those rows) and fix it with balanced candidate sets. Under balancing, a fine-tuned Qwen3-4B reranker still reaches 0.72 accuracy (0.57 on unseen actions). They also measure the gap between oracle reranking and deployment, where the system must propose the action itself. A proposer combining retrieval of known actions with open-vocabulary page and LM proposals reaches 0.52 end-to-end, versus 0.745 with the oracle candidate set. A similar but milder confound and gap appear on Mind2Web. The authors acknowledge that the "IRL-inspired" scorer isn't real inverse RL, that the split holds out sessions rather than users, and that many appendix results are single-seed. Results are also highly sensitive to evaluation context length.
Critical strengths
Graphical visualization of the method in figure 1 and 2
Critical weaknesses
- The writing is confusing and hard to follow. Right away, it introduces concepts that look simple on surface but simply mean nothing: what is “training support membership”, “support-identifiability rate (SIR)”, “open-vocabulary DOM/LM proposals.”, “target-anchored construction”, etc. Writing should be clear and meaningful. This draft is far from it.
- Poor description of the data and method. What is Opera? why did you choose it? what training are you performing (since your method is focused on training, as mentioned in abstract and 108)? I needed to look it up, which should have been properly explained in the paper.
- This work does not meaningfully advance the state of web agent research. It does not focus on more benchmarks web benchmarks beyond Mind2Web, which was released in 2023. There are a lot of high quality benchmarks that came out even in the past year (e.g. WebChain)
- It lacks substantial baselines, like recent LLM-based approaches like GenericAgent from BrowserGym (which can be powered by open-weight models like Qwen3.8 32B). To show that their contribution is useful, they need substantial baselines.
- Significant concepts missing. It does not discuss top-k-based approach like Dense Markup Ranker from WebLINX or the accessibility-tree based approach from WorkArena.
- Narrow contribution: the authors describe their work as “train-only candidate contract, row-level likelihood ledger, and explicit target-OOV analysis for a closed-choice factorized scorer.” This sentence does not make much sense, but from what i could gather, it appears that this approach is only useful during the training process, yet they do not release a trained model.
Questions for authors
It feels the paper is primarily written by Claude. Is that the case? It is okay if it’s for a draft, but I expect the final paper to be properly rewritten to be clear and direct about the contributions. Otherwise, it's not a good use of reviewers’ time if the paper is written in a way that's hard to parse and not informative.