- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Web$hell, a framework that provides a unified interface for web agents to search for information, interact with webpages, and revisit stored evidence. Preliminary analysis shows that previous web search and navigation methods do not generalize well across web tasks, showing limitations of using only certain types of access model. To mitigate the this problem, the proposed approach represents content and controls as addressable objects, uses a deterministic compiler to preserve their relationships, and caches page states for selective access. The proposed approach is reported to obtain the best performance among the evaluated methods on multiple benchmarks, generally with fewer steps and comparable or lower LLM inference costs.
Critical strengths
- This paper aims to address a practical problem that web agents usually need both information gathering and browser interaction.
- The system design is well-motivated and coherent, and the evaluation on multiple benchmarks shows the effectiveness of the proposed approach.
Critical weaknesses
- It seems that the main underlying LLM is gpt-5.6-luna, and it will be better if more models can be tested to see how well the proposed framework can fit different models, and especially whether some components is dependent on specific model choices.
- There is a lack of ablation studies on the components of the framework, and the source of improvement is not fully isolated since it changes too many aspects at the same time.
Questions for authors
- Does the cost mesurement includes all the aspects, such as fetching and compilation, if not, what are the end-to-end latency and total resource costs, including all different aspects?
- At many times, the screenshot of the web pages can contain helpful information, would it be feasible to include such information in the proposed framework?
- I'm wondering whether there will be errors or exceptions when processing and compiling the web pages. Since real-world pages can be noisy and contain errors themselves, how to ensure the processing can handle different kinds of cases?
Thanks for the detailed response. The repeated runs, the full 420 question ablation and the extra model families address most of my concerns. the cost explanation (cache-write pricing) makes sense now. Raising my score.
One thing left. On full WebArena-Verified HybridSB is 0.649 vs 0.671 for Web$hell, so the margin there is about 2 pts, not the 7-11 pts vs HybridSBF in your table. Please put HybridSB in Table 3 and adjust the claim. The cost difference ($0.017 vs $0.066) is the stronger point here anyway.
Also the extra model family results are 100 tasks and one run each, so some margins are within noise. would be good to say so in the paper.