- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The author found a new problem in agent systems: web agents assume that combining facts that are true about different entities is also true, such as "Issue A is open" and "Issue B is assigned to Alice" implies "the issue is open and assigned to Alice." ELVA solves this problem by keeping provenance of each piece of evidence, i.e., which page, item and action it was collected from, and dropping the evidence when the provenance is lost, and the agent stops when the evidence is complete. When evidence is missing, the agent backtracks to observe rather than making assumptions.
Critical strengths
The paper points out a real and under-studied failure in web agents: true facts get combined the wrong way, so the agent wrongly claims success. Framing verification as something that has to start during evidence collection, not only at the end, is a useful and fresh perspective.
ELVA uses explicit rules for when evidence can be reused and when it expires, so every completion decision can be traced and inspected. It also comes with a formal guarantee, which is uncommon for web-agent systems.
The method beats strong baselines on all five WebArena-Verified sites, and a controlled diagnostic shows it reduces both false “done” claims and missed valid evidence. Ablations then show which parts of the system actually drive the gains.
Critical weaknesses
The task compiler is hand-written regex rules containing domain terms such as “issue,” “project,” and “product,” and the system is tested on only one benchmark with one LLM.
The paper does not compare against LLM-based or learned verifiers on the same checkpoints, and it reports no token, call, or latency costs. Without these it is hard to tell whether ELVA beats simpler alternatives in practice, and the modest gains on some sites (under 2 pp on Reddit and Map) make this matter more.
In practice, “inadmissible conjunction” looks a lot like the verifier hallucinating a link between two unrelated facts, which raises the question of whether a stronger LLM verifier with entity-aware context would solve it without the whole runtime. The paper doesn’t measure how often this failure actually happens in real agent errors compared with ordinary hallucination or reasoning mistakes, and it doesn’t compare against LLM-based verifiers, so the main motivation is asserted rather than shown.
The framework is hard to be moved to new tasks because there are many hand-tuned thresholds and penalties.
Questions for authors
How common is the problem? In ELVA’s or the baselines’ failed trajectories, what fraction of false completions are actually caused by inadmissible conjunction, compared with ordinary hallucination or reasoning errors? If this failure is rare, the motivation for the whole runtime is much weaker.
Comparison to LLM verifiers. How does CIV compare with an LLM-based verifier (e.g., a strong model given the full trajectory with entity IDs kept) on the same 1,600 checkpoints? Can a stronger model avoid this failure on its own, without the hand-built pipeline?
Cost. What are the token usage, number of LLM calls, and wall-clock time per task for ELVA compared with the baselines? Without this, it’s unclear whether the 6.15 pp gain comes from better verification or just from more steps and compute.
Generalization and hand-tuning. How many compiler rules and hyperparameters (thresholds, penalties, priors) were tuned on WebArena, and were they tuned on the same tasks used for evaluation? Have you tried ELVA on a different benchmark (e.g., WorkArena or Mind2Web-Live) or with a different policy LLM?
Compiler coverage. On how many evaluation tasks does the compiler miss or mistranslate a requirement? Since fail-closed checks can’t catch requirements the compiler never extracted, this directly limits the guarantee.