- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
VERA acquires functional application knowledge through open-ended web exploration. It specifies expected outcomes before execution, verifies them using before/after screenshots, and admits supported claims into persistent knowledge. Recovery mechanisms support continued exploration, while contextual risk annotations document potentially sensitive operations.
Critical strengths
1.The paper addresses a meaningful reliability problem: successful tool execution does not necessarily justify retaining a functional claim. The overall workflow is understandable and appropriately separates outcome verification from risk assessment.
2.The controlled admission comparison on identical traces is useful. Precision improves from 82.5% to 97.0% while retaining almost all human-supported functions, providing encouraging evidence for evidence-based admission.
Critical weaknesses
1.The specific methodological contribution is not sufficiently isolated. The admission baselines compare verification against accepting proposals or executor success. They do not establish the value of fixing expectations before execution over ordinary screenshot-based outcome verification. A matched verifier without precommitted expectations would help clarify the incremental contribution.
2.The evaluation is narrow and does not establish downstream usefulness. The core admission study contains only 45 functions from two applications. Moreover, Appendix H demonstrates symbolic compatibility rather than actual downstream task execution. Consequently, both generalization and the practical benefit of the induced functional knowledge remain insufficiently demonstrated.
3.The exploration gains are not evaluated under comparable total cost. On SauceDemo, Full uses 138 high-level calls versus 34 for Linear. Although these costs are transparently reported, higher attained coverage under substantially more interaction does not establish a cost-effective advantage for the recovery mechanism.
Questions for authors
How does VERA compare with an otherwise identical outcome verifier without frozen expectations?
Thank you for your comments on the contribution, evaluation scope, and exploration cost.
W1 and Q1: Expected outcomes and knowledge verification
VERA's functional hypothesis specifies a location, action, possible preconditions, and an expected observable outcome. Fixing the expectation before execution gives the verifier a criterion for assessing the original claim against subsequent observations.
Following your suggestion, we compared VERA with a verifier without a preset expected outcome using the same actions and screenshot pairs:
Admission evidence support is the fraction of admitted attempts that are supported; supported-attempt retention is the fraction of supported attempts admitted. The comparison covers the two shopping sites and RealWorld, including 166 attempts with complete screenshots. VERA's two metrics are higher by 3.9 and 2.7 percentage points, respectively. These results support knowledge admission based on a preset functional claim: the expected outcome provides a clear object of verification while retaining attempts supported by observational evidence.
W2: Evaluation scope and downstream use
We chose sites whose functions and outcomes can be checked individually, and added three runs on the non-shopping RealWorld content community. Initial attempt-level evidence assessments yield:
Four Practice Shopping attempts with incomplete screenshots are excluded from the proportions. In RealWorld, VERA verified article browsing, filtering, editing, and publishing. Its admission evidence support rate was 90.4%, and all clearly supported attempts in the initial review were retained. The system also recorded contextual risks for evidence-gathering actions, illustrating this knowledge-acquisition process in a non-shopping setting. The RealWorld candidate scan included an additional instruction to check visible top navigation.
For downstream use, the controlled admission study on identical traces increases knowledge precision from 82.50% to 96.97% while retaining 32/33 supported functions. Appendix H shows that all three exported models support symbolic planning: two yield plans to order completion, while the third lacks a necessary transition and cannot reach that goal. This illustrates how knowledge admission affects symbolic reachability.
W3: Exploration mechanism and cost
Recovery provides further opportunities to collect evidence when the current interface's candidates are exhausted or verification does not support a claim. All conditions use the same ordinary-attempt cap; extra recovery calls are counted in the cost.
On SauceDemo, mean functional coverage across three runs was 59.1% for Full and 39.4% for Linear. Their respective totals were 138 and 34 high-level calls. Full's total comprised 62 ordinary attempts and 76 recovery/reset calls. These figures show the observed coverage gain alongside its interaction cost.