- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces RecursiveWebJudge for evaluating long-horizon web-agent trajectories. The method divides trajectories into windows and decomposes evaluation rubrics into verifiable requirements. It records local assessments and factual evidence, adaptively selects additional windows, and decides when to stop. The collected evidence is then aggregated according to the rubric’s logical structure. Experiments on AgentRewardBench use three multimodal judge backbones to assess task success and non-progressing repetition, reporting up to a 26.6% relative improvement in average macro-F1 over the strongest baseline using the same backbone.
Critical strengths
- The paper addresses the practical challenge of evaluating web-agent trajectories that exceed the judge’s context window.
- Experiments cover three multimodal backbones from two model families and show substantial improvements in repetition detection on Qwen-7B.
Critical weaknesses
-
The evaluation covers only task success and non-progressing repetition, which map to relatively simple AND and ANY rules. It remains unclear how well the framework would handle more complex aspects of web-agent behavior, such as action intent, malicious actions, or recovery from errors.
-
The method mainly combines context compression, iterative inspection, and task-specific decision rules, using prompts to select windows and decide when to stop. While the design is reasonable, its novelty over existing approaches is limited, particularly given that TRACE already incorporates adaptive inspection and cross-step evidence accumulation.
-
Qwen-7B’s repetition false positive rate reaches 70.0% in Table 2. Despite the gains in macro-F1, this level of false alarms raises concerns about practical reliability. An analysis of whether these errors come from misreading individual windows or from the ANY rule propagating local mistakes would help explain this result.
-
The efficiency claim is not yet supported by a full cost comparison. The method sometimes inspects fewer windows, but it also makes additional controller calls and repeatedly processes accumulated evidence. Without reporting total token usage and runtime, it is difficult to tell whether it is actually more efficient than exhaustive inspection.
Questions for authors
Equation 7 returns a negative verdict when all observed, relevant local labels are negative. How does this rule distinguish between “no repetition has been found” and “no repetition occurred,” particularly when some windows remain uninspected or repetition can only be identified by linking evidence across windows?
6 comments
Comment 2: The novelty over existing approaches is limited: the method combines context compression, iterative inspection, and task-specific decision rules, uses prompts to select windows and decide when to stop, and TRACE already incorporates adaptive inspection and cross-step evidence accumulation.
Thanks for raising this important point. We have added a mechanism-level comparison with IRCoT and TRACE to Appendix C.1 and ablations of rubric decomposition and factual notes to Section 6.2 (Table 3). We clarify our novelty below.
Firstly, where our method differs from TRACE. We agree that adaptive inspection and cross-step evidence accumulation are shared with TRACE, which is why TRACE is a same-backbone baseline in Table 1 of the paper. What RecursiveWebJudge adds is to couple the acquisition loop with structured local evidence, i.e., a local assessment (+, −, ?, NA) and a factual note recorded for each inspected window (Eq. 4), and with rubric-specific aggregation that derives the verdict from this evidence according to the rubric's logical structure (Eq. 6–7). The trajectory is not compressed before judging: each window is observed in its raw multimodal form in an isolated call, and only the returned assessments and notes are passed on. Table 1 compares the methods.
Table 1: Comparison with IRCoT and TRACE
| IRCoT | TRACE | RecursiveWebJudge | |
|---|---|---|---|
| Setting | Multi-step QA over a retrieval corpus | Trajectory evaluation | Trajectory evaluation |
| What guides the next step | The latest reasoning sentence, used as a retrieval query | A triage pass over a text-only overview of the full trajectory, then model-chosen analysis actions | The accumulated local assessments and notes, starting from the first and last windows |
| What is accumulated | Retrieved paragraphs and reasoning steps | Step-cited findings from the analysis actions | Per-window local assessments (+, −, ?, NA) with factual notes |
| How the verdict is derived | Read from the final reasoning chain | Free-form synthesis of the findings | Rubric-specific aggregation: AND reasons over the ordered evidence; ANY applies a deterministic rule and defers only residual cases |
Secondly, the gains come from what TRACE lacks, not from what it shares with us. Table 2 varies one component of our framework at a time, with two baselines for reference.
Table 2: Isolating each component (COMB / Success mF1 / Repetition mF1)
| Method | Iterative acquisition | Structured local evidence | Rubric-specific aggregation | Llama-11B | Qwen-3B | Qwen-7B |
|---|---|---|---|---|---|---|
| WebVoyager | – | – | – | 39.5 / 54.8 / 24.2 | 39.0 / 54.7 / 23.3 | 42.7 / 65.7 / 19.6 |
| TRACE | ✓ | – | – | 27.7 / 36.2 / 19.3 | 31.6 / 39.6 / 23.7 | 40.6 / 62.1 / 19.1 |
| Ours, First+Last | – | ✓ | ✓ | 46.6 / 51.0 / 42.3 | 42.5 / 56.5 / 28.6 | 56.1 / 63.8 / 48.5 |
| Ours, AggFlip | ✓ | ✓ | swapped | 37.6 / 34.2 / 41.0 | 24.0 / 24.6 / 23.3 | 32.1 / 40.9 / 23.3 |
| Ours | ✓ | ✓ | ✓ | 52.6 / 54.7 / 50.4 | 48.4 / 58.7 / 38.1 | 62.9 / 68.6 / 57.3 |
The results reveal three key findings:
- The mechanisms shared with TRACE do not account for the gains. With adaptive inspection and cross-step accumulation, TRACE still detects little repetition: its Repetition recall is 3.9, 7.5, and 2.6 on Llama-11B, Qwen-3B, and Qwen-7B, versus 68.0, 26.9, and 85.2 for our method (Table 11 of the paper).
- Our components alone outperform TRACE without any adaptive inspection. First+Last involves no iteration and no controller, yet exceeds TRACE on both rubrics on every backbone. Compared with WebVoyager, which inspects one window, it improves Repetition by 18.1, 5.3, and 28.9 points while keeping Success within 3.8 points.
- Adaptive acquisition adds further gains, mainly on Repetition. Adding adaptive acquisition to First+Last improves Repetition by 8.1, 9.5, and 8.8 points and Success by 3.7, 2.2, and 4.8 points, as repetitive behavior typically arises in intermediate interactions that the endpoint windows miss.
Thirdly, on prompt-based selection and stopping.
- The main gains do not rely on the prompt-based controller: First+Last and Exhaustive make no controller calls, and both exceed TRACE on every backbone (Tables 1-2 of the paper).
- Aggregation is not left to a prompt alone: ANY applies a deterministic rule and defers only residual cases to the judge (Eq. 7).
- For the controller itself, we now validate its stopping decisions directly (Appendix C.5) and localize its failure modes (Appendix C.6); see also Comment 5.
In summary, most of our gains come from structured local evidence and rubric-specific aggregation, which TRACE does not have. The adaptive loop, which TRACE also has, adds further gains on top.
Comment 3: The Repetition false positive rate reaches 70.0% on Qwen-7B, raising concerns about practical reliability. An analysis of whether these errors come from misreading individual windows or from the ANY rule propagating local mistakes would help.
Thanks for this constructive suggestion. Following it, we have added exactly this analysis to Appendix C.7 (Table 16).
Firstly, misreading versus rule propagation. The two sources you distinguish leave different traces: a local mistake propagated by the ANY rule yields a false positive resting on a single positive window, whereas consistent misreading yields several. We therefore count the windows labeled positive for each positive Repetition verdict under exhaustive inspection (Table 3).
Table 3: Share (%) of Repetition verdicts resting on zero, one, or at least two positive windows (Table 16 in the revised paper)
| Backbone | False positives: 0 / 1 / ≥2 | True positives: 0 / 1 / ≥2 |
|---|---|---|
| Llama-11B | 1.0 / 37.3 / 61.8 | 0.2 / 38.7 / 61.1 |
| Qwen-3B | 7.4 / 36.8 / 55.9 | 30.1 / 31.9 / 38.0 |
| Qwen-7B | 3.9 / 26.4 / 69.8 | 1.8 / 9.2 / 89.0 |
The false positives stem from how the backbone interprets individual windows rather than from the ANY rule, as two observations show:
- Most false positives rest on several positive windows. On Qwen-7B, 69.8% rest on at least two independently positive windows, 26.4% on a single one, and 3.9% on the residual case of Eq. 7. They are thus consistent misreadings across windows, as in Case 5, rather than one local mistake propagated by the rule.
- A stricter rule does not separate them from true positives. Requiring two positive windows would remove 30.3% of the false positives on Qwen-7B at the cost of 11.0% of the true positives, but on Llama-11B and Qwen-3B it would remove true positives at a similar or higher rate (38.9% vs. 38.3%; 62.0% vs. 44.2%).
Reducing them therefore depends on the backbone's ability to observe each window accurately, rather than on a different aggregation rule.
Secondly, practical reliability. A false positive rate is informative only together with detection: Repetition is positive in 80.0% of the trajectories (Section 5), so the rate is computed on the remaining 20%. Table 4 therefore reports precision and recall for all methods.
Table 4: Repetition precision, recall, and macro-F1 with Qwen-7B (from Table 11 of the paper)
| Method | Precision | Recall | mF1 |
|---|---|---|---|
| WebVoyager | 84.9 | 3.0 | 19.6 |
| WebJudge | 88.9 | 0.7 | 17.4 |
| Universal Verifier | 95.2 | 0.6 | 17.2 |
| TRACE | 87.6 | 2.6 | 19.1 |
| Mind2Web 2 | 89.4 | 28.3 | 39.6 |
| Ours | 83.8 | 85.2 | 59.5 |
| Claude Opus 4.8 (full trajectory) | 80.4 | 89.5 | 50.7 |
Our method detects repetition at a precision comparable to the baselines, and its remaining false positives reflect a difficulty shared by a frontier judge:
- The baselines' low false positive rates come from not detecting. Four of the five baselines almost never flag repetition, missing more than 97% of the repetitive trajectories. Our method keeps a comparable precision (83.8 vs. 84.9–95.2) while raising recall from at most 28.3 to 85.2.
- The negatives are hard even for a frontier full-trajectory judge. Claude Opus 4.8 attains a lower precision than ours (80.4 vs. 83.8) at a similar recall, which implies an even higher false positive rate (about 87%, derived from Table 11 and the 80.0% positive rate). The false positives thus reflect a limitation of the backbone's perception rather than a flaw of the framework.
In summary, the false positives come from the backbone misreading windows, not from the ANY rule. Even a frontier judge has the same difficulty, and our method detects far more repetition than the baselines.
Thank you for recognizing the practical importance of the problem and the breadth of our evaluation! We highly appreciate your concerns and your precise question on Eq. 7. We conducted the suggested analyses and revised the paper accordingly, with all changes highlighted in blue.
Thanks for this thoughtful point. We clarify the scope of our evaluation and how the framework extends to these aspects.
Firstly, why we evaluate these two rubrics. AgentRewardBench is the only public web-agent benchmark we identified with expert annotations for both an outcome-based and a behavioral rubric (Section 5). Its third rubric, Side Effect, has a positive rate of 6.5% and an ambiguous definition, so we excluded it. We are not aware of expert-annotated long-horizon trajectories for action intent or malicious actions, and therefore do not claim results on them.
Secondly, how these aspects map onto the framework. AND and ANY are the two logical primitives from which such rubrics are built, and the framework takes the rubric text and its aggregation type as input (Algorithm 1).
In summary, the aspects you mention can be expressed with the same AND/ANY rules. Testing them is important future work, but it requires annotated benchmarks that are not yet available.