acceptodds
Under review as a conference paper at ICLR 2027

Recursive LLM-as-a-Judge for Long-Horizon Web-Agent Trajectories

Abstract

Recent advances in large language models (LLMs) have enabled web agents to perform increasingly complex, long-horizon tasks. Reliable evaluation of long-horizon web-agent trajectories requires effective evidence acquisition and joint reasoning over collected evidence. However, evidence acquisition is challenging when trajectories exceed the judge's context window, as exhaustive inspection is inefficient when evidence is sparse and fixed selection cannot adapt to uncertainties uncovered during evaluation. Moreover, evidence aggregation must then account for both cross-window dependencies and each rubric's logical structure. To address these challenges, we introduce RecursiveWebJudge, an LLM-as-a-judge framework inspired by recursive reasoning that couples adaptive evidence acquisition with rubric-specific aggregation. The framework first partitions trajectories into context-compatible windows and decomposes rubrics into verifiable requirements. For each selected window, the judge records local assessments and factual notes for these requirements in a shared evidence pool, using accumulated evidence and unresolved requirements to guide subsequent window selection and stopping. It then reasons over the accumulated evidence across windows to derive a final verdict, applying AND when all requirements must hold and ANY when a single qualifying instance suffices. We evaluate RecursiveWebJudge on AgentRewardBench, a benchmark of expert-annotated multimodal web-agent trajectories, using 3B, 7B, and 11B judge backbones to assess task success and non-progressing repetition. It improves average macro-F1 across task success and repetition detection by up to 26.6% over the strongest baseline, with its 7B judge surpassing full-trajectory Claude Code evaluation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM-as-a-Judge, web agents, long-horizon evaluation, multimodal trajectories, long-context reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yGyuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RecursiveWebJudge for evaluating long-horizon web-agent trajectories. The method divides trajectories into windows and decomposes evaluation rubrics into verifiable requirements. It records local assessments and factual evidence, adaptively selects additional windows, and decides when to stop. The collected evidence is then aggregated according to the rubric’s logical structure. Experiments on AgentRewardBench use three multimodal judge backbones to assess task success and non-progressing repetition, reporting up to a 26.6% relative improvement in average macro-F1 over the strongest baseline using the same backbone.

Critical strengths
  • The paper addresses the practical challenge of evaluating web-agent trajectories that exceed the judge’s context window.
  • Experiments cover three multimodal backbones from two model families and show substantial improvements in repetition detection on Qwen-7B.
Critical weaknesses
  • The evaluation covers only task success and non-progressing repetition, which map to relatively simple AND and ANY rules. It remains unclear how well the framework would handle more complex aspects of web-agent behavior, such as action intent, malicious actions, or recovery from errors.

  • The method mainly combines context compression, iterative inspection, and task-specific decision rules, using prompts to select windows and decide when to stop. While the design is reasonable, its novelty over existing approaches is limited, particularly given that TRACE already incorporates adaptive inspection and cross-step evidence accumulation.

  • Qwen-7B’s repetition false positive rate reaches 70.0% in Table 2. Despite the gains in macro-F1, this level of false alarms raises concerns about practical reliability. An analysis of whether these errors come from misreading individual windows or from the ANY rule propagating local mistakes would help explain this result.

  • The efficiency claim is not yet supported by a full cost comparison. The method sometimes inspects fewer windows, but it also makes additional controller calls and repeatedly processes accumulated evidence. Without reporting total token usage and runtime, it is difficult to tell whether it is actually more efficient than exhaustive inspection.

Questions for authors

Equation 7 returns a negative verdict when all observed, relevant local labels are negative. How does this rule distinguish between “no repetition has been found” and “no repetition occurred,” particularly when some windows remain uninspected or repetition can only be identified by linking evidence across windows?

6 comments

Thank you for recognizing the practical importance of the problem and the breadth of our evaluation! We highly appreciate your concerns and your precise question on Eq. 7. We conducted the suggested analyses and revised the paper accordingly, with all changes highlighted in blue.


Comment 1: The evaluation covers only task success and non-progressing repetition, which map to relatively simple AND and ANY rules. It is unclear how the framework handles more complex aspects such as action intent, malicious actions, or recovery from errors.

Thanks for this thoughtful point. We clarify the scope of our evaluation and how the framework extends to these aspects.

Firstly, why we evaluate these two rubrics. AgentRewardBench is the only public web-agent benchmark we identified with expert annotations for both an outcome-based and a behavioral rubric (Section 5). Its third rubric, Side Effect, has a positive rate of 6.5% and an ambiguous definition, so we excluded it. We are not aware of expert-annotated long-horizon trajectories for action intent or malicious actions, and therefore do not claim results on them.

Secondly, how these aspects map onto the framework. AND and ANY are the two logical primitives from which such rubrics are built, and the framework takes the rubric text and its aggregation type as input (Algorithm 1).

  • Malicious actions follow ANY logic: a single window containing a harmful or unauthorized action settles the verdict, just as one qualifying loop settles Repetition.
  • Action intent can likewise be cast as ANY over violations: one window in which the agent acts outside the user's intent is a qualifying instance.
  • Recovery from errors is already part of Success. The factual notes record the outcome of each window so that the aggregator can reason about whether later actions preserve or undo earlier progress (Sections 4.3–4.4). In Case 3, the judge reconciles a confirmed submission with later failed attempts and reaches the correct verdict.

In summary, the aspects you mention can be expressed with the same AND/ANY rules. Testing them is important future work, but it requires annotated benchmarks that are not yet available.


Comment 2: The novelty over existing approaches is limited: the method combines context compression, iterative inspection, and task-specific decision rules, uses prompts to select windows and decide when to stop, and TRACE already incorporates adaptive inspection and cross-step evidence accumulation.

Thanks for raising this important point. We have added a mechanism-level comparison with IRCoT and TRACE to Appendix C.1 and ablations of rubric decomposition and factual notes to Section 6.2 (Table 3). We clarify our novelty below.

Firstly, where our method differs from TRACE. We agree that adaptive inspection and cross-step evidence accumulation are shared with TRACE, which is why TRACE is a same-backbone baseline in Table 1 of the paper. What RecursiveWebJudge adds is to couple the acquisition loop with structured local evidence, i.e., a local assessment (+, −, ?, NA) and a factual note recorded for each inspected window (Eq. 4), and with rubric-specific aggregation that derives the verdict from this evidence according to the rubric's logical structure (Eq. 6–7). The trajectory is not compressed before judging: each window is observed in its raw multimodal form in an isolated call, and only the returned assessments and notes are passed on. Table 1 compares the methods.

Table 1: Comparison with IRCoT and TRACE

IRCoTTRACERecursiveWebJudge
SettingMulti-step QA over a retrieval corpusTrajectory evaluationTrajectory evaluation
What guides the next stepThe latest reasoning sentence, used as a retrieval queryA triage pass over a text-only overview of the full trajectory, then model-chosen analysis actionsThe accumulated local assessments and notes, starting from the first and last windows
What is accumulatedRetrieved paragraphs and reasoning stepsStep-cited findings from the analysis actionsPer-window local assessments (+, −, ?, NA) with factual notes
How the verdict is derivedRead from the final reasoning chainFree-form synthesis of the findingsRubric-specific aggregation: AND reasons over the ordered evidence; ANY applies a deterministic rule and defers only residual cases

Secondly, the gains come from what TRACE lacks, not from what it shares with us. Table 2 varies one component of our framework at a time, with two baselines for reference.

Table 2: Isolating each component (COMB / Success mF1 / Repetition mF1)

MethodIterative acquisitionStructured local evidenceRubric-specific aggregationLlama-11BQwen-3BQwen-7B
WebVoyager–––39.5 / 54.8 / 24.239.0 / 54.7 / 23.342.7 / 65.7 / 19.6
TRACE✓––27.7 / 36.2 / 19.331.6 / 39.6 / 23.740.6 / 62.1 / 19.1
Ours, First+Last–✓✓46.6 / 51.0 / 42.342.5 / 56.5 / 28.656.1 / 63.8 / 48.5
Ours, AggFlip✓✓swapped37.6 / 34.2 / 41.024.0 / 24.6 / 23.332.1 / 40.9 / 23.3
Ours✓✓✓52.6 / 54.7 / 50.448.4 / 58.7 / 38.162.9 / 68.6 / 57.3

The results reveal three key findings:

  1. The mechanisms shared with TRACE do not account for the gains. With adaptive inspection and cross-step accumulation, TRACE still detects little repetition: its Repetition recall is 3.9, 7.5, and 2.6 on Llama-11B, Qwen-3B, and Qwen-7B, versus 68.0, 26.9, and 85.2 for our method (Table 11 of the paper).
  2. Our components alone outperform TRACE without any adaptive inspection. First+Last involves no iteration and no controller, yet exceeds TRACE on both rubrics on every backbone. Compared with WebVoyager, which inspects one window, it improves Repetition by 18.1, 5.3, and 28.9 points while keeping Success within 3.8 points.
  3. Adaptive acquisition adds further gains, mainly on Repetition. Adding adaptive acquisition to First+Last improves Repetition by 8.1, 9.5, and 8.8 points and Success by 3.7, 2.2, and 4.8 points, as repetitive behavior typically arises in intermediate interactions that the endpoint windows miss.

Thirdly, on prompt-based selection and stopping.

  • The main gains do not rely on the prompt-based controller: First+Last and Exhaustive make no controller calls, and both exceed TRACE on every backbone (Tables 1-2 of the paper).
  • Aggregation is not left to a prompt alone: ANY applies a deterministic rule and defers only residual cases to the judge (Eq. 7).
  • For the controller itself, we now validate its stopping decisions directly (Appendix C.5) and localize its failure modes (Appendix C.6); see also Comment 5.

In summary, most of our gains come from structured local evidence and rubric-specific aggregation, which TRACE does not have. The adaptive loop, which TRACE also has, adds further gains on top.


Comment 3: The Repetition false positive rate reaches 70.0% on Qwen-7B, raising concerns about practical reliability. An analysis of whether these errors come from misreading individual windows or from the ANY rule propagating local mistakes would help.

Thanks for this constructive suggestion. Following it, we have added exactly this analysis to Appendix C.7 (Table 16).

Firstly, misreading versus rule propagation. The two sources you distinguish leave different traces: a local mistake propagated by the ANY rule yields a false positive resting on a single positive window, whereas consistent misreading yields several. We therefore count the windows labeled positive for each positive Repetition verdict under exhaustive inspection (Table 3).

Table 3: Share (%) of Repetition verdicts resting on zero, one, or at least two positive windows (Table 16 in the revised paper)

BackboneFalse positives: 0 / 1 / ≥2True positives: 0 / 1 / ≥2
Llama-11B1.0 / 37.3 / 61.80.2 / 38.7 / 61.1
Qwen-3B7.4 / 36.8 / 55.930.1 / 31.9 / 38.0
Qwen-7B3.9 / 26.4 / 69.81.8 / 9.2 / 89.0

The false positives stem from how the backbone interprets individual windows rather than from the ANY rule, as two observations show:

  1. Most false positives rest on several positive windows. On Qwen-7B, 69.8% rest on at least two independently positive windows, 26.4% on a single one, and 3.9% on the residual case of Eq. 7. They are thus consistent misreadings across windows, as in Case 5, rather than one local mistake propagated by the rule.
  2. A stricter rule does not separate them from true positives. Requiring two positive windows would remove 30.3% of the false positives on Qwen-7B at the cost of 11.0% of the true positives, but on Llama-11B and Qwen-3B it would remove true positives at a similar or higher rate (38.9% vs. 38.3%; 62.0% vs. 44.2%).

Reducing them therefore depends on the backbone's ability to observe each window accurately, rather than on a different aggregation rule.

Secondly, practical reliability. A false positive rate is informative only together with detection: Repetition is positive in 80.0% of the trajectories (Section 5), so the rate is computed on the remaining 20%. Table 4 therefore reports precision and recall for all methods.

Table 4: Repetition precision, recall, and macro-F1 with Qwen-7B (from Table 11 of the paper)

MethodPrecisionRecallmF1
WebVoyager84.93.019.6
WebJudge88.90.717.4
Universal Verifier95.20.617.2
TRACE87.62.619.1
Mind2Web 289.428.339.6
Ours83.885.259.5
Claude Opus 4.8 (full trajectory)80.489.550.7

Our method detects repetition at a precision comparable to the baselines, and its remaining false positives reflect a difficulty shared by a frontier judge:

  • The baselines' low false positive rates come from not detecting. Four of the five baselines almost never flag repetition, missing more than 97% of the repetitive trajectories. Our method keeps a comparable precision (83.8 vs. 84.9–95.2) while raising recall from at most 28.3 to 85.2.
  • The negatives are hard even for a frontier full-trajectory judge. Claude Opus 4.8 attains a lower precision than ours (80.4 vs. 83.8) at a similar recall, which implies an even higher false positive rate (about 87%, derived from Table 11 and the 80.0% positive rate). The false positives thus reflect a limitation of the backbone's perception rather than a flaw of the framework.

In summary, the false positives come from the backbone misreading windows, not from the ANY rule. Even a frontier judge has the same difficulty, and our method detects far more repetition than the baselines.

Reviewer_JajDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RecursiveWebJudge, an LLM-as-a-Judge framework for evaluating long-horizon web-agent trajectories. The method combines trajectory windowing, rubric decomposition, adaptive evidence acquisition, and rubric-specific aggregation to improve evaluation reliability. Experiments on AgentRewardBench with three LLM backbones demonstrate improvements over existing baselines, particularly in detecting non-progressing repetition.

Critical strengths
  1. The paper is well-written and well-organized, with clear explanations of the motivation and proposed framework.
  2. The authors provide a code repository link, which facilitates reproducibility and enhances the credibility of the experimental results.
Critical weaknesses
  1. Limited methodological novelty. The proposed framework mainly combines existing techniques, including trajectory windowing, rubric decomposition, adaptive evidence selection, and logical aggregation. The claimed recursive reasoning mechanism appears closer to iterative evidence retrieval, and its novelty over existing approaches is unclear.
  2. Unclear efficiency benefits. The exhaustive inspection baseline achieves higher overall performance across all three backbones, while the proposed method skips only a limited proportion of windows. Moreover, the additional LLM controller calls introduce computational overhead. Without detailed latency and token consumption comparisons, the claimed efficiency advantages remain unconvincing.
  3. Insufficient ablation studies. The current ablations do not adequately isolate the contributions of key components, such as rubric decomposition and factual evidence notes. Furthermore, swapping the AND/ANY aggregation rules is a relatively trivial ablation, as the resulting performance degradation is largely expected.
  4. High false positive rate. Despite improvements in repetition detection, the method exhibits a high false positive rate, reaching 70.0% on Qwen-7B. This raises concerns about the reliability of the proposed framework in distinguishing repetitive behavior from normal agent interactions.
Questions for authors

See above weaknesses.

7 comments

Thank you for recognizing the clarity of our writing and the reproducibility of our code! We highly appreciate your concerns and suggestions. We conducted the suggested experiments and revised the paper accordingly, with all changes highlighted in blue.


Comment 1: The methodological novelty is limited: the framework mainly combines existing techniques, and the recursive mechanism appears closer to iterative evidence retrieval.

Thanks for raising this important point. We have added a mechanism-level comparison with IRCoT and TRACE to Appendix C.1 and ablations of rubric decomposition and factual notes to Section 6.2 (Table 3). We clarify our novelty below.

Firstly, where our method differs. We agree that trajectory windowing, rubric decomposition, iterative evidence selection, and logical aggregation each have precedents. What RecursiveWebJudge adds is to couple the acquisition loop with structured local evidence, i.e., a local assessment (+, −, ?, NA) and a factual note recorded for each inspected window (Eq. 4), and with rubric-specific aggregation that derives the verdict from this evidence according to the rubric's logical structure (Eq. 6–7). Table 1 compares our method with iterative retrieval (IRCoT) and the closest trajectory judge (TRACE).

Table 1: Comparison with IRCoT and TRACE

IRCoTTRACERecursiveWebJudge
SettingMulti-step QA over a retrieval corpusTrajectory evaluationTrajectory evaluation
What guides the next stepThe latest reasoning sentence, used as a retrieval queryA triage pass over a text-only overview of the full trajectory, then model-chosen analysis actionsThe accumulated local assessments and notes, starting from the first and last windows
What is accumulatedRetrieved paragraphs and reasoning stepsStep-cited findings from the analysis actionsPer-window local assessments (+, −, ?, NA) with factual notes
How the verdict is derivedRead from the final reasoning chainFree-form synthesis of the findingsRubric-specific aggregation: AND reasons over the ordered evidence; ANY applies a deterministic rule and defers only residual cases

Secondly, why it is more than iterative evidence retrieval. Iterative retrieval appends the retrieved content to one growing context, which is infeasible when the trajectory exceeds the judge's context window. We therefore inspect each window in an isolated call that returns only a compact record, and the controller and the aggregator reason solely over these records (Tables 7–9 of the paper). Solving subproblems in isolated contexts and returning their results to the parent is the recursive pattern (Zhang et al., 2025; Yang et al., 2026), which we apply at a single level and hence describe as inspired by recursive reasoning. As a result, the largest prompt stays below 8k tokens regardless of trajectory length (Table 12 in the revised paper).

Thirdly, the gains come from these components rather than from the iterative loop. Table 2 varies one component of our framework at a time, with two baselines for reference.

Table 2: Isolating each component (COMB / Success mF1 / Repetition mF1)

MethodIterative acquisitionStructured local evidenceRubric-specific aggregationLlama-11BQwen-3BQwen-7B
WebVoyager–––39.5 / 54.8 / 24.239.0 / 54.7 / 23.342.7 / 65.7 / 19.6
TRACE✓––27.7 / 36.2 / 19.331.6 / 39.6 / 23.740.6 / 62.1 / 19.1
Ours, First+Last–✓✓46.6 / 51.0 / 42.342.5 / 56.5 / 28.656.1 / 63.8 / 48.5
Ours, AggFlip✓✓swapped37.6 / 34.2 / 41.024.0 / 24.6 / 23.332.1 / 40.9 / 23.3
Ours✓✓✓52.6 / 54.7 / 50.448.4 / 58.7 / 38.162.9 / 68.6 / 57.3

The results reveal two key findings:

  1. Structured local evidence and rubric-specific aggregation are the main source of the gains. Without any iteration, First+Last inspects only the first and last windows, one more than WebVoyager, yet improves Repetition by 18.1, 5.3, and 28.9 points on Llama-11B, Qwen-3B, and Qwen-7B, while keeping Success within 3.8 points. The main gains therefore do not stem from iterative retrieval.
  2. Adaptive acquisition adds further gains, mainly on Repetition. Adding adaptive acquisition to First+Last improves Repetition by 8.1, 9.5, and 8.8 points and Success by 3.7, 2.2, and 4.8 points, as repetitive behavior typically arises in intermediate interactions that the endpoint windows miss.

Together, these results clarify our novelty: most of our gains come from structured local evidence and rubric-specific aggregation, which iterative retrieval does not have. The acquisition loop adds further gains on top.


Comment 2: The efficiency benefits are unclear: exhaustive inspection achieves higher overall performance, only a limited proportion of windows is skipped, the controller adds overhead, and latency and token comparisons are missing.

Thanks for this valuable concern. We agree that the efficiency claim was not sufficiently supported. We have revised the efficiency-related statements in the abstract, Section 1, and Section 6.2, clarified the budget that our setting controls in Section 5, and added token and latency comparisons to Appendix C.4 (Tables 12–13).

Firstly, which budget matters in our setting, and how our framework meets it. We study judges whose context window is smaller than the trajectory, so a judge must either discard evidence or distribute it over multiple calls. Multiple calls are thus a requirement of the setting rather than an overhead of our method, and the binding constraint is the per-call context. Table 3 shows that our framework meets this constraint.

Table 3: Inference budget and accuracy with Qwen-7B (Table 12 in the revised paper)

MethodLargest prompt per call (mean / max)CallsTotal input tokensCOMB / S / R
WebVoyager5.0k / 7.1k2.09.9k42.7 / 65.7 / 19.6
WebJudge11.1k / 29.1k31.255.0k40.2 / 63.1 / 17.4
Universal Verifier4.9k / 6.6k31.239.9k42.1 / 67.1 / 17.2
Mind2Web 25.8k / 8.0k6.532.9k50.4 / 61.1 / 39.6
TRACE15.5k / 31.0k5.940.8k40.6 / 62.1 / 19.1
Ours5.6k / 7.8k23.059.6k63.8 / 68.1 / 59.5

Our largest prompt stays below 8k tokens, on par with WebVoyager, Universal Verifier, and Mind2Web 2 (6.6k–8.0k) and about a quarter of WebJudge and TRACE (29.1k–31.0k), while our method achieves the best result on every metric. Total cost inevitably trades off against accuracy; ours is comparable to that of WebJudge, and we report it for transparency.

Secondly, token and latency comparison with exhaustive inspection. Table 4 compares adaptive and exhaustive inspection on all three backbones.

Table 4: Cost of adaptive and exhaustive inspection (Table 13 in the revised paper)

BackboneMethodWindows readInput tokens (controller)Inference time (s)
Llama-11BExhaustive13.2103.6k2785
Llama-11BOurs12.299.4k (3.5k)2304
Qwen-3BExhaustive13.265.1k735
Qwen-3BOurs6.027.6k (1.4k)289
Qwen-7BExhaustive13.264.6k1036
Qwen-7BOurs11.759.6k (2.9k)984
  • Tokens and latency. Adaptive inspection reads fewer windows and uses fewer input tokens on every backbone, and its total inference time is 0.39×, 0.95×, and 0.83× that of exhaustive inspection on Qwen-3B, Qwen-7B, and Llama-11B.
  • Controller overhead. The controller's calls are text-only and account for 3.5–5.1% of the input tokens.
  • Proportion of skipped windows. We agree that it is limited for stronger controllers: adaptive inspection reads 45% of the windows on Qwen-3B but 89% and 92% on Qwen-7B and Llama-11B. The savings thus follow the controller's stopping behavior (Appendix C.6).

Thirdly, on exhaustive inspection achieving higher performance. Exhaustive is a variant of our own framework rather than an external baseline, and both variants exceed every baseline on every backbone. Its advantage is small on Qwen-7B and Llama-11B (0.2 and 0.9 COMB; Table 2 of the paper), and the larger gap on Qwen-3B stems from its controller stopping too early (Appendix C.6).

In summary, our framework keeps every call within a small context while achieving the best accuracy. Compared with exhaustive inspection, adaptive inspection reads fewer windows and tokens, at a small accuracy cost when the controller is calibrated. We claim no efficiency advantage beyond these measurements.


Comment 3: The ablations do not isolate the contributions of key components such as rubric decomposition and factual notes, and swapping the AND/ANY rules is a relatively trivial ablation.

Thanks for this constructive suggestion. Following it, we have added ablations of rubric decomposition and factual notes to Section 6.2 (Table 3). Since both components also inform the controller (Eq. 5), we ablate them under exhaustive inspection, which reads every window and thus isolates their effect on observation and aggregation.

Table 5: Ablation of evidence components under exhaustive inspection (Table 3 in the revised paper)

BackboneVariantSuccess mF1Repetition mF1COMB
Llama-11BExhaustive54.553.754.1
Llama-11Bw/o Decomposition53.353.053.2
Llama-11Bw/o Notes51.153.952.5
Qwen-3BExhaustive57.652.154.8
Qwen-3Bw/o Decomposition57.351.954.6
Qwen-3Bw/o Notes55.242.048.6
Qwen-7BExhaustive66.958.562.7
Qwen-7Bw/o Decomposition65.858.262.0
Qwen-7Bw/o Notes62.758.560.6

The results reveal three key findings:

  1. Factual notes matter, especially for the weakest backbone. Removing them reduces COMB by 1.6, 6.2, and 2.1 points on Llama-11B, Qwen-3B, and Qwen-7B. On Llama-11B and Qwen-7B, the drop falls on Success (3.4 and 4.2 points); on Qwen-3B, it falls mainly on Repetition (10.1 points).
  2. Rubric decomposition has a small but consistent effect, mainly through evidence acquisition. Under exhaustive inspection, removing it lowers both rubrics on every backbone, by 0.2–0.9 COMB. Under adaptive inspection, where decomposition also guides window selection, removing it lowers Success by 1.4–2.5 points.
  3. AggFlip is a sanity check, now complemented by component-level analyses. We agree that its direction is expected. It verifies that errors shift toward the failure mode implied by each rule and quantifies the cost of a mismatched rule (15.0–30.8 COMB). Beyond it, First+Last and Exhaustive isolate evidence acquisition (Table 2 of the paper), Table 5 isolates the evidence components, and swapping the observer, controller, and aggregator one at a time isolates each role (Appendix C.6).

In summary, the revised ablations now cover every key component. Factual notes matter the most, while rubric decomposition mainly helps the judge decide which windows to inspect.

Reviewer_ENipOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RecursiveWebJudge, a training-free framework for evaluating long-horizon web-agent trajectories with context-limited multimodal models. The method partitions trajectories into windows and decomposes evaluation rubrics into verifiable requirements. Starting from the first and last windows, the judge collects local assessments and factual notes in a shared evidence pool, which guides subsequent window selection and stopping. Evaluation includes task completion and no-progress repetition. The authors report improvements over existing evaluators in the average macro-F1 across success judgment and repetition detection. Ablations examine adaptive inspection and aggregation, while additional analyses investigate sensitivity to trajectory length and backbone-dependent inspection behavior.

Critical strengths
  1. The research problem has practical value and the evaluation dimensions are well defined. The work targets long multimodal trajectories, where evidence is scattered. Evaluation includes task completion and no-progress repetition.
  2. The paper shows clear gain in pruning no-progress repeitition and includes failure analysis. With Qwen2.5-VL-7B-Instruct, COMB improves from 50.4 (the strongest baseline) to 63.8. The ablations and appendix case studies also demonstrate the value of intermediate evidence and show the specific forms that judgment errors take.
Critical weaknesses
  1. The incremental contribution relative to prior work has not been sufficiently isolated. Iterative evidence gathering, cumulative evidence, and cross-window investigation all have precedents, such as IRCoT and TRACE. The existing pipeline comparisons lack experiments that control for budget and other modules.
  2. Early stopping may miss decisive updates or counter-evidence, and its reliability lacks targeted validation. “The evidence read so far supports the conclusion” does not guarantee that “the unread evidence will not overturn the conclusion.” Mandatory reading of the first and last windows and cautious prompting can only mitigate this risk.
  3. The analysis of small models’ decision-making capability and of the limits of the gains is insufficient. At 3B, the adaptive approach clearly lags behind exhaustive inspection, while at 11B the model tends toward broad inspection. The paper describes these phenomena but does not identify which stage is the bottleneck: window selection, observation, integration, or stopping. Moreover, the main gains come from Repetition, and COMB can easily mask differences in per-dimension capability.
Questions for authors
  1. Could the authors validate the main results on a more recent, similarly sized backbone, such as Qwen3-VL-8B or Qwen3.5-9B? Given the framework’s reliance on evidence interpretation and adaptive decision-making, this would help establish whether the reported gains generalize to current small VLMs.
  2. How does the controller determine that the unread windows will not contain corrections, retractions, or counter-evidence?
  3. Could the authors localize the capability bottleneck by fixing the evidence, supplying correct notes, or separately swapping the controller, observer, or aggregator?
5 comments

Thank you for recognizing the practical value of our research problem! We highly appreciate your concrete suggestions in Q1–Q3. We conducted the suggested experiments and revised the paper accordingly, with all changes highlighted in blue.


Comment 1: The incremental contribution over prior work is not sufficiently isolated.

Thanks for raising this important point. We have added a mechanism-level comparison with IRCoT and TRACE to Appendix C.1 and ablations of rubric decomposition and factual notes to Section 6.2 (Table 3). We clarify our contributions below.

Firstly, where our method differs. We agree that iterative evidence gathering, cumulative evidence, and cross-window investigation each have precedents. Our acquisition loop is inspired by IRCoT (cited in Section 1), and TRACE, which combines all three, is already a same-backbone baseline in Table 1 of the paper. What RecursiveWebJudge adds is to couple this loop with structured local evidence, i.e., a local assessment (+, −, ?, NA) and a factual note recorded for each inspected window (Eq. 4), and with rubric-specific aggregation that derives the verdict from this evidence according to the rubric's logical structure (Eq. 6–7). Table 1 compares the three methods.

Table 1: Comparison with IRCoT and TRACE

IRCoTTRACERecursiveWebJudge
SettingMulti-step QA over a retrieval corpusTrajectory evaluationTrajectory evaluation
What guides the next stepThe latest reasoning sentence, used as a retrieval queryA triage pass over a text-only overview of the full trajectory, then model-chosen analysis actionsThe accumulated local assessments and notes, starting from the first and last windows
What is accumulatedRetrieved paragraphs and reasoning stepsStep-cited findings from the analysis actionsPer-window local assessments (+, −, ?, NA) with factual notes
How the verdict is derivedRead from the final reasoning chainFree-form synthesis of the findingsRubric-specific aggregation: AND reasons over the ordered evidence; ANY applies a deterministic rule and defers only residual cases

Secondly, isolating the contribution. Table 2 varies one component of our framework at a time, with two baselines for reference.

Table 2: Isolating each component (COMB / Success mF1 / Repetition mF1)

MethodIterative acquisitionStructured local evidenceRubric-specific aggregationLlama-11BQwen-3BQwen-7B
WebVoyager–––39.5 / 54.8 / 24.239.0 / 54.7 / 23.342.7 / 65.7 / 19.6
TRACE✓––27.7 / 36.2 / 19.331.6 / 39.6 / 23.740.6 / 62.1 / 19.1
Ours, First+Last–✓✓46.6 / 51.0 / 42.342.5 / 56.5 / 28.656.1 / 63.8 / 48.5
Ours, AggFlip✓✓swapped37.6 / 34.2 / 41.024.0 / 24.6 / 23.332.1 / 40.9 / 23.3
Ours✓✓✓52.6 / 54.7 / 50.448.4 / 58.7 / 38.162.9 / 68.6 / 57.3

The results reveal three key findings:

  1. Structured local evidence and rubric-specific aggregation are the main source of the gains. Without any iteration, First+Last inspects only the first and last windows, one more than WebVoyager, yet improves Repetition by 18.1, 5.3, and 28.9 points on Llama-11B, Qwen-3B, and Qwen-7B, while keeping Success within 3.8 points. The gains therefore do not stem from a larger inspection budget.
  2. Both the evidence and the aggregation rule matter. Swapping the aggregation rules lowers Success by 20.5–34.1 points and Repetition by 9.4–34.0 points, showing that aggregation must match each rubric's logical structure. With per-window observations held fixed under exhaustive inspection, removing the factual notes from the aggregator's input reduces COMB by 1.6, 6.2, and 2.1, and the weakest backbone relies on them the most.
  3. Adaptive acquisition adds further gains, mainly on Repetition. Adding adaptive acquisition to First+Last improves Repetition by 8.1, 9.5, and 8.8 points and Success by 3.7, 2.2, and 4.8 points, as repetitive behavior typically arises in intermediate interactions that the endpoint windows miss.

Together, these results isolate our contribution: the gains stem mainly from structured local evidence and rubric-specific aggregation, which IRCoT and TRACE lack, while the iterative loop shared with these methods adds further gains on top.


Comment 2: The existing pipeline comparisons lack experiments that control for budget and other modules.

Thanks for this valuable suggestion. We have clarified the budget that our setting controls in Section 5 and added a cost analysis to Appendix C.4.

Firstly, which budget matters in our setting. We study judges whose context window is smaller than the trajectory. A judge must therefore either discard evidence or distribute it over multiple calls, so multiple calls are a requirement of the setting rather than an overhead specific to our method. The binding constraint is the per-call context, and this is the budget we control. Total cost inevitably trades off against accuracy; our focus is reliable judgment under a fixed per-call context, and we report the total cost for transparency.

Secondly, budget-controlled comparisons. Table 3 reports the per-call and total budgets of all methods with Qwen-7B.

Table 3: Inference budget and accuracy with Qwen-7B (Table 12 in the revised paper)

MethodLargest prompt per call (mean / max)CallsTotal input tokensCOMB / S / R
WebVoyager5.0k / 7.1k2.09.9k42.7 / 65.7 / 19.6
WebJudge11.1k / 29.1k31.255.0k40.2 / 63.1 / 17.4
Universal Verifier4.9k / 6.6k31.239.9k42.1 / 67.1 / 17.2
Mind2Web 25.8k / 8.0k6.532.9k50.4 / 61.1 / 39.6
TRACE15.5k / 31.0k5.940.8k40.6 / 62.1 / 19.1
Ours5.6k / 7.8k23.059.6k63.8 / 68.1 / 59.5
  • Per-call budget. Our largest prompt stays below 8k tokens, on par with WebVoyager, Universal Verifier, and Mind2Web 2 (6.6k–8.0k) and about a quarter of WebJudge and TRACE (29.1k–31.0k), while our method achieves the best result on every metric.
  • Total cost. Our total input is comparable to WebJudge and 1.46× that of TRACE, with COMB higher by 23.6 and 23.2 points.
  • Comparison with exhaustive inspection. Adaptive inspection reads fewer windows and uses fewer input tokens than exhaustive inspection on every backbone (Table 13 in the revised paper).

Beyond the budget, all methods share the backbone, rubric text, verdict labels, output format, and window size (Section 5), and Table 2 varies one component of our framework at a time.

In summary, under the per-call context budget that defines our setting, our method matches the most economical baselines while achieving the best accuracy, and its gains hold at a comparable inspection budget and at a comparable total cost.


Comment 3: Early stopping may miss later corrections, retractions, or counter-evidence, and its reliability lacks targeted validation.

Thanks for this insightful concern. We agree that no judge that reads only part of a trajectory can guarantee that the unread windows will not overturn its verdict. We have therefore validated early stopping directly by comparing adaptive and exhaustive inspection, and added this analysis to Section 6.2 and Appendix C.5.

Firstly, how the controller decides to stop.

  • Early stopping skips only intermediate windows: the last window, which reflects any later correction or retraction, is always inspected.
  • For the intermediate windows, the Control prompt (Table 7 of the paper) asks whether unseen evidence could still change the answer, so the stopping decision adapts to the rubric: one qualifying instance settles an ANY rubric, whereas an early failure does not rule out later completion under an AND rubric.
  • Accordingly, Qwen-7B inspects intermediate windows 7.8–14.1 percentage points less often under Repetition than under Success (Figure 4; see also Case 4).

Secondly, targeted validation. Table 4 reports how many errors early stopping causes and how many it avoids.

Table 4: Effect of early stopping (Table 14 in the revised paper)

RubricBackboneErrors that reading all windows would fixEarly stops that reading all windows would turn wrong
SuccessLlama-11B8%9%
SuccessQwen-7B6%3%
SuccessQwen-3B44%5%
RepetitionLlama-11B1%2%
RepetitionQwen-7B11%7%
RepetitionQwen-3B41%6%

The results reveal three key findings:

  1. With a calibrated controller, early stopping is not a meaningful error source. For Qwen-7B and Llama-11B, reading every window would fix only 1–11% of the errors while turning 2–9% of the early stops wrong, on both rubrics. Consistently, adaptive inspection matches or exceeds exhaustive inspection on Success and stays within 3.4 points on Repetition for both backbones (Table 2 of the paper).
  2. Reading every window is not an upper bound. In every setting, 2–9% of the early stops are correct only because the judge stopped, as unread windows carry observation noise as well as counter-evidence.
  3. The residual risk is confined to the weakest controller. On Qwen-3B, reading every window would fix 44% of the Success errors and 41% of the Repetition errors, because its controller stops too early (Comment 4). This is a calibration issue that diminishes with stronger backbones: the shares drop to 6% and 11% on Qwen-7B.

In summary, early stopping skips only intermediate windows and is not a meaningful error source under a calibrated controller; its remaining cost is confined to the weakest backbone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.