acceptodds
Under review as a conference paper at ICLR 2027

InEx: Intrinsic and Extrinsic Evidence for Trustworthy Evaluation of Latently Verifiable Tasks

Abstract

Many real-world agent tasks are latently verifiable: a correct answer or success condition exists in principle, but the corresponding ground truth is not cheaply available during evaluation. Reliable evaluation must therefore determine whether an agent's claimed outcome is supported by evidence rather than merely judge the plausibility of its final answer. We introduce InEx, an evidence-based evaluator that combines two complementary sources: intrinsic evidence already contained in the submitted trajectory and extrinsic evidence acquired through additional environment interaction. InEx-I reconstructs task-relevant intrinsic evidence by inspecting local observation-action transitions and determines whether the resulting evidence chain supports the claimed outcome. When intrinsic evidence alone is insufficient, InEx-E selectively interacts with the environment to acquire the missing extrinsic evidence. A scaffolded handoff carries the structured intrinsic analysis into extrinsic verification so that the two components form a continuous evidence chain. We implement InEx-I by specializing an open vision-language model through supervised fine-tuning, reinforcement learning, and confidence-aware learning from imperfect supervision, and implement InEx-E with a frontier model capable of environment interaction. We evaluate InEx on a diverse suite of agent benchmarks spanning web, mobile, and desktop interaction. InEx consistently achieves the highest evaluator accuracy among the compared methods and produces more reliable rewards that improve downstream agent training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent evaluation, LLM-as-a-judge, Agent-as-a-judge, Latently verifiable tasks, Multimodal agents, Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_dDPiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces InEx, which combines the reconstruction of internal evidence from trajectories with external environmental evidence actively gathered by an evaluator to assess agent tasks that lack cheap online ground truth, and further uses the evaluation results as reward signals for downstream agent training.

Critical strengths
  1. The paper focuses on an important and underexplored issue: how to reliably determine if an agent truly completes a task when the task has objective success conditions but the ground truth can't be directly accessed during evaluation. Dividing evidence into intrinsic and extrinsic types is a clear and practically meaningful perspective, especially for explaining failures where the 'answer seems reasonable but the trajectory doesn’t support it.'

  2. I like how the paper organizes step-by-step trajectory checking, evidence chain aggregation, and active environment verification into a complete process. The same InEx-I checkpoint is directly evaluated on WebVoyager, WebArena, AndroidWorld, and OSWorld, and it outperforms WebJudge across the board, which shows that the learned trajectory checking capability has some cross-environment transfer value, not just specific engineering optimizations for one site.

  3. The experiments are fairly comprehensive. Besides 4,000 trajectories across the four environments, the paper also provides component ablations, handoff ablations, 3B model experiments, reward error type analysis, and downstream PAE training. The analysis of success precision and failure recall, in particular, is more relevant to the real risk when the evaluator is used as a reward model than simply reporting accuracy.

Critical weaknesses
  1. The core training process is underdisclosed, making current results difficult to reproduce and interpret. The appendix does not report SFT/RL data scale, task composition, ratio of real vs. synthetic samples, specific RL algorithms, and key hyperparameters; The Value function and task distribution factor (rho_i) in formula (12) are also not clearly defined. Generating SFT traces, providing RL rewards, and implementing the "frontier model" for InEx-E do not provide specific models or versions. Given that SFT and RL bring about 13.6 and 4.9 percentage point improvements respectively, these deficiencies directly affect primary performance sources rather than secondary implementation details.

  2. There is a clear capability and computational budget asymmetry between full InEx and the main table baseline. InEx uses task-specific training with 7B evaluator, frontier model, and up to 20 additional steps of environmental interaction, while Claude, GPT-5, Qwen3-VL, and WebJudge in Table 1 are mainly passive trajectory evaluators. The paper discusses interactive evaluation methods such as Agent-as-a-Judge and ProRe in related work, but does not compare them under the same environmental interface and interaction budget. Table 4's "Frontier I w/o scaffold" is a helpful control experiment but only reported on WebVoyager, still insufficient to support strong SOTA conclusions for the full method in four environments.

Questions for authors

See the weaknesses above.

6 comments
AuthorsOpenReview ↗

Response to Reviewer dDPi (1/3)

We sincerely thank the reviewer for recognizing the importance of this underexplored problem and the clarity of the intrinsic/extrinsic perspective, the transfer of a single InEx-I checkpoint across four environments, and the comprehensiveness of our experiments, including the success-precision and failure-recall analysis. We address the questions below.

W1a. What SFT/RL data, RL algorithm and key hyperparameters were used to train InEx-I, and how are Value(·) and ρ_i defined?

The revised Appendix B specifies InEx-I's training (SFT and RL data, RL algorithm, hyperparameters, Value and ρ), summarized in the table. W1b below names the model in each role. We will release the training code and the trained InEx-I weights (Appendix B.5), and the predictions, prompts, aggregation code and train/evaluation exclusion list as specified in Appendices E.5 and F.4.

ItemRevised Appendix B
SFT data (B.1–B.2, Eq. 9)5,942 examples. Real: 942 verified GPT-4o traces on 1,200 human-labeled PAE web trajectories, kept when the verdict matched the label. Synthetic: 5,000 negatives, 2,000 by perturbing steps of successful trajectories and 3,000 from a non-interactive trajectory generator. Real : synthetic = 942 : 5,000
RL data (B.3)300 WebVoyager seed tasks, none in the evaluation set; trajectories from a GPT-4o agent; evaluator initialized from the SFT checkpoint, rollouts sampled on-policy
Reward sources (B.3, Eq. 11)113 tasks: rule-based checkers, reward = agreement with the rule label, c = 1. 187 tasks with reference answers: GPT-5.0 judge, binary reward R and confidence c ∈ [0, 1]
RL algorithm (B.4, Eq. 13)GRPO, each rollout's clipped objective scaled by its weight; G = 16 rollouts per trajectory, clip ε = 0.2, unweighted KL penalty β = 0.04 to the SFT checkpoint, λ = 0.2
Batch and optimization (B.2, B.4)2 trajectories per micro-batch, gradients accumulated over 16 (32 trajectories, 512 rollouts per update); full fine-tuning, SFT learning rate 2×10⁻⁵, RL 10⁻⁶; SFT roughly one pass over its data (about 10⁸ tokens); RL uses each seed task about 30 times
Compute (B.5)32 NVIDIA A100 40GB GPUs

Value and ρ (Eq. 12 and B.4), for rollout j of trajectory i on website k_i: Value(R_ij, c_ij, ρ_i) = (1 + λR_ij) · c_ij · ρ_i, ρ_i = clip(p_target(k_i) / p_sample(k_i), 0.5, 2), p_target(k) ∝ 1 / a_k, where a_k is the SFT checkpoint's accuracy on website k, measured on SFT data only, and p_sample(k) is its share of the accumulated batch. Weights are divided by their batch mean and held constant in the gradient. The confidence c is 1 for checker labels and the judge's confidence otherwise. The factor ρ favors websites where the SFT checkpoint is weakest, within fixed bounds.

The verified traces teach the workflow, and the synthetic negatives show plausible but unsupported trajectories. Uniform-weight RL raises WebVoyager accuracy from 85.2 after SFT to 88.5. The Value weighting brings it to 90.1. The weighted model is ahead in every environment, although on OSWorld the difference lies within run-to-run variation (Appendix F.8, Table 13). See Appendix B (B.1–B.5), marked in blue.

W1b. Which specific models generate the SFT traces, provide the RL rewards, and serve as the InEx-E frontier model?

The table names the model in each role.

RoleModelWhere in the revised paper
SFT evaluation tracesGPT-4oApp. B.1
RL rewardsRule-based checkers (113 seed tasks); GPT-5.0 as reward judge (187 seed tasks with reference answers)App. B.3
InEx-E frontier modelGPT-5.0, also in the frontier rows of Table 4 and in standalone InEx-E (Table 8)App. F.4, F.9
InEx-I backboneQwen2.5-VL-7B-Instruct, trained by us (SFT, then RL)App. B
Agent producing RL trajectoriesGPT-4oApp. B.3
Agents producing evaluated trajectoriesClaude 3.5 Sonnet (WebVoyager, WebArena); GPT-5.6 Luna (AndroidWorld); GPT-5.6 Sol (OSWorld)App. E.1, E.4
Table 1 baselinesClaude 3.5 Sonnet, GPT-5.0, Qwen3-VL-8B as direct judges; WebJudge in its released best (7B) configuration, in which its released 7B model scores the screenshots and GPT-5.0 does key-point extraction and the final judgmentSec. 5.1, App. F.4

Appendix E.4 names these agents. InEx uses frontier models in two places only. The first is the training of InEx-I, where GPT-4o writes the SFT traces and drives the agent whose trajectories are judged during RL, and GPT-5.0 judges the rewards of 187 seed tasks. The second is InEx-E, which gives the final verdict. At inference, InEx-I runs on open weights. The GPT-5.0 judge and InEx-E use the same model. See the new Appendix F.13 (Table 18), marked in blue.

AuthorsOpenReview ↗

Response to Reviewer dDPi (2/3)

W2a. Does the Table 1 comparison account for full InEx's extra capability and compute budget?

Giving GPT-5.0 the environment actions and step budget of InEx-E does not close the gap. This frontier I+E baseline has the same frontier model, actions and budget as InEx-E but no trained scaffold. It reaches 86.6 on the macro average, 5.2 points below InEx, although it uses 184.2K API tokens per WebVoyager trajectory and InEx 32.2K. The table lists what each evaluator uses, its accuracy and its cost per trajectory on WebVoyager:

EvaluatorLocal modelFrontier modelEnv. actionsWebVoyager acc. (%)Macro acc. (%)Latency (s)Total tokens (K)API tokens (K)
GPT-5.0 judge (Table 1)–GPT-5.0No81.3±0.781.511.513.813.8
WebJudge (Table 1)Released 7BGPT-5.0No86.7±0.686.315.363.418.3
Frontier I+E w/o scaffold (Table 4)–GPT-5.0Same as InEx-E86.8±1.486.641.3184.2184.2
InEx-I (Table 1)Trained InEx-I (7B)–No90.1±0.389.518.671.30
InEx (Table 1)Trained InEx-I (7B)GPT-5.0Yes91.4±0.391.835.4103.532.2

This baseline is the most expensive evaluator we measured. On the macro average it is also below InEx-I alone, which uses neither a frontier model nor the environment at inference. W2b and W2c below report ProRe under the same interface and this baseline in all four environments. Costs are in Appendix F.4 (Table 9); the baseline's cost and four-environment results are in Appendix F.12 (Table 17), marked in blue.

W2b. How does full InEx compare with interactive evaluators such as Agent-as-a-Judge and ProRe under the same environment interface and interaction budget?

We ran ProRe (Dai et al., 2026), which probes the environment state, with its released code and default settings on AndroidWorld, the only environment its released code targets. As shown in the table below, under the same environment interface, the same frontier model and step caps of the same order, InEx is 8.5 points more accurate than ProRe with its default probing agent, and InEx-I alone, without any environment action, is above both ProRe runs. Agent-as-a-Judge cannot be run as published. It judges code-generation agents against a hand-annotated requirement hierarchy per task, with tools that read the agent's workspace. Our tasks give one instruction and no requirement list, with evidence in a browser, emulator or desktop. W2c reports a generic agentic judge in its place, the frontier I+E baseline. The interactive GUI evaluators of Cui et al. (2026) and Shi et al. (2026), cited in Section 2, are concurrent work; we will add a comparison with them.

Specifically, GPT-5.0, the model of InEx-E, replaces ProRe's retired default reasoner, Gemini-2.5-Pro. Both judge the same 1,000 trajectories from the same restored start states through the official actions. Accuracy in %, mean ± std over three runs:

EvaluatorReasoner / final verdictProbing agentEnv. step budgetAndroidWorld
GPT-5.0 judge (Table 1)GPT-5.0–083.2±0.7
ProReGPT-5.0V-Droid-8B (default)≤ trajectory length (16.6 avg.)86.0±0.7
ProReGPT-5.0M3A (GPT-5.0)≤ trajectory length (16.6 avg.)83.5±1.4
InEx-I (Table 1)InEx-I–092.1±0.3
InEx (Table 1)GPT-5.0InEx-E (GPT-5.0)15 (4.3 avg. when acting)94.5±0.3

ProRe with its default probing agent reaches 86.0%, in line with the 86.2% its paper reports with a GPT-5 reasoner. InEx-I alone reaches 92.1%, so 6.1 of the 8.5 points do not depend on environment interaction. Details are in the new Appendix F.10 (Table 15), and those of the generic agentic judge in Appendix F.12, marked in blue.

AuthorsOpenReview ↗

Response to Reviewer dDPi (3/3)

W2c. Can Table 4's Frontier I+E w/o scaffold control be reported beyond WebVoyager?

Yes. The control now covers all four environments and stays below InEx in each, by 5.2 points on the macro average. In this control, GPT-5.0 inspects the trajectory, gathers environment evidence and gives the verdict without the InEx-I scaffold. It has the same official actions and interaction budget as InEx-E (up to 20 steps, 15 on AndroidWorld). Unlike standalone InEx-E (Table 8), it first carries out the intrinsic stage itself, prompted to follow InEx-I's workflow. Accuracy in %, mean ± std over three runs on the same 1,000 trajectories:

EvaluatorEnv. actionsWebVoyagerWebArenaAndroidWorldOSWorldMacro
GPT-5.0 judge (Table 1)No81.3±0.784.5±0.983.2±0.777.1±1.881.5
Frontier I+E w/o scaffoldYes86.8±1.488.6±2.187.1±0.883.9±2.186.6
InEx-I (Table 1)No90.1±0.391.1±0.792.1±0.384.6±1.389.5
InEx (Table 1)Yes91.4±0.393.0±0.594.5±0.388.2±1.091.8

With environment access, GPT-5.0 beats its direct judge in every environment, but it reaches InEx-I's level only on OSWorld (83.9% against 84.6%). The gain of InEx appears when environment evidence is added on top of the trained scaffold. Details are in the new Appendix F.12 (Table 17), marked in blue.

Reviewer_haMVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces InEx, an evaluator for latently verifiable agent tasks, where objective success conditions exist but ground-truth outcomes are not readily available during evaluation. InEx combines intrinsic evidence reconstruction (InEx-I), which uses a vision-language model trained with SFT and RL to analyze action-conditioned trajectories, with extrinsic evidence acquisition (InEx-E), which uses a frontier model to selectively verify unresolved claims through environment interaction. A scaffolded handoff connects the two components by passing structured intrinsic evidence to the extrinsic evaluator. The authors evaluate InEx on four benchmarks spanning web, mobile, and desktop environments, reporting higher evaluator accuracy than the compared baselines without environment-specific retraining. They also report improvements in downstream agent training when InEx is used as the reward evaluator.

Critical strengths

S1. Well-motivated evaluation problem

The paper addresses an important problem in agent evaluation: determining task success when the necessary evidence is incomplete or distributed across long interaction trajectories. Its emphasis on evidence-grounded verification rather than plausible final answers is well motivated, particularly when evaluator outputs are used as training rewards. The distinction between trajectory-internal evidence and additional environment verification provides a reasonable framework for addressing this problem.

S2. Broad evaluation across diverse agent environments

The experiments cover four benchmarks spanning web, mobile, and desktop interaction, with 1,000 trajectories per benchmark (Table 1). InEx consistently outperforms the evaluated baselines, achieving improvements of 4.5–6.7 percentage points over WebJudge. Importantly, the same trained InEx-I checkpoint is used across all four environments without environment-specific fine-tuning, providing evidence of cross-environment generalization.

S3. Demonstrated benefits for downstream agent training

The paper evaluates the practical utility of its evaluator by using InEx-derived rewards for downstream agent training (Section 5.3). Under the same PAE training framework, replacing the original VLM-as-Judge reward source with InEx improves agent success rates across the evaluated configurations (Table 2). This controlled experiment provides evidence that improved evaluator accuracy can translate into better agent training outcomes, rather than serving only as an evaluation metric.

Critical weaknesses

W1. The contribution of intrinsic evidence reconstruction is not isolated from evaluator post-training

On Qwen2.5-VL-7B, structured prompting improves accuracy only from 71.6% to 72.8%, whereas SFT and RL raise it to 85.2% and 90.1%, respectively (Table 3). These results demonstrate the effectiveness of the trained InEx-I pipeline but do not establish that step-wise, action-conditioned evidence reconstruction is preferable to simpler evaluator designs under comparable training. WebJudge does not resolve this attribution issue because it is not a backbone- and training-matched control. A more controlled comparison would train a direct binary judge or a generic CoT evaluator using the same backbone, training data, and SFT/RL budget.

W2. Missing comparisons with existing interactive evaluation methods

Although Section 2 discusses Agent-as-a-Judge and recent GUI evaluators that employ environment-grounded verification, the main experiments compare InEx only against generic judges and the trajectory-only WebJudge (Table 1). The frontier I+E without scaffold (Table 4) and standalone InEx-E (Table 8) ablations provide useful internal controls but do not compare against existing interactive evaluators. Consequently, it remains unclear whether InEx improves upon prior approaches that already acquire or verify evidence through environment interaction. The paper should include comparisons with representative interactive evaluators or explain specific benchmark incompatibilities and provide adapted comparisons where feasible.

W3. Ambiguous distinction between latent and automatic verifiability

The paper defines latently verifiable tasks as having an objective ground truth that is not cheaply available during evaluation (Section 3). However, WebArena, AndroidWorld, and OSWorld provide official task evaluators that determine reference outcomes, while these evaluators and their privileged information are withheld from the evaluated models (Section 5.1, Appendix E.4). This raises the question of whether latent verifiability is an intrinsic task property or merely an evaluator-access constraint. If the latter, the paper should provide clearer operational criteria distinguishing this setting from automatically verifiable tasks whose existing checkers are simply withheld. Otherwise, almost any automatically verifiable task could be made latently verifiable by restricting evaluator access, weakening the conceptual significance of the proposed task class.

Questions for authors

Please check weakness section above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer haMV (1/2)

We sincerely thank the reviewer for recognizing the well-motivated evaluation problem, the broad evaluation across four environments with a single reused InEx-I checkpoint, and the demonstrated benefits for downstream agent training. We address the questions below.

W1. Is step-wise evidence reconstruction better than a direct binary or generic CoT judge trained with the same backbone, data and SFT/RL budget?

To separate the evaluation design from training, we ran two controls. In the first, GPT-5.0 follows our step-wise intrinsic-evidence workflow by prompt, without training. In the second, a generic CoT judge (free-form reasoning, then a verdict) is trained from the same Qwen2.5-VL-7B-Instruct checkpoint as InEx-I, with the same SFT examples and RL post-training. Only its SFT targets differ. Each trace is replaced by a CoT judgment that ends in the example's reference verdict, written by GPT-5.0 with this verdict in view. We trained the CoT judge and not a direct binary judge. Accuracy in %, mean ± std over three runs:

EvaluatorTrainingWebVoyagerWebArenaAndroidWorldOSWorldMacro
GPT-5.0 direct judge (Table 1)None81.3±0.784.5±0.983.2±0.777.1±1.881.5
GPT-5.0 + intrinsic-evidence workflow (new)None83.5±0.585.8±1.585.4±1.580.3±0.683.8
Qwen2.5-VL-7B CoT judge (new)SFT78.4±0.781.6±1.285.4±0.873.2±1.579.6
Qwen2.5-VL-7B CoT judge (new)SFT+RL80.3±1.682.2±0.686.7±0.375.6±4.981.2
InEx-I (Table 1)SFT+RL90.1±0.391.1±0.792.1±0.384.6±1.389.5

The CoT judge reaches 80.3% on WebVoyager, against 90.1% for InEx-I. The gap is already there after SFT alone, with 78.4% against 85.2% (Table 3). This control was trained with the same SFT examples and RL post-training as InEx-I yet ends far below it, while the workflow alone improves an untrained frontier model in all four environments. The setup is in the new Appendix F.11 (Table 16), marked in blue.

W2. How does InEx compare with existing interactive evaluators, such as Agent-as-a-Judge and recent GUI evaluators?

We compared InEx with two interactive evaluators. The first is ProRe (Dai et al., 2026), a GUI evaluator that probes the environment, run with its released code and default settings. The second is a generic agentic judge, the frontier I+E baseline of Table 4, which has the same frontier model, actions and budget as InEx-E. As shown in the table below, InEx is 8.5 points more accurate than ProRe with its default probing agent on AndroidWorld and, on the macro average over all four environments, 5.2 points above the agentic judge. Agent-as-a-Judge cannot be run on our benchmarks as published: it judges code-generation agents against a manually annotated hierarchy of requirements per task, with tools that read the agent's workspace, while our tasks give one instruction and no requirement list, with evidence in a browser, emulator or desktop. The interactive GUI evaluators of Cui et al. (2026) and Shi et al. (2026), cited in Section 2, are concurrent work; we will add a comparison with them.

Specifically, ProRe's released code covers AndroidWorld only, and GPT-5.0, the model of InEx-E, replaces its retired default reasoner, Gemini-2.5-Pro. The step caps of ProRe and InEx-E are of the same order (Appendix F.10). The generic agentic judge stands in place of Agent-as-a-Judge and is run in all four environments. In it, GPT-5.0 carries out the intrinsic stage itself, prompted to follow InEx-I's workflow, and then the extrinsic stage with the same actions and budget as InEx-E. The trained InEx-I is not used. All interactive evaluators start from the same restored states and use the official actions. Accuracy in %, mean ± std over three runs on the same 1,000 trajectories:

EvaluatorFrontier modelEnv. actionsWebVoyagerWebArenaAndroidWorldOSWorldMacro
GPT-5.0 judge (Table 1)GPT-5.0No81.3±0.784.5±0.983.2±0.777.1±1.881.5
ProRe, V-Droid-8B (default)GPT-5.0Yes––86.0±0.7––
ProRe, M3A (GPT-5.0)GPT-5.0Yes––83.5±1.4––
Frontier I+E w/o scaffoldGPT-5.0Yes86.8±1.488.6±2.187.1±0.883.9±2.186.6
InEx-I (Table 1)noneNo90.1±0.391.1±0.792.1±0.384.6±1.389.5
InEx (Table 1)GPT-5.0Yes91.4±0.393.0±0.594.5±0.388.2±1.091.8

ProRe with its default probing agent reaches 86.0%, in line with the 86.2% its paper reports with GPT-5, and InEx-I alone, without environment actions, is above both ProRe runs. The frontier I+E baseline beats the direct judge but stays below InEx in all four environments. InEx's gain therefore comes from adding environment evidence to the trained scaffold, not from interaction alone. See Appendix F.10 (Table 15) and the new Appendix F.12 (Table 17), marked in blue.

AuthorsOpenReview ↗

Response to Reviewer haMV (2/2)

W3. Is latent verifiability a task property or an evaluator-access constraint, and what separates it from automatically verifiable tasks whose checkers are withheld?

Latent verifiability is not merely an evaluator-access constraint. Our definition (Section 3) asks whether the ground truth is cheaply available during evaluation. What matters is the cost of obtaining a check, not only who may see an existing one. Where checkers exist, they took manual work on a fixed set of curated tasks. OSWorld (Xie et al., 2024) reports about 1,800 person-hours for building its 369 tasks, and about 2 person-hours per task for developing and checking the evaluation script alone. WebArena's validators and AndroidWorld's success-checking logic were also hand-written per task. WebVoyager and Online-Mind2Web, on live websites, provide no checkers. Only 22.3% of WebVoyager tasks have a fixed golden answer (He et al., 2024), and its main metric is human evaluation. Online-Mind2Web (Xue et al., 2025) relies on human annotation because a fixed ground truth does not always exist. For open-ended or changing outcomes, a fixed golden answer cannot decide success. In our own training data, we hand-wrote rule-based checkers for 113 of the 300 WebVoyager RL seed tasks. For the other 187, the reference answer alone does not decide correctness, so a GPT-5.0 judge assesses each outcome against it (Appendix B.3).

Withholding a checker is therefore not what makes a task latently verifiable. A task with a cheap check, such as the many math and coding problems checked by a rule, a solver, execution or existing unit tests, stays automatically verifiable whether or not the evaluator sees the check. Some web, mobile and desktop benchmarks hand-write checks for their curated tasks; WebVoyager has none. Our evaluation targets the case of new tasks without a cheap check, for which writing one per task does not scale. Withholding the official evaluators of WebArena, AndroidWorld and OSWorld emulates it on benchmark instances, and the evaluators then serve as trusted references. The revised Section 3 states this criterion, marked in blue.

Dear Reviewer haMV,

Thank you again for your thoughtful review. As the discussion period is nearing its end, we wanted to kindly follow up on our rebuttal. When time permits, we would greatly appreciate it if you could review our response and let us know if you have any further questions or comments. We would be happy to provide any additional clarification.

Thank you for your time.

Best,Authors

Reviewer_Ws7VOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes InEx for evaluating agent tasks whose success is objective but not cheaply verifiable. InEx-I trains a vision-language model to inspect local observation-action transitions and aggregate them into an evidence chain. A frontier-model component, InEx-E, receives this analysis for every instance and selectively interacts with the environment before issuing the final verdict. The authors evaluate the system on web, mobile, and desktop trajectories and report improvements in evaluator accuracy and downstream agent training.

Critical strengths
  1. The distinction between evidence already present in a trajectory and evidence that requires further interaction is useful. Inspecting local transitions can expose unsupported claims that a final-answer judge misses, while the handoff provides a concrete way to guide further verification toward unresolved claims.

  2. The evaluation design addresses several important concerns. The same trained intrinsic checkpoint is reused across four environments, Appendix E.2 specifies task-level exclusion between training and WebVoyager evaluation, and the reward diagnostics distinguish false positives from false negatives. The downstream training comparison also connects evaluator quality to a practical use of the reward signal.

Critical weaknesses
  1. Table 6 is inconsistent with the stated aggregation protocol. Appendix E.5 groups the websites into five fixed categories and computes Overall over all trajectories, so every method should share the same group weights. Using the displayed percentages, the combination 2 × GPT-5.0 + WebJudge − 4 × Claude ranges from −63.5 to −62.7 across the five groups, but equals −65.5 in the Overall column. A weighted average cannot lie outside that range, and rounding each reported value to one decimal place cannot explain the discrepancy. The group counts and per-run correct counts are needed to reconcile these results before relying on the reported comparison. This identifies a reporting inconsistency; it does not establish which entries are incorrect.

  2. The intrinsic evaluator's training procedure is not sufficiently specified. Appendix B.4 writes the sample weight as Value(R, c, ρ), but does not define this function, how the task-distribution factor is calculated, or the actual RL update represented by Eq. (13). Appendix B also omits the training-set sizes and mixture proportions, optimization settings, and exact frontier models used for supervision. The gain from SFT to RL therefore cannot be reproduced from the provided description, and Table 3 does not isolate whether confidence weighting contributes beyond ordinary RL with the same supervision. The promised release would help, but these are central method details needed to assess the present submission.

Questions for authors
  1. Appendix D.3 clarifies that the frontier model issues the final verdict on every instance, including those without environment actions. With the trained intrinsic scaffold and frontier model fixed, how does a configuration with environment actions disabled compare with full InEx? The current 90.1% to 91.4% comparison changes both the final judge and access to external evidence, so it does not isolate the value of interaction.

  2. How is the agent's terminal environment state preserved or restored before each evaluator run, and how do you prevent probing actions from completing or modifying the task being assessed? Appendix E.4 fixes the reference outcomes and Appendix C restricts interaction to official actions, but the latter still includes state-changing operations. The relevant verdict should concern the submitted agent's outcome rather than a state subsequently produced by the evaluator.

  3. Section 2 already cites ProRe [1], which combines trajectory-derived claims with evidence collected through targeted state probing. How does InEx compare with this approach on a shared mobile setting when evaluator backbones, permitted observations/actions, and interaction budgets are controlled? This would help establish the benefit of the trained intrinsic scaffold relative to an existing interactive evaluation design, beyond the generic frontier I+E baseline in Table 4.

[1] Dai G, Jiang S, Cao T, et al. ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaboration[C]//International Conference on Learning Representations. 2026.

7 comments
AuthorsOpenReview ↗

Response to Reviewer Ws7V (1/4)

We sincerely thank the reviewer for the careful check of Table 6 and for recognizing the distinction between trajectory evidence and evidence that requires further interaction, the reuse of one trained InEx-I checkpoint across four environments with task-level exclusion between training and WebVoyager evaluation, and the reward diagnostics that separate false positives from false negatives. We address the questions below.

W1. Why is Table 6 inconsistent with the aggregation protocol of Appendix E.5?

The inconsistency comes from two different averaging targets, and we thank the reviewer for pointing it out. In Tables 6 and 8, each group value was an average over the websites of its group, while the Overall value is an average over all 1,000 trajectories. Averages over websites and over trajectories weigh the data differently, so the group columns did not combine into the Overall column. The reported numbers are correct. Only the group columns of these two tables used website averages, and no number in the main text is affected. We now use one target throughout. Every group value is computed over all trajectories of its group, like Overall. The group values move by at most 1.8 points (GPT-5.0 on Knowledge/Learning, 85.2 to 83.4) and by at most 0.3 for InEx-I and InEx, and no Overall value changes (accuracy in %, mean ± std over three runs; group sizes in the header):

EvaluatorCommerce/Services (200)Knowledge/Learning (340)Developer Platforms (135)Search/Navigation (130)News/Sports (195)Overall (1,000)
Claude 3.5 Sonnet75.3±0.880.5±1.481.5±0.782.8±1.274.2±0.378.7±0.8
GPT-5.076.5±0.983.4±1.186.7±0.788.2±0.974.2±0.381.3±0.7
Qwen3-VL-8B74.5±2.076.1±2.475.8±1.576.7±2.773.3±1.575.3±1.9
WebJudge83.5±0.588.8±0.888.1±0.089.5±0.983.6±0.586.7±0.6
InEx-I87.2±0.392.6±0.591.4±0.492.1±0.486.3±0.390.1±0.3
InEx-E (Table 8)81.7±0.384.6±0.285.4±0.486.7±0.480.2±0.383.5±0.1
InEx89.7±0.393.5±0.692.1±0.492.6±0.488.2±0.091.4±0.3

Per-run correct counts (run 1 / run 2 / run 3):

EvaluatorCommerce/Services (200)Knowledge/Learning (340)Developer Platforms (135)Search/Navigation (130)News/Sports (195)Overall (1,000)
Claude 3.5 Sonnet151 / 152 / 149272 / 279 / 270110 / 111 / 109108 / 109 / 106145 / 144 / 145786 / 795 / 779
GPT-5.0154 / 154 / 151288 / 281 / 282118 / 117 / 116116 / 114 / 114145 / 145 / 144821 / 811 / 807
Qwen3-VL-8B153 / 145 / 149267 / 251 / 258103 / 100 / 104100 / 96 / 103146 / 140 / 143769 / 732 / 757
WebJudge166 / 168 / 167300 / 305 / 301119 / 119 / 119115 / 117 / 117162 / 164 / 163862 / 873 / 867
InEx-I174 / 174 / 175314 / 314 / 317123 / 124 / 123119 / 120 / 120168 / 168 / 169898 / 900 / 904
InEx-E163 / 163 / 164287 / 288 / 288115 / 115 / 116113 / 113 / 112157 / 156 / 156835 / 835 / 836
InEx180 / 179 / 179320 / 318 / 316124 / 125 / 124121 / 120 / 120172 / 172 / 172917 / 914 / 911

With trajectory-weighted groups, the reviewer's combination 2 × GPT-5.0 + WebJudge − 4 × Claude ranges from −66.4 to −64.5 across the groups. Its Overall value of −65.5 lies inside that range. InEx-I and InEx remain above WebJudge in every group, and InEx remains above InEx-I.

Tables 6 and 8, their captions and Appendix E.5 now use and state trajectory weighting. The per-run counts (Table 11) and the websites of each group (Table 12) are in the new Appendix F.7, marked in blue.

AuthorsOpenReview ↗

Response to Reviewer Ws7V (2/4)

W2a. Can the InEx-I training procedure be fully specified?

We thank the reviewer for the suggestion; the revised Appendix B now defines the Value function, the task-distribution factor and the RL objective, and gives the data sizes, optimization settings and supervision models:

ComponentSpecification
SFT data942 verified traces (GPT-4o traces on 1,200 human-labeled PAE trajectories, kept if the verdict matches) + 5,000 synthetic negatives (2,000 step perturbations, 3,000 from the non-interactive generator); 5,942 in total
RL data300 WebVoyager seed tasks, all excluded from the evaluation set; trajectories from a GPT-4o agent; evaluator rollouts on-policy, starting from the SFT checkpoint
Reward labels187 seed tasks with reference answers: GPT-5.0 judge, giving a binary reward R and confidence c ∈ [0, 1]; the remaining 113: rule-based checkers, c = 1
Sample weight (Eq. 12)Value = (1 + λR) · c · ρ with λ = 0.2, per rollout; normalized over the batch, stop-gradient
Task factor ρPer website: target share / share in the batch, clipped to [0.5, 2]; target share ∝ 1 / accuracy of the SFT checkpoint on that website's SFT data
RL update (Eq. 13)GRPO; 16 rollouts per trajectory; 32 trajectories (512 rollouts) per update; clip 0.2; KL 0.04 to the SFT checkpoint
OptimizationQwen2.5-VL-7B-Instruct, full fine-tuning; SFT learning rate 2×10⁻⁵, RL 10⁻⁶; default context length; screenshots at native resolution; SFT: roughly one pass over its data (about 10⁸ tokens); RL: each seed task used about 30 times

The weight counts checker labels fully, scales judge labels by the judge's confidence, and grows with the reward. The factor ρ shifts training toward the websites where the SFT checkpoint is weakest, within fixed clipping bounds. The update is GRPO with a KL penalty to the SFT checkpoint. The weight scales each rollout's clipped objective and leaves the KL term unweighted. Supervision comes from GPT-4o traces for SFT and from hand-written checkers or a GPT-5.0 judge for RL. Details are in Appendix B (B.1–B.4), marked in blue.

W2b. Does confidence weighting help beyond ordinary RL with the same supervision?

To isolate the contribution of the Value weighting, we ran the control: the same RL post-training with every sample weight set to 1, without the Value weighting of Eq. (12). Everything else is identical (SFT checkpoint, seed tasks, reward labels, GRPO settings, training length). Accuracy in %, mean ± std over three evaluation runs:

ConfigurationWebVoyagerWebArenaAndroidWorldOSWorldMacro
SFT only85.2––––
+ RL, uniform weights (new)88.5±0.988.7±0.790.6±0.184.0±1.688.0
+ RL, Value weights (InEx-I)90.1±0.391.1±0.792.1±0.384.6±1.389.5

Ordinary RL with the same supervision gives most of the WebVoyager gain over SFT, and the Value weighting adds 1.6 points there. The weighted model is ahead in all four environments, by 1.5 points on the macro average. On OSWorld the difference (0.6) lies within run-to-run variation. See Appendix F.8 (Table 13) and the added row in Table 3, marked in blue.

AuthorsOpenReview ↗

Response to Reviewer Ws7V (3/4)

Q1. How does full InEx compare with the same InEx-I scaffold and frontier model when environment actions are disabled?

We ran this control. With the same GPT-5.0 model, inputs and InEx-I scaffold but with environment actions disabled (action space {stop}), InEx stays within run-to-run variation of InEx-I alone in all four environments. The gain of full InEx therefore appears only when environment actions are enabled. Accuracy in %, mean ± std over three evaluation runs:

ConfigurationFinal verdictEnv. actionsWebVoyagerWebArenaAndroidWorldOSWorldMacro
Frontier I+E, no InEx-I scaffold (Table 4)GPT-5.0Yes86.8±1.488.6±2.187.1±0.883.9±2.186.6
InEx-I onlyInEx-INo90.1±0.391.1±0.792.1±0.384.6±1.389.5
InEx, no actions (new)GPT-5.0No90.4±0.891.4±0.892.1±0.784.5±0.989.6
InExGPT-5.0Yes91.4±0.393.0±0.594.5±0.388.2±1.091.8

Enabling actions adds 2.2 points on the macro average. The complementary control is the frontier I+E baseline, with the same frontier model and environment access but no trained scaffold. It is below InEx in all four environments. See the new Appendix F.9 (Table 14), Appendix F.12 (Table 17) and the added row in Table 4, marked in blue.

Q2. How is the agent's terminal environment state preserved or restored before each evaluator run, and how are probing actions kept from completing or modifying the assessed task?

We restore the state by replay. Before every interactive evaluator run, the agent's recorded actions are replayed on a fresh environment instance. Each step is checked against the stored trajectory, using the environment state where the benchmark exposes it and the screenshot otherwise. A failed replay is repeated, up to five attempts. The evaluation sets hold only trajectories restorable this way.

Reference outcomes are fixed at collection (Appendix E.4) and never recomputed, and the stored trajectory is never modified. Probing can therefore change neither of them. State-changing official actions are not hard-blocked. They are limited by the action space, the step budget and the instruction to verify unresolved claims rather than rerun the task. GPT-6 Luna screened 1,000 InEx-E runs with environment actions (250 per environment), read each run's full context and found no case of the evaluator completing or modifying the task. See the new Appendix C.6 and the revised Appendix E.1, in blue.

Reviewer_5VTdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes InEx, an evaluator for agent tasks whose success conditions are objective but not cheaply accessible during evaluation. InEx-I uses a trained vision-language model to inspect local observation–action transitions and construct an intrinsic evidence chain. This analysis is passed to InEx-E, a frontier-model evaluator that can interact with the environment to obtain missing evidence before producing a final verdict. The same intrinsic evaluator is tested across web, mobile, and desktop benchmarks without environment-specific fine-tuning. The paper reports higher evaluation accuracy than the compared baselines, fewer false-positive success judgments on WebVoyager, and improvements in downstream web-agent training when InEx supplies reward labels.

Critical strengths

The paper addresses a practical evaluation problem with a well-motivated combination of evidence sources. Inspecting recorded transitions can reveal unsupported completion claims, while additional environment interaction can resolve questions that the submitted trajectory cannot answer. The distinction is clear and useful for evaluating interactive agents.

Critical weaknesses

The practical cost–benefit trade-off is insufficiently evaluated. Appendix F.4 reports the frequency and interaction latency of extrinsic verification on WebVoyager, but does not report end-to-end evaluation latency or total inference cost relative to InEx-I and WebJudge.

Questions for authors

What is the total evaluation cost relative to the baselines, including intrinsic analysis and frontier-model processing for cases with no environment actions?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5VTd

We sincerely thank the reviewer for recognizing the practical problem InEx addresses and the clear distinction between recorded and newly acquired evidence.

W1/Q1. What is InEx's end-to-end cost relative to InEx-I and the baselines?

On WebVoyager, InEx averages 35.4 s and 32.2K API tokens per trajectory, against 18.6 s and no API tokens for InEx-I alone and 15.3 s and 18.3K for WebJudge. The table reports latency and tokens for every evaluator of Table 1 on the 1,000 trajectories:

EvaluatorWebVoyager acc. (%)Latency (s)Total tokens (K)API tokens (K)
Claude 3.5 Sonnet78.710.111.211.2
GPT-5.081.311.513.813.8
Qwen3-VL-8B75.36.88.00
WebJudge86.715.363.418.3
InEx-I90.118.671.30
InEx, no interaction (76.4%)28.588.817.5
InEx, with interaction (23.6%)57.6151.279.9
InEx, all trajectories91.435.4103.532.2

Values are per trajectory, tokens in thousands; API tokens are sent to external APIs, the rest come from self-hosted models. The latencies exclude the time for restoring the environment state by replay, which was not measured; only the interactive evaluators need this replay (Appendix C.6).

InEx-I is close to WebJudge in latency and tokens (18.6 s against 15.3 s, 71.3K against 63.4K tokens) and makes no API calls. On roughly three quarters of trajectories without environment action, InEx takes about ten seconds longer than InEx-I alone. InEx is the most expensive evaluator in the table. The extra cost buys the highest accuracy and fewer false-positive successes than InEx-I alone (success precision rises from 93.7 to 97.6). This matters when verdicts are used as training rewards. Details are in Appendix F.4 (Table 9), marked in blue.

Dear Reviewer 5VTd,

Thank you again for your thoughtful review. As the discussion period is nearing its end, we wanted to kindly follow up on our rebuttal. When time permits, we would greatly appreciate it if you could review our response and let us know if you have any further questions or comments. We would be happy to provide any additional clarification.

Thank you for your time.

Best,Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers, the AC, and the SAC for the constructive feedback and discussion. After the rebuttal, Reviewers 5VTd and haMV remain positive, and Reviewer Ws7V raised the rating to weak acceptance. Reviewer dDPi's follow-up concerns novelty, which was not raised in the review, and we have clarified our contributions in our reply.

We are grateful for the recognition of our work's strengths:

  • (5VTd, Ws7V, haMV, dDPi) The distinction between intrinsic and extrinsic evidence is clear and useful.
  • (Ws7V, haMV, dDPi) The same trained InEx-I checkpoint is reused across web, mobile and desktop environments without fine-tuning.

The key issues addressed in the rebuttal are:

  • Comparison with existing interactive evaluators (Ws7V, haMV, dDPi). With the same frontier model and environment actions, InEx is 8.5 points more accurate than ProRe on AndroidWorld and 5.2 points above a generic agentic judge on the macro average over four environments.
  • Design versus training (haMV). A CoT judge with the same checkpoint, examples and RL post-training reaches 81.2% on the macro average, against 89.5% for InEx-I.

We also added the training details, the evaluation cost, the controls without the Value weighting and without environment actions, and the state-restoration procedure, and clarified the definition of latent verifiability.

We have incorporated all new results into the revision. We welcome further feedback and will incorporate the remaining discussions in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.