acceptodds
Withdrawn from ICLR 2027

VRC: Visual Reward Consistency in GUI-Agent Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual reward models can classify recorded trajectories accurately yet provide weak feedback for policy learning. We examine this gap in graphical user interface (GUI) reinforcement learning using paired visual evidence on BrowserGym/MiniWoB. Four data-matched verifier objectives compare binary cross-entropy with invariance and failure-sensitivity losses. A separate policy experiment compares a BCE verifier's single-view reward with five-view soft-min scoring from the combined-loss verifier. Across three seeds, sensitivity supervision achieved the highest validation AUROC, while BCE had the lowest banner inconsistency. On 2,160 evaluation episodes across 24 task families, BCE reward achieved 33.06% success and the soft-min reward achieved 32.22%. Neither the original comparison nor a post-hoc crossed seed-task analysis resolved an improvement from soft-min. Most held-out successes came from one task family, and the soft-min reward showed weak separation between successful and failed policy trajectories. The policy experiment changes verifier training and aggregation together and excludes the sensitivity-only verifier. The observed result concerns this reward combination; the downstream value of sensitivity supervision remains open.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, GUI agents, visual reward models, reward consistency, reward evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_uydqOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an ablation study of visual reward model training, comparing four training objectives using ~2k training examples:

  1. BCE 2. BCE + Invariance Loss 3. BCE + Sensitivity Loss 4. BCE + Invariance + Sensitivity. The experiments show that BCE + Sensitivity Loss achieves the best offline validation performance. However, the authors do not evaluate every variant in downstream RL. I think this paper‘s novelty is quite low and experiments lack information.
Critical strengths
  1. The paper provides a controlled comparison of different visual reward model training objectives using the same training data.
  2. Paper is easy to understand.
Critical weaknesses
  1. Lack of Novelty: The paper primarily combines well-established techniques, such as BCE, invariance regularization, and sensitivity-based ranking losses, and conducts ablation studies on their combinations. I did not gain any substantial new insights from the experiments.
  2. Limited Training Data: The paper uses only 600 original trajectories, augmented into 2,229 training examples, to train the visual reward models. This dataset is relatively small, especially for drawing general conclusions about visual reward model training.
  3. Unconvincing Conclusions: I find the overall conclusions of the paper unconvincing. The experiments are conducted on a relatively small dataset, and the incomplete downstream ablations make it difficult to draw meaningful conclusions about the effectiveness of different training objectives.
  4. Poor Figure Quality: The figures seem to be generated by AI and lack professional polish.
Questions for authors
  1. Incomplete Downstream Ablations: Why was the BCE + Sensitivity Loss variant, which achieved the best offline validation performance, not evaluated in downstream RL?
  2. Limited Training Data: The reward models are trained using only 600 original trajectories augmented into 2,229 examples. How sensitive are the conclusions to the training data scale?
Reviewer_HreaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies visual reward consistency in reinforcement learning for GUI agents. It trains verifiers with invariance and failure-sensitivity losses and uses multi-view soft-min aggregation to produce policy rewards. Experiments on MiniWoB compare four verifier objectives and evaluate two reward configurations downstream. Failure-sensitivity supervision improves offline discrimination, but the combined method does not yield a statistically resolved improvement in policy success.

Critical strengths
  1. The offline ablation is reasonably controlled. All four objectives use the same augmented data, helping distinguish the contribution of auxiliary losses from that of data augmentation under the tested setup.

  2. Outcome evaluation and reporting are transparent. Final success is measured by an independent validator. Negative results, seed variability, and task-level outcomes are reported, supporting a realistic assessment of the method.

Critical weaknesses
  1. The novelty and overall contribution are limited. The method primarily combines classification, consistency regularization, margin losses, and conservative aggregation, with limited novelty in the individual components. As an empirical study, it also lacks sufficient controlled evidence to explain the failure mechanisms, leaving relatively little generalizable insight.

  2. Key experiments are missing. The downstream comparison changes both the verifier objective and aggregation method, while omitting the sensitivity-only verifier that performs best offline. Validation negatives are entirely constructed, with no common evaluation on natural failures. Experiments are restricted to a small MiniWoB setting, without validation in realistic application environments such as OSWorld, limiting external validity.

Questions for authors
  1. Can you provide comparisons that hold the verifier fixed while varying aggregation, together with downstream results using sensitivity-only rewards?

  2. How do the four verifiers perform on a shared set of natural policy failures? To what extent does the high validation AUROC depend on simple cues introduced by negative-example construction?

  3. Is there evidence from realistic application environments supporting the applicability of these observations beyond the current MiniWoB setting?

Reviewer_afdAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies different verifier designs for reinforcement learning in GUI-agent environments. It outlines a series of different conditions to use verifiers or aggregated rewards, and measures downstream GRPO performance with a detailed task-category breakdown.

Unfortunately, I find there are significant issues with the writing, missing baselines and models for comparison, unjustified design choices, and limited results (both in terms of scope and relevance). Most apparent, the paper is not written with a sequential reader in mind, i.e., many terms are not introduced (but assume knowledge of the full paper). I don’t want to accuse the authors of anything without evidence, but this paper has the hallmarks of a largely or fully AI-generated paper, but without the care to make the paper comprehensible to an actual human reader.

Critical strengths
  • Reporting is honest and detailed
  • The general problem setting is relevant, and the paper is framed as a tightly scoped contribution
Critical weaknesses
  • In general, i found the paper difficult to follow, with akward writing, unintutive structure, symbols that are not introduced, etc. This made it somewhat difficult to properly review. Examples:
    • The abstract is actually very hard to read - metrics, names of verifier conditions are thrown together, i would suggest to rewrite, and use cursive writing or capitalize to distinguish verifier objectives. Example: „Neither the original comparison nor a post-hoc crossed seed-task analysis resolved an improvement from soft-min“ is very hard to parse, "post-hoc cross-seed task analysis" makes sense in context (i.e. when knowing the entire paper), but fails it's purpose in an abstract that should summarize the paper for a reader
    • L55: „Explicit invariance did not improve banner consistency over augmented-data BCE, while sensitivity supervision improved validation discrimination“ again borders on incomprehensible. At this point of the paper, it is unclear what „explicit invariance“, „banner consistency“, „augmented-data BCE“, „sensitivity supervision“ or „validation discrimination“ mean
    • Caption of Figure 1: „Four verifier objectives use matched paired evidence: banner pairs retain labels; stale-initial and unsuccessful-prefix records provide changing pairs“ - I assume „banner pair retain labels“ is a verifier objective..but difficult to parse
    • Some symbols are not introduced, e.g. line 158, z_theta or p_theta(e) are not explained, in fact z is never introduced or explained
  • The proposed methodology seems quite simple, just uses a single policy and training environment. It is unclear how generalizable these findings are
  • The work does not compare against baselines from related work
  • I feel there is only a weak empirical result, the proposed interventions do not yield an advantage over baseline binary rewards. While this is fair, and i appreciate the honesty of the authors, this does not make it a convincing contribution
  • Many design decisions are not justified or properly ablated, (e.g.
    • no goal input to the verifier seems like a weird choice, and is not properly justified
    • choice of additional verifier objectives (L_inv, L_sens)
    • the choice of terminal views
    • cropping with just a 2px window
    • the beta-factor in in reward aggregation
  • There are some issues with the experiments:
    • Confounding of verifier objectives and aggregation
    • Validation (Table 1) seems really saturated (in terms of AUROC), showing small variance between objectives, there are a lot of reported metrics, but few values are interpreted
    • not clear if the learned separation reward function is effective

Minor: There are weird line-breaks, not sure which setting is used, e.g. already in the first line, the line break „application-„ „s“ is weird

Questions for authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.