acceptodds
Under review as a conference paper at ICLR 2027

SeekJudge: A Practical Reward Framework for Reinforcement Learning in Computer-Use Agents

Abstract

Deciding whether a trajectory actually fulfills its instruction determines how we measure computer-use agents on long-horizon graphical-user-interface tasks and how we train them with reinforcement learning (RL). This judgment has long relied on rule-based evaluation. Rule-based evaluation often disagrees with human intention and becomes outdated when an app updates or its online content drifts. Existing model-based judges attempt to address these problems, but their judging accuracy remains limited. We propose the SeekJudge framework, whose four agents, a Condense, a Ground, a Seek and an Analyze agent, reach a verdict through a Seek–Analyze loop over the trajectory. To train a specialized model on densely labeled trajectories, we propose a seed-driven distillation pipeline that expands a few human-labeled seed trajectories into K such trajectories. To evaluate step-level judgments, we build CUAStepBench, a human-annotated benchmark that pairs trajectory verdicts with dense step labels. Beyond accuracy, SeekJudge costs far less than a closed-source large model. We further propose rollout overlap, a reward-server design that reduces the overhead of a reward model in RL training. To our knowledge, SeekJudge is the first model-based reward to match or surpass native rule-based reward in online RL, measured by downstream success rate on held-out RL test goals. In offline judging, SeekJudge-9B also exceeds rule-based evaluation by F1 on AgentRewardBench.

Primary area:
foundation or frontier models, including LLMs
Keywords:
computer-use agents, LLM-as-a-judge, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jgbDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SeekJudge, a model-based reward/judge framework for computer-use agents (CUAs). The authors decompose long-trajectory judging into localization and extraction, and implement it with four agents sharing a 9B backbone (Condense, Ground, Seek, Analyze): Condense and Ground compress the trajectory into a text timeline, and the Seek–Analyze loop queries screenshot evidence on demand. For training, they use seed-driven distillation, where teacher models such as DeepSeek-V3.2 and Gemini-3.0-Flash generate supervision data, followed by SFT into Qwen3.5-9B. The model outputs 7-dimensional trajectory scores and 9-class step labels, which are mapped by a regressor to trajectory scores and step scores for offline evaluation and online RL rewards. The authors also build CUAStepBench and propose a rollout-overlapped reward server. Experiments include GRPO on three OSWorld domains—Chrome, Impress, and OS—as well as three offline reward benchmarks.

Critical strengths
  • The problem is important. Reward modeling for long CUA trajectories is indeed a key bottleneck in current RL training, and how to combine step-level and trajectory-level signals is worth studying.
  • The system is relatively complete. The paper not only proposes a judge but also includes a benchmark, a reward server, online RL, and efficiency analysis, representing substantial engineering effort.
  • The rollout overlap design has practical value. Table 5 shows that overall overhead drops from 18.3s to 7.9s, a reduction of about 56.8%, which is useful for RL training scheduling.
  • The cost and context analysis is detailed. Although its assumptions still need verification, it at least provides a discussable engineering direction.
Critical weaknesses

1、 Limited novelty, the core is an engineering combination

Ideas such as multi-agent systems, tool use, PRMs, step-level rewards, and trajectory compression have already appeared extensively in prior work. The main contribution here looks more like combining these components into a system with teacher distillation. The 7-dimensional scoring and 9-class step taxonomy are largely heuristic definitions, lacking theoretical grounding or sufficient human-consistency validation.

2、 Experimental comparisons are not sufficiently fair

SeekJudge’s training relies on distillation from strong teachers such as DeepSeek-V3.2, Gemini-3.0-Flash, and Codex GPT-5.5, whereas baselines such as CUAJudge use only GPT-5-mini and OS-Themis uses Qwen3-VL-8B, so the resources and supervision signals are clearly unequal. When compared with rule-based reward, the rule reward requires no additional distillation, human annotation, or teacher calls, while SeekJudge uses a large amount of extra supervision. In Table 2, the average test success improvement is about 3.3 points with a standard error of about 1.15, i.e., about 2.9 SE, which is not overwhelming; moreover, on Qwen3-VL-8B in the OS domain, SeekJudge reaches 24.17, lower than rule-based reward’s 26.67, indicating that the improvement is unstable.

3、 Offline evaluation evidence is weak and carries bias and contamination risks

CUAStepBench is built by the authors. Although the paper claims leakage control, only about 20 trajectories are used for criteria prompt search, and no inter-annotator agreement (IAA) is reported. In Table 3, SeekJudge-9B’s step-level F1 is only 38.0, essentially on par with CUAJudge’s 37.9 and far below the human expert’s 82.2, showing that step-level judgments are not reliable. In Table 4, on OmniGUIRewardBench, the Rule baseline is 100.0 while SeekJudge-9B is 85.1, indicating that it still clearly underperforms rule-based rewards on tasks where rules can be defined. Yet the authors generalize the conclusion to replacing rule-based reward, which is not sufficiently supported. In addition, teacher models may have already seen public benchmarks, and the paper does not sufficiently perform n-gram, embedding, or task-level decontamination verification.

4、 RL experiments are small-scale, but the conclusions are too strong

Each domain has only 12–24 training/test tasks, and training is 75 steps, so both task quantity and diversity are limited. Table 11 shows that SeekJudge still has ΔJ−ΔG = 4.2 of unconfirmed improvement, indicating a risk of reward hacking. In Table 6, SeekJudge’s training success is 30.06, clearly lower than rule-based reward’s 43.00, showing that its reward signal is not fully aligned with true success. The authors claim to be “the first model-based reward to match or surpass native rule-based reward,” but this is verified only on three small-scale OSWorld domains, with degradation in some domains, and no cross-platform Android/Web RL results are given.

Questions for authors

1、Can you provide complete results, confidence intervals, and significance tests for each domain across multiple random seeds?

2、How do you prove that the teacher models have not seen CUAStepBench, AgentRewardBench, or OmniGUIRewardBench during training? Was task-level decontamination performed?

3、When comparing with rule-based reward, did you include SeekJudge’s distillation, teacher calls, and training costs? Was the comparison conducted under the same compute/data budget?

4、Why is SeekJudge lower than rule-based reward on Qwen3-VL-8B in the OS domain? How do you explain this drop?

3 comments
AuthorsOpenReview ↗

Response to Reviewer jgbD (Part 1/2)

Response to Reviewer jgbD (RE3)

Before the point-by-point response, we point out the following two major misunderstandings in this review, each of which can be checked against a single table.

(1) The review compares the step-level F1 of SeekJudge-9B, 38.0, with the 82.2 of the human expert and concludes that step-level judgments are unreliable. In Table 3, 82.2 is the trajectory-level F1 of the expert, and the step-level cells of the expert row are empty.

(2) The review reads the 100.0 of Rule on OmniGUIRewardBench as evidence that SeekJudge underperforms rule-based reward. The caption of Table 4 states that the labels of OmniGUIRewardBench are produced by its execution rules, so Rule reaches 100.0 by construction, and its entries are set in italics and excluded from the comparison. On the human-labeled AgentRewardBench in the same table, Rule reaches 67.1 and SeekJudge-9B reaches 76.6.

Each of these errors takes a correct number from the paper and attaches it to the wrong quantity. We ask the reviewer to verify these two points against the paper and to reconsider the assessment in light of the corrections below.

RE3-Q1. Novelty, and the basis of the seven dimensions and nine step classes.

Through an RL ablation and the step-level F1 against human labels, we show that the seven dimensions and nine classes both improve RL and agree with human step labels, and the novelty of the paper lies in the two results below.

To our knowledge, this paper is the first in two respects.

  • We are the first to show that a model-based reward matches or surpasses native rule-based reward on held-out test success in online RL.
  • We are the first to build a CUA reward benchmark with a human label on every executed step across platforms, with 5,634 step labels over 177 applications.

We also separate localization from extraction in a CUA judge, after showing that a verdict degrades as more screenshots share one forward pass even when the decisive screenshot is present.

The review lists multi-agent systems, tool use, PRMs, step-level rewards and trajectory compression as prior ideas. Prior work has used each of them, and none of the prior work that we know of has produced a model-based reward that matches native rule-based reward in online RL.

The seven dimensions and nine classes are supported at two levels.

  • Effect on RL. Adding the step term derived from the nine classes raises test success from 33.62 to 38.84 in Table 6, and R-Table 3 (RE1-Q2) shows the same effect on a second setting.
  • Agreement with humans. Merged into correct and erroneous steps by the criterion of the CUAStepBench labels, whether a step harms the final result, the nine classes give the step-level F1 of Table 3, where SeekJudge-9B is the best among small models (RE3-Q4).

RE3-Q2. Is the comparison fair?

Yes. Through controlled comparisons on the same backbone, we show that the comparison is fair and that the SeekJudge framework leads without any distillation.

The paper already contains such comparisons, and we added more (R-Table 1 and R-Table 2, RE1-Q1).

  • (1) On GPT-5-mini, the closed-source model that CUAJudge was released with, the SeekJudge framework exceeds CUAJudge on all three benchmarks without any distillation.
  • (2) On the untrained Qwen3-VL-8B, the SeekJudge framework exceeds OS-Themis on all three benchmarks and exceeds CUAJudge on CUAStepBench (Tables 3 and 4).
  • (3) Without training, Qwen3.5-9B and Qwen3-VL-8B perform comparably in the same framework, which shows that the untrained reference in the paper is fair. Distillation on the same data improves both backbones, and the fine-tuned Qwen3-VL-8B is within 2.2 points of SeekJudge-9B.

Every entry that needs a fitted regressor follows the held-out protocol of RE1-Q3.

AuthorsOpenReview ↗

Response to Reviewer jgbD (Part 2/2)

RE3-Q3. Author-built CUAStepBench, about 20 anchor trajectories, and annotator agreement.

Through a multi-stage annotation check and a blind independent expert, we show that the labels of CUAStepBench are reliable, and the about 20 trajectories touch no benchmark label.

The about 20 human-labeled trajectories are anchors for the criteria prompt. They define what counts as success and touch no benchmark label (Appendix E.1).

During the annotation of CUAStepBench, the same annotator checked each trajectory twice, a Codex audit flagged 51 suspicious labels as candidates, and two human annotators reviewed every flagged label and made all corrections, correcting 12 and discarding 7. An independent expert who did not take part in the annotation blindly labeled all 278 trajectories and reaches a trajectory-level F1 of 82.2 (Appendix J).

RE3-Q4. The step-level F1 of 38.0 against the 82.2 of the human expert.

The comparison mixes two levels. 82.2 is a trajectory-level F1, and among step-level numbers, SeekJudge-9B has the best step-level F1 of all small models.

The step-level cells of the expert row in Table 3 are empty, and the paper reports no human step-level F1. 38.0 should be compared with other step-level numbers. Among small models, SeekJudge-9B has the highest step-level F1, 38.0 against 21.2 for CUAJudge on Qwen3-VL-8B and 15.5 for OS-Themis, and it matches CUAJudge on GPT-5-mini at 37.9.

RE3-Q5. Rule reaches 100.0 on OmniGUIRewardBench.

Rule reaches 100.0 by construction, because the labels of OmniGUIRewardBench are produced by its execution rules, and on human labels SeekJudge-9B is well above Rule.

The caption of Table 4 states this, and the entries of Rule are set in italics and excluded from the comparison. This column measures agreement with the rules and does not show that the rules are correct. Two results show that the rules and humans disagree.

  • On the human-labeled AgentRewardBench, Rule reaches 67.1 and SeekJudge-9B reaches 76.6.
  • In Figure 1(a), on the 71 trajectories where the rule and the judge disagree, blind human raters side with the judge 48 times.

RE3-Q6. Scale of the RL experiments, reward hacking, platforms and cost.

Through a bound on reward hacking and the cost of distillation, we show that SeekJudge has the most rule-confirmed gain among model-based rewards and needs only a one-time cost.

Scale and platforms. RL runs on OSWorld because the comparison with rule-based reward needs rules as the test metric, and the step limits follow ACuRL so that our results are comparable to prior CUA RL work. The offline evaluation covers Android, Windows, Ubuntu, macOS and the web. The training success of Table 6 is addressed in correction (3) above.

Reward hacking. ΔJ − ΔG treats the rules as ground truth, and the rules disagree with human judgment (RE3-Q5), so it is an upper bound on hacking. Under this bound, SeekJudge has the largest rule-confirmed gain ΔG among the three model-based rewards, 9.5 against 5.1 for CUAJudge and 5.0 for OS-Themis, and the smallest unconfirmed share of its J gain, 31% against 83% and 37%.

Cost. Distillation is a one-time cost. The teacher API costs below 6,000 RMB, about 850 USD, with a proxy model trained on the outputs of Gemini-3.0-Flash serving most Analyze, Condense and Ground calls. SFT takes 41.4 hours on eight RTX A6000 GPUs, 331 GPU hours in total. Every reward uses the same 75 RL steps and the same GPUs.

All new experiments and related changes in this response have been added to the revised paper.

Dear Reviewer jgbD,

We would appreciate it if you could let us know whether our clarifications address your concern. If there are any aspects that you feel would benefit from further explanation, we would be happy to elaborate.

Reviewer_mJgxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents SeekJudge, a reward framework that separates evidence localization from screenshot analysis. Four roles share a distilled 9B model to produce trajectory and step scores, which are combined into a trajectory-level reward for GRPO. The paper also introduces CUAStepBench and a reward server that overlaps judging with environment interaction.

Critical strengths
  1. I think the topic is timely and realistic. Developing multi-agent systems for GUI tasks is essential but lacks discussion.
  2. The framework is well motivated. Separating evidence localization from screenshot analysis directly addresses the observed interference from multiple trajectory images.
  3. Beyond offline judging, SeekJudge achieves higher average downstream RL success than native rule-based rewards on both actor backbones. The densely annotated CUAStepBench and measurements of context usage and training overhead further strengthen the empirical contribution.
Critical weaknesses
  1. The sources of the online RL gains are not fully isolated. SeekJudge combines specialized model training, continuous trajectory scores, a step-score term, and reward calibration, whereas CUAJudge and OS-Themis use native binary verdicts without comparable calibration. The contribution of the core architecture to downstream RL improvements therefore remains less clear.
  2. Transfer to environments without rule-based supervision remains insufficiently validated. The default RL calibrator is fitted on approximately 1K target-environment rollouts labeled by native rules. Appendix F usefully shows that an offline calibrator still outperforms rule-based reward, but only in one setting. With online RL limited to three domains containing 12–24 training tasks each, broader applicability to new applications without target-environment rule labels remains uncertain.
Questions for authors
  1. Could you extend the reward-granularity ablation to other RL settings and provide comparable calibration for competing judges where applicable, to separate judging quality from reward-design improvements?
  2. How well does a frozen, independently fitted calibrator perform across all six RL settings, or on new applications without target-environment rule labels?
3 comments
AuthorsOpenReview ↗

Response to Reviewer mJgx (Part 1/1)

Response to Reviewer mJgx (RE2)

We thank the reviewer for recognizing the motivation of our work. The two questions are exactly the ones that separate judgment quality from reward design.

RE2-Q1. Extending the reward ablation, with comparable calibration for competing judges.

Yes. By extending the ablation to a second RL setting, we show that the gain comes mainly from the step term, which comes from the per-step judgment of SeekJudge, and on the same model with a held-out calibration SeekJudge still exceeds CUAJudge.

We extended the ablation of Table 6 from UI-TARS Impress to UI-TARS OS.

R-Table 3. Test success (%) of SeekJudge reward ablations with the UI-TARS actor. Impress is from Table 6, and OS is new.

SettingSeekJudge w/o continuous score and step termSeekJudge w/o step termSeekJudge
UI-TARS Impress35.933.638.8
UI-TARS OS21.718.328.9

The full SeekJudge is the best in both settings, and removing the step term costs 5.2 and 10.6 points. This shows that the RL gain of SeekJudge comes mainly from the step term.

The step term is computed from the class that SeekJudge assigns to every executed step. CUAJudge and OS-Themis natively output only a binary verdict per trajectory, and the step labels of CUAJudge in Table 3 come from our step prompt. The gain from reward design is therefore also a gain from the way SeekJudge judges.

We checked calibration on the offline side, with the same model for both frameworks. On GPT-5-mini, the model that CUAJudge uses, the SeekJudge regressor is fit on the outputs of GPT-5-mini with a task-level five-fold cross-fit, so every trajectory is scored by a regressor that never saw its task.

R-Table 1. Trajectory-level F1 (%) on the same backbone without training. GPT-5-mini SeekJudge uses a held-out regressor.

BackboneFrameworkAgentRewardBenchCUAStepBenchOmniGUIRewardBench
GPT-5-miniCUAJudge62.668.583.8
GPT-5-miniSeekJudge77.172.585.6

SeekJudge exceeds CUAJudge on all three benchmarks. This shows that, with the same model and a held-out fit, the advantage comes from the framework itself.

RE2-Q2. A frozen calibrator across settings and without target-environment rule labels.

By replacing the RL calibrator with a frozen offline fit, we show that the gain of SeekJudge does not come from target-environment rule labels in both settings that we tested.

The offline fit is fit once on the labels of the three offline benchmarks, frozen before RL, and it sees no RL rollout.

R-Table 9. Test success (%) with the UI-TARS actor and the offline fit. Impress is from Appendix F, and OS is new.

SettingRule-basedSeekJudge, RL calibratorSeekJudge, offline fit
UI-TARS Impress30.438.836.5
UI-TARS OS25.628.927.8

In both settings, the frozen offline fit is within 2.3 points of the RL calibrator. This shows that SeekJudge does not need rule-labeled rollouts from the target environment to obtain its gain.

For RL on an entirely new application, test success itself has no rule to measure it and needs a separate evaluation protocol, and we list this as future work.

All new experiments and related changes in this response have been added to the revised paper.

Thanks for your response, and I think many of my concerns have been addressed. I will keep my positive score.

Thank you for reading our response and for your continued positive assessment of the paper. We are glad that the extended reward ablation and the frozen offline calibrator results addressed many of your concerns

Reviewer_CBLuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SeekJudge, a reward framework for computer-use agents. It first describes what changed on the screen after each action and identifies the interface element the action targeted. It then uses these descriptions to select screenshots for closer inspection before judging whether the task was completed. Four agents share a 9B model trained on data generated by larger models. The system scores seven aspects of a trajectory and assigns a label to each step. For RL, it combines a calibrated trajectory score with the average step score. The paper also introduces CUAStepBench, with 278 trajectories and 5,634 labeled steps, and a reward server that processes part of the trajectory while the agent is still acting. Experiments cover three offline judging benchmarks and online RL with two actor models in three OSWorld domains. The reported mean test success rate is 3.3 percentage points higher than that obtained with rule-based rewards.

Critical strengths
  1. The method addresses a practical problem in GUI evaluation. Inspecting relevant screenshots separately is a sensible way to check details that may be missed when many images are read together. Processing some of the evidence during the rollout also has a measured benefit: Table 5 reports a 16.6% reduction in total round time.
  2. The paper tests whether the judge can help train an agent.The online RL experiments are more informative than an offline classification comparison alone. CUAStepBench also provides human labels for individual actions across multiple platforms, which could be useful for studying where GUI agents make mistakes.
Critical weaknesses
  1. The experiments do not isolate the benefit of the proposed judging framework. The reported gains combine changes to the backbone, training data, judging procedure, and reward. The framework without task-specific fine-tuning uses Qwen3-VL-8B, whereas SeekJudge-9B is fine-tuned from Qwen3.5-9B. There is no before-and-after comparison on the same backbone. Table 7 tests the framework with two strong models rather than SeekJudge-9B. This provides some evidence for the design, but it does not show whether the same gains hold for the deployed model or whether a simpler judge trained on comparable data would perform similarly. The RL comparison has a related problem. SeekJudge uses target-domain calibration and a continuous reward with an additional step term, while CUAJudge and OS-Themis use their uncalibrated binary outputs. Table 6 partly examines reward format, but only using UI-TARS on Impress tasks. The current results therefore do not establish how much of the improvement comes from the evidence-search procedure rather than additional supervision or a different reward. For the main offline results, the score regressor and threshold are fitted using labels from the same benchmarks used for evaluation (Appendix E.2). The cross-fit and cross-benchmark results in Table 9 retain most of the performance. This reduces the concern about overfitting, but these held-out results should be the main comparison. Using a shared threshold does not, by itself, ensure that the evaluation data were held out during calibration.

  2. The evidence for reliable rewards on long tasks is limited. The RL experiments use only 12-24 training tasks and 12-23 test tasks per domain, with episodes capped at 15 training steps and 25 test steps. Training lasts 75 updates. This is limited evidence for the long-trajectory setting that motivates the paper. The 3.3-point gain is averaged over training checkpoints. The reported uncertainty is based on three pairs of repeated runs, but the paper does not specify which settings were repeated or report independent repeats for all six settings. I would like to see whether the gain is stable across tasks, trajectory lengths, and training runs. The step labels are not yet a convincing training signal. SeekJudge-9B reaches 38.0 error-step F1 and 33.1 recall on CUAStepBench, while CUAJudge using GPT-5-mini with the supplied step prompt reaches 37.9 F1 (Table 3). The default method averages step scores into one trajectory reward and assigns the same advantage to every action within a trajectory. The benefit of this average is tested in only one setting, and the direct per-step variant performs worse in four of the six settings (Appendix B). Averaging does not necessarily remove incentives to add steps: when the mean step score is negative, adding zero-scored steps raises the reward unless the trajectory score drops enough to offset it. This is a potential failure mode, not evidence that the trained policy actually exploits it. The claim about replacing rule-based rewards also needs qualification. The RL calibrator uses roughly 1K rule-labeled rollouts from the target domains, and the paper does not explicitly state whether all test goals were excluded from these rollouts. Appendix F removes this calibration in one setting, with a positive result, but does not establish the same result across the other settings. The experiments therefore do not yet show that the method works as well where rule-based supervision is unavailable. Finally, CUAStepBench deliberately retains cases whose outcome is unclear to the teacher judge. The independent expert check in Appendix J is helpful, but it does not report agreement on the step labels. It is also unclear how labels are assigned when the recorded evidence is genuinely insufficient. These issues matter when this benchmark is used to support the reliability of detailed step judgments.

Questions for authors

None

4 comments
AuthorsOpenReview ↗

Response to Reviewer CBLu (Part 1/3)

Response to Reviewer CBLu (RE1)

We thank the reviewer for a careful review.

RE1-Q1. How much of the gain comes from the judging framework itself?

Through controlled same-backbone comparisons, we show that the judging framework and the distillation training each bring a gain on their own.

On the same GPT-5-mini without any training, the SeekJudge framework exceeds CUAJudge on all three benchmarks (R-Table 1). On the same backbone before and after distillation, training improves both backbones, and the fine-tuned Qwen3-VL-8B comes within 2.2 points of SeekJudge-9B (R-Table 2).

Same backbone, no training. We ran the SeekJudge framework on GPT-5-mini, the closed-source model that CUAJudge was released with.

R-Table 1. Trajectory-level F1 (%) on the same backbone without training. GPT-5-mini SeekJudge uses a held-out regressor (RE1-Q3).

BackboneFrameworkAgentRewardBenchCUAStepBenchOmniGUIRewardBench
GPT-5-miniCUAJudge62.668.583.8
GPT-5-miniSeekJudge77.172.585.6

No distillation and no extra data enter this comparison, and SeekJudge exceeds CUAJudge on all three benchmarks, by up to 14.5 points. This shows that the framework itself brings a gain before any training.

Same backbone, before and after training. We fine-tuned Qwen3-VL-8B on the same distillation data as SeekJudge-9B, and we ran the untrained Qwen3.5-9B, the base model of SeekJudge-9B, in the same framework.

R-Table 2. Trajectory-level F1 (%) before and after SFT on the same distillation data. Fine-tuned rows use the held-out regressor (RE1-Q3).

BackboneDistillation SFTAgentRewardBenchCUAStepBenchOmniGUIRewardBench
Qwen3-VL-8Bno61.271.377.3
Qwen3-VL-8Byes75.972.582.3
Qwen3.5-9Bno67.671.080.4
Qwen3.5-9B (SeekJudge-9B)yes75.873.784.5

Without training, Qwen3.5-9B and Qwen3-VL-8B perform comparably under the same framework. This shows that the untrained Qwen3-VL-8B in the paper is a fair untrained reference. Distillation improves both backbones on all three benchmarks, and after training, Qwen3-VL-8B is within 2.2 points of SeekJudge-9B. This shows that the gain from the distillation data does not depend on a particular backbone.

In the component ablation of Table 7, adding the Seek–Analyze loop on top of Condense and Ground raises trajectory F1 by 2.3 and step F1 by 7.0.

RE1-Q2. Does the RL gain come from judgment quality or from reward design?

Through an ablation of the reward components in two RL settings, we show that the gain comes mainly from the step term, which is itself a product of the SeekJudge judgment.

The continuous score and the step term are computed from the outputs of the SeekJudge judgment, a class for every executed step and a score on each of seven dimensions. CUAJudge and OS-Themis natively output one binary verdict per trajectory, and the step labels of CUAJudge in Table 3 come from our step prompt (RE1-Q6). The gain from the reward design is therefore also a gain of the SeekJudge judgment.

To measure the two components, we extended the ablation of Table 6 from UI-TARS Impress to UI-TARS OS.

R-Table 3. Test success (%) of SeekJudge reward ablations with the UI-TARS actor. Impress is from Table 6, and OS is new.

SettingSeekJudge w/o continuous score and step termSeekJudge w/o step termSeekJudge
UI-TARS Impress35.933.638.8
UI-TARS OS21.718.328.9

The full SeekJudge is the best in both settings, and removing the step term costs 5.2 and 10.6 points. This shows that, in both settings, the step term is the main source of the RL gain of SeekJudge.

The calibration component is addressed in RE1-Q7, where we replace the RL calibrator with an offline fit that sees no RL rollout.

AuthorsOpenReview ↗

Response to Reviewer CBLu (Part 2/3)

RE1-Q3. Held-out regressor results as the main offline comparison.

Through a task-level held-out cross-fit of every fitted entry, we show that the conclusions of the paper hold on held-out numbers, which the main tables of the revised paper now also report.

The judge model never sees benchmark labels. Only a small regressor after the judge is fit, which combines the seven dimension scores into one score, together with a decision threshold.

For every fitted SeekJudge entry in the paper, we redid a task-level five-fold cross-fit. Each evaluated trajectory is scored by a fold model whose training data excludes its task, and repeated runs of one task fall into the same fold. The threshold maximizes Youden's J on the training rows of that fold, and results are averaged over 10 split seeds.

R-Table 4. Trajectory-level F1 (%) under the joint fit of the paper and the task-level cross-fit, with std over 10 seeds.

RowBenchmarkJoint fit (paper)Cross-fit
SeekJudge-9BAgentRewardBench76.675.8 (0.53)
SeekJudge-9BCUAStepBench74.273.7 (0.34)
SeekJudge-9BOmniGUIRewardBench85.184.5 (0.17)

F1 changes by at most 0.8 points. Under the cross-fit, SeekJudge-9B still exceeds every small-model baseline and leads CUAJudge with GPT-5-mini by 5.2 points on CUAStepBench and by 13.2 points on AgentRewardBench. This shows that the conclusions of the paper do not depend on fitting the regressor on the evaluation set.

RE1-Q4. Evidence on long trajectories is limited.

Through two offline length studies, on CUAStepBench and on our new human-annotated CUAStepBench-Long, we show that the advantage of SeekJudge holds as trajectories grow to hundreds of steps.

The step limits of our RL follow ACuRL, with 15 steps in training and 25 steps at test, so that our results are comparable to prior CUA RL work. We therefore measure long trajectories offline. First, we split CUAStepBench into three equal parts by the number of steps.

R-Table 5. Trajectory-level F1 (%) on CUAStepBench in three length bins. The number of trajectories in each bin is in parentheses.

Judge≤ 14 steps (94)15–26 steps (92)> 26 steps (92)
CUAJudge, GPT-5-mini59.873.672.3
OSReward, OS-Shepherd-9B72.762.769.0
CUAJudge, Qwen3-VL-8B61.768.869.9
OS-Themis, Qwen3-VL-8B56.157.865.0
SeekJudge-9B75.472.674.4

SeekJudge-9B exceeds every small-model baseline in all three bins, and its F1 varies by only 2.8 points across bins, against 13.8 points for CUAJudge with GPT-5-mini. This shows that SeekJudge-9B is the most stable judge across trajectory lengths.

Second, we built CUAStepBench-Long by human annotation, 18 trajectories with a mean length of 272 steps. A single-forward judge has to cap the number of input images on such trajectories, so we ran CUAJudge on Qwen3-VL-8B under a growing cap.

R-Table 6. Trajectory-level F1 (%) on CUAStepBench-Long over 6 to 16 runs. CUAJudge runs Qwen3-VL-8B on four GPUs under an image cap.

Judge \ Max images6496128no cap
CUAJudge, Qwen3-VL-8B69.873.2OOM–
SeekJudge-9B–––78.2

CUAJudge peaks at 73.2 with 96 images and runs out of memory on four GPUs at 128. SeekJudge-9B reaches 78.2 without any cap and with less than a quarter of that GPU memory. This shows that SeekJudge-9B keeps its advantage on trajectories of hundreds of steps, where a single-forward judge reaches its memory limit.

RE1-Q5. Checkpoint averaging and repeated runs behind the 3.3-point gain.

Through two other readings of the same runs, we show that the ranking of the rewards does not depend on averaging the checkpoints.

The three repeated pairs are CUAJudge on UI-TARS Impress, SeekJudge on UI-TARS Impress and SeekJudge on Qwen3-VL-8B OS, and the revised paper states this.

Averaging the five test evaluations was the protocol fixed before training, to reduce the variance of a single checkpoint. R-Table 7 gives two other readings of the same runs.

R-Table 7. Test success (%) averaged over the six RL settings under three readings of each training run.

ReadingRuleCUAJudgeOS-ThemisSeekJudge
mean of 5 evaluations (paper)26.5324.8124.0829.81
mean of last 2 evaluations28.2024.1821.7430.88
last evaluation28.7823.0121.6030.29

SeekJudge is the best reward under all three readings. This shows that the ranking of the rewards does not depend on how the checkpoints are averaged. The per-setting results are in the revised appendix.

AuthorsOpenReview ↗

Response to Reviewer CBLu (Part 3/3)

RE1-Q6. The step labels are not yet a convincing training signal.

Through a comparison under the same step prompt and an analysis of rollout lengths, we show that SeekJudge-9B gives the best step labels among small models, that the step term raises test success, and that the policy does not exploit it to add steps.

On 38.0 against 37.9. CUAJudge does not output step-level judgments, and its step labels in Table 3 come from our step prompt, as the † mark shows. The same prompt on Qwen3-VL-8B reaches only 21.2. The comparison therefore shows that a 9B model matches GPT-5-mini in executing our step-level definition. Among small models, SeekJudge-9B has the highest step-level F1, and on the same Qwen3.5-9B backbone, distillation raises step-level F1 from 30.9 to 38.0.

On averaging the step scores. Appendix B shows that injecting step scores directly into the per-step advantage is unstable, and we leave per-step credit assignment to future work. The paper uses the step score as part of the trajectory reward, and R-Table 3 shows that removing it lowers test success in both tested settings.

On the incentive to add zero-score steps. We counted the average number of interaction steps per training rollout.

R-Table 8. Average interaction steps per training rollout (cap 15), averaged over the six RL settings.

Rewardupdates 1–15updates 61–75change
Rule10.189.82−0.36
CUAJudge10.249.55−0.69
OS-Themis10.2210.11−0.11
SeekJudge10.188.84−1.34

Under SeekJudge, rollouts become shorter in 5 of the 6 settings, and they shorten the most among the four rewards. This shows that the policy does not exploit this incentive. The path redundancy dimension lowers the trajectory score for redundant steps, and the meaningless class gives repeated actions a negative score.

RE1-Q7. Is SeekJudge effective without rule supervision in the target domain?

Yes. By replacing the RL calibrator with an offline fit that sees no RL rollout, we show that the gain of SeekJudge does not come from the rule-labeled rollouts of the target domain.

We keep everything else unchanged. The offline fit is fit once on the three offline benchmarks before RL, and it sees no RL rollout.

R-Table 9. Test success (%) with the UI-TARS actor and the offline fit. Impress is from Appendix F, and OS is new.

SettingRule-basedSeekJudge, RL calibratorSeekJudge, offline fit
UI-TARS Impress30.438.836.5
UI-TARS OS25.628.927.8

In both settings, the offline fit is within 2.3 points of the RL calibrator. This shows that the gain of SeekJudge does not come from the rule-labeled rollouts that the RL calibrator uses.

RE1-Q8. Retained uncertain cases and agreement on the CUAStepBench labels.

Through a benchmark whose samples we did not select and a blind independent expert, we show that keeping uncertain cases does not favor our method and that the labels of CUAStepBench agree with the expert.

Filtering uses only the confidence of the teacher, independent of whether the teacher was right, and all judges are evaluated on the same retained set. On AgentRewardBench, whose samples we did not select, SeekJudge-9B leads CUAJudge with GPT-5-mini by 13.2 points, more than the 5.2 points on CUAStepBench (R-Table 4). This shows that the filtering does not favor our method.

An independent expert who did not take part in the annotation blindly labeled all 278 trajectories and reaches a trajectory-level F1 of 82.2 against the final labels (Table 3, Appendix J). During annotation, the same annotator checked each trajectory twice, a Codex audit flagged 51 suspicious labels as candidates, and two human annotators reviewed every flagged label and made all corrections, correcting 12 and discarding 7.

All new experiments and related changes in this response have been added to the revised paper.

Dear Reviewer CBLu,

We would appreciate it if you could let us know whether our clarifications address your concern. If there are any aspects that you feel would benefit from further explanation, we would be happy to elaborate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.