- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studied interaction-as-a-judge: let a computer-use agent (CUA) to interact with the applications (websites, games, etc) before sending the interaction trajectory to LLM-as-a-judge. It checks (1) whether interaction-as-a-judge better aligns with human preference, compared to static LLM-as-a-judge, (2) whether interaction-as-a-judge more accurately verifies the CUA’s generated code, and (3) whether interaction-as-a-judge helps the CUA better improve itself, in both inference-only iterative improvements and RL training.
Critical strengths
The research problem (interaction-as-a-judge) is of reasonable value.
Both inference-time agent iteration and RL-based training received convincing improvements from the proposed interaction-as-a-judge feedback. Especially, when using interaction-as-a-judge in RL, model performance outscores its counterpart using code-only judging by 20.1 points under GPT-sol interactive eval.
Two useful datasets are collected with human verification. The first one has user preferences on agent-generated applications. The second one verifies whether the agent-written code completed a user request.
Critical weaknesses
- The authors wrote the abstract and most of the intro as if they were the first to use interaction trajectories to evaluate agent-generated applications. This is pretty misleading. Then, in line 46, the authors further said:
While prior works have explored this paradigm (interaction-as-a-judge), …… we are the first to systematically study it in a controlled fashion. Specifically, we investigate how dimensions such as interaction strategy and agent capability affect evaluation accuracy for both pairwise human preference prediction and correctness verification.
This claim is also ambiguous and vague. In this paper’s experiments, which have been done in a similar way (or on a different task) by previous work? Which are original? There is not a comprehensive explanation.
-
Furthermore, the authors did not exhaustively analyze which “interaction strategy” is the best. In figure 3, all three interactive strategies have similar performance (differences within 2 points). Also, the authors did not analyze: taking token cost into consideration, which strategy is the most cost-effective?
-
The “agent capability” experiment is also ambiguous. Both CUA and the judge are changed in the experiments (luna, terra, sol). So I don’t understand what model capability the experiment analyzes, the CUA or the judge?
Questions for authors
Can you elaborate how you do the human annotation for AnonCorrectness? The details of this part are missing from the paper.
2 comments
Response Part 2
“taking token cost into consideration, which strategy is the most cost-effective?”
We chose a fixed interaction budget to keep the same visual token budget for our reward models and LLM judges, otherwise those costs would vary. Also, we chose a 30 step interaction budget as we found that when given an unlimited turn budget computer-use agents (for both methods) stopped by themselves around 30 steps; increasing step budget for programmatic would likely provide limited additional benefit in this setup. Programmatic computer-use also has to predict from only the initial state which means that error accumulates over time. Lastly, we do not argue that iterative computer-use is always better than programmatic or even heuristic computer-use and cost should certainly be a factor when choosing between each method. Our goal was to show that performance scales between these three computer-use methods including with cost which was not shown in prior work.
However, we do agree this would be an interesting experiment to run. We run two additional experiments: training a reward model with programmatic with GPT 5.6 Luna on xhigh and iterative with GPT 5.6 Luna for 14 iterations rather than 30. The costs of these two experiments are matched. Iterative still improves over programmatic in the full code and App.tsx settings. When there is no code, the two perform similarly. We believe this is because when there is sufficient code, quality matters more since the reward model can extract the majority of basic information from the code. When there is no code, simply having more visual information (30 steps vs 14 steps) can outweigh the benefits of higher quality, but more expensive interactions. We have included this in a revision to our paper in Appendix E.5.
| Setting | Programmatic (xhigh), 30 steps | Iterative (medium), 14 steps | Gap |
|---|---|---|---|
| Full code | 70.2 | 70.9 | +0.7 |
| App.tsx only | 70.1 | 70.4 | +0.3 |
| Interaction only | 68.2 | 68.1 | -0.1 |
“Both CUA and the judge are changed in the experiments (luna, terra, sol).”
We agree with the reviewer and ran a full cross matrix experiment where we varied the computer-use agent and the judging agent separately. Performance is measured as number of broken requirements correctly identified.
| Judge \ CUA | Luna | Terra | Sol |
|---|---|---|---|
| Luna | 62.4% | 64.3% | 61.5% |
| Terra | 66.1% | 67.4% | 68.6% |
| Sol | 66.1% | 68.3% | 69.9% |
We find that we observe scaling across both axes; increasing computer-use but keeping the judge fixed improves performance and vice-versa. The main exception is when Luna as a Judge uses Sol computer-use traces; we hypothesize that this is because Luna is a significantly weaker model than Sol and isn’t able to utilize that information properly. Similarly, when Luna is the computer-use agent, Terra and Sol perform the same as the judge (despite Sol being higher capability than Terra), again hinting at issues with a Luna and Sol capability gap. We have included this in a revision of our paper in Appendix F.1. We believe this experiment is important so once all the revisions are in, we will likely incorporate it into the main paper. Thank you for your suggestion.
“Can you elaborate how you do the human annotation for AnonCorrectness?”
As mentioned on page 3, the human annotator manually reviews the game and annotation to verify its correctness. The annotator does this by actually playing the game to verify that the annotation is broken. Working requirements are done in a similar manner, except with the working games and requirements. The requirements are straightforward and non-ambiguous to a human (e.g., “the player must not fall through the floor”). If the question is in regards to how we collected human complaints and human queries in the first place, we can only share limited details during review to avoid de-anonymizing our work. In short, human annotations were collected at scale from everyday users of a widely deployed online platform. We will further document this process in our camera-ready.
Response Part 1
We first want to thank the reviewer for their time, their questions, and their positive sentiment towards our work. We especially want to thank the reviewer for pointing out the general usefulness of interaction-as-a-judge, our convincing inference-time and reinforcement learning experiments, and how useful the new data resources we are releasing will be for the research community. If our answers to your questions are helpful, we would greatly appreciate an update to our score.
We acknowledge that there are numerous past works that explore interaction for evaluation and generation refinement. The differentiation of our work is to more thoroughly understand this emerging paradigm through (1) demonstrating its alignment with human preferences, (2) providing evidence that the paradigm scales as computer use improves, and (3) studying it at an unprecedented data scale.
Human Preference Reward Modeling. Human preferences are a standard practice used for training reward models and post-training including supervised fine-tuning and reinforcement learning (e.g., RLHF, DPO). To our knowledge, no prior work explores the use of computer-use agent interaction traces for human preference reward modeling, simply using an LLM judge in the loop with rubrics. Both human preferences and rubrics-based approaches are valid and widely used in practice, but we are the first to show that interaction-based methods can work with human preferences and reward modeling.
Capability Scaling Effects. One of the key insights we provide is that there seems to be a scaling effect linking better computer-use capabilities and methods to better downstream evaluation and generation. Our work provides the first evidence that changing the computer-use agent’s base model (e.g., GPT 5.6 Luna to GPT 5.6 Sol) or its execution method (heuristic, programmatic, and iterative computer-use) improves performance. This suggests that Interaction-as-a-Judge will scale as researchers develop better models and methods, which was not clear prior to our work.
Dataset Size, Scale, and Realism. We have collected significant amounts of data and are contributing and evaluating over two datasets one of which contains 6.8k tasks (13.6k applications) across 7 diverse categories and the other containing 412 requirements/tasks across 222 games. For some comparison with relevant prior work, AppEvalPilot releases 194 tasks and evaluates on at most 54 tasks in the paper, WebGen-Agent evaluates over 647 requirements/tasks and 101 websites (and the data comes from an entirely different publication, WebGen-Bench), and Verification Horizon evaluates over a private set of 671 web development tasks. Including both of our datasets, we are releasing and evaluating on an order of magnitude more data than these prior works. Additionally, many of these benchmarks use synthetically generated tasks; all our tasks are either direct user requests or derived from real user complaints. As a final point, we are fully releasing our agentic traces, not just the final applications which we believe will be of significant help to future research in this area.
We want to emphasize that we are building upon prior work and believe that we are providing significant value to the community through our data and experiments. We have revised our manuscript to more effectively clarify the scoping and differentiation in the abstract/introduction.