- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper try to study the adaption of general domain computer use models to specialized browser-based booking workflows. Firstly, the authors use SFT training on Qwen3.5-35B-A3B models, in order to teach tool protocols, browser interactions, and user communication. Then, they use trajectory-level reinforcement learning (RL) to improve task execution. Besides, the system includes a browser harness that matches the production model-facing protocol, an Iron User simulator that updates task ground truth during clarification dialogues, and an evaluation pipeline combining LLM judgments with deterministic rules. The paper reports that SFT model improves strict task success from 16.7% to 74.1%. In a separate RL evaluation, SFT+RL increases the composite reward from 0.553 to 0.725 compared with SFT alone. The appendix provides human evaluations of the user simulator and Judge, as well as training stability experiments.
Critical strengths
1. The task setting is practically relevant and goes beyond GUI interaction.
The paper considers more than filling out forms correctly. It also covers asking for missing information, discussing alternatives, explaining why a request cannot be completed, and reporting outcomes honestly. Updating the ground truth to reflect what the user agrees to is a reasonable way to handle changing goals during an interaction.
2. The system is fairly complete, and task-specific training brings substantial gains.
The authors bring together a browser harness, user simulator, evaluator, and post-training pipeline. In Table 1, SFT improves the same base model’s success rate by 57.4 percentage points. This shows that task-specific training is effective for the evaluated booking workflow.
3. The authors include human checks of key components and acknowledge limitations in the results.
The appendix includes human evaluations of Iron User and the Judge. The authors also note that a success rate close to Claude’s does not establish statistical equivalence, and that repeated validation during training cannot replace independent training runs. These are useful qualifications that keep the claims in perspective.
Critical weaknesses
1. The contribution beyond standard domain adaptation is still unclear.
It is not particularly surprising that a general-purpose computer-use model performs poorly in a specialized domain. The paper needs to explain more clearly what makes this specialization gap difficult and why the proposed designs are needed to address it.
Using SFT to learn task-specific knowledge and interaction protocols, followed by RL to improve task performance, is a common approach. There is nothing wrong with using existing algorithms, but the paper then needs to show what additional value its system design provides. The most promising components are Iron User with dynamic ground-truth updates and the staged Judge, yet neither is directly compared with a simpler alternative.
As a result, it is hard to tell whether the gains mainly come from standard task-specific training or from these particular designs. It is also unclear how well the designs would carry over to other tasks. Could the authors compare dynamic ground truth with fixed targets or a simpler simulator, and the staged Judge with a simpler evaluation setup? Which failure cases do these components improve, and by how much?
2. The relationship between training and evaluation data is unclear, making generalization difficult to assess.
Section 5 states that training–evaluation overlap was not measured, and Section 6.3 does not clearly explain how the evaluation subset in Table 1 relates to the training data. The relationship among the 399 tasks in Section 5, the booking subset in Table 1, and the RL evaluation in Table 2 also needs clarification.
Having many different hostnames does not necessarily mean that website layouts, generation templates, or task instances are independent. It is therefore difficult to tell whether the model performs better on unseen tasks or mainly adapts to distributions already covered during training. This does not imply data leakage, but it limits what can be concluded about generalization.
Please clarify the size, source, and scenario composition of each evaluation set, as well as how the sets relate to one another. Could the authors also report overlap with the SFT/RL training data at the task, website, and generation-template levels, and provide results on a clearly separated held-out subset?
3. RL improves the composite reward, but its effect on strict task success remains unclear.
In Table 2, SFT+RL increases the composite reward from 0.553 to 0.725 compared with SFT alone. However, this reward combines several task and behavioral signals. A higher reward could reflect more completed tasks, fewer formatting errors, or better compliance with other rewarded behaviors. It does not directly tell us how much strict task success has improved.
The authors acknowledge that the metrics in the two tables are not directly comparable, but a comparison of SFT and SFT+RL on the same held-out tasks using strict success is still missing. In addition, the Judge’s overall accuracy is 82%, but the paper does not show whether its errors differ across the evaluated models. It is therefore unclear how these errors might affect the comparisons.
Could the authors report strict binary success rates on the same held-out tasks, with confidence intervals that account for the paired evaluation? If human checks are available, do they support the same model ranking and direction of improvement? Breaking down the results into correct bookings, appropriate abandonment, Unknown outcomes, and technical failures would also help show where the RL gains come from.
Questions for authors
Refer to weakness.