- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CAVEAT studies whether CUAs preserve user objectives when truthful marketplace presentation favors other products. Nine simulated marketplaces implement eight steering mechanisms, with matched controls preserving catalogs, product facts, and the unique user-optimal purchase. Across five model families on 52 tasks, optimal purchasing drops from 78.6% to 17.3% under steering. Single-mechanism experiments and trajectory analyses examine search closure, preference interpretation, and unresolved evidence. A task-specification and verification harness substantially improves selected models, including an 80-percentage-point gain on five large-catalog tasks. Post-training a 27B model also improves performance on held-out product types.
Critical strengths
-
Matched controls preserve the decision problem while varying presentation, and purchase outcomes are programmatically scored. The large degradation across five model families, plus increased purchases of the same promoted products, convincingly demonstrates steering in this setting, and that the problem is real and imoprtant.
-
Single-mechanism and targeted ablations move beyond aggregate failure rates. The prompting baseline, harness-component ablations, and held-out product-type evaluation make the mitigation results more informative than an unstructured prompt-only comparison.
-
I think it's also an interesting failure mode that previous works have not explored.
Critical weaknesses
-
The verification tool responsible for most of the Hard-set improvement is not specified at the implementation level. Appendix H describes its purpose and the agent’s coverage protocol, but not the tool’s full input schema or how it checks submitted coverage/evidence and approves a candidate.
-
Mitigation evidence is preliminary, as the authors acknowledge: Hard has five tasks, post-training transfer uses 16 non-laptop Shop tasks, and the cross-environment check has four Stay tasks. These results demonstrate promising interventions but establish their generality less strongly than the broader benchmark findings.
Questions for authors
-
The paper mentions Fara, but I don't see it in Figure 3. Also, it would be better to evaluate more model families, like at least one from Anthropic (Given that they are often good at CUA tasks, and they are usually safer compared to other providers. For example, they are shown to be less "goal-directed" compared to other model families in Shayegani et al. 2026, "Just Do It!? Computer-Use Agents Exhibit Blind Goal-Directedness"). There are also specifically trained open CUA models out there, like UI-TARS, OpenCUA, etc., that are worth being evaluated.
-
The paper mentions small models, but then refers to 27B. There are much smaller models available like the ones I mentioned in the previous points, so 27B does not really count as small, and I think the claims of the paper need to be revised a bit, or actuall smaller models like 4B-7B-8B need to be tested.
-
Please specify how the verification tool checks candidate evidence and coverage, beyond the protocol in Appendix H. Is validation deterministic or model-based, and can an agent purchase after a failed or omitted check? Section 6.1 already states that the harness has no hidden catalog access.
-
Which tasks or trajectories informed harness design and prompt tuning? Please provide per-task mitigation outcomes and clarify the task-clustered uncertainty for the five-task Hard and four-task Stay evaluations.
-
In the preference-order experiment, the global optimum is never found. Among inspected nondominated alternatives, what user-specified rule determines the correct trade-off? How do you separate unsupported priority claims in rationales from demonstrably worse purchase choices?
-
I could not find any details about the computational resources used, especially for the training of the 27B model. Can you please specify the resources for reproduction? Also, can you provide some insights into the resources needed for smaller models of sizes 4B to 8B?
Dear AC and Reviewers,
We thank all reviewers for their thoughtful and constructive engagement during the discussion. All three reviewers now recommend acceptance: Reviewer NYF8 with a score of 4, and Reviewers tjFk and f3V9 with 3.
We are encouraged that the reviews recognize the following strengths.
We have uploaded a revised PDF with all changes highlighted in blue.
evaluateusage and search effort.CAVEAT gives the community a controlled way to measure whether agents stay faithful to users when the environment has its own incentives, and shows that the resulting failures can be diagnosed and substantially reduced. We thank the reviewers and the AC again for their time and engagement. Their careful comments have made the paper stronger.