- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Summary
This paper studies a setting in which websites require AI agents to complete useful work, such as coding or image labeling, before providing access to protected content. Through controlled browser experiments involving multiple models and agent implementations, the authors investigate whether agents can manage the cost of this work-based access mechanism. The results show that agents often continue submitting work despite excessive or unreliable requirements, and that repeated pressure can also weaken safety refusals in some configurations.
Critical strengths
Strengths
-
The paper introduces a novel and relevant problem setting. Rather than assuming that web content is freely accessible once an agent locates it, the work considers an interaction in which access must be earned through verifiable contributions. This perspective connects agent autonomy with the economic incentives of content providers and raises questions about cost control, stopping decisions, and the reliability of agent behavior that are not captured by conventional browsing benchmarks.
-
The evaluation examines several dimensions of agent behavior. The experiments include multiple models and agent implementations, as well as websites with different work requirements and access behaviors. This makes it possible to study not only whether agents can complete work, but also whether they stop when requirements become unreasonable, continue when promised access is never delivered, or change their behavior under repeated rejection.
-
The findings reveal a meaningful gap between stated preferences and actual agent behavior. Agents can claim to have a low spending limit when asked directly but still complete many more tasks during execution. The safety experiments also suggest that a model's direct refusal behavior may not fully predict its behavior when embedded in a persistent agent loop. These observations provide useful empirical evidence about the interaction between persistence, incentives, and safety.
Critical weaknesses
Weaknesses
-
The controlled environment may not capture the full complexity of real content-access mechanisms. The experiments use ten locally hosted website templates and primarily two forms of work: coding and image labeling. Real publishers may have different incentives, verification standards, task values, and content-access policies. The paper would be stronger if it more clearly established how the selected settings map to realistic publisher-agent interactions and which conclusions are expected to generalize beyond the controlled environment.
-
The evaluation does not fully quantify the economic value of the exchange. The number of tasks completed is a useful measure of work expenditure, but it is not equivalent to the actual cost of the work or the value received by the website owner. Different submissions may vary substantially in quality, usefulness, and computational cost. A more explicit model of the utility of completed work, the value of the protected content, and the cost of continued attempts would make the conclusions about rational stopping and excessive payment more precise.
-
The safety findings should be interpreted carefully across models and agent configurations. The observed deterioration in refusal behavior is substantial for some configurations but not uniform across all tested models. Since the paper also shows that the agent implementation strongly affects whether runs continue or stop, it remains important to separate the effects of repeated access pressure from those of the agent scaffold, task design, and model-specific behavior. Additional controls and a clearer characterization of the conditions under which refusals deteriorate would strengthen the broader safety claim.
Questions for authors
-
Clarification The paper studies whether agents provide work in exchange for content access and independently verifies work submissions. Could the authors clarify how submission quality is measured across coding and image-labeling tasks, and whether the reported work counts distinguish correct, useful contributions from merely completed submissions? This affects how strongly the results support the claim that websites can obtain valuable work rather than simply induce agents to expend effort.
-
Assessment-critical evidence The observed deterioration in refusal behavior appears to vary substantially across models and agent implementations. Could the authors clarify how robust the safety finding is across the tested model–agent configurations, including the corresponding denominators and uncertainty for the key harmful-response rates? In particular, how much of the reported effect can be attributed to repeated access pressure itself, as opposed to the agent scaffold or other differences between direct prompting and agent execution? This distinction is important for assessing the scope of the paper's safety conclusions.
We thank the reviewer for the careful review and for recognizing the novelty of the setting and the gap between what agents say and do! Below we address each point.
W1 (realistic settings). Work as payment is not deployed on today's websites, so we cannot measure it in the wild. Controlled websites let us change one thing at a time (the price, whether the website delivers, how the agent can pay) and check every submission. To keep the pages realistic, five of our ten website templates are local recreations of real websites (e.g., RTINGS and the U.S. National Park Service), and the pages are server-backed web apps in a real Chromium browser. What we expect to generalize is how the agents behave: they choose sensibly between offers shown side by side, but have no limit on the total they spend when a website keeps asking for more. How long they persist depends on the model and the agent implementation.
W2 (economic value). Besides task counts, we report the tokens consumed: the median successful 32-problem run used 2.2 to 3.8 million tokens depending on the model, about 70k to 120k per problem, and took 18 to 42 minutes. We report tokens rather than dollars because tokens convert directly into cost at any price, whereas prices differ between models, change over time, and differ for the same model across providers. The value the website receives is the verified correct work (see Q1).
W3 and Q2 (safety finding). The safety result mainly shows a possibility: a website that keeps rejecting the work can push a model into harmful responses. DeepSeek-V4-Flash shows that this does happen: it refused almost all unsafe requests when asked directly (2 harmful replies out of 40), yet produced harmful responses in 69 of its 128 agent runs. GPT-5.6-Luna kept refusing (4 of 128 runs), so the effect indeed depends on the model. The pressure comes only from the website: the user's request is benign, and the website repeats one fixed rejection. Within the same runs in setting F, 12 of DeepSeek-V4-Flash's 47 first responses were harmful, before any rejection, and 31 of its 38 second responses, after one (G: 7 of 50, then 23 of 45). So the agent setting raises the rate a bit, and repeated rejection much more. We will further clarify this in the paper.
Q1 (quality of the work). Both validators are deterministic. A coding solution counts as correct only if it passes all public and hidden private LiveCodeBench tests, and an image label must match the GQA reference (ignoring case and whitespace, with "grey" and "gray" treated as equal). Every submission is checked, whether or not the website grants access. We report submissions and correct work separately: participation counts any nonblank submission, while every count of completed work, such as the largest workload or the correct pairs under moving goalposts, counts only correct solutions. So the websites did obtain valid work: the four strongest models solved both coding problems in every run with every agent.