- Rating
- 1(−3)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(−1)
- Soundness
- 1(−1)
Read review
Paper summary
The paper measures whether an AI assistant keeps the user thinking or hands over the answer. Eleven behaviors are scored, such as asking the user’s own view before advising, handing a decision back to the user, and correcting a wrong premise instead of building on it. Three LLM judges from three model families score each behavior from 0 to 3 on 320 prompts: 200 single-turn professional requests, 16 five-turn conversations, and 40 prompts where the user pushes the model to skip the questions. Four behaviors are dropped from the composite score because neither the judges nor a blind human panel agree on them.
Seven open-weight models from six families run twice: with no system prompt, and with a system prompt describing the wanted behaviors. With no system prompt, every model answers outright and almost never asks the user first; two proprietary models behave the same way. With the system prompt, every model’s composite score roughly doubles. A generic system prompt of the same length adds almost nothing, and the gain holds on held-out prompts, in multi-turn and adversarial settings, and under a judge from outside the evaluated families. The paper also measures the costs of the prompt: a responder that only asks questions beats every baseline, so a second score credits asking only when the model delivers the answer once the user replies; prompted models ask first on half the requests the prompt told them to answer directly; accuracy on calculation questions drops.
Overall important problem with a good benchmark and actionable result
Critical strengths
-
The best result is that every model tested stops answering outright and starts asking the user first when a system prompt tells it to. This holds on all seven open-weight models from six families, and a generic prompt of the same length adds almost nothing (3.2 points). HumanAgencyBench (Sturgeon et al., arXiv 2025) had already shown that models rarely ask first by default, but could not say whether that is a training choice or a limit of the models. Reversal on every family says training choice, which is the premise the pro-worker AI argument depends on.
-
I appreciated that the paper tests its own score harder than most benchmark papers do. They build a responder that only asks questions, show it beats every real model, and add a delivery check that drops it to 3 to 8 points. They measure how often prompted models ask when the user wanted an answer. They drop four behaviors because humans could not agree on them either.
Critical weaknesses
-
“PWB measures interaction form and does not establish effects on worker skill or decisions.” Then what does a high score tell someone deciding whether to use this prompt? The paper’s own results show the prompt that raises the score also makes the model worse in two ways. When the user clearly wants an answer, the prompted model asks a question instead 47% of the time. On calculation questions, it gets the answer right 6.7 points less often, because it holds the answer back. Making the user think may be worth those costs, but the paper cannot say, because no user is ever measured. The title and the name “pro-worker” promise that unmeasured outcome. Either narrow the title and abstract to what is measured, how the model behaves, or run the study already described in future work: give real users the model with and without the prompt and test what they remember and how often they accept a wrong answer.
-
“Cognitive forcing and complementarity account for 42% (Qwen3.5) to 97% (Gemma) of each model’s ∆PWI.” Almost the whole result comes from two behaviors: asking the user first, and handing the decision back. Were those two checked against human judgment? The judges agree worst on asking first when there is no prompt (κ = 0.41), and the human study has no asking-first items. The human study has “two controls,” but the paper does not say which behaviors. If handing back the decision was not one, then the two behaviors that drive the result were scored by LLM judges only, and the eleven-behavior list in the abstract describes a bigger instrument than the one behind the numbers. Name the controls and add a human check on asking-first items.
Questions for authors
Every scored prompt is one where asking the user first is the right move, so a model that asks on every request gets full credit. Your 30 direct-answer prompts show prompted models ask first on half of them when they should have answered, but those prompts do not count toward the score. Why not include them, so asking when the user wanted an answer costs points?
ai slop
Did the authors even read their own paper before submitting it?
Section 7 literally contains
prompt_checks/NEXT_VERSION.mdin the main text. The paper reads more like an unfinished AI-generated draft than a carefully written and reviewed academic submission.The fact that such an obviously unpolished manuscript received a score of 4 (Clear Acceptance) is genuinely astonishing.
I am not claiming that a Markdown file path alone proves AI authorship. But when a paper leaves this kind of material in its main text, it raises serious questions about how carefully the authors reviewed their own work.
This is exactly the kind of low-effort, AI-slop-like academic writing that should receive much closer scrutiny, not a Clear Acceptance.