- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies human–AI complementarity in safety-relevant tasks, comparing human, AI, and hybrid performance across multiple types of tasks. The authors evaluate Top-N assistance, subtask delegation, and confidence-based and learned routing methods. The results show an opportunity for human–AI complementarity, but assistance may reduce humans’ ability to correct AI errors. In addition, routing does not consistently outperform the better single source. The paper argues that human–AI oversight needs to preserve human corrections and find reliable ways to determine when to select human or AI answers.
Critical strengths
- The scale of the data and ratings is relatively large, and the evaluation suite covers a relatively rich range of tasks.
- The paper tests various possible strategies for human–AI complementarity and answer routing, providing a valuable reference for related research.
- It distinguishes accuracy from the ability to correct AI errors, which is useful for evaluating human–AI collaboration.
Critical weaknesses
- Some basic details of human experiments are unclear:
- What are the human raters’ ability levels, and can these levels be considered the same? On this basis, how are raters assigned to different assistance conditions? How is it decided who works unassisted, who sees candidate answers, and who uses AI to decompose tasks? Is the assignment random?
- What specific task instructions do human raters receive? Is there any practice before the formal experiment? In particular, how is the use of AI explained? Since the paper concerns human–AI collaboration, human raters’ priors about using AI may greatly affect the experimental results.
- Can the same participant answer the same question under different conditions and therefore be influenced by their previous answer?
- The quality-control process for human ratings is unclear. Please explain how the authors check that human raters take the rating tasks seriously and what exclusion criteria are used.
- Although the paper designs several forms of human–AI collaboration, it lacks evaluation of more advanced AI usage (e.g., assistance through various forms of agents) and usage in more realistic settings (e.g., free-form, repeated multi-turn conversations without restricting how AI is used).
- Top-N and ISD use Sonnet and Gemini, respectively. Method and model effects are therefore difficult to separate, limiting the interpretation of differences in the effects of the two methods.
Questions for authors
- Free-text answers are graded using semantic match. How exactly is semantic-match grading implemented?
Grading and study protocol clarifications
Thank you for your review! Several of these procedural details are described in the paper. We summarize them below with section references and provide additional detail where needed.
For semantic-match grading, BBEH-Safety and FACTS use GPT-5.4 mini to compare the extracted answer with the question and references. The rubric accepts equivalent meaning, minor wording differences, listed alternatives and numerical agreement within 1%. Missing, blank and “unknown” answers are incorrect. The model returns a JSON correctness verdict and reason. The revised appendix reproduces the full prompt.
Addressing your weakness concerns:
Appendix B.2 describes recruitment through separate Prolific studies. Each study had a fixed condition; participants received the unassisted, Top-N or ISD interface of the study they joined. Exclusions prevented participation in another condition for the same dataset. All conditions used the same qualification filters: “Qualified AI taskers,” “Fact-Checking” and an 80–100% approval rate. Conditions were not assigned based on ability, and we did not administer ability tests. Equal baseline ability and randomized assignment between conditions are therefore not established. Item allocation prioritized fewer-rated items with random tie-breaking.
Section 4.1 describes the assistance interfaces. Top-N showed candidate answers and rationales; ISD explained its editable, pre-filled subtasks and requested answers for uncertain ones. The revised appendix reproduces the saved instructions. There were no practice tasks or pretests, and prior beliefs about AI were not measured. Some instruction wording differed across conditions, limiting causal interpretation.
Participants could join only one condition per dataset, preventing answers to the same question under different assistance conditions (Appendix B.2).
There were no additional attention/engagement checks or manual quality exclusions. We retained the first three non-preview submissions per item and condition, yielding 35,613 ratings; another 3,869 were later quota overshoots.
Our evaluation concerns the constrained interfaces described in Section 4.1. Unrestricted, repeated interaction with an acting agent is outside its scope and remains an important extension.
Section 4.1 explains that model diversity and budget constraints motivated Top-N/Sonnet and ISD/Gemini. Each is compared with its respective model baseline. The design does not isolate method effects when comparing the assistance arms.