acceptodds
Under review as a conference paper at ICLR 2027

Towards Human-AI Complementarity for AI Safety

Abstract

Safely overseeing AI is increasingly challenging for the humans doing it, as they must evaluate longer and more complex outputs with every new model generation. To mitigate this challenge, we could leverage complementarity: where humans and models make different mistakes, combining their judgments could yield an oversight signal stronger than either produces alone. We study this opportunity in realistic, safety-relevant settings, contributing the largest known suite of such tasks (3,957 instances across 8 datasets and 6 domains) as well as a platform and data pipeline for measuring human, AI, and human-AI performance under matched conditions. We collected 35,613 judgements from 416 human raters, finding a considerable opportunity for a complementarity effect, but also that the assistance methods we tested appeared to induce over-reliance rather than a synergy from coordinating human-AI teams. Confidence-based routing yields little improvement on average over the better single source. These results show that, while complementarity is an important and feasible engineering target for safe oversight, achieving it remains an open challenge for future work.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Human-AI complementarity, AI safety, scalable oversight, human-AI collaboration, over-reliance, confidence-based routing, uncertainty calibration

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_2kZiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies human–AI complementarity in safety-relevant tasks, comparing human, AI, and hybrid performance across multiple types of tasks. The authors evaluate Top-N assistance, subtask delegation, and confidence-based and learned routing methods. The results show an opportunity for human–AI complementarity, but assistance may reduce humans’ ability to correct AI errors. In addition, routing does not consistently outperform the better single source. The paper argues that human–AI oversight needs to preserve human corrections and find reliable ways to determine when to select human or AI answers.

Critical strengths
  • The scale of the data and ratings is relatively large, and the evaluation suite covers a relatively rich range of tasks.
  • The paper tests various possible strategies for human–AI complementarity and answer routing, providing a valuable reference for related research.
  • It distinguishes accuracy from the ability to correct AI errors, which is useful for evaluating human–AI collaboration.
Critical weaknesses
  • Some basic details of human experiments are unclear:
    • What are the human raters’ ability levels, and can these levels be considered the same? On this basis, how are raters assigned to different assistance conditions? How is it decided who works unassisted, who sees candidate answers, and who uses AI to decompose tasks? Is the assignment random?
    • What specific task instructions do human raters receive? Is there any practice before the formal experiment? In particular, how is the use of AI explained? Since the paper concerns human–AI collaboration, human raters’ priors about using AI may greatly affect the experimental results.
    • Can the same participant answer the same question under different conditions and therefore be influenced by their previous answer?
    • The quality-control process for human ratings is unclear. Please explain how the authors check that human raters take the rating tasks seriously and what exclusion criteria are used.
  • Although the paper designs several forms of human–AI collaboration, it lacks evaluation of more advanced AI usage (e.g., assistance through various forms of agents) and usage in more realistic settings (e.g., free-form, repeated multi-turn conversations without restricting how AI is used).
  • Top-N and ISD use Sonnet and Gemini, respectively. Method and model effects are therefore difficult to separate, limiting the interpretation of differences in the effects of the two methods.
Questions for authors
  • Free-text answers are graded using semantic match. How exactly is semantic-match grading implemented?
1 comment
AuthorsOpenReview ↗

Grading and study protocol clarifications

Thank you for your review! Several of these procedural details are described in the paper. We summarize them below with section references and provide additional detail where needed.

For semantic-match grading, BBEH-Safety and FACTS use GPT-5.4 mini to compare the extracted answer with the question and references. The rubric accepts equivalent meaning, minor wording differences, listed alternatives and numerical agreement within 1%. Missing, blank and “unknown” answers are incorrect. The model returns a JSON correctness verdict and reason. The revised appendix reproduces the full prompt.

Addressing your weakness concerns:

  1. Appendix B.2 describes recruitment through separate Prolific studies. Each study had a fixed condition; participants received the unassisted, Top-N or ISD interface of the study they joined. Exclusions prevented participation in another condition for the same dataset. All conditions used the same qualification filters: “Qualified AI taskers,” “Fact-Checking” and an 80–100% approval rate. Conditions were not assigned based on ability, and we did not administer ability tests. Equal baseline ability and randomized assignment between conditions are therefore not established. Item allocation prioritized fewer-rated items with random tie-breaking.

  2. Section 4.1 describes the assistance interfaces. Top-N showed candidate answers and rationales; ISD explained its editable, pre-filled subtasks and requested answers for uncertain ones. The revised appendix reproduces the saved instructions. There were no practice tasks or pretests, and prior beliefs about AI were not measured. Some instruction wording differed across conditions, limiting causal interpretation.

  3. Participants could join only one condition per dataset, preventing answers to the same question under different assistance conditions (Appendix B.2).

  4. There were no additional attention/engagement checks or manual quality exclusions. We retained the first three non-preview submissions per item and condition, yielding 35,613 ratings; another 3,869 were later quota overshoots.

  5. Our evaluation concerns the constrained interfaces described in Section 4.1. Unrestricted, repeated interaction with an acting agent is outside its scope and remains an important extension.

  6. Section 4.1 explains that model diversity and budget constraints motivated Top-N/Sonnet and ISD/Gemini. Each is compared with its respective model baseline. The design does not isolate method effects when comparing the assistance arms.

Reviewer_kCUoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies complementarity between humans and AI, specifically in the context of AI safety. The authors combined 8 benchmarks to form a dataset of about 4000 test cases on which they evaluated human-AI performance, as well as the performance of humans and AI separately. They report that humans could improve accuracy by up to 12.7 pp, and, in some cases, they perform better than when two AIs are combined. In some cases, complementarity fails, even decreasing performance.

Critical strengths
  • The evaluation is extensive, with a vast array of configurations tested, showing substantial effort in experimental coverage.
  • The focus on safety evaluations is timely and important.
Critical weaknesses
  • Novelty: Much of the work appears adapted from Xu et al. That work appears to study a closely related question, and the present paper's main difference seems to be the data used. The novelty may only be putting existing safety-related benchmarks together to investigate similar questions. The paper does not discuss in depth how its results compare with prior work. For example, the finding of inconsistent complementarity was also reported by Xu et al. (2026). That there is some opportunity for complementarity is also known. The contributions appear incremental relative to prior work because the main additions are the new benchmark combination and the safety setting.
  • Confound between model and assistance type: top-N is used with Sonnet, while a different method is used with Gemini. It is therefore unclear whether differences in performance come from the model or from the type of assistance.
  • Ethics statement raises substantial concerns: No ethics review prior to performing the study, no content warning, no data use notice or explicit consent, and Prolific identifiers are stored. Similar studies share such data (e.g., Xu et al. (2026) share data on request), so it is unclear why the ratings are not released.
Questions for authors

Clarification requests

  • How many human evaluators took part, and how long did the study take them on average?
  • Each answer was rated by 3 raters. Did all three ever give different answers in some cases, and how was such a tie broken?
  • Why was top-N used with Sonnet and the other method with Gemini only, rather than the same model for both? Why not use only Sonnet, since seems to perform better on most of the dataset. How can errors be attributed to the model versus the type of assistance?
  • When duplicate candidates in the top-N are collapsed (L201), how often does this happen, and can the user end up seeing only a top-1 answer?
  • Did randomizing the order around top-N reduce assistance errors compared with leaving the confidence/ranking in? Could it happen that participants assumed the answers were ordered and mostly choose the first one as if the order was meaningful?
  • The paper says confidence is removed in the top-N setting. Is it also removed from the wording of the answers? E.g. in the example figure 2, the third option seems to be written more confidently than the previous two.
  • What exactly is new compared with Xu et al. ? Both in terms of methodology and results observed.
  • The ratings are not released. Is this because of privacy or because consent was not obtained?

Suggestions for future work (these do not affect my assessment)

  • Compare white-box and black-box confidences, to show whether the white-box approaches improve over black-box. The authors may wish to reconsider whether 'white-box' is the most precise term for methods such as linear probes.

Minor issues and typos (these do not affect my assessment)

  • L149 - Possible typo: I see 3 interaction types overall instead of 4
  • Figure 6 has unclear legend
  • The large number of results presented in the main body hinders obtaining deeper insight due to lack of space.
6 comments
AuthorsOpenReview ↗

Clarifications and presentation updates

Thank you for your review! Addressing your clarification requests in order:

  1. All 416 Prolific participants mentioned in the paper were human evaluators. Each had one hour to complete as many questions as they could.

  2. Three-way disagreement occurred on 385 unassisted, 280 Top-N and 158 ISD items, out of 3,957 per condition. The largest answer-equivalence cluster wins; ties use mean reported confidence, then deterministic lexical order.

  3. We wanted to evaluate two model families, but budget constraints prevented testing both methods with both models. We compare Top-N/Sonnet and ISD/Gemini against their respective model baselines. This design cannot separate model effects from assistance-method effects when comparing Top-N with ISD.

  4. Yes, Top-N could display one candidate. Among 11,861 saved payloads linked to retained ratings, 600 were singletons (5.06%): 597 CulturalBench and three BBEH-Safety. Ten retained ratings lacked payloads. Original pre-deduplication outputs were unavailable, so these counts do not establish the exact duplicate-collapse rate. Binary tasks were capped at two candidates.

  5. We did not compare shuffled with ranked presentation, so we cannot establish whether shuffling reduced errors. We also cannot rule out participants assuming the first option was preferred despite the randomized order.

  6. We hid numeric rank and confidence, but retained the answer and rationale wording, including verbal confidence. You are right that the example is therefore not confidence-free.

  7. We extend Xu et al. in three main ways: collecting all three human conditions across 3,957 safety-relevant items, rather than restricting assistance to a low-confidence subset; evaluating additional learned and internal-state routing methods; and directly comparing human–AI with AI–AI complementarity. This lets us examine whether human corrections remain available—including on confidently wrong model predictions—and whether richer routing methods can recover them.

  8. Ratings are withheld pending checks of source-dataset redistribution restrictions. We also did not obtain study-specific consent, as disclosed in the ethics statement.

We have also addressed the minor issues: corrected the count to three underlying interaction types, clarified Figure 6’s legend, and moved a secondary figure to the appendix to make more room for discussion.

AuthorsOpenReview ↗

Ethics and participant data

Thank you for raising the ethics concerns. Unfortunately, no prior ethics review was obtained, and we did not provide study-specific consent, withdrawal information, content warnings or a privacy/data-use notice. We acknowledge these omissions. We received no participant complaints, study withdrawals or deletion requests.

Participants were identified by pseudonymous Prolific IDs; we did not collect names or email addresses. The IDs supported session matching and cross-condition exclusions. Residence was available for 394 participants: 169 in the UK, 148 in the US, 41 in EU countries and 36 elsewhere; 22 were unavailable.

These data were collected solely for this study. Participant-level access is restricted to the project mentees and supporting organisation. We will not use these data for further studies and will delete them shortly after the ICLR review process concludes. Ratings remain withheld pending checks of source-dataset redistribution restrictions.

We have since updated the rating platform to include study-specific consent forms and will use these in subsequent collections.

Thank you for the transparent response. To help assess appropriate remediation, could you clarify why ethics review was not sought? Specifically, was an ethics review body (institutional or via the supporting organization) available to the authors for this study? If so, would retrospective review be possible before publication?

Reviewer_xNzvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes that human and AI system are complementary in AI safety oversight. This study makes a useful empirical contribution by providing extensive experimental evidence that human-AI complementarity does indeed exist while showing that the evaluated assistance and routing methods do not consistently realize these gains.

Critical strengths
  • This study makes a useful empirical contribution by providing extensive experimental evidence for studying human-AI complementarity.
  • The negative results that the assisted and hybrid settings cannot consistently achieve the gains reported in the paper help clarify the gap between the potential for complementarity and the performance achieved by the evaluated methods
Critical weaknesses
  • For the committee-level results in Table 4, does the item-level bootstrap consider potential dependence across items as the same participants answer many questions? If not, how does this affect the interpretation of the confidence intervals?

  • Table 4 reports the paired accuracy differences but does not fully explain how their confidence intervals are constructed. How are multiple comparisons handled? If multiple comparisons are not adjusted for, individual significant differences should be interpreted carefully.

  • Appendix B.2 reports that 286 CulturalBench responses were scored by regarding the listed answers as true, and other as false.Even three additional responses could not be judged by the rule, the author handles them by hand. Could this affect the results, including the comparisions across conditions?

Questions for authors

See weakness.

1 comment
AuthorsOpenReview ↗

Statistical analysis and CulturalBench sensitivity

Thank you for your review!

Table 4’s item bootstrap does not account for dependence from participants answering multiple questions. Its intervals describe item variation conditional on the observed committees, rather than uncertainty for new participants or committees, and may understate uncertainty.

Our paired reanalysis uses the same resampled items for both systems and computes their accuracy difference within each replicate. Confidence intervals come from this bootstrap distribution. We report Holm corrections for 126 Table 4 comparisons; no aggregate unassisted-human routing gain survives correction. The original stars are labelled unadjusted and exploratory.

An additional crossed item–participant bootstrap supports the reductions in useful human corrections under assistance. This concerns individual judgments, not corrected committee-level intervals.

For CulturalBench, replay identifies 267 inferred responses and three manual judgments across 253 items; we could not reconcile the earlier count of 286. We believe interpreting listed options as true and omitted options as false is reasonable. Excluding all affected items from every condition preserves the opportunity reductions, including when all CulturalBench items are excluded. One Gemini+ISD routing gain no longer survives correction (adjusted p=.0470 before exclusion, .2605 afterward). These sensitivity analyses are reported in the revised appendix.

Reviewer_tfJsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a curated collection of AI safety datasets, supplemented with human judgments, to investigate human-AI complementarity in this domain. The authors find that humans and AI models exhibit slightly different failure patterns on the benchmark and argue that combining their strengths could offer a promising opportunity to improve AI safety. However, their evaluations do not identify a method that successfully leverages this complementarity to improve benchmark performance, either by providing humans with AI assistance or by routing decisions between humans and AI models. Nevertheless, the authors argue that human-AI complementarity should be considered and leveraged when designing methods for AI safety.

Note: This is an emergency review completed on short notice. I am reading as an informed outsider rather than as a domain expert, so my assessment focuses on clarity, soundness of the core argument, and the strength of the evidence as presented. I have not independently verified proofs, re-run experiments, or checked the related work in depth, and the area chair should weight this review accordingly, particularly on questions of novelty relative to prior work.

Critical strengths

The dataset appears to be well curated, the experiments are thoughtfully designed, and the results highlight the potential for humans and AI models to complement each other. In particular, the dataset, together with the collected human judgments, could serve as a valuable resource for future research.

Critical weaknesses

I am not entirely convinced of the practical importance of human-AI complementarity for AI safety. While the results reveal a gap that could potentially be closed by combining human and AI judgments, the overall error rate remains around 20%. This raises the question of whether improvements from leveraging complementarity would be substantial enough to meaningfully advance AI safety, or whether improving the underlying methods is more important. Furthermore, as AI models become more capable, it is unclear whether the observed complementarity will persist.

Questions for authors

Q1: How do the authors expect this field to evolve? Do they believe there will always be tasks on which humans can provide correct judgments when AI models fail? Or could human judgments become increasingly less relevant as AI models and the methods used to route decisions between humans and AI improve?

Minor comment: The Sonnet and Gemini models are not properly cited. It would be helpful to provide their full model names, including the respective providers, at least upon first mention.

2 comments
AuthorsOpenReview ↗

Response to Reviewer tfJs

Thank you for your review!

On Q1: We expect the field to focus on identifying where humans still add value and developing routing and assistance methods that use those strengths. As models and routers improve, human input may be needed less often. However, higher average capability need not eliminate useful human corrections: Morris et al. (2026) describe how models “can achieve expert-level performance on many challenging tasks while failing at others that are simple for most people.” We do not claim that human advantages will always persist; this needs to be measured as models evolve.

Thanks for the model name callout! We have added the full names at first mention: Anthropic Claude Sonnet 4.6 and Google Gemini 3 Flash Preview.

We understand the concern about the practical importance of human–AI complementarity for AI safety. In our evaluation, perfect selection between Sonnet and unassisted humans raises accuracy from 65.30% to 78.04%, a 36.71% relative reduction in model error. The remaining 21.96% error limits deployment, but we believe reducing errors by this amount would still be valuable for oversight. Our tested routers do not reliably achieve this gain; realizing it remains the research challenge. Improving models and using complementary human judgments are compatible directions.

I thank the authors for their reply. It addresses my concerns, however, I also noticed concerns from reviewer kCUo regarding the ethics of the study. I'm not an expert in this domain, so I would leave that judgment to the other reviewers/AC/ethics reviewers. I will maintain my score with my low confidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.