acceptodds
Under review as a conference paper at ICLR 2027

Better Reviews, Busier Area Chairs: Why a Little Human Judgment Matters in AI-Assisted Review

19 views

Abstract

Growing submission volumes have intensified interest in AI-assisted peer review, but its benefits depend on the expert effort required to turn reviews into reliable decisions. We study the downstream workload of Area Chairs (ACs), whose capacity to integrate reports and determine what requires further assessment can become a bottleneck. We develop a Bayesian framework linking AI-assisted reports to the AC's downstream workload, defined as the minimum additional assessment cost required to estimate paper quality at a fixed reliability target. We establish a paradox: AI-assistance can improve both individual report accuracy and the panel's aggregate assessment while increasing AC workload, even when the AC chooses assessments adaptively and optimally. We further show that a small amount of independent reviewer judgment may be sufficient to eliminate this increase and substantially reduce AC workload. Experiments using expert-annotated real reviews from PeerReview Bench are consistent with these theoretical findings. Our results highlight the value of preserving independent human judgment when designing AI-assisted review processes.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI-Assisted Peer Review, Human-AI collaboration

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Fnh7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes a theoretical framework that studies downstream workload of area chairs regarding the AI-assisted peer reviews. Authors develop a Bayesian framework that defines AC's downstream workload as the minimum additional assessment cost required to estimate paper quality at a fixed reliability target. They theoretically show that AC's workload would be increased, and the experimental results also support this.

Critical strengths
  • This paper is very well-written (and easy to follow) and addresses important research questions in the era of AI-assisted peer reviews.
  • As I read the paper, the model formulation and theorems are sound.
Critical weaknesses
  • I did not found a substantive weakness in this work.
Questions for authors

According to the results, I'm wondering what would be the optimal human:AI reviewer ratio to achieve both accurate and efficient (including AC workload) peer review system. As Figure 1(b) shows, do we only need 8% of human reviewers?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Fnh7

We thank the reviewer for their time and for the valuable question, which helped us improve the paper. We are also glad that the reviewer found the paper clear and theorems sound.

Response to Q1

Thank you for this question. In Figure 1(b), a human input of 8% is enough to avoid the increase in AC workload in the instance shown, where human input is the weight that each reviewer places on their own judgment within an AI-assisted report. This is the least input that avoids the increase, and the submission does not say how much input is optimal. We have added a subsection, "Optimal Reviewer Input" (Appendix C.9), to answer this in terms of the same weight, and we summarize it below.

What "optimal" means here. For a system that is both accurate and efficient, we fix the accuracy requirement, a tolerable error in the estimate of paper quality, and look for the input that minimizes the AC's workload needed to reach it.

The optimal input. The optimal input is higher than the 8% that avoids the increase. Write ωH,i\omega_{H,i} for the error variance, on attribute ii, of the average of the KK human-only reports, and ωD,i\omega_{D,i} for that of the average of the KK AI drafts, which includes the error shared across drafts. In the attribute-specific setting of Section 4, Proposition C.4 shows that the AC's workload is lowest when the weight on the reviewer's own judgment on attribute ii is

ei∗=ωD,iωD,i+ωH,i,e_i^\ast=\frac{\omega_{D,i}}{\omega_{D,i}+\omega_{H,i}},

and that this holds for every assessment problem and every accuracy target. The optimal human weight is thus the AI-draft panel's share of the combined error, and it grows with the error shared across AI drafts, which does not average out across reviewers.

In short, 8% is the least human input that avoids the workload increase in that instance, not the optimum. The input that is best for the AC is higher (Proposition C.4). If reviewer effort is also counted, input at the threshold of Section 4 already gives a lower total workload than human-only reviewing, whatever the cost of reviewer effort (Appendix C.7, Theorem C.1). We reproduce the new subsection below.


C.9 Optimal Reviewer Input

Section 4 characterizes the least reviewer input that avoids the increase in AC workload. Here we characterize the input that minimizes this workload, in the attribute-specific setting of Section 4. Recall that on attribute ii the panel means of human-only reports and of AI drafts have error variances ωH,i=σi/K\omega_{H,i}=\sigma_i/K and ωD,i=τi+γi/K\omega_{D,i}=\tau_i+\gamma_i/K, where σi\sigma_i, τi\tau_i, and γi\gamma_i are the variances of the human error, the shared AI error, and the draft-specific AI error. Under input E=diag(e1,…,ed)E=\mathrm{diag}(e_1,\ldots,e_d), the panel mean of assisted reports has error variance ωA,i(ei)=(1−ei)2ωD,i+ei2ωH,i\omega_{A,i}(e_i)=(1-e_i)^2\omega_{D,i}+e_i^2\omega_{H,i}, a convex quadratic in eie_i that is minimized at

ei∗:=ωD,iωD,i+ωH,i=Kτi+γiKτi+γi+σi.e_i^\ast:=\frac{\omega_{D,i}}{\omega_{D,i}+\omega_{H,i}}=\frac{K\tau_i+\gamma_i}{K\tau_i+\gamma_i+\sigma_i}.

Let E∗:=diag(e1∗,…,ed∗)E^\ast:=\mathrm{diag}(e_1^\ast,\ldots,e_d^\ast).

Proposition C.4. For every diagonal input EE and every rubric, assessment menu, positive cost vector, and target ε>0\varepsilon>0, WA(ε;E∗)≤WA(ε;E)W_A(\varepsilon;E^\ast)\le W_A(\varepsilon;E).

Proof. Since ei∗e_i^\ast minimizes ωA,i\omega_{A,i} for every ii, ΩA(E∗)⪯ΩA(E)\Omega_A(E^\ast)\preceq\Omega_A(E) for every diagonal EE. The first part of the proof of Proposition 3.2 uses the two panels only through the order of their covariances; applied to the panels with covariances ΩA(E∗)\Omega_A(E^\ast) and ΩA(E)\Omega_A(E), it gives the claim. □\square

The AC's workload is therefore lowest when each report weights the reviewer's own judgment by the AI-draft panel's share of the combined error variance. This weight increases with the shared error τi\tau_i, which does not average out across reviewers. It is larger than the threshold of Section 4: with ri=ωD,i/ωH,ir_i=\omega_{D,i}/\omega_{H,i}, $\ e_i^\ast = r_i /(r_i +1) > (r_i-1)_+ /(r_i +1) = e_{H, i} $.

Reviewer_e3m3OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper theoretically (using a Bayesian approach) and experimentally (semi-synthetic) analyzes whether AI-assisted peer review can reduce the burden on the AC’s decision-making process for a conference paper. The analysis shows that AI-assisted review can increase the burden on the AC (to reach the same reliability target) even when each individual AI-assisted report is more accurate, e.g., by 48% in an illustrative two-attribute example, and that retaining a small amount of independent human judgment within each AI-assisted review can be sufficient to avoid this increase in burden.

Critical strengths
  • The related work section is well written, providing an overview of where this paper is positioned. Some additional explanation of why such burden on AC has increased in recent years would be beneficial.

  • This paper is a theoretical paper (which lies outside my expertise; I have also notified the AC) analyzing the burden that AC would take when AI-assisted peer review takes place. Most of the current literature focuses on the burden on human reviewers, and I think such an approach to assessing the burden on AC is helpful to the community.

Critical weaknesses

Please see below.

Questions for authors

The paper’s main message, as I understand it, is that each AI-assisted peer review should contain some independent human judgment to keep the AC’s verification burden low. However, the analysis assumes an AC who knows E and all parameters (e.g., AC will already know which parts of each report reflect human judgment and how reliable they are). In practice, the AC cannot directly observe this, and identifying or assessing the independent human contribution in each report (e.g., by reading disclosed original assessments and AI interactions, as in the ICLR 2027 policy) would itself add to the AC’s workload. Does the workload reduction still hold once this cost is taken into account?

1 comment
AuthorsOpenReview ↗

Response to Reviewer e3m3

We thank the reviewer for their time and valuable feedback. We also appreciate the reviewer's encouraging comments on our approach.

Response to S1

Thank you for this suggestion. We made two additions.

In the Introduction, alongside the NeurIPS 2025 example, we added an example from journals.

Journals face the same constraint: Operations Research, a premier journal in the fields of operations research and management/decision sciences, permits Associate Editors, the journal counterparts of conference ACs, to recommend desk rejection with brief justifications in order to conserve limited editorial capacity [1].

In the Related Work (Section 1.1), we added an explanation of why the AC stage could be the bottleneck:

Downstream, reviews become decisions at the AC stage, where one experienced person reconciles several reports and decides what still needs checking. Experienced people are the slowest-growing resource in an expanding field. Submissions grow with newcomers, experienced chairs only with years spent in the field, so this stage becomes a bottleneck as the field grows [2].

[1] Amy Ward. Statement from the Operations Research Journal EIC Regarding LLM Use. INFORMS Open Forum, 2026.

[2] NeurIPS Communications Chairs. Reflections on the 2025 Review Process from the Program Committee Chairs. NeurIPS Blog, 2025.

Response to Q1

Thank you for this question. We added a subsection, "Unobserved Reviewer Input" (Appendix C.8), to address it. We show that the AC only needs to know that reviewer input is at least the threshold of Section 4, and does not need to know the exact amount EE or any parameter of the AI drafts. In this case, an AC who handles AI-assisted reports exactly as human-only reports is guaranteed a workload no higher than under human-only reviewing (Theorem C.3). If the AC has a more precise estimate of the human contribution, for example from the original assessments disclosed under the ICLR 2027 policy, the workload can be lowered further.

In Section 4, we added a short paragraph pointing to this subsection, which we reproduce below

C.8 Unobserved Reviewer Input

Section 4 takes the AC to know the input level EE and the model parameters. In practice, the AC cannot observe EE and may not know the error structure of the AI drafts. Here we show that the AC needs neither. Knowing that reviewer input is at least the threshold is enough to avoid the workload increase, with no cost of identifying the human contribution in each report.

Suppose the reports are assisted with some input EE that the AC does not observe, and the AC proceeds exactly as under human-only reviewing. For a fixed allocation t∈R+Lt\in\mathbb{R}^L_+, the AC combines the panel mean YˉA\bar Y^A with the AC's own assessments as if the reports were human-only,

m~(t):=V~(t)(Λ−1m0+ΩH−1YˉA+∑ℓ:tℓ>0tℓaℓZℓ),V~(t):=(Λ−1+ΩH−1+J(t))−1,\tilde m(t):=\tilde V(t)(\Lambda^{-1}m_0+\Omega_H^{-1}\bar Y^A+\sum_{\ell:t_\ell>0} t_\ell a_\ell Z_\ell),\qquad\tilde V(t):=(\Lambda^{-1}+\Omega_H^{-1}+J(t))^{-1},

and evaluates the allocation by the human-only risk RH(t)=θ⊤V~(t)θ\mathcal{R}_H(t)=\theta^\top\tilde V(t)\theta. This uses only what the AC would need without AI assistance.

Theorem C.3. Suppose ΩA(E)⪯ΩH\Omega_A(E)\preceq\Omega_H. Then for every fixed allocation t∈R+Lt\in\mathbb{R}^L_+,

E[(θ⊤m~(t)−q)2]≤RH(t).\mathbb{E}[(\theta^\top\tilde m(t)-q)^2]\le\mathcal{R}_H(t).

Consequently, every allocation that reaches the target ε\varepsilon under human-only reviewing also reaches it here, and the AC needs a workload of at most WH(ε)W_H(\varepsilon) without knowing EE.

Proof. Write YˉA=x+ξ\bar Y^A=x+\xi with ξ∼N(0,ΩA(E))\xi\sim\mathcal{N}(0,\Omega_A(E)), and Zℓ=aℓ⊤x+ηℓZ_\ell=a_\ell^\top x+\eta_\ell with ηℓ∼N(0,1/tℓ)\eta_\ell\sim\mathcal{N}(0,1/t_\ell) for tℓ>0t_\ell>0. Since V~(t)(Λ−1+ΩH−1+J(t))=I\tilde V(t)(\Lambda^{-1}+\Omega_H^{-1}+J(t))=I,

m~(t)−x=V~(t)(Λ−1(m0−x)+ΩH−1ξ+∑ℓ:tℓ>0tℓaℓηℓ).\tilde m(t)-x=\tilde V(t)(\Lambda^{-1}(m_0-x)+\Omega_H^{-1}\xi+\sum_{\ell:t_\ell>0}t_\ell a_\ell\eta_\ell).

For fixed tt, the variables m0−xm_0-x, ξ\xi, and (ηℓ)(\eta_\ell) are independent with mean zero, so the three terms in parentheses have covariances Λ−1\Lambda^{-1}, ΩH−1ΩA(E)ΩH−1\Omega_H^{-1}\Omega_A(E)\Omega_H^{-1}, and J(t)J(t). The error covariance is therefore

V~(t)(Λ−1+ΩH−1ΩA(E)ΩH−1+J(t))V~(t)=V~(t)+V~(t) ΩH−1(ΩA(E)−ΩH)ΩH−1 V~(t)⪯V~(t),\tilde V(t)(\Lambda^{-1}+\Omega_H^{-1}\Omega_A(E)\Omega_H^{-1}+J(t))\tilde V(t) =\tilde V(t)+\tilde V(t)\,\Omega_H^{-1}\big(\Omega_A(E)-\Omega_H\big)\Omega_H^{-1}\,\tilde V(t) \preceq\tilde V(t),

because ΩA(E)⪯ΩH\Omega_A(E)\preceq\Omega_H. Taking the quadratic form in θ\theta gives the bound. The last claim follows by taking the human-only optimal allocation, whose cost is WH(ε)W_H(\varepsilon). □\square

By Theorem 4.1, the condition holds for every uniform input e∈[eH,1]e\in[e_H,1]. A venue policy that secures this minimum input therefore lets the AC ignore how each report was produced, and the AC’s workload is no higher than the human-only workload WH(ε)W_H(\varepsilon).

Knowing the actual report covariance ΩA(E)\Omega_A(E) can lower the workload further, to WA(ε;E)≤WH(ε)W_A(\varepsilon;E)\le W_H(\varepsilon), so reading disclosures is optional and pays when its cost is below that additional saving.

Reviewer_ceY2OpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the downstream stage of the peer-review process. Its central argument is that AI can make individual reviews more accurate while still increasing the subsequent verification burden on Area Chairs. Retaining a small amount of independent human judgment may alleviate this problem. The paper considers "whether information across reviews is complementary" and "whether the remaining uncertainty is concentrated in issues that are costly to verify". Its theoretical framework refines the question of "how much error remains" into "where the remaining errors are and how much it costs to resolve them".

Critical strengths
  • The paper is novel and offers a valuable research perspective. Few studies on AI review focus on Area Chairs.
  • It expands the evaluation objective from "whether an individual review is accurate" to "how much expert work is required to ultimately reach a reliable judgment". This is valuable for assessing the practical benefits of AI-assisted review.
  • The paper provides formal theoretical results that go beyond simply discussing review homogeneity. The dimensionality reduction in Theorem 3.3 is elegant.
  • The paper offers an insightful conclusion: even when the panel's estimate of overall paper quality improves, the cost-weighted remaining uncertainty may still increase AC workload.
Critical weaknesses
  • The paper studies risks arising from AI review, such as review homogeneity and the effects of errors on collective judgment, but lacks citations to relevant existing work on AI review risks (for example, [1] AI Review Is a Systemic Risk to Peer Review: Toward a Blockchain-Supported Claim-Level Ledger for Accountability, ICML 2026). The Introduction and Related Work could further strengthen their connection to research on systemic risks and accountability in AI review [1]. Such an addition would help clarify the positioning of this work.

  • There is a substantial gap between the theory and the experiments. The theory studies the minimum cost-weighted independent verification required to achieve a fixed error level in paper-quality estimation. The experiments study how many items must be inspected in a model-ranked list of review comments to find three valid concerns.

    The connection between the two is unclear:

    • Why does finding three valid concerns imply that the same level of decision reliability has been achieved?
    • Why can three localized issues and one decisive technical error be compared by counting items?
    • Why can comments with different verification difficulties be counted as having equal verification costs?
  • The experiments do not provide statistically significant validation. The theory emphasizes that the panel's overall estimate improves while the AC becomes busier. In the experiments, however, the AI panel has worse coverage and fewer valid items among the first five. Moreover, the only statistically significant difference is in LFP, which measures repetition rather than AC workload.

I will decide whether to raise or lower my score based on the authors' rebuttal.

Questions for authors
  • How do the experiments correspond to the theory on a one-to-one basis? How do the experimental symbols and values (NFP, LFP, FP5, R3) map to the theoretical quantities (Ω or r)?
  • Should human input also incur a cost in the model?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ceY2, Part 1

We sincerely thank the reviewer for the constructive and detailed comments. We also appreciate the reviewer’s acknowledgment of the strengths of our work.

Response to W1

Thank you for pointing out this gap. We now discuss systemic risk in both the Introduction and the Related Work. Specifically, we added the following paragraph to Section 1.1.

Systemic risks, accountability and governance of AI review. Beyond the quality of individual reviews, a growing body of work treats AI review as a systemic risk to the review process and asks how it should be governed. One line of work detects and monitors LLM-written reviews [2][3]; another evaluates LLM-use policies experimentally [4] or designs review mechanisms [5]. On accountability, Miao et al. [1] view AI review as a systemic risk and propose ledgers that record each review claim so that authors can dispute it and ACs can rule on it. Venues have begun adopting detection and disclosure requirements [6][7][8]. Our results identify a systemic risk of this kind that is invisible at the level of individual reviews. Assistance can make every report more accurate while leaving the panel less informative and placing more burden on the AC. It could also guide the design of mechanisms.

On positioning. Our finding can be read as a systemic risk of AI review: it arises even when every individual review is more accurate. It also guides the mechanism design. Requiring reviewers to retain a small amount of independent judgment is enough to keep the AC's burden from growing.

[1] Miao et al. AI Review Is a Systemic Risk to Peer Review: Toward a Blockchain-Supported Claim-Level Ledger for Accountability. ICML, 2026.

[2] Liang et al. Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews. arXiv, 2024.

[3] Rao et al. Detecting LLM-Generated Peer Reviews. PLoS ONE, 2025.

[4] Kim et al. Use and Effects of LLMs in Peer Review: A Randomized Experiment and Survey at ICML 2026. arXiv, 2026.

[5] Gans. Designing AI-Augmented Peer Review. NBER, 2026.

[6] ICLR 2026 Program Chairs. ICLR 2026 Response to LLM-Generated Papers and Reviews. ICLR Blog, 2025.

[7] ICML 2026 Program Chairs. On Violations of LLM Review Policies. ICML Blog, 2026.

[8] ICLR. Policy on LLM Use for Writing Reviews or Meta Reviews. 2026.

Response to Q1 and Part of W2: Connection between Model and Experiment

Thank you for this important comment. The submission left the connection implicit, and the revision now states it explicitly. The three specific questions in W2 are answered in the next comment.

Making the connection explicit. Section 5 now opens by stating what the experiments examine and points to the table of correspondences (lines 404–407). In Section 5.1, each measurement is introduced together with the quantity in the model it stands for. For example:

  • FP items and the FP rate (lines 414–419). After the definition of the fully positive criterion, we add: "An FP item is thus an expert-certified assessment of one important aspect of the paper, and the FP rate of a report is our measure of individual report accuracy, the counterpart of Assumption 2.1." We also give an example of an FP item.
  • AI reports and human-informed revisions (lines 423–425, 448–451). We state that the released AI reports stand for the draft-dominated endpoint E=0E=0, and the exposure fraction α\alpha is our proxy for the input weight EE.
  • Item groups and coverage (lines 455–457). We state that an item group corresponds to one aspect of the paper, which may involve one or several attributes xix_i, and NFPN_{\mathrm{FP}} could be read as the amount of important information the panel carries.
  • Repetition (lines 459–462). We explain why LFPL_{\mathrm{FP}} matters. Repetition is the observable proxy of the shared error.
  • Inspection and workload (lines 463–471). We state that each inspection is an assessment task at unit cost, confirming three FP concerns plays the role of the reliability target ε\varepsilon, FP5FP_5 is the information obtained at a fixed cost, and R3R_3 is the inspection cost of reaching the target.

Appendix D.3 (Table 3) collects the correspondences, reproduced here.

ModelExperiment proxy
Attributes xx of the paperItem groups; each group concerns one or more attributes
Individual accuracyFP rate of a report
Human-only report YjHY_j^HHuman report HH
AI draft, E=0E=0AI report AA
Input weight EEExposure fraction α\alpha
Information in the panel, Ωg−1\Omega_g^{-1}NFPN_{\mathrm{FP}}, distinct FP concerns in the panel
Shared error TTLFPL_{\mathrm{FP}} in excess of the human-panel level
Assessment task, cost cℓc_\ellInspection of one representative item at unit cost
Target ε\varepsilonConfirm three distinct FP concerns
Minimum workload Wg(ε)W_g(\varepsilon)R3=min⁡{pos3,15}R_3=\min\lbrace \mathrm{pos}_3,15\rbrace
Risk Rg(t)R_g(t) at fixed effortFP5FP_5
AuthorsOpenReview ↗

Response to Reviewer ceY2, Part 2

Response to the Rest of W2

Why three confirmed concerns stand for a reliability level. Each confirmed FP concern removes the AC's uncertainty about one aspect of the paper, so the number of confirmed concerns measures how much of that uncertainty has been resolved, and a fixed number of confirmed concerns plays the role of a fixed reliability target.

We chose three for a reason specific to the data. A report holds about two FP concerns on average and a panel five to six. Three is therefore the smallest target that a single report does not meet on average, so the AC must combine reports to reach it, while most panels can still supply it. A target of five would use up nearly all the FP concerns an average panel holds, and some panels could not supply it. A target of two would not require a panel at all. The comparison does not rest on this choice alone. FP5FP_5, which fixes the inspection cost instead of the target, gives a reading that involves no target. The revision now explains this choice in Appendix D.2.

Localized issues versus a decisive error. The submission defined the criterion but did not clearly spell out what it implies. We use the fully positive criterion defined by the authors of the dataset [9]: an item counts only if domain experts rate it correct, significant, and supported by sufficient evidence, that is, positive on all three dimensions. Every counted item is therefore an expert-certified concern about an important aspect of the paper, and issues that experts do not rate as significant are not counted. The revision now says this where the criterion is introduced (lines 414–419), gives an example of an FP item that identifies a temporal leakage invalidating a paper's primary prediction task (lines 416–419).

Verification costs. Thank you for raising this point. Verification costs do differ across comments, and our experiment does not capture this. Each inspection is counted at unit cost. The dataset contains no measurement of how costly a comment is to verify, and this cost is not a property of the comment alone, since it also depends on the AC's expertise. We now note this simplification in Section 6 and leave the measurement of verification effort to future work.

[9] Kim et al. On the Limits and Opportunities of AI Reviewers: Reviewing the Reviews of Nature-Family Papers with 45 Expert Scientists. arXiv, 2026.

Response to W3

Thank you for this comment. We agree that the contrasts between human and AI panels are not statistically significant. Our sample is small (54 papers), and we now state this explicitly as a limitation in Section 6. We would like to note two things. First, every point estimate moves in the direction of the paradox: AI reports have a higher individual FP rate, while AI panels have lower coverage, fewer FP concerns among the first five inspections, and a higher inspection burden (Table 1). We read this as support for the possibility of the paradox in real reviews. Second, the remedy is statistically significant. With limited human input, inspection burden falls (ΔR3=−1.15\Delta R_3=-1.15 [−1.94,−0.44][-1.94,-0.44]) and distinct coverage rises (ΔNFP=+0.98\Delta N_{\mathrm{FP}}=+0.98 [0.70,1.26][0.70,1.26]) at 25% exposure (Table 2).

Response to Q2

Thank you for this question. Our theory extends to this setting, and its conclusion is unchanged: a small amount of independent reviewer judgment remains worth its cost. We added a subsection, "Costly Reviewer Input" (Appendix C.7), that makes this precise.In brief, once reviewer effort is added to the objective, we prove two results. First, assisted reviewing with input eHe_H has a lower total workload than human-only reviewing, whatever the cost of reviewer effort (Theorem C.1). Second, the AC's saving is first order in the input, so a small amount of input also lowers the total workload relative to unmodified AI drafts unless reviewer effort is expensive relative to AC verification (Proposition C.2).

Theorem C.1. Let eHe_H be the threshold of Theorem 4.1. For every rubric, assessment menu, positive cost vector, and target ε>0\varepsilon>0 with WH(ε)<∞W_H(\varepsilon)<\infty, and every hR>0h_R>0,

Wtot(ε;eH) ≤ Wtot(ε;1)−hRK(1−eH) < Wtot(ε;1).W_{\mathrm{tot}}(\varepsilon;e_H)\ \le\ W_{\mathrm{tot}}(\varepsilon;1)-h_RK(1-e_H)\ <\ W_{\mathrm{tot}}(\varepsilon;1).

Proposition C.2. Under the hypotheses of Theorem 3.3, suppose ε<ε∗(eI)\varepsilon<\varepsilon_\ast(eI) for all e∈[0,δ)e\in[0,\delta), for some δ>0\delta>0. Then the right derivative of the total workload at e=0e=0 is hRK−2W0 w∗⊤ΩD−1w∗h_RK-2W_0\,w^{\ast\top}\Omega_D^{-1}w^{\ast}. In particular, if hRK<2W0w∗⊤ΩD−1w∗h_RK<2W_0 w^{\ast\top}\Omega_D^{-1}w^{\ast}, some positive input level gives a strictly lower total workload than AI drafts.

The conclusion of Section 4 therefore carries over: input eHe_H gives a lower total workload than human-only reviewing whatever the cost of reviewer effort, and a small input also does better than AI drafts unless reviewer effort is expensive relative to AC verification. The proofs and discussions are in Appendix C.7.

The authors have addressed my concerns, so I have decided to raise my score.

We are very glad to hear that our response has addressed your concerns. We sincerely thank you for your valuable comments and thoughtful feedback, which have helped us improve the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We thank all reviewers and the AC for their time and for helping us improve the paper. We have uploaded a revised version. The main changes are listed below.

Main text

  • Introduction and Related Work. Added a discussion of the systemic risks of AI review, an example from a journal, and an explanation of why the AC stage can become a bottleneck. (Reviewers ceY2, e3m3)
  • Section 4. Added a closing paragraph pointing to the three new appendix results below. (Reviewers ceY2, e3m3, Fnh7)
  • Section 5. Added explicit connections between the experimental measurements and the model quantities, with an example of a fully positive review item. (Reviewer ceY2)
  • Section 6. Expanded the discussion of limitations. (Reviewer ceY2)

Appendix

  • C.7 Costly Reviewer Input: total workload when reviewer input is counted. (Reviewer ceY2)
  • C.8 Unobserved Reviewer Input: the AC does not need to know the reviewer input weight, only that it meets the threshold of Section 4. (Reviewer e3m3)
  • C.9 Optimal Reviewer Input: the input weight that minimizes the AC's workload. (Reviewer Fnh7)
  • D.2 and D.3: the choice of the inspection target, and a table of correspondences between model quantities and experimental proxies. (Reviewer ceY2)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.