- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work proposes a theoretical framework that studies downstream workload of area chairs regarding the AI-assisted peer reviews. Authors develop a Bayesian framework that defines AC's downstream workload as the minimum additional assessment cost required to estimate paper quality at a fixed reliability target. They theoretically show that AC's workload would be increased, and the experimental results also support this.
Critical strengths
- This paper is very well-written (and easy to follow) and addresses important research questions in the era of AI-assisted peer reviews.
- As I read the paper, the model formulation and theorems are sound.
Critical weaknesses
- I did not found a substantive weakness in this work.
Questions for authors
According to the results, I'm wondering what would be the optimal human:AI reviewer ratio to achieve both accurate and efficient (including AC workload) peer review system. As Figure 1(b) shows, do we only need 8% of human reviewers?
Response to Reviewer Fnh7
We thank the reviewer for their time and for the valuable question, which helped us improve the paper. We are also glad that the reviewer found the paper clear and theorems sound.
Response to Q1
Thank you for this question. In Figure 1(b), a human input of 8% is enough to avoid the increase in AC workload in the instance shown, where human input is the weight that each reviewer places on their own judgment within an AI-assisted report. This is the least input that avoids the increase, and the submission does not say how much input is optimal. We have added a subsection, "Optimal Reviewer Input" (Appendix C.9), to answer this in terms of the same weight, and we summarize it below.
What "optimal" means here. For a system that is both accurate and efficient, we fix the accuracy requirement, a tolerable error in the estimate of paper quality, and look for the input that minimizes the AC's workload needed to reach it.
The optimal input. The optimal input is higher than the 8% that avoids the increase. Write ωH,i for the error variance, on attribute i, of the average of the K human-only reports, and ωD,i for that of the average of the K AI drafts, which includes the error shared across drafts. In the attribute-specific setting of Section 4, Proposition C.4 shows that the AC's workload is lowest when the weight on the reviewer's own judgment on attribute i is
ei∗=ωD,i+ωH,iωD,i,
and that this holds for every assessment problem and every accuracy target. The optimal human weight is thus the AI-draft panel's share of the combined error, and it grows with the error shared across AI drafts, which does not average out across reviewers.
In short, 8% is the least human input that avoids the workload increase in that instance, not the optimum. The input that is best for the AC is higher (Proposition C.4). If reviewer effort is also counted, input at the threshold of Section 4 already gives a lower total workload than human-only reviewing, whatever the cost of reviewer effort (Appendix C.7, Theorem C.1). We reproduce the new subsection below.
C.9 Optimal Reviewer Input
Section 4 characterizes the least reviewer input that avoids the increase in AC workload. Here we characterize the input that minimizes this workload, in the attribute-specific setting of Section 4. Recall that on attribute i the panel means of human-only reports and of AI drafts have error variances ωH,i=σi/K and ωD,i=τi+γi/K, where σi, τi, and γi are the variances of the human error, the shared AI error, and the draft-specific AI error. Under input E=diag(e1,…,ed), the panel mean of assisted reports has error variance ωA,i(ei)=(1−ei)2ωD,i+ei2ωH,i, a convex quadratic in ei that is minimized at
ei∗:=ωD,i+ωH,iωD,i=Kτi+γi+σiKτi+γi.
Let E∗:=diag(e1∗,…,ed∗).
Proposition C.4. For every diagonal input E and every rubric, assessment menu, positive cost vector, and target ε>0, WA(ε;E∗)≤WA(ε;E).
Proof. Since ei∗ minimizes ωA,i for every i, ΩA(E∗)⪯ΩA(E) for every diagonal E. The first part of the proof of Proposition 3.2 uses the two panels only through the order of their covariances; applied to the panels with covariances ΩA(E∗) and ΩA(E), it gives the claim. □
The AC's workload is therefore lowest when each report weights the reviewer's own judgment by the AI-draft panel's share of the combined error variance. This weight increases with the shared error τi, which does not average out across reviewers. It is larger than the threshold of Section 4: with ri=ωD,i/ωH,i, $\ e_i^\ast = r_i /(r_i +1) > (r_i-1)_+ /(r_i +1) = e_{H, i} $.