- Rating
- 1
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper introduces LogJudge, which detects AI-written text based on an LLM judge’s probabilities over human/machine answer tokens. FusionJudge then averages this score with a separately elicited numerical probability. The paper introduces the Manugram benchmark. Its finding is that answer-token probabilities break ties in verbalized scores and improve recall at low false-positive rates.
Critical strengths
- Focusing on low false-positive rates is a practically relevant consideration for the given cost of incorrectly flagging human writing.
- The tie-breaking ablation is helpful to get evidences regarding the mechanism behind the gains observed.
- The benchmark and independent analysis of AI-polished human writing may have some potential.
Critical weaknesses
- The core interpretation is not well grounded. The probability of generating “human” in a given prompt is not necessarily a calibrated probability of human authorship. It can be used as a classification score, but the paper does not demonstrate its interpretation as an authorship probability. Characterizing the method as recovering a finer underlying judgment thus goes beyond what is supported by the evidence.
- The comparison doesn't isolate the readout mechanism enough. BaseJudge and LogJudge use different output instructions, and different calls. Their difference may have been captured by instruction-induced differences in judgment, numerical quantization or inference variability, instead of different renderings of the same judgment. In particular, BaseJudge requests 2 decimal places explicitly. Higher-precision output, repeated elicitation, and different label verbalizations will be needed to understand the degree to which the improvement is anchored to this baseline setup.
- The fusion is not well-motivated. Two scores in don't define a common probability scale, or motivate equal weighting. The tie-breaking analysis describes some of the gain, but doesn't justify the probabilistic interpretation of averaging. FusionJudge is better justified as a heuristic score ensemble, than as a principled probability estimate.
- The evaluation does not demonstrate deployment-level false-positive control or domain robustness. Main results use thresholds chosen retrospectively on the evaluated human texts, and backbone-selection uses the evaluation splits as well. It is unclear how performance with independently-calibrated, fixed thresholds would be. Moreover, the domain-shift experiment alters the supervised detector’s training data while the evaluation slices remain fixed. It turns out that the zero-shot curves are constant by construction, which certainly does not show robustness to altering their test distribution.
- Methodological novelty and reliability evidence is still scarce. Answer-token probability readout has recognized precedents, and the fusion is a straightforward average. More robust task-specific contribution beyond enhanced score resolution is desired. Appendix V also notes large score shifts for a minority of repeated calls, but assesses decision agreement at .5 instead of actual low-FPR operating thresholds, leaving deployment stability undecided.
Questions for authors
- Do the gains hold when BaseJudge outputs higher-precision probabilities or when repeated estimates are averaged?
- What are the FPR and TPR achieved using independently calibrated thresholds that are transferred between domains?
- How sensitive are those decisions that are made at or close to the operating thresholds to label wording, label order, and repeated API-calls?
4 comments
Follow-up before the end of the discussion period
Thank you again for your review. As the discussion period ends soon, we would like to ask whether our response addresses your concerns. The revised PDF includes the analyses you suggested, and the key results are as follows. With the same call settings, LogJudge outperforms BaseJudge, also when BaseJudge outputs scores with higher precision (G1). Thresholds calibrated on held-out human texts keep the false-positive rate near 1%, and under the larger domain shifts only LogJudge and FusionJudge stay below 1% (G2). Decisions at the 1%-FPR threshold are stable under repeated calls (G4) and label order (W2 of our reply). Reviewer WpPs, who raised related points (G1, G2), found the response convincing. We believe these additions make the paper considerably stronger, and we thank you for suggesting them. We would be glad to provide any further analysis before the deadline.
Response to Authors
Thanks for the careful revisions. Since the main experiments are extensive, with a complete dataset. I increase Soundness and Informativeness. However, I stick with negative recommendation because of limited novelty and no principled mechanism. Reading top-k answer-token log-probabilities are common (e.g., G-Eval, P(True)), and the coarseness of verbalized confidence vs. token probabilities is well known. Once calibrated-probability framing is withdrawn (W1, W3), FusionJudge is simply an unweighted 0.5 average of unaligned scores, with no mechanistic explanation for when it does, and when it does not help.
Thank you for the follow-up
Thank you for reading the revision, for raising Soundness and Informativeness, and for noting that the experiments are extensive and the dataset complete.
Novelty. G-Eval and P(True) read token probabilities outside detection, and Sec. 2 cites them. That verbalized confidence is coarse is also known. What was not shown is what this costs a detector at a strict false-alarm budget. With the same judge, the readout changes AUROC by at most 0.033 on any split but raises TPR@1%FPR by 0.145 (gpt-5.6-luna) and 0.128 (gemma-4-31b). The written answer's resolution, not the judge's ranking, limits recall. The paper also makes the readout work with the few log-probabilities an API returns, by keeping the residual mass. The readout is one of the contributions, next to FusionJudge and the Manugram benchmark (about 30k texts).
Mechanism of FusionJudge. FusionJudge is a score ensemble, and Sec. 5 ("Why the readout matters, and why averaging helps", Figure 5) explains when it helps. Most of the gain does not need the two scores to share a scale. Within each written value, LogJudge breaks the ties. This rule alone lifts the four-split AT-HM from 0.653 (BaseJudge) to 0.829. Where LogJudge is confident, it also moves texts across written values, which gives the rest of the gain up to 0.848. Averaging therefore helps when the written answer orders texts well but takes only a few values, as here. It does not help at much stricter budgets, below about 0.3% FPR, where LogJudge alone is ahead (App. "Fusion Weight and Combination Rule"). Any weight from 0.05 to 0.95 gives 0.838 to 0.849, so the equal weight is not tuned.
The other reviewers found the problem well framed and Manugram's validation "broader and more credible" (Lma8), and the approach "simple and practical" with Manugram "a useful resource" (WpPs). WpPs raised their score after the revision.
Thank you again for the constructive discussion.
Response to Reviewer xghj
Thank you for the detailed review and for finding the low-FPR focus "practically relevant" and the tie-breaking ablation "helpful".
W1. The probability of answering "human" is not a calibrated probability of authorship.
Thank you for pointing this out. We did not intend to suggest a calibrated probability of authorship, and we changed the wording to rule out this reading. The paper now calls the judge scores detection scores, oriented so that a higher score means more human-like, and treats them only as such: no claim reads a score as a probability.
Changes and details: Sec. 3 defines the scores this way, and the text speaks of finer resolution, not a finer judgment.
W2. Different instructions, precision and calls mean the readout is not isolated.
Thank you for listing these factors. We tested each, and at equal compute the readout gain holds:
Changes and details: Sec. 5, App. "Test-Time Compute", "Label Order and Wording" and "Logprob Capture Reproducibility".
W3. FusionJudge is a heuristic score ensemble, not a principled probability estimate.
We agree that FusionJudge is a score ensemble, and the revision states this explicitly. The equal weight is answered in the general comment (G3): it was fixed in advance as the simplest choice, and no other weight or per-document rule does clearly better, including a tie-break rule that needs no common scale.
Changes and details: Sec. 3, App. "Fusion Weight and Combination Rule".
W4. Thresholds and backbones are chosen on the evaluation data, and the domain-shift test fixes the zero-shot test data.
We agree that with oracle thresholds the zero-shot values are flat by construction, and we added both checks you suggest (G2). In the domain-shift test every detector, zero-shot included, now sets its threshold on the training domain and is evaluated on another. Under Mid- and Far-OOD, only LogJudge and FusionJudge keep false alarms below 1%, while the trained detector flags up to 17% of the human texts.
Changes and details: Sec. 5, App. "Calibrated Thresholds" and "Baseline Backbone Selection".
W5. Novelty and reliability evidence are scarce: the readout has precedents, the fusion is a plain average, a contribution beyond score resolution is desired, and App. V (now Y) checks stability at 0.5, not at the low-FPR threshold.
The method is simple by design, which we see as a strength (as does WpPs: "Simple and practical approach"), and its novelty is what it shows for detection: at strict false-alarm rates, a strong judge's written answer is limited by its resolution, not its ranking, and recovering it raises recall at 1% FPR from 50% to 65% (76% with FusionJudge). Beyond the readout, the paper contributes Manugram, with AI-Scientist abstracts and rephrased human text as its own class, and two findings: under domain shift the judges keep false alarms under control where a trained detector does not (G2), and rephrasing affects the two detector families in opposite ways. Reliability is now checked at the 1%-FPR threshold, and the G4 gains hold under a bootstrap with refitted thresholds.
Changes and details: Sec. 2, Sec. 5, App. "Benchmark Comparison" and "Logprob Capture Reproducibility".
Q1. Do the gains hold with more decimals or averaged repeated answers?
At equal compute, yes. With reasoning, four decimals or averaged calls lift BaseJudge to or above LogJudge, at extra compute, and FusionJudge stays at least as strong at an equal number of calls (W2, G1).
Q2. What FPR and TPR do calibrated thresholds give when transferred between domains?
Between MG-Sci fields, the realized FPR averages 1.6% for FusionJudge and 1.9% for LogJudge. Under the larger shifts, only these two stay below 1% (W4, G2).
Q3. How sensitive are decisions near the threshold to label wording, order and repeated calls?
Decisions near the threshold are stable under label order and repeated calls. Other label words reduce recall at 1% FPR but largely keep the ranking (W2, G4).
Thank you again for your review. We are happy to clarify any remaining point.