acceptodds
Under review as a conference paper at ICLR 2027

Question of Confidence: A Zero-Shot AI-Text Detector and Benchmark Suite

Abstract

Ask a large language model whether a person or a machine wrote a text, and it answers in words. Before it writes “human” or “machine”, the model already holds a probability distribution over the possible answers. We show that this distribution is a better detection signal than a probability the model is asked to write out, its stated probability. We propose two detectors built on it. LogJudge asks the model once and scores the text by the token probability it places on “human” versus “machine” before it answers. FusionJudge averages the token probability with the stated probability. Since a detector must rarely accuse a human, we count how many machine-written texts each detector catches while flagging at most 1% of human-written ones. Two state-of-the-art detectors, Fast-DetectGPT and Binoculars, catch 52% and 51%. With a low-cost commercial model as the judge, the stated probability catches 50%, LogJudge 65%, and FusionJudge 76%. The stated probability orders texts well, but it takes only a few rounded values, so many texts tie right where the decision is made. The token probability is fine-grained and separates them. Their average, FusionJudge, keeps the stated probability’s order, breaks its ties with the token probability, and follows the token probability where it is confident. Current detectors are near ceiling on benchmarks written by 2023-era models, so we release Manugram, about 30,000 texts that recreate a standard benchmark with four current models and add scientific abstracts, some of them written by automated AI-Scientist systems. Averaged over the four benchmarks, FusionJudge is the strongest detector we tested. Neither detector needs training, so their accuracy holds when the topic changes, while a trained detector’s accuracy collapses. They also read only the few probabilities that commercial APIs already return, which means strong API models can be used as detectors without any extra work. We release the code and data.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
machine-generated text detection, zero-shot detector, benchmarking, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xghjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper introduces LogJudge, which detects AI-written text based on an LLM judge’s probabilities over human/machine answer tokens. FusionJudge then averages this score with a separately elicited numerical probability. The paper introduces the Manugram benchmark. Its finding is that answer-token probabilities break ties in verbalized scores and improve recall at low false-positive rates.

Critical strengths
  1. Focusing on low false-positive rates is a practically relevant consideration for the given cost of incorrectly flagging human writing.
  2. The tie-breaking ablation is helpful to get evidences regarding the mechanism behind the gains observed.
  3. The benchmark and independent analysis of AI-polished human writing may have some potential.
Critical weaknesses
  1. The core interpretation is not well grounded. The probability of generating “human” in a given prompt is not necessarily a calibrated probability of human authorship. It can be used as a classification score, but the paper does not demonstrate its interpretation as an authorship probability. Characterizing the method as recovering a finer underlying judgment thus goes beyond what is supported by the evidence.
  2. The comparison doesn't isolate the readout mechanism enough. BaseJudge and LogJudge use different output instructions, and different calls. Their difference may have been captured by instruction-induced differences in judgment, numerical quantization or inference variability, instead of different renderings of the same judgment. In particular, BaseJudge requests 2 decimal places explicitly. Higher-precision output, repeated elicitation, and different label verbalizations will be needed to understand the degree to which the improvement is anchored to this baseline setup.
  3. The fusion is not well-motivated. Two scores in [0,1][0,1] don't define a common probability scale, or motivate equal weighting. The tie-breaking analysis describes some of the gain, but doesn't justify the probabilistic interpretation of averaging. FusionJudge is better justified as a heuristic score ensemble, than as a principled probability estimate.
  4. The evaluation does not demonstrate deployment-level false-positive control or domain robustness. Main results use thresholds chosen retrospectively on the evaluated human texts, and backbone-selection uses the evaluation splits as well. It is unclear how performance with independently-calibrated, fixed thresholds would be. Moreover, the domain-shift experiment alters the supervised detector’s training data while the evaluation slices remain fixed. It turns out that the zero-shot curves are constant by construction, which certainly does not show robustness to altering their test distribution.
  5. Methodological novelty and reliability evidence is still scarce. Answer-token probability readout has recognized precedents, and the fusion is a straightforward average. More robust task-specific contribution beyond enhanced score resolution is desired. Appendix V also notes large score shifts for a minority of repeated calls, but assesses decision agreement at .5 instead of actual low-FPR operating thresholds, leaving deployment stability undecided.
Questions for authors
  1. Do the gains hold when BaseJudge outputs higher-precision probabilities or when repeated estimates are averaged?
  2. What are the FPR and TPR achieved using independently calibrated thresholds that are transferred between domains?
  3. How sensitive are those decisions that are made at or close to the operating thresholds to label wording, label order, and repeated API-calls?
4 comments
AuthorsOpenReview ↗

Response to Reviewer xghj

Thank you for the detailed review and for finding the low-FPR focus "practically relevant" and the tie-breaking ablation "helpful".

W1. The probability of answering "human" is not a calibrated probability of authorship.

Thank you for pointing this out. We did not intend to suggest a calibrated probability of authorship, and we changed the wording to rule out this reading. The paper now calls the judge scores detection scores, oriented so that a higher score means more human-like, and treats them only as such: no claim reads a score as a probability.

Changes and details: Sec. 3 defines the scores this way, and the text speaks of finer resolution, not a finer judgment.

W2. Different instructions, precision and calls mean the readout is not isolated.

Thank you for listing these factors. We tested each, and at equal compute the readout gain holds:

  • Call settings, precision and prompt wording (general comment, G1): called exactly like LogJudge, BaseJudge stays at a TPR@1%FPR of 0.058 to 0.126 at any precision, against 0.648 for LogJudge.
  • Repeated API calls: the order BaseJudge < LogJudge < FusionJudge holds in every capture (general comment, G4).
  • Repeated elicitation: averaging repeated answers does help BaseJudge, since it also breaks ties, but each added call is extra test-time compute. Without reasoning, as in LogJudge's call, even three averaged BaseJudge samples stay far below one LogJudge call. With reasoning, only four averaged BaseJudge calls slightly pass FusionJudge.
  • Label order and words: reversing the label order leaves 93% of the decisions at the threshold unchanged. Other label words largely keep the ranking but reduce recall at 1% FPR, because the judge becomes fully certain on the most machine-like texts and ties them at 0. Human/machine avoids this.

Changes and details: Sec. 5, App. "Test-Time Compute", "Label Order and Wording" and "Logprob Capture Reproducibility".

W3. FusionJudge is a heuristic score ensemble, not a principled probability estimate.

We agree that FusionJudge is a score ensemble, and the revision states this explicitly. The equal weight is answered in the general comment (G3): it was fixed in advance as the simplest choice, and no other weight or per-document rule does clearly better, including a tie-break rule that needs no common scale.

Changes and details: Sec. 3, App. "Fusion Weight and Combination Rule".

W4. Thresholds and backbones are chosen on the evaluation data, and the domain-shift test fixes the zero-shot test data.

We agree that with oracle thresholds the zero-shot values are flat by construction, and we added both checks you suggest (G2). In the domain-shift test every detector, zero-shot included, now sets its threshold on the training domain and is evaluated on another. Under Mid- and Far-OOD, only LogJudge and FusionJudge keep false alarms below 1%, while the trained detector flags up to 17% of the human texts.

Changes and details: Sec. 5, App. "Calibrated Thresholds" and "Baseline Backbone Selection".

W5. Novelty and reliability evidence are scarce: the readout has precedents, the fusion is a plain average, a contribution beyond score resolution is desired, and App. V (now Y) checks stability at 0.5, not at the low-FPR threshold.

The method is simple by design, which we see as a strength (as does WpPs: "Simple and practical approach"), and its novelty is what it shows for detection: at strict false-alarm rates, a strong judge's written answer is limited by its resolution, not its ranking, and recovering it raises recall at 1% FPR from 50% to 65% (76% with FusionJudge). Beyond the readout, the paper contributes Manugram, with AI-Scientist abstracts and rephrased human text as its own class, and two findings: under domain shift the judges keep false alarms under control where a trained detector does not (G2), and rephrasing affects the two detector families in opposite ways. Reliability is now checked at the 1%-FPR threshold, and the G4 gains hold under a bootstrap with refitted thresholds.

Changes and details: Sec. 2, Sec. 5, App. "Benchmark Comparison" and "Logprob Capture Reproducibility".

Q1. Do the gains hold with more decimals or averaged repeated answers?

At equal compute, yes. With reasoning, four decimals or averaged calls lift BaseJudge to or above LogJudge, at extra compute, and FusionJudge stays at least as strong at an equal number of calls (W2, G1).

Q2. What FPR and TPR do calibrated thresholds give when transferred between domains?

Between MG-Sci fields, the realized FPR averages 1.6% for FusionJudge and 1.9% for LogJudge. Under the larger shifts, only these two stay below 1% (W4, G2).

Q3. How sensitive are decisions near the threshold to label wording, order and repeated calls?

Decisions near the threshold are stable under label order and repeated calls. Other label words reduce recall at 1% FPR but largely keep the ranking (W2, G4).

Thank you again for your review. We are happy to clarify any remaining point.

AuthorsOpenReview ↗

Follow-up before the end of the discussion period

Thank you again for your review. As the discussion period ends soon, we would like to ask whether our response addresses your concerns. The revised PDF includes the analyses you suggested, and the key results are as follows. With the same call settings, LogJudge outperforms BaseJudge, also when BaseJudge outputs scores with higher precision (G1). Thresholds calibrated on held-out human texts keep the false-positive rate near 1%, and under the larger domain shifts only LogJudge and FusionJudge stay below 1% (G2). Decisions at the 1%-FPR threshold are stable under repeated calls (G4) and label order (W2 of our reply). Reviewer WpPs, who raised related points (G1, G2), found the response convincing. We believe these additions make the paper considerably stronger, and we thank you for suggesting them. We would be glad to provide any further analysis before the deadline.

Reviewer_xghjOpenReview ↗

Response to Authors

Thanks for the careful revisions. Since the main experiments are extensive, with a complete dataset. I increase Soundness and Informativeness. However, I stick with negative recommendation because of limited novelty and no principled mechanism. Reading top-k answer-token log-probabilities are common (e.g., G-Eval, P(True)), and the coarseness of verbalized confidence vs. token probabilities is well known. Once calibrated-probability framing is withdrawn (W1, W3), FusionJudge is simply an unweighted 0.5 average of unaligned scores, with no mechanistic explanation for when it does, and when it does not help.

AuthorsOpenReview ↗

Thank you for the follow-up

Thank you for reading the revision, for raising Soundness and Informativeness, and for noting that the experiments are extensive and the dataset complete.

Novelty. G-Eval and P(True) read token probabilities outside detection, and Sec. 2 cites them. That verbalized confidence is coarse is also known. What was not shown is what this costs a detector at a strict false-alarm budget. With the same judge, the readout changes AUROC by at most 0.033 on any split but raises TPR@1%FPR by 0.145 (gpt-5.6-luna) and 0.128 (gemma-4-31b). The written answer's resolution, not the judge's ranking, limits recall. The paper also makes the readout work with the few log-probabilities an API returns, by keeping the residual mass. The readout is one of the contributions, next to FusionJudge and the Manugram benchmark (about 30k texts).

Mechanism of FusionJudge. FusionJudge is a score ensemble, and Sec. 5 ("Why the readout matters, and why averaging helps", Figure 5) explains when it helps. Most of the gain does not need the two scores to share a scale. Within each written value, LogJudge breaks the ties. This rule alone lifts the four-split AT-HM from 0.653 (BaseJudge) to 0.829. Where LogJudge is confident, it also moves texts across written values, which gives the rest of the gain up to 0.848. Averaging therefore helps when the written answer orders texts well but takes only a few values, as here. It does not help at much stricter budgets, below about 0.3% FPR, where LogJudge alone is ahead (App. "Fusion Weight and Combination Rule"). Any weight from 0.05 to 0.95 gives 0.838 to 0.849, so the equal weight is not tuned.

The other reviewers found the problem well framed and Manugram's validation "broader and more credible" (Lma8), and the approach "simple and practical" with Manugram "a useful resource" (WpPs). WpPs raised their score after the revision.

Thank you again for the constructive discussion.

Reviewer_WpPsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LogJudge, a zero-shot AI-text detector that derives detection scores from label-token probabilities at the answer position, and FusionJudge, which combines these scores with the model's explicitly stated probabilities. The authors also introduce the Manugram benchmark and report that both methods outperform the evaluated baselines on average in oracle TPR@1%FPR.

Critical strengths
  1. Simple and practical approach. The methods require no additional training and can use the top-k token probabilities available through commercial APIs.
  2. Focus on low false-positive rates. The evaluation emphasizes a practically relevant operating regime, and the analysis of score ties near the decision threshold offers useful insight into the performance gains.
  3. Valuable benchmark contribution. Manugram covers multiple domains, generators, and paraphrased texts, providing a useful resource for evaluating AI-text detectors.
Critical weaknesses
  1. The treatment of residual probability mass requires targeted validation. Assigning a neutral value of 0.5 to probability mass outside the top-k window or unmapped to valid labels needs further justification. Please report label-absence rates and residual-mass distributions near the decision threshold, and evaluate how different k values and residual-mass treatments affect TPR@1%FPR.
  2. The gains have not been sufficiently disentangled from output precision and prompt format. BaseJudge is instructed to output only two decimal places, whereas LogJudge uses finer-grained token probabilities and different output instructions, so the gains cannot yet be attributed entirely to "different readouts of the same judgment." Please increase BaseJudge's output precision, repeat the comparison under the same oracle TPR@1%FPR protocol, and report the number of distinct scores and the sizes of tied groups near the threshold.
  3. The evaluation does not sufficiently support claims of cross-domain stability or low-FPR deployment. Oracle TPR@1%FPR is a reasonable measure of discriminative ability, but its threshold depends on human-class labels from the evaluation set, and backbone selection also uses the evaluation data. The domain-shift experiment fixes the test set and varies only the supervised methods' training sources, which does not establish that zero-shot methods remain stable when the test distribution changes.
Questions for authors

Most of my concerns and questions are written in the weaknesses. If the authors can address most of them in the rebuttal responses, I will consider raise my overall score.

3 comments
AuthorsOpenReview ↗

Response to Reviewer WpPs

Thank you for the concrete requests. We are glad you found the approach "simple and practical", the analysis of ties near the threshold a "useful insight into the performance gains", and Manugram "a useful resource". We ran every analysis you asked for, and each one supports the paper's claims: keeping the residual mass is what orders the most machine-like texts (W1), the gain comes from the readout when the BaseJudge call is matched (W2), and thresholds fixed in advance keep false alarms near the target, with only LogJudge and FusionJudge staying below it under strong domain shift (W3).

W1. Is counting the mass outside the two labels as 0.5 justified, and how do label absence, the residual near the threshold, k and its treatment matter?

The value 0.5 is not critical, but keeping the residual is: counting it as human gives the same TPR@1%FPR, while dropping it ties the most machine-like texts. The reason is label absence: near the threshold, the gpt-5.6-luna API returns only "machine" for every text, without "human". There, 1 − p(machine), the judge's remaining doubt, is what orders the texts, and the residual is small (median at most 0.0012 per split). Four-split mean TPR@1%FPR:

Treatment of the residual rluna, k = 1luna, fullgemma, k = 1gemma, k = 20
Neutral, p(human) + r/2 (LogJudge)0.6480.6480.5090.512
Human, 1 − p(machine)0.6480.6480.5090.512
Renormalized0.0000.0000.0000.509
Machine, p(human)0.0000.0000.0000.509

The gpt-5.6-luna API returns one or two candidates, which is enough. gemma-4-31b returns all 20, so there the treatment matters only at k = 1.

Changes and details: Sec. 5 ("Residual mass"), new App. "LogJudge: Window Size and Residual Mass".

W2. BaseJudge writes two decimals under other instructions, so precision or prompt format may explain the gain.

Thank you, this point led us to compare the two calls in detail. With the call fully matched, the gain comes from the readout. The details and the table are in the general comment (G1). In short: called exactly like LogJudge, BaseJudge reaches a four-split TPR@1%FPR of only 0.058 at two decimals, and 0.102 and 0.126 with four decimals or no limit, against 0.648 for LogJudge. With reasoning and four decimals, BaseJudge reaches LogJudge's level (0.651 against 0.648), but only by reasoning on 63% of the calls, at higher test-time compute. On your two specific requests: the written answer takes 91 distinct values, LogJudge 6,529, and in the matched call the extra digits repeat a few values, so the tie group at the threshold grows to about 150 human texts. With two additional judge prompts, the order BaseJudge < LogJudge < FusionJudge stays the same.

Changes and details: Sec. 3 and Sec. 5 ("Same test-time compute") state the call settings. New App. "Test-Time Compute", "Same Judge, Two Readouts" and "Other Judge Prompts".

W3. Oracle thresholds and backbone selection use the evaluation data, and the domain-shift test keeps the zero-shot detectors' test data fixed.

You are right that a deployed detector must fix its threshold in advance. We used the oracle threshold because it is the standard for comparing detectors (Hans et al., 2024), and we added a check for each of the three points you raise. The details and the full table are in the general comment (G2). In short:

  • Thresholds set on held-out human texts give a mean realized FPR of at most 1.1% and change TPR by at most 0.011 on the three large splits.
  • Backbones chosen without the evaluated split match ours in 14 of 16 cases, and LogJudge and FusionJudge are unchanged.
  • Under domain shift, every detector, zero-shot included, now sets its threshold on the scenario's training data, so the zero-shot detectors also face a changed test distribution. Under Mid- and Far-OOD, LogJudge and FusionJudge are the only detectors that keep false alarms below 1%, while the trained detector flags up to 17% of the human texts, and they still catch 55 to 64% of the machine texts.

This supports the paper's claim directly: the zero-shot judges keep their false alarms under control when the genre changes, which a trained detector does not.

Changes and details: Sec. 5 ("Calibrated thresholds" and "Zero-shot versus trained under domain shift"), App. "Calibrated Thresholds" (new domain-shift figure) and "Baseline Backbone Selection".

We hope these results address your concerns. If any point is still open, we would be glad to provide further analyses during the discussion.

Thank you for the detailed response and additional experiments. The authors have addressed most of my concerns regarding residual-mass treatment, output precision and prompt matching, and so on.

I find the response sufficiently convincing and will therefore raise my score to 3.

Thank you for considering our response and for your positive reassessment.

Reviewer_Lma8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines how to read an LLM judge's answer when using LLM-as-judge for AI-text detection. The authors argue that asking the judge to state a probability or label directly is too coarse and produces many ties at a 1% false-alarm rate. They therefore propose LogJudge, which reads the top-k log-probabilities of “human/machine” at a single answer token and computes the expected human-likeness; FusionJudge then averages this score with the stated probability. The authors also build Manugram, covering current generators, scientific abstracts, AI-Scientist abstracts, and rephrased corpora, and validate the methods on four splits and on ICLR abstracts. The main result is that, with the same judge, the log-probability readout leaves AUROC largely unchanged but substantially improves recall at low FPR; FusionJudge is strongest on average and requires no training.

Critical strengths

1.1. The problem is well framed, rather than attributing the gain to a stronger judge, the authors identify a resolution bottleneck in stated probabilities near the 1% FPR and use tied values and ROC operating points to explain why LogJudge works. This observation has practical relevance for LLM-as-judge detection. 2. 3.2. Manugram doesn’t just reuse Ghostbuster, it regenerates the machine text, adds scientific abstracts and rephrased corpora, and runs domain-shift, rephrasing, and self-recognition checks. Compared with reporting AUROC only on the old Ghostbuster, this validation is broader and more credible.

Critical weaknesses

1.The methodological contribution is more about how to derive a detection score from the model’s output than about proposing a new detection mechanism. AUROC remains almost unchanged, the gains are concentrated at low-FPR thresholds, and FusionJudge is merely an equal-weight average, lacking calibration or a selection rule. 2.MG-AI-Sci contains only 100 human texts, so its 1% threshold is effectively determined by a single document; the AT-HM differences on this small sample should therefore not be overinterpreted. In addition, the rephrasing attack covers only one generator and a limited corpus, so the conclusions have limited generalizability.

Questions for authors
  1. Is FusionJudge’s equal-weight average pre-specified, or was it selected on a validation set? Has it been compared with a simple rule that switches between LogJudge and FusionJudge according to the FPR range?
  2. Can the 1% threshold and small-sample AT-HM for MG-AI-Sci be accompanied by bootstrap confidence intervals or leave-one-out threshold-stability results?
  3. Given the log-probability jitter in Appendix V, is the AT-HM gap between FusionJudge and LogJudge robust under repeated calls or resampling?
  4. How can the ICLR abstract flag rate be presented to readers so that it is not mistaken for the share of LLM-written abstracts? Under the label-free setting, can a more intuitive calibration explanation or a sensitivity/upper-and-lower-bound analysis be provided?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Lma8

Thank you for the positive assessment and the precise questions. We are glad you see the resolution bottleneck near 1% FPR as practically relevant and the Manugram validation as "broader and more credible".

W1. The contribution is a way to derive a score rather than a new detection mechanism, AUROC barely changes, and FusionJudge is a plain average without calibration or a selection rule.

We agree that the method is simple, and we see this as a strength, as WpPs also notes ("Simple and practical approach"). It needs no training, only the few log-probabilities commercial APIs return, and no parameter to tune. A deployed detector must rarely flag a human author, so its recall at a small false-alarm rate decides its value. The written answer takes only 91 distinct values (6,529 for LogJudge), so many texts tie at the 1% threshold. AUROC hardly registers these ties, but recall does. With the same judge and evidence prompt, and with BaseJudge allowed to reason first, the readout raises the four-split TPR@1%FPR by 0.145 for gpt-5.6-luna and by 0.128 for gemma-4-31b, while AUROC changes by at most 0.033 on any split. On calibration, the equal weight needs no fitting (Q1), thresholds set on held-out data give the same results, and to make sure the judge scores are not read as calibrated probabilities, the paper now states explicitly that they are detection scores.

Changes and details: Sec. 2 ("Confidence readouts in LLM-as-judge") states what is new compared with earlier token-probability readouts. Sec. 3 (detection scores), App. "Calibrated Thresholds".

W2. MG-AI-Sci has only 100 human texts, and the rephrasing attack uses one generator and a limited corpus.

We agree that MG-AI-Sci is small and that differences on it alone should not be over-interpreted (Q2). We include it because AI-Scientist abstracts are a new kind of machine text that none of the benchmarks we compare covers, and our conclusions do not depend on it: without MG-AI-Sci, LogJudge still leads Fast-DetectGPT and Binoculars (AT-HM 0.792 against 0.712 and 0.714). The rephrasing corpora contain about 24,000 texts from news, essays, fiction and scientific abstracts. All rephrasing is done by gpt-5.6-luna, the hardest generator for both judges, so the test is a demanding case. We agree that more rephrasing models would strengthen it.

Changes and details: Sec. 5 ("Split by split", and the limitations now name the single rephrasing model), App. "Detailed General Results".

Q1. Was the equal weight fixed in advance, and does a rule that switches between LogJudge and FusionJudge by FPR range help?

Yes, it was fixed in advance, without a validation set. This is answered in the general comment (G3). On the switching rule: it uses LogJudge only where LogJudge leads, which on the four-split mean is below about 0.3% FPR, so at 1% FPR the rule equals FusionJudge.

Changes and details: App. "Fusion Weight and Combination Rule".

Q2. Can the MG-AI-Sci results get bootstrap intervals or a leave-one-out check of the threshold?

Yes, both are added, and our conclusions hold: on the four-split mean, the gains of FusionJudge over every detector and of LogJudge over BaseJudge hold under a bootstrap that refits the threshold on every resample, for example FusionJudge − LogJudge +0.081 AT-HM (95% interval 0.032 to 0.125). This is answered in the general comment (G4), with the full table. For MG-AI-Sci alone, the checks confirm your point: with 100 human texts its 1% threshold rests on a single text, and leave-one-out moves the TPR@1%FPR of LogJudge and FusionJudge from 0.56 to at most 0.72. This is why the paper bases its conclusions on the four-split mean, while MG-AI-Sci adds AI-Scientist text that no other benchmark we compare covers. As you note, differences on this split alone should not be over-interpreted.

Changes and details: Sec. 5 ("Metrics"), App. "Bootstrap Confidence Intervals".

Q3. Does the gap between FusionJudge and LogJudge hold under repeated calls and resampling?

Yes, under both. This is answered in the general comment (G4). In addition, across the four captures the gap is 0.082 ± 0.008 AT-HM, and at least 0.073 in every capture.

Changes and details: Sec. 5 ("Stable across repeated calls"), App. "Logprob Capture Reproducibility" (Appendix V in the submission, now Appendix Y).

Q4. How can the ICLR flag rate be kept from being read as the share of LLM-written abstracts?

The paper now states that a flag rate is not a share and adds the sensitivity analysis you suggest. If at most about 1% of human abstracts are falsely flagged, as in 2020, the 2025 flag rate of 11.1% corresponds to roughly 9 to 19% machine-like abstracts, depending on the recall.

Changes and details: Sec. 5 ("In the wild: ICLR abstracts"), ICLR appendix ("From flag rate to share").

We are happy to clarify any remaining point.

AuthorsOpenReview ↗

Follow-up before the end of the discussion period

Thank you again for your positive and helpful review. As the discussion period ends soon, we wanted to check whether our response answers your questions. The revised PDF includes the analyses you suggested. In particular, performance holds for any fusion weight from 0.05 to 0.95 (G3), the gap between FusionJudge and LogJudge holds in every repeated capture (Q3 of our reply), and the new ICLR analysis separates the flag rate from the share of machine-like abstracts (Q4 of our reply). We believe these additions make the paper stronger, and we thank you for suggesting them. We are happy to clarify anything further in the time left.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response: main concerns and revisions

We thank the reviewers for their careful and constructive reviews. They found the focus on low false-positive rates practically relevant (WpPs, xghj), the analysis of ties near the threshold insightful (WpPs) and "well framed" (Lma8), the approach "simple and practical" (WpPs), and Manugram "a useful resource" (WpPs) whose validation is "broader and more credible" (Lma8). We ran every requested analysis and uploaded a revised PDF. All code and data are released. The main concerns are answered here, the others in each reply.

G1. Is the gain due to the readout or to the BaseJudge call? (WpPs, xghj)

After submission, we noticed that the two calls use different settings. BaseJudge used the endpoint's default settings, under which gpt-5.6-luna reasons on about half of the calls. LogJudge uses reasoning effort "none", which the API requires for log-probabilities. This gave BaseJudge more test-time compute, so the paper's comparison favored the baseline. We reran BaseJudge with LogJudge's call settings and also read the digit tokens of that call (gpt-5.6-luna, four-split mean):

Detector and callReasoningTPR@1%FPR
BaseJudge, matched call, 2 / 4 decimals / no limitno0.058 / 0.102 / 0.126
Digit-token probabilities of the same matched callno0.509
BaseJudge (paper, default settings)about half0.503
LogJudge (paper, ours)no0.648
FusionJudge (paper, ours)0.761

With reasoning and four decimals or no limit, BaseJudge reaches LogJudge's level, at higher test-time compute. Two additional judge prompts keep the order BaseJudge < LogJudge < FusionJudge.

Changes and details: Sec. 3, Sec. 5 ("Same test-time compute"), App. "Test-Time Compute", "Same Judge, Two Readouts" and "Other Judge Prompts".

G2. Thresholds, backbones and domain shift (WpPs, xghj)

Post hoc TPR@FPR thresholds are standard for comparing detectors (Hans et al., 2024), but a deployed detector must fix its threshold in advance, so we added the checks the reviewers asked for. Thresholds set on held-out human texts give a mean realized FPR of at most 1.1% and change TPR by at most 0.011 on the three large splits. Backbones chosen without the evaluated split match ours in 14 of 16 cases, and our detectors are unchanged. Under domain shift, every detector, zero-shot included, now sets its threshold on the human texts of the scenario's training data. Under Mid- and Far-OOD, only LogJudge and FusionJudge keep false alarms below 1% (Fast-DetectGPT 1.1 to 1.2%), while the trained detector flags up to 17% of the human texts (realized FPR / TPR):

DetectorIn-DomainNear-OODMid-OODFar-OOD
FusionJudge1.3% / 0.9162.1% / 0.9070.1% / 0.5550.2% / 0.557
LogJudge1.0% / 0.7092.0% / 0.7440.9% / 0.6410.6% / 0.576
Fast-DetectGPT1.2% / 0.6071.2% / 0.6161.2% / 0.6151.1% / 0.605
Binoculars1.5% / 0.5321.4% / 0.5257.0% / 0.7147.2% / 0.727
RoBERTa-Sentinel (trained)1.9% / 0.9731.0% / 0.67016.7% / 0.5666.5% / 0.215

Changes and details: Sec. 5 ("Calibrated thresholds" and "Zero-shot versus trained under domain shift"), App. "Calibrated Thresholds" and "Baseline Backbone Selection".

G3. The equal weight of FusionJudge (Lma8, xghj)

We fixed the plain average in advance as the simplest choice, without a validation set, and tested it afterwards. Any weight from 0.05 to 0.95 gives a four-split AT-HM of 0.838 to 0.849 (0.848 at equal weight). Three other combination rules give 0.829 to 0.832.

Changes and details: Sec. 3 ("FusionJudge"), App. "Fusion Weight and Combination Rule".

G4. Uncertainty and repeated calls (Lma8, xghj)

In a bootstrap that refits the threshold on every resample, these gains hold:

Four-split meanAT-HM difference95% interval
FusionJudge − LogJudge+0.0810.032 to 0.125
FusionJudge − BaseJudge+0.1950.137 to 0.236
FusionJudge − Fast-DetectGPT+0.2120.076 to 0.258
FusionJudge − Binoculars+0.2200.100 to 0.281
LogJudge − BaseJudge+0.1140.031 to 0.174

We also captured the 2,337 texts near the 1%-FPR threshold three more times. The order BaseJudge < LogJudge < FusionJudge holds in every capture, and for LogJudge only 1 of 400 random control texts changed sides at the threshold.

Changes and details: Sec. 5 ("Metrics" and "Stable across repeated calls"), App. "Bootstrap Confidence Intervals" and "Logprob Capture Reproducibility".

Other changes: The paper now states explicitly that the judge scores are detection scores (xghj), and the related work states what is new compared with earlier token-probability readouts (xghj). New analyses cover the residual probability mass (WpPs), label order and wording (xghj), and the ICLR flag rate (Lma8). The limitations now name the single rephrasing model (Lma8).

The main claims and results of the paper are unchanged.

AuthorsOpenReview ↗

Summary of the discussion

We thank the reviewers and the AC, and briefly summarize the discussion.

The reviewers found the focus on low false-positive rates practically relevant (WpPs, xghj) and the resolution problem "well framed" (Lma8). The analysis of ties near the threshold explains the gains (WpPs), and the tie-breaking ablation gives evidence on the mechanism (xghj). WpPs called the approach "simple and practical", since it needs no training and works with the log-probabilities of commercial APIs. Manugram is "a useful resource" (WpPs), with a validation "broader and more credible" (Lma8).

The revision added the analyses the reviewers asked for. We reran BaseJudge with LogJudge's call settings, and LogJudge's lead grows, also when BaseJudge writes probabilities with more decimals under these settings (G1). Thresholds calibrated on held-out human texts keep the realized FPR near 1%, and under the larger domain shifts only LogJudge and FusionJudge stay below 1% (G2). The equal fusion weight was fixed in advance, and any weight from 0.05 to 0.95 gives a similar AT-HM (G3). In bootstrap intervals and three repeated captures, the order BaseJudge < LogJudge < FusionJudge holds (G4). New analyses also cover residual mass, label order and wording, the ICLR flag rate, and why FusionJudge helps (Sec. 5).

After the revision, two of the three reviewers recommend acceptance. WpPs found the response convincing and raised the score from 2 to 3. Lma8 rated the paper 3, and our reply addresses each of their questions. xghj raised Soundness and Informativeness, found the experiments "extensive, with a complete dataset", and now rates the remaining soundness concerns as minor or fixable. xghj's recommendation is based on novelty and on the mechanism of FusionJudge, and our last reply answers both.

On novelty, G-Eval and P(True) read answer-token probabilities outside detection, and Sec. 2 cites them. The paper shows what the coarse written answer costs at a strict false-alarm budget. With the same judge, the readout changes AUROC by at most 0.033 on any split but raises TPR@1%FPR by 0.145 (gpt-5.6-luna) and 0.128 (gemma-4-31b).

On the mechanism, Sec. 5 and Figure 5 show when averaging helps. Breaking the written answer's ties with LogJudge alone lifts AT-HM from 0.653 to 0.829, and this step does not need the two scores to share a scale. Moving texts where LogJudge is confident gives the rest, up to 0.848. Below about 0.3% FPR, LogJudge alone is ahead. The paper treats the scores as detection scores, and these results do not rely on calibrated probabilities.

The main claims and results of the paper are unchanged. At 1% FPR, averaged over the four benchmarks, the stated probability (BaseJudge) catches 50% of machine-written texts, LogJudge 65% and FusionJudge 76%, against 52% for Fast-DetectGPT and 51% for Binoculars. All code and data are released.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.