acceptodds
Under review as a conference paper at ICLR 2027

DeskReject-Copilot: A Dataset, Approach, and LLM Benchmark for Submission Screening

Abstract

The number of submissions to journals and conferences has grown rapidly (e.g., ICLR: 2,213 in 2020 → 19,814 in 2026). This puts the peer review system under increasing strain. While prior work studies LLMs for writing reviews, their use for screening submissions for desk rejection has been identified as promising, but has neither been realized as a system nor evaluated empirically, in part because no labeled data existed. We introduce the first dataset of its kind: 1,104 ICLR desk rejections from 2020–2026, fully double-annotated with 20 rejection reasons. Building on it, we present DeskReject-Copilot, a detection pipeline that covers seven formal rejection reasons and provides a tailored configuration for each check. Instantiating it with ten multimodal LLMs released between May 2024 and August 2026 yields macro-averaged F1 scores between 0.75 and 0.99 in the balanced setting and between 0.71 and 0.97 in the imbalanced setting. The results reveal a marked asymmetry across desk-reject reasons: small open-weight models already achieve near-perfect detection for a subset of reasons, whereas for others, satisfactory performance is achieved only by a single closed-weight, high-cost model, with most current top-tier models falling short. Our findings suggest that DeskReject-Copilot can serve as an effective screening assistant for program chairs, albeit with a deliberate choice of model per desk-rejection reason. We release our dataset and code.

Primary area:
datasets and benchmarks
Keywords:
desk rejection, automated screening, peer review, large language models, benchmark, dataset

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_DG7vOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Authors propose DeskReject-Copilot, a detection pipeline that covers 7 formal rejection reasons and provides a tailored configuration for each check. Using ten multimodal LLMs, macro F1 scores are between 0.75 and 0.99, but the results reveal there is a marked asymmetry across the specific reasons. Authors argue that the proposed method can be served as an effective screening assistant for peer review systems.

Critical strengths
  • Despite there are lots of relevant works for automating paper reviewing, this work studies a relatively under-explored research problem, which is automating desk rejection decisions.
  • Most frontier models show very high accuracy, whereas some categories are relatively hard to detect, which supports a potential for automating the entire process.
Critical weaknesses
  • My biggest concern is that the problem that authors resolve is not interesting to the ML community. Although automating desk rejection decisions is underexplored area of research, as the results in Table 1--3 suggests, most frontier models can do the job easily by simply prompting those. Also, the method authors propose is not interesting as well since it is just a simple prompting. I'm not sure what lessons readers can get after reading this paper. At least I think this paper is not suitable as an ICLR paper.
Questions for authors

None.

2 comments

We thank the reviewer for recognising that automating desk-rejection decisions is an under-explored problem, and that the high accuracy of frontier models points to a real potential for automation. We address the points below.



P1: Quality of Frontier Models

In Table 2, we show that under the given settings and constraints, only Claude Opus 5 reliably detects incorrect templates. All other models remain below the trivial baseline. We therefore only partially agree that current frontier models can already perform this task. But even if all current frontier models could already detect desk rejections, we would still see the need to demonstrate this empirically. Empirical evidence of the capabilities and limitations of AI for paper screening can signal to journals and conferences that they should engage with this topic in more depth. We also see this work as a starting point for further research. Our approach does not yet cover all of the desk-rejection reasons we identified in our dataset, or those that may emerge in the coming years. To what extent models can be applied to the remaining desk-rejection reasons, and how well they perform on them, is still an open question that requires further research.



P2: Methodological contribution

Taking into account the available resources and data, we were able to show that AI can support the screening process, and that this does not necessarily require highly technical approaches. We therefore understand the reviewer’s point in principle. However, we do not necessarily see this as a weakness: we show that very promising results can already be achieved with the right input data and prompts. Furthermore, we believe that our two main contributions should be considered separately:

(1) Dataset: Our dataset is the first of its kind: a labeled dataset of real desk rejections whose initial submissions are freely available online. Researchers can use it for further work, and journals or conferences that want to build their own systems can use it to test them more easily and effectively. The dataset is also AI-agnostic, so deterministic approaches can use it to evaluate their systems as well. With it, future research can focus on methods for detecting desk rejections without having to worry as much about which data to use.

(2) Approach and Benchmark: Our second contribution already provides valuable insights for 7 of the 20 desk-rejection reasons we identified. Within the given constraints, we showed that many models deliver promising results when given the right input data. We also showed that for some reasons model performance is very similar (e.g., front page, prompt injection), while for others it differs considerably (e.g., template, incomplete manuscript). These findings come together in our discussion, where we argue that the future of automated screening lies in combining small open-weight models, closed-weight top-tier models, deterministic rule-based approaches, and humans to achieve the best overall result.

Although the methodological contribution of this work is limited, we make important contributions to datasets, approaches and benchmarks, and to transferring research findings into practice.

P3: ICLR Relevance

The Program Chair of ICML 2027, Andrew Gordon Wilson, asked on LinkedIn for ideas on how to handle the overwhelming number of submissions [1]. In the invitation to Area Chairs (ACs), ICML 2027 then announced plans to provide an AI tool for paper screening [2]. The ML community is therefore already engaging with automated screening from a practitioner’s perspective. If the paper is accepted, we are optimistic that it will spark interesting discussions at ICLR from both a scientific and a practical perspective. Transferring research into practice is often a challenge. Publishing this work, however, could act as a catalyst for bringing automated paper screening from research into practice, where the practice in this case is the scientific publishing system itself. Automated screening matters most for venues that receive a large number of submissions. The ML community, and therefore the ICLR community, is particularly affected, as the number of submissions to ICLR 2027 shows. Because ICLR is so strongly affected, we are convinced that it is the right venue to publish and discuss contributions of this kind. We see this as a great opportunity to put research findings into practice quickly.

[1] Wilson, A. G. (2026). LinkedIn. Retrieved November 10, 2026, from https://www.linkedin.com/feed/update/urn:li:activity:7507821221892407296/

[2] Feuer, B. (2026). LinkedIn. Retrieved November 10, 2026, from https://www.linkedin.com/feed/update/urn:li:activity:7520477927533121537/

Thanks for the response. I'm still not convinced that deciding whether or not to desk-reject a paper is an interesting ML problem. Also, since frontier models show nearly perfect scores, I'm pessimistic of the value of the benchmark -- of course, it still has some values as authors mentioned, but I think it is not a level of ICLR acceptance. This could be my personal taste, but I always expect challenges when new benchmarks came out. I'll keep my score.

Reviewer_PaGBOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problem of predicting desk-rejection decision for scientific papers. The authors collect 1k ICLR desk-rejects from the past few years, and evaluate several models in predicting them, reaching relatively high results.

Critical strengths

The problem is extremely important and timely (especially given the 60k+ submissions in the current ICLR). The data is certainly useful and the results are likely to inform decision makers. While results are not perfect, I don't think they have to be, it probably makes sense to have a human in the loop verifying the outputs any way (this is not to say off course the efforts shouldn't be made to improve them further)

Critical weaknesses
  • As noted by the authors, for some of the DR reasons, calling an LLM might be an overkill (e.g., too many pages, wrong template or even potentially prompt injection could be identified using traditional coding much cheaply). This is especially clear for the wrong template, where most LLMs seem to be very bad anyway, and the most expensive ones are needed. While I realize the authors noted this, this is a clear limitation that could have been solved with not that much effort (using coding-agents), which has a huge effect on both price and quality
  • While the current results are already of value, it seems they cover <40% of the cases, and particularly the invalid reference is the most dominant DR reason. Given the use of LLMs for this task, why did the authors choose to exclude the external categories?
  • While no training is deployed, it would be more rigorous to use some of the data as validation data to tune your hyperparameters/design choices
  • Could it be that models have seen these papers during training and know whether they were DRed or not?
  • What is the model generation study?
Questions for authors
  • The annotation process is somewhat unclear to me. Given that almost all papers had an official reason, what was the role of the human annotators (raters)? Did they rate only the official reason? Verify it against the paper? Other?
  • I am not sure what the outstanding papers are used for
7 comments

We thank the reviewer for describing the problem as extremely important and timely, and for finding the data useful for decision makers. We also share the view that a human in the loop is the right setting for such a system. We address the questions and concerns below.



W1: Necessity of LLMs for simple checks

We understand this point of view well. However, we see a major advantage of the specialized pipeline in its flexibility. As we found in our analysis and observed while creating the dataset, desk rejects can differ considerably, even within a single category. For Wrong Template, for example, the reasons why a submission violates the template are highly varied. The advantage of deterministic approaches is, of course, that they are easier to trace and to verify. Given the high variation within the categories, however, the flexibility that LLMs offer should be taken into account and valued. As we also state in our discussion, we see the potential future of automated screening in a mixture of small open-weight models, closed-weight top-tier models, and deterministic, rule-based approaches, in combination with humans. In this work, we wanted to investigate what LLMs can achieve on their own. Deploying such a system in practice requires weighing several factors against each other, such as cost, runtime, traceability, and data privacy. Depending on how an individual conference or journal weights these factors, corresponding decisions have to be made regarding the architecture and the choice of technologies.


W2: Selection of Desk-Reject Reasons (Invalid Reference)

We also considered this and decided that LLMs are not a suitable method for this category. We see the knowledge cutoff as the main problem for LLMs here, so we do not see much potential in using an LLM natively for this task. An LLM cannot reliably verify references to work published after its training data ends, which makes it a poor fit for checking bibliographies of new submissions. Prior work has already addressed identifying incorrect references in bibliographies, using approaches that do not rely on LLMs at all [1].


[1] Zhenyue Zhao, Yihe Wang, Toby Stuart, Mathijs De Vaan, Paul Ginsparg, and Yian Yin. Llm hallucinations in the wild: Large-scale evidence from non-existent citations, 2026. URL https://arxiv.org/abs/2605.07723.



W3: Validation Split

We would have liked to do exactly that, as it would have been the clearly better approach. However, we faced the challenge that the dataset is very small for some categories. We deliberately decided not to split the already small dataset any further, so that we could use all available data for testing. To validate the prompts, we used synthetic data instead (details can be found in Appendix C). We acknowledge that synthetic data has drawbacks. However, given the limited amount of available data, we chose synthetic data for validation so that the full set of real data would remain available for testing. In our future research section, we note that future work could examine more closely what suitable synthetic data in this domain could look like.

W4: Data Contamination

The papers themselves and the official desk-rejection reasons might be part of the models’ training data. However, we believe that there is most likely no data contamination, for the following reasons:

  1. The labels that the models output, and against which they are evaluated, cannot have been part of the models’ training data. We created the dataset as part of this project and did not publish it online before the test runs. So although the papers and the official desk-rejection reasons may have been in the training data, the labels that actually matter were not. The official desk-rejection reasons also differ, in some cases substantially, from the wording of the labels (see our “Incomplete Manuscript” example in another point of our response).

  2. Prompt injection has only existed as a desk-rejection reason since ICLR 2026. However, some of the models were released in 2024 or early 2025. This data therefore cannot have been in their training data, because it did not yet exist at that time. This does not rule out contamination for all of the data, but it does indicate that the data cannot be entirely compromised by contamination.

  3. We also tested for data contamination after the fact. As reference date for every ICLR edition, we used the release of its reviews on OpenReview, which takes place in November of the preceding year (e.g. 11 November 2025 for ICLR 2026). For six models, the knowledge cutoff (or, if none is stated, the release date) lies before at least one of these reference dates. For GPT-4o (cutoff 10/2023), ICLR 2024 to 2026 are unseen. For GPT-4.1 (06/2024), ICLR 2025 and 2026 are unseen. For Gemini 3.1 Pro (01/2025), GPT-5.4 (08/2025), Qwen2.5-VL-72B (release 02/2025) and Qwen3-VL-235B-Thinking (release 09/2025), ICLR 2026 is unseen. All earlier editions may have been part of the training data. Contamination would lead to higher recall on these possibly seen papers than on the unseen ones. We compared both groups for all categories with at least five cases each. The key results are shown in the following table:


ModelCutoffUnseenTmpl. seenTmpl. unseenTmpl. ΔPages seenPages unseenPages ΔFront seenFront unseenFront ΔIn-text seenIn-text unseenIn-text Δ
GPT-5.408/202520260.420.29−0.130.850.99+0.141.001.000.000.891.00+0.11
Gemini 3.1 Pro01/202520260.520.49−0.031.001.000.001.001.000.001.001.000.00
GPT-4o10/20232024–260.000.05+0.050.800.76−0.041.001.000.00–––
GPT-4.106/20242025–260.120.14+0.020.800.85+0.051.001.000.00–––
Qwen2.5-VL-72B02/2025†20260.210.17−0.040.600.85+0.251.001.000.000.440.38−0.06
Qwen3-VL-235B-Thinking09/2025†20260.120.14+0.020.900.99+0.091.001.000.001.001.000.00

Tmpl. = Template, Front = Front page.


For four models, recall on possibly seen and unseen papers differs by at most 0.05 in every category. Only GPT-5.4 and Qwen2.5-VL-72B show larger differences, in both directions: lower recall on unseen papers for Wrong Template, higher recall for Too Many Pages. No model performs consistently better on papers it may have seen. This suggests that data contamination has at most a minor influence on our results. Four models are not covered, because their cutoffs lie after the ICLR 2026 reviews. We are currently working on adding it to the paper as an appendix.


W5: Model Generation Study

For the GPT models (as a closed-weight model family) and the Qwen models (as an open-weight model family), we selected three different models each. From OpenAI, these are GPT-4o (05/2024), GPT-4.1 (04/2025), and GPT-5.4 (03/2026). From Qwen, these are Qwen2.5-VL-72B-Instruct (02/2025), Qwen3-VL-235B-A22B-Thinking (09/2025), and Qwen3.8-27B (08/2026). An overview of these models is also given in Table 7. The models within each family were deliberately chosen from different release periods. Our assumption was that newer generations should perform better than their predecessors. With this, we wanted to examine two things: how the ability to detect desk rejections has changed across the models of a model family, and whether current frontier models are needed for this task at all or whether older models can be used as well.

Q1: Annotation Process

The official desk-rejection reasons on OpenReview are not standardized. They are free text that each area chair (AC) can write however they like. The reasons are also not always clear about what they mean, so they often need interpretation. The raters added structure and standardization through their labeling. The following five real desk-rejection reasons from OpenReview show this. For each, we give the label our raters assigned:

  1. “Extensive missing figures.” → Incomplete Manuscript
  2. “Incomplete: figures are not displayed (no graphics; just alt-text)” → Incomplete Manuscript
  3. “The submitted PDF is a placeholder and not a valid submission.” → Incomplete Manuscript
  4. “The submitted PDF only contains a title and an abstract.” → Incomplete Manuscript
  5. “not a full paper; violating anonymity requirement.” → Incomplete Manuscript, Breach of Anonymity – [Front Page / In-Text / …]

Although all five reasons received the label “Incomplete Manuscript“, they share no common wording, and the underlying causes differ: in some cases figures are missing, while in others the paper contains only a title and an abstract. Reason 5 also states that the anonymity requirement was violated. We distinguish between several forms of deanonymization, and the phrase “violating anonymity requirement” does not say which one applies. The rater therefore examined the document more closely to assign the right deanonymization label. Through this labeling, each paper received its label or labels without ambiguity, giving a ground truth against which the models can be benchmarked.

Q2: Outstanding Paper


The idea behind giving the outstanding papers to the models is that the models can use them as a visual reference. This way, the models don’t depend only on the written instructions, which we also provide. They can also compare the outstanding paper visually with the paper under review to spot differences. We also ran an ablation study in which the outstanding papers were not provided as a reference. Qwen3-VL-235B-Thinking was no longer available on OpenRouter at the time the ablation study was conducted, so no results are available for this model. The key results are shown in the following table:

ModelP (with ref.)R (with ref.)F1 (with ref.)P (w/o ref.)R (w/o ref.)F1 (w/o ref.)Δ\DeltaF1
GPT-4o1.000.040.071.000.170.28+0.21
GLM-4.6V1.000.130.240.750.270.39+0.15
GPT-4.11.000.150.260.970.180.31+0.05
Qwen2.5-VL-72B0.970.180.311.000.250.41+0.10
GPT-5.41.000.330.500.970.200.33−0.17
Kimi K2.60.930.400.560.840.470.60+0.04
Qwen3.8-27B0.890.490.630.980.410.58−0.05
Gemini 3.1 Pro0.960.500.660.970.430.59−0.07
Claude Opus 50.990.970.980.970.960.96−0.02

The results show a trend: The three strongest models achieve a slightly lower F1 when they do not receive an outstanding paper as a reference. Most of the weaker models, in contrast, improve when the reference paper is omitted. GPT-5.4 is an exception, as its F1 drops from 0.50 to 0.33 without the reference paper. We are currently working on adding the ablation study to the paper as an appendix.

Reviewer_FnDmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigated the performance of desk rejection of papers using LLM models. It built a dataset of 1,104 ICLR desk rejections from 2020-2026, annotated against 20 rejection reasons. It also conducted the DeskReject-Copilot, a pipeline that checks a submission against seven of rejection reasons, with a per-reason input configuration. It evaluated ten multimodal LLMs released between May 2024 and August 2026 in a balanced and an imbalanced setting. It concluded that the auto desk rejection could achieve desired performance if the model is well selected and configured.

Critical strengths

The dataset is a genuine first and is built properly. No public labeled desk-reject corpus existed, and the annotation is well done also. The experiment is well setup to ensure the confidence of the results such as the small sample uncertainty is well handled.

Critical weaknesses

The main concerns come from the experimental study: such as the one category that discriminates between models is the one whose prompt was never developed against the real failure mode. Each model was run once, so no result has a run-to-run error bar. Also, positives and negatives come from different populations, and the design cannot separate the violation from that difference.

Questions for authors

none

2 comments

We thank the reviewer for describing our dataset as a genuine first that is built properly, and for the positive assessment of the annotation and of how we handle small-sample uncertainty. We address the concerns below.

P1: Prompt development (for Wrong Template)

We understand the concern. However, most papers in our dataset that use an incorrect template deviate substantially from the correct one, for example through heavily reduced margins or a template from a different conference. These cases are therefore close to the synthetic data we used to validate the prompt, so the results depend only marginally on subtle violations. This does not mean that detecting small deviations is unimportant, and one model, Claude Opus 5, already manages to do so. Overall, however, we agree with you and believe that future research should examine this point in more detail. 


P2: One run

We agree with this point in principle. Due to cost constraints, we were unfortunately only able to run each model once. This may limit what can be concluded about any individual model. However, because we evaluated 10 different models in 10 independent runs, we believe our results do allow conclusions about whether AI models can, in principle, be used in the screening process. For example, all models achieve 100% accuracy for the “Front Page” reason, and 9 out of 10 models do so for “Prompt Injection”. So although multiple runs could yield more reliable results for individual models, our data already provide valuable insights into the general use of AI in the screening process. We refrain from additional runs for cost reasons, but we are happy to add them if this makes a decisive difference to the reviewer’s opinion. However, the time remaining before the end of the rebuttal period is too short to run the experiments and analyse the results. Accordingly, we would do so for the camera-ready version.

P3: Samples

It is correct that the data come from different pools: the negatives are generally initial submissions, while the positives are accepted papers. However, they differ only slightly, and only in a few cases. We accounted for these differences in our analysis and experiments. For example, in two of the years, authors were given an additional page after the initial submission. This affects the “Too Many Pages” category, because we pass the pages around the page limit to the model. We designed the experiment so that, for the positive samples from these years, the pages passed to the model were adjusted accordingly. If you could point us to specific situations you are particularly concerned about, we would be happy to address them in more detail.

Thank you again for your review.

We posted our response on November 14 and have since uploaded a revised version (changes highlighted in blue). We would appreciate your feedback on whether it addresses your concerns, and please let us know if there is anything else we should add or change.

Reviewer_Fjq4OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces DeskReject-Copilot, a dataset of 1,104 desk-rejected ICLR submissions and an LLM-based pipeline for screening submission violations. The authors evaluate ten multimodal LLMs on seven categories using category-specific prompts and inputs. Results vary substantially across models and categories, and the intended use is to flag submissions for further human inspection.

Critical strengths

A practically relevant application.

Submission compliance checks create real administrative work, so this is a useful problem to study. Collecting actual desk-rejection records and organizing the official reasons into a taxonomy could provide a helpful starting resource for future work.

Useful annotation checks and detailed reporting.

I appreciate that two annotators independently categorize the official rejection reasons, with high agreement reported. The complete detection prompts, category-level confusion matrices, and Wilson confidence intervals also make the setup easier to inspect and the results easier to interpret.

A thoughtful discussion of low-prevalence deployment.

Section 7.1 makes an important point: strong results on a sampled evaluation set do not necessarily translate into high alert precision across actual conference submissions. It is good to see this limitation discussed explicitly, and keeping human inspection in the proposed workflow is sensible.

Critical weaknesses

The dataset coverage is narrower and more skewed than the overall size suggests.

Although the dataset spans 2020–2026, it covers only ICLR, and 82.2% of the records come from 2026 (Table 4). The seven evaluated categories contain just 320 positive label instances, including 210 from 2026. They also exclude the largest category, Invalid Reference, which accounts for 650 labels (Tables 6 and 15).

This makes the headline dataset size a somewhat misleading guide to the scope of the evaluation. I am also concerned that negatives are sampled roughly uniformly across years, while positives follow a very different distribution. Without examining the effect of this mismatch or reporting results by year, it is hard to tell how well the findings hold across changing policies, submission years, or venues.

Several categories offer little model discrimination or too little evidence for reliable conclusions.

All ten models achieve perfect F1 on Front Page, and nine do so on Prompt Injection (Table 2). These categories therefore contribute little to distinguishing the models. At the same time, Prompt Injection has only four positives, and Non-Anonymised Link has only six. For Prompt Injection, even perfect recall comes with a 95% confidence interval of [0.51, 1.00] (Table 33).

Reporting uncertainty is helpful, but it does not resolve the lack of positive examples. Front Page and Prompt Injection also account for two-sevenths of the macro-average weight despite their near-ceiling results. The micro metrics provide another perspective, but I would still like to see larger samples and more challenging cases before treating these results as evidence of reliable detection.

The labeling criteria and selective verification procedure make the evaluation difficult to interpret.

My main concern is whether the benchmark consistently measures the same target. Positive labels come from official desk-rejection reasons, while the manual review of negatives treats some minor formatting deviations as violations, even when it is unclear whether they would justify desk rejection (Section 3; Appendix E). These criteria need to be reconciled more clearly.

Only negatives flagged by a model were manually inspected. Of the 125 inspected samples, 29 were excluded as violations rather than relabeled and retained (Section 5.2). As a result, the final test set partly depends on model outputs, while possible violations missed by every model remain unchecked. This creates a risk of selective-verification bias, and it is unclear how much the filtering affects model rankings. An independent audit using consistent criteria would make the evaluation much more convincing.

The value of the specialized pipeline over simpler alternatives is not established.

The pipeline mainly consists of seven separate prompts with manually chosen inputs. That can be a reasonable design, but the paper does not show what it gains over simpler approaches. The only explicit predictive baseline is “Always violation”; there are no substantive comparisons with applicable rule-based checks or a generic LLM screening prompt.

This is especially relevant because template and page-limit violations account for 246 of the 320 positives. I would also have liked to see ablations testing the contribution of page images, reference papers, page selection, and separate versus joint prompting. A benchmark paper does not need a new algorithm, but stronger baselines and design validation are needed here to establish the pipeline’s incremental value.

Questions for authors

Q1. What is the intended target: any formal submission violation, or a violation that warrants desk rejection? How are official rejection reasons reconciled with minor formatting deviations found during manual inspection? Would the rankings and conclusions hold if negatives were independently audited and confirmed violations were relabeled and retained?

Q2. How sensitive are the overall results to the year distribution and the near-ceiling categories? Could you provide results by year and category, along with an analysis of whether the main model comparisons hold when saturated categories are excluded?

Q3. What does the specialized pipeline gain over applicable rule-based checks and a generic LLM screening prompt? Could ablations of the key inputs and prompting choices establish which parts of the design contribute meaningfully?

3 comments

We thank the reviewer for recognising the practical relevance of the problem and the value of our dataset and taxonomy as a resource for future work. We also appreciate the positive remarks on the double annotation, the detailed reporting and our discussion of low-prevalence deployment. We address the questions and concerns below.


Q1 + W3: Narrow and skewed dataset

Our system is a research prototype that has to be adapted to the requirements of each conference or journal. In this work, however, we applied a very strict standard, under which even minor deviations from the guidelines lead to a desk reject. We did this so that models that detected minor violations would not be scored lower than models that missed them. As described in Appendix E, template violations and their severity in particular have to be assessed case by case, so minor template violations may not automatically justify a desk reject. For the evaluation in this work, we nevertheless keep the rule that even minor violations of the official guidelines justify a desk reject. As requested, we re-evaluated the dataset with both the original and the relabeled data. The results are shown in the following table.
 The ranking is shown in brackets after each entry.

ModelBal. – No reviewBal. – ExcludedBal. – RelabeledImb. – No reviewImb. – ExcludedImb. – Relabeled
GPT-5.40.84 (5)0.85 (5)0.85 (5)0.75 (8)0.78 (8)0.79 (7)
Gemini 3.1 Pro0.90 (3)0.91 (3)0.91 (3)0.85 (2)0.89 (2)0.90 (2)
Claude Opus 50.97 (1)0.99 (1)0.98 (1)0.93 (1)0.97 (1)0.97 (1)
GLM-4.6V0.80 (8)0.81 (8)0.81 (8)0.78 (7)0.79 (7)0.77 (8)
Kimi K2.60.88 (4)0.90 (4)0.90 (4)0.84 (3)0.87 (3)0.88 (3)
Qwen3.8-27B0.90 (2)0.92 (2)0.92 (2)0.83 (4)0.86 (4)0.86 (4)
–
GPT-4o0.78 (9)0.78 (9)0.77 (9)0.73 (9)0.74 (9)0.73 (9)
GPT-4.10.82 (6)0.83 (6)0.83 (6)0.80 (5)0.82 (5)0.79 (6)
–
Qwen2.5-VL-72B0.74 (10)0.75 (10)0.74 (10)0.70 (10)0.71 (10)0.70 (10)
Qwen3-VL-235B-Thinking0.81 (7)0.83 (7)0.83 (7)0.79 (6)0.81 (6)0.80 (5)
–
Always violation0.670.680.690.350.360.37

The table shows the F1 scores. The analysis shows that the F1 scores do not exceed a certain level.

Q2 + W1/W2: Saturated and underpowered categories

The 2026 concentration reflects the data source: ICLR 2026 had far more desk rejections with explicit reasons than earlier editions. We do not expect this to bias the results. All models are used zero-shot, so they cannot learn a correlation between year and label. In addition, every prompt is conditioned on the guidelines, page limit and reference paper of the submission’s own year. The changes to the submissions from year to year are also only minor, if there are any at all. We agree that Front Page and Prompt Injection do not discriminate between models and that categories with very few positives allow only indicative conclusions. This is why we report Wilson intervals. Per-model, per-category results are already reported in Table 2 and Appendix F, so comparisons without the saturated categories can be read directly from them.

Q3 + W4: Specialized Pipeline and Ablations

We see a major advantage of the specialized pipeline in its flexibility. Our analysis and the process of building the dataset showed that desk rejects can vary considerably, even within a single category. For Wrong Template, for example, submissions violate the template in many different ways. Deterministic approaches have the clear advantage of being easier to trace and verify. Given the high variation within categories, however, the flexibility that LLMs offer should also be taken into account. As stated in our discussion, we see the future of automated screening in a combination of small open-weight models, top-tier closed-weight models and deterministic, rule-based approaches, together with human oversight. In this work, we set out to investigate what LLMs can achieve on their own. Deploying such a system in practice means weighing several factors against each other, including cost, runtime, traceability and data privacy. How these choices about architecture and technology are made depends on how a given conference or journal weights these factors. We revisited each of the seven categories to identify where an ablation would be meaningful and could provide further insight. We concluded that an ablation is only meaningful for Wrong Template, where we removed the reference paper (the outstanding paper) from the input. Qwen3-VL-235B-Thinking was no longer available on OpenRouter when we ran the ablation, so no results are available for this model. The key results are shown in the following table:

ModelP (with ref.)R (with ref.)F1 (with ref.)P (w/o ref.)R (w/o ref.)F1 (w/o ref.)Δ\DeltaF1
GPT-4o1.000.040.071.000.170.28+0.21
GLM-4.6V1.000.130.240.750.270.39+0.15
GPT-4.11.000.150.260.970.180.31+0.05
Qwen2.5-VL-72B0.970.180.311.000.250.41+0.10
GPT-5.41.000.330.500.970.200.33−0.17
Kimi K2.60.930.400.560.840.470.60+0.04
Qwen3.8-27B0.890.490.630.980.410.58−0.05
Gemini 3.1 Pro0.960.500.660.970.430.59−0.07
Claude Opus 50.990.970.980.970.960.96−0.02

The results show a clear trend. The three strongest models achieve a slightly lower F1 without the outstanding paper as a reference, while most of the weaker models improve when it is omitted. GPT-5.4 is an exception: its F1 drops from 0.50 to 0.33 without the reference paper. We are currently adding the ablation study to the paper as an appendix. In your review, you also suggest a similar check for Too Many Pages. We considered this as well, but do not see a meaningful way to design an ablation here. Without the images and text of page n and the overflow page n + 1, the model has no basis for deciding whether the page limit was respected. If you have further ideas on how this could be done, we would be grateful for them and would be happy to provide the corresponding experiments.

Thanks for the detailed response and additional experiments. The relabeling analysis, contamination analysis, and template ablation are helpful. However, the dataset imbalance and underpowered categories remain concerns. The selective verification issue is also not fully resolved, and the comparison with simpler baselines is still missing. I'll raise my score to 2, but I don't think the current evidence supports a higher rating. I leave the final judgment to the AC.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal Phase

We sincerely thank the Area Chair and all reviewers for their time and constructive feedback, which has allowed us to further improve the paper. All changes in the paper are highlighted in blue.

Strengths noted by the reviewers

  • Important, timely and under-explored problem (Fjq4, PaGB, DG7v)
  • First labeled desk-rejection dataset, built properly, with high annotation quality (Fjq4, FnDm, PaGB)
  • Transparent reporting (Fjq4) and careful handling of small-sample uncertainty (FnDm)
  • Thoughtful discussion of deployment with a human in the loop (Fjq4, PaGB)

Concerns addressed

  • Selective verification (Fjq4): We re-evaluated all models without manual review, with confirmed violations excluded, and with confirmed violations relabeled. Rankings remain essentially unchanged.
  • Reference paper (PaGB, Fjq4): We clarified its role as a visual formatting reference and added an ablation. It improves the strongest models and helps detect subtle deviations.
  • Risk of data contamination (PaGB): We added an analysis based on each model's knowledge cutoff. It shows no consistent advantage on possibly seen papers. Likely reason: even if the source papers have been seen in model training, our dataset cannot have been seen, since we newly constructed it.
  • Clarifications + Discussion (PaGB, FnDm): We clarified the annotation process, defined the model generation study, extended the discussion of rule-based vs. LLM-based screening, and explained how differences between sample pools are handled.

Reviewer Fjq4 acknowledged these additions and raised their score.

Remaining points

  • Fit to ICLR (DG7v): Reviewer DG7v remains concerned that the benchmark is too close to saturation for frontier models and that the approach offers limited methodological novelty for an ICLR paper. Respectfully, we hold a different view, covering both research and practice. On the research side, performance varies considerably across categories: only one model reliably detects incorrect templates, while all others stay below the trivial baseline. On the practical side, ML conferences are also among the venues most affected by rising submission numbers, and ICML 2027 has announced plans to use an AI screening tool. The community is already engaging with this problem in practice, and can benefit from the insights of our paper.
  • Further experiments (Fjq4, PaGB, FnDm): Some possible comparisons with alternative approaches remain open, including with rule-based, repeated runs, and coverage of further categories such as Invalid Reference. While we are highly interested in pursuing these questions in future work, they are out of scope for the present paper, which covers a new dataset, an approach, and a broad evaluation already.

Once more, we want to thank all reviewers for their valuable input.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.