- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Authors propose DeskReject-Copilot, a detection pipeline that covers 7 formal rejection reasons and provides a tailored configuration for each check. Using ten multimodal LLMs, macro F1 scores are between 0.75 and 0.99, but the results reveal there is a marked asymmetry across the specific reasons. Authors argue that the proposed method can be served as an effective screening assistant for peer review systems.
Critical strengths
- Despite there are lots of relevant works for automating paper reviewing, this work studies a relatively under-explored research problem, which is automating desk rejection decisions.
- Most frontier models show very high accuracy, whereas some categories are relatively hard to detect, which supports a potential for automating the entire process.
Critical weaknesses
- My biggest concern is that the problem that authors resolve is not interesting to the ML community. Although automating desk rejection decisions is underexplored area of research, as the results in Table 1--3 suggests, most frontier models can do the job easily by simply prompting those. Also, the method authors propose is not interesting as well since it is just a simple prompting. I'm not sure what lessons readers can get after reading this paper. At least I think this paper is not suitable as an ICLR paper.
Questions for authors
None.
2 comments
Thanks for the response. I'm still not convinced that deciding whether or not to desk-reject a paper is an interesting ML problem. Also, since frontier models show nearly perfect scores, I'm pessimistic of the value of the benchmark -- of course, it still has some values as authors mentioned, but I think it is not a level of ICLR acceptance. This could be my personal taste, but I always expect challenges when new benchmarks came out. I'll keep my score.
We thank the reviewer for recognising that automating desk-rejection decisions is an under-explored problem, and that the high accuracy of frontier models points to a real potential for automation. We address the points below.
P1: Quality of Frontier Models
In Table 2, we show that under the given settings and constraints, only Claude Opus 5 reliably detects incorrect templates. All other models remain below the trivial baseline. We therefore only partially agree that current frontier models can already perform this task. But even if all current frontier models could already detect desk rejections, we would still see the need to demonstrate this empirically. Empirical evidence of the capabilities and limitations of AI for paper screening can signal to journals and conferences that they should engage with this topic in more depth. We also see this work as a starting point for further research. Our approach does not yet cover all of the desk-rejection reasons we identified in our dataset, or those that may emerge in the coming years. To what extent models can be applied to the remaining desk-rejection reasons, and how well they perform on them, is still an open question that requires further research.
P2: Methodological contribution
Taking into account the available resources and data, we were able to show that AI can support the screening process, and that this does not necessarily require highly technical approaches. We therefore understand the reviewer’s point in principle. However, we do not necessarily see this as a weakness: we show that very promising results can already be achieved with the right input data and prompts. Furthermore, we believe that our two main contributions should be considered separately:
(1) Dataset: Our dataset is the first of its kind: a labeled dataset of real desk rejections whose initial submissions are freely available online. Researchers can use it for further work, and journals or conferences that want to build their own systems can use it to test them more easily and effectively. The dataset is also AI-agnostic, so deterministic approaches can use it to evaluate their systems as well. With it, future research can focus on methods for detecting desk rejections without having to worry as much about which data to use.
(2) Approach and Benchmark: Our second contribution already provides valuable insights for 7 of the 20 desk-rejection reasons we identified. Within the given constraints, we showed that many models deliver promising results when given the right input data. We also showed that for some reasons model performance is very similar (e.g., front page, prompt injection), while for others it differs considerably (e.g., template, incomplete manuscript). These findings come together in our discussion, where we argue that the future of automated screening lies in combining small open-weight models, closed-weight top-tier models, deterministic rule-based approaches, and humans to achieve the best overall result.
Although the methodological contribution of this work is limited, we make important contributions to datasets, approaches and benchmarks, and to transferring research findings into practice.
P3: ICLR Relevance
The Program Chair of ICML 2027, Andrew Gordon Wilson, asked on LinkedIn for ideas on how to handle the overwhelming number of submissions [1]. In the invitation to Area Chairs (ACs), ICML 2027 then announced plans to provide an AI tool for paper screening [2]. The ML community is therefore already engaging with automated screening from a practitioner’s perspective. If the paper is accepted, we are optimistic that it will spark interesting discussions at ICLR from both a scientific and a practical perspective. Transferring research into practice is often a challenge. Publishing this work, however, could act as a catalyst for bringing automated paper screening from research into practice, where the practice in this case is the scientific publishing system itself. Automated screening matters most for venues that receive a large number of submissions. The ML community, and therefore the ICLR community, is particularly affected, as the number of submissions to ICLR 2027 shows. Because ICLR is so strongly affected, we are convinced that it is the right venue to publish and discuss contributions of this kind. We see this as a great opportunity to put research findings into practice quickly.
[1] Wilson, A. G. (2026). LinkedIn. Retrieved November 10, 2026, from https://www.linkedin.com/feed/update/urn:li:activity:7507821221892407296/
[2] Feuer, B. (2026). LinkedIn. Retrieved November 10, 2026, from https://www.linkedin.com/feed/update/urn:li:activity:7520477927533121537/