- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper argues that a simple text-based TF-IDF + ridge regression model can predict reviewers’ average scores, and that giving the model access to more manuscript content substantially improves its predictive performance. It also finds that one reviewer’s score still helps predict another’s after controlling for the manuscript-based prediction.
Critical strengths
The experimental design is simple and clear, and the paper is easy to follow. The presentation-control analyses and fixed-budget reading experiments offer useful evidence that access to manuscript content beyond the title and abstract helps explain the observed performance gains.
Critical weaknesses
- The evaluation against stronger neural/LLM-based scorers is quite limited, relying primarily on a single comparator, NAIPv2-8B. Moreover, this comparison does not isolate scoring capability from input scope: the sparse model uses broad manuscript text, whereas NAIPv2-8B receives only titles and abstracts. Since the sparse model is weaker under matched title-and-abstract inputs, the reported advantage mainly demonstrates the benefit of broader manuscript access rather than a stronger scoring method under comparable input conditions.
- Incomplete manuscript provenance leaves potential temporal leakage unresolved. Some inputs may include revisions made after the target scores were assigned, allowing predictions to rely on text influenced by the review process.
Questions for authors
- Could the authors compare against more LLM-based scoring baselines, ideally with matched manuscript access? Since the main advantage seems tied to broader text access, this would clarify whether the gain reflects the scoring method itself or simply the information available.
- How well do these findings generalize beyond ICLR? The current analysis is restricted to a single machine-learning conference, and it is unclear whether the same predictable patterns would hold across other venues or scientific fields with different reviewing criteria and writing conventions.
- To what extent does the “predictable core” reflect topic- or subfield-specific lexical correlations? Since the presentation controls do not explicitly account for topic, could the authors report topic-controlled analyses or cross-subfield generalization results to assess whether the learned signal transfers beyond these correlations?