- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes a supervised framework (EGRD) for assessing research idea novelty against prior work. The main idea is to explictly model how much of a target idea is covered by prior work and what remains novel, including potentially novel combinations of existing components. The authors also construct an LLM-annotated dataset (IdeaDelta) for training and evaluation. Experiments show substantial improvement over the evaluated baselines on IdeaDelta, with limited agreement with human judgements on RINoBench.
Critical strengths
-
I like the overall problem formulation: explicitly modeling prior-work coverage and residual novelty rather than relying on a holistic LLM judgement.
-
The authors conducted a fairly extensive set of experiments, including comparisons with multiple baselines, ablation studies, and transfer evaluation on a human-annotated benchmark.
Critical weaknesses
-
My biggest concern is whether EGRD is actually needed given how IdeaDelta is constructed. The IdeaDelta novelty labels are generated by an LLM estimating several intermediate variables, which are then combined using a manually designed scoring function. This annotation pipeline is already a complete novelty assessment method. I'm not entirely sure why we need to train another, much more complex model to reproduce its judgements. A natural baseline would be to directly apply the IdeaDelta annotation pipeline to RINoBench and compare it with EGRD. If EGRD performs worse, it could potentially be motivated as a more efficient distillation model, but the paper would need to establish that narrative with more experiments.
-
Relatedly, I'm concerned about the evaluation setup. EGRD's architecture and auxiliary training signals are closely aligned with the variables used to construct IdeaDelta's gold labels. In some sense, the model is designed around the mechanism that generated its evaluation labels. This makes the comparison with prompted LLM baselines somewhat unfair.
-
I'm also not convinced that the residual decomposition actually produces meaningful semantic units. The units are learned latent representations without unit-level supervision, and the support matrix is also learned from aggregated coverage signals. The ablations show that these components help downstream prediction, but they don't really demonstrate whether the learned units correspond to meaningful, human-interpretable research contributions, or whether the predicted alignments with prior work are correct.
-
The interaction residual seems to be a coarse approximation of compositional novelty. The method uses within-prior co-support to determine whether a combination has been covered. However, a prior supporting both components A and B doesn't necessarily mean it covers the specific relationship or mechanism between them in the target idea. I'm not sure the current formulation can distinguish an existing combination from a genuinely novel way of combining the same components.
Questions for authors
Could the authors provide some examples of the learned semantic units and their alignment with prior work? Is there any human evaluation showing that the decomposition captures meaningful research contributions?