- Rating
- 1(−1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper leverages existing response data to investigate which items can effectively differentiate models that have similar total scores. The authors discover that the impact of individual items exhibits a reproducible pattern after controlling for accuracy and item discrimination. Furthermore, selecting items tailored to a specific group proves more effective at preserving the relative rankings of held-out items
Critical strengths
-
The experimental design is solid, considering the multiple control experiments.
-
This research uses performance metrics outside the target benchmark, reducing the circular justification.
-
The author clearly acknowledges the generalization scope and statistical control in the limitations section.
Critical weaknesses
-
It is still unclear whether the observed effect can already be explained by a standard IRT model.
-
The differences between the ability groups may partly come from differences in model families rather than ability itself.
-
It is not clear whether the new metric is really necessary, since simpler baselines such as within-group random item selection are not included in the comparison.
No proper rebuttal during the discussion phase
Questions for authors
- Do simulated responses from a fitted IRT model show similar residual fingerprints?
- Do the results hold when the selection and test sets contain different model families?
- How much does the method improve over random selection within the same group, selection by discrimination, and selection using the metric without residualization?