- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper introduces Kangaroo, a benchmark of 3,886 questions from 140 German mathematics exams (1998–2025), including 1,746 multimodal items and human mean scores for 115 exams. Four vision–language models are evaluated under negative marking with abstention. GPT-5 achieves 88.34% accuracy, but all models show 23–43 percentage-point multimodal gaps. Ensemble and human scores are negatively correlated (), with the association weakening after controlling for visual content. Model-score calibrations do not reliably outperform historical grade means (MAE 4.12 vs. 4.16).
Critical strengths
-
The benchmark combines authentic multimodal exams, contest-faithful scoring, and 23 years of human results, providing a valuable resource for studying evaluation validity.
-
Regression controls, year-cluster bootstrap, and expanding-window evaluation on 75 held-out exams strengthen the longitudinal human–model analysis.
Critical weaknesses
The proposed reference-anchor evaluation may be poorly matched to the capabilities of the tested models.
The models outperform human mean scores by 35–47 percentage points on average (Appendix B), while achieving near-ceiling accuracy on text-only questions (92–99%, Table 1). This raises a concern that their scores may be insufficiently sensitive to difficulty differences relevant to typical students. The substantial multimodal performance gaps further suggest that model scores are strongly influenced by visual-content characteristics, which need not correspond to human difficulty.
The interpretation of reference-anchor failure conflates aggregate human performance with exam difficulty.
Human mean scores depend not only on exam difficulty but also on student ability, preparation, and participant composition, which may change across years. A fixed-ability reference anchor could therefore track changes in exam difficulty without necessarily predicting the means of changing student cohorts. The results support a narrower conclusion about the predictive utility of the tested model scores rather than the validity of fixed-ability anchors in general.
The explanation based on visual composition is not fully established.
The positive association between visual share and human scores decreases from +1.35 to +0.38 after adding year fixed effects, with the latter confidence interval including zero (Appendix Table 12). Thus, the evidence that visually richer exams are easier for students is not robust to year controls.
Questions for authors
-
Can the authors clarify whether the proposed reference anchor is intended to predict aggregate human mean scores or measure exam difficulty independently of changing student cohorts? What evidence supports the broader conclusion about human–model difficulty alignment?
-
What evidence rules out ceiling effects as a major reason for the negative result? Can the authors analyze whether model score variability and human–model correlations change with model ability, using the four evaluated models?
-
Appendix Table 15 reports a recent-three-year baseline with MAE 3.55, outperforming the reported model-score calibrations. Does adding model scores improve upon this stronger baseline, with paired confidence intervals?
2 comments
Thank you for your detail response. The revised manuscript undergone sufficient effort. Your rebuttal resolved many of my concerns. Howerver, construct validity remains limited and the negative result should not be interpreted as proof of zero predictive value. These are meaningful limitations, but they are less serious now that the manuscript acknowledges the restricted scope. I will raise the soundness to 2 and keep my overall assesment as weak rejection.
We thank the reviewer for the detailed reading.
W1 and Q2 (ceiling effects and model ability). We agree that the four models score far above the pupils and approach the ceiling on text-only items. On whole exams, however, their scores are not compressed. Within grade groups, their standard deviation across exams is 6.2 to 8.6 points against 4.3 for the human mean score, only 0.9% to 20.9% of exams reach 95 %max, and the correlations with the human mean score lie between −0.26 and −0.32 although the mean scores range from 76.3 to 88.4 %max. On multimodal items alone, where the models score 58.1 to 77.0 %max, the slopes lie between −0.51 and −0.05 per 10 points and every interval includes zero.
We have also extended the ability range in both directions (table in the general comment). Ministral 3 8B scores 53.5 %max, the closest to the pupils' 41.1, and no exam reaches 95. Its score does not improve the forecast and its slope is negative as for the original models, so the result holds from 53.5 to 88.4 %max without a binding ceiling. We report this run as exploratory because 17.5% of its responses hit the output limit or needed the fallback parser; two other treatments of these responses give forecast differences of +0.05 and +0.12. At the top the ceiling does bind. GPT-6.1 Sol scores 99.4 %max, earns the full score on 83.5% of exams, and with a within-grade standard deviation of 1.5 points says almost nothing about how exams differ. The four newer models lose only 0.3 to 11.0 points on multimodal items, against 23 to 43 for the original models, and their exam scores change by −0.9 to +0.1 points per 10 points of visual share, against −1.7 to −3.9 for the original models and −3.1 for Ministral 3 8B. The negative slope disappears together with this dependence, as C2 implies. We agree that the models' dependence on visual content need not reflect human difficulty, and we will state that only one run tests an anchor matched to pupil ability.
W2 and Q1 (what the anchor test measures). We agree, and we change the paper accordingly. The human mean score depends on both the exam and the pupils who took it, and the official summaries, which give a mean score and a score histogram per exam, cannot separate the two. To answer Q1, the test concerns the prediction of human mean scores and does not measure exam difficulty independently of the cohorts. A fixed-ability anchor that tracks difficulty predicts the human mean score if cohort ability changes slowly compared with exam difficulty, which is plausible for national cohorts of several hundred thousand pupils, and we will state this assumption. Under it we can compare changes within a grade group. Year-to-year changes in the ensemble score and the human mean score correlate at −0.20 [−0.40, 0.00], and at −0.07 [−0.26, 0.12] with the change in visual share held fixed. We agree that a failed forecast does not invalidate fixed-ability anchors in general and that our conclusion about disagreement on perceived difficulty went beyond the evidence. We will remove it, state the finding as the failure to predict human mean scores, and state in Sections 3.3 and 5 that agreement on difficulty requires item-level responses, which the official summaries do not contain.
W3 (visual composition). We agree. Section 6.2 reports that the association for pupils falls from +1.35 to +0.38 [−0.65, 1.40] after accounting for year, yet Sections 6.2 and 6.4 and the Discussion still describe more visual exams as easier for pupils and give composition as the main reason the anchor fails. With year effects the association holds for models (−2.23 [−3.47, −0.78]), but visual share and year correlate at 0.82 within grade groups, which leaves too little independent variation to establish its direction for pupils. We will state the composition effect for models only and explain why its direction for pupils remains open.
Q3 (recent-three-year baseline). Model scores do not improve on the mean of the previous three reference years (MAE 3.55). We added the change in the model score from its own mean over those years, with one slope shared by all grade groups and fitted on earlier years, so a zero slope returns the baseline. The paired MAE difference is −0.13 [−0.42, 0.17] for the ensemble and lies between −0.10 and +0.04 for the four models and the seven new runs, and every interval includes zero. Even the widest interval allows a gain of at most 0.42 points, or 12% of the baseline error, so a small gain remains possible but a useful one is not supported. We will move this baseline and the paired differences into the main forecast table.