acceptodds
Under review as a conference paper at ICLR 2027

Can LLMs Serve as Reference Anchors for the German Kangaroo Competition?

Abstract

Benchmarks for mathematical reasoning rarely preserve the incentives, visual content, and long-running human reference data of real exams, especially outside English. We introduce Kangaroo, a longitudinal benchmark built from the German Mathematical Kangaroo archive from 1998–2025: 140 exams and 3,886 multiple-choice items across five grade groups, 1,746 of them multimodal, with official human results for 115 exams across 23 years. We evaluate four frontier vision–language models in a single-turn exam mode with explicit abstention under the contest's negative marking. GPT-5 is strongest at 88.34% accuracy, but every model loses between 23 and 43 percentage points on multimodal items. We then ask whether model scores can serve as reference anchors for human exam performance, that is, whether a model's score on an exam helps predict the mean score of the pupils who took it. Across the 115 exams, human and model scores are negatively correlated, and the correlation shrinks toward zero once we account for grade and the share of multimodal items. Calibrations fit on earlier years predict later years no better than the historical grade mean: 4.12 against 4.16 points of mean absolute error for an equal-weight ensemble. Current model scores therefore cannot serve as reference anchors for human exam performance.

Primary area:
datasets and benchmarks
Keywords:
multimodal benchmarks, mathematical reasoning, vision-language models, human-model comparison, abstention

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nFyuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper introduces Kangaroo, a benchmark of 3,886 questions from 140 German mathematics exams (1998–2025), including 1,746 multimodal items and human mean scores for 115 exams. Four vision–language models are evaluated under negative marking with abstention. GPT-5 achieves 88.34% accuracy, but all models show 23–43 percentage-point multimodal gaps. Ensemble and human scores are negatively correlated (r=−0.490r=-0.490), with the association weakening after controlling for visual content. Model-score calibrations do not reliably outperform historical grade means (MAE 4.12 vs. 4.16).

Critical strengths
  • The benchmark combines authentic multimodal exams, contest-faithful scoring, and 23 years of human results, providing a valuable resource for studying evaluation validity.

  • Regression controls, year-cluster bootstrap, and expanding-window evaluation on 75 held-out exams strengthen the longitudinal human–model analysis.

Critical weaknesses

The proposed reference-anchor evaluation may be poorly matched to the capabilities of the tested models.

The models outperform human mean scores by 35–47 percentage points on average (Appendix B), while achieving near-ceiling accuracy on text-only questions (92–99%, Table 1). This raises a concern that their scores may be insufficiently sensitive to difficulty differences relevant to typical students. The substantial multimodal performance gaps further suggest that model scores are strongly influenced by visual-content characteristics, which need not correspond to human difficulty.

The interpretation of reference-anchor failure conflates aggregate human performance with exam difficulty.

Human mean scores depend not only on exam difficulty but also on student ability, preparation, and participant composition, which may change across years. A fixed-ability reference anchor could therefore track changes in exam difficulty without necessarily predicting the means of changing student cohorts. The results support a narrower conclusion about the predictive utility of the tested model scores rather than the validity of fixed-ability anchors in general.

The explanation based on visual composition is not fully established.

The positive association between visual share and human scores decreases from +1.35 to +0.38 after adding year fixed effects, with the latter confidence interval including zero (Appendix Table 12). Thus, the evidence that visually richer exams are easier for students is not robust to year controls.

Questions for authors
  • Can the authors clarify whether the proposed reference anchor is intended to predict aggregate human mean scores or measure exam difficulty independently of changing student cohorts? What evidence supports the broader conclusion about human–model difficulty alignment?

  • What evidence rules out ceiling effects as a major reason for the negative result? Can the authors analyze whether model score variability and human–model correlations change with model ability, using the four evaluated models?

  • Appendix Table 15 reports a recent-three-year baseline with MAE 3.55, outperforming the reported model-score calibrations. Does adding model scores improve upon this stronger baseline, with paired confidence intervals?

2 comments

We thank the reviewer for the detailed reading.

W1 and Q2 (ceiling effects and model ability). We agree that the four models score far above the pupils and approach the ceiling on text-only items. On whole exams, however, their scores are not compressed. Within grade groups, their standard deviation across exams is 6.2 to 8.6 points against 4.3 for the human mean score, only 0.9% to 20.9% of exams reach 95 %max, and the correlations with the human mean score lie between −0.26 and −0.32 although the mean scores range from 76.3 to 88.4 %max. On multimodal items alone, where the models score 58.1 to 77.0 %max, the slopes lie between −0.51 and −0.05 per 10 points and every interval includes zero.

We have also extended the ability range in both directions (table in the general comment). Ministral 3 8B scores 53.5 %max, the closest to the pupils' 41.1, and no exam reaches 95. Its score does not improve the forecast and its slope is negative as for the original models, so the result holds from 53.5 to 88.4 %max without a binding ceiling. We report this run as exploratory because 17.5% of its responses hit the output limit or needed the fallback parser; two other treatments of these responses give forecast differences of +0.05 and +0.12. At the top the ceiling does bind. GPT-6.1 Sol scores 99.4 %max, earns the full score on 83.5% of exams, and with a within-grade standard deviation of 1.5 points says almost nothing about how exams differ. The four newer models lose only 0.3 to 11.0 points on multimodal items, against 23 to 43 for the original models, and their exam scores change by −0.9 to +0.1 points per 10 points of visual share, against −1.7 to −3.9 for the original models and −3.1 for Ministral 3 8B. The negative slope disappears together with this dependence, as C2 implies. We agree that the models' dependence on visual content need not reflect human difficulty, and we will state that only one run tests an anchor matched to pupil ability.

W2 and Q1 (what the anchor test measures). We agree, and we change the paper accordingly. The human mean score depends on both the exam and the pupils who took it, and the official summaries, which give a mean score and a score histogram per exam, cannot separate the two. To answer Q1, the test concerns the prediction of human mean scores and does not measure exam difficulty independently of the cohorts. A fixed-ability anchor that tracks difficulty predicts the human mean score if cohort ability changes slowly compared with exam difficulty, which is plausible for national cohorts of several hundred thousand pupils, and we will state this assumption. Under it we can compare changes within a grade group. Year-to-year changes in the ensemble score and the human mean score correlate at −0.20 [−0.40, 0.00], and at −0.07 [−0.26, 0.12] with the change in visual share held fixed. We agree that a failed forecast does not invalidate fixed-ability anchors in general and that our conclusion about disagreement on perceived difficulty went beyond the evidence. We will remove it, state the finding as the failure to predict human mean scores, and state in Sections 3.3 and 5 that agreement on difficulty requires item-level responses, which the official summaries do not contain.

W3 (visual composition). We agree. Section 6.2 reports that the association for pupils falls from +1.35 to +0.38 [−0.65, 1.40] after accounting for year, yet Sections 6.2 and 6.4 and the Discussion still describe more visual exams as easier for pupils and give composition as the main reason the anchor fails. With year effects the association holds for models (−2.23 [−3.47, −0.78]), but visual share and year correlate at 0.82 within grade groups, which leaves too little independent variation to establish its direction for pupils. We will state the composition effect for models only and explain why its direction for pupils remains open.

Q3 (recent-three-year baseline). Model scores do not improve on the mean of the previous three reference years (MAE 3.55). We added the change in the model score from its own mean over those years, with one slope shared by all grade groups and fitted on earlier years, so a zero slope returns the baseline. The paired MAE difference is −0.13 [−0.42, 0.17] for the ensemble and lies between −0.10 and +0.04 for the four models and the seven new runs, and every interval includes zero. Even the widest interval allows a gain of at most 0.42 points, or 12% of the baseline error, so a small gain remains possible but a useful one is not supported. We will move this baseline and the paired differences into the main forecast table.

Thank you for your detail response. The revised manuscript undergone sufficient effort. Your rebuttal resolved many of my concerns. Howerver, construct validity remains limited and the negative result should not be interpreted as proof of zero predictive value. These are meaningful limitations, but they are less serious now that the manuscript acknowledges the restricted scope. I will raise the soundness to 2 and keep my overall assesment as weak rejection.

Reviewer_wbVHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a German mathematics benchmark based on the Kangaroo competition and uses it to investigate whether LLMs can serve as reference anchors for comparing human performance across exams over time.

Critical strengths

1- The main strength of this paper is its focus on a non-English language, which is valuable for broadening LLM research beyond English and improving our understanding of model capabilities in non-English contexts.

Critical weaknesses

1- My main concern is the novelty of this paper. The authors introduce an MMLU-style benchmark for evaluating mathematical reasoning in German. However, as the results show, current LLMs can solve these problems with relatively high accuracy, raising concerns about whether this type of evaluation remains sufficiently challenging and informative. The benchmark may therefore provide limited new insights into the capabilities of contemporary models. Furthermore, the evaluation covers only four models and does not include more recent models, such as newer versions of Qwen, GPT, and Claude Opus. Evaluating a broader range of state-of-the-art models would provide a more comprehensive assessment of the benchmark's relevance and contribution.

2- Another concern is potential data contamination, which could affect the paper's conclusions. Older exam questions have had more opportunities to be included in LLM training data than more recent questions. Consequently, models may perform differently on older exams partly because of potential exposure during training rather than their actual mathematical reasoning capabilities. The authors should acknowledge this possibility and, if feasible, investigate its impact on their findings. Without such analysis, it remains unclear whether potential contamination influences the observed relationships between model performance and human performance across exams over time.

Questions for authors

I have no questions.

2 comments

We thank the reviewer for the assessment.

W1 (novelty and newer models). We agree that current models solve most of these problems. We have run GPT-6.1 Sol, GPT-6 Luna, Claude Haiku 5.5 and Qwen3.8 Flash on all 3,886 items under the same protocol (table in the general comment). They score between 92.1 and 99.4 %max, and GPT-6.1 Sol earns the full score on 83.5% of the 115 exams. Our question is whether a model's exam score can serve as a fixed reference for comparing pupil cohorts across years, and high accuracy does not settle it. Krishnaveti and Rawat (2024) already use GPT-4 as a fixed examinee to trace the difficulty of SAT Math from 2008 to 2023. The German Kangaroo contest pairs the exact exams with 23 years of official results covering 15.6 million participations, so we can test that assumption directly. None of the nine models passes: no model improves the forecast of later human mean scores over either the grade group's historical mean or the mean of the previous three years.

As models approach the ceiling, their scores vary less across exams. The within-grade standard deviation is 1.5 points for GPT-6.1 Sol against 4.3 for the pupils, so its score says little about how exams differ, and stronger performance does not make models reference anchors. The negative within-grade association in the submission is specific to the original models and disappears in the four newer models, which lose far less accuracy on multimodal items. We will give the reference-anchor question more prominence in the abstract and the contribution list, where the benchmark currently comes first.

W2 (contamination). We agree that exposure may vary with exam age, as the submission acknowledges, but three results speak against it as the source of our findings. At equal item composition, accuracy changes per later decade by −0.52 to +1.12 points and every interval includes zero, so no model shows the advantage on older items that memorization would produce. GPT-5 is no less accurate on the March 2025 contest, which falls after its documented knowledge cutoff, than on the 2013-2024 items (+3.87 points [−1.89, 9.26]). Finally, the estimate with year effects compares grade groups within one contest year and shows no positive association (−0.32 [−1.99, 1.33]), and the recent-three-year test removes exposure that changes smoothly with exam age. We will report these checks in an appendix table and keep exposure as a stated limitation.

Thank you for the detailed response. My concern regarding the evaluation of newer models has been addressed. However, I still have concerns about the paper's overall novelty and the significance of its core idea for ICLR. Despite the clarification, my overall assessment remains unchanged.

Reviewer_SEYPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces "Kangaroo", a longitudinal benchmark of multiple-choice math exams (1998–2025) with both text-only and multimodal items, alongside official pupil results. Four frontier VLMs are evaluated under the negative-marking rule with explicit abstention. While models are near ceiling on text-only items, they drop significantly on multimodal ones. The main question is whether model scores can serve as a "reference anchor" to compare pupil cohorts across years. However, model and pupil scores are negatively correlated. Furthermore, model scores neither provide information beyond the exam's multimodal share nor improve future score predictions via expanding-window calibration.

Critical strengths

The question is valuable, and the test is falsifiable. Using a fixed model “examinee” to separate harder exams from weaker cohorts, and so track how pupils perform in a long-running contest over the years, is a good idea. The experiments are reasonably thorough, the ablations are well designed, and the negative result is reported honestly.

Critical weaknesses

1、 Each model is run only once, with a single prompt and no option reordering. If model scores are to serve as a reference, averaging over repeated runs and perturbing option order are necessary rather than optional. Otherwise the subsequent fits are affected by run-level noise. 2、 The paper does not cite closely related work. SMART-840 already reports no significant correlation between model and child difficulty on Kangaroo problems, using item-level pupil statistics. M3Kang builds a Kangaroo benchmark in 108 languages, including German, and compares models with results from over 68,000 students.

Questions for authors

1、 Can the authors provide evidence that training-data contamination has little effect on the results? Older exams have been public longer and are more likely to have been seen in training. Without such evidence, the conclusions could be substantially affected, or even unreliable. 2、 Is there anything specific about German in the context of mathematical problems? The paper’s own language ablation suggests that switching between major languages has little effect on models’ math performance. The German-language dataset therefore does not appear to be a contribution in itself. 3、 The analyses rely on additive linear models. Have the authors tried non-linear fits, or an interaction between model score and visual share, so that visual share can change how model and pupil scores relate rather than only adding a separate term? Could this give a better fit?

1 comment

We thank the reviewer for the constructive assessment and for pointing us to related work.

W1 (single run, option order). We agree that a reference should not rest on one run, and we have run both requested checks for Sonnet 4.5. A repeat of all 3,886 items about a year after the original run gives 77.48% accuracy against 77.35% (+0.13 points [−0.93, 1.13]), with the same outcome on 88.75% of items. Exam scores of the two runs correlate at 0.69, the within-grade slopes are −1.60 and −1.61, and neither run nor their mean improves the forecast. A third run shuffles the answer options on the 3,409 items that allow it; the others print letters inside option images or refer to them in the text. On the reordered items the model chooses each position in 18.8% to 21.3% of its answers, its accuracy lies between 82.3% and 85.0% for each position of the correct option, and item outcomes agree with the repeat on 86.77% of items. The forecast result is unchanged (+0.14 [−0.12, 0.40]), and the slope keeps its sign but weakens to −1.06 [−2.59, 0.20], against −1.94 [−3.39, −0.60] for the mean of the three runs. A single run thus estimates the slope with visible noise, while the forecast conclusion holds across runs. For GPT-5 and Qwen3-VL we have a second pass only on 200 items from the language ablation (same outcome on 96.0% and 92.5%), so full repeats and reordering remain untested for them. We will report the three Sonnet runs.

W2 (SMART-840 and M3Kang). We agree that both papers should have been cited. Both compare models and pupils on individual items, and the absence of a significant correlation between model and child difficulty in SMART-840 is consistent with our finding. We ask whether a model's score on a whole exam predicts the national cohort's mean, using 23 years of official results with 15.6 million partiicipations and forecasting later exams. A reference anchor needs this property, item-level agreement neither implies nor excludes it, and Krishnaveti and Rawat (2024) already assume it when they use GPT-4 as a fixed examinee to trace the difficulty of SAT Math from 2008 to 2023. We will add the three works with a paragraph on how ours relates to them and restrict our claim that Kangaroo is distinct to the long series of official human results and the forecast test.

Q1 (contamination). We cannot rule out training exposure for any public exam, but three results speak against it as the source of our findings. If older exams were memorized, models should score higher on older items. At equal composition (visual subtype, grade group, point value), accuracy changes per later decade by +1.12 [−0.30, 2.45] points for GPT-5, +1.05 [−0.46, 2.43] for Qwen3-VL, +0.88 [−0.38, 2.35] for Grok 4 Fast and −0.52 [−2.12, 0.95] for Claude Sonnet 4.5, so no model shows that advantage. GPT-5 is no less accurate on the March 2025 contest, which falls after its documented knowledge cutoff of 30 September 2024, than on the 2013-2024 items (+3.87 points [−1.89, 9.26]). Finally, the estimate with year effects compares grade groups within one contest year and shows no positive association (−0.32 [−1.99, 1.33]), and the recent-three-year test (Q3 of Reviewer nFyu) removes exposure that changes smoothly with exam age. We will report these checks in an appendix table and keep exposure as a stated limitation.

Q2 (German). We agree that German is not a contribution in itself, as the language ablation suggests. The German contest is useful because its official summaries give the national cohort's mean score on the exact exams of 23 years, and an anchor test needs the exams as the pupils took them. We will describe the contribution as these exams paired with their official results and drop the language from it.

Q3 (non-linear fits and interaction). We fitted both. An interaction of the ensemble score with visual share raises the in-sample R² from 0.326 to 0.356, with an interval for the interaction term that narrowly includes zero, and implies a slope of −2.95 at a visual share of 30% and −0.92 [−2.91, 1.01] at 60%. A squared model score gives an R² of 0.354. Neither fit improves forecasts of later years. With a slope shared across grade groups, the interaction calibration reaches an MAE of 3.78 against 3.89 for visual share alone in the same form (−0.11 [−0.30, 0.10]), and fitted per grade group it reaches 4.48, above the grade mean's 4.16. We will add both fits to the appendix.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers and the AC

We thank the three reviewers for their careful reading. We make two changes to address points that several reviewers raised.

Our test asks whether the exam scores of the tested models help predict the mean score of the pupils who took each exam. They do not. As Reviewer nFyu points out, the pupils' mean depends on both the exam and the cohort who took it. The test therefore cannot tell us whether models and pupils disagree on what makes an exam hard. We will remove the conclusion about disagreement on perceived difficulty from the abstract. In the abstract, introduction and conclusion, we will state that the exam scores of the tested models do not help predict the mean scores of pupil cohorts and therefore cannot serve as reference anchors in the form we test.

We have completed seven further full runs on all 3,886 items. Four use newer models (GPT-6.1 Sol, GPT-6 Luna, Claude Haiku 5.5, Qwen3.8 Flash). Two repeat Claude Sonnet 4.5, once unchanged about a year after the original run and once with the answer options in shuffled order. We also ran a small recent model, Ministral 3 8B, as an exploratory check because its mean score of 53.5 %max is the closest to the pupils' 41.1.

The table gives results for every series on the 115 exams with official results. Within grade groups, SD is the standard deviation of exam scores, r is their correlation with the human mean score, and the slope gives the change in the human mean score per 10-point increase in the model score (C1). The two ΔMAE columns show how forecast error changes on the 75 held-out exams. The first replaces the grade group's historical mean with a calibration on the model score fitted to earlier years in that grade group (C3, baseline MAE 4.16). The second starts with the mean of the previous three reference years, a stronger baseline with an MAE of 3.55, and adds the change in the model score from its own recent mean. A negative ΔMAE indicates a better forecast. We report year-cluster bootstrap 95% intervals.

SeriesMean %maxSDrSlope per 10 pointsΔMAE vs. grade meanΔMAE added to recent three years
Human mean score41.14.3
GPT-5 (submission)88.46.2−0.27−1.85 [−3.47, −0.17]+0.00 [−0.45, 0.45]+0.00 [−0.30, 0.30]
Qwen3-VL (submission)85.67.2−0.26−1.57 [−3.03, −0.23]−0.05 [−0.51, 0.36]−0.10 [−0.34, 0.12]
Grok 4 Fast (submission)79.97.6−0.29−1.65 [−3.75, 0.06]+0.00 [−0.64, 0.66]−0.04 [−0.44, 0.48]
Claude Sonnet 4.5 (submission)76.38.6−0.32−1.60 [−2.69, −0.55]+0.09 [−0.27, 0.43]−0.03 [−0.13, 0.07]
Claude Sonnet 4.5, repeat76.58.4−0.31−1.61 [−2.54, −0.72]+0.09 [−0.23, 0.39]−0.09 [−0.25, 0.07]
Claude Sonnet 4.5, shuffled options78.17.6−0.19−1.06 [−2.59, 0.20]+0.14 [−0.12, 0.40]+0.02 [−0.02, 0.07]
GPT-6.1 Sol99.41.5+0.14+4.02 [−0.68, 9.24]+0.12 [0.01, 0.25]+0.01 [−0.06, 0.08]
Qwen3.8 Flash95.63.4−0.14−1.80 [−4.32, 0.33]+0.07 [−0.18, 0.33]+0.03 [−0.05, 0.11]
Claude Haiku 5.593.64.5+0.07+0.63 [−1.27, 2.47]+0.23 [−0.02, 0.52]0.00 [−0.04, 0.04]
GPT-6 Luna92.15.6+0.11+0.85 [−0.66, 2.28]+0.22 [0.03, 0.41]−0.02 [−0.09, 0.03]
Ministral 3 8B53.510.3−0.18−0.74 [−1.54, −0.05]+0.09 [−0.15, 0.31]+0.04 [−0.04, 0.14]

No series clearly improves either forecast. The differences range from −0.10 to +0.23 points against baseline errors of 4.16 and 3.55. The only two intervals that exclude zero indicate a larger error. The negative slope of the four original models reappears in the Sonnet repeat and in Ministral 3 8B. The four newer models, however, have closed most of the gap between text-only and multimodal items. Their scores no longer fall as the visual share of an exam increases, and the negative slope disappears with that dependence, as the submission explains (C2, details in the response to Reviewer nFyu). Our conclusion therefore rests on the forecast test, as it does in the Discussion of the submission. We will report all runs in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.