acceptodds
Under review as a conference paper at ICLR 2027

A Text-Free Preschool Mathematics Benchmark: What Its Scores Do and Do Not Measure

Abstract

We present a procedurally generated benchmark for preschool mathematics — counting, comparing, adding and subtracting with counters, pattern continuation, conservation of number and equal sharing — designed for a cascaded agent of small models rather than a single large one. The benchmark is rendered entirely without text: identity, size, position, container contents, holding state, episode phase and counting boards are encoded as colour, geometry and layout, so that no task can be solved by reading. Where a family admits a non-verbal answer the agent must demonstrate the result by acting, and both cascade stages are educated on corpora generated from the same distribution, shipped with the benchmark. We evaluate 21 configurations over 608 evaluation episodes and find that the training method, not the architecture, accounts for the largest gap in the results: full-parameter training of a 2B front end reaches 0.809 where low-rank training of the same model reaches 0.480, and a quarter of the education data with full-parameter training beats all of it with low-rank training. The front end bounds overall performance, with the back end contributing a family-specific competence profile that does not scale monotonically. We then audit the benchmark itself, computing for each family the score attainable by an agent that neither perceives nor reasons. Sixteen of 19 families carry usable signal, two are saturated, and one has none: conservation of number reports a peak success rate of 0.53 against a floor of 0.500, so its apparent difficulty is an answer prior being collected, and no configuration clears its own constant strategy by more than one item. We argue that this floor computation belongs in the benchmark's release gate rather than in post-hoc analysis, and that for families with skewed answers the headline metric must be balanced accuracy reported against chance.

Primary area:
datasets and benchmarks
Keywords:
benchmark design and validity, evaluation methodology, vision-language models, visual reasoning without language, cascaded small models, early mathematics learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_SdojOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a benchmark that measures the ability of a pair of small weights (< 8B) models to reason on a rendered image of geometric objects: a VLM describes the image into text, to which an LLM decides on the action to take.

Some of the reported key findings: 1) full parameter training outperforms LoRA on their proposed benchmark; 2) the VLM is the bottleneck of the overall performance; 3) some of the proposed benchmark tasks are saturated by the models tested.

Due to a very heavy usage of AI for writing, the motivation / experimental setup / results are all very hard to parse. The manuscript uses a lot of non-conventional keywords (e.g., “a cascade of small models” (line 50) or “cascade stages are educated on corpora” (line 21)) or metrics (“five-year-old criterion” (line 304)) that obscure the content. None of these keywords are sufficiently described / defined, and the reader has to infer their meaning from the context. My best understanding of “cascade” is a pair of models where the output of the first model becomes the input to the second model; “educated” means being finetuned; “five-year-old criterion” means how long tshe LLM action trajectory is compared to the gold trajectory.

Figures and tables are also very difficult to parse. See Figure 1 for example. To my best understanding, the difference in (b) and (c) is that (c) seems to be a rendered image after an LLM attempts to use the action “reach+grasp” on the blue block in (b). However, I have no clue as to what the other parts of the figure is trying to convey.

At this stage of the manuscript, it is unclear what the exact contribution of this paper is.

Critical strengths

N/A

Critical weaknesses
  1. (Major) The main research question of this paper is obscure
  • As of right now, the paper progresses from 1) building a synthetic benchmark -> 2) training some chosen pairs of (VLM, LLM) on in-distribution benchmark data and evaluating the resulting setup on the proposed benchmark -> 3) analyzing which proposed benchmark tasks have been saturated by the trained models.

  • The paper only evaluates models trained on this synthetic dataset. This makes it hard to decouple the usefulness of the benchmark from the effect training dynamics. Evaluating existing models would help validate the quality / calibration of the benchmark.

  • The same goes for the “audit” part of the paper. If the models are all finetuned on a synthetic dataset in-distribution, it is not difficult to make them saturate on the benchmark.

  1. (Major) The main motivation of this paper is unclear
  • The paper says this benchmark is “for a cascade of small models” (line 50). To my best knowledge, a cascade refers to a (VLM, LLM) pair, where a VLM first describes the input image, and the LLM reasons on that output text. However, it is never described why the VLM is not being directly evaluated on solving the task directly from the input image. Why is there a need to evaluate this pair, instead of a single model? The paper suggests that “decomposing a system into perception and planning modules is an old idea” (lines 101-102), but the VLMs are already decomposed into such modules. Internally, there is a visual encoder that perceives the image input, and there are transformer decoder layers that do that planning.

  • The paragraph titled “Educational agents and preschool mathematics” under the related work section is very confusing. The paper suggests that “Whether a model can be taught a competence it does not acquire zero-shot” (line 89-90) is somehow relevant to early-childhood education, then cites a paper that uses an interactive AI to assist children learn. Here are some detailed questions.

    • Machine learning is all about models learning to solve the task it was trained on. This concept is not new.
    • It is unclear how this concept is relevant to early-childhood education.
    • It is also unclear how the proposed experimental setup is relevant to an AI agent that assists children learn, nor “children’s over-trust of conversational agents” (line 99).
  1. (Major) The experimental setup is unclear.
  • The 19 tasks are never fully described. Table 2 in the Appendix only lists the names of the tasks, not what they are. What exactly is the input / output structure? What is the model expected to do?

  • The paper refers to “answer modality” (lines 136-137) and classifies each task as either “say” or “act” (Table 2, page 14). It seems like the authors are classifying a task as a “say” modality if the final required action is “say” from a choice of 8 (lines 264-265). Note that this is not a conventional definition of a modality. Since the second stage of the benchmark passes through an LLM, the only modality relevant here is “text.”

  • The ablations are run (Table 6) without a clear purpose. The (VLM, LLM) pairs are chosen seemingly arbitrarily, and full parameter finetuning and LoRA finetuning are also mix-and-match ed without a clear explanation. A good experimental design would ablate one design choice at a time for a particular purpose.

Questions for authors
  1. Can the authors clarify the motivation and the main research question of the paper?

  2. Can the authors clarify the experimental setup?

Reviewer_UPN8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a synthetic benchmark of preschool-style mathematics tasks and studies how a trained pair of small models performs on it. The paper also evaluates and criticizes its own proposed benchmark by auditing the benchmark.

Critical strengths
  1. no strength is observed.
Critical weaknesses
  1. This paper appears to be heavily AI generated.

  2. The paper doesn't have a central contribution: it moves between creating a benchmark, comparing fine-tuning methods, and auditing benchmark validity. Authors pick a niche domain (text-free preschool mathematics), test small opensource models after finetuning on the generated set, and later do self-audit that find the benchmark problematic.

Questions for authors
  1. Procedurally generating images could potentially be a good source as a benchmark. Authors could have focused their efforts on that and try to create a benchmark that tests frontier visual reasoning models.
Reviewer_rSYqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a procedurally generated benchmark of preschool mathematics-inspired tasks for small multimodal cascades. It includes 19 task families across four difficulty levels, with 608 evaluation episodes. A vision front end converts text-free rendered images into structured descriptions, and a language back end uses these descriptions and textual instructions to select actions. The paper evaluates different model configurations and training methods, examines the contribution of a hand-engineered repair layer, and audits task-level scores using answer-frequency baselines and performance spread. Its intended contribution is both a controlled evaluation environment and an analysis of what benchmark scores can and cannot establish about model capabilities.

Critical strengths
  1. The benchmark provides a controlled and inspectable evaluation environment. Procedural generation, symbolic success checks, and the separation between visual perception and action selection provide useful infrastructure for studying failures under controlled conditions. The released implementation also makes important evaluation choices accessible for scrutiny.
  2. The paper examines limitations of its own evaluation rather than reporting success rates alone. The repair ablation and the analysis of answer-frequency baselines expose substantial dependencies and weaknesses in the reported scores. This transparency is valuable. In particular, identifying a task family whose best raw accuracy barely exceeds a constant-answer baseline is a useful caution against interpreting aggregate accuracy as direct evidence of mathematical competence.
Critical weaknesses
  1. The intended scientific use of the benchmark is insufficiently established. My concern is not that a synthetic benchmark must demonstrate an immediate real-world application. Rather, the paper needs to explain what specific capability it measures and what scientific conclusion its scores enable. These tasks combine visual recognition, structured-description generation, instruction following, state tracking, and action selection. It remains unclear how success or failure should be attributed to mathematical reasoning rather than these other demands. The connection to preschool mathematics motivates the task content, but does not by itself validate the benchmark as a measure of an analogous capability in AI systems. The paper should explain which competing explanations its task design distinguishes and what additional diagnostic value it provides over existing childhood-inspired cognitive benchmarks and controlled interactive environments. Without this account, the experiments establish performance within a particular engineered pipeline, while the broader significance of the resulting capability profiles remains uncertain.
  2. The repair layer substantially limits the interpretation of the central results. In the reported G1 repair ablation, success falls from 509/608 episodes with repair to 137/608 without it, and 12 task families fall to zero. This makes the information available to the repair layer central to the evaluation. The supplementary implementation appears to do more than normalize malformed descriptions. For example, _recover_counter_inventory reads counter identities and locations from world.objects and the held object from world.hand, then reconstructs object entries and regional counts. These quantities may be available without evidence that the visual front end previously observed them. Although this is not direct access to answer labels, it constitutes an additional state-information channel whose relationship to the claimed observation protocol needs justification. Consequently, the results do not yet adequately support attributing performance to the learned visual–language cascade or interpreting the front end as the principal performance bottleneck. A distinction between formatting repairs, information reconstructed from accessible action history, and information read from simulator state is necessary to assess these claims.
Questions for authors
  1. What specific scientific question is the benchmark designed to answer? Which conclusions about mathematical reasoning, perception, or interactive state tracking can its scores support, and how does the evaluation distinguish these explanations? Clarifying this would materially affect my assessment of the benchmark’s contribution.
  2. What information is the agent legitimately allowed to observe, and how does each state-dependent repair respect that boundary? In particular, can the counter inventory and regional counts reconstructed by _recover_counter_inventory always be recovered from the agent’s previous observations and actions? Evidence separating formatting corrections from simulator-state information would materially affect my assessment of the reported capability attributions.
  3. How should the floor audit distinguish performance above a constant-answer baseline from evidence of input-dependent competence or discrimination between models? For F CONSERVE NUMBER, the best raw accuracy is 17/32, compared with a constant-answer baseline of 16/32, while the reported balanced accuracy is approximately 0.479 against a chance level of 1/3. These statistics answer different questions. Please clarify the precise conclusion supported by the “no signal” designation, including its uncertainty, and whether the zero floor assigned to action-based families is intended only as a lower bound.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.