- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Sci-ρ, a programmatically generated (i.e., "symbolic") benchmark for STEM problems that require both text and images for solution. The benchmark comprises 606 expert-authored Python templates spanning five disciplines: mathematics, physics, chemistry, biology, and computer science. Approximately 62% of the templates are adapted from past exam questions and existing benchmarks, while the remainder are original. Problem descriptions were translated into six additional languages (Indonesian, Chinese, Arabic, Hindi, Kazakh, and Swahili) by Gemini 3 Pro and verified by native speakers, whereas the image content remains in English. Each template generates 10 variants, resulting in a total of 42,420 instances.
The authors evaluated 17 vision-language models (VLMs) using metrics such as average accuracy and worst-case accuracy, as well as step-wise precision, recall, and F1 scores judged GPT-5.4.
Critical strengths
1. A reusable and contamination-resistant resource that extends the scope of symbolic robustness evaluation from English mathematics to a broader domain. Traditional symbolic or dynamic visual benchmarks (such as DynaMath) have primarily been confined to mathematics and English. Sci-ρ provides executable templates and step-by-step solution processes covering five major STEM fields and seven languages. Because test instances can be regenerated, researchers can continuously track robustness performance without relying on static test sets that may have leaked into training data. The resource employs a curriculum-based classification architecture (comprising 113 topics) alongside relevant metadata—such as source, difficulty, and question type—to facilitate in-depth analysis. This is a key reason why the paper holds significant value for the academic community.
2. Comprehensive and transparent evaluation. The study evaluates 17 models across various families and scales, reporting "worst-case" metrics at both the answer and solution-step levels. The research team cross-validated step-level evaluation results using evaluators from different model families and made all prompts, evaluation criteria, and decoding parameters publicly available in the appendix. Consequently, the resulting empirical findings are verifiable and, in principle, reproducible.
Critical weaknesses
1. Lack of control conditions to support key claims regarding robustness. The difference between and was not disentangled from sampling noise. The GPT-5.4 and Qwen3.5 models used a temperature of 1.0 during decoding, whereas all other models used 0.0 (see Table 7); furthermore, the noise floor resulting from repeated sampling was not reported. Nor was it demonstrated that the difficulty levels across variants were comparable. The ground truth value in Figure 1 is 0.01, implying that a 1% tolerance effectively demands a perfectly correct answer. Additionally, assertions regarding specific subjects or perturbation levels rely on very small subsets (Biology: 23 templates; Colors: 10 templates) without providing confidence intervals. Given that robustness is the central theme of the paper, these shortcomings limit the informative value the benchmark can currently provide.
2. Secondary analyses suffer from confounding factors and insufficient validation.
- Multilingual support. Since the images retain English labels and the prompts mandate "strict use of English" for reasoning (see Table 8), only the problem description itself is non-English.
- IAR. This ratio utilizes average attention values calculated per token. Although the number of image tokens () remains constant across languages, the number of text tokens () fluctuates based on tokenizer efficiency; consequently, the observed patterns (lowest for Chinese, highest for Hindi and Kazakh) may simply be an artifact of this normalization method.
- Step-based metrics. These evaluations rely on GPT-5.4 as the judge and lack human verification.
- Inconsistencies between text and tables. Certain claims contradict the data presented in the paper's tables. For example, lines 389–390 state that, across almost all models, the field of "Mathematics" experienced the largest drop from F to F$_{wst}$; however, according to Table 2, the decline in this field was not the greatest, and the counts for the various fields shown in Figure 5 do not match the dataset's actual content.
Questions for authors
-
The benchmark scores are actually quite high for the models (GPT-5.4 88.8%, Qwen3.5-27B 82.3%), implying that they can solve most of the problems; this diminishes the significance of the benchmark itself.
-
What is the worst-case acc when sampling the same instance for 10 times? And how does the result change if you use a 0 temperature?
-
Does the 1% tolerance refer to relative error or absolute error? Additionally, how should cases where the ground truth is close to zero be handled? Would the main conclusions change if the tolerance set to 5%?
-
What are the syntax parsing failure rates for the answers, broken down by model and language? This is crucial for understanding performance degradation in low-resource languages.
-
Can the impact of language be observed even when using summed attention or controlling for the number of text tokens?
4 comments
W2.2: IAR Normalization Artifact
| Language | Text tokens | Normalized IAR | Summed IAR |
|---|---|---|---|
| Chinese | 43.5 | 0.139 | 0.561 |
| English | 45.8 | 0.158 | 0.587 |
| Arabic | 55.7 | 0.169 | 0.564 |
| Indonesian | 63.6 | 0.177 | 0.549 |
| Swahili | 85.9 | 0.189 | 0.504 |
| Kazakh | 114.5 | 0.221 | 0.491 |
| Hindi | 158.6 | 0.235 | 0.438 |
We thank the reviewer for this insightful point. Following the suggestion, we recomputed the IAR using summed attention and compared it with the per-token normalized version (table above). Under per-token normalization, the IAR increases monotonically with the number of text tokens, from 0.139 for Chinese (43.5 tokens) to 0.235 for Hindi (158.6 tokens). Under summed attention, the ordering largely reverses, decreasing from 0.587 for English to 0.438 for Hindi. Both metrics thus follow the number of text tokens, in opposite directions. We therefore agree that tokenizer efficiency is a valid confounding factor. Since text-token counts vary across languages while image-token counts stay constant, the aggregation method can shift the IAR independently of the model's actual reliance on the image, and our results cannot disentangle language from this factor or from related confounders such as text length. We will therefore remove the claim that the IAR reflects language-dependent visual reliance from our main findings and report both metrics in Section 4.3 as an exploratory analysis. We will present this as a cautionary finding, since attention-based measures in multilingual VLMs are sensitive to tokenization, and suggest controlling for text length and interventional methods (e.g., image ablation) as future work. This revision does not affect our main contributions.
W2.3: No Human Verification of Step-Level Metrics
We disagree that the step-level evaluation lacks human input. All gold solutions were written by contributors and verified by domain experts (Section 3.1, Appendix A.2). The judge model strictly compares model steps to these human-authored standards (Tables 9 and 10) rather than independently determining correctness. Additionally, replacing the judge with Gemini 2.5 Pro (Appendix E) shows that the gap between and remains stable at 15.5--25.0 points across all subjects (Table 13).
Q1: Benchmark Difficulty / Saturation
We disagree. High is where our argument starts. Symbolic benchmarks were built because static scores overstate what models can do. GSM-Symbolic (Mirzadeh et al., ICLR 2025) found that LLMs with high GSM8K scores lose accuracy when only the numbers or names in a question change, and DynaMath (Zou et al., 2025), FinChain (Xie et al., 2026), and EngTrace (Gull et al., 2026) test the same weakness in visual mathematics, finance, and engineering. is the score a static benchmark would report. counts a template as solved only when the model gets all 10 variants right, and by this measure GPT-5.4 drops from 88.8 to 73.9 and no open-weight model reaches 52 (Table 1). The best model we tested misses at least one variant on a quarter of the templates, so the benchmark is not saturated. Each template can generate more variants, and can only fall as grows, so Sci- can be made harder without new annotation.
Q2: Repeated Sampling and Temprature 0
Answered in W1.1
Q3: Tolerance Definition and Sensitivity
| Model | Before: 1% (paper) | After: 5% |
|---|---|---|
| GPT-5.4 | 88.8/73.9 | 91.1/78.4 |
| Gemini-2.5-Pro | 85.9/62.7 | 88.8/69.3 |
| Qwen3.5-27B | 82.3/51.2 | 84.7/55.1 |
| Llama-4-Scout | 69.2/33.7 | 71.8/37.7 |
The 1% tolerance refers to relative error (that is, a prediction is accepted if ). We do not apply special handling when the ground truth is zero, so such cases require an exact match. To examine sensitivity of tolerance, as the reviewer suggested we re-scored all responses with a 5% tolerance (table above). Relaxing the tolerance raises by only 2.3–2.9 points and narrows the gap between and only slightly across all four models. The ranking of models is also unchanged. Therefore, our main conclusion that VLMs show a substantial gap between average-case and worst-case accuracy holds under a looser tolerance.
Q4: Answer Parsing Failure Rates
We counted the responses with no extractable answer and will add the full table to the appendix. GPT-5.4 has almost none (at most 0.02% in any language), yet it still loses 14.9 points from to . Both Gemini models and Llama-4-Scout also stay below 2.2% in every language. Rates are higher for some other models, mostly outside English, which matches the language degradation the paper reports.
Q5: Language Effect Under Summed Attention / Token-Count Control
Answered in W2.2
W2.4: Inconsistencies Between Text and Tables/Figures
Thanks for catching these. The tables are correct; the mistakes are just in the text, and we will fix them. Since all our findings are computed from the tables, none of the conclusions change. Lines 389--390 discuss which subject has the largest drop, but our main claim is that stays below for every subject and model, which is clearly shown in Table 2. For Figure 5, we reran the attention analysis on the final 606 templates and found the exact same ranking (Chinese lowest, Kazakh and Hindi highest).
Thank you for your response; I will take the other reviewer's comments into account before making my final decision.
We thank the reviewers for their constructive feedback and recognition of Sci-ρ's contributions. Below are our point-by-point responses to all weaknesses and questions raised.
W1.1: Gap Not Disentangled from Sampling Noise
The decoding settings are in Table 7. To separate the gap from sampling noise, we ran an additional experiment on the English subset. We gave each model the same variant 10 times, at the temperature used in the paper, and took the worst case over those repeats instead of over 10 different variants. The 10-variant results are the paper's runs. We can clearly see that the gaps exist for both temp = 1.0 and temp = 0.0.
For every model, the worst case drops more across variants than across repeats. Qwen3-VL-8B, for example, loses 36.8 points across variants but only 2.0 across repeats. Our templates change only the sampled values between variants, so the variant gap measures the effect of varying the problem, as we argue in Section 4.2.
W1.2: Comparability of Difficulty Across Variants
All variants of a template share the same underlying reasoning: they are generated from the same problem structure and solved with the same sequence of solution steps, and only the sampled parameters (e.g., numerical values, shapes, colors, or function types) differ. Difficulty is therefore comparable across variants by construction.
W1.3: Strictness of the 1% Tolerance
Even with a 1% tolerance, exact-match grading can still mark a response as incorrect because of rounding or precision differences, and prior benchmarks that score only final answers (including DynaMath) share this limitation. This is precisely why Sci-ρ includes an alternative grading scheme, namely step-level evaluation against gold solutions, which assesses the reasoning process independently of the precision of the final numerical answer. Under this evaluation, the gap between average-case and worst-case performance remains substantial (e.g., GPT-5.4's F drops from 88.0 to 67.8 in Mathematics), indicating that the robustness gap is not merely an artifact of strict numerical tolerance.
W1.4: Small Subsets Without Confidence Intervals
Biology is small because few IGCSE and A-Level Biology problems actually need an image (Section 3.1). We added 95% bootstrap confidence intervals, resampling over templates. For GPT-5.4, Biology Awst is 47.8 [26.1, 69.6], still substantially lower than every other subject (for example, 24.5 points [3.1, 45.9] below Mathematics). Color's error rate is 37.0% [16.0, 60.0], higher than every other perturbation type. Both claims in the paper hold up with the intervals. And since each template is code, these subsets are easy to grow later.
W2.1: Limited Scope of the Multilingual Setting
We view this as a deliberate design choice rather than a weakness. The problem description is where the conditions, given quantities, and the question itself are stated, so a model must fully understand the non-English text to solve the problem. Keeping the images in English holds the visual input constant across all seven languages, so the language of the text is the only thing that varies. This setting is also realistic: non-English-speaking students routinely work with problems written in their native language alongside English-labeled diagrams from international textbooks and exam papers. Finally, we instruct models to reason in English so that their reasoning steps can be compared consistently against our gold solutions in the step-level evaluation. As reported, performance drops are driven by the text language despite identical images (e.g., InternVL3.5-4B gains 0.5 points in Chinese but drops 15.3 points in Swahili; Table 11, Figure 4).