acceptodds
Under review as a conference paper at ICLR 2027

Sci-: A Multilingual Visually-Grounded Symbolic Benchmark for STEM Problems

Abstract

Symbolic benchmarks have emerged as a key approach to assess model robustness under minor modifications to STEM-related questions. However, existing symbolic benchmarks mostly remain limited to mathematical reasoning, lack visual grounding, and are predominantly in English. In this work, we introduce Sci- ( ), a dynamic benchmark for visually-grounded STEM problems spanning five subjects and seven languages, comprising 4,242 problem templates (606 per language) crafted by domain experts, including Olympiad medalists. Each template is implemented as executable Python code that generates diverse but equivalent problem instances by varying numerical values, visual patterns, geometric shapes, color schemes, and function types, resulting in 42,420 instances in total, each paired with reasoning steps and ground-truth solutions. We evaluated 17 state-of-the-art VLMs and discovered a noticeable gap between worst-case accuracy (defined as the proportion of problem templates that a model answers correctly across every generated variation) and average accuracy. We also discovered that smaller models show noticeable performance degradation across languages, whereas proprietary and larger models remain robust. Step-level evaluation reflects this same trend, revealing a significant gap between average F1 and worst-case F1 scores. Finally, our inspection of attention heads of a VLM reveals substantial cross-lingual variation in the relative attention allocated to image tokens compared to text tokens. Our work highlights the importance of evaluation beyond static benchmarks as a metric to measure the quality of VLMs.

Primary area:
datasets and benchmarks
Keywords:
Vision Language Models, Robustness, Reasoning, STEM, Multilingual, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_mTCvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Sci-ρ, a programmatically generated (i.e., "symbolic") benchmark for STEM problems that require both text and images for solution. The benchmark comprises 606 expert-authored Python templates spanning five disciplines: mathematics, physics, chemistry, biology, and computer science. Approximately 62% of the templates are adapted from past exam questions and existing benchmarks, while the remainder are original. Problem descriptions were translated into six additional languages ​​(Indonesian, Chinese, Arabic, Hindi, Kazakh, and Swahili) by Gemini 3 Pro and verified by native speakers, whereas the image content remains in English. Each template generates 10 variants, resulting in a total of 42,420 instances.

The authors evaluated 17 vision-language models (VLMs) using metrics such as average accuracy and worst-case accuracy, as well as step-wise precision, recall, and F1 scores judged GPT-5.4.

Critical strengths

1. A reusable and contamination-resistant resource that extends the scope of symbolic robustness evaluation from English mathematics to a broader domain. Traditional symbolic or dynamic visual benchmarks (such as DynaMath) have primarily been confined to mathematics and English. Sci-ρ provides executable templates and step-by-step solution processes covering five major STEM fields and seven languages. Because test instances can be regenerated, researchers can continuously track robustness performance without relying on static test sets that may have leaked into training data. The resource employs a curriculum-based classification architecture (comprising 113 topics) alongside relevant metadata—such as source, difficulty, and question type—to facilitate in-depth analysis. This is a key reason why the paper holds significant value for the academic community.

2. Comprehensive and transparent evaluation. The study evaluates 17 models across various families and scales, reporting "worst-case" metrics at both the answer and solution-step levels. The research team cross-validated step-level evaluation results using evaluators from different model families and made all prompts, evaluation criteria, and decoding parameters publicly available in the appendix. Consequently, the resulting empirical findings are verifiable and, in principle, reproducible.

Critical weaknesses

1. Lack of control conditions to support key claims regarding robustness. The difference between Aavg\mathcal{A}_{avg} and Awst\mathcal{A}_{wst} was not disentangled from sampling noise. The GPT-5.4 and Qwen3.5 models used a temperature of 1.0 during decoding, whereas all other models used 0.0 (see Table 7); furthermore, the noise floor resulting from repeated sampling was not reported. Nor was it demonstrated that the difficulty levels across variants were comparable. The ground truth value in Figure 1 is 0.01, implying that a 1% tolerance effectively demands a perfectly correct answer. Additionally, assertions regarding specific subjects or perturbation levels rely on very small subsets (Biology: 23 templates; Colors: 10 templates) without providing confidence intervals. Given that robustness is the central theme of the paper, these shortcomings limit the informative value the benchmark can currently provide.

2. Secondary analyses suffer from confounding factors and insufficient validation.

  • Multilingual support. Since the images retain English labels and the prompts mandate "strict use of English" for reasoning (see Table 8), only the problem description itself is non-English.
  • IAR. This ratio utilizes average attention values ​​calculated per token. Although the number of image tokens (∣I∣|I|) remains constant across languages, the number of text tokens (∣T∣|T|) fluctuates based on tokenizer efficiency; consequently, the observed patterns (lowest for Chinese, highest for Hindi and Kazakh) may simply be an artifact of this normalization method.
  • Step-based metrics. These evaluations rely on GPT-5.4 as the judge and lack human verification.
  • Inconsistencies between text and tables. Certain claims contradict the data presented in the paper's tables. For example, lines 389–390 state that, across almost all models, the field of "Mathematics" experienced the largest drop from F to F$_{wst}$; however, according to Table 2, the decline in this field was not the greatest, and the counts for the various fields shown in Figure 5 do not match the dataset's actual content.
Questions for authors
  1. The benchmark scores are actually quite high for the models (GPT-5.4 88.8%, Qwen3.5-27B 82.3%), implying that they can solve most of the problems; this diminishes the significance of the benchmark itself.

  2. What is the worst-case acc when sampling the same instance for 10 times? And how does the result change if you use a 0 temperature?

  3. Does the 1% tolerance refer to relative error or absolute error? Additionally, how should cases where the ground truth is close to zero be handled? Would the main conclusions change if the tolerance set to 5%?

  4. What are the syntax parsing failure rates for the answers, broken down by model and language? This is crucial for understanding performance degradation in low-resource languages.

  5. Can the impact of language be observed even when using summed attention or controlling for the number of text tokens?

4 comments

We thank the reviewers for their constructive feedback and recognition of Sci-ρ\rho's contributions. Below are our point-by-point responses to all weaknesses and questions raised.

W1.1: Gap Not Disentangled from Sampling Noise

The decoding settings are in Table 7. To separate the gap from sampling noise, we ran an additional experiment on the English subset. We gave each model the same variant 10 times, at the temperature used in the paper, and took the worst case over those repeats instead of over 10 different variants. The 10-variant results are the paper's runs. We can clearly see that the gaps exist for both temp = 1.0 and temp = 0.0.

ModelAavg\mathcal{A}_{avg} (10 variants)Awst\mathcal{A}_{wst} (10 variants)GapAavg\mathcal{A}_{avg} (10 repeats)Awst\mathcal{A}_{wst} (10 repeats)Gap
GPT-5.4 (temp. 1.0)92.477.914.592.388.43.9
Qwen3.5-27B (temp. 1.0)86.659.726.986.867.819.0
Qwen3-VL-8B (temp. 0)69.532.736.868.766.72.0
InternVL3.5-8B (temp. 0)69.637.632.068.860.78.1

For every model, the worst case drops more across variants than across repeats. Qwen3-VL-8B, for example, loses 36.8 points across variants but only 2.0 across repeats. Our templates change only the sampled values between variants, so the variant gap measures the effect of varying the problem, as we argue in Section 4.2.

W1.2: Comparability of Difficulty Across Variants

All variants of a template share the same underlying reasoning: they are generated from the same problem structure and solved with the same sequence of solution steps, and only the sampled parameters (e.g., numerical values, shapes, colors, or function types) differ. Difficulty is therefore comparable across variants by construction.

W1.3: Strictness of the 1% Tolerance

Even with a 1% tolerance, exact-match grading can still mark a response as incorrect because of rounding or precision differences, and prior benchmarks that score only final answers (including DynaMath) share this limitation. This is precisely why Sci-ρ\rho includes an alternative grading scheme, namely step-level evaluation against gold solutions, which assesses the reasoning process independently of the precision of the final numerical answer. Under this evaluation, the gap between average-case and worst-case performance remains substantial (e.g., GPT-5.4's F\mathcal{F} drops from 88.0 to 67.8 in Mathematics), indicating that the robustness gap is not merely an artifact of strict numerical tolerance.

W1.4: Small Subsets Without Confidence Intervals

Biology is small because few IGCSE and A-Level Biology problems actually need an image (Section 3.1). We added 95% bootstrap confidence intervals, resampling over templates. For GPT-5.4, Biology Awst\mathcal{A}_{wst} is 47.8 [26.1, 69.6], still substantially lower than every other subject (for example, 24.5 points [3.1, 45.9] below Mathematics). Color's error rate is 37.0% [16.0, 60.0], higher than every other perturbation type. Both claims in the paper hold up with the intervals. And since each template is code, these subsets are easy to grow later.

W2.1: Limited Scope of the Multilingual Setting

We view this as a deliberate design choice rather than a weakness. The problem description is where the conditions, given quantities, and the question itself are stated, so a model must fully understand the non-English text to solve the problem. Keeping the images in English holds the visual input constant across all seven languages, so the language of the text is the only thing that varies. This setting is also realistic: non-English-speaking students routinely work with problems written in their native language alongside English-labeled diagrams from international textbooks and exam papers. Finally, we instruct models to reason in English so that their reasoning steps can be compared consistently against our gold solutions in the step-level evaluation. As reported, performance drops are driven by the text language despite identical images (e.g., InternVL3.5-4B gains 0.5 points in Chinese but drops 15.3 points in Swahili; Table 11, Figure 4).

W2.2: IAR Normalization Artifact

LanguageText tokensNormalized IARSummed IAR
Chinese43.50.1390.561
English45.80.1580.587
Arabic55.70.1690.564
Indonesian63.60.1770.549
Swahili85.90.1890.504
Kazakh114.50.2210.491
Hindi158.60.2350.438

We thank the reviewer for this insightful point. Following the suggestion, we recomputed the IAR using summed attention and compared it with the per-token normalized version (table above). Under per-token normalization, the IAR increases monotonically with the number of text tokens, from 0.139 for Chinese (43.5 tokens) to 0.235 for Hindi (158.6 tokens). Under summed attention, the ordering largely reverses, decreasing from 0.587 for English to 0.438 for Hindi. Both metrics thus follow the number of text tokens, in opposite directions. We therefore agree that tokenizer efficiency is a valid confounding factor. Since text-token counts vary across languages while image-token counts stay constant, the aggregation method can shift the IAR independently of the model's actual reliance on the image, and our results cannot disentangle language from this factor or from related confounders such as text length. We will therefore remove the claim that the IAR reflects language-dependent visual reliance from our main findings and report both metrics in Section 4.3 as an exploratory analysis. We will present this as a cautionary finding, since attention-based measures in multilingual VLMs are sensitive to tokenization, and suggest controlling for text length and interventional methods (e.g., image ablation) as future work. This revision does not affect our main contributions.

W2.3: No Human Verification of Step-Level Metrics

We disagree that the step-level evaluation lacks human input. All gold solutions were written by contributors and verified by domain experts (Section 3.1, Appendix A.2). The judge model strictly compares model steps to these human-authored standards (Tables 9 and 10) rather than independently determining correctness. Additionally, replacing the judge with Gemini 2.5 Pro (Appendix E) shows that the gap between F\mathcal{F} and Fwst\mathcal{F}_{wst} remains stable at 15.5--25.0 points across all subjects (Table 13).

Q1: Benchmark Difficulty / Saturation

We disagree. High Aavg\mathcal{A}_{avg} is where our argument starts. Symbolic benchmarks were built because static scores overstate what models can do. GSM-Symbolic (Mirzadeh et al., ICLR 2025) found that LLMs with high GSM8K scores lose accuracy when only the numbers or names in a question change, and DynaMath (Zou et al., 2025), FinChain (Xie et al., 2026), and EngTrace (Gull et al., 2026) test the same weakness in visual mathematics, finance, and engineering. Aavg\mathcal{A}_{avg} is the score a static benchmark would report. Awst\mathcal{A}_{wst} counts a template as solved only when the model gets all 10 variants right, and by this measure GPT-5.4 drops from 88.8 to 73.9 and no open-weight model reaches 52 (Table 1). The best model we tested misses at least one variant on a quarter of the templates, so the benchmark is not saturated. Each template can generate more variants, and Awst\mathcal{A}_{wst} can only fall as MM grows, so Sci-ρ\rho can be made harder without new annotation.

Q2: Repeated Sampling and Temprature 0

Answered in W1.1

Q3: Tolerance Definition and Sensitivity

ModelBefore: 1% (paper)After: 5%
GPT-5.488.8/73.991.1/78.4
Gemini-2.5-Pro85.9/62.788.8/69.3
Qwen3.5-27B82.3/51.284.7/55.1
Llama-4-Scout69.2/33.771.8/37.7

The 1% tolerance refers to relative error (that is, a prediction y^\hat{y} is accepted if ∣y^−y∣≤0.01∣y∣|\hat{y} - y| \le 0.01|y|). We do not apply special handling when the ground truth is zero, so such cases require an exact match. To examine sensitivity of tolerance, as the reviewer suggested we re-scored all responses with a 5% tolerance (table above). Relaxing the tolerance raises Aavg\mathcal{A}_{avg} by only 2.3–2.9 points and narrows the gap between Aavg\mathcal{A}_{avg} and Awst\mathcal{A}_{wst} only slightly across all four models. The ranking of models is also unchanged. Therefore, our main conclusion that VLMs show a substantial gap between average-case and worst-case accuracy holds under a looser tolerance.

Q4: Answer Parsing Failure Rates

We counted the responses with no extractable answer and will add the full table to the appendix. GPT-5.4 has almost none (at most 0.02% in any language), yet it still loses 14.9 points from Aavg\mathcal{A}_{avg} to Awst\mathcal{A}_{wst}. Both Gemini models and Llama-4-Scout also stay below 2.2% in every language. Rates are higher for some other models, mostly outside English, which matches the language degradation the paper reports.

Q5: Language Effect Under Summed Attention / Token-Count Control

Answered in W2.2

W2.4: Inconsistencies Between Text and Tables/Figures

Thanks for catching these. The tables are correct; the mistakes are just in the text, and we will fix them. Since all our findings are computed from the tables, none of the conclusions change. Lines 389--390 discuss which subject has the largest drop, but our main claim is that Fwst\mathcal{F}_{wst} stays below F\mathcal{F} for every subject and model, which is clearly shown in Table 2. For Figure 5, we reran the attention analysis on the final 606 templates and found the exact same ranking (Chinese lowest, Kazakh and Hindi highest).

Thank you for your response; I will take the other reviewer's comments into account before making my final decision.

Reviewer_ddTYOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces Sci-ρ, a multilingual symbolic benchmark for evaluating VLM robustness across five STEM subjects and seven languages. It consists of 606 executable templates generating multiple problem instances. Experiments on different VLMs reveal substantial gaps between average-case and worst-case accuracy, as well as performance degradation in lower-resource languages. Additional analyses examine reasoning consistency, attention patterns, and failure modes.

Critical strengths
  1. The benchmark extends dynamic robustness evaluation to multiple STEM disciplines and languages.
  2. Executable Python templates enable scalable problem generation with verified answers and reasoning steps.
  3. The paper evaluates different VLMs and provides additional analyses of reasoning consistency, multilingual robustness, and failure modes.
Critical weaknesses
  1. The benchmark construction, evaluation methodology, and overall analysis closely resemble DynaMath (Zou et al., ICLR 2025). Both works transform visually grounded problems into executable Python templates, generate multiple variants through numerical and visual perturbations, and evaluate reasoning robustness using average-case and worst-case accuracy. Even the perturbation categories and the use of ten variants per template largely overlap. More importantly, the central empirical finding—that VLMs exhibit substantial performance gaps between average-case and worst-case accuracy—has already been established by DynaMath.
  2. While Sci-ρ extends the evaluation to additional STEM subjects and languages, it remains unclear what fundamentally new methodological contribution or scientific insight this extension provides. The paper does not sufficiently distinguish its contributions or demonstrate that the expanded settings reveal qualitatively new robustness challenges beyond those already identified in prior works.
Questions for authors
  1. The multilingual benchmark translates only the textual descriptions while keeping the visual content in English. This introduces a potential confound between multilingual reasoning and cross-lingual alignment with English visual annotations. Consequently, the observed performance differences across languages cannot be attributed solely to language-dependent reasoning capabilities. The paper would benefit from fully localized visual inputs or controlled experiments that disentangle language understanding from cross-lingual visual grounding.
  2. The additional step-level and attention analyses do not convincingly explain the underlying causes of reasoning failures. The step-level evaluation relies on GPT-5.4 as an LLM judge without sufficient validation against human annotations. Meanwhile, the attention analysis is limited to a single model and interprets differences in attention allocation as changes in visual reliance without causal evidence.
4 comments

We thank the reviewers for their constructive feedback and recognition of Sci-ρ\rho's contributions. Below are our point-by-point responses to all weaknesses and questions raised.

W1 & W2: Relation to DynaMath and Novelty of the Extension

We thank the reviewer for the careful comparison. As W1 and W2 are closely related, we address them together.

The main contribution of this work is not a novel dataset construction method, but the dataset itself. We also note that DynaMath is not the first symbolic benchmark. In fact, the fact that Sci-ρ\rho follows a similar construction paradigm reinforces the validity of our approach and suggests that we are employing established best practices for building symbolic evaluation datasets. We appreciate the reviewer for recognizing this.

We disagree that this makes our contribution merely incremental. DynaMath established that VLMs are fragile when solving English mathematics problems. Whether this fragility extends to other STEM disciplines, across different languages, and at the level of intermediate reasoning steps has remained an open question. To the best of our knowledge, Sci-ρ\rho is the first symbolic, visually grounded benchmark specifically designed to investigate these dimensions.

Compared to DynaMath, which covers nine mathematical topics in English and scores only final answers, Sci-ρ\rho spans five STEM subjects (Mathematics, Physics, Chemistry, Biology, and Computer Science) across 113 curriculum-aligned topics from Cambridge IGCSE and A-Level, and covers seven typologically diverse languages (English, Arabic, Chinese, Hindi, Indonesian, Kazakh, and Swahili), including low-resource ones. Every template is also paired with a gold step-by-step solution, which enables step-level evaluation (precision, recall, F\mathcal{F}, and worst-case Fwst\mathcal{F}_{wst}).

These extensions reveal findings that DynaMath cannot reveal. First, language is an independent axis of robustness: frontier models (GPT-5.4, Gemini 2.5) barely degrade across languages, while open models drop substantially in lower-resource languages, e.g., InternVL3.5-14B falls from 60.0 in English to 43.5 in Swahili (Table 11, Figure 4). Second, perturbations break the reasoning process, not only the final answer: GPT-5.4's step-level F\mathcal{F} drops from 88.0 to 67.8 in Mathematics under worst-case evaluation, and this gap holds at 15.5--25.0 points under an independent Gemini 2.5 Pro judge (Table 2, Appendix E). We will make these distinctions more explicit in Sections 1 and 2 of the revised version.

Q1: Confound from English-Only Visual Content

Many diagrams in Sci-ρ\rho are language-agnostic (e.g., function graphs, geometric shapes, circuits, and molecular structures labeled only with symbols and numbers), so localizing the images would have minimal impact and in some cases is not applicable at all. We therefore expect re-rendering the diagrams in localized languages not to change the overall trend. Moreover, since the images are identical across all seven languages, the visual input is held constant and only the language of the text varies. Under this setup, higher-resource languages show no notable degradation, whereas smaller models degrade noticeably in lower-resource languages (Table 11, Figure 4). This suggests that the language of the problem plays a meaningful role in the performance differences we observe. Such evaluation setting is still realistic, where non-English speakers work with English-labeled figures from international textbooks and exam papers.

Q2.1: Step-Level Evaluation Lacks Human Validation

We disagree that the step-level evaluation lacks human input. All gold solutions were written by contributors and verified by domain experts (Section 3.1, Appendix A.2). The judge model strictly compares model steps to these human-authored standards (Tables 9 and 10) rather than independently determining correctness. Additionally, replacing the judge with Gemini 2.5 Pro (Appendix E) shows that the gap between F\mathcal{F} and Fwst\mathcal{F}_{wst} remains stable at 15.5--25.0 points across all subjects (Table 13).

Q2.2: Attention Analysis: Single Model and No Causal Evidence

In response to a related concern from Reviewer mTCv (W2.2), we recomputed the IAR using summed attention instead of the per-token average. Under per-token normalization, the IAR increases monotonically with the number of text tokens, from 0.139 for Chinese to 0.235 for Hindi, whereas under summed attention the ordering largely reverses, decreasing from 0.587 for English to 0.438 for Hindi. Since both metrics follow the number of text tokens in opposite directions, our results cannot disentangle the effect of language from confounding factors such as tokenizer efficiency and text length, and therefore cannot support claims about visual reliance. We will remove the claim that the IAR reflects language-dependent visual reliance from our main findings and report both metrics in Section 4.3 as an exploratory analysis of attention allocation. We will present this as a cautionary finding, since attention-based measures in multilingual VLMs are sensitive to tokenization, and suggest extending the analysis to more models, controlling for text length, and using interventional methods (e.g., image ablation or attention masking) as future work. This revision does not affect our main contributions.

Thank you for the clarification. My main concern is whether the expanded settings yield sufficiently informative scientific insights beyond prior work. Lower performance in low-resource languages is expected (simply due to a lack of data), and the step-level degradation is not sufficiently analyzed to explain what it reveals beyond final-answer degradation. For Q1, the expectation that localization would not change the trend requires supporting evidence, although I consider this a minor issue. Overall, my concerns about novelty and experimental analysis remain, so I will keep my score.

Thank you for the follow-up. To further show that the step-level results reveal insights that final-answer accuracy cannot, we ran a new analysis. We used the existing outputs of five models (GPT-5.4, Qwen3-VL-8B, InternVL3.5-8B, Molmo2-8B and Gemma-3-4B) on the English set, which has 606 templates with 10 variants each. The step judge (Section 4.2, Appendix E) reads the human-written gold solution and checks, step by step, whether each step appears in the model's answer.

The first thing we found is that a correct answer does not always come with complete reasoning. For each model, 16.7–30.5% of its correct answers miss at least one gold step, from 16.7% for GPT-5.4 to 30.5% for Gemma-3-4B. Both Aavg\mathcal{A}_{avg} and Awst\mathcal{A}_{wst} count these answers as solved, so a final-answer score treats them the same as full solutions.

Next, we looked at the step where wrong answers start to go wrong. For each of the 9,565 wrong answers, we took the first gold step the model missed and sorted it by type. We consider five types: (1) reading the diagram, values, or question, (2) interpreting the problem and applying relevant knowledge, (3) applying the correct formula, (4) performing the calculation, and (5) rounding and giving the final answer. We used GPT-5 as a judge for this. The table below shows, for each model, how its wrong answers are spread across these types. For four of the five models, the most common first mistake is reading a value from the figure, table or question, and for Gemma-3-4B it is the second most common. Reading and subject knowledge together make up 43–52% of each model's wrong answers, which fits the error analysis in Section 4.4. A final-answer score would only tell us that these answers are wrong. It would not tell us where the mistake began.

ModelReadingKnowledgeFormula/unitsCalculationRounding/answer
GPT-5.439.612.716.926.34.4
Qwen3-VL-8B35.813.219.528.52.8
InternVL3.5-8B34.514.216.231.14.1
Molmo2-8B32.714.720.229.43.0
Gemma-3-4B28.814.322.031.93.0

Finally, we checked whether a model fails the variants of one problem at the same place. We took the templates where a model solved some variants and failed at least two, and where the gold solution has the same number of steps in every variant. For each of these templates, we identified the step at which most of the failed variants first went wrong. For example, if three of four failed variants first go wrong at step 2, step 2 is that template's most common failure step, and the template contributes three of its four failed variants to the count. For every model, 75–91% of failed variants first went wrong at their template's most common failure step. In other words, a model tends to break at the same step again and again. This again gives a step-by-step picture of the robustness gap reported in the paper, which a final-answer score cannot give.

We will include these reports in the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.