acceptodds
Under review as a conference paper at ICLR 2027

From Seeing to Solving: Benchmarking and Diagnosing Symbolic Reasoning in Large Vision-Language Models

15 views

Abstract

Large vision-language models (LVLMs) remain unreliable on symbolic reasoning tasks, yet final-answer accuracy alone provides limited insight into whether failures arise from incorrect problem recovery or downstream symbolic solving. We curate the Multimodal Constraint Satisfaction Problem Benchmark (MMCSP-Bench), which provides matched structured-text, standard-image, and out-of-distribution (OOD) image views of Sudoku, Nonogram, and Graph Coloring, with all candidate solutions verified against the original constraints. We further propose the Declarative Constraint Satisfaction Problem Interface (DeCSP), a controlled symbolic interface that replaces unconstrained solver-program generation with predefined constraint-operator selection and deterministic execution. Across seven LVLMs, DeCSP achieves 96.24% mean valid-solution accuracy on structured text, compared with 8.19% for direct answering and 43.43% for PAL. However, DeCSP accuracy decreases to 42.57% on standard images and 28.19% on OOD images, revealing substantial challenges in multimodal symbolic reasoning. Controlled evaluations with fixed solvers, correct structures, and shared transcriptions disentangle visual-recovery failures from symbolic-generation failures, showing that recovery quality and final validity measure distinct dimensions of reasoning. Our stage-targeted training further shows that recovery improvements do not consistently transfer to direct answering, while program-training effects vary across models. Code and data are available at https://anonymous.4open.science/r/Seeing2Solving-37532.

Primary area:
datasets and benchmarks
Keywords:
vision-language models, symbolic reasoning, multimodal benchmarks, diagnostic evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_15jpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MMCSP-Bench, a benchmark containing matched text, standard-image, and rendering-shifted versions of 300 synthetic constraint satisfaction problems. The tasks cover Sudoku, Nonogram, and Graph Coloring. It also proposes DeCSP, in which a vision-language model selects from seven predefined operators and a deterministic backend constructs and solves the resulting CSP.

Across seven models, DeCSP achieves 96.24% mean validity on text inputs, decreasing to 42.57% on standard images and 28.19% on rendering-shifted images. The paper uses matched views, shared transcriptions, exact-recovery analysis, and stage-specific training to separate perception, interface, execution, and verification failures. This diagnostic framework is useful. However, the main DeCSP task appears closer to task routing and output-format compliance than general symbolic formulation.

Critical strengths
  1. Verifying candidate solutions against the original problem is a sound design choice. It avoids giving credit to a model that correctly solves an incorrectly transcribed surrogate problem.

  2. The matched text and image views make it possible to distinguish visual recovery errors from downstream symbolic or interface failures. The exact-recovery analysis shows clearly that correct transcription and valid solving are not equivalent.

Critical weaknesses
  1. DeCSP receives the task identity and selects from only seven operators that are nearly partitioned by task. Canonical task-correct insertion reaches 100%, suggesting that the learned component performs constrained routing rather than general symbolic formulation.

  2. The results are highly sensitive to interface syntax. Normalizing one Nonogram operator alias converts 79 failures into valid solutions, so a notable portion of the measured error reflects naming conventions rather than perception or reasoning.

  3. The benchmark covers three small synthetic CSP families, and the OOD variants mainly modify fonts, line positions, strokes, or node rendering. The experiments support conclusions about renderer sensitivity, but not broad visual or symbolic generalization.

Questions for authors
  1. Given that task identity nearly determines the valid operators, what part of DeCSP should be viewed as learned symbolic formulation?
  2. How many failures are caused only by operator naming or formatting?
  3. Is the OOD claim intended to cover renderer changes only, rather than changes in problem structure?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 15jp #1

We thank the reviewer for recognizing the value of checking solutions against the original problem, matched text and image problems, and checking whether the transcribed problem is correct.

DeCSP deliberately reduces the VLM’s symbolic solving burden to help distinguish errors in reading and recording the puzzle from errors in selecting constraints or solving it.

  • Matched views: Text → Standard image → OOD image. The puzzle stays the same. We test how presenting it as an image, or changing its rendering, affects solving.
  • Solving axis: Base → PAL → DeCSP. The VLM generates an answer, writes a solver program or selects predefined constraint operators, respectively. Comparing these routes with correct puzzle data separates failures in these steps from image-transcription errors. DeCSP’s backend constructs the constraints and searches for a solution (Sections 4.1–4.2).

We address each question and concern below.

Question 1

Given that task identity nearly determines the valid operators, what part of DeCSP should be viewed as learned symbolic formulation?

Answer:

The learned step is choosing which predefined constraint operators match the stated rules; the backend then constructs constraints and searches for a solution. On the added 30 problems with varying rules, DeCSP solves 60.00% with VLM-selected operators, compared with 0.00% when operators are selected from the task name alone.

In the main benchmark, each task has fixed rules, so the task name is enough to select the required operators. Appendix E.1 tests whether VLMs select the correct operators from reworded rules. Seven VLMs each receive 132 rule descriptions; the table reports how often they select exactly the required operators:

TestDeCSPFixed task-name rule
Unseen wording63.42%13.64%

These 30 new text-only variants are separate from the main benchmark’s 100 Sudoku problems and their views. They retain 9×9 grids, digits 1–9 and the label “Sudoku,” but require only two or three of four rules: preserve printed digits, and avoid repeated digits within rows, columns or boxes. Each is solvable under its stated rules but not all standard Sudoku rules; the tested combinations are absent from the prompt’s single-operator demonstrations. The table reports validity using the same backend, with the last row supplying correct operators directly instead of using VLM selections.

MethodQwen3.5-9BQwen3.5-4B
Fixed task-name rule0.00%0.00%
DeCSP60.00%60.00%
Correct operators supplied100.00%100.00%

Every successful DeCSP case uses exactly the required operators.

Question 2

How many failures are caused only by operator naming or formatting?

Answer:

Correcting the operator name alone resolves 79 failures with correct puzzle data, 66 with standard images and 41 with OOD images. Appendix B.2 of our original submission reports the correct-data result. We now apply the same correction to saved image responses, changing only col_runs to column_runs; the transcribed puzzle data, backend and execution settings stay unchanged.

Each row contains 2,100 evaluations (seven VLMs × 300 problems). Before correction, the backend rejects this operator name in 3.76% of correct-structure evaluations, 3.19% of standard-image evaluations and 2.14% of OOD-image evaluations. The table shows valid-solution rates:

InputBeforeAfterChange (pp)
Correct structures96.24%100.00%+3.76
Standard images42.57%45.71%+3.14
OOD images28.19%30.14%+1.95

This mismatch accounts for all recorded format rejections at the operator-selection stage; five image cases still fail because their transcribed puzzles or returned solutions are incorrect.

Another 542 standard-image and 781 OOD-image transcriptions cannot be parsed as JSON; these failures cannot be attributed solely to naming or formatting without establishing whether the puzzle was transcribed correctly.

Question 3

Is the OOD claim intended to cover renderer changes only, rather than changes in problem structure?

Answer:

Yes. Our original OOD design changes how the puzzle is drawn, while keeping the puzzle and its rules fixed (Section 3.2; Appendix A.1). Appendix A.1 states: “Across all three tasks, the OOD view changes only rendering attributes; the source problem, public structure, and original-problem verifier remain unchanged.” We compare the same VLM and solving method before and after changes to fonts, strokes or node appearance.

AuthorsOpenReview ↗

Response to Reviewer 15jp #2

Concern 1

DeCSP receives the task identity and selects from only seven operators that are nearly partitioned by task. Canonical task-correct insertion reaches 100%, suggesting that the learned component performs constrained routing rather than general symbolic formulation.

Answer:

The restricted operator interface is intentional: it reduces the VLM’s symbolic solving burden so that we can examine errors in reading and transcribing the image. Section 4 of our original submission specifies that DeCSP “restricts the model output to selecting predefined constraint operators.”

For this control, we supply the correct operators ourselves and run DeCSP’s backend on either correct puzzle data or saved VLM transcriptions. The correct-data result comes from the original Table 4; the two image conditions are added controls. Each row covers seven VLMs × 300 problems (N = 2,100), including transcriptions that cannot be parsed.

Correcting only the operator name retains one original timeout and yields 45.71% on standard images; supplying correct operators in the control below yields 45.76%.

InputValidityChange from correct input (pp)
Correct structures100.00%0.00
Standard transcriptions45.76%−54.24
OOD transcriptions30.14%−69.86

With correct operators supplied, the low image scores show that producing usable, correct puzzle data from images remains a major bottleneck. Among OOD transcriptions that can be parsed, validity is 100.00% for the 551 that reproduce the puzzle exactly and 10.68% for the 768 that do not.

Concern 2

The results are highly sensitive to interface syntax. Normalizing one Nonogram operator alias converts 79 failures into valid solutions, so a notable portion of the measured error reflects naming conventions rather than perception or reasoning.

Answer:

Correcting the operator name eliminates all failures with correct puzzle data, but scores on image inputs remain low.

Appendix B.2 reports: “This replay yields 79 failure-to-valid transitions and provides a direct measurement of interface-name sensitivity.” We now apply the same col_runs → column_runs correction to saved image responses, keeping the transcribed puzzle data, backend and execution settings unchanged. Across 2,100 evaluations per input condition, validity rises from 96.24% to 100.00% with correct structures, 42.57% to 45.71% with standard images and 28.19% to 30.14% with OOD images.

Our paper also reports 94 valid standard-image solutions from transcriptions that do not exactly match the original puzzle (Section 6.1; Appendix D.2). We therefore check the transcribed puzzle and the final answer separately, and record name mismatches as interface errors.

Concern 3

The benchmark covers three small synthetic CSP families, and the OOD variants mainly modify fonts, line positions, strokes, or node rendering. The experiments support conclusions about renderer sensitivity, but not broad visual or symbolic generalization.

Answer:

Our OOD experiment deliberately changes only how the same puzzle is drawn. Section 3.2 states: “OOD is restricted to presentation changes and does not alter task rules or problem structures”. Because the puzzle and its rules stay fixed, a performance drop reflects sensitivity to its visual presentation, not increased difficulty of the symbolic problem. Our paper reports the following DeCSP results across seven VLMs and the three task families, with 300 problems per VLM and view (N = 2,100):

Standard imagesOOD imagesChange (pp)
42.57%28.19%−14.38

Turning graph images into usable, correct descriptions also remains a bottleneck on larger graphs. We added 100 graphs with 9–12 nodes, compared with 6–8 nodes in the main benchmark, and evaluated five VLMs. Each receives either the correct graph data or a standard image. DeCSP uses VLM-selected operators; Known-Program uses a prewritten, task-correct solver. Each method obtains its own transcriptions of the same images. Tables report validity on the same 100 problems; changes are image minus correct-structure scores:

DeCSP.

VLMCorrect structuresStandard imagesChange (pp)
Qwen3.5-4B100.00%0.00%−100.00
Qwen3.5-9B100.00%0.00%−100.00
Qwen3-VL-32B100.00%0.00%−100.00
Qwen3.8-27B100.00%2.00%−98.00
Mistral-3.2-24B100.00%0.00%−100.00

Known-Program.

VLMCorrect structuresStandard imagesChange (pp)
Qwen3.5-4B100.00%1.00%−99.00
Qwen3.5-9B100.00%0.00%−100.00
Qwen3-VL-32B100.00%1.00%−99.00
Qwen3.8-27B100.00%3.00%−97.00
Mistral-3.2-24B100.00%0.00%−100.00
AuthorsOpenReview ↗

Follow-up to Reviewer 15jp

Dear Reviewer 15jp,

Thank you again for your thoughtful feedback. Our responses clarify DeCSP’s diagnostic role and the scope of renderer-OOD evaluation, and include additional operator-name correction results and rule-combination tests.

As the discussion period draws to a close, we would appreciate any further feedback you may have. Please let us know if any points remain unclear or require further discussion; we would be happy to clarify them.

Best regards, The Authors

Reviewer_ticoOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper introduces MMCSP-Bench to distinguish visual problem-recovery failures from downstream symbolic-solving failures.

    • It contains 300 source problems across Sudoku, Nonogram, and Graph Coloring. Each problem has structured-text, standard-image, and OOD-image views, resulting in 900 evaluation inputs. Candidate solutions are verified against the original constraints rather than the model’s recovered problem.
    • DeCSP restricts model output to predefined constraint operators. A deterministic backend constructs the constraints and searches for a solution.
  • The evaluation combines controlled diagnosis with stage-targeted training.

    • Fixed-solver, correct-structure, and shared-transcription comparisons examine different parts of the pipeline. The paper also evaluates whether improvements to visual recovery and program generation transfer to final solving performance.
Critical strengths
  • The controlled evaluation provides a useful alternative to final-answer accuracy alone.

    • The matched views preserve the original problem, and the fixed-solver, correct-structure, and shared-transcription controls reduce ambiguity about which component changes between comparisons. These are substantive controls rather than comparisons between unrelated tasks.
    • The joint analysis of recovery and validity is particularly useful. A valid answer can follow an incorrect reconstruction, while an exact reconstruction can still be followed by failure. Evaluating these outcomes separately avoids treating final correctness as evidence of faithful problem understanding.
  • The intervention studies test whether the diagnosis leads to useful improvements.

    • Visual repair improves exact recovery by 35.74 percentage points and DeCSP accuracy by 35.77 points on average. This shows why improvements to an intermediate component should be evaluated in the final solving route.
    • The paper also reports negative and model-dependent results. For example, joint visual and program repair improves average PAL accuracy but degrades Gemma4. These results are informative, although their interpretation depends on the caveats in Weakness 4.
  • (minor) good figures

Critical weaknesses
  1. The additional diagnostic value over existing benchmarks is not sufficiently established.

    • Separating perception from reasoning is already a central concern in prior work, as the paper acknowledges. VGRP-Bench (Ren et al., 2025) also evaluates perception, rule-following, and solving at multiple granularities. The paper should explain what new, reproducible diagnosis MMCSP-Bench provides beyond these evaluations.
    • I do not expect a benchmark to contribute a larger dataset, a new metric, and a new capability simultaneously. The strongest candidate here is the diagnostic protocol, so a focused comparison should show which failure mechanisms it reveals that existing analyses would miss.
  2. The main DeCSP results may not measure symbolic reasoning.

    • In the released prompt (code/mmcsp/prompts.py), DeCSP receives the task name, and each operator description names its task, for example "enforce every Nonogram column run clue". The prompt also instructs the model to exclude operators from other tasks. On the main benchmark, operator selection therefore reduces to matching the task name, and Appendix E.1 itself treats a deterministic task router as a fixed-task baseline.
    • Appendix B.2 reports that normalizing col_runs to column_runs converts 79 failures into valid solutions. This equals all correct-structure DeCSP failures implied by Table 3, so DeCSP would reach 100% on structured text after alias handling. The comparison of 96.24% with 43.43% and 8.19% therefore reflects how much work is delegated to the backend.
    • On images, DeCSP accuracy (42.57%) is close to the Known-Program control (46.10%), so it mainly measures transcription quality. The unseen-wording study in Appendix E is the only setting where operator selection is non-trivial (63.42% exact recovery), and it deserves a more central role.
  3. The PAL and Base baselines face restrictions that the main text does not disclose.

    • The released PAL audit (methods/pal.py) rejects imports, try, lambda, class, yield, and every method call except append, so common calls such as .get, .items, and .copy are rejected. Programs also run under a 2-second CPU limit, while the prompt only tells the model not to import modules.
    • Program or interface checks account for 25.29% of PAL first failures on standard images, compared with 3.19% for DeCSP (Table 16). PAL solves only 1.00% of structured-text Nonograms on 5×5 and 6×6 grids and produces 176 truncated programs under the 1,024-token limit. Base is told to return only <answer>JSON</answer>, which leaves no room for explicit reasoning.
    • The gap between DeCSP and these baselines is therefore partly an artifact of their constraints. The central comparisons should be reassessed with the audit rules disclosed to the model, standard Python features, and adequate budgets.
  4. The intervention results rest on a different instance set and in-distribution training.

    • Appendix C.2 states that the visual-repair transfer uses Graph Coloring instances graph-s20260813-* rather than the primary benchmark, and the released data contains only gcv2-*-s20260831 instances. Its Base baselines differ sharply from the main results. Qwen3.5-9B reaches 17.67% versus 1.00% and 1.67% in Table 3, and its −17.50-point change drives most of the −3.20 mean. The claim that recovery gains do not transfer to direct answering therefore rests on an unreleased, non-primary set.
    • Program adapters are trained on solver programs for the same three tasks, so 100% accuracy for Qwen3-VL-32B is largely expected. The renderers used for visual training are not specified, which leaves unclear whether the OOD view remains out of distribution.
    • In Table 13, six of seven models obtain identical accuracy across three "independently trained" program adapters (SD 0.00). The authors should clarify how the runs differ, since the reported intervals may otherwise understate variability.
  5. The task coverage and model evaluation leave the scope of the findings unclear.

    • The benchmark contains only 300 source problems across three task families, and the 900 inputs are matched views rather than distinct puzzles. Its coverage is narrower than VGRP-Bench's 20 puzzle types, and the OOD setting changes presentation rather than task rules. The paper should show that its diagnoses remain stable across freshly generated instances and difficulty levels.
    • The primary panel consists of models in the 4B to 32B range, with a 1,024-token limit per call and thinking disabled for several Qwen configurations. It does not establish whether representative strong GPT, Gemini, or Claude models still struggle under adequate reasoning and output budgets.
    • A focus on smaller, locally deployable models could be reasonable, but that target population and its resource constraints should be explicit. Parameter count alone does not establish an on-device setting.
Questions for authors
  • Additional value over prior benchmarks

    • What specific, reproducible diagnosis does MMCSP-Bench enable beyond VGRP-Bench, CrossWordBench, and earlier frameworks that separate perception from reasoning?
    • Could the authors demonstrate a finding that changes how a model or intervention should be assessed? A new metric or a larger collection is not required.
  • What DeCSP and the baselines measure

    • How does DeCSP compare with a deterministic task router on the main benchmark, and does the alias replay bring correct-structure DeCSP to 100%?
    • How do PAL results change when the audit rules are stated in the prompt, standard Python features are allowed, and the token budget is sufficient? How does Base perform when explicit reasoning is permitted before the final answer?
    • Which central findings remain once alias errors, schema violations, audit rejections, and truncation are separated from semantic errors?
  • Intervention studies

    • Why does the visual-repair transfer use graph-s20260813-* rather than the primary Graph Coloring set, and will this set be released? Do the Base transfer results hold on the primary instances?
    • Which renderers were used for visual training? How do the three program-adapter runs differ in seed and data?
  • Generality and intended model population

    • Do the main diagnostic patterns reproduce on additional generated problems, across difficulty levels, and with representative strong reasoning-enabled models from the GPT, Gemini, or Claude families? A prespecified subset could suffice if full evaluation is too costly.
    • If stronger models solve most instances reliably, is the intended contribution a diagnostic benchmark for smaller models under constrained budgets? Please clarify that scope and its practical motivation.

  • The matched-input controls and original-problem verification are valuable. However, the main DeCSP numbers largely reflect task-name matching and transcription, the baselines face undisclosed restrictions, and the transfer conclusions rest on a non-primary set. The paper therefore does not yet establish what the benchmark adds or whether its findings concern semantic abilities.
  • I would consider raising my score if the authors show a clear incremental diagnostic contribution, evaluate baselines under disclosed and adequate settings, separate interface artifacts from semantic failures, and confirm the intervention results on the primary benchmark. The benchmark need not defeat every frontier model, but its intended model population and evaluation purpose must be justified.
9 comments
AuthorsOpenReview ↗

Response to Reviewer tico #1

We thank the reviewer for recognizing the value of our controlled evaluation and our analysis of both how accurately VLMs read problems from images and whether their answers satisfy the original constraints. our contribution connects controlled diagnosis with targeted training and reevaluation: identify which component fails, then test whether improving it helps final solving. MMCSP-Bench provides the matched problems for this pipeline.

Evaluation design (Figure 1; Section 5.2).

  • Input axis: Text → Standard image → OOD image. The underlying CSP stays fixed. We test whether the VLM correctly reads the digits, clues or graph connections from the image, and how rendering changes affect this.
  • Solving axis: Base → PAL → DeCSP. With the problem information fixed, the VLM writes an answer, generates a complete solver program or selects predefined constraint operators, respectively. DeCSP deliberately reduces the VLM’s symbolic solving burden: its backend constructs constraints from the extracted problem information and selected operators, then searches for a solution.

These controls guide separate training for transcribing images into JSON and generating solver programs; we then test whether the training improves final answers. We address each question and concern below.

Question 1 — Additional value over prior benchmarks

Question 1.1

What specific, reproducible diagnosis does MMCSP-Bench enable beyond VGRP-Bench, CrossWordBench, and earlier frameworks that separate perception from reasoning?

Answer:

Our diagnostic protocol reveals model-specific differences that aggregate solving accuracy alone would obscure. VGRP-Bench evaluates perception, rule following, and end-to-end solving, including fine-tuning, while CrossWordBench examines clue answering, crossing-letter consistency, and grid parsing. Complementing these evaluations, we provide identical image-derived problem descriptions to DeCSP and CDT and use the same solver backend, isolating the effect of the symbolic interface.

A concrete example is the Qwen3.5-9B comparison detailed in our answer to Question 1.2: a controlled operator-name correction removes the entire 17.50-point CDT–DeCSP gap. This identifies an effective interface-level intervention while keeping the image transcriptions and solver backend fixed. Our contribution is this controlled attribution of observed failures to actionable components, alongside separate measurements of transcription fidelity and original-problem validity.

Question 1.2

Could the authors demonstrate a finding that changes how a model or intervention should be assessed? A new metric or a larger collection is not required.

Answer:

Our protocol identifies a targeted intervention that removes an apparent advantage of constraint generation. We compare DeCSP and CDT using identical image transcriptions and the same solver backend. On 600 Qwen3.5-9B cases, CDT achieves 64.00% validity versus 46.50% for DeCSP. Correcting only col_runs to column_runs recovers 105 DeCSP failures—exactly all cases in which CDT succeeds and DeCSP fails—bringing DeCSP to the same 64.00%.

This diagnosis changes the intervention choice: correcting interface binding is sufficient to close the observed gap, without retraining the model or changing the supplied transcriptions. The paired comparison and controlled replay identify both the failure source and an effective remedy.

AuthorsOpenReview ↗

Response to Reviewer tico #2

Question 2 — What DeCSP and the baselines measure

Question 2.1

How does DeCSP compare with a deterministic task router on the main benchmark, and does the alias replay bring correct-structure DeCSP to 100%?

Answer:

With correct structures in the main benchmark, the task-name baseline and name-corrected DeCSP both reach 100.00%. DeCSP selects predefined operators; the baseline selects the task’s fixed operator set. Both use the same backend.

The fixed baseline evaluates all 300 source problems. DeCSP covers seven VLMs × 300 sources, or 2,100 evaluations per condition:

Method on correct structuresValidity
Task-name-only baseline100.00%
DeCSP, original96.24%
DeCSP, after name correction100.00%

Changing only col_runs to column_runs resolves all 79 correct-structure failures (Appendix B.2). We extend this correction to image responses, with 2,100 evaluations per condition:

Name-rejection rates before repair: Correct structures: 3.76%; Standard images: 3.19%; OOD images: 2.14%.

InputBeforeAfterChange (pp)
Correct structures96.24%100.00%+3.76
Standard images42.57%45.71%+3.14
OOD images28.19%30.14%+1.95

The much lower image scores after correction show that naming alone does not account for image failures.

Changed rules provide a separate test of selection beyond task-name routing. We constructed 30 new text Sudoku variants, separate from the primary instances, with the same task label but different combinations of known rules. Each variant is solvable under its stated rules but unsolvable if all standard Sudoku rules are imposed. The fixed task-name baseline solves 0.00%, while VLM-selected DeCSP solves 60.00% for both Qwen3.5-9B and Qwen3.5-4B.

Question 2.2

How do PAL results change when the audit rules are stated in the prompt, standard Python features are allowed, and the token budget is sufficient? How does Base perform when explicit reasoning is permitted before the final answer?

Answer:

Broader Python support raises PAL by 13.33 percentage points, and allowing Base to write reasoning raises it by 34.44 points at the same 8,192-token cap. DeCSP retains an 8.89-point accuracy advantage in this Qwen control; in the separate GPT-5.5 control, PAL matches its accuracy and DeCSP uses 70.28% fewer API tokens.

Qwen3.5-9B: PAL controls. All settings use the same 90 text problems (30 per task), with thinking off. Executor comparisons reuse generated programs. Expanded controls raise CPU/total runtime limits from 2/3 to 20/30 seconds. Output means include failures but exclude input tokens and backend computation.

PAL labels give prompt/executor pairs: Orig. = original; Discl. = original Python rules disclosed; Strict = original Python rules; Broad = broader Python support.

Method (prompt/executor)Cap (tokens)ValidityMean output tokens
PAL (Orig./Orig.)1,02452.22%432.24
PAL (Discl./Strict)8,19252.22%1,867.11
PAL (Discl./Broad)8,19265.56%1,867.11
PAL (Discl./Broad)32,76865.56%5,690.04
DeCSP32,76874.44%33.33

Increasing the original PAL cap to 4,096 or 8,192 tokens alone leaves validity at 52.22%.

Base: written reasoning. On the same Qwen problems, allowing an explanation before the final answer improves validity at a fixed cap; built-in thinking remains off.

Prompt (output cap)ValidityMean output tokens
Answer only (8,192)8.89%119.00
Reasoning allowed (8,192)43.33%5,716.94
Reasoning allowed (16,384)56.67%9,253.61

At 32,768 tokens, Base validity remains 56.67%.

Thinking controls. Qwen exhausts both 16,384- and 32,768-token thinking caps without producing a program. An 8,192-token program call using the first 4,096 tokens of an existing 32,768-token thought response reaches 58.89% (strict-rule prompt) and 57.78% (broader-rule prompt). Full sweeps, continuation settings and token accounting appear in the supplementary budget tables.

GPT-5.5: matched accuracy with fewer API tokens. This separate test uses 30 source problems in three views, high reasoning effort and a 32,768-token cap. Replaying the same PAL programs with the AST-node limit raised from 1,000 to 4,096 closes the accuracy gap; the original 2-second CPU and 3-second elapsed-time limits are retained.

Method (AST node cap)ValidityAPI tokens / case
PAL (1,000)72.22%4,913.78
PAL (4,096)98.89%4,913.78
DeCSP98.89%1,460.19

At the same 98.89% validity, DeCSP uses 70.28% fewer API tokens. These counts include input and output from all calls, including thinking, transcription and unsuccessful responses; backend computation is excluded. AST replay adds no model tokens.

AuthorsOpenReview ↗

Response to Reviewer tico #3

Question 2.3

Which central findings remain once alias errors, schema violations, audit rejections, and truncation are separated from semantic errors?

Answer:

Two findings remain: the evaluated open-weight VLMs often produce incorrect or unusable image transcriptions, and a transcription need not be exact to yield a valid solution. In the control with author-supplied operators (Canonical), we provide the correct operators and run the same backend on correct problem information or saved image transcriptions. Each row covers 2,100 evaluations, including unparseable responses:

InputValidityChange from correct input (pp)
correct structures100.00%0.00
standard transcriptions45.76%−54.24
OOD transcriptions30.14%−69.86

This gap is not explained solely by unparseable output. Among the 1,319 parseable OOD transcriptions from the same evaluation, 551 exactly match the original structures and 768 do not. Each row below uses its own subgroup as the denominator:

OOD transcriptionValidity
Matches the original structure100.00%
Parseable, but not an exact match10.68%

Conversely, 94 valid standard-image answers come from non-exact transcriptions (Section 6.1; Appendix D.2).

An incorrect operator name can prevent solving even when the problem has been read correctly. The 79 correct-structure failures of our operator-selection method, DeCSP, become valid solutions after correcting only the operator alias.

Baseline restrictions explain part of the method gap. On 90 Qwen3.5-9B text problems (30 per task), with the disclosed-rule prompt, broader Python support raises PAL from 52.22% to 65.56%; allowing Base to write reasoning raises it from 8.89% to 43.33% at the same 8K cap. DeCSP reaches 74.44%, leaving an 8.89-point gap to PAL. These controls show how the results change when interface restrictions and token budgets change. A remaining invalid answer, by itself, still does not show that the VLM misunderstood the constraints.

Question 3 — Intervention studies

Question 3.1

Why does the visual-repair transfer use graph-s20260813-* rather than the primary Graph Coloring set, and will this set be released? Do the Base transfer results hold on the primary instances?

Answer:

DeCSP’s gains persist on all 100 primary Graph Coloring v2 problems; Base changes are model-dependent. The earlier transfer set is prepared for release.

Appendix C.2 describes the separate transfer set and paired before/after evaluation: “The transfer baseline is evaluated on these records, rather than imported from the primary benchmark.” That set used the earlier graph generator. Its 300 source problems, 600 matched images and renderer are prepared for release.

We evaluate standard and renderer-OOD images with the same three trained visual components. The 3,200 results cover two VLMs, 100 sources, two views, two methods and four visual states (untrained plus three trained). Scores combine both views; post-training scores average three runs, with failures retained. Exact transcription measures whether DeCSP’s recovered graph matches the original completely.

VLM / metricBeforeAfterChange (pp)
Qwen3.5-9B / Base validity4.00%1.33%−2.67
Qwen3.5-9B / DeCSP validity32.50%42.17%+9.67
Qwen3.5-9B / exact transcription16.00%12.50%−3.50
Mistral-Small-3.2-24B / Base validity4.50%7.33%+2.83
Mistral-Small-3.2-24B / DeCSP validity5.50%12.50%+7.00
Mistral-Small-3.2-24B / exact transcription1.00%1.00%0.00

DeCSP exceeds its untrained baseline in all three runs for both VLMs. Qwen’s Base changes from 4.00% to 1.33% on primary v2, compared with 53.00% to 0.50% on the earlier Graph set. The large decline does not recur on the primary set, whose Base accuracy starts at 4.00%. DeCSP’s gains occur without gains in mean exact transcription, reinforcing the need to assess final validity and transcription accuracy separately.

Question 3.2

Which renderers were used for visual training? How do the three program-adapter runs differ in seed and data?

Answer:

We train the visual component on images generated with the legible (clear rendering) configuration. For OOD evaluation, we use Sudoku jitter (position shifts), Nonogram thin (thin strokes) and Graph Coloring bold (bold strokes).

We independently train three versions of the component that generates solver programs, using different random seeds and the same training data.

Reviewer_jkHPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MMCSP-Bench, which provides matched text, standard-image, and OOD-image versions of Sudoku, Nonogram, and Graph Coloring problems to diagnose symbolic reasoning in VLMs. Controlled experiments across seven VLMs distinguish visual-recovery errors from downstream symbolic-generation failures, with all candidate solutions verified against the original constraints. Stage-targeted training shows that better recovery can improve solver-based pipelines without consistently improving direct answering, highlighting the importance of evaluating intermediate correctness separately from final solution validity.

Critical strengths
  1. The paper examines where visual symbolic solving fails, rather than treating final-answer accuracy as a sufficient measure of reasoning. Matched text and image versions preserve the underlying constraints, enabling meaningful comparisons between problem recovery and downstream solving.

  2. Every candidate is checked against the source constraints, accepting any valid solution. This reveals an important distinction: successful solving can conceal transcription errors, while correct transcription does not guarantee downstream success.

  3. Fixed-solver, correct-structure, and shared-transcription comparisons help separate recovery errors from symbolic-generation failures. The results show that external execution alone is insufficient; reliability also depends on how constraint formulation, implementation, and search are divided between the model and backend.

  4. Better visual recovery substantially improves DeCSP but does not consistently benefit direct answering, while program training helps some models and harms others. This challenges the assumption that improving perception or code generation automatically improves end-to-end reasoning, motivating explicit intermediate supervision and route-specific evaluation.

Critical weaknesses
  1. In the primary benchmark, the required operator set is largely predictable from the task name, while the backend handles constraint construction and search. Although the unseen-wording study tests more demanding selection, it still uses the same seven predefined operators. The results therefore demonstrate the reliability of a restricted interface more clearly than the ability to formulate unfamiliar symbolic problems.

  2. All primary calls receive a 1,024-token limit, but generating a complete solver requires substantially more output than selecting operator names. The reported 176 truncated PAL programs show that this restriction materially affects performance; disabling thinking for several models further limits capability comparisons. Budget sweeps and matched total-compute comparisons are needed to establish how much of DeCSP’s advantage survives adequate resources for competing methods.

  3. Correct transcription followed by failure need not indicate incorrect constraint reasoning: normalizing col_runs to column_runs alone converts 79 failures into valid solutions. Likewise, non-exact transcription can preserve enough constraints to produce a valid answer. Separating serialization errors, semantic constraint errors, and search failures would make the claimed perception–reasoning decomposition more precise.

  4. Visual training optimizes image-to-JSON recovery while updating components also used by direct answering. Consequently, degraded direct-answer performance could reflect disrupted alignment or task-specific adaptation, rather than showing that improved perception fails to support reasoning. The experiments establish that transfer is inconsistent, but additional controls—such as supplying improved transcriptions to an unchanged solver model—are needed to explain why.

Questions for authors
  1. How do you distinguish constraint reasoning from task-name routing in DeCSP, and would its gains persist on unfamiliar constraint combinations?

  2. Why use the same 1,024-token limit for operator selection and complete solver generation, and does DeCSP’s advantage persist when PAL receives sufficient generation and reasoning budgets?

6 comments
AuthorsOpenReview ↗

Response to Reviewer jkHP #1

We thank the reviewer for recognizing the value of our matched text–image comparisons and our finding that improving an individual component does not necessarily improve final answers. our method, DeCSP, deliberately reduces the VLM’s symbolic solving burden to help distinguish errors in reading the image from errors in subsequent symbolic solving.

Evaluation design (Figure 1; Section 5.2).

  • Input axis: Text → Standard image → OOD image. Text-to-image comparisons test how accurately the VLM reads digits, clues or graph connections from an image. Standard-to-OOD comparisons test the effect of rendering changes while the underlying CSP stays fixed.
  • Solving axis: Base → PAL → DeCSP. With the correct problem information supplied, these routes compare direct solving, solver-program generation and selecting constraint operators according to the stated rules. They progressively delegate symbolic work to the backend, helping locate errors in these solving steps.

DeCSP’s design.

  • VLM: Select predefined constraint operators; for image inputs, first read the problem information into a structured JSON description.
  • Backend: Construct constraints from that description and the selected operators, then search for a solution.

We address each question and concern point by point below.

Question 1

How do you distinguish constraint reasoning from task-name routing in DeCSP, and would its gains persist on unfamiliar constraint combinations?

Answer:

Our method, DeCSP, can select constraints according to the stated rules, rather than rely only on the task name. On the added combination test, it retains advantages over PAL of 20.00 and 33.33 percentage points for Qwen3.5-9B and Qwen3.5-4B, respectively.

We constructed 30 new text-only Sudoku variants. Each uses a 9×9 grid and digits 1–9, but its instructions require only two or three of the four familiar rules: preserve printed digits, and avoid repeated digits in rows, columns or boxes. The task label stays “Sudoku.” Each variant has a solution under its stated rules, but no solution if all standard Sudoku rules are imposed. DeCSP solves 60.00% with either VLM; the fixed task-name baseline, which selects all four standard rules, solves 0.00%. The tested combinations are absent from the prompt examples.

PAL’s generated solver programs are rerun after uniformly removing eligible redundant imports and standalone example-running code; solver functions and execution limits are unchanged. The table reports valid-solution rates on the same 30 problems:

MethodQwen3.5-9BQwen3.5-4B
Fixed task-name rule0.00%0.00%
PAL40.00%26.67%
DeCSP60.00%60.00%
DeCSP − PAL (pp)+20.00+33.33

For each VLM, all 18 successful DeCSP cases select exactly the required operators. In a separate backend control, supplying the correct operators directly instead of using VLM selections solves all 30 problems (100.00%).

Appendix E.1 reports that the seven VLMs select exactly the required set of operators in 63.42% of cases, compared with 13.64% for the task-name-only baseline.

AuthorsOpenReview ↗

Response to Reviewer jkHP #2

Question 2

Why use the same 1,024-token limit for operator selection and complete solver generation, and does DeCSP’s advantage persist when PAL receives sufficient generation and reasoning budgets?

Answer:

After relaxing PAL’s restrictions, DeCSP retains an 8.89-point accuracy advantage on the Qwen3.5-9B text control; for GPT-5.5, PAL matches its accuracy, while DeCSP uses 70.28% fewer API tokens.

The original 1,024-token cap sets a common per-call budget and controls evaluation cost (Appendix B.1; Table 8). The Qwen3.5-9B controls use the same 90 text problems (30 per task), with thinking off; executor comparisons reuse generated programs.

PAL labels give prompt/executor pairs: Orig. = original; Discl. = original Python rules disclosed; Strict = original Python rules; Broad = broader Python support.

The expanded PAL controls raise the CPU limit from 2 to 20 seconds and the total runtime limit from 3 to 30 seconds, and test broader Python support.

Mean output tokens are calculated over all 90 responses, including failures; input tokens and backend computation are excluded.

Method (prompt/executor)Cap (tokens)ValidityMean output tokens
PAL (Orig./Orig.)1,02452.22%432.24
PAL (Orig./Orig.)4,09652.22%499.63
PAL (Orig./Orig.)8,19252.22%545.14
PAL (Orig./Broad)8,19252.22%545.14
PAL (Discl./Strict)8,19252.22%1,867.11
PAL (Discl./Broad)8,19265.56%1,867.11
PAL (Discl./Broad)16,38465.56%3,141.42
PAL (Discl./Broad)32,76865.56%5,690.04
DeCSP16,38474.44%33.33
DeCSP32,76874.44%33.33

GPT-5.5: matched accuracy with fewer API tokens. We additionally evaluate 30 source problems in three views, with high reasoning effort and a 32,768-token cap. Increasing PAL’s program-size limit from 1,000 to 4,096 AST nodes raises its validity from 72.22% to 98.89%, matching DeCSP; the original 2-second CPU and 3-second elapsed-time limits are retained. At this accuracy, DeCSP uses 1,460.19 API tokens per case versus 4,913.78 for PAL, a 70.28% reduction. These counts include input and output from all model calls, including transcription, thinking and unsuccessful responses; solver-backend computation is excluded.

Qwen3.5-9B thinking controls. The built-in thinking mode exhausts the 16,384- or 32,768-token cap without producing a program. We also test Continuation: a separate program call with the first 4,096 tokens of an existing 32,768-token thought response as input context and an 8,192-token output cap; total output counts the full thought response plus the continuation. The two prompts state the original or expanded Python permissions, both use the broader executor, and changes are relative to the same prompt with thinking off.

Strict-rule prompt.

Mode (program cap)ValidityChange (pp)Mean total output tokens
Off (16,384)65.56%0.003,141.42
Off (32,768)65.56%0.005,690.04
Continuation (8,192)58.89%−6.6733,596.98

Broader-rule prompt.

Mode (program cap)ValidityChange (pp)Mean total output tokens
Off (16,384)51.11%0.003,112.23
Off (32,768)51.11%0.005,296.77
Continuation (8,192)57.78%+6.6733,709.24

Concern 1

In the primary benchmark, the required operator set is largely predictable from the task name, while the backend handles constraint construction and search. Although the unseen-wording study tests more demanding selection, it still uses the same seven predefined operators. The results therefore demonstrate the reliability of a restricted interface more clearly than the ability to formulate unfamiliar symbolic problems.

Answer:

We agree that DeCSP’s correct-structure results demonstrate the reliability of a restricted interface. We deliberately ask the VLM to choose predefined operators instead of constructing a complete solver, so that errors in reading and transcribing the image are easier to identify.

Section 4 of our original submission states that DeCSP “restricts the model output to selecting predefined constraint operators.” We supply correct operators and run DeCSP’s backend on correct structures or saved image transcriptions. The correct-structure result is reported in Table 4; the two image conditions are added controls. Each condition contains 2,100 evaluations (seven VLMs × 300 source problems), including failures to parse the image transcriptions.

InputValidityChange from correct input (pp)
Correct structures100.00%0.00
Standard image transcriptions45.76%−54.24
OOD image transcriptions30.14%−69.86

This motivates training the VLM to transcribe images into JSON problem descriptions, then checking whether final solving improves (Sections 5.3 and 6.3).

AuthorsOpenReview ↗

Response to Reviewer jkHP #3

Concern 2

All primary calls receive a 1,024-token limit, but generating a complete solver requires substantially more output than selecting operator names. The reported 176 truncated PAL programs show that this restriction materially affects performance; disabling thinking for several models further limits capability comparisons. Budget sweeps and matched total-compute comparisons are needed to establish how much of DeCSP’s advantage survives adequate resources for competing methods.

Answer:

The expanded controls establish an accuracy advantage on the Qwen text subset and an API-token advantage at matched accuracy for GPT-5.5. As detailed in Question 2, broader Python support and larger generation budgets bring PAL to 65.56% on the 90 Qwen3.5-9B text problems, compared with 74.44% for DeCSP. For GPT-5.5, relaxing the program-size limit brings PAL to the same 98.89% validity as DeCSP, while DeCSP uses 70.28% fewer API tokens.

The 176 truncated PAL programs in our paper come from standard-image calls. The added Qwen controls isolate program generation and execution using text inputs; the GPT extension includes text, standard-image and OOD-image views. These comparisons measure accuracy and model-token usage, rather than matched total computation across the model and solver backend.

Concern 3

Correct transcription followed by failure need not indicate incorrect constraint reasoning: normalizing col_runs to column_runs alone converts 79 failures into valid solutions. Likewise, non-exact transcription can preserve enough constraints to produce a valid answer. Separating serialization errors, semantic constraint errors, and search failures would make the claimed perception–reasoning decomposition more precise.

Answer:

Correcting the operator name eliminates all failures when the correct problem description is supplied, but accuracy remains much lower with image inputs. In our method, DeCSP, all 79 correct-structure failures arise from the col_runs/column_runs mismatch. Appendix B.2 reports the correction as “a direct measurement of interface-name sensitivity.” We distinguish these errors in calling the solver from errors in reasoning about constraints.

We extended the same correction to saved image responses, keeping the inputs and backend unchanged. Each input condition has 2,100 evaluations.

InputBeforeAfterChange (pp)
Correct structures96.24%100.00%+3.76
Standard images42.57%45.71%+3.14
OOD images28.19%30.14%+1.95

We also report 94 valid answers to standard-image problems whose transcriptions did not exactly match the original problem (Section 6.1; Appendix D.2), so we measure transcription accuracy and solution validity separately.

For PAL, where the VLM generates the solver program, we record whether that program is rejected by the code checks, fails during execution, or returns an invalid solution. An invalid answer alone does not tell us whether the program encoded the constraints incorrectly or made an error when searching for a solution.

Concern 4

Visual training optimizes image-to-JSON recovery while updating components also used by direct answering. Consequently, degraded direct-answer performance could reflect disrupted alignment or task-specific adaptation, rather than showing that improved perception fails to support reasoning. The experiments establish that transfer is inconsistent, but additional controls—such as supplying improved transcriptions to an unchanged solver model—are needed to explain why.

Answer:

We test this concern by keeping the answering VLM fixed and changing only the supplied transcriptions. On the earlier transfer set, supplying transcriptions from the visually trained component raises Qwen3.5-9B’s validity from 22.22% to 33.33%, an 11.11-point improvement (95% CI: +6.11 to +16.11).

We use 90 problems from the earlier transfer set, 30 per task, in both image views. All successful cases come from its earlier-version Graph Coloring instances, which admit valid colorings based on node labels alone. This control tests the effect of changed transcriptions; it does not establish improved use of graph edges or identify the mechanism of the original decline after model updates. Post-training scores average three visual-training runs; the 95% intervals describe uncertainty from sampling source problems.

VLMOriginal transcriptionsPost-training transcriptionsChange [95% CI] (pp)
Qwen3.5-9B22.22%33.33%+11.11 [+6.11, +16.11]
Mistral-3.2-24B22.22%25.19%+2.96 [−1.85, +7.22]

On all 100 primary Graph Coloring v2 problems, DeCSP gains 9.67 points for Qwen and 7.00 for Mistral; Base changes by −2.67 and +2.83 points, respectively. The large legacy-set Qwen decline does not recur, and exact transcription does not improve on average.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Discussion

Dear Area Chair,

Thank you for overseeing the discussion. We would like to clarify how Seeing2Solving evaluates the same MMCSP-Bench problems along two axes (Figure 1). On the input axis, we present each problem as structured text, as a standard image, and as an image with out-of-distribution (OOD) rendering. On the solving axis, the model answers directly (Base), generates a complete solver program (PAL), or selects constraint operators (DeCSP). We check transcriptions against the source structure and verify solutions against the original constraints to distinguish problem recovery errors from downstream symbolic-generation errors.

DeCSP deliberately reduces the model’s symbolic solving burden. The model selects predefined operators, while a deterministic backend constructs the constraints and searches for a solution. All three reviewers asked whether the task name alone could account for the model’s operator choices on the primary benchmark (jkHP C1/Q1, tico C2/Q2.1, 15jp C1/Q1). We kept the Sudoku label fixed and varied the combinations of familiar rules on 30 text-only boards. With the same backend, task-name routing solved 0/30, Qwen3.5-4B and Qwen3.5-9B each solved 18/30 with their selected operators, and the backend solved 30/30 when given the correct operators (Appendix E.1).

The ratings from Reviewers jkHP, tico, and 15jp changed from (2, 2, 2) to (2, 3, 3) during the discussion. Both tico and 15jp raised their ratings to weak acceptance, stating that the additional experiments addressed most of their concerns. Reviewer jkHP has not yet posted a follow-up. Our rebuttal and revised manuscript explicitly address both of jkHP’s questions and all four concerns. Appendix E.1 tests task-name routing (Q1/C1). Appendix E.4.3 reports PAL budget sweeps and per-case API token counts (Q2/C2). The token counts exclude solver computation. We also isolated operator-naming errors in saved outputs (C3, Section 6.2 and Appendix B.2) and varied only the transcriptions supplied to a fixed answering model to test visual-training transfer (C4, Appendix C.2).

We have highlighted revised passages and the heading of the new Appendix E.4 in light yellow. We thank all three reviewers for their suggestions and would welcome any further discussion of these results.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.