- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MMCSP-Bench, a benchmark containing matched text, standard-image, and rendering-shifted versions of 300 synthetic constraint satisfaction problems. The tasks cover Sudoku, Nonogram, and Graph Coloring. It also proposes DeCSP, in which a vision-language model selects from seven predefined operators and a deterministic backend constructs and solves the resulting CSP.
Across seven models, DeCSP achieves 96.24% mean validity on text inputs, decreasing to 42.57% on standard images and 28.19% on rendering-shifted images. The paper uses matched views, shared transcriptions, exact-recovery analysis, and stage-specific training to separate perception, interface, execution, and verification failures. This diagnostic framework is useful. However, the main DeCSP task appears closer to task routing and output-format compliance than general symbolic formulation.
Critical strengths
-
Verifying candidate solutions against the original problem is a sound design choice. It avoids giving credit to a model that correctly solves an incorrectly transcribed surrogate problem.
-
The matched text and image views make it possible to distinguish visual recovery errors from downstream symbolic or interface failures. The exact-recovery analysis shows clearly that correct transcription and valid solving are not equivalent.
Critical weaknesses
-
DeCSP receives the task identity and selects from only seven operators that are nearly partitioned by task. Canonical task-correct insertion reaches 100%, suggesting that the learned component performs constrained routing rather than general symbolic formulation.
-
The results are highly sensitive to interface syntax. Normalizing one Nonogram operator alias converts 79 failures into valid solutions, so a notable portion of the measured error reflects naming conventions rather than perception or reasoning.
-
The benchmark covers three small synthetic CSP families, and the OOD variants mainly modify fonts, line positions, strokes, or node rendering. The experiments support conclusions about renderer sensitivity, but not broad visual or symbolic generalization.
Questions for authors
- Given that task identity nearly determines the valid operators, what part of DeCSP should be viewed as learned symbolic formulation?
- How many failures are caused only by operator naming or formatting?
- Is the OOD claim intended to cover renderer changes only, rather than changes in problem structure?
5 comments
Response to Reviewer 15jp #2
Concern 1
DeCSP receives the task identity and selects from only seven operators that are nearly partitioned by task. Canonical task-correct insertion reaches 100%, suggesting that the learned component performs constrained routing rather than general symbolic formulation.
Answer:
The restricted operator interface is intentional: it reduces the VLM’s symbolic solving burden so that we can examine errors in reading and transcribing the image. Section 4 of our original submission specifies that DeCSP “restricts the model output to selecting predefined constraint operators.”
For this control, we supply the correct operators ourselves and run DeCSP’s backend on either correct puzzle data or saved VLM transcriptions. The correct-data result comes from the original Table 4; the two image conditions are added controls. Each row covers seven VLMs × 300 problems (N = 2,100), including transcriptions that cannot be parsed.
Correcting only the operator name retains one original timeout and yields 45.71% on standard images; supplying correct operators in the control below yields 45.76%.
| Input | Validity | Change from correct input (pp) |
|---|---|---|
| Correct structures | 100.00% | 0.00 |
| Standard transcriptions | 45.76% | −54.24 |
| OOD transcriptions | 30.14% | −69.86 |
With correct operators supplied, the low image scores show that producing usable, correct puzzle data from images remains a major bottleneck. Among OOD transcriptions that can be parsed, validity is 100.00% for the 551 that reproduce the puzzle exactly and 10.68% for the 768 that do not.
Concern 2
The results are highly sensitive to interface syntax. Normalizing one Nonogram operator alias converts 79 failures into valid solutions, so a notable portion of the measured error reflects naming conventions rather than perception or reasoning.
Answer:
Correcting the operator name eliminates all failures with correct puzzle data, but scores on image inputs remain low.
Appendix B.2 reports: “This replay yields 79 failure-to-valid transitions and provides a direct measurement of interface-name sensitivity.” We now apply the same col_runs → column_runs correction to saved image responses, keeping the transcribed puzzle data, backend and execution settings unchanged. Across 2,100 evaluations per input condition, validity rises from 96.24% to 100.00% with correct structures, 42.57% to 45.71% with standard images and 28.19% to 30.14% with OOD images.
Our paper also reports 94 valid standard-image solutions from transcriptions that do not exactly match the original puzzle (Section 6.1; Appendix D.2). We therefore check the transcribed puzzle and the final answer separately, and record name mismatches as interface errors.
Concern 3
The benchmark covers three small synthetic CSP families, and the OOD variants mainly modify fonts, line positions, strokes, or node rendering. The experiments support conclusions about renderer sensitivity, but not broad visual or symbolic generalization.
Answer:
Our OOD experiment deliberately changes only how the same puzzle is drawn. Section 3.2 states: “OOD is restricted to presentation changes and does not alter task rules or problem structures”. Because the puzzle and its rules stay fixed, a performance drop reflects sensitivity to its visual presentation, not increased difficulty of the symbolic problem. Our paper reports the following DeCSP results across seven VLMs and the three task families, with 300 problems per VLM and view (N = 2,100):
| Standard images | OOD images | Change (pp) |
|---|---|---|
| 42.57% | 28.19% | −14.38 |
Turning graph images into usable, correct descriptions also remains a bottleneck on larger graphs. We added 100 graphs with 9–12 nodes, compared with 6–8 nodes in the main benchmark, and evaluated five VLMs. Each receives either the correct graph data or a standard image. DeCSP uses VLM-selected operators; Known-Program uses a prewritten, task-correct solver. Each method obtains its own transcriptions of the same images. Tables report validity on the same 100 problems; changes are image minus correct-structure scores:
DeCSP.
| VLM | Correct structures | Standard images | Change (pp) |
|---|---|---|---|
| Qwen3.5-4B | 100.00% | 0.00% | −100.00 |
| Qwen3.5-9B | 100.00% | 0.00% | −100.00 |
| Qwen3-VL-32B | 100.00% | 0.00% | −100.00 |
| Qwen3.8-27B | 100.00% | 2.00% | −98.00 |
| Mistral-3.2-24B | 100.00% | 0.00% | −100.00 |
Known-Program.
| VLM | Correct structures | Standard images | Change (pp) |
|---|---|---|---|
| Qwen3.5-4B | 100.00% | 1.00% | −99.00 |
| Qwen3.5-9B | 100.00% | 0.00% | −100.00 |
| Qwen3-VL-32B | 100.00% | 1.00% | −99.00 |
| Qwen3.8-27B | 100.00% | 3.00% | −97.00 |
| Mistral-3.2-24B | 100.00% | 0.00% | −100.00 |
Follow-up to Reviewer 15jp
Dear Reviewer 15jp,
Thank you again for your thoughtful feedback. Our responses clarify DeCSP’s diagnostic role and the scope of renderer-OOD evaluation, and include additional operator-name correction results and rule-combination tests.
As the discussion period draws to a close, we would appreciate any further feedback you may have. Please let us know if any points remain unclear or require further discussion; we would be happy to clarify them.
Best regards, The Authors
Response to Reviewer 15jp #1
We thank the reviewer for recognizing the value of checking solutions against the original problem, matched text and image problems, and checking whether the transcribed problem is correct.
DeCSP deliberately reduces the VLM’s symbolic solving burden to help distinguish errors in reading and recording the puzzle from errors in selecting constraints or solving it.
We address each question and concern below.
Question 1
Answer:
The learned step is choosing which predefined constraint operators match the stated rules; the backend then constructs constraints and searches for a solution. On the added 30 problems with varying rules, DeCSP solves 60.00% with VLM-selected operators, compared with 0.00% when operators are selected from the task name alone.
In the main benchmark, each task has fixed rules, so the task name is enough to select the required operators. Appendix E.1 tests whether VLMs select the correct operators from reworded rules. Seven VLMs each receive 132 rule descriptions; the table reports how often they select exactly the required operators:
These 30 new text-only variants are separate from the main benchmark’s 100 Sudoku problems and their views. They retain 9×9 grids, digits 1–9 and the label “Sudoku,” but require only two or three of four rules: preserve printed digits, and avoid repeated digits within rows, columns or boxes. Each is solvable under its stated rules but not all standard Sudoku rules; the tested combinations are absent from the prompt’s single-operator demonstrations. The table reports validity using the same backend, with the last row supplying correct operators directly instead of using VLM selections.
Every successful DeCSP case uses exactly the required operators.
Question 2
Answer:
Correcting the operator name alone resolves 79 failures with correct puzzle data, 66 with standard images and 41 with OOD images. Appendix B.2 of our original submission reports the correct-data result. We now apply the same correction to saved image responses, changing only
col_runstocolumn_runs; the transcribed puzzle data, backend and execution settings stay unchanged.Each row contains 2,100 evaluations (seven VLMs × 300 problems). Before correction, the backend rejects this operator name in 3.76% of correct-structure evaluations, 3.19% of standard-image evaluations and 2.14% of OOD-image evaluations. The table shows valid-solution rates:
This mismatch accounts for all recorded format rejections at the operator-selection stage; five image cases still fail because their transcribed puzzles or returned solutions are incorrect.
Another 542 standard-image and 781 OOD-image transcriptions cannot be parsed as JSON; these failures cannot be attributed solely to naming or formatting without establishing whether the puzzle was transcribed correctly.
Question 3
Answer:
Yes. Our original OOD design changes how the puzzle is drawn, while keeping the puzzle and its rules fixed (Section 3.2; Appendix A.1). Appendix A.1 states: “Across all three tasks, the OOD view changes only rendering attributes; the source problem, public structure, and original-problem verifier remain unchanged.” We compare the same VLM and solving method before and after changes to fonts, strokes or node appearance.