acceptodds
Withdrawn from ICLR 2027

From Pixels to Symbols: Machine-Native Symbolic World Models for Multimodal Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) are often trained and evaluated with visual inputs designed for human perception rather than machine reasoning. Inspired by the reverse of graphical user interface (GUI), we propose ASCII maps as a compact, discrete, symbolic world model that convert visual scenes into layouts encoding walls, free space, objects, targets, and action-relevant entities. To evaluate the performance of ASCII map as a symbolic visual representation, we introduce SymbolicBench, which covers short-range path planning, long-horizon navigation, and object-centered action planning across maze, floorplan, and agent environments using grid-like images. Across top-tier proprietary and open-source MLLMs, ASCII-based inputs substantially outperform image-only inputs, achieving from 2 to 83 times improvement, while reducing token usage. Further analysis shows that these gains persist across different task complexity levels and generalize beyond grid-like images to real-world images for spatial reasoning tasks with strong robustness. variations. Together, these findings support compact symbolic world representations as an effective interface between visual scene structure and MLLM reasoning, enabling more reliable spatial reasoning and planning with fewer input tokens. The source code will be fully released upon paper acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MLLM, Symbolic World Models, Spatial Reasoning, Symbolic Visual Representation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RXfsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this work, the authors evaluate the performance of ASCII maps as a symbolic visual representation instead of RGB, the traditional and common representation. Basically, they replace images in the input of VLMs with ASCII maps, but only for a few tasks defined in the paper.

They curate a benchmark with 3 tasks and customized eval metrics for each one. The tasks are short-range path planning, long-horizon navigation, and object-centered action planning.

They run evals on multiple open- and closed-source models with images, images + ASCII maps, and ASCII maps alone, and they show that ASCII maps are significantly (sometimes 80 times) better than images only.

Critical strengths

1.The improvements with ASCII maps on the tasks the paper introduced are significant and large. 2.The question the authors are bringing up is valuable: what is the best representation of the visual world as the input for VLMs?

Critical weaknesses

1.One baseline that is missing in the SIG-tiny experiment is using the SIG GT image instead of the ASCII map and comparing that with the ASCII results. The improvement there might come from the clear structure of the objects on the map.

2.There are not many real-world and established benchmarks used for this conversion test. The only real-world task is SIG-tiny, in which the numbers are confounded.

3.In the intro, the authors ask three questions. The third one is “Scope: Under what task conditions do the benefits of symbolic structure outweigh the loss of visual detail?”, but the authors don’t mention what kind of tasks don’t benefit from ASCII mapping.

Questions for authors

1.Run more real-world and established spatial benchmarks, such as VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs.

2.Show spatial tasks where ASCII mapping might not be helpful, and explain why.

3.Run baselines with SIG GT images.

  1. what was the reasoning behind choosing those 3 specific tasks?
Reviewer_E9S5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ASCII maps as structured inputs for multimodal language models. SymbolicBench evaluates maze navigation, floorplan navigation, and object-centered planning. Across seven models, ASCII inputs substantially improve performance over images and reduce input tokens, with additional experiments on driving scenes and representation robustness.

Critical strengths

1.The motivation is clear: explicitly representing obstacles and object locations can make spatial planning easier for existing models. 2.Improvements are substantial across models and tasks; GPT maze exact match increases from 0.177 to 0.897. 3.Symbol, resolution, corruption, and predicted-map experiments provide useful evidence beyond clean-input comparisons.

Critical weaknesses

1.The novelty relative to existing symbolic-grid representations is insufficiently explained. SpatialEval already represents mazes using ASCII and compares text-only, image-only, and combined inputs; Visualization-of-Thought uses emoji grids for navigation and visual state tracking. The broader tasks and robustness analyses are useful, but what new representational principle or empirical insight distinguishes this work from extending these existing approaches? 2.Equation (7) computes AAR from sets of action-location tuples. Coordinates specify where actions occur, but set overlap does not preserve their order. Are tuples collected only after valid sequential execution, or directly from the predicted plan? Reporting task success for the existing Mini-BEHAVIOR predictions would clarify whether improved coverage translates into executable plans. 3.Table 3 and Section 4.3 report that each score averages five corruption types. Could the authors show the per-type results, to establish whether the average hides a corruption type on which the benefit disappears? The main evaluation prompts and inference settings would also help readers interpret the baselines. Moreover, there's a wrong reference to a table in Appendix B.1.

Questions for authors

Please refer to the weakness。

Reviewer_qLgeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper turns grid-like pictures (mazes, floor plans, Mini-BEHAVIOR scenes) into ASCII text maps using hand-written rules. It finds the grid lines, uses colors to tell walls from floor, and uses DINO-X to find objects. Then it gives the text map to an MLLM instead of, or along with, the picture. On their new SymbolicBench (300 mazes, 300 floor plans from CubiCasa5K, 300 Mini-BEHAVIOR scenes), seven models do 2 to 83 times better with the text map than with the picture, and use fewer tokens. They also test driving scenes from SIGBench-tiny by turning the ground-truth grid into ASCII, and the appendix adds a trained model that predicts that grid from the image.

Critical strengths
  1. The experiment is simple, easy to reproduce, and covers seven models from six families.

  2. The token savings are real, and the noise, symbol and grid-size tests are a reasonable start.

  3. The path metrics (LWS, PES, DPF) make sense when there are several shortest paths.

Critical weaknesses
  1. With an essentially exact text copy of the grid, the model no longer has to read the picture, and LLMs are good at text puzzles. Earlier work already showed VLMs often do worse with images than LLMs do with text, and BALROG found that adding images often hurts agents.

  2. The images are drawn from structured data (maze grids, a grid world, floor plans that the authors binarized and overlaid with grid lines, App. A.1), and the converter relies on those exact drawing choices. So the ASCII map is basically the answer key, and I'd expect a simple search algorithm to solve Maze and Floorplan exactly from it. The missing test is letting the MLLM build the ASCII map from the image itself.

  3. An ASCII map is a state, not a world model, since nothing predicts what happens next. The intro's question about which symbols help only gets a small symbol-swap test, and there's no task where visual detail matters, so I feel this paper claims more than it shows

  4. It needs a proofread. There's a broken sentence in the abstract, "Tab. ??" (Line872) and a repeated sentence in App. B.1, a wrong ECR legend in Fig. 3, and several typos. "Token usage" and the reasoning settings are never defined.

Questions for authors
  1. What happens if the MLLM writes the ASCII map from the picture itself and then plans?

  2. How does the 2D ASCII layout compare with a plain list of coordinates?

  3. Are Tables 1, 3 and 5 on the same cases? How many runs, and how much do they vary?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.