- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this work, the authors evaluate the performance of ASCII maps as a symbolic visual representation instead of RGB, the traditional and common representation. Basically, they replace images in the input of VLMs with ASCII maps, but only for a few tasks defined in the paper.
They curate a benchmark with 3 tasks and customized eval metrics for each one. The tasks are short-range path planning, long-horizon navigation, and object-centered action planning.
They run evals on multiple open- and closed-source models with images, images + ASCII maps, and ASCII maps alone, and they show that ASCII maps are significantly (sometimes 80 times) better than images only.
Critical strengths
1.The improvements with ASCII maps on the tasks the paper introduced are significant and large. 2.The question the authors are bringing up is valuable: what is the best representation of the visual world as the input for VLMs?
Critical weaknesses
1.One baseline that is missing in the SIG-tiny experiment is using the SIG GT image instead of the ASCII map and comparing that with the ASCII results. The improvement there might come from the clear structure of the objects on the map.
2.There are not many real-world and established benchmarks used for this conversion test. The only real-world task is SIG-tiny, in which the numbers are confounded.
3.In the intro, the authors ask three questions. The third one is “Scope: Under what task conditions do the benefits of symbolic structure outweigh the loss of visual detail?”, but the authors don’t mention what kind of tasks don’t benefit from ASCII mapping.
Questions for authors
1.Run more real-world and established spatial benchmarks, such as VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs.
2.Show spatial tasks where ASCII mapping might not be helpful, and explain why.
3.Run baselines with SIG GT images.
- what was the reasoning behind choosing those 3 specific tasks?