acceptodds
Under review as a conference paper at ICLR 2027

Iconic Format Costs Language Models Accuracy: A Preregistered Three-Format Study

Abstract

It is a common claim in the cognitive sciences that mental representations come in different formats. A representational format is a way of information organization that dictates a representation's specific computational profile. For example, discursive representations—notably, natural language representations—are decomposable into constituent representations that combine in certain canonical ways. Iconic representations, meanwhile, are representations that represent an object by way of mirroring its arrangement. A typical example of an iconic representation is that of a typical map, which mirrors, in miniature, the actual distance relations between the places it depicts. Whether language models build internal models of spatial structure is disputed and whether any such models are iconically formatted has rarely been asked. This set of pre-registered studies aims to make progress on this question via a multi-pronged strategy of evaluating behavior and model mechanisms. Study 1 evaluates three open-weight and two frontier models on informationally equivalent but differently formatted reasoning tasks. Study 3, meanwhile, subjects these models to the same tasks but removes their written-out reasoning, asking for the answer alone. More specifically, these studies ask the language model to supply correct answers to natural language sentences, diagrams, and "tabular" formats. These studies find that the open-weight models perform worse on iconic tasks compared to discursive or tabular tasks, and by a wide margin (−0.92 log-odds against tables in Study 1; −25.1 percentage points in the graph family, and −6.0 to −13.0 percentage points in the other three, depending on the scorer), suggesting that contemporary language models struggle to reason over icons. The frontier models are at or near ceiling in all three formats outside the graph family, so outside it these tasks are uninformative about them; within it, they performed worse on the iconically formatted questions (5.2 and 6.0 percentage points). Removing written-out reasoning narrows the gap (−0.40 log-odds). Because studies 1 and 3 are behavioral, they cannot tell us about whether language models use, or even have, iconic representations. Study 2 endeavored to test this by probing the residual streams of the model while performing these tasks. The outcome of the study was negative, insofar as the results were entirely determined by choice of readout, but supplied an important methodological result. That result is that linear decodability from the residual stream cannot discriminate between a language model's having iconic representations versus a discursive representations of spatial facts. Three conclusions follow from this work. First, there is some evidence that current language models struggle to reason over icons. Second, a common method for studying the internal representations of language models, linear probing, cannot settle questions about format in particular. Third, these results point to better designs for such studies: controls that detect when a readout is blind to format, and causal tests of whether a model uses the information it encodes.

Primary area:
interpretability and explainable AI
Keywords:
representational format, reasoning in language models, iconicity, linear probing

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_T7hDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper seeks to understand how language models process inputs that are ostensibly informationally equivalent but are presented in altered modalities (in iconic, tabular, or discursive textual formats).

They evaluate a mix of open and proprietary models on several hand-designed tasks, which require models to answer simple multiple choice questions about stimuli that are presented in the different formats mentioned above. They find that models do not process all modalities with equal degree of accuracy, and that linear probes, by their nature, are unable to fully distinguish different formats.

The ideas in this paper are extremely interesting, but I believe there is a deep lack of conceptual clarity (elaborated upon later) and that there are severe weaknesses in the reporting of results, partly due to those sections being seemingly AI-generated. I believe the work is in need of severe revisions before it is fit for publication

Critical strengths
  • The paper covers a fair deal of conceptual ground with respect to its experiments and I found the problem of using linear probes to distinguish representational formats under simple linear transformations interesting.

  • As language models and multimodal VLMs continue to process ever more modalities of data it is important to understand the extent to which models are sensitive to these difference (if at all). This paper gives preliminary evidence of subtle differences depending on how text input is formatted, even if the informational content is mostly identical

Critical weaknesses
  • I do not believe the different representations are trivially informationally equivalent. They may be equivalent in the eyes of the experiment designers who used the same source dataset to construct the stimuli, but to the eyes of the observing system there may be small discrepancies between formats which, while not necessarily task relevant on the surface, might influence task performance. Since a central claim is that surface form matters (motivation for the tabular condition), if one format makes some kinds of information more easily discernible (e.g., the iconic format for the network diagrams), would we still consider them informationally equivalent? Or does this latter example tap more into the notion of computational equivalence that the authors note?

  • It is argued that the tabular condition is introduced to control for the fact that the diagram input while containing the same information has different surface properties, namely being arranged spatially. I understand the argument the authors lay out in the paper, but am not really convinced that the tables are “a discursive representation with superficial iconic form.” The kind of form they take is highly distinct from diagrams where information is encoded by spatial features and is less flexible in its semantic content than discursive representations due to the tighter syntax. All of which is to say is that I’m not sure that tabular falls between iconic and discursive if the latter two are imagined to be at the poles of a spectrum. Indeed, perhaps some of the lack of differences between tabular and discursive results can be explained in terms of this lack of conceptual clarity.

  • While the opening introduction is largely clear. the bulk of the results section is hard to parse and appears to be 100% AI-generated (as indicated by Pangram <5% false positive rate). The issue isn’t that it is AI generated per se, but that the current wording severely hurts the readability of the paper. I strongly recommend that the authors rewrite this section

  • The presentation of the results is of mixed quality. For example Fig 2 is largely redundant with the tables presented earlier in the paper. The only added information for the by-task family graph is the individual points corresponding to each model, but even that is better represented as grouped bar graphs with error bars (when possible).

  • For the residual stream analyses there are inadequate justifications for why certain probing decisions were made, even while noting the choice of readout makes a difference. 
For example, “mean-pooling over prompt tokens and reading the last token give different results.”
Why would one consider doing one over the other? I can surmise, but this should be explicitly mentioned in the text.

  • There are many results that aren’t discussed or commented on strongly enough given the framing of the paper. For example, what are we to make of the fact that models perform worse on ‘iconic’ tasks where we would expect people to perform better with iconic vs. discursive descriptions. The lack of human baselines also makes it difficult to assess how many of these effects are directionally sound vs. potentially just being noise (the lack of error bars on reported statistics adds to this issue).

  • The pooled model results are helpful to include, but I strongly recommend including the by-model breakdown in the main text (since there is quite a range of models used in terms of size/performance). Additionally for open models, could the authors also include average accuracy with error bars (stochasiticity will be introduced if non-greedy decoding is used)?

  • There are interesting points raised about representational analysis and comparison of different modalities of the same data. Several interesting papers from the field are referenced, which suggest relatively easy to implement strategies relative to linear probes, but these are not conducted.

  • There is an overall dearth of error analyses on, say, the chain of thoughts or even in the patterns of errors that might give a reader insight into why we might be seeing the deficits that we do.

Questions for authors
  • Why do the authors think we see advantages for non-iconic formats for tasks that seem to require visual processing?

  • Could the authors conduct a simple human experiment baseline to ground these results?

  • Could the section on residual stream probes be reorganized into a general section on representational probing and strengthened with non-probing analyses?

1 comment

Thanks for the comments. Regarding informational equivalence, we believe that the three renderings are informationally equivalent. Informational equivalence concerns whether one rendering can be recovered from another, whereas computational equivalence concerns how easily a reader can draw inferences from each rendering. A rendering that makes some facts easier to discern differs computationally, not informationally, and this is what the study sets out to measure. To make sure that the renderings were informationally equivalent, we ran an additional, unregistered check during the review period: a deterministic parser for each rendering, which rebuilt the generator’s structure exactly from every rendering of every item (§3).

Re the tabular condition: the design requires only that the table state the same facts as the sentences while sharing the surface properties of the diagram. It is not part of our argument that the table lies between the other two formats on a single scale, and we have made this clear in §1.

Presentation: we agree, and §5 has been rewritten, with the scoring terms defined at the outset. Per-model results are now given in Appendix C. The open-weight models used greedy decoding, so single runs were reported; for the hosted models, the scored answer differed on 1.4% and 2.8% of 144 trials run twice. Study 2 has been removed and will be reported separately.

Re the non-iconic formats’ better performance: the iconic deficit rules out reading answers off the arrangement as the models’ dominant strategy, since that strategy would predict an advantage, which was not observed. The small value of reasoning with diagrams suggests, though it does not confirm, the hypothesis that the models fail to transcribe the drawing into a form their reasoning can use; Appendix B reports one such failure. We agree that a human baseline would help ground the comparison, but unfortunately we could not run one during the discussion period.

Reviewer_Lgz5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper evaluates whether LLMs struggle to perform tasks that are presented in an iconic (image-like) format when compared with natural language or tabular formats. The results are evaluated with and without chain-of-thought (though importantly this is not in the models' native reasoning mode). When evaluated with a preregistered scorer, decreased performance is observed for iconic formats, but the results are more equivocal when scoring mistakes are corrected. A probe analysis does not identify any reliable signatures of representational format.

Critical strengths
  • This paper addresses an interesting question about representational format in LLMs.
  • The studies are preregistered, and deviations from the preregistration are clearly reported.
  • The paper is clearly written and the experiments are well documented.
Critical weaknesses
  • It is very much expected than a text-only LLM will struggle with tasks that are presented in a 2D image-like format. VLMs would be a more reasonable model class for this evaluation as they have the native ability to interpret 2D images.
  • For LLMs, the ability to interpret 2D images rendered via text will depend strongly on reasoning. Native reasoning ability was disabled and models were only allowed to reason via chain-of-thought outputs, which should also lead to predictable impairments on these tasks.
  • The analysis with the preregistered scorer is not informative because it makes basic mistakes in parsing the models' outputs. This metric is not informative simply because it was preregistered. These results give a misleading impression -- when analyzed properly, the effect of iconic formats is much more equivocal and appears to only play a significant role for the graph task.
  • The tables and figures are missing important annotations. There is no caption indicating what metric (accuracy?) is reported in the tables. The figures do not have any error bars.
Questions for authors

How do VLMs or reasoning-enabled LLMs perform on these tasks?

1 comment

Thanks for the comments. We disagree that the deficit is to be expected. Larkin and Simon’s account predicts an advantage for diagrams over sentences stating the same facts, and two recent studies find that ASCII layouts improve text-only models’ spatial reasoning relative to prose descriptions (see Huang et al., 2026; Rajpal et al., 2026). Neither of those studies separated iconicity from other features of the ASCII format, however. This study does so via the tabular control, which shares the surface properties of the diagram without being iconic.

We used text-only models because the question of format has to be tested with the input channel held fixed. A VLM shown an image of the diagram would receive it in a different modality and a different format, so the comparison would be confounded. Meanwhile, we disabled internal reasoning so that every model reasons only in the visible channel that Study 3 removes. Results may well differ with it enabled, but we have not been able to run reasoning-enabled models since submission.

We agree that the registered scorer should not lead, and have made the appropriate adjustments, leading instead with the key-resolved estimate. §6 explains the scorer’s bias: in FU5, where the key is manipulated, the registered scorer gives the key contrast +0.87 log-odds, as opposed to +0.10 under key resolution. Under key resolution, the deficit is largest in graph (−25.1 percentage points) and −6.0 percentage points outside it. The interaction with reasoning outside graph holds under both scorers. The tables are now captioned with the metric they report.

Reviewer_u8G8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article investigates the impact of using different representation formats for the same information on the accuracy of language model inference. The author designed three forms of information expression, including discursive, tabular, and ASCII graphic with spatial mapping relationships, and conducted pre registration experiments on four types of inference tasks and five language models. The results indicate that when the model is required to provide a brief inference process, the accuracy of Iconic format is significantly lower than the other two formats, with the performance gap of Graph task being particularly prominent. Further experiments revealed that after removing explicit reasoning requirements, this gap actually narrowed.

In addition, the paper analyzes the decodability of spatial information within the model through Residual Stream Probing and discusses the theoretical limitations of Linear Probing in identifying representation formats. The author believes that linear decodability is not sufficient to prove that the model adopts a specific representation organization.

Critical strengths

Starting from the difference between visual representation and discourse representation, this paper explores the influence of information organization forms on the inference performance of language models, and introduces the Tabular format as a control condition. The research motivation is clear, and the experimental design is targeted to a certain extent.

The study covers four types of tasks, five models, and two inference instruction conditions, and verifies the impact of input formats on model accuracy through a mixed effects model and multiple sensitivity analyses.

The author disclosed in detail the pre registration bias, score correction, and data generation issues, and objectively discussed the unexpected results and unsuccessful Probe experiments. In addition, the analysis of the difference between linear decodability and actual representation forms has certain methodological value.

Critical weaknesses

Although the Tabular format controls some surface features, Iconic still involves additional character parsing, letter mapping, and relationship extraction processes. Especially in Graph tasks, the model may first have difficulty correctly identifying the graph structure. The existing experiments have not fully distinguished the influence of these factors on Iconicity itself.

When considering the random slope of the model, the complete sample analysis under the modified score is no longer significant (p=. 77), but returns to significance after excluding some question types. In addition, some tasks have imbalanced answer distribution and posterior sample exclusion, which affects the robustness of the conclusions.

Study 2 only conducted exploratory analysis on Qwen2.5-1.5B and failed to reliably identify internal representation formats, therefore its theoretical increment and empirical contribution are relatively limited.

Study 1 and Study 3 found the impact of format and inference instructions on accuracy, but Study 2 failed to explain the internal mechanisms behind these differences. Therefore, the paper has not yet formed a complete chain of evidence from behavioral observation to mechanism verification.

Questions for authors

Can the author further distinguish the impact of Iconicity itself and the cost of ASCII graphic parsing? Especially in Graph tasks, is the performance degradation mainly due to errors in graph structure recognition or subsequent inference errors?

After adjusting the score, the random slope analysis of the complete sample was no longer significant (p=. 77), but after excluding some question types, it became highly significant again. How does the author explain this difference? Does it affect the generalization of the main conclusions?

Given the low success rate of Iconic format answer parsing, can the author further distinguish the accuracy loss caused by inference errors and output format not meeting the requirements?

What is the new contribution of this paper compared to existing research on the invariance of reversible linear transformations? Study 2 failed to reliably identify internal representation formats. How do its results support the findings of previous behavioral experiments?

1 comment

Thanks for these comments. We address them point by point.

Iconicity versus the cost of parsing ASCII: the current design does not completely separate these, and §8 now distinguishes costs that are relevant to iconicity (e.g., reading a relation off an arrangement) from those incidental to it (e.g., the letter key and the rarity of ASCII text). The preregistered follow-up FU5 addresses some of these worries. We found in that follow-up that the letter key did not account for the drawing’s cost, though only marginally, and that adding the table’s rows beneath the drawing raised accuracy (C2 = +1.62).

In graph, two open-weight models failed to recover a node’s degree from the drawing (Table 3). That said, a separately preregistered intervention study finds degree linearly decodable at the key entry. This suggests that the information is present in the activations, but we decline to speculate as to whether the failure on graph lies in reading the drawing or in what the model does with what it has read, and the paper reflects this.

The random slope: under key resolution the fit gives p = .77 because all of Claude Opus 4.8’s errors on tables fall in a single cell (contain direct-count), where the diagram’s legend states a convention the sentences do not (see Appendix D, entry 4). With that cell excluded, the estimate is −1.17 (SE 0.16), and −1.13 to −1.40 leaving out any one model. Admittedly, five models remain a small sample, as we note in §8.

Parse failures: a readable answer is recovered from diagrams less often than from tables or sentences (b = −1.37; Appendix C), and the design cannot separate this from the reported accuracy deficit, as acknowledged in §8. Key resolution and name recovery recover two kinds of answer that the registered scorer missed, namely answers given as letter labels and answers phrased as sentences.

Study 2 has been removed and will be reported separately.

Reviewer_EfS8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the impact of different information representation formats on the reasoning accuracy of language models. The authors construct four types of spatial reasoning tasks and present identical information in three formats: ASCII diagrams, tables, and natural language. Experiments are conducted on five models. The results show that models generally achieve lower accuracy with diagrammatic representations than with the other two formats. Additionally, the authors attempt to investigate internal spatial representations through linear probing, but the results remain inconclusive.

Critical strengths
  1. Meaningful research question. The study contributes to understanding how different information representation formats affect the performance of language models.

  2. Experimental design. The experiments cover multiple spatial reasoning tasks and models, and the inclusion of tables as a control condition provides additional value to the experimental design.

Critical weaknesses
  1. Limited novelty in the actual research contributions. Previous studies have investigated the effects of input formats on language model reasoning performance, including ASCII diagram understanding and format sensitivity. Although this paper introduces tables as a control condition, its main findings largely confirm previously observed phenomena without clearly explaining the mechanisms underlying the performance differences. Therefore, the actual research contributions offer limited novelty.

  2. Inconclusive findings from the internal representation analysis. The paper aims to investigate whether language models possess and utilize iconic internal representations. However, Study 2 conducts linear probing experiments only on Qwen2.5-1.5B and ultimately fails to distinguish iconic from propositional representations or explain the accuracy differences observed in the behavioral experiments. Consequently, the paper does not adequately address its stated research question concerning internal representations, and the independent contribution of this study remains limited.

  3. Poor organization of tables and insufficient discussion of related work. Several tables presenting major experimental results lack numbering and captions, including those in Sections 5.1, 5.2, and 5.5. The table in Section 5.2 is even split across different pages, affecting readability and presentation quality. Furthermore, although the Related Work section cites numerous classical studies in cognitive science and representation theory, it provides insufficient discussion of recent studies directly addressing language model input formats, ASCII understanding, and format bias. This weakens the paper's justification of its novelty.

Questions for authors
  1. Compared with previous studies, what new theoretical or empirical findings does the three-format comparison provide?

  2. Given that Study 2 fails to identify the internal representation format of the model, could the authors further clarify its independent contribution and explain how it supports the paper's primary research objectives?

  3. Could the authors standardize the numbering and captions of the main experimental tables and provide additional comparisons with recent directly relevant studies to better establish the paper's novelty?

1 comment

Thank you for the comments. Re: novelty, we believe that there are two results that go beyond the finding that models are sensitive to the format of their input. First, there is no difference between the sentential and tabular formats (b = −0.01), but there is a large difference between the tabular and diagrammatic formats, even though both are monospaced, symbol-dense, and two-dimensional. The format work we are aware of does not hold these surface properties fixed while varying the iconicity of the representation. The second new result is that asking for reasoning helps much less with diagrams (+2.3 percentage points outside the graph family, key-resolved) than with tables and sentences (+9.3 and +11.2 percentage points, respectively). The direction of the deficit is also informative. Larkin and Simon predict that tasks should be easier to answer from a diagram than from sentences stating the same facts, and two recent studies find that ASCII layouts do help text-only models reason spatially, relative to prose descriptions (Huang et al., 2026; Rajpal et al., 2026). Neither of those studies had a tabular control, however, so it is unclear whether the benefit came from iconicity or from other features of ASCII layout. These comparisons are now in §1–2.

Study 2 has been removed and will be reported separately, and all tables are now numbered and captioned.

Comments on the paper

1 comment

We thank all the reviewers for their considered and helpful comments. The revision makes the following changes. First, Study 2 has been removed and will be reported separately, since three reviewers found its contribution limited. Second, the abstract and introduction now lead with the key-resolved estimate. Third, §2 now discusses work on input-format sensitivity, along with two recent studies in which ASCII layouts improve spatial reasoning relative to prose descriptions. Fourth, we ran an additional, unregistered check on the informational equivalence of the renderings. Specifically, a deterministic parser for each rendering recovered the generator’s structure exactly from every rendering of every item. Fifth, §5 has been rewritten, with the scorers and exclusion frames defined at the outset, and all tables are now appropriately numbered and captioned, including figure two, which now has 95% intervals. Sixth and finally, §8 addresses the choice of text-only models and the disabling of internal reasoning, and distinguishes the costs that are relevant to iconicity from those that are incidental to it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.