acceptodds
Under review as a conference paper at ICLR 2027

Beneath the Surface: Investigating LLMs' Capabilities for Communicating with Subtext

Abstract

Human communication is fundamentally creative, and often makes use of subtext—implied meaning that goes beyond the literal content of the text. Here, we systematically study whether language models can use subtext in communicative settings, and introduce three new evaluation suites to assess these capabilities. Our evaluation settings range from writing & interpreting allegories to playing multi-agent and multi-modal games inspired by the rules of board games like Dixit. We find that out of the box, frontier models exhibit a bias towards overly literal, explicit communication. However, a shared common ground between agents can significantly help reasoning models like Gemini-2.5-pro and GPT-5 to communicate subtext to an intended audience sharing the common ground. In our environment Visual Allusions, we find 25%-40% (absolute) reduction in literal clues with a common ground of short stories. Similarly, in our storytelling environment The Aesopian Author, an author agent's success rate in communicating a forbidden idea to a critic, which an inquisitor interprets as benign, drops from 54% to 28% when critic loses the common ground with the author. In The Aesopian Author, we also find stronger models can convey subtext that a weaker monitor model might miss. For allegory understanding, we find paratextual and persona conditions to significantly shift the interpretation of subtext. Overall, our work provides quantifiable measures for an inherently subjective phenomenon like subtext and reveals both strengths and weaknesses of current LLMs in these settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
subtext, creativity, multi-agent communication, multi-modal evaluation, pragmatics, common ground

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_N1QKOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether LLMs can communicate through subtext, i.e., convey implied meaning beyond the literal content of a text. The authors introduce three evaluation environments: Visual Allusions, Historical Allegories, and The Aesopian Author. Through these experiments, the authors find that frontier models, out of the box, tend toward overly literal and explicit communication. However, when agents share a common ground, reasoning models such as Gemini-2.5-Pro and GPT-5 reduce the proportion of obvious clues by 25–40% (absolute). Allegory interpretation is also substantially influenced by paratextual factors, such as the author's identity and the reader's persona.

Critical strengths

1. Well-designed and thorough experiments. The environments cover complementary aspects of subtext, ranging from selective communication in a board-game setting to interpreting and generating historical allegories. I particularly appreciate how the authors operationalize a highly subjective notion like subtext as measurable communicative outcomes.

2. An interesting and meaningful research question. Rather than asking only whether LLMs can understand implicit meaning, the paper asks whether they can convey meaning selectively to different audiences, an important aspect of human communication closely tied to pragmatics and Theory of Mind. Several findings stand out to me, including models’ tendency to produce overly obvious clues, the limited evidence that they spontaneously infer common ground, and the effect of author and reader identities on allegory interpretation. These results are interesting and point to specific limitations in how LLMs handle implicit communication.

Critical weaknesses

I do not have any major concerns about the experimental design or findings, but I have two minor suggestions:

  1. Some figures, particularly Figures 2 and 3, contain relatively small text and could benefit from larger fonts to improve readability.

  2. I also wonder whether the proposed environments could serve as a useful starting point for future mechanistic interpretability studies of subtext understanding. For example, the Historical Allegories experiments show that contextual cues can substantially change a model's interpretation of the same story, raising an interesting question of whether these cues alter the internal representation of implicit meaning or primarily affect how existing information is used in the final interpretation. I think a brief discussion of such directions could further highlight the broader value of the proposed benchmarks.

Questions for authors

None

1 comment

Thank you so much for going through our paper and providing thoughtful comments. We are glad that you found our experiments well-designed and thorough and the research question as interesting and meaningful. We have noted down your comments and will be address the size of figures in the revision. We also agree that the historical allegories task can be fruitful for the mechanistic interpretability studies. We currently have a single line in the Discussion section (line 478) that alludes to this but we will make sure to extend it as you suggested in the revision.

Reviewer_uHf4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper explores whether modern large language models (LLMs) can create and understand hidden meanings in communication. It introduces three tests: the visual social game Visual Allusions, the artificial historical stories in Historical Allegories, and the multi-agent storytelling game The Aesopian Author. The results show that models often give clues that other agents can understand, but they communicate more selectively when they know which information they share. The study also finds that culturally familiar names and different reader perspectives can affect how models identify historical events. Although these experiments are creative and potentially useful, the current results do not clearly prove that the models are using or understanding hidden meanings. The observed effects could instead come from task difficulty, prompts influencing model behavior, information leakage, random disagreements between models, or general ambiguity.

Critical strengths
  1. Transparency: The appendices report many prompts, model parameters, game rules, synthetic data procedures, example generations, and an artifact link. The paper also explicitly discusses dual-use concerns and the limitations of interpreting reasoning traces.
  2. Broad model coverage: The experiments include several model families and capability tiers. Historical Allegories used several types of hard negatives and reverses answer order; and Visual Allusions includes limited human check.
Critical weaknesses
  1. In Visual Allusions, the terms "obvious," "obscure," and "just-right" are defined based on how many players choose the correct card. These categories describe the results of the game, not the type of language used in the clue. The main problem is that "just-right" is a very broad and unclear category because it groups together all outcomes between zero and all players recognizing the card. For example, if there are ten possible receivers, a clue understood by only one person and a clue understood by nine people are both labeled "just-right," even though one is almost obscure and the other is almost obvious. Even in the default four-player game, where there are three receivers, the results of one and two correct guesses receive the same label, despite showing different levels of accessibility. As a result, the "just-right" percentage hides important information about how well a clue controls who understands it. Models with very different communication behaviors may therefore appear equally successful. At a minimum, the authors should report the full distribution of how many receivers correctly identify each clue. If a clue is intended for a specific partner, they should also report how often that partner understands it and how often unintended receivers do so.

  2. Model specification, deployment, and context accounting are inadequate: The paper lists most of the evaluated models and some settings, such as temperature, output limits, and reasoning budgets. However, the information is incomplete and does not allow a fully reproducible or fair comparison. First, the paper does not explain how Gemma-3-27B was run, including its hardware, software, and configuration. Second, the paper reports output limits but not input context limits, actual prompt lengths, or truncation details. Since experiments use different amounts of conversation history, these differences could affect the results. Third, models use different reasoning budgets and output limits. Therefore, performance differences may come from these settings rather than model capability alone. Finally, the paper calls models "stronger," "weaker," or "bigger" without clear evidence or a consistent definition of capability. For example, Gemini-2.5-Flash-Lite is described as weaker mainly because it costs less per token. The authors should report exact model versions, deployment details, input/output token statistics, truncation and retry rates, and a clear separation between model identity and inference settings.

  3. Evaluation and scoring procedures are insufficiently documented: The paper does not explain how open-ended responses from critics, deep-readers, and historians are converted into accuracy scores in Tables 2, 10, and 11. Without clear scoring rules, the results are difficult to reproduce, and potential scoring bias cannot be assessed. Human validation is also limited, as both checks rely on a single co-author without clear annotation guidelines, blinding, or reliability measures. These checks are useful but do not provide strong independent validation.

Questions for authors

Please refer to weaknesses

6 comments

Thank you for going through our work and providing thoughtful comments. Below we try to address your concerns:

  1. Concerning obvious, obscure, and just-right clues as a measure

We agree that measuring the fraction of the three types of clues can hide important information about the behavior of the agents in the game. That’s the reason why we also provide many more measures in addition to these. In the paper, we introduce a measure called Spark Coefficient, which exactly measure what you stated in your review: “If a clue is intended for a specific partner, they should also report how often that partner understands it and how often unintended receivers do so”. In lines 181-185, we define the spark coefficient between two players as the rate at which the two players selectively communicate with each other during the game. A very high spark coefficient (close to 1) between player i and player j, means that clues given by player i were only correctly guessed by player j, and vice-versa. The metric was specifically used to study the behavior of models sharing the common ground and whether the two models sharing the common stories create clues that are only correctly guessed by the player with access to these stories. These results are provided in Figure 2c and discussed in the lines 244-247, and we find the highest spark coefficient for Gemini-2.5-pro, which while is low in the absolute value i.e. 0.20, but is statistically significant, meaning the rate at which the model’s clues are recognized by the partner player sharing the same stories is higher than other unintended players.

In lines 199-206, we also describe the human validation experiment, where the human annotator was shown the voting deck and the clue generated by an LLM during the gameplay. The annotator is then asked to choose, which card corresponds to the storyteller’s card, and we measure how often the human annotator correctly guesses the storyteller’s card based on the clue alone. This helps us to measure whether the model generated clues are literal or unambiguous enough to be correctly decoded by a human annotator. Hence, this measure is explicitly about the language used in the clue and goes beyond the results of the game. We find that on average 91% of the times the human annotator is able to correctly guess the cards selected by different models.

Given the review, we also ran some new experiments to measure the distribution of the number of receivers correctly guessing the clues for different models. The table below shows results from 4 models in 20-game tournament in the regular case without shared context (each game has 4 players, hence 3 maximum receivers for a sender):

Storyteller# cluesMean # correct0 correct1 correct2 correct3 correct
GPT-5872.801.1%4.6%6.9%87.4%
GPT-5-mini892.960.0%1.1%2.2%96.6%
Gemini-2.5-Flash912.980.0%0.0%2.2%97.8%
Gemini-2.5-Pro882.830.0%4.5%8.0%87.5%
All3552.890.3%2.5%4.8%92.4%

As we can see, the average number of receivers that correctly guess the clue are very close to the maximum value of 3, which is also evident from the mode of the distribution being at 3 (and with a big margin).

We also ran experiments for the shared context case, following the same experimental procedure as detailed in the paper but due to the shortage of time, used 5 games per model instead of 20:

Partnered storyteller# cluesMean # correct0123Partner correctOutsider correctPartner-only
Gemini-2.5-Flash462.724.3%4.3%6.5%84.8%89.1%91.3%2.2%
Gemini-2.5-Pro361.7522.2%16.7%25.0%36.1%72.2%51.4%13.9%
GPT-5362.195.6%22.2%19.4%52.8%80.6%69.4%8.3%
GPT-5-mini371.8424.3%16.2%10.8%48.6%59.5%62.2%8.1%

For models like Gemini-2.5-Pro, GPT-5, GPT-5-mini, the mean players correctly guessing the clue falls significantly, and while the mode of the distribution remains at 3, the other values now also have non-trivial amount of coverage. For Gemini-2.5-Pro and GPT-5, the rate at which the partner player (“intended receiver”) correctly guesses its clues is also significantly higher than the unintended players, consistent with the results with the spark coefficients.

We hope this addresses the first concern of the reviewer.

Response continues in the next comment

  1. Model specification, deployment, and context details.

About Gemma-3-27b, we used the google ai studio API to access it, instead of locally hosting the model. Thank you for bringing this up, we will explicitly state this in the final version. Concerning the input context limits, while we do cite the technical reports or model cards with the information, we also detail them below for the models used in the paper

  • Gemini-2.5-Pro, Gemini-2.5-Flash, Gemini-2.5-Flash-lite: 1,048,576
  • GPT-5, GPT-5-mini: 400,000
  • GPT-6-sol:1,050,000
  • Claude Sonnet 4.5, Claude Haiku 4.5: 200,000
  • Gemma-3-27b: 128,000

We would make sure to include these in the final version.

The prompt lengths for different models on our environments are also provided below:

Visual Allusions in the Regular as well as Shared Context Case We report the number of prompt tokens for each of the three stages in the game i.e. storyteller, card play, and vote. Regular denotes the case without any shared context and Shared is for the shared context case.

ModelRegular: StorytellerRegular: CardplayRegular: VoteShared: StorytellerShared: CardplayShared: Vote
GPT-5624762094380233902335821529
GPT-5-mini839583575812255382550622961
Gemini-2.5-Pro227422761749198941990419376
Gemini-2.5-Flash227422761749198941990419376
Gemma-3-27B227622801753198961990619379
Claude Sonnet 4.5968595846713288332874025868
Claude Haiku 4.51003299707099291802912626254

The gemini and gemma models have significantly smaller prompt tokens for each stage. This is attributed to the image tokenizers of these models being more efficient in terms of number of tokens compared to the other models. This can potentially explain the slight edge in performance that we observe for Gemini-2.5-pro over GPT-5 and Claude Sonnet 4.5.

Aesopian Author

ModelAuthor (write)Author (reflect)CriticInquisitor
Gemini 2.5 Pro3368468729461188
Gemini 2.5 Flash53977172––
Gemini 2.5 Flash-Lite–––1189
GPT-6 Sol19132782––
GPT-547636702––
GPT-5-mini38455114––
Claude Sonnet 4.634595063––

The numbers here are average across different rounds during the gameplay. Note that for Critic only Gemini-2.5-pro was used for the experiments, hence the rest of the cells are empty for Critic. Similarly, for inquisitors we considered Gemini-2.5-pro and Gemini-2.5-Flash-Lite only. The two numbers for the author are for the writing stage where the agent is instructed to write a new story and the reflection stage where it is given the interpretations of the two agents and asked to devise a strategy for the next round of the game. Critic’s number of tokens are almost thrice the amount for the inquisitor because the critic also is given the literary history of the author agent during the game. GPT-6-sol has a notably smaller number of tokens as the author agent, as the model tends to generate shorter stories and reflections, and models like GPT-5 and Claude Sonnet 4.6 have much wordy stories and reflections (we will add the token lengths for generations in the revision too, but emit here for brevity).

Historical Allegories

ModelCriticHistorianDeep Reader
GPT-5121012141203
GPT-5-mini121012141203
Gemini-2.5-Pro119912021191
Gemini-2.5-Flash119912021191
Gemma-3-27B120712101199
Claude Sonnet 4.5137813831370
Claude Haiku 4.5137813831370

We report the average number of prompt tokens considering all of the allegories in the dataset. Since the inputs for all the models are fixed, the differences in the number of tokens across models are low.

We will make sure to include all these numbers in the revision.

Response continues in the next comment

3 About the use of terms like “weaker”, “stronger”.

We agree that in the current manuscript the terms have been used loosely without sufficient justification. In Section 4.2 on the Aesopian Author environment, we used gemini-2.5-flash-lite for a less capable (weaker) model for the inquisitor agent compared to gemini-2.5-pro (Lines 381-383). In the current manuscript we base this on the inference cost, but we will add in the final version, the benchmark comparison from the Gemini-2.5-Flash-Lite [1] model card, which shows that the model underperforms Gemini-2.5-Pro on established benchmarks like including GPQA Diamond, AIME 2025, and Humanity's Last Exam. We agree that even these could be considered as imperfect proxies, and we will highlight them as such in the paper. The other place where this terminology is used is for the common ground experiments in the Visual Allusions section (Lines 244-247), where we state the stronger models like Gemini-2.5-pro, GPT-5, and Sonnet 4.5 outperform other models. In the revision, instead of using the term stronger, we will replace it with “more recent reasoning models”.

4. Scoring for Historical Allegories Task. Thanks for asking this question. We use an LLM-judge (Gemini-2.5-flash) to evaluate whether the analysis generated by the models explicitly references the historical event that was used to generate the allegory originally. Specifically, the following prompt was used:

Please read the following analysis and determine if it correctly and explicitly identifies the historical event "{event_name}" as a historical subtext or allegory in relation to the story. Enclose your decision in <decision></decision> tags (either "Yes" or "No") and your explanation in <explanation></explanation> tags.

Analysis:
{analysis}

We also tried a key-word match approach to check if original historical events are mentioned in the model’s analysis. We found it to underestimate the performance of the models, but the relative trends between persona and paratextual conditions were identical.

We agree that this is an important detail and we will make sure to include this in the revision.

[1] https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-2-5-Flash-Lite-Model-Card.pdf?utm_source=chatgpt.com . Page 4

Reviewer_fBDrOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary
  • This paper investigates LLMs can use subtext to communicate. Subtext is defined as content that is communicated between two agents that is not explicitly stated in an uttered message.

  • The paper devises three evaluation environments for investigating subtext generation in LLMs: visual allusion, historical allegory, and the aesopian author.

  • The paper uses applies multiple interventions to understand how LLMs attempt to communicate subtext in different settings. The findings are intuitive. Common ground that is shared by the author and audience helps with communicating subtext. Para-textual information helps communicate subtext.

  • While our findings suggest that current frontier models out of the box tend to struggle with these tasks and are often too explicit with their communication, the biggest reasoning models do tend to outperform their smaller counterparts, especially when provided with a common ground, hinting towards improving capabilities of LLMs towards creative communication.

Critical strengths
  • The paper is well-written and easy to follow.
  • The paper presents an interesting and novel environment for investigating subtext use in language models
  • The paper makes multiple interesting observations such as (1) stronger models can use subtle subtext that weaker models cannot recognize (2)
Critical weaknesses
  • I imagine the subtext that an agent can communicate can be sensitive to the historical allegories it has in its context. Is there an analysis of how communicable subtext changes as you change the historical events in the author's prompt? Are there examples of agents composing historical events in creative ways to communicate subtext that is markedly different from any of the individual historical events?
Questions for authors
  • Why doesn't seeing the current story only have much of an impact when the gemini 2.5 lite inquisitor is used instead of 2.5 pro (figure 3b)? Shouldn't we expect the success rate to decline in this scenario as well?
  • The reporting of "obvious", "obscure", and "just-right" feel a bit coarse. Does the paper also report the fraction of the receiver agents that understood different meanings from the communicated text?
3 comments

Thank you for going through our paper and providing thoughtful comments. We are glad that you found our proposed environments interesting and novel, and found the paper to make multiple interesting observations. Below we try to answer your concerns and questions:

1. About Sensitivity of Subtext to In-context Allegories.

We are unsure, which experiments are you specifically talking about here but our best guess is that the question is about the experiments on Visual Allusions environment with two players sharing a common set of stories (these stories are fictional stories from Tell Me A Story Dataset [1]). We will answer the question based on this guess, but we are happy to respond again if there is a misunderstanding on our end. About the sensitivity of the results on Visual Allusion to the set of stories provided in-context, we did not explicitly test for that in the current paper. We do provide examples where models used the stories in context to craft clues that compose elements of those stories to visual elements in the cards. In lines 249-252, we describe a case where the model Gemini-2.5-pro, chose a card with a Turkey on it and provided a clue referencing a story involving thanksgiving dinner: “The family meeting was even more awkward than I remembered” , which the other player sharing the same story in the context was able to recognize and not the other players.

For the discussion period, we also ran some new experiments to understand the impact of the number of shared stories provided to the two partner players on the performance. Similar to the experimental setup of the paper, we had 2 gemini-2.5-pro players sharing the stories compete with 2 copies of gemini-2.5-flash based players, and ran 5 games per a given number of stories. The results are detailed in the table below:

# StoriesPair scoreOpponent scoreGapSpark (partners)Spark (other pairs)
028.1 ± 1.125.7 ± 3.02.40.02−0.01
228.3 ± 2.125.3 ± 4.43.00.07−0.01
527.6 ± 2.923.6 ± 5.04.00.03−0.01
1029.3 ± 0.821.7 ± 4.77.60.21−0.04
2029.9 ± 1.622.0 ± 4.07.90.19−0.04

When the number of shared stories is low i.e. <=5, we see very low values of Spark Coefficient between the partner players, meaning they fail to communicate exclusively with each other during the game by making use of the shared context. As more stories are provided, there is a significant jump in the spark coefficients. A similar jump is also observed in the gap between the overall score on the game for the partner players (gemini-2.5-pro models) sharing the context and the opponent players (gemini-2.5-flash models) that lack it.

2. Response to Question 1. Why doesn't seeing the current story only have much of an impact when the gemini 2.5 lite inquisitor is used instead of 2.5 pro (figure 3b)? Shouldn't we expect the success rate to decline in this scenario as well?

We explain this in the paper in lines 416-420. We find that when we use Gemini-2.5-Flash-Lite as the inquisitor agent, it tends to miss out on some minor details in the story, which are picked up by a more capable critic agent i.e. Gemini-2.5-pro. Hence, the success rate is already high at 84% without requiring literary history. Compare this to when we used Gemini-2.5-pro as the inquisitor, where even with the literary history the success rate at the best case remains at 54%. Hence, providing the literary history when using Gemini-2.5-flash-lite as the inquisitor only marginally improves the success rate from 84% to 86%, because the base rate is already quite high.

3.Response to Question 2. The reporting of "obvious", "obscure", and "just-right" feel a bit coarse. Does the paper also report the fraction of the receiver agents that understood different meanings from the communicated text?

Thank you for this question. We ran some new experiments, to get the distributions of the number of receivers understanding the generated clues in Visual Allusions. The table below shows results from 4 models in 20-game tournament in the regular case without shared context (each game has 4 players, hence 3 receivers):

Storyteller# cluesMean # correct0 correct1 correct2 correct3 correct
GPT-5872.801.1%4.6%6.9%87.4%
GPT-5-mini892.960.0%1.1%2.2%96.6%
Gemini-2.5-Flash912.980.0%0.0%2.2%97.8%
Gemini-2.5-Pro882.830.0%4.5%8.0%87.5%
All3552.890.3%2.5%4.8%92.4%

As we can see, the average number of receivers that correctly guess the clue are very close to the maximum value of 3, which is also evident from the mode of the distribution being at 3 (and with a big margin).

We also ran experiments for the shared context case, following the same experimental procedure as detailed in the paper but due to the shortage of time, used 5 games per model instead of 20:

Partnered storyteller# cluesMean # correct0123Partner correctOutsider correctPartner-only
Gemini-2.5-Flash462.724.3%4.3%6.5%84.8%89.1%91.3%2.2%
Gemini-2.5-Pro361.7522.2%16.7%25.0%36.1%72.2%51.4%13.9%
GPT-5362.195.6%22.2%19.4%52.8%80.6%69.4%8.3%
GPT-5-mini371.8424.3%16.2%10.8%48.6%59.5%62.2%8.1%

For models like Gemini-2.5-Pro, GPT-5, GPT-5-mini, the mean players correctly guessing the clue falls significantly, and while the mode of the distribution remains at 3, the other values now also have non-trivial amount of coverage. For Gemini-2.5-Pro and GPT-5, the rate at which the partner player (“intended receiver”) correctly guesses its clues is also significantly higher than the unintended players, consistent with the results with the spark coefficients.

We hope this answers your question.

Reviewer_fBDrOpenReview ↗

Response to Authors

I thank the authors for their thoughtful responses. The new experiments on Visual Allegory are helpful (that is the game I was referring to) and the breakdown of the correct share of the Visual Allegory game is also helpful to see. As I have given the maximum, I keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.