acceptodds
Under review as a conference paper at ICLR 2027

Do LLMs Act on What They Know? From Partner Representations to Cooperative Actions

Abstract

Cooperation with unfamiliar partners requires adapting to communication conventions that are not known in advance. We study this problem in a controlled Hanabi-derived environment with scripted hint generation, LLM-controlled receiving decisions, and frozen model weights. Across eight LLMs, linear probes recover intent conventions substantially more accurately than target conventions, yet receiving choices do not consistently agree with the sender's convention. We compare probe-predicted and ground-truth conventions presented either as general rules or as externally computed action recommendations. Rule statements yield modest and model-dependent changes in cooperation, whereas action translation produces larger gains on average. In a Qwen3-8B case study, matched-state statement reversals reveal much greater sensitivity to action recommendations than to rule statements. Activation transfers from oracle-action and non-oracle hint-restatement donors improve intent accuracy on both action classes, but the tested alternatives do not reliably reproduce these benefits. Together, these results distinguish convention decodability, sensitivity to convention information, and cooperative performance, and highlight limitations in turning available partner information into receiving decisions.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM, unfamiliar partner, ZSC, zero-shot coordinate

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SNXkOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper provides an analysis of the inferences that LLMs make in social interactions, through the lens of a game similar to Hanabi. In this setting agents can provide a hint to a game partner, where the hint follows a convention about which card is targeted and how the hint should be interpreted. The other agent then has to make an inference about the appropriate action. Probes are used to identify the extent to which the models seem to be representing the intent and target information and formulating the correct response. A total of 8 LLMs are evaluated in this setting via these measures, both when paired against the same LLM or against a different LLM. Further experiments explore how performance is affected by intervening with an appropriate activation vector or fine-tuning. The key results are that intent is recovered more accurately than target, and interventions via activation transfer and fine-tuning both improve performance.

Critical strengths

The main strength of this paper is its in-depth analysis of convention inferences in a controlled setting. In addition, the use of multiple converging methods -- not just probing but also activation transfer and fine-tuning -- provides deeper insight than would have been obtained with probing alone.

Critical weaknesses

The primary weakness of the paper is that it has relatively limited scope. While the use of the constrained domain makes it possible to carefully analyze the behavior of the model, I would want to see that similar results hold outside this limited setting.

Another significant limitation is that there are no reported statistical significance tests or error bars on most of the critical results (although some results are marked with an asterisk, which may denote statistical significance but I couldn't find an explanation in the text). All qualitative statements about the results (such as one model or method producing an improvement over another) need to be supported by statistical analysis. At present the key results depend entirely on reported differences in proportions and are difficult to evaluate. This applies to results in 5.2-5.7 and Tables 1-4, and Figures 4 and 5 both need error bars.

There is not a detailed discussion of the limitations of the work, which could give the impression that the results are broadly generalizable.

Questions for authors

For making the claim that models are worse at inferring target than intent, it seems like the relevant baselines should be included. For example, there may be a smaller set of possible targets than possible intents, so the task is simply easier. How do you correct for this?

3 comments

We thank the reviewer for the careful reading and for finding the converging methods useful. Changes for this review have a green highlight in the revised PDF, on top of the other reviewer's text color where that reviewer raised the same point. This comment addresses W1, the next addresses the Question and W3, and the last addresses W2.

W1. "The primary weakness of the paper is that it has relatively limited scope. While the use of the constrained domain makes it possible to carefully analyze the behavior of the model, I would want to see that similar results hold outside this limited setting."

The central contrast is not specific to one model, and it holds under both single-token answers and free generation. In every LLM we evaluated, giving the true convention as a rule (told) changes intent doing by −0.015 to +0.109, while replacing the rule with the current action (told + verb) adds a further +0.056 to +0.583 (App. F, Tables 10–11, 240 games per cell), and told + verb has the highest score. These are differences between point values without per-model intervals, so we read the pattern as directional.

The contrast points the same way over pairs of different LLMs and under free generation. Over the 56 ordered cross-play pairings (App. J, board-level intervals), the mean score is 5.14 under raw, 5.41 under told, and 6.98 under told + verb, so translating the true convention into the current action raises the cross-play score by +1.57* over the rule statement. On 480 recorded Qwen3-8B states (App. C.3), under free generation the rule raises intent doing by +0.040*, and translating it into the current action raises it by a further +0.279*.

The controlled environment is what makes this checkable. The scripted sender fixes the meaning of every hint, so each decision is scored against a known convention (our response to LnLx, 3/3). Extending the test to other cooperative tasks, richer or learned conventions, generated hints, and more than two players is future work, and App. K states this scope.

These changes are in Sec. 6 (pointer to the limitations), App. K, and App. J (cross-play means).

Question. "For making the claim that models are worse at inferring target than intent, it seems like the relevant baselines should be included. For example, there may be a smaller set of possible targets than possible intents, so the task is simply easier. How do you correct for this?"

The option sets have the same size. The target decision is the leftmost or the rightmost of the hinted cards, and the intent decision is PLAY or DISCARD, so both decisions are binary with chance 0.500. The convention labels used to evaluate knowing are also nearly balanced. At the final turn, their majority-class rates are 0.504 for target and 0.517 for intent in LLM games, and 0.500 and 0.531 on scripted boards (App. L.3, Table 21). The revision states the chance level in Sec. 5.3 and in the captions of Tables 1–2, and adds that equal chance does not make the two axes equally difficult.

The observation-feature classifier measures how readily each convention can be predicted from the recorded history without using LLM hidden states (revised prompt, new App. L.3). In LLM games at the final turn, its accuracy is 0.844 for target and 0.990 for intent, compared with 0.656 and 0.833 for the hidden-state probe. The target–intent difference therefore already appears in the observable events.

The paper's main claim does not depend on this difference. Under told, the true convention is given and no convention inference is needed (Qwen3-8B, Table 2, submitted prompt, 240 games per condition; intervals in App. J, Table 16). Target doing then rises from 0.515 to 0.821, while intent doing stays at 0.599, against 0.591 under raw. Intent, the axis the history reveals more readily, is the one the model does not consistently apply, so the rule-versus-action result on intent concerns application, not inference difficulty.

W3. "There is not a detailed discussion of the limitations of the work, which could give the impression that the results are broadly generalizable."

The limitations section (App. J in the submission, App. K now) was not referenced from the main text. The conclusion (Sec. 6) now points to it, and the revision extends it with two items. One states that knowing is recoverability by an external linear probe, which does not distinguish observable cues retained in the representation from a convention estimate formed by the LLM, and that the behavioral contrasts separately test how supplied conventions change decisions. The other states that the response-interface check covers only Qwen3-8B on recorded states, not complete games. App. K also states that the results concern controlled receiving decisions in a two-player environment with scripted senders.

These changes are in Sec. 5.3, the captions of Tables 1–2, Sec. 6, App. K, and the new App. L.3.

W2. "There are no reported statistical significance tests or error bars on most of the critical results (although some results are marked with an asterisk, which may denote statistical significance but I couldn't find an explanation in the text). ... This applies to results in 5.2-5.7 and Tables 1-4, and Figures 4 and 5 both need error bars."

The revision adds intervals for every result the review lists. Sec. 5.1 now defines the asterisk: it marks a paired difference whose board-level 95% bootstrap interval excludes zero (20,000 resamples). Pairs share the deal and convention assignment, or the recorded state, and the four games of a board are resampled together. With these intervals, every asterisk of the submission is unchanged. The table below gives where the statistics for each result are.

ResultIn the revision
Table 1, Sec. 5.2Intervals for every cell of seven of the eight models in App. J, Tables 14–15 (the hidden states of Qwen3-32B were not retained)
Table 2, Sec. 5.3Intervals for every cell in App. J, Table 16; asterisks on the score differences quoted in Sec. 5.3, with their intervals in App. J
Figure 4Error bars for doing and score
Figure 5, Sec. 5.4Single-cell interval half-widths in the caption (0.14–0.29 points); asterisks on the cross-play differences quoted in Sec. 5.4, with intervals for the 56-pairing means in App. J
Table 3, Sec. 5.5Asterisks on the mean aligned Δ in the table; intervals for every cell in App. J, Table 17
Table 4, Sec. 5.6Asterisks in the table; intervals for every cell in App. J, Table 18
Sec. 5.7Asterisk on the quoted adapter difference; score intervals in App. J, Table 19, and the adapter comparison in App. I

Every comparison stated in Secs. 5.2–5.7 holds with these intervals. Recomputing Tables 1–4 and Figures 4–5 from the recorded runs reproduces every published value except five entries of Table 4, which now use the recomputed values. These differ from the submitted ones by at most 0.005, with no change in sign or conclusion (App. J). App. F now states the sources of its Gemma-3-12B row and of the Fig. 5 self-play cells.

Reviewer_LnLxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates the gap between what LLMs know about their partners and how they act on that information in cooperative settings. Using a controlled Hanabi-derived environment, the authors evaluate eight LLMs and distinguish between two aspects: (1) knowing, whether a partner's communication conventions can be decoded from the model's hidden representations, and (2) doing, whether the model's receiving decisions are consistent with those conventions. The paper compares two sources of convention information, namely predictions from linear probes and ground-truth conventions, and presents this information either as general rule statements or as explicit action recommendations.

Critical strengths

The paper explicitly separates the decodability of partner conventions, the consistency of receiving decisions with those conventions, and the resulting game performance. This distinction provides a useful framework for analyzing LLM cooperation beyond aggregate task performance.

Critical weaknesses
  1. Limited experimental scope and insufficient comparisons with existing partner-reasoning methods. All experiments are conducted in a single Hanabi-derived environment with a small set of predefined communication conventions and scripted hint generation. The evaluation primarily focuses on controlled receiving decisions rather than fully autonomous multi-agent cooperation. Moreover, the main comparisons involve different information-presentation conditions designed by the authors. The paper does not directly compare against representative partner-reasoning methods, such as ProAgent, under a unified evaluation setting. In particular, it remains unclear whether an existing LLM agent equipped with explicit partner reasoning or task decomposition could address the convention-application problem without relying on the activation interventions investigated in this paper. Given these limitations, it is unclear whether the observed knowing–doing gap generalizes to other cooperative tasks, more complex and diverse communication conventions, or partners employing different communication strategies.
  2. Linear probe accuracy provides limited evidence for the interpretation of the knowing–doing gap. The core measurement of knowing relies on supervised linear probes trained on concatenated hidden states from all model layers. However, high probe accuracy does not necessarily establish that the LLM itself has formed a partner-convention representation that it can directly use for action selection. The hidden states may simply encode observable information from the interaction history, from which an external classifier can infer the partner's convention. The paper lacks comparisons with simpler convention-inference baselines, such as directly predicting partner conventions from observable interaction histories or using text-based convention inference. These limitations weaken the evidence supporting the interpretation of the observed knowing–doing gap, especially regarding what the probe results establish about the model's internal knowledge of partner conventions.
Questions for authors
  1. Could the authors compare the hidden-state linear probes against simpler convention-inference baselines, such as methods that directly infer the sender's convention from observable interaction histories? Such comparisons would help clarify what additional information is captured by the hidden-state probes.
  2. How would the main results change if the LLM were explicitly instructed to first infer the sender's convention and then apply it step by step to the current hint? Including a simple Infer-then-Act baseline that does not rely on ground-truth conventions would help clarify whether external action translation is necessary to address the convention-application problem.
4 comments

We thank the reviewer for the constructive questions and for finding the separation of knowing, doing, and score useful. Changes for this review are in blue in the revised PDF, or purple where tXrA raised the same point. We ran the analyses requested in Q1 and Q2. Our response is split into three comments covering the Infer-then-Act baseline, a decomposed pipeline without ground truth, and the observation baselines and scope. The new analyses use Qwen3-8B and report target and intent separately.

Q2. "How would the main results change if the LLM were explicitly instructed to first infer the sender's convention and then apply it step by step to the current hint? Including a simple Infer-then-Act baseline that does not rely on ground-truth conventions would help clarify whether external action translation is necessary to address the convention-application problem."

Without a supplied partner convention, target doing remained near chance across the tested response forms, while thinking improved intent doing without external action translation. We compared Infer-then-Act with plain thinking and single-token answers on the same 120 receiving states. Infer-then-Act adds both a worked example and a six-step procedure to plain thinking. Every target interval includes 0.500, whereas the true rule in NOTES gives 0.778 target doing in complete games on the same boards (App. L.4). In the pre-specified comparison, the additional changes of Infer-then-Act over plain thinking were +0.033 on target and +0.083 on intent, with both 95% intervals including zero. In the two comparisons with single-token answers, which were added after the fact, intent doing increased by +0.200* under plain thinking and +0.283* under Infer-then-Act, without a supplied ground-truth convention for the partner. These results evaluate the final decisions rather than the convention inferred during reasoning.

Setup (new App. L.4 gives the prompt additions verbatim, the decoding settings, intervals, and per-action results). Qwen3-8B answers 120 receiving states fixed before any query, from 30 evaluation boards (60 target and 60 intent questions, 30 per correct answer), with the same option order in every response form. The prompt contains the public history and the receiver's own sending convention, but supplies no ground-truth or probe-predicted convention for its partner. Plain thinks for up to 8,192 tokens. Infer-then-Act also adds a worked example to the rules and, immediately before the question, a six-step procedure that reconstructs the revealed cards, keeps the conventions consistent with them, and applies that reading to the current hint. The reference is the single-token answer on the same states. Only the final choice is scored.

Results (point values):

Response formTarget doingIntent doing
Single token0.5500.433
Plain thinking0.5170.633
Infer-then-Act0.5500.717

The test uses the revised prompt built after submission (same game, conventions, and boards), and its procedure adopts ProAgent's analyze-then-act structure (App. L.4). Of the 120 Infer-then-Act traces, 39 reached the 8,192-token budget, and values for the closed traces are in App. L.4.

W1. "It remains unclear whether an existing LLM agent equipped with explicit partner reasoning or task decomposition could address the convention-application problem without relying on the activation interventions investigated in this paper."

The decomposed pipeline improves receiving decisions without activation interventions or a supplied ground-truth partner convention. In complete Qwen3-8B games, the LLM judges individual public events and code aggregates those judgments into a rule used for the final decision. This directly tests whether explicit task decomposition can support convention-dependent behavior in our setting. The pipeline is a diagnostic case of reasoning support, while a unified comparison with a full ProAgent implementation remains future work.

The pipeline (new App. L.5). Code pairs revealed cards with the partner hints that marked them and derives each card's marked end and playability at the time of the hint. The model then judges, with thinking, whether these public-event facts rule out each candidate convention. Code aggregates those judgments and writes into NOTES each convention component on which the highest-scoring hypotheses agree. The model makes the receiving decision after thinking for up to 1,024 tokens. No step uses the partner's ground-truth convention.

Results (revised prompt, 30 boards, 120 games per row, point values).

ReceiverDecisionScore (0–15)Target doingIntent doing
UnassistedSingle token5.480.5170.593
UnassistedThinking, 1,024 tokens5.780.5120.619
Pipeline, no ground truthThinking, 1,024 tokens7.310.7120.891

The pipeline is above the unassisted single-token receiver by +1.82* in score, +0.192* in target doing, and +0.292* in intent doing. Against the unassisted receiver with the same final thinking protocol, the gains are +1.53*, +0.186*, and +0.266* (App. L.5, Table 27, with intervals).

The model's per-event judgments agree with the elimination rule in 0.927 of cases, and once code has written a rule, target decisions that receive it follow its end in 0.945 of cases (App. L.5). The pipeline generates about 112k tokens per game, far more than the unassisted receiver, and its gains come with this added computation. The pipeline is in the new App. L.5, with the revised prompt in App. L.1.

Q1 / W2. "Could the authors compare the hidden-state linear probes against simpler convention-inference baselines, such as methods that directly infer the sender's convention from observable interaction histories? Such comparisons would help clarify what additional information is captured by the hidden-state probes."

In the tested baselines, the hidden-state probe exceeds the TF-IDF classifier on the same prompt text in every reported column (point estimates), while a classifier using structured received-side events exceeds the probe. Thus, the observable interaction history provides strong evidence for the convention, and convention information is also linearly recoverable from the receiver's hidden states. We use knowing for this external decodability measure and separately test how supplying convention information changes decisions (Sec. 5.3).

Setup (new App. L.3, Table 21, which adds scripted test boards, turn averages, and the full feature set). Every classifier uses the probe's training turns (scripted boards), labels, and test splits. The probe keeps the paper's fixed regularization, and the others choose theirs on a validation split of the training boards. Entries are LLM games, final turn (revised prompt).

ClassifierTargetIntent
Hidden-state probe (all layers, linear)0.6560.833
Bag-of-words (TF-IDF) on the same prompt text0.6250.792
Observation features, received-side events only0.8440.990
Majority class0.5040.517

W1. "All experiments are conducted in a single Hanabi-derived environment with a small set of predefined communication conventions and scripted hint generation ... it is unclear whether the observed knowing–doing gap generalizes to other cooperative tasks, more complex and diverse communication conventions, or partners employing different communication strategies."

The single controlled environment is a deliberate choice that makes the diagnosis checkable against a known convention. A scripted sender fixes the convention, so every decision has a known correct meaning, and with a 2 × 2 convention space every candidate convention can be checked against the history. Scripted sending isolates the receiving decision, so changes in score can be attributed to it. Within the environment, every reported cell averages over all four conventions, and in all eight LLMs giving the current action moves intent doing more than giving the rule (App. F, point values).

App. K, linked from the conclusion, states this scope. Extending the test to other cooperative tasks, richer or mixed conventions, and partners whose sending is learned or generated is future work.

These changes are in the Abstract, Sec. 1, Sec. 5.2 (knowing as linear decodability), Sec. 6 (pointer to the limitations), App. K, and the new App. L.3.

Thank you to the authors for the detailed responses and additional experiments. I appreciate their efforts to address my concerns, particularly the new observable-history baselines, probe controls, and Infer-then-Act and decomposed pipeline experiments. These additions address several of my original questions and strengthen the paper.

However, my main concerns remain. First, the evaluation is still limited to a single Hanabi-derived environment with scripted hint generation, leaving the generalizability of the findings to other cooperative tasks and non-scripted multi-agent interactions unclear. Second, although the decomposed pipeline achieves encouraging improvements, it relies heavily on external structured processing and substantially greater inference costs. Without better isolating these factors, it is difficult to determine how much of the improvement can be attributed to LLM-based partner reasoning rather than external processing and additional computation.

While I acknowledge the improvements made in the revision, these limitations remain important to my overall assessment. Therefore, I will maintain my original recommendation.

Reviewer_tXrAOpenReview ↗
Rating
3(+2)
Confidence
2(−1)
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper studies whether LLMs act on the partner information they store internally. They use a two-player variant of Hanabi where the sender follows a 2x2 hint convention. The hinting is all scripted and the LLM only makes the receiving decisions, which are read off the next token logits. Linear probes can recover the intent convention, but the target convention is only near-chance, and the model's decisions largely ignore the sender's convention. Across 8 LLMs, stating the convention as a rule makes very little difference but providing the externally computed action makes a big difference. Analysis of Qwen3-8B suggests the model does encode both the hint type and the convention but doesn't combine them. A LoRA adapter learns this combination and the bottleneck then shifts to inferring the convention.

Critical strengths
  • The evaluation is wide, ranging across models and model pairs. 8 open-weight LLMs from a number of families are assessed with 56 cross-play pairs, with 240 games each.
Critical weaknesses
  • The probe is a supervised classifier on around 150k concatenated features from all the layers. That demonstrates the convention is linearly recoverable by an external readout, not that the model has a representation of it that it can use in its own computation. It is not quite right to say that "act on what they know".
  • The action translation conditions give the model the answer. In the told + verb case, the external code simply writes the correct slot and action into NOTES so the model is following instructions rather than partner adaptation. Same with loop + verb where the convention to decision step is moved outside the model.
Questions for authors

N/A

1 comment

We thank the reviewer for the clear summary and for noting the breadth of the evaluation. Changes for this review are in red in the revised PDF, or purple where LnLx raised the same point.

W1. "The probe is a supervised classifier on around 150k concatenated features from all the layers. That demonstrates the convention is linearly recoverable by an external readout, not that the model has a representation of it that it can use in its own computation. It is not quite right to say that 'act on what they know'."

Knowing is an operational measure: whether the partner's convention is linearly recoverable from the receiver's hidden states by a trained external probe. Probing studies of Othello and chess models likewise read latent board state from hidden states with trained probes (Li et al., 2023b; Nanda et al., 2023; Karvonen, 2024), and we use "know" in this operational sense. We acknowledge that the title can read more strongly, so the revision states this definition in the Abstract and Sec. 1. The behavioral contrasts measure a different quantity, how decisions change when convention information is supplied in the prompt (Qwen3-8B, submitted prompt). On 480 recorded states, stating the true convention as a rule (told) raises target doing by +0.312* [+0.265, +0.360] but intent doing by only +0.021 [−0.006, +0.048], and the PLAY rate stays at 0.758 (App. C.3). A convention given explicitly is thus applied to target decisions but not consistently to intent decisions, and this result does not rely on the probe.

The capacity controls test whether a supervised probe on this many features merely fits its labels (new App. L.2, revised prompt). All three probes fit their training turns from 48 scripted boards perfectly. At the final turn of 12 new boards, however, the probe trained on true labels reaches 0.677, compared with 0.479 when labels are shuffled across games and 0.448 when random vectors replace the hidden states. The best tested single-layer readout (layer 5) reaches 0.747, compared with 0.740 for the all-layer probe, at the final turn of the 36 scripted test boards. The probe therefore reads a convention-related signal that generalizes to new boards and can be read without concatenating all layers, rather than merely fitting its training labels. Whether this signal reflects observable cues retained in the representation or a convention estimate formed by the LLM is not determined by the probe, and App. K states this limit.

The review's summary sentence ("encode both the hint type and the convention but doesn't combine them") refers to answer-position readouts (Sec. 5.6, App. H.5). In Qwen3-8B, linear readouts decode the hint type and the convention stated in NOTES at 0.92 and 0.98, and the action they jointly imply at 0.698 balanced accuracy. The revision reports this as a difference in linear decodability at the answer position, consistent with the behavioral gap between told and told + verb.

These changes are in the Abstract, Sec. 1 (definition of knowing), Sec. 2 (prior probing studies), Sec. 5.2 (last sentence), Sec. 5.6 and App. H.5 (readout interpretation), App. K, and the new App. L.2.

W2. "The action translation conditions give the model the answer. In the told + verb case, the external code simply writes the correct slot and action into NOTES so the model is following instructions rather than partner adaptation. Same with loop + verb where the convention to decision step is moved outside the model."

The action-translation conditions are diagnostic references for the rule-statement conditions, not a form of partner adaptation. They replace the general rule statement in NOTES with the slot and action that the supplied convention implies for the current hint, which is the difference the contrast is designed to test. On 480 recorded Qwen3-8B states (App. C.3), stating the true convention as a rule raises target doing by +0.312*, and translating it into the current action raises it by a further +0.131*. For intent, the corresponding changes are +0.021 and +0.317*. The contrast thus shows how the receiver's decisions change when the same convention information is given as a general rule or as its application to the current hint.

loop + verb plays the same role for the probe-predicted convention. Over 240 Qwen3-8B games, it raises intent doing over loop by +0.149* [+0.123, +0.174] (Table 2).

The contrast also holds under free generation. We re-queried the 480 recorded states (submitted prompt, 60 boards, 120 per axis and correct answer) with free generation (new App. C.3, Table 8, with intervals). The rule adds +0.267* to target doing and +0.040* to intent doing, and the action then adds +0.163* and +0.279*. The contrast therefore holds under both single-token answers and free generation.

These changes are in Sec. 5.3 (two sentences), the new App. C.3 (Table 8), Sec. 6, and App. K.

Comments on the paper

1 comment

We thank the three reviewers for careful, constructive reviews. The revised PDF marks each change by the review it answers, with text color for tXrA (red), LnLx (blue), or both (purple) and a green highlight for SNXk, and the main text stays within 10 pages.

Notation used in all responses. Doing is the fraction of receiving decisions that agree with the sender's convention. Knowing is the accuracy of a probe that predicts the sender's convention. Both have chance 0.500, and higher is better. Score is the game score (0–15, higher is better). An asterisk marks a paired difference whose board-level 95% bootstrap interval excludes zero, and brackets give these intervals (20,000 resamples).

Main findings after the revision

  1. In all eight LLMs, the sender's convention is linearly decodable from the receiver's hidden states, more accurately for intent than for target and least accurately for target in LLM games (Table 1, App. J).
  2. In all eight LLMs, giving the current action raises intent doing more than giving the true convention as a rule (App. F, point values). For Qwen3-8B the rule does raise target doing, and the contrast holds under free generation (Table 2, App. C.3).
  3. Activation transfers from donor runs given the correct action or a restated hint improve intent doing on both action classes, while the tested fixed transformations do not (Sec. 5.6). A LoRA adapter can learn the application step (Sec. 5.7).

Shared concerns and what we did

ConcernRaised byWhat we did
Decodability alone does not establish usable representationstXrA, LnLxDefined knowing as recoverability by an external linear probe (limit in App. K); added capacity controls (App. L.2)
No comparison with convention inference from the observable historyLnLxAdded observation-feature and bag-of-words baselines (App. L.3)
told + verb writes the answertXrA, LnLxStated as a diagnostic condition; re-queried 480 recorded states under free generation
No infer-then-act or partner-reasoning baselineLnLxRan an Infer-then-Act baseline without ground truth and a decomposed hypothesis-elimination pipeline (App. L)
Missing asterisk definition and incomplete uncertainty reportingSNXkRestored the asterisk definition; added board-level intervals and error bars
Target may simply be easierSNXkBoth axes are binary with chance 0.500. Under the default protocol, the true rule changes Qwen3-8B's target decisions much more than its intent decisions, with convention inference removed
Limited scope and insufficient discussion of limitationsSNXk, LnLxLinked the limitations from the conclusion; extended them

Where the changes are (colored in the PDF)

LocationChange
Abstract, Sec. 1Knowing defined operationally as linear decodability
Sec. 2Sobotka et al. added; description of prior probing studies made precise
Sec. 5.1Asterisk definition restored (board-level)
Sec. 5.2What decodability measures, stated explicitly
Sec. 5.3Chance level of each axis, response-interface result
Sec. 5.4Asterisks on cross-play differences; interval pointer
Sec. 5.6, App. H.5Readout interpretation clarified; readout settings, cross-validation unit, and both action labels stated
Sec. 5.7, App. IAdapter training data stated in full; paired score difference and intervals
Sec. 6Pointer to the limitations
Tables 1–4, Figs. 4–5Chance level and direction in captions, error bars in Fig. 4, asterisks in Tables 3–4, interval pointers; five Table 4 entries recomputed
App. C.3 (new)Response-interface check on 480 recorded states
App. FSource of the Gemma-3-12B row and of the Fig. 5 self-play cells
App. J (new)Bootstrap intervals for Tables 1–4, Figs. 4–5, and the LoRA results
App. KLimitations extended
App. L (new)Capacity controls, observation baselines, Infer-then-Act, and the pipeline, with the revised prompt

Prompt used by the new analyses. The response-interface check (App. C.3) uses the submitted prompt. The other new analyses use a revised prompt built after submission (same game, conventions, and boards) and are reported with it in App. L.

Scope. Knowing is recoverability by an external linear probe. Behavioral contrasts measure how supplied conventions change decisions, and activation transfers measure how donor activations change them. told + verb and loop + verb are diagnostic conditions. The results are established in a controlled two-player environment whose scripted senders make every decision checkable against a known convention.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.