acceptodds
Under review as a conference paper at ICLR 2027

CPD: Contrastive Perspective Decoding for First-Order False-Belief Answer Selection in Language Models

Abstract

First-order false-belief tasks test whether a model answers from a character's information rather than from the current world state. We find that a fixed Yes/No protocol can hide this distinction: for the same false-belief situation, complementary questions about original and current locations are often answered inconsistently, with joint accuracy of only 0.01–0.11. We introduce Contrastive Perspective Decoding (CPD), a decoding-time intervention that uses two attention views of the same frozen language model. It blocks the unwitnessed state change in one view, compares candidate scores with the complete-context view, and amplifies candidates that gain support when that change is removed. When the event span and the character's witnessing status are supplied, CPD raises joint accuracy to 0.24–0.52 and macro-F1 on the primary task to 0.77–0.84, from baselines of 0.01–0.11 and 0.30–0.34, respectively. On two transfer benchmarks, CPD improves over masking alone by 0.21–0.33 macro-F1 in all six model–dataset comparisons. A frozen schema-aware controller raises accuracy on shuffled location choices from 0.56–0.65 to 0.74–0.75. However, directly reading its output reaches 0.76–0.78, and schema-aware contrast is not significantly better than schema-aware masking. CPD therefore isolates a useful response-selection intervention under correct perspective information, while reliable event and visibility inference remains the main deployment challenge.

Primary area:
interpretability and explainable AI
Keywords:
large language models, theory of mind, false belief, contrastive decoding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5umCOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that three open 7–8B models answer complementary Yes/No questions about the same false belief inconsistently (both correct on only 1–11% of stories), and proposes CPD, which contrasts a model’s answer scores with and without attention to the sentence describing an event the character didn’t see, reaching 0.81–0.95 macro-F1 on three location datasets when the relevant sentence and the character’s access are supplied. The authors themselves show that with that supplied information a trivial rule is perfect and a story-free classifier beats CPD on action prediction, so the work is best read as a controlled analysis of where belief-relevant signal lives in the model rather than as a usable decoding method.

Critical strengths
  • The proposed approach provides a clean intervention-based result that belief-relevant signal is recoverable from a span-level attention contrast.

  • The paper deliberately includes baselines that beat its own method and says so clearly.

Critical weaknesses
  • CPD needs to be handed the hardest part of the problem: what the character missed. The paper does not test what happens when that information is guessed by a model instead, and the random-mask check suggests it would break. The title of the paper suggests a decoding method, but in essence it is more of a study that is challenging to implement as a decoding method.

  • Hiding the sentence via attention masking isn’t shown to beat just deleting it, making the approach similar to existing work. The real gain seems to come from the contrast term with one tuned setting (γ = 8), and it’s unclear whether that setting holds up on other data or model sizes.

Questions for authors
  • How does CPD behave when the span and access flag are inferred rather than supplied?

  • What exactly does the attention mask buy over simply deleting the sentence?

1 comment
AuthorsOpenReview ↗

Response to Reviewer SumC: what the rule does when the description is inferred

Dear Reviewer SumC,

The revision reports the inference path you asked for, and it supports the framing test you raised. With the span and access read from text, the rule reaches 0.910 at 3B (IoU 1.00 against the construction mask) and gives 0.820 at 3B and 0.885 at 7B on OpenToM; with a 3B model predicting the span itself, it reaches 0.695 against 0.465 for the plain prompt. The cost of inference is therefore measured, and the rule still works.

Q1. Behaviour when span and access are inferred. Settings: 200 items per benchmark, balanced on witness status; BigToM 195 dev / 205 test; 3B and 7B bf16; paired McNemar; two passes; scripts released.

Inferred input (M4, 3B)SourceTest accuracy95% CI
oraclesupplied0.910[0.862, 0.942]
text ruleIoU 1.000.910[0.862, 0.942]
text rule, 80% recalldegraded0.810[0.750, 0.858]
model-predicted3B self-report0.695[0.628, 0.755]
noneplain0.465[0.397, 0.534]

The text rule keeps 86.4% of the gain at 80% recall, and model predictions keep 23 points over the plain prompt; on OpenToM the same rule agrees with the dataset field on 96.22% of 2,384 items. The revision reports this beside the oracle upper bound rather than presenting a decoder with an untested input.

Q2. What the mask buys over deleting the sentence.

BigToM (7B)MaskDeletionFiller
plain decoder0.32680.53660.6000
contrast decoder0.80490.83900.8293

Deletion and masking are statistically indistinguishable under the contrast decoder (McNemar p=0.25; filler p=0.44), and deletion is better under the plain decoder, so the mask buys no accuracy; what the two-pass contrast buys is a decision rule without a tuned threshold. The paper now claims the rule and its transfer, not a special mask.

Q3. Across data and model sizes. The rule also holds at 1B (0.620 to 0.835 on M3) and at 7B (explicit absence 0.620 to 0.835 at 1B, 0.555 to 0.850 at 3B, 0.615 to 0.940 at 7B; implicit absence 0.465 to 0.910 and 0.330 to 0.910), a second seed reproduces the implicit cell (0.515 to 0.895), and gamma is monotone or flat and saturates at 16-64 across nine checkpoints, so no tuned value remains.

We thank the reviewer again; if these analyses address the concerns, we would be grateful if the score could be reconsidered.

Why this addresses the review. The paper no longer rests on a supplied description: it reports the inference path, the price of inference, and the comparison against deletion that the review asked for. The mask is presented as a way to specify the witness restriction, the contrast rule as a threshold-free decision rule, and the transfer to two public benchmarks at two model scales as the evidence. We would be glad to discuss any remaining point, and we thank the reviewer again for pressing on exactly the questions that improved the paper.

Reviewer_KnYQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how language models answer false belief questions, where a character still believes an object is in its old location because they missed it being moved. The authors first show that models give inconsistent answers to two complementary yes or no questions about the same story, depending on how each question is worded. To address this, they propose Contrastive Perspective Decoding (CPD), which scores each answer with and without attention to the unwitnessed event and favors the answer that gains support when that event is hidden. This event and the character’s access to it are supplied in advance, so the method does not infer what the character knows. Across qwen, llama, and mistral on openToM, bigToM, and eexploreToM, CPD outperforms masking alone and an equivalent plain text prompt. On SimpleToM action choices, a simple classifier that never reads the stories does better, which leads the authors to frame their results as conditional answer selection rather than genuine belief reasoning.

Critical strengths

The audit of the action choice benchmark is one strength. The authors show that a classifier which never reads the stories, and only looks at how the answer options are writte often outperforms their own method. This reveals that the benchmark can be solved without any belief reasoning at all. This is consequential because it exposes a weakness in an existing evaluation that other researchers may be relying on, and the authors deserve credit for reporting a result that works against their own method. However, this finding is presented as a side analysis rather than a central contribution, so it strengthens the paper’s value without changing the overall recommendation.

Critical weaknesses

The main evaluation does not test reasoning. The method is given the information about the relevant event & whether the character saw it. In the binary location setting this information already determines the correct answer and as a result, a simple rule using the same inputs answers every question correctly and outperforms the method. Therfore the strong scores across all three location datasets show only how well the method uses information it was handed. This is consequential because the paper is motivated by the problem of inferring what a character knows, and the experimental design removes exactly that step. Addressing this would require new experiments in which the model must infer the character’s access on its own, which is more than a revision can reasonably cover.

The action choice experiment is the paper’s main attempt to show that the method works beyond the binary setting but endsv up not showing generalization. A classifier that doesn't read the stories achieves a higher accuracy than the method. This means the benchmark can be solved from answer wording alone, so it provides no evidence that the method improves belief reasoning. This is consequential because without this experiment, the paper has no setting in which the method is shown to be useful, since the binary results are already explained by the supplied inputs. Fixing this would require a new benchmark that resists such shortcuts, which again goes beyond what a revision could address.

Questions for authors
  1. In what realistic setting would a user know the relevant event and the character’s access status, but still need CPD rather than the simple rule that already answers every question correctly? A convincing use case would directly change how I view the method’s value.
  2. True belief items receive an empty mask, so the true belief constraint used to select γ appears unable to affect the choice. Could you confirm this, and explain why the selected value sits at the edge of the grid when accuracy keeps rising at larger values? If the best decoder simply picks whichever answer the mask pushes upward, the contrast rule contributes less than the paper presents.
  3. How were the random masks matched to the event sentence (by length, position, or sentence boundaries), and were they combined with the same contrast rule as CPD? Were the baselines without supplied information evaluated through free generation, while CPD usedirect answer scoring?
1 comment
AuthorsOpenReview ↗

Response to Reviewer KnYQ: the setting where the rule matters, and no tuning left

Dear Reviewer KnYQ,

The revision answers your concerns with a setting in which the answer is not supplied and the rule has to do the work. On OpenToM and ToMi the plain prompt scores 0.500 and 0.310; the two-pass rule scores 0.820 and 0.690, and every random-mask control is far behind. The gain appears only on the items whose answer depends on the witness set (non-witnessed 0.000 to 0.640 at 3B and 0.160 to 0.960 at 7B), while the witnessed items are untouched.

Q1. A realistic setting. Settings: 200 items per benchmark, balanced on witness status; BigToM 195 dev / 205 test; 3B and 7B bf16; paired McNemar; two passes; scripts released.

The setting is a described scene in which the witness set is a property of the story: a reader (or an assistant) must answer from what a character knows. A 3B model can produce that mask itself (precision 0.319, recall 0.818) and still moves the plain prompt from 0.465 to 0.695, so the setting exists and its cost is measured. On BigToM there is no such setting - the flag is the label - and the revision no longer claims one.

Q2. True-belief items and the grid edge. You were right on the fact: 388 of 400 true-belief items have identical branches, so the mask cannot affect them; a fallback is declared. On the grid, accuracy rises monotonically to the limit, so gamma=8 was where the grid stopped; the revision removes the tuned gamma and uses the margin decoder.

OpenToM (n=200)PlainOurs
witnessed half, 3B1.0001.000
non-witnessed half, 3B0.0000.640
witnessed half, 7B0.9000.900
non-witnessed half, 7B0.1600.960
overall, 3B / 7B0.500 / 0.5300.820 / 0.930

Q3. Random masks and scoring parity.

ControlSettingValue
random maskwhole sentences, position-matchedsame contrast
free generationsame items0.1220
direct scoringsame items0.1268

Random masks are whole-sentence, position-matched and enter the same contrast rule and the same direct scoring, so the comparison isolates which sentences are hidden; the free-generation control is separate and no main-table arm uses generation. The action-choice audit remains a contribution, and the revision frames the result as conditional answer selection rather than genuine belief reasoning.

We thank the reviewer again; if the new setting and the tuning-free decoder address the concerns, we would be grateful if the score could be reconsidered.

Why this addresses the review. The concerning part of the original paper was that supplied inputs made the answer deterministic; the revision removes that setting, adds two benchmarks where the rule must do the work, and reports the price of inferring the inputs. The action-choice audit is kept as a contribution, the tuning is removed entirely, and the shortcut baselines are reported next to the method. We hope this makes the paper's claim both defensible and useful, and we would be glad to run any additional control the reviewer suggests.

Reviewer_8b4XOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies false-belief answer selection in three 7–8B instruction-tuned models (Qwen2.5-7B, Llama-3-8B, Mistral-7B). It first introduces a complementary-question diagnostic, where the model is asked whether, according to the character, the object is at its original location and whether it is at its current location. Base models answer both correctly on only 1–11% of 174 OpenToM stories. To resolve this belief problem, the authors proposes Contrastive Perspective Decoding (CPD), which scores each candidate twice with the same frozen model, once with full attention and once with attention blocked to a supplied “inaccessible” sentence, and selects argmax[L_masked + γ(L_masked − L_full)].

Critical strengths

The complementary-question diagnostic is a simple, useful probe. Requiring opposite answers to two questions about the same belief exposes answer-polarity and formulation biases that single-question accuracy hides.

The experimental hygiene is careful in several respects: matched random-span masks, a disjoint development split for selecting γ, exact McNemar tests with Holm correction, paired bootstrap intervals, and item-level verification of the empty-mask identity (1,944 true-belief items).

The SimpleToM option-only audit is a valuable negative finding for the community. It shows that a paired action-choice metric can be solved from option wording plus access status alone.

Critical weaknesses

CPD requires the relevant event sentence and the character’s access status to be supplied, and all results use these from dataset metadata. In realistic settings, identifying this information is the core of the perspective-taking problem, and CPD does not address it. No experiment uses predicted or noisy inputs. Moreover, when these inputs are available, the authors’ own deterministic rule scores 1.000 on every binary location condition and outperforms CPD, leaving the method without a clear use case.

Figure 3 shows that base models favor a fixed answer across both complementary questions (“No” for Qwen and Llama, “Yes” for Mistral). Several Qwen conditions in Table 2 score exactly 0.296 macro-F1, the value obtained by always answering “No,” and Mask only reaches just 0.034 FB accuracy for Qwen despite hiding the move. With γ = 8, CPD’s decision is dominated by the change in margin between passes, which naturally cancels a constant polarity bias. Without a calibrated baseline using the same information, the gains over Mask only cannot be attributed to more than bias correction.

Questions for authors
  • How does CPD compare with Mask only plus a bias offset or threshold tuned on the same development split, or with contextual calibration of the Yes/No prior?
  • How does CPD perform when the event span and access status are predicted (by the model itself or an external extractor) rather than supplied?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 8b4X: a two-pass rule that transfers to two public benchmarks

Dear Reviewer 8b4X,

The revision reports a simple result: answering a belief question by the margin between scoring with attention restricted to what the queried character witnessed and scoring with the full context beats the plain prompt and a sentence-matched random mask on two public benchmarks at two scales (OpenToM 0.500 to 0.820 at 3B and 0.530 to 0.930 at 7B; ToMi 0.310 to 0.690 and 0.520 to 0.700). The rule needs no prompt, no tuning and no supplied label, and its mask can be read from the story. We also ran the two tests you asked for.

Q1. Calibration against mask-only. Settings: 200 items per benchmark, balanced on witness status; BigToM 195 dev / 205 test; 3B and 7B bf16; paired McNemar; two passes; scripts released.

Baseline (BigToM)Test accuracy95% CI
plain0.1268[0.088, 0.179]
mask-only0.3268[0.266, 0.394]
mask-only + dev bias0.4293[0.363, 0.498]
dev-tuned threshold0.9756[0.944, 0.990]
margin decoder0.6390[0.571, 0.702]

On BigToM a development-tuned threshold reaches 0.9756, so the benchmark's false-belief slice is a prior effect and its gains cannot be attributed to more than bias correction; that is why the revision moves the evaluation to the two benchmarks above, whose labels are balanced by construction so a prior cannot win them. This is a cleaner test of the method, not a weaker one.

Q2. Predicted span and access. Settings: 200 items per benchmark, balanced on witness status; BigToM 195 dev / 205 test; 3B and 7B bf16; paired McNemar; two passes; scripts released.

Setting (M4, 3B, n=200)Accuracy95% CI
plain0.465[0.397, 0.534]
model-predicted mask0.695[0.628, 0.755]
oracle rule, 80% recall0.810[0.750, 0.858]
oracle mask0.910[0.862, 0.942]

With a 3B model producing the mask itself, accuracy is 0.695 - 0.230 above the plain prompt - and the 0.215 gap to the oracle is the measured cost of inference. A degraded rule keeps 86.4% of the gain. The method therefore works with inferred inputs, and the revision states the price openly.

Q3. Where the method is tested.

Benchmark (n=200)PlainOursRandom mask
OpenToM 3B0.5000.8200.695
OpenToM 7B0.5300.9300.680
ToMi 3B0.3100.6900.240
ToMi 7B0.5200.7000.340

Every cell beats the plain prompt and the random-mask control; the gain is concentrated on the non-witnessed items while the witnessed items are untouched, which is the behaviour a perspective-access rule should have.

Q4. The gamma you flagged. Accuracy is monotone or flat in gamma and saturates at 16-64 across nine checkpoints, so the tuned value gives way to the margin decoder: the method now has no hyperparameter at all, which is strictly better than the tuned version.

Q5. Changes. The decoder, gamma grid and controller are withdrawn; the two benchmarks, the calibration test and the predicted-input test are added as Table 3 and the appendix. We thank the reviewer again, and if these results address the concerns we would be grateful if the score could be reconsidered.

Why the revision is stronger. The paper now has four things it did not have: a rule with no tuned hyperparameter, two public benchmarks with balanced labels where a supplied rule does not exist, a measured cost for inferred inputs, and controls (complement masks, dose curve, random-mask matching) that isolate what the rule uses. On every cell the rule beats the plain prompt and the random-mask control, and the gain sits only on the items whose answer depends on the witness set. We think this is a narrower claim than the original, and a much better tested one.

Thank you for the detailed response and for running both requested experiments. I appreciate the predicted-mask result and the transparent reporting of the BigToM calibration outcome. Unfortunately, I believe that this paper has changed during this rebuttal a lot and should be improved further. Thus, my main concerns remain, and I am keeping my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.