- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper shows that three open 7–8B models answer complementary Yes/No questions about the same false belief inconsistently (both correct on only 1–11% of stories), and proposes CPD, which contrasts a model’s answer scores with and without attention to the sentence describing an event the character didn’t see, reaching 0.81–0.95 macro-F1 on three location datasets when the relevant sentence and the character’s access are supplied. The authors themselves show that with that supplied information a trivial rule is perfect and a story-free classifier beats CPD on action prediction, so the work is best read as a controlled analysis of where belief-relevant signal lives in the model rather than as a usable decoding method.
Critical strengths
-
The proposed approach provides a clean intervention-based result that belief-relevant signal is recoverable from a span-level attention contrast.
-
The paper deliberately includes baselines that beat its own method and says so clearly.
Critical weaknesses
-
CPD needs to be handed the hardest part of the problem: what the character missed. The paper does not test what happens when that information is guessed by a model instead, and the random-mask check suggests it would break. The title of the paper suggests a decoding method, but in essence it is more of a study that is challenging to implement as a decoding method.
-
Hiding the sentence via attention masking isn’t shown to beat just deleting it, making the approach similar to existing work. The real gain seems to come from the contrast term with one tuned setting (γ = 8), and it’s unclear whether that setting holds up on other data or model sizes.
Questions for authors
-
How does CPD behave when the span and access flag are inferred rather than supplied?
-
What exactly does the attention mask buy over simply deleting the sentence?
Response to Reviewer SumC: what the rule does when the description is inferred
Dear Reviewer SumC,
The revision reports the inference path you asked for, and it supports the framing test you raised. With the span and access read from text, the rule reaches 0.910 at 3B (IoU 1.00 against the construction mask) and gives 0.820 at 3B and 0.885 at 7B on OpenToM; with a 3B model predicting the span itself, it reaches 0.695 against 0.465 for the plain prompt. The cost of inference is therefore measured, and the rule still works.
Q1. Behaviour when span and access are inferred. Settings: 200 items per benchmark, balanced on witness status; BigToM 195 dev / 205 test; 3B and 7B bf16; paired McNemar; two passes; scripts released.
The text rule keeps 86.4% of the gain at 80% recall, and model predictions keep 23 points over the plain prompt; on OpenToM the same rule agrees with the dataset field on 96.22% of 2,384 items. The revision reports this beside the oracle upper bound rather than presenting a decoder with an untested input.
Q2. What the mask buys over deleting the sentence.
Deletion and masking are statistically indistinguishable under the contrast decoder (McNemar p=0.25; filler p=0.44), and deletion is better under the plain decoder, so the mask buys no accuracy; what the two-pass contrast buys is a decision rule without a tuned threshold. The paper now claims the rule and its transfer, not a special mask.
Q3. Across data and model sizes. The rule also holds at 1B (0.620 to 0.835 on M3) and at 7B (explicit absence 0.620 to 0.835 at 1B, 0.555 to 0.850 at 3B, 0.615 to 0.940 at 7B; implicit absence 0.465 to 0.910 and 0.330 to 0.910), a second seed reproduces the implicit cell (0.515 to 0.895), and gamma is monotone or flat and saturates at 16-64 across nine checkpoints, so no tuned value remains.
We thank the reviewer again; if these analyses address the concerns, we would be grateful if the score could be reconsidered.
Why this addresses the review. The paper no longer rests on a supplied description: it reports the inference path, the price of inference, and the comparison against deletion that the review asked for. The mask is presented as a way to specify the witness restriction, the contrast rule as a threshold-free decision rule, and the transfer to two public benchmarks at two model scales as the evidence. We would be glad to discuss any remaining point, and we thank the reviewer again for pressing on exactly the questions that improved the paper.