- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper sorts in-context tasks by the source their output depends on (surface identity, pretrained semantics, or a computed intermediate) and probes each with interventions on Qwen3-8B, including a correspondence dose in low-resource MT, byte corruption of the source, and held-out-digit and intermediate-rescue conditions on two synthetic tasks, plus a five-family check. Code is described as a supplementary archive; no repository is linked.
Critical strengths
- Some confound controls are well chosen. Tokenization-preserving word substitution sends semantic accuracy to roughly chance (0.18–0.28) in all five families (Table 8), ruling out tokenization as the cause of the corruption collapse.
- The held-out-digit control (Table 4) exposes a support-overlap artifact, and the paper retracts its own feature-identification reading accordingly.
Critical weaknesses
-
Each finding restates something already known. That correct pairings help translation is expected. An unreadable source breaking semantic mappings but not arbitrary symbol bijections mirrors the task-recognition/task-learning split of Pan et al. (2023), and an 8B model answering with thinking disabled failing a multi-step composition that a supplied intermediate fixes matches Dziri et al. (2023) and the scratchpad gains of Nye et al. (2021). The framework adds no predictive content, since Section 3.1 concedes that is read off the same interventions it explains, and Appendix E finds a held-out signal only for the surface route ().
-
The computed-intermediate claim outlives the paper's own controls. With the query digit held out, numeric joint, explicit-rule, and direct accuracy fall to 0.025/0.205/0.365, yet the five-family rescue (Appendix D) is still measured against the overlapped joint baseline. Appendix K blames support overlap, but (0.525) gives no gain over (0.58) though more demonstrations only raise overlap. The relational joint score moves from 0.48 to 0.58 with decode budget and is omitted cross-family for truncation, while enabling thinking lifts numeric joint accuracy from 0.58 to 0.78 (Appendix B). What remains looks like a limit of single-pass decoding, not of availability during adaptation.
Questions for authors
- At , what is joint accuracy on episodes where the query digit recurs? With 80% recurrence and 0.025 otherwise, the reported 0.58 implies about 0.72, which reuse alone doesn't explain.
- Does the relational direct probe stay near chance with thinking enabled?
- What are the absolute byte-corrupted chrF++ scores behind Table 3? A floor would make the collapse uninformative.