- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The study Supervision Before Structure demonstrates that explicit causal architecture in world models is redundant for determining question answerability under full supervision, but provides an effective inductive bias that improves label efficiency when answerability labels are scarce.
Critical strengths
This paper raises a seemingly interesting question.
Critical weaknesses
- This paper is very hard to read as there seems to be no editing at all. It thus lacks the most basic presentation requirements for scientific results and discovery. For example, lines 216- 258 lack clarity, as the entire paragraph is presented as one block. Moreover, the experimental setup and tables are impossible to understand.
- The primary limitation is that the claimed label-efficiency benefit of structured world models is demonstrated under an artificial supervision regime in which identifiability labels are selectively withheld while closely related counterfactual and evidence-acquisition targets remain fully available. When these supervision sources are jointly reduced, the reported advantage largely disappears. Furthermore, the fact that removing or randomizing mechanism supervision does not degrade performance raises questions about whether the improvement reflects meaningful causal structure rather than a generic architectural inductive bias. Overall I think these findings substantially weaken the paper's central claim and its practical significance.
Questions for authors
I encourage the authors to carefully revise their work before considering resubmission.
Thank You reviewer
"When these supervision sources are jointly reduced, the advantage largely disappears." We agree, and the new experiments make the point stronger than the submitted table did. With the evidence-gain target confined to the labelled queries, the structured model has no forty-label lead on five of eight environments; with all three targets confined, the structured model is behind the control on five of eight (Table 4). You wrote that these findings weaken the paper's central claim. The central claim of the revised paper is those findings: the margin is not label efficiency, it requires an auxiliary target that contains the label, and it owes nothing to causal semantics (Section 4).
"Removing or randomising mechanism supervision does not degrade performance." Agreed, and the revision says it in those terms: what survives when both auxiliary targets are fully available is specific to the full structured model, with or without mechanism labels, so it owes nothing to causal semantics. A generic K-expert head given to the control does not reproduce it, and we have not isolated the ingredient further (Section 4, "What remains"; Appendix D).
Presentation. The paper is rewritten. Old lines 216–258 are now four short paragraphs with one claim each, and the numbers they carried are in Table 4 and Appendix Table 6. Every table caption now defines its columns, and the two models are drawn side by side in Figure 4.
Does the restructured Section 4 address your concern about the central claim?