- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies an interesting question in on-policy context distillation: whether simple, reusable instructions can be more effective teacher privileges than instance-specific gold answers. Using autoformalization as a testbed, the authors show that targeted formatting instructions often preserve in-domain performance while substantially outperforming gold privileges under cross-dataset transfer.
Critical strengths
-
The paper challenges the assumption that more privileged information necessarily leads to better distillation. The distinction between instance-specific supervision and transferable behavioral guidance is interesting and potentially useful beyond autoformalization.
-
The experiments cover multiple datasets and two model families. The error decomposition helps connect the observed improvements to concrete failure modes rather than treating instruction design as a black box.
Critical weaknesses
-
The strongest OOD results rely on "matched" instructions. It is unclear how these instructions would be selected without knowing the target domain or evaluating multiple candidates. This matters for whether the reported gains reflect genuine transfer or favorable instruction selection.
-
Table 8 suggests that much of the advantage comes from avoiding the degradation caused by gold privileges, rather than consistently improving over the no-privilege baseline. These are different conclusions, and the paper should distinguish them more clearly.
-
The weaker results on ProntoQA suggest that formatting instructions mainly help when syntactic errors dominate. The paper would benefit from a clearer characterization of when instruction privileges are preferable to gold, rather than presenting them as broadly superior.
-
Solver verdict accuracy does not necessarily guarantee faithful FOL translation. Moreover, experiments use only one training seed, making it difficult to assess the stability of the reported improvements.
Questions for authors
-
How are matched instructions selected in a genuinely unseen target domain? Can the authors demonstrate a target-blind selection strategy?
-
Would combining gold privileges with targeted instructions outperform either alone? This could help determine whether the problem is the gold information itself or how it influences the teacher.
-
Can the authors evaluate the semantic fidelity of generated FOL independently of solver verdict accuracy?