- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper studies a policy that must infer a changed legality rule from accept/refuse feedback alone. It splits a deployable policy into propose, infer, and decide stages, varies one stage at a time, and introduces "planting" a chosen rule among the hypotheses as a diagnostic. It concludes that proposal, not probe planning, is the bottleneck. A benchmark of 38 rule shifts derived from three codebases is released.
Critical strengths
- The question is focused, and the stage-wise intervention design and planting diagnostic are clean ideas.
- Success criteria are fixed in advance, and negative results are reported honestly.
Critical weaknesses
- The main conclusion follows from the design. The policy draws 24 random hypotheses, and in five of six shapes none of them can express the deployed rule. If the correct rule is not in the hypothesis set, no inference or planning stage can recover it, and planting it must help. Locating the bottleneck at proposal is therefore a consequence of the setup, not an empirical finding.
- The planning result is confounded. Planning is compared on a hypothesis set that does not contain the deployed rule, so little gain should be expected. In addition, with H=40 attempts over 36 actions, most actions are tried within an episode anyway, which leaves little room for query selection to matter. The claim that planning adds nothing is not tested under conditions where it could.
- The proposer appears under-provisioned. The only change to proposal that helps is drawing more hypotheses, and the gain shows no sign of saturating. This points to too few hypotheses in this particular policy rather than a general finding that proposal is the bottleneck.
Questions for authors
See Weaksnss.