- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors explore novel halting methods for recursive reasoners. While prior work either uses a fixed number of iterations or halting based on ACT, the authors propose margin-based halting. Additionally, they propose a margin-based method of returning predictions if halting did not occur.
Critical strengths
The paper provides substantial motivational theory for their methods.
Critical weaknesses
The paper itself seems sound and provides efficiency gains but 1) the results are unconvincing, with a simple decode lock performing on-par with the proposed margin-based methods and 2) the majority of the paper is spent on motivational theory rather than experiments in actually challenging domains.
First, the paper makes several assumptions about recursive reasoners, e.g., contractive trajectories and separation assumptions. While prior theory works on similar assumptions like approaching a fixed-point, the paper here shows few convincing empirical results. Would the authors be able to provide empirical results for the separation assumption? Could we quantify the contractiveness, e.g., with an empirical contraction factor?
Additionally, I would like to see more experiments. Currently, the results might be unconvincing since the baselines in both domains are already near the results from the oracle. Seeing noticable improvements in a task with more headroom would strengthen the paper a lot. Currently, my read of the paper would suggest that the much more decode lock metric works similarly well than the proposed methods.
Questions for authors
Can we provide evidence for the assumptions? Are there results for domains where margin-based halting provides substantial improvements over decode-lock?