- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper uses offline lookahead on full training data to teach a network to instantly pick the best affordable feature groups at test time. And it cuts inference down to sub-milliseconds, fixes tied-choice ordering bias, and outperforms top baselines on 5 out of 6 real-world benchmarks.
Critical strengths
The paper's experiments are clean and honest. There is no data leakage, the stats are done properly, and the authors also explain why their 2-step planning didn't help on real datasets instead of trying to hide it.
Critical weaknesses
My biggest concern is that the whole story about multi-step offline planning doesn't hold up on real data. Across all six real datasets and every single one of the 30 splits, validation always picks the 1-step model and completely avoids H2. Even worse, H2 is worse than H1 on several tasks. This means the real-world performance has nothing to do with the multi-step planning the authors are selling.
Questions for authors
Looking at Table 13, across all 30 validation splits on the six real benchmarks, the validation set never once selects H2, and H2 is often substantially worse than H1. Can the authors provide any empirical evidence or intuition showing that two-step planning actually helps on real-world tabular data?