- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper examines Transformers for Nested Logical Filtering. Here, boolean queries are evaluated across table rows. The authors compare CoT with Program of Thought, showing that direct textual outputs become susceptible to token errors as answer length grows. They evaluate on WTQ and DataBench.
Critical strengths
- Authors give a formal characterization of there approach and the limitations of short CoT LLMs.
- There is a good discussion on the distinction on correctness of an answer vs the ability to reliably generate it.
- Experiments are well thought out and directly support the claims across synthetic and public benchmarks.
Critical weaknesses
- Limited novelty beyond theoretical contribution. NC1 result builds on the problem formulation, and the noise-robustness separation reflects the inference algorithm. The paper needs to be more clear which results are new vs established.
- Assumptions limit application: They favor PoT but are not directly characterizing LLM generation errors.
- Experiment comparison: PoT gets 8 rounds, CoT does not have an equivalent correction mechanism. Experiments do not cleanly separate the prosed explanation.
- Limited size: only 20 synthetic samples, 40 per group on the public. Please note which deep seek checkpoint is used for the backbone.
Questions for authors
- Can the authors highlight which results are new beyond existing results?
- Fairness: can the authors provide an apple to apple comparison between PoT and CoT?
- Reproducibility: Specify model checkpoint and which samples used from public datasets.