- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The author claims that current large language models perform poorly at inductive reasoning. The author proposes IMPA, an RL-based mapping framework that incorporates CoT, cases, patterns, and rationality. The author also proposes RLP for paradigm generation. The author conducts experiments and claims that their paradigm improves the quality of the generated results. The paradigm first extracts cases and patterns from the initial CoT to form a graph. Then, a verification module provides a reward based on soft consistency.
Critical strengths
- The reward module can be self-verified without relying on an external reward model.
Critical weaknesses
- I think the author, in general, should be more careful about writing. e. g., in the first paragraph of the introduction, the author makes some very strong claims about the limitations of LLMs (
tend to be suited for instance-level answers rather than autonomously inducting abstract, reusable, and general patterns) or the underlying mechanism of LLMs (treat CoT as the overall trajectory of policy optimization, without explicit knowledge abstraction). The authors should either include at least some toy experiments to support the claim or soften the tone. - The author chooses BigBenchHard and other general reasoning benchmarks. Also, in the related work, the author seems to care only about general knowledge-based reasoning rather than inductive reasoning. The author should consider either properly scoping the paper as inductive reasoning (consider changing the benchmark to inductive reasoning-specific ones such as InductiveBench[1]) or broadening the scope of the paper to be a general knowledge-reasoning paper.
[1] InductionBench: LLMs Fail in the Simplest Complexity Class
Questions for authors
How is the IAMP framework working at inference time? Does it still have the outer-loop section, and is there any difference between the training-time behavior?