- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes a new prompting based method called ReLTA, a framework that improves LLM reasoning by improving few-shot Chain-of-Thought (CoT) examples. So the method first identifies some difficult questions using a combination of prediction error and entropy, the authors call it the difficulty score. The method then uses a larger model to generate a reasoning trace for per selected questions. At inference time, cosine similarity is used to reterive relevant examples using a reasoning-aware retriever and are then provided to the smaller model as few-shot examples. The paper shows their method across eight reasoning benchmarks and report improvements over existing prompting baselines.
Critical strengths
I think the paper is generally well written and easy to follow.
Critical weaknesses
I find several issues with the motivation of the paper.
- The main baseline of ReLTA is active prompting, that came out in 2023/24, therefore i think ReLTA is significantly late and is not a method that I believe relevant for current age of language models.
- On top of that the main contribution of the paper is very small, like it is mostly confined to introducing the difficulty score which is a weighted sum of already used Entropy and Error rate (the main addition of the paper). I overall think that this is very minimal contribution.
- On top of that similar to active prompting, a larger model is required for generating CoTs. Having a method that would have not required this larger model would have improved the contribution of the paper.
- The improvements over a simple baseline are also very minimal like 2%.
Questions for authors
- Why are all coder models using in the evaluation when the benchmarks are not coding-based.
- What is the reason behind using 0.7 and 0.3 for calculating difficulty score?
Dear Reviewers,
As the discussion period is coming to an end, I kindly encourage you to review and respond to the authors' rebuttal as soon as possible.
Thank you very much for your time and valuable contributions to the review process.
Best regards,
Area Chair