- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
I believe this paper should be desk-rejected for the reasons I laid out in my official comment and documented in my AI documentation note. But, my understanding of the paper is that the authors have identified a gap in long-context reasoning evaluation. LLMs can retrieve correct evidence, but may apply it to incorrect cases or document versions (here they are dealing specifically with tasks having multiple documents or "scenarios"). They introduce a benchmark SCB and metric FBNB on which they evaluate 10 LLMs, finding varying performance. I also have opinionated critiques of the writing style, structure and prioritization of dense paragraphs over providing intuition or insightful results.
Critical strengths
- Authors point out a genuine gap in long-context reasoning evaluations.
Critical weaknesses
- Violation of the ICLR style guidelines, as documented in my Official Comment: https://openreview.net/forum?id=jwevBT2MGb¬eId=WCNzl2kSsx
- The authors prioritize providing long, dense paragraphs over giving readers intuition or elevating experiments that would clearly illustrate novel findings rather than redundant, muddled figures like Figure 1:
- I'd suggest that the authors provide more figures in their paper, for results but also the example from the second paragraph of their introduction could have been made into a motivational figure to provide the reader with better intuition for the gap the authors are pointing out.
- Table 1 comes across as messy and an "info-dump" rather than a justifiable simplification of the related work on similar benchmarks and how the authors' proposed benchmark is novel, which is what such a table should be aiming to be.
- Examples would have been nice to include in section 3.2
- Writing clarity is not up to ICLR standards:
- A high number of "clause :/; clause" sentences which I associate with LLM-generations. I know the authors disclose the use of AI writing assistants, but I'd encourage them to go through the paper and simplify things and make sure they expressing their message in their own words as much as possible.
- Re: Section 2. I'm not sure why this isn't just titled "Related Work" in keeping with standard practices
Frankly, I stopped reading after completing section 3. I did the style guide violation investigation after I read section 2. Then, I skimmed the results and later sections, but, due to what I believe to be a desk-reject-worthy violation and the clear use (in my opinion) of minimally edited LLM-generated prose, I did not provide my full effort to critique the paper starting from Section 4. If the paper had just one of: a violation or LLM-generated writing, I would have felt obliged to provide the authors with a more meaningful review to aid them in resubmitting. As is, I frankly felt it was not worth my time.
Questions for authors
Why did you edit the PDF after LaTeX compilation?