- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces IPBench to test whether language models can judge support for an ideational proposition in text. The task first asks whether a passage actually engages with the proposition, then classifies its stance as ACCEPT / REJECT / NEUTRAL. The benchmark contains 516 examples from 172 passages of classical Chinese thought and three propositions, annotated by two experts. The main finding is over-engagement : models often treat unrelated passages as relevant, while author memory and proposition orthodoxy do not provide a stable explanation. The paper also proposes a proposition space for comparing texts in a shared ideological space, but this idea is still tested on a very small setup (e.g., only three propositions, one corpus, and an almost fixed prompt), so its generality remains unclear.
Critical strengths
I find the research question interesting, especially the separation of engagement and stance and the observation of over-engagement . This is a useful perspective, and the overall idea has potential. I would like to see a broader evaluation and a more carefully polished paper.
Critical weaknesses
- The paper does not even use the full 9-page ICLR limit.
- The dataset is small and narrow, and the selection process is incomplete. IPBench has only 172 passages and three propositions, and the passages were already filtered for possible relevance. The full screening record was not kept, so the engagement / over-engagement results may depend heavily on the current dataset construction. The main experiments also rely on one fixed zero-shot prompt, with no checks across prompts, more propositions, or other corpora.
Questions for authors
- The relation to ACL 2026 Semantic-Aware Logical Reasoning via a Semiotic Framework is not clear enough. Both study abstract propositions and stance relations. What is the key difference between IP/TWPI and that work? RepublicQA could also serve as an external test of whether over-engagement exists beyond IPBench.
- The current evaluation is too narrow. It uses only three propositions, one pre-filtered corpus, and mostly one fixed prompt. Please test more propositions, other corpora, and prompt settings (eg. prompt variation and few-shot) to show that over-engagement is actually robust.
- The annotation and proposition space need stronger evidence. The first expert's label is directly used as gold, even though engagement with abstract propositions can be subjective. Disagreements should be adjudicated or evaluated separately. The proposition space is also mostly conceptual; a real cross-author or cross-period comparison would make this claim much more convincing.
Dear Reviewers, as the rebuttal deadline is approaching, please actively engage in the discussion based on the authors' responses. Thank You!