acceptodds
Under review as a conference paper at ICLR 2027

IPBench: Evaluating Text-Warranted Proposition Inference in Language Models

Abstract

Natural language inference (NLI) judges a hypothesis that is drawn from its premise and is true or false of a situation. We study the judgement of a hypothesis with the two opposite properties: it is exogenous, given before any premise is read, and it is abstract and ideational, a judgement that can be accepted or rejected but not verified. We call such a hypothesis an ideational proposition and the judgement of its relation to a text Text-Warranted Proposition Inference (TWPI): one of four labels, not applicable, accepted, rejected, or engaged without a determinate stance, which separates the prior question of engagement from the subsequent question of stance. To measure it we introduce IPBench: 516 cells formed from 172 passages of classical Chinese thought and 3 propositions, labelled by two experts under released semantic conventions (), of which 174 are engaged, with a protocol that scores engagement and stance separately and measures over-engagement as a quantity of its own. Under one bare prompt frozen before any run, four models reach four-way accuracy of only .70 to .78, and the missing accuracy is over-engagement: every model engages more cells than the gold standard contains, by a factor of 1.03 to 1.33, while accept and reject are rarely confused, and two models route the surplus to the neutral label where two route it to a stance. Two pre-registered controlled experiments find that prior knowledge, whether the model's memory of a passage's author or the orthodoxy of the proposition, has little and no consistent effect on the surplus. Applied over a fixed set of propositions, the judgement maps every text to a point of a shared proposition space in which the thought of different authors and periods can be compared; IPBench is not a new judge and its tables are not a ranking, but a testbed for the operation that such a space presupposes.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Text-Warranted Proposition Inference, Natural Language Inference, Propositional Reasoning, Language Model Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NK4VOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces IPBench to test whether language models can judge support for an ideational proposition in text. The task first asks whether a passage actually engages with the proposition, then classifies its stance as ACCEPT / REJECT / NEUTRAL. The benchmark contains 516 examples from 172 passages of classical Chinese thought and three propositions, annotated by two experts. The main finding is over-engagement : models often treat unrelated passages as relevant, while author memory and proposition orthodoxy do not provide a stable explanation. The paper also proposes a proposition space for comparing texts in a shared ideological space, but this idea is still tested on a very small setup (e.g., only three propositions, one corpus, and an almost fixed prompt), so its generality remains unclear.

Critical strengths

I find the research question interesting, especially the separation of engagement and stance and the observation of over-engagement . This is a useful perspective, and the overall idea has potential. I would like to see a broader evaluation and a more carefully polished paper.

Critical weaknesses
  • The paper does not even use the full 9-page ICLR limit.
  • The dataset is small and narrow, and the selection process is incomplete. IPBench has only 172 passages and three propositions, and the passages were already filtered for possible relevance. The full screening record was not kept, so the engagement / over-engagement results may depend heavily on the current dataset construction. The main experiments also rely on one fixed zero-shot prompt, with no checks across prompts, more propositions, or other corpora.
Questions for authors
  • The relation to ACL 2026 Semantic-Aware Logical Reasoning via a Semiotic Framework is not clear enough. Both study abstract propositions and stance relations. What is the key difference between IP/TWPI and that work? RepublicQA could also serve as an external test of whether over-engagement exists beyond IPBench.
  • The current evaluation is too narrow. It uses only three propositions, one pre-filtered corpus, and mostly one fixed prompt. Please test more propositions, other corpora, and prompt settings (eg. prompt variation and few-shot) to show that over-engagement is actually robust.
  • The annotation and proposition space need stronger evidence. The first expert's label is directly used as gold, even though engagement with abstract propositions can be subjective. Disagreements should be adjudicated or evaluated separately. The proposition space is also mostly conceptual; a real cross-author or cross-period comparison would make this claim much more convincing.
Reviewer_6myqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Text-Warranted Proposition Inference, a task for determining whether a text engages with a predefined abstract ideational proposition and, if so, whether it accepts, rejects, or engages without a determinate stance toward that proposition. The authors construct IPBench using 172 passages of classical Chinese philosophical writing paired with three predefined propositions, yielding 516 annotated text–proposition pairs. Two expert annotators achieve high agreement (Cohen's kappa score is 0.92). The benchmark is evaluated using zero-shot prompting across several language models, with the primary four models achieving four-way accuracies of approximately 70-78%.

The main empirical observation is that many errors occur at the engagement boundary rather than between acceptance and rejection. All four primary model settings predict more engaged instances than the gold standard, although the magnitude and destination labels differ across models. The authors further investigate whether knowledge of an author's philosophical position or a proposition's doctrinal orthodoxy explains this behavior, finding no consistent evidence for either explanation.

Critical strengths
  1. A useful operational distinction between engagement and stance. The paper identifies a meaningful limitation of conventional stance labels: a text that does not engage a proposition should not necessarily receive the same label as a text that engages it without endorsing or rejecting it. The proposed two-stage formulation and commitment-position criterion provide a relatively explicit way to operationalize this distinction. This is particularly relevant for intellectual-history and document-level interpretation tasks, where attributing an unsupported belief to an author is a consequential error.

  2. Careful annotation and informative error decomposition. The benchmark benefits from expert annotation with substantial agreement score, explicitly stated semantic conventions, separate engagement and stance metrics, and reported rerun variability. The authors also preregister their experimental readings and acknowledge important limitations. The distinction between errors at the engagement boundary and errors between ACCEPT and REJECT provides more useful diagnostic information than four-class accuracy alone.

Critical weaknesses
  1. The benchmark is too narrow to substantiate the generality and distinctiveness of the proposed task.

IPBench contains only three propositions and 172 passages, all drawn from a single selected corpus of classical Chinese philosophical writing. Two propositions concern moral priorities, and the corpus was assembled by selecting passages potentially relevant to the propositions. The selection funnel was not preserved (Section 3.1), making the sampling process difficult to reproduce and leaving the reported engagement prevalence highly dependent on corpus construction.

Most importantly, only 12 of 516 instances are labeled NEUTRAL, despite the distinction between NA and NEUTRAL being central to the proposed task. This makes it difficult to establish whether the benchmark adequately tests that distinction across diverse linguistic and conceptual contexts.

High agreement between two experts following shared semantic conventions supports annotation consistency, but it does not by itself establish broader construct validity. Sensitivity to alternative reasonable conventions, independent propositions, and additional corpora is not evaluated.

There is also a significant novelty-positioning issue. Prior stance detection work, notably Mohammad et al. (2016, Section 2.1), already discusses the difference between genuinely neutral stance and insufficient evidence to infer a stance, although those cases are merged in its evaluation. TWPI's formalization may still constitute a useful contribution, but the paper does not sufficiently demonstrate why its additional distinctions and philosophical framing lead to a substantively new evaluation capability beyond existing stance and relevance-detection formulations.

  1. The experimental design does not adequately distinguish a fundamental engagement-inference failure from prompting and decision-calibration effects.

The headline result is obtained from essentially one fixed zero-shot prompting configuration per model setting. The paper does not systematically evaluate prompt sensitivity, explicit engagement-first prompting, evidence-grounded decisions, or established relevance-plus-stance baselines.

This matters because the authors themselves acknowledge in Section 3.4 that OAI, precision, and recall depend on the operating point and are not standalone measures of capability. The experiments report discrete labels without continuous engagement scores, threshold sweeps, or AUC measurements. Consequently, the observed tendency to predict too many engaged instances could reflect decision calibration or elicitation choices rather than an intrinsic inability to determine engagement.

The results under thinking-enabled settings reinforce this concern: recall increases while precision decreases, consistent with a substantially different operating point.

Furthermore, the paper's characterization of over-engagement as the principal source of missing accuracy needs qualification. In the primary settings, false engagements account for approximately 39-55% of four-way errors (Table 1, FAS); the larger reported 61-77% engagement-boundary error share also includes missed engagements. This distinction is important when interpreting the dominant failure mechanism. DeepSeek's pooled OAI excess is also below the authors' estimated rerun noise floor.

Finally, the analysis of author knowledge is based on measuring a model's prior beliefs and associating them with its errors, rather than directly randomizing or manipulating source attribution in the evaluated passages. This limits the causal interpretation of the reported negative result.

Questions for authors

Q1. Robustness of the central failure mode. Does the reported over-engagement persist under an explicit two-stage prompt that first requires a text-grounded engagement decision and only subsequently requests stance? How sensitive are the results to prompt paraphrasing, evidence requirements, or changes in the engagement decision threshold? A comparison with continuous engagement scores, PR curves, or AUC would help determine whether the observed errors primarily reflect discrimination ability or operating-point calibration.

Q2. Construct validity and baseline comparisons. Can the authors provide evidence that the NA/NEUTRAL distinction generalizes beyond the three selected propositions, particularly given that only 12 instances are labeled NEUTRAL? How stable are the results when the second expert's labels are used as the evaluation reference? I would also like to see comparisons against an always-NA baseline (which achieves approximately 66.3% accuracy given the reported class distribution) and a conventional relevance-detection-plus-stance pipeline.

Q3. Interpretation of the prior-knowledge experiments. The author-memory analysis appears to measure associations between independently elicited model priors and false engagement, rather than directly intervene on source attribution. Could the authors clarify which causal conclusion is justified by this design? Would masking or manipulating author-identifying information change the engagement predictions?

Reviewer_C2SDOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces the "ideational proposition" (IP)—an exogenous, abstract hypothesis—and a new task (TWPI) that evaluates whether a language model can first determine if a text engages with an IP (Engagement) and second, what stance it takes (Stance). To measure this, the authors construct IPBench, a dataset of 516 cells (172 classical Chinese texts paired with 3 propositions) annotated by experts with high agreement (κ=.92\kappa = .92), and design a specialized protocol that explicitly measures "over-engagement" (asserting a text engages a proposition when it does not). Testing four frontier LLMs reveals that models achieve only 70–78% four-way accuracy, with the vast majority of their errors (61–77%) occurring at the engagement boundary. Every model over-engages, hallucinating commitments that are not in the text, though the surplus exits through model-specific labels (e.g., some output NEUTRAL, others output ACCEPT/REJECT). Through controlled pre-registered experiments, the authors demonstrate that this over-engagement is not consistently explained by prior knowledge, such as memorization of the author or the orthodoxy of the proposition; rather, models can correctly identify a stance when prompted, but fail to verify whether the text actually makes the comparison at all.

Critical strengths
  1. The Conceptual Decoupling of "Engagement" and "Stance". This is the paper’s most brilliant contribution. In standard Natural Language Inference (NLI) or stance detection, the model assumes the text is already talking about the hypothesis. The authors correctly identify that this assumption is a luxury we cannot afford in real-world text analysis. By defining an Ideational Proposition (IP)—an exogenous, abstract hypothesis—they force the model to answer a much harder, two-step question: Does the text even engage with this idea? and If so, what is its stance? This mirrors human reading perfectly: we constantly evaluate whether an author is addressing the question at hand before we judge their opinion on it.

  2. Quantifying "Over-Engagement". LLMs are notorious for "yapping" and trying to find relevance where there is none. The paper beautifully isolates this failure mode. The introduction of metrics like OAI (Over-Activation Index) and FAS (False Activation Share) is a methodological gem. The finding that models hallucinate engagement (e.g., seeing a passage about "profit" and wrongly assuming it's making a statement about "law") because they rely on general topical familiarity rather than actual textual commitment is a profound insight into how LLMs "read".

  3. Exceptional Methodological Rigor. The authors did not just prompt an LLM and publish the results. They built a pre-registered, highly controlled experiment. The inter-annotator agreement is exceptional for such an abstract, interpretive task. They established a "noise floor" by running the models twice, ensuring that the reported over-engagement is not just API variance. The controlled experiments (testing if prior knowledge/orthodoxy drives the error) are cleanly executed.

  4. The Philosophical Depth. The theoretical grounding in Brandon's "commitment positions" and Kintsch & van Dijk's "macrostructure" gives the paper a depth that is often missing in AI evaluations. The models are failing because they lack a schema for what the text is supposed to be about.

Critical weaknesses
  1. The Scale is Extremely Small. This is the biggest hurdle for this paper. 516 cells (172 passages × 3 propositions) is tiny by modern benchmark standards. While the annotation is deep and high-quality, evaluating only four models on three propositions from a single corpus (classical Chinese thought) makes it very difficult to claim this is a generalizable capability test for LLMs. A model might over-engage here simply because ancient Chinese texts require specific historical grounding.

  2. The "Bare Prompt" Constraint. The authors use a "bare prompt" with zero examples to test the models. While this is meant to measure intrinsic capability, it is a slightly unfair way to evaluate modern LLMs. If you give a model 3 few-shot examples of NA (Not Applicable) versus NEUTRAL, would its over-engagement drop? The paper argues that reasoning fine-tuning worsens abstention, but it feels like they are testing the model's out-of-the-box behavior rather than its true capability under optimal prompting.

  3. The "Ideational Proposition" Definition is Very Hard to Scale. The definition of an IP requires it to be "exogenous, abstract, and ideational." Generating a pool of these propositions that are broad enough to apply to any text, yet specific enough to allow for stance detection, is incredibly hard. It essentially requires a domain expert (a philosopher or historian) to write them. This makes the benchmark hard to scale to thousands of texts without massive human labor.

  4. The "Singular Gold Standard" Issue. The gold standard is the first annotator's label. Even though the second annotator agreed 96% of the time, the remaining 4% of disagreements are exactly where the philosophical ambiguity lies. If a model disagrees with the first annotator but aligns with the second, it is scored wrong. For a task this subjective, a "majority vote" or a consensus-based gold standard would be more robust.

Questions for authors

First, I want to understand the impact of the "bare prompt" constraint on your central claim that models lack the judgment to distinguish NA from engagement. Did you test any few-shot prompting (for example, providing two or three examples of NA versus NEUTRAL in the prompt)? If few-shot prompting substantially reduces over-engagement, then the failure is a prompt artifact rather than a fundamental lack of capability, which would weaken the paper's strong claim that models "are not equipped with this judgement."

Second, regarding the gold standard being a single annotator's label, you report an excellent κ of .92, but there are still roughly 4% of cells where the two experts disagreed. Did you analyze how often the models' "false engagement" errors on those specific disagreement cells actually align with the second expert's annotation? If the models are being penalized for agreeing with the second human expert, this would artificially inflate the measured over-engagement and affect your conclusions about the direction and volume of the errors.

Third, the benchmark is built entirely on 172 passages of classical Chinese philosophical texts. To what extent do you believe the over-engagement phenomenon is driven by the unique rhetorical style of classical Chinese (where arguments are highly implicit and contextual) rather than being a general property of LLMs? If you were to run a small pilot with even 50 passages from modern English political speeches or legal texts, do you expect the OAI (over-engagement) to remain above 1.0, or does the model become better calibrated when the text type is more familiar to its pretraining distribution?

Comments on the paper

1 comment

Dear Reviewers, as the rebuttal deadline is approaching, please actively engage in the discussion based on the authors' responses. Thank You!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.