IPBench: Evaluating Text-Warranted Proposition Inference in Language Models

Natural language inference (NLI) judges a hypothesis that is drawn from its premise and is true or false of a situation. We study the judgement of a hypothesis with the two opposite properties: it is exogenous, given before any premise is read, and it is abstract and ideational, a judgement that can be accepted or rejected but not verified. We call such a hypothesis an ideational proposition and the judgement of its relation to a text Text-Warranted Proposition Inference (TWPI): one of four labels, not applicable, accepted, rejected, or engaged without a determinate stance, which separates the prior question of engagement from the subsequent question of stance. To measure it we introduce IPBench: 516 cells formed from 172 passages of classical Chinese thought and 3 propositions, labelled by two experts under released semantic conventions ($\kappa =.92$), of which 174 are engaged, with a protocol that scores engagement and stance separately and measures over-engagement as a quantity of its own. Under one bare prompt frozen before any run, four models reach four-way accuracy of only .70 to .78, and the missing accuracy is over-engagement: every model engages more cells than the gold standard contains, by a factor of 1.03 to 1.33, while accept and reject are rarely confused, and two models route the surplus to the neutral label where two route it to a stance. Two pre-registered controlled experiments find that prior knowledge, whether the model's memory of a passage's author or the orthodoxy of the proposition, has little and no consistent effect on the surplus. Applied over a fixed set of propositions, the judgement maps every text to a point of a shared proposition space in which the thought of different authors and periods can be compared; IPBench is not a new judge and its tables are not a ranking, but a testbed for the operation that such a space presupposes.