acceptodds
Under review as a conference paper at ICLR 2027

Answer Set Consistency of LLMs for Question Answering

Abstract

Large Language Models (LLMs) can contradict themselves when answering factual questions, especially when asked to enumerate all entities that satisfy a question. We formalize such self-contradiction as answer-set inconsistency: given two enumeration questions whose answers satisfy a set-theoretic relation (equivalence, disjointness, containment, etc.), the LLM generates responses violating the relation. To diagnose this phenomenon, we create Answer-Set Consistency Benchmark (ASCB), a benchmark dataset comprising 600 tuples of enumeration questions (2400 questions in total) over which a variety of set-theoretic relations hold, and propose related metrics to quantify answer-set inconsistency. Our evaluation of 18 state-of-the-art LLMs reveals pervasive inconsistency across models, even when the LLM can identify the correct relation. We also analyze likely sources of these failures and use relation-aware prompting as a lightweight exploratory intervention. Making the relation explicit often improves consistency, but gains are uneven and can be offset by abstention or prompt misinterpretation. Our results position answer-set consistency as a complementary dimension for evaluating LLM-based question answering.

Primary area:
datasets and benchmarks
Keywords:
LLMs, Question Answering, Consistency

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ngymOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The work introduces the Answer-Set Consistency Benchmark (ACSB), evaluating the ability of LLMs to produce answers to question groups of 4 (quadruples), while abiding to set-theoretic relationships among them. The benchmark consists 600 quadruples and totals 2400 questions. Its items are made up of verbatim and adapted questions from three knowledge-graph question-answering benchmarks (LC-QuAD 2.0, QQALD, and QAWiki). They also include a completely novel set of 150 questions, that the authors constructed.

The authors reports results in three evaluation settings across 18 LLMs, including a variety of providers, sizes, and open-/closed-weight status. They generally find that, without knowledge of the relationships, models produce results that don’t abide by the intended relationships. This improves when either provided with or previously asked about the intended relationship between a question pair (pair within the quadruple).

Critical strengths
  1. Released dataset: The work introduces ASCB, a dataset that evaluates the ability to produce results that adhere to set-theoretic relationships among a question-quadruple. Results vary strongly across models. The manual verification of all benchmark items by the authors also stands out positively. I skimmed through the dataset and the properties described in the paper are accurately reflected.
  2. Well written: The work motivates their benchmark well, provides good running examples and explanatory tables, and presents the experimental results in a clear way.
  3. Extensive experimental results: The authors evaluate the dataset in three evaluation settings across 18 LLMs. The chosen evaluation settings are also convincing. The authors also report extensive ablations and further analyses in the appendix.
Critical weaknesses
  1. Dataset size/novel contribution: While I understand the combinatorial effect increasing the dataset size has on the experimental results (and the resulting costs), it would have been great to include more questions synthesized by the authors (increasing the ratio of novel questions to existing/copied questions).
  2. (Minor) discrepancies in released artifacts: Please see questions and comments below.
Questions for authors

The E1,∗E_{1,*} results are a little unclear to me.

Q1: Why do you mix the repeated base setting (as claimed in the paper) with the CtE setting and report the mean of both (summary_idk_worst.py, L193)?

Q2: The provided results indicate one additional run for this setting, with some models missing these additional base runs (551/600 for gpt-4.1-nano, 296 for deepseek-reasoner, 148 for gpt-oss-20b, llama-3.1-8b, mistral-small-24b, and 0 for grok-3-mini, llama-3.1-70b)?

Unrelated to E1,∗E_{1,*}:

Q3: Given that you don’t focus on factually correct answers (e.g. the correct list of all fifa world cup winners), how do you see this connecting to the broader QA benchmark landscape that grounds nearly all work in answering factually correct given a reference source (e.g. documents, web-search, etc.)?

Additional Comments:

  1. While overall very well written, the writing degrades strongly in section 3.2 and parts of 3.3. Suddenly lots of LLM-isms in terms of terminology and writing style appear, which strongly contrast the very well written abstract, introduction and related work discussion. Also, the discussion sections appears to repeat this. Please rewrite these parts for clarity.
  2. The claim of “a novel handcrafted benchmark” should be narrowed to match that ASCB mostly builds on existing QA pairs and is manually augmented.
  3. While a cosmetic change, I would have found results for models grouped by families/providers more informative instead of the chosen ordering, given how dense the table is.
  4. Reproducibility: I couldn’t reproduce Tables 9-11 from the provided code. I also only found 300/600 GPT-oss-20b Oracle results.
5 comments

We thank the reviewer for their constructive feedback. We appreciate the opportunity to explain how ASCB was constructed from existing QA resources. While several source questions served as starting points, developing a consistent set of related enumeration questions required substantial transformation and manual verification. We conducted an additional question-reuse analysis to quantify this contribution and clarify the benchmark's construction. Finally, we addressed the reported reproducibility issues by correcting the evaluation pipeline, completing the released data, and updating the corresponding results. We have incorporated the relevant changes into the revised manuscript and provide detailed responses below.

W1 and Additional Comments (2): We agree that “a novel handcrafted benchmark” is imprecise and will use “a manually curated benchmark”. As Section 3.2 and Appendix G explain, QAWiki, LC-QuAD, and QALD provided starting points, not ready-made question sets. Three authors substantially revised and verified questions to construct related enumeration quadruples. In truth, few input questions made it unmodified to the final dataset, but rather many served as inspiration. This saved time versus inventing questions completely from scratch.

To quantify reuse, we compared the 1,800 questions in ASCB's three source-based subsets with original English questions and paraphrases. We checked exact and formatting-normalized matches (Unicode, case, whitespace). For nonmatches, we retrieved the five nearest source questions using bge-small-en-v1.5 embeddings and cosine similarity, then assessed whether answer conditions, entities, time, scope, or constraints changed. Q2Q_2 inherited Q1Q_1's classification because they are equivalent; nonverbatim Q3Q_3 and Q4Q_4 were classified as substantial changes when they introduced the required subset or complement condition, apart from identified close source rephrasings.

We found 338 verbatim matches (18.8%), 23 formatting-only variants (1.3%), 248 close rephrasings (13.8%), and 1,191 substantial changes or questions without close counterparts (66.2%). Only 35 of 900 Q3/Q4Q_3/Q_4 questions matched source wording verbatim.

Furthermore, the separate synthetic subset comprises 600 final questions that were manually selected and revised from LLM-generated candidates, rather than directly adopting the generated outputs.

Regarding benchmark size, ASCB contains 600 manually verified quadruples (2,400 questions). Its size already supports significant findings: Appendix J reports one-sided paired McNemar improvements over Base for CtE in 79/90 and Oracle in 83/90 model-relation comparisons (p<0.05). These results show that ASCB is large enough to support the paper’s main empirical conclusions while maintaining a consistent, manually verified evaluation structure.

We added the reuse audit to Section 3.2 and Appendix G and will release its code.

Q1–Q2: The repeated-Q1Q_1 control E1,∗E_{1,*} measures answer stability when the same enumeration question is asked again, for comparison with equivalent rephrasings. To conserve evaluation budget, we reused Q1Q_1 responses from CtE runs and collected an additional isolated Base Q1Q_1 response where available. The Base repetitions were incomplete. When both scores exist, line 193 averages repeated-Base and CtE scores equally at model level; otherwise, it uses CtE alone. We clarified this mixed-context procedure in the manuscript.

Q3: Answer-set consistency complements factual accuracy. For example, winners of the FIFA World Cup from Latin America must be a subset of all winning national teams. Correct answers necessarily respect this containment, but incomplete or incorrect answers can also satisfy it. Consistency is therefore necessary, not sufficient, for correctness, and can be evaluated without exhaustive reference sets.

Following reviewer jYSP's question, we analyzed 92 ASCB groups with author-reviewed candidate answer sets. Across 18 models, Q3/Q4Q_3/Q_4 satisfied disjointness in 66.4% of Base cases on average, but both answers exactly matched their candidate sets in only 25.8%. This illustrates why consistency cannot substitute for factual correctness. The approach can also complement document- or search-grounded QA by assessing both evidence-based accuracy and relational consistency. We clarified Section 3.1 and added Appendix K.

Additional Comments (1): We revised Sections 3.2–3.3 and the Discussion to clarify benchmark construction, tasks, and interpretation.

Additional Comments (3): Revised result tables group models by family and provider and order them for easier comparison.

Additional Comments (4): We fixed the reproduction pipeline and added a single-command exporter for Tables 9-11, correcting data paths and metric mappings. We also found that GPT-oss-20b Oracle outputs contained only 300 of the expected 600 complete groups. We uploaded the missing results and updated the manuscript tables.

Thank you for the detailed response to the review. Please see my response to the individual sections below:

W1 and AC2: I believe adding these details to Section 3.2 improves the presentation of the authors' work and eases my concerns about the ratio of "novel" questions/benchmark items. Could you clarify why you chose to describe these statistics as "provisionally" in the revised manuscript?

Q1-Q2: In the revised description in the manuscript LL335-340, you describe that this fallback is applied to every model. However, in the code, it's only applied to Llama-3.1-70b and Grok-3-mini. Therefore, the missing repeats for the other models mentioned in my initial review remain missing.

Q3: Thank you; this answers my question. While this doesn't concern my review, I am unable to see any revisions to Section 3.1.

AC1: Unfortunately, the provided revisions seem to add content in response to my/other reviewers' concerns instead of actually revising the presentation of these sections. This remains an open comment.

AC3: The ordering has improved. I'm still not sure why both Llama models aren't grouped, and also why GPT-oss isn't listed alongside the other GPT models. I would appreciate this fix.

AC4: I can confirm the 300 missing GPT-oss-20b Oracle results are now provided. However, these added results don't seem to be reflected in Table 3.

AC4: Similarly, while the export code for Tables 12-14 (formerly 9-11) is now provided, the values don't seem to match. Please refresh your results tables.

We thank the reviewer for their careful reading and constructive feedback. Below, we address the additional concerns raised.

W1 and AC2: Thank you for the question. ‘‘Provisional“ applies only to the semantic classifications, the 338338 verbatim and 2323 formatting-only matches were established by reproducible text checks. For the remaining 1,4391{,}439 questions, we retrieved the five closest source questions using cosine similarity for ranking only. We classified a question as a close rephrasing when it preserved meaning, target entities, answer conditions, and scope. Changes to those elements were classified as substantial. We did not use an uncalibrated similarity threshold because similar wording can imply different answers. The initial audit reviewed Q1Q_1 against its source anchor or retrieved candidates, inherited Q1Q_1’s label for Q2Q_2, and generally treated Q3/Q4Q_3/Q_4 subset or complement questions as substantial changes unless a close source rephrasing was found. This yielded 248248 close rephrasings and 1,1911{,}191 substantial changes or questions without a close counterpart. These counts are provisional because inherited labels and construction rules did not independently establish every question’s provenance. A human reviewer is now checking each nonmatching question against its retrieved candidates. We will report updated counts and clarify the procedure in Section 3.2 and Appendix G during the rebuttal period.

Q1-Q2: Thank you for pointing this out. We apologize for the ambiguity in our previous description. In the code accompanying that revision, the CtE fallback was applied only to Llama-3.1-70b and Grok-3-mini because they were the only two models with no independent Q1Q_1 repeats at the time. For models with some repeats, the remaining questions were indeed left missing, as you observed. To remove this ambiguity, we have generated the missing independent Q1Q_1 repeats for the other models, including Llama-3.1-70b. For the 16 models with complete coverage, our proposed final E1,∗E_{1,*} analysis uses all 600 independently repeated Q1Q_1 questions and no CtE answers. The remaining exceptions are Gemini-2.0-flash (590/600) and Grok-3-mini (0/600). We could not obtain the remaining responses from these original models through the APIs available to us. We currently retain their other results and marked E1,∗CONE_{1,*}^{CON} and E1,∗SIME_{1,*}^{SIM} values as unavailable. If the reviewer prefers a model set with complete repeated-Q1Q_1 coverage, we are also open to replacing them with Gemini-3.1-flash-lite and Grok-4.3, respectively, and running all experimental conditions for the replacement models.

Q3: We revised Section 3.1 to clarify that answer-set consistency is necessary but insufficient for factual correctness and complements reference-based QA. We also added a pointer to Appendix K, which examines this distinction empirically.

AC1: We agree that Sections 3.2, 3.3, and the Discussion should be as clear as earlier sections. They were revised in the preceding version, but many edits were not highlighted in blue because they rewrote or reorganized existing text rather than added rebuttal material. This may have obscured the changes.

Following the reviewer’s feedback, we made a further editing pass focused on conciseness and readability. We shortened Section 3.3’s introduction and clarified the differences among Base, CtE, and Oracle. We also simplified the Discussion and divided longer passages into shorter paragraphs. We retained the RQ1-RQ5 structure to keep the conclusions connected to the research questions, while making each answer more direct. We hope that these revisions address the reviewer's concern by improving the presentation of the existing material. We appreciate the feedback, which has helped us make the writing more consistent with the rest of the paper.

A3: Previously, tables separated open- and closed-source models, with open-source models ordered by parameter count. This separated GPT-oss-20b from other OpenAI models and the two Llama models from each other. We now group models by family, as suggested.

AC4: We apologize for the oversight. The GPT-oss-20b Oracle results had been uploaded but were missing from Table 3-4, which we have now corrected.

AC4 (Table 12-14): We regenerated these tables from per-question outputs and highlighted changes in blue. The previous Table 12 export could mark an incorrect relation prediction as contradiction-free when answers also violated the expected relation. The corrected metric verifies that the predicted relation holds. Tables 13-14 now use the main experiment’s worst-case null-answer policy, exclude missing classifications, and apply the specified partition and difference metrics. These corrections affect the auxiliary relation-prediction analysis reported in the Appendix L. Table 3 uses fixed ASCB relations and does not depend on these scores, so the paper’s main answer-set consistency finding is unchanged.

Reviewer_jYSPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper examines a reliability issue in enumeration question answering: answer lists generated by LLMs for related questions may contradict one another. For example, entities listed as members of a subgroup may be absent from the model’s own list of the broader group. To systematically investigate this phenomenon, the authors introduce ASCB, which organizes 2,400 English questions into 600 quadruples with known equivalence, containment, disjointness, and set-difference relations. They evaluate 18 models under three conditions: independent answering, identifying relations before answering, and answering with the correct relations explicitly provided. The results show that models may correctly identify relations between questions while still generating answer sets that violate those relations; explicitly providing relation information generally improves measured consistency. Through this benchmark and cross-model analysis, the paper complements conventional factual accuracy evaluation with a way to assess whether related answers are mutually consistent.

Critical strengths
  1. The research problem has practical relevance. Per-question accuracy does not directly capture consistency across answers to related questions. By checking explicit set-theoretic relations without requiring exhaustive gold answers for every question, the paper provides a reusable, complementary evaluation dimension for question-answering systems.
  2. The dataset has a clearly defined structure and undergoes manual verification. The relations within each quadruple make the evaluation target explicit and allow consistency to be assessed without collecting exhaustive gold answers. The final items were manually revised and independently verified by at least two authors, providing support for the benchmark’s quality.
  3. The evaluation covers a broad range of models. The experiments include 18 models from different families and of different sizes. This demonstrates that the observed consistency problems occur across multiple models and broadens the empirical relevance of the findings.
Critical weaknesses
  1. The scoring does not fully capture semantic consistency. Different names for the same entity may be counted as inconsistent. Although the appendix quantifies alias-related issues, it does not report relation scores after entity normalization. The handling and filtering of abstentions and empty answers also require clarification.
  2. The fixed relation structure limits the scope of the conclusions. Although the questions draw on multiple sources and topics, every item follows the same quadruple structure: two equivalent questions and two questions defining disjoint, exhaustive subsets. Real-world enumeration questions can involve other structures. For example, employees participating in project A and those participating in project B may partially overlap without either group containing the other. Such relations are not primary evaluation targets in the current benchmark.
  3. Factual omissions are not sufficiently distinguished from failures to understand logical relations. Inconsistent answer sets may arise because a model misunderstands a relation, but they may also result from incomplete factual recall. For example, a model may understand that boarding students are a subset of all students in a class, yet omit a student when enumerating the full class, thereby violating containment. This indicates unreliable output, but does not necessarily demonstrate a failure to understand containment.
Questions for authors
  1. Do the main findings hold after entity normalization? Appendix D quantifies cases in which different names refer to the same entity. Could the authors normalize these names and report the resulting consistency scores, model rankings, and gains from CtE and Oracle, to establish how much the main conclusions are affected?
  2. How exactly do abstentions and empty answers affect scoring? Please clarify how idk, no answer, and empty responses are handled in each metric, including the denominator calculations. For results that filter abstentions, is an entire relation instance excluded whenever any answer set required to evaluate that relation is unavailable? Under this treatment, what are the scores and effective sample sizes for each relation?
  3. How does the fixed quadruple structure limit the scope of the conclusions? The current benchmark primarily evaluates equivalence, containment, and disjoint partitions. Why were relations such as partial overlap without containment not included as primary evaluation targets? Is there any validation on other relation structures? If not, please clarify which conclusions apply specifically to the current template.
  4. What evidence distinguishes factual omissions from failures to apply set relations? A model may understand containment but produce inconsistent answer sets because it omits an entity. Have the authors conducted controlled experiments, such as providing complete, verified facts before evaluating answer-set consistency? If not, how does the current analysis support its conclusions about the main causes of inconsistency?
2 comments

We thank the reviewer for the thoughtful comments on entity normalization, abstention handling, benchmark structure, and answer-set inconsistency. These points help clarify ASCB’s scope, which is to assess whether generated answers respect known relations, independently of entity recall accuracy. We conducted additional analyses to assess robustness and study the relationship between consistency and factual knowledge.

W1/Q1: We agree that aliases should not be mistaken for inconsistencies. We therefore repeated the evaluation using Wikidata entity normalization. Substantial inconsistency persists: CtE improves mean consistency for 17/18 models and Oracle for all 18. Appendix D and expanded Table 5 report original and normalized scores and gains.

Using the same abstention policy as Table 3, mean Base CON changes from 30.99% to 32.31%, CtE from 59.07% to 58.50%, and Oracle from 62.12% to 61.47%. Normalized gains over Base are 26.19 (CtE) and 29.17 (Oracle) points, versus 28.08 and 31.13 before normalization. Scores may decrease when previously disjoint surface forms resolve to the same entity. Thus, lexical variation contributes measurement noise but does not explain the observed inconsistency.

W2/Q3: ASCB's fixed four-question structure enables systematic evaluation of relations established from question semantics without enumerating complete answer sets. Partial overlap requires a shared entity and one exclusive to each set. For example, containment between “tributaries of the Rhine” and “left-bank tributaries of the Rhine” follows from their definitions, whereas partial overlap with “rivers entirely within Switzerland” depends on the actual entities. Numerical or temporal constraints could create such cases but poorly fit ASCB's dichotomous partitions. Adding Q5Q_5 would require new verified cases and potentially artificial constraints. We clarified this limitation and identify partial overlap as future work.

Q2: We explicitly accounted for model abstentions to prevent models from appearing consistent simply by declining to answer. In our main evaluation, relations involving unavailable answers receive the worst score. We additionally report a sensitivity analysis treating unavailable individual answers as empty sets. Since ASCB reference answers are nonempty, standardized “idk,” no-answer, and empty responses are treated as unavailable. In Table 3, relations involving unavailable answers receive their worst score: binary CON metrics and E1,2SIME_{1,2}^{SIM}/E4,1∖3SIME_{4,1\setminus3}^{SIM} receive 0; D3,4SIMD_{3,4}^{SIM} receives 1. In Table 4's sensitivity analysis, each unavailable answer is replaced individually by ∅\varnothing; other answers in its group remain unchanged. Neither approach drops groups or changes denominators: n=600n=600 per relation for both CON and SIM.

W3/Q4: We agree that containment violations do not necessarily imply a misunderstanding of the underlying set-theoretic relation. Our evaluation explicitly separates relation classification (Task 1.1) from answer enumeration (Task 1.2), and distinguishes answer-set inconsistency from contradiction (Section 3.1 and Appendix L). Table 13 further shows that even when GPT-5-nano correctly identifies containment, its generated answers satisfy it in only 35.3% and 27.6% of cases for N3,1N_{3,1} and N4,1N_{4,1}, respectively, under the Base setting. Thus, correct relation recognition does not guarantee consistent answer generation.

To further investigate the role of factual knowledge, we annotated gold answer sets for 92 question groups, requiring exact agreement between at least one human annotator and two independent LLM agents, plus approval by another human reviewer. For GPT-4.1-mini and GPT-5-nano, we supplied an unlabeled list of entities from each group's broad reference set before asking Q1Q_1-Q4Q_4 separately, without mappings or relation labels, and compared this with fresh no-list runs.

Mean consistency increased across all five relations, from 46.5% to 67.2% for GPT-4.1-mini and 57.2% to 71.5% for GPT-5-nano. For GPT-4.1-mini, equivalence improved 58.7%->83.7%, disjointness 35.9%->44.6%, and set difference 15.2%->32.6%. Across both containment relations, GPT-4.1-mini improved from 113/184 (61.4%) to 161/184 (87.5%); GPT-5-nano from 111/184 (60.3%) to 139/184 (75.5%).

Of GPT-4.1-mini's containment failures, a reference narrow-set entity was absent from its broad answer in 16/71 without the list and 15/23 with it. In a separate task, directed containment classification was correct in 44/184 cases (23.9%) for GPT-4.1-mini versus 167/184 (90.8%) for GPT-5-nano. Yet GPT-5-nano had 19 containment failures with available answers in the list condition, all correctly classified in that separate task. Entity availability therefore contributes to consistency but does not fully explain failures; recognizing relations and generating compatible sets remain distinct. We clarified this in Section 3.4, Appendix K.2, and Limitations/Future Work.

Reviewer_jYSPOpenReview ↗

Official Review of Submission45436 by Reviewer jYSP

Thank the authors for their thorough rebuttal with new experiments addressing entity normalization, abstention handling, and the distinction between factual omissions and relation-understanding failures; accordingly, we have raised our score.

Reviewer_UDUCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates consistency in enumeration question answering: whether the answers listed by a model when responding to several related questions are mutually consistent. The authors constructed the Answer-Set Consistency Benchmark, comprising 600 question quadruples and a total of 2,400 questions, and evaluated 18 models. The paper compares consistency under different set relations and discusses the causes of inconsistency through repeated questioning and error analysis. The authors also experimented with having models first determine the relations before answering, or, upon detecting inconsistencies, providing the correct relations and having the models answer again. The experiments found that even when models can identify the relations between questions, they do not necessarily provide consistent answers; these two approaches typically improve consistency.

Critical strengths
  1. I believe one valuable aspect of this paper is that it expands the scope of enumeration question answering tasks from "how well each question is answered" to "whether the answers to several related questions align with one another." Section 3.2 organizes equivalence, containment, disjointness, and set-difference relations into sets of four questions each, ultimately forming 600 question quadruples, each of which was independently verified by at least two authors. This design makes cross-question consistency directly verifiable and eliminates the need to prepare complete gold-standard answer sets for each question in order to detect some errors.

  2. The paper’s distinction between relation identification and actual responses also provides valuable insights. For example, in Table 6 of Appendix I, Gemini-2.5-pro achieved a 91.33% accuracy rate in identifying equivalence relations, but the equivalence consistency of its independent responses to individual questions, as shown in Table 3, was only 29.50%. This demonstrates that relation identification accuracy cannot substitute for the verification of actual answers. Appendix K further analyzes answer-set consistency based on whether the predicted relations were correct, while Appendices A–D examine the effects of abstentions, formatting, and aliases, providing supplementary insights for understanding these results.

Critical weaknesses

The paper provides valuable descriptive results for RQ1–RQ3. My main questions focus on RQ4 and RQ5:

  1. My main question is whether the existing experiments are sufficient to determine the primary cause of the inconsistencies.

    Lines 470–472 of Section 5 suggest that inconsistencies between equivalent questions stem primarily from stochasticity, while those under other types of relations stem primarily from semantic factors (that is, misunderstandings of the question or the relation). This conclusion is stronger than the assertion that “both types of factors may play a role” and requires more direct evidence.

    The paper uses the results of repeated responses to the same question as a control and compares them with the results from equivalently phrased questions. This demonstrates that even when the input remains unchanged, the model’s answers can vary. However, rephrasing a question might also cause the model to list different facts or omit different answers. Therefore, the fact that consistency scores are similar in both cases does not establish that stochasticity is the primary cause of inconsistency between equivalent questions.

    When comparing consistency scores for different set relations, differences in scoring rules must also be taken into account. Consider the following hypothetical example:

    • Equivalence relation: Both questions should be answered as {A, B}. The model answers {A, B} and {A}, respectively. Because the two answers differ, they are judged to be inconsistent.
    • Disjointness relation: The two questions should be answered with {A, B} and {C, D}, respectively. The model answers {A} and {C, D}, respectively. Although B is again omitted, the two answer sets still have no common elements, so the disjointness relation holds.

    Therefore, even if both types of questions exhibit the same degree of omission, they may still yield different consistency scores. If these omissions result from random fluctuations, such score discrepancies may occur even if the model has not misunderstood the relation. Section 3.4, lines 329–331, assumes that stochasticity plays a similar role for different relations, but this premise still needs to be verified. Based solely on these score discrepancies, it is not yet possible to attribute inconsistencies under certain relations primarily to semantic understanding issues.

    The discussions and error cases in Appendices L and M provide some clues, but they are still insufficient to determine which cause is dominant. I hope the authors will first repeat each question formulation multiple times on a representative small subset, comparing the variability for the same formulation with the changes resulting from switching formulations.

  2. The existing comparisons do not clearly indicate how much improvement relation-aware prompting itself provides.

    Section 3.3 refers to answering questions one by one in independent dialogues as "Base"; answering sequentially within the same dialogue after first determining the relation is termed "CtE." The authors also introduce an "Oracle" setup: upon detecting inconsistent answers, the model is informed of the correct relation and asked to respond again. Section 4 reports that CtE improved average consistency on 17 out of 18 models, while Oracle outperformed Base on all 18 models.

    These results indicate that the entire response process has the potential to improve consistency, but they are insufficient to determine the additional effect of the relation information itself. CtE’s subsequent responses can reference previous answers, and Oracle also allows the model to respond again after receiving feedback. These changes alone may improve consistency. For example, when two equivalent questions appear in the same dialogue, the model may reuse the previous answer even without first determining the relation.

    I hope the authors will add two direct comparisons using a representative model:

    a. Keep the question order unchanged and have the model answer the questions sequentially within the same dialogue without first determining the relation, then compare this to CtE.

    b. Starting from the same set of initially inconsistent answers and retaining the same context, one group receives only the prompt "Please check and answer again," while the other receives the correct relation; both groups then answer again.

    These two controls would allow for a more direct examination of whether determining the relation first or providing the correct relation offers more benefit than simply referring to the context and answering again after checking.

Questions for authors
  1. For the examples that changed from inconsistent to consistent under CtE and Oracle, is it possible to conduct a sample check of the correctness and completeness of the answers? Have there been cases where the answers became more consistent, but factual errors or omissions actually increased?

  2. While using AI-assisted checking to review the supplementary code, I discovered several inconsistencies: the repeated-question control mixed results from different conditions; the set-difference scoring was stricter than defined in the main text; and the treatment of abstentions in the significance test differed from that in the main table. Are these discrepancies due to issues with the code implementation, inaccuracies in the paper’s description, or differences in file versions? After verification, do the relevant results and conclusions need to be revised?

1 comment

We thank the reviewers for their constructive feedback. We conducted additional experiments on answer-set inconsistency, correctness, and relation-aware prompting and dialogue context, corrected evaluation issues, and updated the manuscript. Detailed responses follow.

W1: Our central finding is that LLM answer sets often violate expected set-theoretic relations even when models identify those relations correctly. Repetition variability, phrasing sensitivity, and factual omissions may all contribute. To assess correctness separately from consistency, we annotated 92 question groups. Each reference set required agreement among at least one human annotator and two independent LLM agents, followed by approval from another human reviewer. Appendix K, Table 9 now reports entity F1 for Q1Q_1-Q4Q_4 under Base, CtE, and Oracle. The annotated dataset is available in the anonymous repository in data/ASCB/en/unified_dataset_w_candidate_answer.tsv.

Across 18 models and 1,100 matched cases, rephrased questions yielded identical answers 61.6% of the time, versus 62.5% for repetitions of the same question. Moreover, among 1,435 Q3Q_3-Q4Q_4 pairs, 1,099 passed disjointness, although 590 omitted at least one reference entity; 72 had omissions without extra entities, and 10 matched the reviewer's single-omission example. Thus, disjointness can hold for incomplete answers. We revised Section 5 to distinguish this observation from causal explanations.

W2: We tested repetition and wording effects separately, generating ten independent Base responses for each of Q1Q_1-Q4Q_4 using GPT-4.1-mini on the 92 annotated groups. Mean within-formulation exact agreement was 76.7% (Q1Q_1), 72.8% (Q2Q_2), 73.4% (Q3Q_3), and 63.2% (Q4Q_4). For equivalent Q1Q_1/Q2Q_2 formulations, agreement was 74.7% within a formulation versus 55.9% across formulations; candidate-entity inclusion disagreement rose from 1.6% to 4.9%. Both stochastic variability and phrasing matter, and variability differs by formulation. We added Appendix K.1 and removed the equal-variability assumption from Section 3.4. We are open to extending our experiments beyond GPT-4.1-mini to include additional models.

W3: The original CtE and Oracle experiments were intended to evaluate lightweight conversational strategies to mitigate inconsistency, rather than isolate the effect of relation information from dialogue context or revision opportunities. In order to investigate the reviewer's concerns about how relation information is introduced, we conducted an additional two experiments with GPT-4.1-mini across all the 600 questions.

  • a) First, keeping dialogue structure and question order fixed, sequential answering without relation classification achieved 74.88% mean consistency, compared with 65.70% for CtE (-9.18 points). CtE also increased abstentions (18.58% versus 6.1%). Thus, relation classification first did not help overall in this setting.

  • b) Second, for initially inconsistent sets, we compared a plain “Please check and answer again” prompt with the same prompt preceded by the correct relation. The latter raised correction rates from 12.0% to 21.1% (equivalence), 15.5% to 44.8% (containment), and 9.2% to 19.0% (set difference). Overall CON rose from 71.67% to 74.07%. These results suggest relation information can aid targeted revision, though not necessarily initial sequential answering.

We can incorporate these findings after extending experiments to all 18 models.

Q1: Consistency is not equivalent to correctness. Using the 92 annotated groups, we assessed omissions and non-reference entities when a model changed from inconsistent under Base to consistent under CtE or Oracle. Across five relations, there were 1,901 Base-to-CtE and 1,963 Base-to-Oracle scoreable gains. Omissions increased in 491 (25.8%) and 505 (25.7%) cases, respectively; non-reference entities increased in 385 and 523 cases. For North by Northwest, CtE/Oracle made two responses consistent by omitting Indiana from both, rather than only one. We now discuss this trade-off in Section 5 and Appendix K.

Q2: We thank the reviewer for identifying these issues.

  • The E₁,∗E_{₁,*} control deliberately reused Q1Q_1 responses from CtE runs to conserve evaluation budget, alongside an additional isolated Base response when available. As Base repetitions were incomplete, we averaged the repeated-Base and CtE scores with equal weight at model level when both existed; otherwise we used CtE alone. This mixed-context design is now clarified.
  • We also aligned abstention handling between the main table and significance tests and clarified the set-difference metric to match its implementation. Recalculation changed the significance status of 23/180 tests. We revised the tables, Oracle–CtE comparisons, and interpretations. The core findings of substantial inconsistency and generally improved consistency under relation-aware prompting remain, subject to the limitations highlighted above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.