- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The work introduces the Answer-Set Consistency Benchmark (ACSB), evaluating the ability of LLMs to produce answers to question groups of 4 (quadruples), while abiding to set-theoretic relationships among them. The benchmark consists 600 quadruples and totals 2400 questions. Its items are made up of verbatim and adapted questions from three knowledge-graph question-answering benchmarks (LC-QuAD 2.0, QQALD, and QAWiki). They also include a completely novel set of 150 questions, that the authors constructed.
The authors reports results in three evaluation settings across 18 LLMs, including a variety of providers, sizes, and open-/closed-weight status. They generally find that, without knowledge of the relationships, models produce results that don’t abide by the intended relationships. This improves when either provided with or previously asked about the intended relationship between a question pair (pair within the quadruple).
Critical strengths
- Released dataset: The work introduces ASCB, a dataset that evaluates the ability to produce results that adhere to set-theoretic relationships among a question-quadruple. Results vary strongly across models. The manual verification of all benchmark items by the authors also stands out positively. I skimmed through the dataset and the properties described in the paper are accurately reflected.
- Well written: The work motivates their benchmark well, provides good running examples and explanatory tables, and presents the experimental results in a clear way.
- Extensive experimental results: The authors evaluate the dataset in three evaluation settings across 18 LLMs. The chosen evaluation settings are also convincing. The authors also report extensive ablations and further analyses in the appendix.
Critical weaknesses
- Dataset size/novel contribution: While I understand the combinatorial effect increasing the dataset size has on the experimental results (and the resulting costs), it would have been great to include more questions synthesized by the authors (increasing the ratio of novel questions to existing/copied questions).
- (Minor) discrepancies in released artifacts: Please see questions and comments below.
Questions for authors
The results are a little unclear to me.
Q1: Why do you mix the repeated base setting (as claimed in the paper) with the CtE setting and report the mean of both (summary_idk_worst.py, L193)?
Q2: The provided results indicate one additional run for this setting, with some models missing these additional base runs (551/600 for gpt-4.1-nano, 296 for deepseek-reasoner, 148 for gpt-oss-20b, llama-3.1-8b, mistral-small-24b, and 0 for grok-3-mini, llama-3.1-70b)?
Unrelated to :
Q3: Given that you don’t focus on factually correct answers (e.g. the correct list of all fifa world cup winners), how do you see this connecting to the broader QA benchmark landscape that grounds nearly all work in answering factually correct given a reference source (e.g. documents, web-search, etc.)?
Additional Comments:
- While overall very well written, the writing degrades strongly in section 3.2 and parts of 3.3. Suddenly lots of LLM-isms in terms of terminology and writing style appear, which strongly contrast the very well written abstract, introduction and related work discussion. Also, the discussion sections appears to repeat this. Please rewrite these parts for clarity.
- The claim of “a novel handcrafted benchmark” should be narrowed to match that ASCB mostly builds on existing QA pairs and is manually augmented.
- While a cosmetic change, I would have found results for models grouped by families/providers more informative instead of the chosen ordering, given how dense the table is.
- Reproducibility: I couldn’t reproduce Tables 9-11 from the provided code. I also only found 300/600 GPT-oss-20b Oracle results.
5 comments
Thank you for the detailed response to the review. Please see my response to the individual sections below:
W1 and AC2: I believe adding these details to Section 3.2 improves the presentation of the authors' work and eases my concerns about the ratio of "novel" questions/benchmark items. Could you clarify why you chose to describe these statistics as "provisionally" in the revised manuscript?
Q1-Q2: In the revised description in the manuscript LL335-340, you describe that this fallback is applied to every model. However, in the code, it's only applied to Llama-3.1-70b and Grok-3-mini. Therefore, the missing repeats for the other models mentioned in my initial review remain missing.
Q3: Thank you; this answers my question. While this doesn't concern my review, I am unable to see any revisions to Section 3.1.
AC1: Unfortunately, the provided revisions seem to add content in response to my/other reviewers' concerns instead of actually revising the presentation of these sections. This remains an open comment.
AC3: The ordering has improved. I'm still not sure why both Llama models aren't grouped, and also why GPT-oss isn't listed alongside the other GPT models. I would appreciate this fix.
AC4: I can confirm the 300 missing GPT-oss-20b Oracle results are now provided. However, these added results don't seem to be reflected in Table 3.
AC4: Similarly, while the export code for Tables 12-14 (formerly 9-11) is now provided, the values don't seem to match. Please refresh your results tables.
We thank the reviewer for their careful reading and constructive feedback. Below, we address the additional concerns raised.
W1 and AC2: Thank you for the question. ‘‘Provisional“ applies only to the semantic classifications, the verbatim and formatting-only matches were established by reproducible text checks. For the remaining questions, we retrieved the five closest source questions using cosine similarity for ranking only. We classified a question as a close rephrasing when it preserved meaning, target entities, answer conditions, and scope. Changes to those elements were classified as substantial. We did not use an uncalibrated similarity threshold because similar wording can imply different answers. The initial audit reviewed against its source anchor or retrieved candidates, inherited ’s label for , and generally treated subset or complement questions as substantial changes unless a close source rephrasing was found. This yielded close rephrasings and substantial changes or questions without a close counterpart. These counts are provisional because inherited labels and construction rules did not independently establish every question’s provenance. A human reviewer is now checking each nonmatching question against its retrieved candidates. We will report updated counts and clarify the procedure in Section 3.2 and Appendix G during the rebuttal period.
Q1-Q2: Thank you for pointing this out. We apologize for the ambiguity in our previous description. In the code accompanying that revision, the CtE fallback was applied only to Llama-3.1-70b and Grok-3-mini because they were the only two models with no independent repeats at the time. For models with some repeats, the remaining questions were indeed left missing, as you observed. To remove this ambiguity, we have generated the missing independent repeats for the other models, including Llama-3.1-70b. For the 16 models with complete coverage, our proposed final analysis uses all 600 independently repeated questions and no CtE answers. The remaining exceptions are Gemini-2.0-flash (590/600) and Grok-3-mini (0/600). We could not obtain the remaining responses from these original models through the APIs available to us. We currently retain their other results and marked and values as unavailable. If the reviewer prefers a model set with complete repeated- coverage, we are also open to replacing them with Gemini-3.1-flash-lite and Grok-4.3, respectively, and running all experimental conditions for the replacement models.
Q3: We revised Section 3.1 to clarify that answer-set consistency is necessary but insufficient for factual correctness and complements reference-based QA. We also added a pointer to Appendix K, which examines this distinction empirically.
AC1: We agree that Sections 3.2, 3.3, and the Discussion should be as clear as earlier sections. They were revised in the preceding version, but many edits were not highlighted in blue because they rewrote or reorganized existing text rather than added rebuttal material. This may have obscured the changes.
Following the reviewer’s feedback, we made a further editing pass focused on conciseness and readability. We shortened Section 3.3’s introduction and clarified the differences among Base, CtE, and Oracle. We also simplified the Discussion and divided longer passages into shorter paragraphs. We retained the RQ1-RQ5 structure to keep the conclusions connected to the research questions, while making each answer more direct. We hope that these revisions address the reviewer's concern by improving the presentation of the existing material. We appreciate the feedback, which has helped us make the writing more consistent with the rest of the paper.
A3: Previously, tables separated open- and closed-source models, with open-source models ordered by parameter count. This separated GPT-oss-20b from other OpenAI models and the two Llama models from each other. We now group models by family, as suggested.
AC4: We apologize for the oversight. The GPT-oss-20b Oracle results had been uploaded but were missing from Table 3-4, which we have now corrected.
AC4 (Table 12-14): We regenerated these tables from per-question outputs and highlighted changes in blue. The previous Table 12 export could mark an incorrect relation prediction as contradiction-free when answers also violated the expected relation. The corrected metric verifies that the predicted relation holds. Tables 13-14 now use the main experiment’s worst-case null-answer policy, exclude missing classifications, and apply the specified partition and difference metrics. These corrections affect the auxiliary relation-prediction analysis reported in the Appendix L. Table 3 uses fixed ASCB relations and does not depend on these scores, so the paper’s main answer-set consistency finding is unchanged.
We thank the reviewer for their constructive feedback. We appreciate the opportunity to explain how ASCB was constructed from existing QA resources. While several source questions served as starting points, developing a consistent set of related enumeration questions required substantial transformation and manual verification. We conducted an additional question-reuse analysis to quantify this contribution and clarify the benchmark's construction. Finally, we addressed the reported reproducibility issues by correcting the evaluation pipeline, completing the released data, and updating the corresponding results. We have incorporated the relevant changes into the revised manuscript and provide detailed responses below.
W1 and Additional Comments (2): We agree that “a novel handcrafted benchmark” is imprecise and will use “a manually curated benchmark”. As Section 3.2 and Appendix G explain, QAWiki, LC-QuAD, and QALD provided starting points, not ready-made question sets. Three authors substantially revised and verified questions to construct related enumeration quadruples. In truth, few input questions made it unmodified to the final dataset, but rather many served as inspiration. This saved time versus inventing questions completely from scratch.
To quantify reuse, we compared the 1,800 questions in ASCB's three source-based subsets with original English questions and paraphrases. We checked exact and formatting-normalized matches (Unicode, case, whitespace). For nonmatches, we retrieved the five nearest source questions using bge-small-en-v1.5 embeddings and cosine similarity, then assessed whether answer conditions, entities, time, scope, or constraints changed. Q2 inherited Q1's classification because they are equivalent; nonverbatim Q3 and Q4 were classified as substantial changes when they introduced the required subset or complement condition, apart from identified close source rephrasings.
We found 338 verbatim matches (18.8%), 23 formatting-only variants (1.3%), 248 close rephrasings (13.8%), and 1,191 substantial changes or questions without close counterparts (66.2%). Only 35 of 900 Q3/Q4 questions matched source wording verbatim.
Furthermore, the separate synthetic subset comprises 600 final questions that were manually selected and revised from LLM-generated candidates, rather than directly adopting the generated outputs.
Regarding benchmark size, ASCB contains 600 manually verified quadruples (2,400 questions). Its size already supports significant findings: Appendix J reports one-sided paired McNemar improvements over Base for CtE in 79/90 and Oracle in 83/90 model-relation comparisons (p<0.05). These results show that ASCB is large enough to support the paper’s main empirical conclusions while maintaining a consistent, manually verified evaluation structure.
We added the reuse audit to Section 3.2 and Appendix G and will release its code.
Q1–Q2: The repeated-Q1 control E1,∗ measures answer stability when the same enumeration question is asked again, for comparison with equivalent rephrasings. To conserve evaluation budget, we reused Q1 responses from CtE runs and collected an additional isolated Base Q1 response where available. The Base repetitions were incomplete. When both scores exist, line 193 averages repeated-Base and CtE scores equally at model level; otherwise, it uses CtE alone. We clarified this mixed-context procedure in the manuscript.
Q3: Answer-set consistency complements factual accuracy. For example, winners of the FIFA World Cup from Latin America must be a subset of all winning national teams. Correct answers necessarily respect this containment, but incomplete or incorrect answers can also satisfy it. Consistency is therefore necessary, not sufficient, for correctness, and can be evaluated without exhaustive reference sets.
Following reviewer jYSP's question, we analyzed 92 ASCB groups with author-reviewed candidate answer sets. Across 18 models, Q3/Q4 satisfied disjointness in 66.4% of Base cases on average, but both answers exactly matched their candidate sets in only 25.8%. This illustrates why consistency cannot substitute for factual correctness. The approach can also complement document- or search-grounded QA by assessing both evidence-based accuracy and relational consistency. We clarified Section 3.1 and added Appendix K.
Additional Comments (1): We revised Sections 3.2–3.3 and the Discussion to clarify benchmark construction, tasks, and interpretation.
Additional Comments (3): Revised result tables group models by family and provider and order them for easier comparison.
Additional Comments (4): We fixed the reproduction pipeline and added a single-command exporter for Tables 9-11, correcting data paths and metric mappings. We also found that GPT-oss-20b Oracle outputs contained only 300 of the expected 600 complete groups. We uploaded the missing results and updated the manuscript tables.