- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper distinguishes between the answers permitted as correct by the original question and the answers a model can actually provide (world ambiguity vs model ambiguity), and defines clarification in terms of answer equivalence. It then proposes CircleClar, which first enumerates candidate answers, generates clarifications for each candidate, independently re-answers the question, and retains only those answers that return to the original candidate. CircleClar achieves the strongest or near-strongest overall performance on answer-set recovery, model-dependent ambiguity detection, and the three-way decision among answering, clarifying, and refusing.
Critical strengths
-
The CircleClar method is relatively straightforward, and its core checks are supported by experimental evidence. Requiring answers to return to the original candidate under validation is more effective than merely requiring them to agree with one another. The component ablations specifically support this distinction.
-
CircleClar achieves the highest Set F1 in eight of the sixteen model–dataset combinations and outperforms both Sampling-50 and 1-Enumeration in all eight QA settings. The parameter analysis, single-answer tests, and stricter checks on answer content also provide useful context for the results.
-
The definition and computation of model-realizable answer support are particularly interesting.
Critical weaknesses
-
The paper has significant readability issues. I spent many hours carefully reading it, but some parts remained confusing. The formalization of clarification and the bijection proof for Proposition 4.1 seemed unnecessary; it would be sufficient to state the properties that a clarification should directly satisfy.
-
As the authors note in the limitations, CircleClar may produce incorrect answers when the model itself hallucinates or relies on incorrect knowledge.
Questions for authors
-
Is the ultimate objective to estimate the answers the model can provide to the original question, or the answers it can consistently produce under a specified clarification? When synonymous clarifications, repeated sampling, or adjusted reference thresholds are used, are the reference answer set and the resulting conclusions stable?
-
D2 requires that the answer produced after clarification belongs to the answer set of the original question. However, this condition alone does not ensure that an irrelevant question that happens to elicit the same answer is a reasonable clarification.
-
I find your topic very interesting. If the readability issues are properly addressed, I may consider raising my score.
4 comments
Thanks for the rebuttal. The Q1 answer resolves my main conceptual concern, and the added limitations look right to me.
Here are some of my concerns regarding your rebuttal.
-
On clarification leakage: I went through the code, and the QA prompt does forbid including the target answer itself, which is good (though it should be in the appendix). But the generator sees the target, so it can entail the answer instead of naming it. Example: given the target Sterling Holloway, the clarification "The snake in the 1967 film was voiced by an actor whose initials are S.H. and who is not Scarlett Johansson. Who is it?" contains no answer string, yet the answer is solvable from the wording alone.
-
The equivalence judge is GPT-5.4-mini (configs/models.yaml), which is itself one of the evaluated models. Self-evaluation makes the absolute numbers hard to trust, that model's own column most of all. Could you rerun the main table with an external judge, or report agreement between two judges?
-
The acceptance threshold is 0.6 while reference inclusion sits at 0.9 (QA) and 0.8 (AmbigInst, SQL). With R=10 re-answers, a candidate whose true recovery rate is 0.4 still passes about 17% of the time by chance. Why the gap, and how does set F1 move with R and tau?
The new results cited in the rebuttal (the relatedness table, the threshold ablation, the overlap table) are not in the PDF or the code, and the Figshare link gives me a 403.
Readability: my difficulty was that "clarification" is never informally defined before the formalization starts. Section 1 points to references (not self-contained), the first example appears only in Section 4.1. A reader must absorb the full formal apparatus of Section 3 without yet knowing what object is being formalized. Also, the everyday meaning of the word (a statement that explains) does not match the technical one (a rewritten question). One paragraph with the Jungle Book example up front would have saved me a lot of backtracking.
Thank you for answering and engaging with us! We really appreciate this.
Regarding Figshare
We double-checked the link on other computers, and it works just fine for us. However, we now also include all the Figshare content in the revised manuscript. We made a top-level comment about this, detailing what the Figshare Number corresponds to in the manuscript. We apologize for this complication.
- As you mentioned, we explicitly constrain the model not to include the answer. In addition to the prompts in Appendix F, we have now explicitly written this there. From our manual inspection throughout the project, we find that the case where the model inserts the answer in “disguise” into the prompt is highly unlikely. Nevertheless, we ran an ablation to address this concern, providing an LLM-as-a-judge with the generated clarification and answer and asking if the answer leaks into the clarification. We find in Tab. 14 (manuscript) that only 0.57% have potential answer leakage.
- Yes, we already ran this experiment in response to a similar request from other reviewers using GPT-6 Luna. Results are in Table 3 Figshare / Table 7 Manuscript where we see them to be highly consistent.
- In the original manuscript, we already provide a sensitivity analysis for the different components in Appendix C4. For the threshold we find it generally robust. However, as CircleClar is heavily precision-oriented (see Fig3 manuscript) we decided to opt for a lower to possibly improve recall. For the number of samples, we find it slightly beneficial to increase from 2 to 5, but it remains quite similar from 5 to 10.
Regarding Readability
Thank you for clarifying. We understand your point now, and we find your suggestion to be quite useful. We updated the manuscript and now explicitly provide an example of a clarification in the 3rd paragraph, where we first discuss clarifications. We also updated the first sentence of this paragraph to provide a clearer description. Thank you again for this point !
Thanks for your rebuttal. I have updated my score from 2 to 3; the rebuttal resolved my main concerns. One reservation remains: the formalization reads as largely definitional to me, so the theory provides vocabulary rather than independent guarantees. I think it is not a "key-level" contribution (I acknowledge that it is a contribution indeed.).
Rebuttal xNTB
Thank you for your review, and we are especially glad you find our core conceptual idea very interesting. Below, we address your points with results in Figshare and have updated our manuscript (changes in blue):
W1: We aimed to make the paper readable, so we take your concern seriously. We would like to kindly ask you if you could pinpoint the specific sections that were hard to read or unclear. For now, we have updated our main text (see revision) to explain the mathematical part of the desiderata and proposition in more detail.
Regarding the mathematical formalization: We think overall that it is important to not only convey the intuition but also to make the idea formally clear and explicit. As in our example, the ideas can be understood quite well intuitively. A key contribution of our paper is specifically this formalization.
Again, we would really appreciate it if you could pinpoint potential sections that you found hard to read to help us improve on this point further. We are happy to further revise the manuscript accordingly during the rebuttal.
W2: Yes, this is correct, and as you mentioned, we state this already in our limitations. However, we want to emphasize that this is a general failure mode for all methods that rely on sampling diversity as a proxy for correctness. On a more abstract meta-level, if a model “believes” (e.g., measured through sampling diversity) something wrong very confidently (because, e.g., it is wrong in the training data), then in general may be almost impossible to find this out without access to external grounding beyond the model.
Q1: Thank you for raising this important point, and we added this point to the revised manuscript (see revision update). The setting we want to target is to find the answers that the model can correctly find, given that it is supplied with the clarification. The reason for this is that, in practice, a user or another agent will select or provide the desired clarification.
We provide ablations for the reference threshold in Fig. 2 and find the results to be stable for varying the reference set construction. Moreover, synonymous clarifications are not a problem, since they yield the same answer and induce the same downstream decisions; accordingly, we treat them as equivalent (Desideratum D3). Regarding repeated sampling, we repeated our reference construction by sampling 3 independent sets and measured pairwise F1 overlap between our original set and the newly obtained sets, and find high overlap in Tab. 7. supporting the robustness under repeated sampling.
Q2: This is a valid observation, and we have added it to our limitations section (see revised manuscript). However, while conceptually correct, this case does likely not occur in practice, as we generate clarifications based on the question and the answer, so no unrelated clarifications will be generated. To support this claim empirically, we conduct the following experiment: For each clarification produced, we ask an LLM-as-a-judge whether the clarification and the question are related or unrelated. The results are in Tab. 4. As you can see, the generated clarifications consistently relate to the original question.
Q3: We appreciate your interest and would welcome any further input or guidance on which points are not readable.
Thank you for your review. We hope we have clarified some points. Most importantly, to best address any remaining issues with readability, we ask you to pinpoint sections that could benefit from further revision.