acceptodds
Under review as a conference paper at ICLR 2027

What is a Clarification? Estimating Model-Dependent Input Ambiguity

Abstract

Ambiguous inputs can admit multiple interpretations, each potentially yielding a different answer. Existing work largely treats this ambiguity as a property of the input, yet for a particular language model, only some alternatives may be known and answerable. We therefore distinguish world ambiguity, capturing all alternatives admitted by an input, from model ambiguity, capturing those accessible to the responding model. To characterize these alternatives, we formalize natural-language clarifications through three desiderata: uniqueness, faithfulness, and answer equivalence. We show that these desiderata induce a one-to-one correspondence between valid answers and equivalence classes of clarifications, implying that any interpretation can be represented either by its answer or by the clarifications that uniquely express it. We leverage this correspondence to introduce CIRCLECLAR, a black-box method that estimates the set of answers available to a model by checking consistency along an answer–clarification–answer cycle. Across QA, task instructions, and Text-to-SQL, CIRCLECLAR achieves strong recovery of these answer sets, yields the strongest overall performance on model-dependent ambiguity detection, and enables effective selection among answering, clarifying, and abstaining. Together, these results demonstrate the practical value of a model-dependent view of ambiguity and its characterization through an answer-clarification correspondence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Ambiguity, Uncertainty Quantification

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xNTBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper distinguishes between the answers permitted as correct by the original question and the answers a model can actually provide (world ambiguity vs model ambiguity), and defines clarification in terms of answer equivalence. It then proposes CircleClar, which first enumerates candidate answers, generates clarifications for each candidate, independently re-answers the question, and retains only those answers that return to the original candidate. CircleClar achieves the strongest or near-strongest overall performance on answer-set recovery, model-dependent ambiguity detection, and the three-way decision among answering, clarifying, and refusing.

Critical strengths
  1. The CircleClar method is relatively straightforward, and its core checks are supported by experimental evidence. Requiring answers to return to the original candidate under validation is more effective than merely requiring them to agree with one another. The component ablations specifically support this distinction.

  2. CircleClar achieves the highest Set F1 in eight of the sixteen model–dataset combinations and outperforms both Sampling-50 and 1-Enumeration in all eight QA settings. The parameter analysis, single-answer tests, and stricter checks on answer content also provide useful context for the results.

  3. The definition and computation of model-realizable answer support are particularly interesting.

Critical weaknesses
  1. The paper has significant readability issues. I spent many hours carefully reading it, but some parts remained confusing. The formalization of clarification and the bijection proof for Proposition 4.1 seemed unnecessary; it would be sufficient to state the properties that a clarification should directly satisfy.

  2. As the authors note in the limitations, CircleClar may produce incorrect answers when the model itself hallucinates or relies on incorrect knowledge.

Questions for authors
  1. Is the ultimate objective to estimate the answers the model can provide to the original question, or the answers it can consistently produce under a specified clarification? When synonymous clarifications, repeated sampling, or adjusted reference thresholds are used, are the reference answer set and the resulting conclusions stable?

  2. D2 requires that the answer produced after clarification belongs to the answer set of the original question. However, this condition alone does not ensure that an irrelevant question that happens to elicit the same answer is a reasonable clarification.

  3. I find your topic very interesting. If the readability issues are properly addressed, I may consider raising my score.

4 comments
AuthorsOpenReview ↗

Rebuttal xNTB

Thank you for your review, and we are especially glad you find our core conceptual idea very interesting. Below, we address your points with results in Figshare and have updated our manuscript (changes in blue):

W1: We aimed to make the paper readable, so we take your concern seriously. We would like to kindly ask you if you could pinpoint the specific sections that were hard to read or unclear. For now, we have updated our main text (see revision) to explain the mathematical part of the desiderata and proposition in more detail.

Regarding the mathematical formalization: We think overall that it is important to not only convey the intuition but also to make the idea formally clear and explicit. As in our example, the ideas can be understood quite well intuitively. A key contribution of our paper is specifically this formalization.

Again, we would really appreciate it if you could pinpoint potential sections that you found hard to read to help us improve on this point further. We are happy to further revise the manuscript accordingly during the rebuttal.

W2: Yes, this is correct, and as you mentioned, we state this already in our limitations. However, we want to emphasize that this is a general failure mode for all methods that rely on sampling diversity as a proxy for correctness. On a more abstract meta-level, if a model “believes” (e.g., measured through sampling diversity) something wrong very confidently (because, e.g., it is wrong in the training data), then in general may be almost impossible to find this out without access to external grounding beyond the model.

Q1: Thank you for raising this important point, and we added this point to the revised manuscript (see revision update). The setting we want to target is to find the answers that the model can correctly find, given that it is supplied with the clarification. The reason for this is that, in practice, a user or another agent will select or provide the desired clarification.

We provide ablations for the reference threshold in Fig. 2 and find the results to be stable for varying the reference set construction. Moreover, synonymous clarifications are not a problem, since they yield the same answer and induce the same downstream decisions; accordingly, we treat them as equivalent (Desideratum D3). Regarding repeated sampling, we repeated our reference construction by sampling 3 independent sets and measured pairwise F1 overlap between our original set and the newly obtained sets, and find high overlap in Tab. 7. supporting the robustness under repeated sampling.

Q2: This is a valid observation, and we have added it to our limitations section (see revised manuscript). However, while conceptually correct, this case does likely not occur in practice, as we generate clarifications based on the question and the answer, so no unrelated clarifications will be generated. To support this claim empirically, we conduct the following experiment: For each clarification produced, we ask an LLM-as-a-judge whether the clarification and the question are related or unrelated. The results are in Tab. 4. As you can see, the generated clarifications consistently relate to the original question.

Q3: We appreciate your interest and would welcome any further input or guidance on which points are not readable.

Thank you for your review. We hope we have clarified some points. Most importantly, to best address any remaining issues with readability, we ask you to pinpoint sections that could benefit from further revision.

Thanks for the rebuttal. The Q1 answer resolves my main conceptual concern, and the added limitations look right to me.

Here are some of my concerns regarding your rebuttal.

  1. On clarification leakage: I went through the code, and the QA prompt does forbid including the target answer itself, which is good (though it should be in the appendix). But the generator sees the target, so it can entail the answer instead of naming it. Example: given the target Sterling Holloway, the clarification "The snake in the 1967 film was voiced by an actor whose initials are S.H. and who is not Scarlett Johansson. Who is it?" contains no answer string, yet the answer is solvable from the wording alone.

  2. The equivalence judge is GPT-5.4-mini (configs/models.yaml), which is itself one of the evaluated models. Self-evaluation makes the absolute numbers hard to trust, that model's own column most of all. Could you rerun the main table with an external judge, or report agreement between two judges?

  3. The acceptance threshold is 0.6 while reference inclusion sits at 0.9 (QA) and 0.8 (AmbigInst, SQL). With R=10 re-answers, a candidate whose true recovery rate is 0.4 still passes about 17% of the time by chance. Why the gap, and how does set F1 move with R and tau?

The new results cited in the rebuttal (the relatedness table, the threshold ablation, the overlap table) are not in the PDF or the code, and the Figshare link gives me a 403.

Readability: my difficulty was that "clarification" is never informally defined before the formalization starts. Section 1 points to references (not self-contained), the first example appears only in Section 4.1. A reader must absorb the full formal apparatus of Section 3 without yet knowing what object is being formalized. Also, the everyday meaning of the word (a statement that explains) does not match the technical one (a rewritten question). One paragraph with the Jungle Book example up front would have saved me a lot of backtracking.

Thank you for answering and engaging with us! We really appreciate this.

Regarding Figshare

We double-checked the link on other computers, and it works just fine for us. However, we now also include all the Figshare content in the revised manuscript. We made a top-level comment about this, detailing what the Figshare Number corresponds to in the manuscript. We apologize for this complication.

  1. As you mentioned, we explicitly constrain the model not to include the answer. In addition to the prompts in Appendix F, we have now explicitly written this there. From our manual inspection throughout the project, we find that the case where the model inserts the answer in “disguise” into the prompt is highly unlikely. Nevertheless, we ran an ablation to address this concern, providing an LLM-as-a-judge with the generated clarification and answer and asking if the answer leaks into the clarification. We find in Tab. 14 (manuscript) that only 0.57% have potential answer leakage.
  2. Yes, we already ran this experiment in response to a similar request from other reviewers using GPT-6 Luna. Results are in Table 3 Figshare / Table 7 Manuscript where we see them to be highly consistent.
  3. In the original manuscript, we already provide a sensitivity analysis for the different components in Appendix C4. For the threshold τ\tau we find it generally robust. However, as CircleClar is heavily precision-oriented (see Fig3 manuscript) we decided to opt for a lower τ\tau to possibly improve recall. For the number of samples, we find it slightly beneficial to increase from 2 to 5, but it remains quite similar from 5 to 10.

Regarding Readability

Thank you for clarifying. We understand your point now, and we find your suggestion to be quite useful. We updated the manuscript and now explicitly provide an example of a clarification in the 3rd paragraph, where we first discuss clarifications. We also updated the first sentence of this paragraph to provide a clearer description. Thank you again for this point !

Thanks for your rebuttal. I have updated my score from 2 to 3; the rebuttal resolved my main concerns. One reservation remains: the formalization reads as largely definitional to me, so the theory provides vocabulary rather than independent guarantees. I think it is not a "key-level" contribution (I acknowledge that it is a contribution indeed.).

Reviewer_uThmOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper first distinguishes world ambiguity (all alternative interpretations of an input) and model ambiguity (those accessible to the model) to set the stage. Then, it argues for three desiderata for natural language clarifications: uniqueness, faithfulness, and answer equivalence, and shows that there is a one-to-one mapping between valid answers and clarifications (equivalence classes). Leveraging this framework, CircleClar is presented, which is a black-box method to identify a set of answers available to a model based on answer-clarification-answer consistency. Experiments on AmbigQA, AmbigQA-HQ (Open-Domain QA), AmbigInst (general instruction), and AMBROSIA (Text-to-SQL) show that CircleClar can better estimate the answer support (set of answers accessible to the model) and better make ambiguity-aware decisions than the baselines.

Critical strengths
  • The distinction between world ambiguity and model ambiguity is convincingly argued. This framing makes the direction of this work compelling.
  • The theoretical grounding of clarification is rigorous for the area, and the proposed methodology (answer-clarification-answer consistency) directly follows from it.
  • Strong empirical results on multiple tasks.
Critical weaknesses
  • Evaluation relies on estimated reference of true underlying model answer support (set of valid answers a given model has access to), which is not directly observable. The evaluation also relies on LLM-based judgements like consistency verification. Both add noise to the evaluation process.
  • Several baselines were designed for ambiguity detection rather than answer set recovery. They thus constitute weaker baselines.
Questions for authors
  • How sensitive are the results to the LLM-based judgements used in the evaluation pipeline?
  • Is there any evidence that CircleClar measures model ambiguity rather than consistency in answer generation (w.r.t. the reliance on answer-clarification-answer consistency)?
1 comment
AuthorsOpenReview ↗

Rebuttal uThm

Thank you for your feedback. We are delighted that you recognized our strong empirical results, our theoretical grounding, and the motivation behind model ambiguity as critical strengths. Below, we address your points. We provide additional experiments in Figshare

W1: We agree that estimating the answer support is not directly observable. To investigate noise in reference set construction, we also evaluated an alternative reference set. To construct the new reference set, we provide the model with the clarification and then evaluate whether the model's beam-search answer is correct and, if so, include it in the reference set. Since beam search is not available for GPT, we used the answer with the highest probability for the GPT family. As such, this approach does not rely on a thresholding mechanism like CircleClar. We present our main text results using the new reference set in Tab. 1 and Figures 1a and 1b. CircleClar also has a clear edge using this reference set. Furthermore, we present in Tab. 2 a direct comparison between the reference set retrieved using the new method and that retrieved by our old method, finding significant overlap, thereby validating the stability of the reference set construction. In addition, we provide ablations of our original reference set construction algorithm across different thresholds τ\tau in Fig. 2. Again, CircleClar’s performance is largely stable across thresholds.

To mitigate judge noise, we now also provide an ablation using a different judge, GPT6 Luna, with results for Qwen 30B in Tab. 3. Results are very similar, suggesting the variation because of the chosen judge model is low.

W2: It is correct that there are no baselines in the literature designed to capture our objective of model-realizable answers. This is because we are also the first work to consider and formalize this setting. We attempted to have the most fair comparison by best adapting the existing baselines for ambiguity detection, e.g. ICE, for the task of answer set recovery. However, if you believe that there is a better and more direct way to implement these for these tasks, or if we are missing a relevant baseline, we would be very grateful if you could point us to them so we can include them as well.

Beyond that, we also explicitly evaluate ambiguity detection, the task these methods were designed for, in Sec. 5.2 (Fig. 4a). Importantly, we use here the original methods (not modified to produce sets) and, as such, compare here to established baselines. CircleClar also performs well in this setting.

Q1: We included experiments with another judge (GPT-6 Luna) to measure the sensitivity in Table 3. → See W1.

Q2: We are not sure if we understood this question correctly. Could you clarify your question further? Our main experiments focus on the recovery of the answer set and, hence, the model ambiguity (see Sec. 5.2 and App. B); we hence see strong evidence that it captures the targeted model ambiguity, which we describe through the lens of recoverable answers.

Thank you again for your review, and we think the additional reference set construction and noise ablation make the paper stronger. We hope we have clarified the other points; we are happy to engage further should anything have remained unclear.

Reviewer_6cUhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem of ambiguity, by distinguishing between world ambiguities (those admitted by an input) and model ambiguities (those a certain model knows and can answer). It formalizes clarifications defining three desiderata and proposes CircleClar, a black-box method that estimates the answer available to a model by checking an answer, clarification. answer cycle. On QA, task instructions, and Text-to-SQL the method is competitive, specifically on small models, and is used to decid when to answer, clarify, or abstain

Critical strengths
  1. The problem has practical relevance. In Fig. 2 the paper shows clearly that the number of accessible interpretations can vary sensibly among models, and this should influence the decision of a model to answer or ask for clarifications
  2. The desiderata for clarifications are clear and useful, not only for the method, but also as criteria to curate datasets (like AmbigQA-HQ)
  3. The method is simple, black box, and applicable to different tasks
  4. The dataset is a useful contribution for the community, with a documented curation process and examples of the correctness
Critical weaknesses

Major

  1. The evaluation looks to be aligned with the method. The reference MT(x)M_T(x) is constructed with the same criterion of CircleClar. The model answers the dataset clarification, and it accept the answer if the most frequent froup matches the candidate beyond a threshold, using the same equivalence judge (mentioned in appendix F). CircleClar then applies almost the same test as the reference, only changing the source of the clarification, while the baselines are scored against a target they do not optimize for. The same happens for the ambiguity-detection and response-policy labels, which might explain why the baselines are close to chance. Can the authors show the same experiment with a reference independent of the recovery criterion?

  2. The definition does not appear to match the operationalization. MT(x)M_T(x) is defined as the support of pT(∣˙x)p_T(\dot | x) (i.e., the answers that the model would produce from the ambiguous input). That said, both the reference and the method measure whether the model can recover an answer when the interpretation is explicit, that is, knowledge the model has but might never produce from xx. These two are different quantities and the paper does not clarify which of the two is relevant for the decision of whether to answer or clarify.

  3. Empirical results are mixed. In the experiments, CircleClar is best in 8 of 16 settings. The largest gains appear on small models and come mostly from the Abstain cases (i.e., recognizing unanswerable cases). On strong models ICE or just enumeration are comparable or better: for example, GPT-5.5 on AmbigQA-HQ gets 77.9 with ICE versus 68.9, and on AmbigQA, ICE is better at ambiguity detection for both the GPT models. Standard deviations are also very large (+/-30-48), and the clearest gains are on AmbigQA-HQ, 100 questions curated by the authors, according to the same desiderata the method is built on.

  4. Limited theoretical contribution. As D3 defines equivalence as the induction of the same answer, the bijection in Prop 4.1 follows directly from the definitions. The world/model difference is also similar to the perceived ambiguity of [1].

[1] Kim, H. J., et al. Aligning Language Models to Explicitly Handle Ambiguity. EMNLP 2024.

Minor

  1. It would be helpful if the authors could report the computational cost too. They have 6 enumerations, one clarification per candidate, and 10 answers per clarification, but no comparison to the baselines at a matched number of calls is reported.
  2. The semantic judge is GPT-5.4-mini, which is also one of the evaluated models, but it is also used to build the reference and in the method's consistency check
  3. Model coverage looks limited, with only four models from two families
  4. Passing the cycle does not guarantee the correctness of the answer, as a consistently reproduced error would pass the check anyways (the it is acknowledged in the paper). This is mitigated in evaluation by intersecting with the dataset answer, but no mitigation is used in deployment
Questions for authors

See weaknesses

4 comments
AuthorsOpenReview ↗

Rebuttal 6cUh

Thank you for feedback! We are happy that you consider the problem has practical relevance, and that our desiderata for clarifications are clear and useful. Below, we address your points. Additional data from new experiments are in Figshare

W1: We appreciate the idea of also testing other methods for retrieving the reference set. We have now also run experiments to construct the reference set using a different mechanism, as follows: We provide the model with the clarification, and then evaluate whether the model's beam-search answer is correct and, if so, include it in the reference set. Since beam search is not available for GPT, we used the answer with the highest probability for the GPT family. As such, this approach does not use a thresholding mechanism like CircleClar. We present our main text results using the new reference set in Tab. 1 and Figures 1a and 1b. CircleClar also has a clear edge using this reference set. Furthermore, we present in Tab. 2 a direct comparison between the reference set retrieved using the new method and that retrieved by our old method, finding significant overlap, thereby validating the stability of the reference set construction. In addition, we provide ablations of our original reference set construction algorithm across different thresholds τ\tau in Fig. 2. Again, CircleClar’s performance is largely stable across thresholds. We thank you again, as we believe that a reference construction independent of our mechanism strengthens the paper, and our evidence shows that the conclusion remains unchanged.

W2: That is an important point you raise, and we revised our manuscript to adequately acknowledge this. The setting we want to target is to find the answers that the model can correctly find, given that it is supplied with the clarification. The reason for this is that, in practice, a user or another agent will select or provide the desired clarification. Mathematically, the two views can be unified by a latent variable model defining p(y∣x)=∫p(y∣c)p(c∣x)dcp(y|x) = \int p(y|c)p(c|x)dc (see underspecification [2]). In practice, as you pointed out, modeling p(y∣x)p(y|x) directly might not yield the answer that would be found when the clarifications are made explicit. In the latent variable model, this would correspond to having p(c∣x)=0p(c|x)=0, so the model does not internally assign probability to this clarification. However, this is not the behavior we want because p(c∣x)p(c|x) should be decided by the user/other agent. Hence, our goal is to find p(y∣c)p(y|c). This is a very important point, and similarly, we want to thank you for pointing it out, as it makes things clearer.

[2] https://proceedings.mlr.press/v267/kirchhof25b.html

W3: An important point is that we specifically designed AmbigQA-HQ because of quality issues in AmbigQA. The key difference is not only that the clarifications do not meet our criteria, but also that AmbigQA misses many clarifications and answers, which we had to supply through extensive manual checks to create AmbigQA-HQ. Thus, performance on AmbigQA is inherently noisy, primarily due to missing answers and clarifications, and any interpretation of performance should prioritize AmbigQA-HQ. We provide some examples of missing clarifications and answers in Tab 5.

CircleClar, while having the most top-1 rankings, does not consistently outperform the other methods in all model-dataset combinations. We want to reinforce that the main contribution of our work is the conceptual formal framework to describe ambiguity and clarifications, and CircleClar is a simple implementation to provide empirical evidence for its validity. Notice that our empirical analysis not only spans model-realizable answer set recovery but also ambiguity detection and a three-way decision problem (answer, clarify, abstain) (Fig. 4a and b), all of which CircleClar performs the best in. The variability of our results is because the F1 score is in [0,1], and it becomes 0 if the model has no knowledge, introducing strong deviations into the metric.

W4: We agree that the bijection follows directly from the desiderata; as such, we also introduced it only as a Proposition instead of a Theorem. Accordingly, we now emphasize in the manuscript that the proposition follows directly from the definitions. As we agree with you that the desiderata provide a useful and novel characterization, we believe that there is value in explicitly stating this Proposition and its implications. While [1] also discusses this subject, to our knowledge, the authors only briefly touch on this idea and do not properly formalize it. In contrast, we provide a clear formalization and demonstrate the difference both mathematically and empirically (e.g., Figure 1).

Thank you again for your review. Discussing these points helped us strengthen the paper, and we hope to have clarified the remaining questions. If there are any remaining points, we are happy to discuss them.

AuthorsOpenReview ↗

Rebuttal 6cUh Continuation - Minor Weaknesses

Minor:

WM1: Thank you for this suggestion. We added an evaluation of the computational costs (number of questions, tokens, and wall time) to Tab 6.

WM2: We are not sure if we understand your point correctly. It is correct that gpt5.4-mini is used as a judge for creating semantic clusters of the sampled answers for all models. However, the actual sampling/reference construction is performed for each model individually, using that respective model. Could you clarify what you mean?

Additionally, we now provide an ablation using a different judge, GPT6 Luna, with results for Qwen 30B in Tab 3. Results are very similar, suggesting the variation across different judge models is low.

WM3: We will provide experiments using Gemma 4 models for the camera-ready deadline.

WM4: We are not sure if we correctly understand your question. Our predicted sets are generated without any access to the evaluation data and are evaluated only at test time. As such, there is no intersection with the dataset answer at prediction time.

Reviewer_6cUhOpenReview ↗

Reply by Reviewer 6cUh

I thank the authors for providing their detailed rebuttal, and for addressing my main concerns. I find that they have correctly closed many of my points, even though I think that my Major 3 is still open. The claim is that AmbigQA is noisy/low quality. The problem is still there though, and it is that their AmbigQA-HQ is built exactly following their method desiderata, giving an intrinsic advantage.

Moreover I think the authors have misunderstood two of my remaining points: Minor 2: the problem with gpt5.4-mini is actually that one single model was used both for building the reference and in the method’s consistency check. This means that the method under evaluation takes part in building the standard it is measured against. The ablation with GPT6 Luna is the right check, but it covers only Qwen 30B. Minor 4: this point was not about data leakage. It is that passing the cycle does not guarantee the correctness of the answer, since an error the model does consistently will also pass the check. In your evaluation this is mitigated by intersecting with the dataset answers, but a deployed system has no such safeguard, so the estimated set may contain confidently wrong answers.

Thank you for taking the time to engage with us. We are glad that we could address many of your points and think they also made the paper clearer.

Note: (We added the new results below to both the manuscript/figshare; however, for the manuscript, we appended them at the very end so as not to change the current table numbers mentioned in other reviews and the top-level comment).

W3

We created an additional dataset, which we call AmbigQA-Extended, that contains the same 96 AmbigQA questions as in AmbigQA-HQ but does not apply any of D1-D3. We make only two kinds of changes: (i) we add missing valid interpretations together with their answers, and (ii) we correct factually wrong answers. The resulting answer sets therefore reflect a general fix to the original dataset (completeness and correctness), as any dataset curation method would do, and not our characterization of clarifications. So we specifically do not perform curated rewriting of clarifications and answers guided by the desiderata D1-D3, as detailed in Appendix B. We provide the results in the updated Manuscript/Figshare in Table 18/Table 10, which shows that even without any desiderata-based curation, CircleClar outperforms all baselines for both Qwen models and remains competitive with the strongest baseline for the GPT models.

We want to point out two other things: (1) AMBROSIA and AmbigInst were not curated by us, and CircleClar is the best or second-best method in all eight model settings on these datasets. (2) Moreover, we believe D1–D3 describe sensible properties that datasets for (model-dependent) ambiguity should fulfill. Violating them changes the construction of MT(x)M_T(x) in practice: E.g., rewrites that change what the question asks (D2) add answers that do not answer the original input, and rewrites that remain ambiguous (D1) make recovery depend on which interpretation the model chooses rather than on what it knows. A dataset that fulfills D1–D3 is therefore not tailored to our method but simply measures the target quantity more faithfully.

WM2:

We have now also added the ablation for the gpt5.4-mini run using GPT-6 Luna as a judge (manuscript/figshare Table 17/Table9). Again we find a very high correlation showing robustness across semantic judges. A full ablation of all models will be done for the camera-ready version; however, for the rebuttal, it is no longer feasible due to computational reasons. However, as this ablation examines judge quality in terms of semantic clustering and, for both Qwen 30 and gpt5.4-mini, we found a high correlation, we are confident this extends to other models, since clustering is independent of the model producing the clusters.

WM4

This is correct, and a deployed system may have confidently wrong answers. However, we want to emphasize that this is a general failure mode for all methods that rely on sampling diversity as a proxy for correctness. On a more abstract meta-level, if a model “believes” (e.g., measured through sampling diversity) something wrong very confidently (because, e.g., it is wrong in the training data), then in general it may be almost impossible to find this out without access to external grounding beyond the model. Nonetheless, we mention and acknowledge this part in our limitations.

Thank you again for engaging with us and for your time. We hope we were able to address your remaining points.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Figshare <-> Manuscript Numbers

One reviewer pointed out that Figshare (https://figshare.com/s/ed11981ce5233597cc8f) does not work for him/her; we have now also included Figshare results in the main manuscript. Here is the conversion of Figshare Table/Figure numbers to Manuscript Table/Figure numbers:

FigshareManuscript
Table 1Table 8
Figure 1a & 1bFigure 10a & 10b
Table 2Table 9
Figure 2Figure 11
Table 3Table 7
Table 4Table 11
Table 5Table 4
Table 6Table 13
Table 7Table 10
Table 8Table 14

We apologize for the inconvenience.

Dear Reviewers,

We thank you again for your effort in reviewing our paper. Your feedback helped us improve our manuscript. We believe that our answer addressed the raised questions. Should any remaining points not have been clarified in our rebuttal, we kindly ask you to point them out so we can provide further clarification before the discussion closes. Should the points be addressed, we would appreciate it if you could reflect this in the final evaluation accordingly.

Thank you again for your effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.