acceptodds
Under review as a conference paper at ICLR 2027

TACO: Improving Uncertainty Decomposition in Black-Box LLMs via Target-Aware Clarification Optimization

13 views

Abstract

Uncertainty decomposition is essential for trustworthy reasoning with large language models (LLMs). Recent approaches for black-box LLMs uncertainty decomposition leverage input clarifications to construct ensembles over different interpretations of the same input. However, the effectiveness of such methods critically depends on the quality of generated clarification sets. Existing clarification generation approaches typically treat clarification generation as a general text generation task and overlook its target-dependent nature: the same clarification may be interpreted differently by different target LLMs, introducing additional uncertainty and degrading uncertainty decomposition. To address this limitation, we propose TACO, a Target-Aware Clarification Optimization framework for black-box LLM uncertainty decomposition. TACO formulates clarification generation as a reinforcement learning problem and directly optimizes the clarification generator using feedback from the target LLM through two complementary reward mechanisms: an uncertainty decomposition reward that improves the separation between ambiguous and clear inputs, and an entropy alignment reward that encourages ambiguity-aware diversity while preserving consistency for clear inputs. Extensive experiments demonstrate that TACO consistently improves uncertainty decomposition performance across diverse target LLMs and tasks.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Uncertainty Decomposition, Large Language Models, Target-Aware Clarification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_p6BGOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses uncertainty decomposition for black-box LLMs within the input clarification ensembling (ICE) framework. The authors argue that existing clarification generators, whether prompted or supervised fine-tuned, ignore the target LLM, so their clarifications can introduce additional epistemic uncertainty that contaminates the decomposition. They propose TACO, which casts clarification generation as a reinforcement learning problem and optimizes the generator with PPO using feedback from the frozen target model. Two rewards guide training: an uncertainty decomposition reward, defined as the batch-level AUROC of ICE data-uncertainty scores, which improves the separation between ambiguous and clear inputs; and an entropy alignment reward, which encourages ambiguity-aware diversity for ambiguous inputs while preserving semantic consistency for clear ones. TACO is evaluated on ambiguity detection (AmbigInst, AmbigQA) and correctness prediction (GSM8K), and the authors report consistent AUROC and Best-F1 gains over baselines including ICE and Spectral Uncertainty. Ablations indicate that both rewards contribute, with the uncertainty decomposition reward driving most of the improvement. Further analyses suggest that TACO lowers clarification-induced model uncertainty while increasing data uncertainty for ambiguous inputs, and that it adapts clarification diversity to input ambiguity.

Critical strengths
  • S1: Addressing a clear gap: ICE-based uncertainty decomposition methods assume that clarifications remove input ambiguity without adding uncertainty of their own. In fact, previous approaches (Hou et al. , Walha et al.) rely heavily on using a frontier model to generate high quality clarifications, therefore side-stepping the critical clarification quality bottleneck at the cost of expensive generation. This is a real gap that is addressed by the paper, where they use RL fine-tuning to get substantial improvements on relatively small clarification models.

  • S2: Consistent substantial gains: TACO improves over all baselines on both ambiguity detection and correctness prediction. On AmbigInst with LLaMA-3-8B, it reaches 90.49 AUROC versus 80.33 for the strongest baseline (Table 1). On GSM8K with Qwen-2.5-7B, it reaches 94.13 versus 89.04. The gains hold across two target LLMs and two generator backbones (Tables 1–2), and they come with lower model uncertainty under clarifications (Table 3).

Critical weaknesses
  • W1: Weakly justified claim of target alignment:
    • The proposed approach clearly improves the overall quality of the generated clarifications, making them more diverse for ambiguous questions and less diverse for unambiguous ones. However, the paper repeatedly claims that the clarifications are aligned to a specific target model, without sufficient justification.
    • Figure 4 does show that the same ground-truth set of clarifications can yield different performance across target models. But this may simply reflect the models’ differing knowledge of the question’s overall domain, i.e., epistemic uncertainty with respect to the original question. It need not reflect how this particular set of clarifications interacts or aligns with each model. Uncertainty of this kind (high epistemic uncertainty to the original question) cannot be reduced by clarifications better adapted to the target model, nor should it be: reducing it would corrupt the epistemic uncertainty signal, which in such cases should remain high.
    • To establish the target-alignment claim convincingly, the authors should evaluate how LoRA adapters trained for one target model (e.g., Qwen-2.5-7B) perform when applied to another target model (e.g., LLaMA-3-8B), and vice versa. This would help isolate whether the improvement stems solely from generally better clarifications, or from both better clarifications and genuine alignment to the target model. A complementary case study comparing the clarifications produced by TACO when fine-tuned on different target models would also be informative: do different fine-tuning targets yield substantially different clarification sets?
Questions for authors
  • Q1: Value function: How is the value function VψV_\psi implemented? Which backbone does it use, and which loss is used to train the critic? These architectural details are omitted from the manuscript and should be included. Moreover, VψV_\psi is computed per instance, whereas RUDR_{UD} is computed per batch. For this component of the reward, VψV_\psi may increase variance instead of reducing it.
  • Q2: Cross-dataset generalization: The method is evaluated on several datasets, but it is not tested whether a clarification LLM fine-tuned on one dataset generalizes to another. This is an important practical question: generalization across domains would remove the need to fine-tune on each target dataset before inference. Please include cross-dataset experiments to address this point.
  • Q3: Ambiguity labels on GSM8K: The rewards defined in Section 3.3 require ground-truth ambiguity labels for each question. However, such labels are not available for GSM8K. How was this handled? Are all GSM8K questions treated as unambiguous?
  • Q4: Train/test split on AmbigInst: AmbigInst reuses a limited set of instruction templates across many inputs, so the random train/test split used in the experiments leaks templates into the training set. A split by instruction template would give a fairer assessment of the method’s impact on performance.
  • Q5: Baseline consistency: Some baseline numbers appear inconsistent. Comparing Tables 1 and 2 for MC=M_C = GLM-4-9B and MT=M_T = LLaMA-3-8B:
    • ICE on AmbigInst in Table 1 (79.54 / 76.78) is identical to Zero-Shot in Table 2.
    • ICE on AmbigQA in Table 1 (67.10 / 71.73) is identical to SFT in Table 2. Please clarify this clarify this discrepancy and report the correct values.
  • Q6: Typo (minor): There is a typo in Equation (4), the objective formulation.
Reviewer_2o53OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper builds on ICE (Hou et al., 2024), which flags an input as ambiguous when a frozen target LLM answers differently under clarifications from a clarifier model. The authors argue that a target can misread clarifications, which ICE then mistakes for the target's own epistemic uncertainty. TACO therefore trains the clarifier for each target with reinforcement learning on a small labelled set, rewarding clarification sets that separate ambiguous from clear inputs. With two open target models, it is reported to beat ICE and other baselines on ambiguity detection and correctness prediction.

Critical strengths
  1. Training the clarifier is a sensible step for ICE-style methods, and the ambiguity-detection gains are consistent, on both targets and against zero-shot, in-context learning (ICL) and supervised fine-tuning (SFT) clarifiers. The recipe is also light, with LoRA and a small labelled set.

  2. Figure 4 is a useful finding. The same human clarifications give very different area under the ROC curve (AUROC) on different targets, so ICE-style aleatoric scores depend on the target.

Critical weaknesses
  1. The training signal is the evaluation label, so the gains may come from supervised ambiguity classification rather than better uncertainty decomposition. Both rewards use the ambiguous/clear labels, and UDR directly maximizes the test metric, AUROC, on in-distribution training examples. Almost all baselines are training-free.

    The clarifier can also steer the score. By eq. 5, a single clarification gives a data-uncertainty score of s=0s=0 for any target. If answer clusters across clarifications do not overlap, the score is log⁡K\log K, however scattered the answers within each clarification are. A clarifier that classifies the input itself would therefore score well.

  2. Target-awareness, the central claim, is never isolated. TACO is trained and tested on the same target, with no cross-target experiment. Figure 4 cannot tell targets that misread clarifications from targets that simply know different things. With llama-3-8b as target, the figure's target-agnostic human clarifications reach a much higher AUROC on AmbigQA than TACO. They are ahead on AmbigInst too, which suggests the bottleneck is finding the right interpretations, not tailoring them.

Questions for authors
  1. What AUROC does a glm-4-9b LoRA classifier trained on the same labels reach on AmbigInst and AmbigQA?
  2. How is KK distributed for ambiguous and clear inputs under ICL, SFT and TACO, and how often is the score exactly zero? How does TACO compare with SFT at a fixed KK?
  3. Could you report a cross-target matrix, training the clarifier with llama-3-8b and testing with qwen-2.5-7b, and the reverse?
  4. Which clarifier does ICE use on each dataset, and which does Spectral Uncertainty use? How does Spectral Uncertainty do with the SFT clarifier?
Reviewer_cqt8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes TACO, a method for adapting the clarification generator in Input Clarification Ensembling to a particular frozen target LLM. It trains the generator with PPO and LoRA using two rewards: a batch-level AUROC reward for distinguishing ambiguous from clear inputs using estimated data uncertainty, and an instance-level reward encouraging diverse clarifications for ambiguous inputs and consistent clarifications for clear inputs. Experiments evaluate ambiguity detection on AmbigInst and AmbigQA and correctness prediction on GSM8K. The reported results improve over ICE and Spectral Uncertainty with two target LLMs, while ablations with two generator backbones suggest that both rewards contribute.

Critical strengths
  1. Adapting clarification generation to the target model is a useful direction. ICE depends on how a model responds to clarified inputs, so optimizing the generator with feedback from that model addresses a relevant limitation of a fixed generator. The framework operates through sampled target-model responses and does not require updating the target model, making the idea applicable to a useful access setting.

  2. The reported improvements are substantial in several settings, and the ablations help explain them. For example, Table 1 reports AmbigInst AUROC of 90.49 versus 80.33 for the strongest listed baseline with LLaMA-3-8B, and AmbigQA AUROC of 75.01 versus 69.08. Improvements also appear with Qwen-2.5-7B. Table 2 compares against prompting and SFT and removes each reward separately, providing useful evidence that the gains are associated with the proposed optimization rather than just the choice of generator backbone.

Critical weaknesses
  1. The contribution over ICE is incremental, and the benefit of target matching is not isolated. The main additions are the two rewards and PPO training of the clarification generator. Table 2 shows benefits from this training, but Figure 4 does not distinguish target adaptation from task-specific fine-tuning.

  2. The experiments do not yet establish more faithful uncertainty decomposition. Better ambiguity AUROC does not rule out semantic drift or disagreement between incorrect answers. Clarification faithfulness is not directly validated, and the clarification-entropy estimator, normalization, and GSM8K training reward remain insufficiently specified.

  3. Citation and statement formatting needs revision. Several sentence-end citations use narrative author-year formatting where parenthetical citations are appropriate. The ethics, reproducibility, and AI-use statements use numbered main sections instead of the template's unnumbered headings. Unresolved placeholders such as “Figure x” and “[]” also remain.

Questions for authors
  1. Can you provide a controlled analysis that distinguishes ambiguity between valid interpretations from disagreement caused by incorrect target-model answers, and checks clarification faithfulness against the available human annotations? In particular, how should the higher data uncertainty in Figure 6 be interpreted given the displayed factual errors?

  2. What are the exact semantic grouping procedure and entropy estimator in Eq. (8), and the normalization functions in Eqs. (7) and (9)? What training reward and labels are used for GSM8K, where the described ambiguous/clear distinction is not supplied by the benchmark?

  3. Would a cross-target evaluation with the two existing models show an independent benefit from matching the generator to its target? In addition, a brief discussion of Suri et al. (2026) and Zhao et al. (2026) would help distinguish their interactive clarification training from the proposed ICE optimization.

  4. Please identify the exact evaluation subsets, including what is meant by the GSM8K “validation split,” and explain which prediction is judged correct on GSM8K. Are baseline response budgets matched, do the five seeds change data subsets, training, or inference sampling, and do the error bars denote standard deviations or standard errors?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.