acceptodds
Under review as a conference paper at ICLR 2027

RaTI: Reliability-aware Truth Inference for Multi-Model Question Answering

Abstract

Large language models (LLMs) have demonstrated strong reasoning capabilities on question answering (QA), while factual errors remain common due to limitations in knowledge coverage. Combining responses from multiple LLMs can improve answer reliability through their complementary knowledge. A central challenge in multi-model QA is determining how much each candidate should influence the final answer because factual reliability varies across candidates. Existing methods typically use an additional LLM for aggregation or treat agreement among generated answers as confidence. However, response agreement does not necessarily indicate factual reliability. Several models may generate the same incorrect answer, giving an unreliable candidate strong consensus and suppressing a less frequent correct answer. To address this limitation, we propose RaTI (**R**eliability-**a**ware **T**ruth **I**nference), a novel reliability-aware truth inference framework for robust multi-model QA aggregation. Rather than relying solely on response consistency, RaTI models the factual reliability of each candidate through the internal representations of LLMs. It further converts the estimated hallucination risks into credibility weights to calibrate semantic Minimum Bayes Risk inference. In this manner, RaTI enables high-reliability candidates to dominate the final aggregation while suppressing misleading false consensus, thereby reducing the reliance on response agreement signals in existing methods. Experiments on six QA benchmarks show that RaTI consistently improves over individual models and representative collaboration baselines, while further analyses confirm the effectiveness of reliability estimation and its robustness under increasing candidate error rates.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Truth Inference, Multi-Model Collaboration Question Answering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mBdXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RaTI (Reliability-aware Truth Inference), a multi-model question answering framework that combines candidate-level hallucination estimation with reliability-weighted Minimum Bayes Risk (MBR) decoding. The central motivation is that agreement among multiple language models does not necessarily imply factual correctness, especially when models produce correlated errors.

RaTI first obtains one candidate response from each of five heterogeneous lightweight language models. It then uses a pretrained hidden-state-based hallucination detector to estimate the factual risk of each response. These risk scores are converted into normalized inverse-risk credibility weights, which are incorporated into an MBR objective based on cosine distances between sentence embeddings. The final prediction is selected from the original candidate set by minimizing the weighted semantic disagreement.

The authors evaluate RaTI on six question answering benchmarks and compare it against individual models, collaborative inference baselines, and alternative truth-inference strategies. The paper also includes analyses of reliability estimation, model complementarity, robustness, computational overhead, and several additional experiments in the appendix. The reported results suggest that instance-level reliability information can improve answer selection in heterogeneous multi-model settings.

Critical strengths
  1. Well-motivated problem formulation. The paper addresses an important limitation of consensus-based multi-model inference: multiple models may agree on an incorrect answer. Distinguishing candidate reliability from inter-model agreement is a meaningful perspective for improving collaborative question answering.

  2. Simple and interpretable framework. The proposed method combines hallucination-risk estimation with weighted semantic consensus in a straightforward manner. The individual components are relatively easy to understand, and the final decision does not require an additional autoregressive aggregation model.

  3. Broad empirical evaluation. The experiments cover six QA benchmarks and compare the method with several lightweight models and collaborative inference baselines. The reported improvements are generally consistent across F1, Exact Match, and BLEU.

  4. Useful component and reliability analyses. The comparison between perplexity-based and hidden-state-based scoring provides evidence that the choice of reliability estimator matters. The reported ROC-AUC improvement from 0.667 to 0.728 supports the usefulness of the hidden-state signal.

  5. Additional experimental evidence. The appendix provides comparisons with uniform and fixed model-level weighting, sensitivity analyses for sentence encoders and weighting functions, ensemble-size and heterogeneity studies, paired bootstrap confidence intervals, and candidate-oracle analysis. These experiments help characterize the behavior and limitations of the proposed framework.

  6. Practical potential. The method avoids iterative inter-model communication and additional autoregressive answer synthesis. This makes it potentially useful in settings where multiple small language models are already available and an inexpensive answer-selection mechanism is desired.

Critical weaknesses
  1. Limited methodological novelty.

The main algorithm is a relatively direct combination of an existing pretrained hallucination detector, inverse-risk weighting, and embedding-based MBR decoding. The hallucination detector is not newly developed or trained by the authors, and the semantic disagreement function is the standard cosine distance between sentence embeddings. Although the integration is practically reasonable, the paper does not clearly establish a substantial algorithmic contribution beyond combining these established components. The relationship to prior confidence-weighted consensus and risk-aware reranking methods could be discussed more carefully.

  1. Insufficient validation of the central reliability assumption.

The method assumes that hallucination scores produced by a pretrained detector provide meaningful relative reliability estimates across different candidate models and answer types. However, the detector is trained on external context-grounded hallucination datasets, while the evaluated tasks include extractive QA, multiple-choice reasoning, relation extraction, and numerical reasoning. The paper reports an overall ROC-AUC of 0.728, but this moderate discrimination performance does not fully establish that the scores are consistently reliable across datasets, model families, or different types of errors.

Moreover, Equation (6) uses inverse-risk weighting, which may greatly amplify differences between small hallucination scores. Since the authors explicitly acknowledge that these scores are not calibrated probabilities, additional analysis of score distributions, failure cases, and the stability of resulting weights would strengthen the reliability argument. The existing weight-mapping sensitivity experiments are useful but do not completely address this issue.

  1. Limited evidence isolating the contribution of reliability-weighted MBR from simpler selection rules.

Appendix D.1 provides a useful comparison with uniform, global model-level, and dataset-level MBR weighting. Nevertheless, the paper does not fully establish whether the combination of reliability scoring and semantic MBR is substantially better than simply selecting the candidate with the lowest predicted hallucination risk. Such a detector-only argmin baseline is particularly important because the proposed method uses a strong pretrained detector and operates on only five candidate answers.

Similarly, comparing against a semantic-consensus-only method with the same candidate set, and examining the interaction between detector quality and MBR selection, would provide clearer evidence about the necessity of the proposed joint decision rule.

  1. Incomplete end-to-end computational efficiency evaluation.

The efficiency analysis primarily reports the number of autoregressively generated tokens. However, RaTI additionally evaluates a Qwen2.5-3B-based hallucination detector on every candidate and computes sentence embeddings for semantic comparison. These operations introduce nontrivial computation and memory requirements, even though they do not generate new output tokens.

Consequently, equal generated-token consumption does not imply equal inference latency, FLOPs, energy consumption, or memory usage. End-to-end measurements, including detector execution, embedding computation, and aggregation, are necessary to substantiate the practical efficiency advantages, particularly when comparing RaTI against a larger single model or conventional voting.

  1. Potential limitations of semantic distance for factual answer selection.

The proposed MBR objective uses cosine distances between sentence embeddings as a proxy for disagreement between candidate answers. However, semantic similarity does not necessarily reflect factual agreement. For example, two responses containing different numerical values, dates, or named entities may be close in embedding space despite expressing contradictory facts.

This issue is particularly relevant for DROP, relation extraction, and mathematical reasoning. The current evaluation does not sufficiently isolate cases where semantic similarity conflicts with factual correctness. More targeted experiments on numerical contradictions, entity substitutions, and semantically similar but factually incompatible answers would help validate the choice of loss function.

  1. Remaining concerns about evaluation scope and reproducibility.

The main evaluation uses context-grounded QA benchmarks, while the broader motivation concerns factual reliability in multi-model question answering. It remains unclear how well the method generalizes to closed-book factual QA, settings without supporting passages, or questions requiring external verification. The additional GSM8K experiment is useful, but its improvement over the strongest collaborative baseline is relatively small.

Furthermore, several benchmarks are evaluated on subsets rather than their full original datasets. The paper reports subset sizes but does not sufficiently explain their selection procedure and the sensitivity of results to the chosen evaluation samples. More detailed reporting of dataset selection, preprocessing, answer normalization, and inference configuration would improve reproducibility.

Questions for authors

How does RaTI compare against a detector-only selection baseline that simply chooses the candidate with the lowest predicted hallucination score? Can the authors report this comparison on all six benchmarks using exactly the same candidate pool?

Can the authors provide per-dataset and per-model-family evaluations of hallucination detection quality, including ROC-AUC and score calibration? In particular, how reliable is the detector for numerical reasoning and multiple-choice QA?

8 comments
AuthorsOpenReview ↗

Response to Weakness 1

We thank the reviewer for this insightful comment. We clarify the methodological novelty and algorithmic contributions of RaTI as follows.

  • Methodological Novelty. The novelty of RaTI lies in shifting multi-model aggregation from consensus-based selection to reliability-aware consensus inference. Rather than relying solely on agreement among candidate responses, RaTI jointly considers their instance-level reliability and semantic relationships. This is particularly important because agreement among models does not necessarily guarantee factual correctness.

  • Algorithmic Contribution. The key algorithmic contribution is to integrate candidate-level reliability into the semantic MBR objective. Specifically, hallucination-derived credibility weights are used to measure the expected semantic disagreement:

    \sum_{j=1}^{n}W_j L(R_i,R_j),$$ where ($W_j$) represents the credibility of candidate ($R_j$), and ($L(R_i,R_j)$) measures semantic disagreement. Unlike conventional MBR, which treats all candidates equally, RaTI assigns greater influence to more reliable candidates when evaluating the consensus risk.
  • Relationship to Existing Methods. RaTI differs from existing confidence-weighted consensus methods in that it estimates the reliability of each candidate response for each individual question, rather than relying primarily on model-level confidence or generation probabilities. It also differs from risk-aware reranking methods, which typically use risk scores to filter, rank, or independently select candidates. Instead, RaTI incorporates reliability weights directly into the MBR objective, allowing reliability and semantic consensus to jointly determine the final answer.

AuthorsOpenReview ↗

Response to Weakness 2 and Question 1

We thank the reviewer for these valuable comments. We provide additional analyses as follows.

  • Reliability Evaluation. The HS estimator achieves an overall ROC-AUC of 0.728, indicating moderate but meaningful discrimination capability. Although EM-based correctness labels may introduce noise by treating semantically equivalent answers as incorrect, RaTI consistently outperforms HS-lower, which directly selects the lowest-risk candidate, demonstrating the complementary benefit of semantic consensus.

    MethodDROPRERACESQuADNewsQATYDIQA
    HS-lower48.075.840.770.140.052.6
    RaTI54.579.743.973.841.253.7
  • Robustness of Reliability-to-Credibility Mapping. We adopt normalized inverse mapping to assign higher weights to lower-risk candidates, with normalization ensuring that weights sum to one and smoothing preventing numerical instability. Although inverse mapping may amplify small risk differences, sensitivity analysis (Appendix D.3, Figure 9) shows similar F1 performance across Uniform, Linear, Exponential, and Inverse mappings (69.6, 70.0, 69.9, and 70.5, respectively). Inverse mapping achieves the best overall EM (57.8), F1 (70.5), and BLEU (65.5), supporting its effectiveness.

AuthorsOpenReview ↗

Response to Weakness 3

We thank the reviewer for this comment. To clarify the contributions of the two main components, Figure 3 reports an ablation study comparing hallucination-based reliability estimation (HS), semantic MBR, and their combination (HS-MBR).

The results show that HS-MBR achieves the best performance across all three metrics, with average F1, EM, and BLEU scores of 70.5, 57.8, and 65.5, respectively. In comparison, HS achieves 67.9, 54.5, and 63.1, while MBR achieves 69.6, 56.5, and 64.2. These results demonstrate that reliability estimation and semantic consensus provide complementary benefits: HS reduces the influence of unreliable candidates, while MBR further exploits semantic agreement. Their combination consistently outperforms either component alone, validating the effectiveness of the proposed framework.

Reviewer_mMqKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper explores how to combine LLM responses to improve answer reliability on question-answering tasks. Specifically, to tackle the gap between response agreement and response quality, the authors propose RaTI (reliability-aware truth inference), a framework that combines the model answers in a weighted MBR manner, with a hallucination detector and semantic differential loss with SBERT. Experiments on six QA datasets (SQuAD, NewsQA, RACE, RE, DROP, and TYDIQA) and five 2-4B base models (Llama, Phi3, Qwen2.5, Gemma2, Ministral3) show that the proposed methods achieve higher performance compared to single-model or multi-model collaboration baselines. Further analysis profiles the model differences and the stages of the errors with the proposed methods.

Critical strengths
  1. This paper is well-written and easy to follow. The visualizations make sense.

  2. The link between reliability and performance is a good angle for improving multi-model collaboration.

Critical weaknesses
  1. The tasks selected are outdated (2017-2020), which limits the insights into the work for the community. Newer QA or newer tasks requiring more knowledge or reasoning can be a better choice, e.g., MATH, GAIA, and GPQA. Beyond short-answer or multiple-choice questions, experiments on long-form QA can also lead to more interesting findings, e.g., research QA.

  2. The experiments are done on weak baselines. Small and old models are chosen without clear motivation (Llama 3.2 and Qwen 2.5 are too old). To study the mixture, different architectures, model sizes, and model families will help readers understand and analyze the behavior. Besides, baselines with equal compute should also be tested, e.g., sampling multiple times and self-refine with hallucination detection

  3. Many components in the proposed method are not well-motivated or ablated. The hallucination detector (HBM-2) is chosen but not well-ablated. For tasks requiring knowledge, adding external search can be very important here. The semantic differential loss is computed with SBERT, whose performance should be validated.

  4. The proposed method is essentially a weighted MBR, which has limited empirical applicability and novelty

Questions for authors
  1. Table 2 is put together with the conclusion; the layout should be reconsidered
5 comments
AuthorsOpenReview ↗

Response to Weakness 1

We thank the reviewer for this valuable suggestion. We would like to clarify that our benchmark selection is intended to cover diverse QA characteristics rather than simply follow dataset publication dates. The six benchmarks used in our main experiments cover factual extraction, relation identification, multi-sentence comprehension, and discrete reasoning, providing complementary settings for evaluating candidate reliability and answer aggregation.

To further examine whether RaTI generalizes to more challenging tasks, we conduct additional experiments on GPQA, a challenging graduate-level question-answering benchmark involving specialized scientific knowledge and reasoning. For this experiment, we employ three stronger candidate models from different model families: Gemma4:26B, GPT-OSS:20B, and Qwen3.8:27B. We compare RaTI with three representative multi-model collaboration methods under the same candidate model setting. The results are presented below.

MethodAccuracy (%)
MoA58.08
Debate59.10
SLM-MUX58.59
RaTI (Ours)61.62

RaTI achieves the highest accuracy of 61.62%, outperforming the strongest baseline, Debate, by 2.52 percentage points. This result provides additional evidence that reliability-aware aggregation remains effective in challenging knowledge-intensive QA scenarios.Furthermore, our existing experiments on GSM8K provide complementary evidence in mathematical reasoning, where RaTI achieves 71.27% accuracy, outperforming LLM-Debate (70.49%), MoA (68.69%), and SLM-MUX (70.60%).

Taken together, the additional GPQA and GSM8K results demonstrate the applicability of RaTI across different task types, including specialized knowledge-intensive QA and multi-step mathematical reasoning. We will incorporate these additional results and corresponding discussions into the revised manuscript.

AuthorsOpenReview ↗

Response to Weakness 2

We thank the reviewer for this valuable suggestion. To provide a stronger empirical comparison, we introduce an additional Hallucination-aware Self-Refinement (HSR) baseline, which generates multiple candidate responses and uses hallucination detection signals to guide the refinement of candidate answers. This baseline allows us to examine whether explicitly incorporating hallucination information into the refinement process can achieve comparable improvements to RaTI.

We evaluate HSR and RaTI on the same six benchmarks, with results reported below.

DatasetMethodEMF1BLEU
DROPHSR51.1061.1458.08
DROPRaTI54.5063.6060.30
NewsQAHSR38.5857.4350.74
NewsQARaTI41.2059.0051.80
RACEHSR36.9252.7947.27
RACERaTI43.9058.8053.30
REHSR76.6687.0483.73
RERaTI79.7088.2085.20
SQuADHSR69.7882.1177.88
SQuADRaTI73.8085.2081.00
TYDIQAHSR52.1766.3360.39
TYDIQARaTI53.7067.9061.60

These results highlight the advantage of RaTI's aggregation mechanism: rather than using hallucination detection only to guide iterative answer refinement, RaTI directly incorporates instance-level reliability into semantic consensus, allowing candidate reliability and inter-candidate agreement to jointly determine the final prediction. This provides further empirical support for the effectiveness of reliability-aware truth inference.The additional experiment and corresponding analysis have been incorporated into the latest manuscript.

AuthorsOpenReview ↗

Response to Weakness 3

We thank the reviewer for the constructive suggestions.

  • Regarding external retrieval, RaTI focuses on reliability-aware aggregation of candidate answers rather than knowledge acquisition or retrieval-augmented generation. Our objective is to investigate how to identify and aggregate reliable responses given a fixed context and candidate set. Introducing external retrieval would change the information available to candidate models and introduce additional variables, making it difficult to isolate the effectiveness of the aggregation mechanism. We therefore consider retrieval augmentation complementary to RaTI and leave its integration for future work.

  • Regarding the hallucination detector, we adopt the pretrained HDM-2 because it provides context-based factuality estimation using internal representations and can be directly applied without task-specific fine-tuning. Our component analysis already compares hidden-state-based hallucination scoring with perplexity-based scoring, together with four truth inference strategies (MV, CRH, DS, and MBR). The results show that hidden-state-based scoring combined with weighted MBR achieves the strongest overall performance, supporting our design choice.

  • Regarding SBERT, we conduct an additional comparison by replacing SBERT-based semantic similarity with LLM-based semantic evaluation while keeping other components unchanged. SBERT consistently outperforms the LLM-based alternative across all six benchmarks and three metrics. The average improvements are 2.52 EM, 1.86 F1, and 2.15 BLEU points. These results demonstrate that SBERT provides a more effective semantic representation for candidate comparison in our setting, while avoiding additional LLM inference overhead.

Reviewer_7uxHOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RaTI, a reliability-aware framework for aggregating answers from multiple LLMs. Instead of relying solely on agreement among candidate responses, RaTI estimates candidate-level hallucination risk using a pretrained hidden-state-based detector and converts these estimates into credibility weights for semantic Minimum Bayes Risk (MBR) selection. Experiments across six QA benchmarks show improvements over individual lightweight models and multi-model aggregation baselines, supported by component ablations, reliability analysis, and robustness experiments.

Critical strengths
  1. Practical approach to multi-model aggregation. Incorporating candidate-level factuality estimates into semantic consensus is a useful approach to handling cases where multiple models agree on an incorrect answer. The framework is relatively simple and does not require additional autoregressive generation for aggregation.

  2. Comprehensive experimental evaluation. The paper evaluates RaTI across six QA benchmarks and several collaborative baselines, with component ablations, reliability analysis, robustness experiments, and statistical significance tests. The results consistently demonstrate improvements in answer quality, providing reasonable empirical support for the approach.

Critical weaknesses
  1. Limited methodological novelty. The proposed framework mainly combines an existing hidden-state-based hallucination detector with reliability-weighted MBR inference. While this integration is useful, both factuality estimation from internal representations and confidence-based aggregation have been extensively studied. The paper does not sufficiently establish how its approach differs fundamentally from prior reliability/factuality-aware aggregation methods. Moreover, the hidden-state estimator is another proxy for factual reliability, rather than a direct measure of correctness, which makes some of the paper's distinctions between reliability and consistency overstated.

  2. Limitations in reliability evaluation and efficiency analysis. The authors evaluate hallucination detection primarily using AUROC and ranking metrics, but do not report calibration metrics such as ECE or Brier score. Given that the estimated risks are directly transformed into aggregation weights, understanding their calibration would strengthen the evaluation.

  3. Limited generalization to recent models and challenging tasks. The experiments rely on relatively outdated LLMs, without evaluating more recent model generations. The evaluation also focuses on established QA benchmarks, with only an additional GSM8K experiment in the appendix. Evaluating more recent models and more challenging reasoning tasks would strengthen the empirical evidence.

  4. Additional methodological concerns: The inverse mapping from hallucination risk to credibility weights (Eq. 6) is not sufficiently justified and may amplify erroneous low-risk predictions. Also, the reliability evaluation in Section 4.3 uses exact match as a correctness label, which may misclassify factually correct paraphrases.

Questions for authors
  1. Reliability estimation: Could the authors report ECE or Brier scores for the hallucination detector? Since these scores determine aggregation weights, it would be useful to understand how well they reflect actual correctness.

  2. Semantic similarity: Why was SBERT-based similarity chosen over more recent LLM-based semantic evaluation approaches? How sensitive are the results to this choice?

  3. Generalization: How well does RaTI perform with stronger candidate models or on more challenging reasoning benchmarks? Does the benefit of reliability weighting persist as the underlying models improve?

  4. Performance on DROP (Section 4.2): Why does HS-based DS outperform the proposed HS-MBR approach on DROP? Could the authors provide some analysis of when MBR is less effective than alternative aggregation strategies?

  5. Line 043: Citation is missing for the claim that multi-model aggregation improves prediction accuracy.

  6. Lines 047–048: The discussion of LLM aggregators seems oversimplified. Their effectiveness may depend on more than identifying erroneous candidates.

  7. Line 066: The claim that existing methods treat candidates uniformly is misleading, as consistency-based methods already account for differences in candidate agreement.

  8. Lines 425–429: The error decomposition in Figure 7 needs more explanation. How are candidate, scoring, and consensus failures distinguished? Please elaborate.

7 comments
AuthorsOpenReview ↗

Response to Weakness 1

We acknowledge the reviewer's concern regarding the methodological novelty of RaTI. Although RaTI builds upon existing hidden-state hallucination detection and MBR techniques, its contribution lies in integrating instance-level candidate reliability estimation into semantic consensus decision-making, introducing a reliability-guided aggregation mechanism.

Conceptual Contributions

  • From answer consistency to candidate-level reliability: Existing aggregation methods primarily rely on answer consistency, model interactions, or confidence signals. However, agreement does not necessarily imply factual correctness, as multiple models may produce the same incorrect answer. RaTI additionally evaluates the reliability of individual candidate answers, allowing aggregation to consider both consensus and trustworthiness.

  • From model-level to instance-level assessment: Unlike methods that assign fixed weights based on overall model performance, RaTI estimates hallucination risk for each candidate response to a specific question. This enables dynamic reliability assessment that accounts for variations in the same model's performance across different questions.

Technical Contributions

The key distinction between RaTI and existing reliability-aware aggregation methods is that candidate reliability is directly incorporated into semantic MBR decision-making, rather than being used solely for candidate filtering, ranking, or independent selection. This leads to two technical contributions:

  • Reliability-weighted Semantic MBR: Traditional MBR assigns equal weights to all candidates when computing semantic risk. In contrast, RaTI incorporates instance-level reliability into the risk objective: Risk⁡(Ri)=∑j=1nWjL(Ri,Rj).\operatorname{Risk}(R_i)=\sum_{j=1}^{n}W_jL(R_i,R_j). Here, (WjW_j) represents the reliability weight of candidate (RjR_j), and (L(Ri,Rj)L(R_i,R_j)) measures their semantic difference. Consequently, disagreement with highly reliable candidates incurs a greater penalty, reducing the influence of unreliable majority opinions and enabling reliability-guided consensus selection.

  • Theoretical Characterization of Reliability Estimation: Beyond the aggregation mechanism, Appendix C establishes an excess-risk bound for the proposed decision rule: Rp∗(R^)−Rp∗(R∗)≤2BTV(p∗,W),R_{p^{\ast}}(\hat{R}) - R_{p^{\ast}}(R^{\ast}) \leq 2B\text{TV}(p^{\ast},W), where (p∗p^{*}) denotes the oracle reliability distribution and (WW) represents the estimated reliability distribution. This result theoretically characterizes how estimation errors affect the final aggregation risk, showing that better reliability estimation leads to a tighter upper bound on the excess oracle risk.

Relationship Between Reliability and Consistency

We agree that hidden-state estimation remains a proxy signal for reliability rather than direct factual verification. Inspired by findings such as INSIDE [1], we leverage information in internal representations to provide an auxiliary assessment of candidate reliability.

Specifically, Consistency measures semantic agreement among candidate answers, whereas Estimated Reliability uses hidden-state signals to estimate the relative hallucination risk of individual candidates. These signals have different sources and are complementary, but are not entirely independent. We will clarify this distinction in the revised manuscript and avoid presenting reliability estimates as calibrated probabilities of factual correctness.

Clarifying the Methodological Novelty

We emphasize that methodological contributions do not necessarily require reinventing every underlying component. Similar to RAG, whose contribution lies in integrating retrieval and generation into a unified framework, RaTI combines instance-level reliability estimation with semantic MBR to establish a reliability-guided consensus mechanism. Its novelty lies in how candidate-level reliability directly influences semantic risk computation and the final aggregation decision, beyond agreement-based aggregation alone.

[1] Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu and Jieping Ye. INSIDE: LLMs’ Internal States Retain the Power of Hallucination Detection. In ICLR, pp. 3056–3076, 2024a.

AuthorsOpenReview ↗

Response to Weakness 2 and Question 1

We thank the reviewer for pointing out the lack of calibration analysis for the hallucination detector. To address this concern, we additionally evaluate the calibration quality of the hallucination detector using Expected Calibration Error (ECE) and Brier Score across all six QA benchmarks.

The results are summarized below:

DatasetECE ↓Brier Score ↓
DROP0.1150.225
NewsQA0.0950.167
RACE0.0560.232
RE0.0630.101
SQuAD0.0240.076
TYDIQA0.1210.182

The results show that the detector achieves relatively low calibration errors across all six benchmarks. In particular, SQuAD and RE exhibit low ECE values of 0.024 and 0.063, respectively, accompanied by Brier Scores of 0.076 and 0.101. Although calibration quality varies across datasets, the consistently bounded errors provide additional evidence that the detector captures meaningful reliability-related signals across different QA tasks.

We emphasize that these scores should not be interpreted as perfectly calibrated probabilities of factual correctness. Rather, they provide complementary evidence about the quality of the detector's reliability estimates. In RaTI, these estimates are used to derive relative reliability weights, which are further integrated into the semantic MBR objective instead of being treated as direct correctness probabilities.

We have incorporated these additional calibration results into the revised manuscript to provide a more comprehensive evaluation of the hallucination detector beyond discrimination and ranking performance.

AuthorsOpenReview ↗

Response to Weakness 3 and Question 3

We thank the reviewer for raising concerns about the generalization of RaTI to more recent models and challenging reasoning tasks.

We conduct an additional experiment on GPQA, a challenging reasoning benchmark, using a more recent and heterogeneous candidate model pool consisting of Gemma4:26B, GPT-OSS:20B, and Qwen3.8:27B. This combination introduces models from different families and with substantially larger parameter scales than those used in our original experiments, allowing us to further examine whether reliability-aware aggregation remains beneficial with stronger candidate models.

We compare RaTI against three representative multi-model aggregation baselines, including Mixture-of-Agents (MoA), Debate, and SLM-MUX. The results are reported below:

MethodGPQA Accuracy (%)
MoA58.08
Debate59.10
SLM-MUX58.59
RaTI (Ours)61.62

RaTI achieves an accuracy of 61.62%, outperforming MoA, Debate, and SLM-MUX by 3.54, 2.52, and 3.03 percentage points, respectively. These results provide additional empirical evidence that reliability-aware aggregation remains effective when using more recent and larger candidate models, rather than being limited to the lightweight models considered in our original experiments.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers, ACs, SACs, and PCs

We sincerely thank all reviewers for their thorough and thoughtful evaluations. We appreciate their constructive suggestions, which have helped us improve the clarity, rigor, and completeness of our work. We have carefully revised the manuscript and conducted additional experiments to address the concerns raised. All modifications have been highlighted in BLUE. Below, we summarize our major revisions and responses to the common concerns.

  • Methodological Novelty and Contributions (Reviewer 7uxH, mMqK, mBdX): We have clarified RaTI's conceptual and technical contributions, emphasizing its instance-level reliability-aware semantic MBR formulation, and added a theoretical analysis in Appendix C to explain the role of reliability estimation in aggregation.
  • Reliability Estimation and Calibration (Reviewer 7uxH, mBdX): We have supplemented the reliability evaluation with ECE and Brier scores across six benchmarks in Appendix D.10, and provided further analysis of detector performance across different datasets and candidate models.
  • Generalization to Challenging Reasoning Tasks (Reviewer 7uxH, mMqK, mBdX): We have extended the evaluation to mathematical reasoning on GSM8K (Table 9, Appendix D.7) and the more challenging GPQA benchmark, where RaTI achieves 61.62%, outperforming representative collaborative baselines.
  • Baseline Comparisons and Ablation Studies (Reviewer 7uxH, mMqK, mBdX): We have added comparisons with the hallucination-score-based candidate selection baseline and different reliability weighting strategies, with results reported in Figure 3 and Appendix D.1 (Table 5), further validating the effectiveness of the proposed joint decision mechanism.
  • Semantic Similarity and Weight Mapping Sensitivity (Reviewer 7uxH, mMqK): We have compared SBERT-based semantic aggregation with LLM-based evaluation and examined alternative sentence encoders (Figure 8, Appendix D.2) and reliability-to-weight mapping functions (Figure 9, Appendix D.3).
  • Statistical Significance and Error Analysis (Reviewer 7uxH, mBdX): We have added paired bootstrap significance tests (Table 8, Appendix D.6) and expanded the analysis of candidate failure, scoring failure, and consensus failure (Figure 7, Section 4.4) to better characterize the strengths and limitations of RaTI.
  • Computational Efficiency and Reproducibility (Reviewer mBdX): We have added an end-to-end latency and GPU memory analysis (Table 2) covering candidate generation, hallucination detection, and MBR decoding, and provided further implementation and experimental details in Appendices A and B.

We hope these revisions and additional experiments adequately address the reviewers' concerns and further demonstrate the effectiveness and practical value of RaTI.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.