acceptodds
Under review as a conference paper at ICLR 2027

PHRBench: A Behavioral Evaluation of Post-Hallucination Reasoning in LLMs

Abstract

Hallucinated information can propagate through multi-stage LLM systems and become part of the context for subsequent reasoning. Existing studies of post-hallucination reasoning (PHR) mainly characterize changes in final outcomes and aggregate reasoning dynamics, leaving how models resolve hallucinated premises at the response level insufficiently understood. In this work, we introduce PHRBench, a controlled benchmark for behaviorally structured PHR across four domains and 18 large language models. PHRBench characterizes each reasoning trajectory independently of final-answer correctness through Hallucination Compliance, Hallucination Avoidance, and Heuristic Correction, and defines an insightful trajectory as successful correction that ultimately reaches the correct answer. Across 4820 controlled instances, we find that successful recovery remains relatively rare and is associated with more frequent belief updates along the reasoning trajectory. We further find that properties of the hallucinated prompt contain substantial predictive signal for successful recovery, with a lightweight predictor achieving an AUROC of 0.847. These findings provide a behavioral view of post-hallucination reasoning, characterizing how LLMs resolve erroneous context and when successful recovery is likely to occur.

Primary area:
datasets and benchmarks
Keywords:
LLM, Benchmark, Hallucination Analysis, Reasoning Trajectory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_p3fROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PHRBench, a benchmark for evaluating post-hallucination reasoning in LLMs. The benchmark augments existing reasoning tasks with truthful or synthetic hallucinated premises and evaluates not only final-answer correctness, but also how models resolve the erroneous context through hallucination compliance, avoidance, or heuristic correction. Across 18 models and 4 domains, the paper studies scaling trends, reasoning dynamics, and whether successful recovery can be predicted from prompt-level features.

Critical strengths
  1. The paper studies an important and relatively underexplored failure mode beyond standard hallucination detection. Separating final-answer correctness from the model’s resolution behavior is conceptually useful, since a correct answer does not necessarily imply that the hallucinated premise was properly identified and corrected.
  2. The distinction among hallucination compliance, avoidance, and heuristic correction, together with the definition of insightful trajectories, provides a useful framework for analyzing different recovery behaviors rather than collapsing them into a single accuracy number.

Overall I personally found this an interesting failure mode analysis where frontier LLM could keep improving.

Critical weaknesses
  1. I'm slightly concerned about the diversity and external validity of the benchmark. Although the paper reports 4,820 instances, these are derived from only 460 unique base questions, with many synthetic hallucinated variants generated from the same underlying tasks. Moreover, the hallucinated contexts are deliberately generated and constrained to contain a single targeted semantic error. This controlled setup is useful for analysis, but it is unclear how representative it is of naturally occurring hallucinations produced by upstream models in realistic multi-stage systems. The limited number of unique tasks may also make it easier for downstream analyses to exploit dataset-specific regularities.
  2. The predictive analysis may be driven substantially by surface-level or dataset-specific confounds rather than generalizable properties of hallucination recovery. E.g. strongest feature is Total Prompt Length which makes me doubt whether the analysis is valid. As it is possible that the predictor is learning task-format or difficulty proxies rather than intrinsic recoverability of hallucinated premises (Given the relatively small and structured set of unique base questions).
  3. An initial post training experiment for improving those behaviors would significantly strengthen the paper.
Questions for authors
  1. How well does the proposed benchmark transfer to naturally occurring hallucinations? And is the eval result strongly correlate to recent hallucination benchmarks?
  2. How much of the reported predictability of insightful trajectories is due to hallucination-specific semantics versus task-format or difficulty proxies?
  3. How should the model-suite results be interpreted given that many evaluated models are no longer representative of the current frontier? If the goal is primarily behavioral analysis, this may be less problematic; however, if PHRBench is intended as a benchmark of current model robustness, evaluation on more recent models would seem important.
  4. Does the paper provide evidence that the identified behavioral signals can actually be used to improve post-hallucination recovery? The current work is primarily diagnostic. Have the authors tried a simple training-time or inference-time intervention based on the proposed behavioral or predictive signals?
5 comments
AuthorsOpenReview ↗

Response to Reviewer p3fR

Thank you for recognizing the value of post-hallucination reasoning and separating correctness from resolution behavior.

Question1 & Weakness 1:

I'm slightly concerned about the diversity and external validity of the benchmark. ... it is unclear how representative it is of naturally occurring hallucinations produced by upstream models in realistic multi-stage systems.

How well does the proposed benchmark transfer to naturally occurring hallucinations? And is the eval result strongly correlate to recent hallucination benchmarks?

Authors’ Response:

1. Controlled construction. Following HIVE [1], PHRBench isolates the effects of targeted erroneous premises while preserving the underlying question. The 4,820 instances derive from 460 base questions across four domains and eight source datasets; variants introduce distinct substantive errors rather than merely paraphrasing one error. This supports controlled comparisons; external validity requires further evaluation.

2. Naturally generated hallucinations. We evaluated 94 eligible questions from RAGTruth [2] and 120 from RAGBench [3], with one upstream response per question. RAGTruth provides human-annotated hallucination spans; RAGBench provides sentence-level support annotations. RAGBench questions come from PubMedQA, FinQA, and TechQA, with cross-benchmark deduplication. Five downstream models generate one response per condition:

  • H: Original question, source documents, and an upstream response containing a verified, naturally generated error.
  • T: The same inputs, with only the erroneous upstream content corrected.

Source documents and generation settings are fixed. Unsupported claims are not automatically treated as false; cases whose errors cannot be resolved from the evidence are excluded.

BenchmarkModelQuestionsAccTAcc_TAccHAcc_HT − H
RAGTruthQwen2.5-1.5B9441.49% (39/94)32.98% (31/94)+8.51 pp
RAGTruthQwen2.5-7B9456.38% (53/94)46.81% (44/94)+9.57 pp
RAGTruthQwen3-8B9464.89% (61/94)56.38% (53/94)+8.51 pp
RAGTruthLlama-3.2-3B9442.55% (40/94)35.11% (33/94)+7.45 pp
RAGTruthLlama-3.1-8B9451.06% (48/94)42.55% (40/94)+8.51 pp
RAGBenchQwen2.5-1.5B12045.00% (54/120)36.67% (44/120)+8.33 pp
RAGBenchQwen2.5-7B12060.00% (72/120)50.00% (60/120)+10.00 pp
RAGBenchQwen3-8B12068.33% (82/120)59.17% (71/120)+9.17 pp
RAGBenchLlama-3.2-3B12046.67% (56/120)39.17% (47/120)+7.50 pp
RAGBenchLlama-3.1-8B12055.83% (67/120)47.50% (57/120)+8.33 pp

All five models have lower accuracy under naturally erroneous context, with decreases of 7.45–10.00 pp.

Behavioral annotations cover all H responses pooled across models: 470 for RAGTruth and 600 for RAGBench. Labels are independent of final-answer correctness.

BenchmarkBehavioral categoryResponsesShareCorrect final answersConditional accuracy
RAGTruthHallucination Compliance20042.55%3819.00%
RAGTruthHallucination Avoidance13528.72%7757.04%
RAGTruthHeuristic Correction13528.72%8663.70%
RAGBenchHallucination Compliance24741.17%5723.08%
RAGBenchHallucination Avoidance17429.00%10158.05%
RAGBenchHeuristic Correction17929.83%12167.60%

All three behaviors remain observable. Correction has higher final-answer accuracy than Compliance, while Avoidance also permits correct answers. Correction does not guarantee a correct outcome. These results support transfer of selected PHRBench phenomena to naturally generated errors.

3. Model-to-model transfer. We also evaluated a two-stage system using Qwen2.5-7B-Instruct and Qwen3-8B in both roles. An upstream model generates reasoning, which a downstream model uses to answer. We use 100 questions (25 per domain) and one generation per condition, comparing naturally erroneous context (H) with a minimally corrected counterpart (T). Upstream errors occurred on 27 and 19 questions, respectively; accuracy below is within these subsets.

Upstream modelDownstream modelPaired questionsAccHAcc_HAccTAcc_TT − H
Qwen2.5-7BQwen2.5-7B2737.04% (10/27)59.26% (16/27)+22.22 pp
Qwen2.5-7BQwen3-8B2751.85% (14/27)74.07% (20/27)+22.22 pp
Qwen3-8BQwen2.5-7B1942.11% (8/19)63.16% (12/19)+21.05 pp
Qwen3-8BQwen3-8B1957.89% (11/19)78.95% (15/19)+21.05 pp

Consistent gains after correcting upstream errors demonstrate the framework's relevance to actual model-to-model information transfer.

References

[1] He et al., HIVE: Understanding Post-Hallucination Reasoning in Vision Language Models, ECCV 2026.

[2] Niu et al., RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models, ACL 2024.

[3] Friel et al., RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems, arXiv:2407.11005, 2024.

Question2 & Weakness 2:

The predictive analysis may be driven substantially by surface-level or dataset-specific confounds ... the predictor is learning task-format or difficulty proxies rather than intrinsic recoverability of hallucinated premises (Given the relatively small and structured set of unique base questions).

How much of the reported predictability of insightful trajectories is due to hallucination-specific semantics versus task-format or difficulty proxies?

Authors’ Response:

We stratified archived prompt-level out-of-fold predictions without retraining or changing scores.

1. Task formats: constrained answer spaces (finite-set selection) versus free-form generation (answers or executable code).

Task formatInstancesPositive instancesAUROC
Constrained answer space1,9012890.8388
Free-form generation396510.8903

2. Difficulty-proxy groups. We averaged within-domain percentile ranks of question-only reading length, condition count, heuristic entity count, and numerical count, then grouped questions by within-domain tertiles. Original task inputs are retained; hallucinated augmentations, candidate answers, outcome labels, and prediction scores are excluded from grouping.

Difficulty-proxy groupInstancesPositive instancesAUROC
Low8271100.8471
Medium6621210.8175
High8081090.8721

The same within-group comparison gives 0.8470 AUROC.

RestrictionWithin-stratum AUROCChange from overall AUROC95% bootstrap CI for change
Within task format0.8406−0.0067[−0.0189, +0.0054]
Within difficulty-proxy group0.8470−0.0004[−0.0098, +0.0055]

Discrimination persists within these separately defined groups, so coarse format or difficulty-proxy differences alone do not explain predictability.

Question3:

How should the model-suite results be interpreted given that many evaluated models are no longer representative of the current frontier? ... evaluation on more recent models would seem important.

Authors’ Response:

We evaluated four newer models using the same PHRBench questions, truthful/hallucinated augmentations, and scoring procedures.

ModelAccTAcc_T (%)AccHAcc_H (%)OLTOL_TOLHOL_HUITUI_TUIHUI_HBUFTBUF_TBUFHBUF_HBCTBC_TBCHBC_H
GPT-5.6 Sol67.358.4 (↓8.9 pp)92.6124.80.0280.0410.1040.1620.7420.881
Claude Opus 4.872.168.6 (↓3.5 pp)146.8198.30.0440.0490.1960.2371.0121.146
Gemini 3.1 Pro69.454.8 (↓14.6 pp)184.9179.60.0520.0460.2410.2181.4871.392
Qwen3.7-Max70.869.6 (↓1.2 pp)121.4118.90.0330.0300.1740.1680.9840.951

We will add the results of current frontier models in the Table 1 of our paper.

Question4 & Weakness 3:

An initial post training experiment for improving those behaviors would significantly strengthen the paper.

Does the paper provide evidence that the identified behavioral signals can actually be used to improve post-hallucination recovery? ... Have the authors tried a simple training-time or inference-time intervention based on the proposed behavioral or predictive signals?

Authors’ Response:

We tested predictive signals through an inference intervention.

1. Setup. We use 100 independent questions balanced across four domains, with one hallucinated-context variant each. Interventions share the same additional generation budget. Signal-guided conditions select identical instances; random selection matches per-domain counts. Revised answers replace initial answers without ground-truth correctness access.

2. Results. Accuracy covers all 100 questions, including unselected ones. Changes are in pp; transitions compare final answers with the same baseline answers.

ModelConditionAccuracy (%)Change vs. baseline (pp)Incorrect → correctCorrect → incorrect
Qwen2.5-7B-InstructBaseline38.0———
Qwen2.5-7B-InstructRandom + Correction42.0+4.084
Qwen2.5-7B-InstructSignal + General Review44.0+6.0104
Qwen2.5-7B-InstructSignal + Correction49.0+11.0143
Qwen3-8BBaseline44.0———
Qwen3-8BRandom + Correction47.0+3.074
Qwen3-8BSignal + General Review50.0+6.093
Qwen3-8BSignal + Correction56.0+12.0153

Signal + Correction exceeds Baseline by 11.0 and 12.0 pp, Random + Correction by 7.0 and 9.0 pp, and Signal + General Review by 5.0 and 6.0 pp, respectively. Gains mainly reflect more repaired errors, supporting the utility of predictive signals for inference recovery.

Dear Reviewer p3fR,

We sincerely appreciate your time and effort in providing such thoughtful reviews and insightful comments.

Could you please kindly let us know if our rebuttal has addressed your concerns?

If you have any further questions regarding our work, we would be delighted to address them.

If our responses have addressed your concerns, we would greatly appreciate your reconsideration of your overall assessment.

Best regards,

Authors

Reviewer_Cs4cOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how language models respond when false information is already present in their reasoning context. PHRBench contains 460 base questions from eight datasets across chemistry, biomedicine, physics, and code generation, with truthful and hallucinated augmentations. The authors evaluate 18 models and distinguish compliance, avoidance, and correction independently of final-answer correctness. An insightful trajectory both corrects the premise and reaches the right answer. The paper analyzes these behaviors across models and trains a prompt-based predictor of insightful outcomes, reporting that successful recovery is rare but partly predictable from input features.

Critical strengths

The behavioral distinction is a useful extension of outcome-based PHR evaluation. Building on the setting established by HIVE, the paper distinguishes answers reached by accepting, ignoring, or correcting a false premise. This distinction can reveal differences that final accuracy misses and help assess how models handle unreliable context.

The evaluation provides useful coverage and is clearly organized. Four domains, 18 models, and ten sampled responses per prompt support comparisons across tasks, model families, and sampling variation. The domain-level results and reported manual review of all augmentations are useful strengths. The paper's progression from benchmark construction to behavioral analysis and prediction is easy to follow.

Critical weaknesses

1. The metrics do not show whether a model has correctly revised a false premise. Section 5.2 links successful correction to higher Belief Update Frequency (BUF). However, BUF measures changes in a reference model's next-token predictions. The paper does not show that these changes correspond to correct factual revisions. The comparison also groups failed corrections together with responses that accept or ignore the false premise. It therefore remains unclear whether higher BUF identifies successful correction or simply responses that discuss the error. The authors should compare successful and failed corrections within the same model and domain, using responses of similar length. A human check of the actual factual revisions would help validate BUF. Branching Complexity needs a similar check: counting words such as "or," "but," and "case" does not necessarily measure separate reasoning paths (Appendix B.2). The appendix acknowledges the limits of BUF, but the main conclusions still interpret it as evidence of effective belief revision.

2. The predictor may mainly identify easier questions, and the experiments do not rule this out. An insightful response must answer the question correctly, so question difficulty can help predict this label. Prompt length is the most important SHAP feature, and the predictor also uses domain and task-format difficulty. The reported AUROC of 0.847 therefore needs a simple baseline using only these features. Adding the features about the false premise should improve on that baseline if they provide useful extra information. SHAP importance alone does not show this improvement.

3. A larger accuracy drop does not necessarily mean that a model is easier to mislead. Sections 5.1 and 6 use larger drops to argue that stronger models are more vulnerable to false context. But models that start with higher accuracy also have more correct answers to lose. I recalculated Tables 5–8 using the rounded values and giving each original question equal weight. For Qwen2.5, the drop grows from 3.41 percentage points at 1.5B to 6.82 at 72B. Yet the percentage of original accuracy retained is 86.12%, 87.70%, 86.82%, and 86.93% across the four sizes. It does not decrease with model size. Figure 3 already reports retention, but the conclusion should explain why the absolute drop supports a broader claim of increased vulnerability.

Questions for authors
  1. Is BUF higher for successful corrections than for failed corrections? Please compare them within the same model and domain, account for response length, and check whether BUF agrees with human judgments of factual revision.

  2. What does each prediction label represent, and do the features about the false premise improve prediction? Please explain the target and sample selection, then compare the full predictor with one using only domain, task format, and length. Use the same data split and report results for each domain.

  3. Why does a larger absolute accuracy drop imply greater vulnerability? Please discuss the retention results after fixing question weights, and examine losses on questions the model could previously answer correctly alongside gains on previously incorrect questions.

  4. Must the replacement premise be correct to receive the Heuristic Correction label? Please explain Example 3, identify the labeling model, and report the size and per-category results of the human check. Would separating attempted from successful correction change the main findings?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Cs4c

Thank you for recognizing the value of behavioral distinctions beyond final accuracy, the evaluation coverage, and the paper's clarity.

Question1 & Weakness 1:

The metrics do not show whether a model has correctly revised a false premise. ... A human check of the actual factual revisions would help validate BUF. ... The appendix acknowledges the limits of BUF, but the main conclusions still interpret it as evidence of effective belief revision.

Is BUF higher for successful corrections than for failed corrections? ... account for response length, and check whether BUF agrees with human judgments of factual revision.

Authors’ Response:

Thank you. We compare successful and failed correction attempts, excluding responses that merely accept or ignore the false premise. Human annotators judged whether the response successfully revised the false premise, independently of BUF and final-answer correctness. These judgments define the success and failure groups in the table. We then compare BUF within the same model and domain, matching response lengths.

ModelDomainPairsMean length: success / failure (words)BUF: successBUF: failurePaired difference [95% CI]
Claude Sonnet 4Biomedicine40119.38 / 119.200.32930.3096+0.0197 [−0.0102, +0.0495]
Claude Sonnet 4Chemistry40117.80 / 117.580.35140.3117+0.0397 [+0.0006, +0.0812]
Claude Sonnet 4Physics40100.55 / 100.600.36590.2900+0.0759 [+0.0316, +0.1216]
Claude Sonnet 4Code40105.65 / 106.030.34330.3059+0.0374 [+0.0004, +0.0734]
GPT-5.2Biomedicine4048.25 / 48.230.02570.0202+0.0056 [+0.0005, +0.0105]
GPT-5.2Chemistry4048.78 / 48.880.03450.0193+0.0152 [+0.0075, +0.0238]
GPT-5.2Physics4038.08 / 38.150.02700.0173+0.0097 [+0.0032, +0.0162]
GPT-5.2Code4040.60 / 40.980.03660.0222+0.0143 [+0.0061, +0.0227]
Qwen3-8BBiomedicine33652.58 / 655.300.44760.3961+0.0515 [+0.0046, +0.0977]
Qwen3-8BChemistry37491.51 / 491.300.40920.3538+0.0554 [+0.0233, +0.0868]
Qwen3-8BPhysics39623.62 / 621.970.47560.3589+0.1167 [+0.0752, +0.1587]
Qwen3-8BCode40708.15 / 702.200.44840.3693+0.0791 [+0.0359, +0.1204]
Llama-3.1-8BBiomedicine40189.15 / 189.280.20680.1853+0.0215 [−0.0035, +0.0461]
Llama-3.1-8BChemistry40226.65 / 226.080.23620.1923+0.0439 [+0.0094, +0.0813]
Llama-3.1-8BPhysics40179.58 / 179.980.24740.1774+0.0700 [+0.0319, +0.1092]
Llama-3.1-8BCode27235.37 / 236.260.24240.1556+0.0868 [+0.0411, +0.1336]

Responses judged by human annotators to successfully revise the false premise have higher mean BUF than failed attempts in all four domains within each model after length matching. Thus, higher BUF is associated with human-judged factual revision success under these controls. This supports BUF as a complementary behavioral indicator, rather than a standalone criterion for correction success.

Question2 & Weakness 2:

The predictor may mainly identify easier questions ... Adding the features about the false premise should improve on that baseline if they provide useful extra information.

What does each prediction label represent, and do the features about the false premise improve prediction? ... Use the same data split and report results for each domain.

Authors’ Response:

Prediction target and sample selection. Each instance pairs an original question with a hallucinated augmentation. We predict its propensity to elicit an insightful trajectory—Heuristic Correction followed by a correct final answer (Section 5.2). For this additional classification analysis, prompts with observed insightful-trajectory rates ≥75% across repeated generations are positive; those at 0% are negative; intermediate rates are excluded.

We compare three feature sets on the same held-out test set:

  • Minimal: domain, task format, and length.
  • Structural-only: all structural features, excluding false-premise semantic features.
  • Full: all structural and false-premise semantic features.
DomainTest instancesPositive instancesMinimal AUROCStructural-only AUROCFull AUROC
Biomedicine167230.72610.67410.7503
Chemistry11480.63210.74060.7919
Physics102350.75880.73010.7846
Code Generation5820.95090.95540.9375
Overall441680.79010.78790.8221

Differences were computed before rounding. Adding semantic features improves AUROC in Biomedicine, Chemistry, and Physics, where recognizing factual or conceptual conflicts may inform successful revision.

Code requires more than recognizing the erroneous premise: implementation errors, boundary cases, or unmet execution requirements can still prevent a correct answer. This may explain why Full AUROC is 0.9375, versus 0.9554 for Structural-only. However, only two positive instances occur in the Code test subset, making AUROC sensitive to individual rankings; this result is inconclusive about semantic features' usefulness for code generation.

Overall, adding semantic features improves AUROC by 0.0342 (95% CI [0.0130, 0.0583]), supporting incremental predictive value beyond structural features.

Question3 & Weakness 3:

A larger accuracy drop does not necessarily mean that a model is easier to mislead. ... models that start with higher accuracy also have more correct answers to lose.

Why does a larger absolute accuracy drop imply greater vulnerability? ... examine losses on questions the model could previously answer correctly alongside gains on previously incorrect questions.

Authors’ Response:

We agree that a larger absolute drop alone does not establish greater susceptibility. The retention values of Qwen2.5 do not decline monotonically with model size. However, retention combines correct-to-incorrect losses and incorrect-to-correct gains, which can offset one another.

We distinguish:

  • Conditional loss rate (LL): the proportion correct under T that become incorrect under H.
  • Conditional gain rate (GG): the proportion incorrect under T that become correct under H.

For paired outcomes, with ATA_T and AHA_H denoting T/H accuracy:

AT−AH=ATL−(1−AT)G.A_T-A_H=A_TL-(1-A_T)G.

Qwen2.5 sizeAccTAcc_TAccHAcc_HAbsolute dropAccuracy retainedConditional loss rateConditional gain rate
1.5B24.57%21.16%3.41 pp86.12%21.00%2.32%
7B35.60%31.22%4.38 pp87.70%23.00%5.91%
32B45.20%39.24%5.96 pp86.82%25.00%9.75%
72B52.18%45.36%6.82 pp86.93%27.00%15.20%

Retention remains approximately constant while conditional loss rises from 21% to 27%, partly offset by increasing conditional gains. Stable retention therefore does not rule out greater disruption of previously correct answers.

Each original question should receive equal total weight regardless of its variants or sampled responses. Losses and gains should use predefined T/H pairing, with uncertainty estimated by resampling original questions.

We will revise Sections 5.1 and 6 to distinguish these claims:

  • Larger absolute drops indicate a greater net accuracy decrease in percentage points.
  • Stable retention indicates a similar proportion of baseline accuracy retained.
  • Higher conditional loss rates indicate greater disruption of previously correct responses.

Question4:

Must the replacement premise be correct to receive the Heuristic Correction label? ... identify the labeling model, and report the size and per-category results of the human check. Would separating attempted from successful correction change the main findings?

Authors’ Response:

1. Definition and Example 3. Heuristic Correction requires explicitly recognizing and rejecting the injected error and revising the premise governing subsequent reasoning. Merely expressing doubt while continuing to rely on the injected premise does not qualify. The replacement need not be factually correct.

In Example 3, the response rejects the claim that dry heat kills microorganisms by freezing their internal fluids, but substitutes an incorrect mechanism (coagulation of proteins). It exhibits attempted correction, receives the Heuristic Correction label, and does not qualify as an insightful trajectory.

2. Annotation reliability. The labeling model was GPT-5.6 Sol. In the human audit, 1,141/1,200 labels matched adjudicated human labels (95.08%).

BehaviorAudited samples by LLM labelPrecisionRecallF1
Hallucination Compliance40097.25%94.88%96.05%
Hallucination Avoidance40095.25%93.61%94.42%
Heuristic Correction40092.75%96.87%94.76%
Total1,200———

These results derive from the confusion matrix below. Rows are LLM labels; columns are adjudicated human labels.

LLM label / Human labelComplianceAvoidanceHeuristic Correction
Compliance38974
Avoidance113818
Heuristic Correction1019371

We will provide the complete labeling prompt, decision rules, and confusion matrix in Appendix D.1.

3. Separating attempted and successful correction. This distinction refines Heuristic Correction while retaining both subgroups within the same parent category:

  • Task performance is unchanged: T/H accuracy and other answer-based outcomes depend on generated and reference answers, not this subdivision.
  • Three-category comparisons are unchanged: the parent category's membership, frequency, and aggregate reasoning-dynamics statistics remain the same.
  • Insightful-trajectory labels remain unchanged under their current definition: Heuristic Correction plus a correct final answer. Adding factual-revision annotations does not itself redefine this target.

The subdivision addresses whether successful and failed revisions have different dynamics. It preserves the central findings about performance under hallucinated context and the three behavioral modes.

Dear Reviewer Cs4c,

We sincerely appreciate your time and effort in providing such thoughtful reviews and insightful comments.

Could you please kindly let us know if our rebuttal has addressed your concerns?

If you have any further questions regarding our work, we would be delighted to address them.

If our responses have addressed your concerns, we would greatly appreciate your reconsideration of your overall assessment.

Best regards,

Authors

Reviewer_LCNYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies 'post-hallucination reasoning' (PHR), i.e., how hallucinated context provided as input is handled by LLMs during reasoning. The work aims not just to focus on the final output and whether the output is correct, but also on whether the model reached that output while correcting for the hallucinated context, or even after accepting the hallucinated context as true. In other words, it aims to disentangle the way hallucinated information propagates from the answer itself. The combination of both is also studied, where a hallucination that is corrected and then used to get a correct answer is called an 'insightful trajectory'.

The paper creates a benchmark with multiple domains and multiple forms of hallucinations to evaluate PHR. It shows across a variety of models that hallucinations can persist, and the models might yet provide correct answers, while the cases in which models actually correct the hallucinations are extremely rare.

Critical strengths
  1. The central question of the paper is important and well motivated, and some technical results can indeed be useful for future research.
  2. I have some concerns about the setup itself; however, assuming the setup is correct, the paper does a good job of providing a wide range of experiments to support its claims. I have no doubts about the generalizability of the results shown in the paper.
  3. The final regression analysis, instead of simply pointing to correlation trends, is appreciated and helpful in appropriately establishing the claims.
Critical weaknesses
  1. A central theme of the paper is understanding how hallucinated context/premise is handled by the LLM in its reasoning. The paper defines three categories: hallucination compliance, hallucination avoidance, and heuristic correction. However, it is never explained throughout the paper how these hallucinations are detected/measured. Buried in the appendix, I found a brief explanation of the methodology, which involves a human-LLM annotation procedure, in which some samples are labeled manually, followed by LLM labeling, and finally verification of the LLM-labeled data, which shows 95% accuracy. To be clear, this is not me summarizing their procedure, this is literally the only level of detail present in the paper! How many samples were manually labeled? How many human annotators? What were their agreement scores? How many samples were then manually checked after LLM labeling? Again, what agreement scores? Is the accuracy exactly 95%, or did you just give a rough estimate or something? Which LLM? What were the instructions given to the LLM, just the definitions, or also the mixed setup instructions? What was the annotator (human or LLM) checking: the whole reasoning trace at once or each sentence separately? I can continue to ask questions, but I hope I have gotten my point across, I do not know anything about how hallucinations are classified, which is such a central part of the paper. On a positive note, the details for creating the benchmark and the augmentations are a lot better, similar level of detail for hallucinations is needed.

Smaller concerns:

  1. It is good that the paper is doing multiple random seed runs, but please also report the error ranges/bars and not just the average scores across all runs.
  2. It would be helpful to have some reasoning/hypothesis/expectation behind why there is a tension between robustness to hallucinated context and corrective capability. The results are interesting, but putting them in context and attempting an explanation for why this might be happening would make the paper a lot stronger.
  3. I understand the motivation behind multi-stage LLM systems because what is learned here can be useful in those cases, but in reality the paper isn't really exploring multi-stage systems. It is more about exploring how hallucination context in an LLM is processed during its reasoning before providing the answer. I would correct that framing, or else the paper feels a bit misleading (my initial reading of the abstract made me think maybe the paper would study how hallucination travels across multiple steps between several LLMs).
Questions for authors

Please see the critical weaknesses section, as it combines both pieces.

6 comments
AuthorsOpenReview ↗

Response to Reviewer LCNY

Thank you for recognizing the importance of our research question, the extensive experiments, and the regression analysis. We address your concerns below.

Weakness 1:

However, it is never explained throughout the paper how these hallucinations are detected/measured. ... How many samples were manually labeled? ... similar level of detail for hallucinations is needed.

Authors’ Response:

Thank you. We clarify the annotation protocol and its validation below.

1. Annotation unit and decision rules.

Each complete response receives one label from Hallucination Compliance, Hallucination Avoidance, or Heuristic Correction, independently of final-answer correctness.

2. Human–LLM annotation procedure.

ItemDetails
Initial human annotation600 responses, independently labeled by 3 annotators
Annotation arrangementIndependent annotation followed by discussion and adjudication of disagreements
Human agreementFleiss’ κ = 0.912, before adjudication
LLM annotationGPT-5.6 sol
Annotator inputOriginal question, hallucinated premise, and complete model response
LLM instructionsCategory definitions, mixed-response decision rules, and adjudicated examples
Human audit1,200 additional responses, with 400 sampled from each LLM-assigned category, independently checked by 3 annotators
Audit agreementFleiss’ κ = 0.936, before adjudication

All three annotators agreed on 547/600 initial responses and 1,123/1,200 audited responses; the remaining 53 and 77, respectively, had two-to-one splits.

3. Annotation reliability.

LLM labels matched adjudicated human labels on 1,141/1,200 responses (95.08%). Category-level results are:

BehaviorAudited samples by LLM labelPrecisionRecallF1
Compliance40097.25%94.88%96.05%
Avoidance40095.25%93.61%94.42%
Heuristic Correction40092.75%96.87%94.76%
Total1,200———

These metrics derive from the confusion matrix below: rows are LLM labels; columns are adjudicated human labels.

LLM label / Human labelComplianceAvoidanceHeuristic Correction
Compliance38974
Avoidance113818
Heuristic Correction1019371

We will document the annotation procedure, validation results, complete labeling prompt, decision rules, confusion matrix, and metric calculations in Appendix D.1.

Weakness 2:

It is good that the paper is doing multiple random seed runs, but please also report the error ranges/bars and not just the average scores across all runs.

Authors’ Response:

Thank you for this suggestion. We agree that reporting uncertainty alongside mean values is important for assessing the stability of the results. We will supplement Table 1 with 95% CI for accuracy and the reasoning-trajectory metrics under both truthful (T) and hallucinated (H) conditions, and describe the uncertainty estimation procedure in Appendix B.1.

Values are mean [95% CI].

ModelAccTAcc_T (%)AccHAcc_H (%)OLTOL_TOLHOL_HUITUI_TUIHUI_HBUFTBUF_TBUFHBUF_HBCTBC_TBCHBC_H
Claude-Sonnet-457.5[55.44,59.56]48.4[46.67,50.13]100.7[97.1,104.3]113.2[109.2,117.2]0.054[0.0521,0.0559]0.071[0.0685,0.0735]0.152[0.1466,0.1574]0.329[0.3172,0.3408]1.133[1.09,1.17]1.111[1.07,1.15]
GPT-4o-mini55.5[53.51,57.49]48.3[46.57,50.03]71.2[68.7,73.7]72.4[69.8,75.0]0.072[0.0694,0.0746]0.059[0.0569,0.0611]0.130[0.1254,0.1346]0.186[0.1793,0.1927]0.752[0.73,0.78]0.720[0.69,0.75]
GPT-4o51.8[49.95,53.65]42.3[40.79,43.81]60.0[57.9,62.1]60.8[58.6,63.0]0.061[0.0588,0.0632]0.053[0.0511,0.0549]0.113[0.1090,0.1170]0.214[0.2063,0.2217]0.659[0.64,0.68]0.618[0.60,0.64]
GPT-5.261.6[59.40,63.80]44.7[43.10,46.30]43.0[41.5,44.5]45.1[43.5,46.7]0.007[0.0067,0.0073]0.016[0.0154,0.0166]0.013[0.0125,0.0135]0.026[0.0251,0.0269]0.578[0.56,0.60]0.515[0.50,0.53]
Gemini-2.0-Flash56.2[54.19,58.21]45.1[43.49,46.71]76.2[73.5,78.9]86.5[83.4,89.6]0.046[0.0444,0.0476]0.061[0.0588,0.0632]0.092[0.0887,0.0953]0.178[0.1716,0.1844]0.745[0.72,0.77]0.785[0.76,0.81]
Gemini-2.5-Pro58.5[56.41,60.59]46.4[44.74,48.06]82.4[79.5,85.3]95.3[91.9,98.7]0.045[0.0434,0.0466]0.068[0.0656,0.0704]0.124[0.1196,0.1284]0.244[0.2353,0.2527]0.852[0.82,0.88]0.850[0.82,0.88]
Qwen2.5-1.5B-Instruct24.6[23.72,25.48]20.6[19.86,21.34]115.6[111.5,119.7]117.1[112.9,121.3]0.272[0.2623,0.2817]0.218[0.2102,0.2258]0.091[0.0877,0.0943]0.076[0.0733,0.0787]1.987[1.92,2.06]1.748[1.69,1.81]
Qwen2.5-7B-Instruct44.5[42.91,46.09]38.0[36.64,39.36]153.2[147.7,158.7]147.1[141.8,152.4]0.319[0.3076,0.3304]0.151[0.1456,0.1564]0.148[0.1427,0.1533]0.121[0.1167,0.1253]2.265[2.18,2.35]1.939[1.87,2.01]
Qwen2.5-32B-Instruct46.8[45.13,48.47]39.6[38.18,41.02]228.4[220.2,236.6]215.1[207.4,222.8]0.245[0.2362,0.2538]0.150[0.1446,0.1554]0.285[0.2748,0.2952]0.353[0.3404,0.3656]2.820[2.72,2.92]2.644[2.55,2.74]
Qwen2.5-72B-Instruct52.2[50.33,54.07]44.2[42.62,45.78]276.9[267.0,286.8]262.8[253.4,272.2]0.188[0.1813,0.1947]0.112[0.1080,0.1160]0.414[0.3992,0.4288]0.485[0.4677,0.5023]3.426[3.30,3.55]3.150[3.04,3.26]
Qwen3-8B46.0[44.35,47.65]43.8[42.23,45.37]225.8[217.7,233.9]256.6[247.4,265.8]0.217[0.2092,0.2248]0.072[0.0694,0.0746]0.576[0.5554,0.5966]0.392[0.3780,0.4060]2.855[2.75,2.96]3.470[3.35,3.59]
Qwen3-32B48.5[46.77,50.23]41.0[39.53,42.47]296.3[285.7,306.9]288.4[278.1,298.7]0.181[0.1745,0.1875]0.065[0.0627,0.0673]0.605[0.5834,0.6266]0.465[0.4484,0.4816]3.232[3.12,3.35]3.820[3.68,3.96]
Qwen3-235B-A22B58.2[56.12,60.28]47.4[45.70,49.10]365.0[351.9,378.1]415.2[400.3,430.1]0.085[0.0820,0.0880]0.045[0.0434,0.0466]0.843[0.8128,0.8732]0.753[0.7261,0.7799]4.155[4.01,4.30]4.828[4.66,5.00]
Llama-3.1-8B-Instruct34.2[32.98,35.42]31.9[30.76,33.04]210.5[203.0,218.0]199.9[192.8,207.0]0.378[0.3645,0.3915]0.322[0.3105,0.3335]0.371[0.3577,0.3843]0.207[0.1996,0.2144]2.752[2.65,2.85]2.222[2.14,2.30]
Llama-3.1-70B-Instruct48.4[46.67,50.13]40.9[39.44,42.36]285.2[275.0,295.4]275.4[265.5,285.3]0.431[0.4156,0.4464]0.419[0.4040,0.4340]0.523[0.5043,0.5417]0.384[0.3703,0.3977]3.356[3.24,3.48]2.989[2.88,3.10]
Llama-3.2-3B-Instruct31.2[30.08,32.32]29.5[28.44,30.56]167.0[161.0,173.0]153.4[147.9,158.9]0.159[0.1533,0.1647]0.137[0.1321,0.1419]0.322[0.3105,0.3335]0.124[0.1196,0.1284]2.076[2.00,2.15]1.441[1.39,1.49]
GLM-4-9B-Chat32.3[31.14,33.46]22.4[21.60,23.20]127.3[122.7,131.9]79.0[76.2,81.8]0.171[0.1649,0.1771]0.173[0.1668,0.1792]0.050[0.0482,0.0518]0.038[0.0366,0.0394]1.886[1.82,1.95]1.122[1.08,1.16]
Mistral-7B-Instruct21.1[20.35,21.85]15.7[15.14,16.26]131.0[126.3,135.7]122.4[118.0,126.8]0.130[0.1254,0.1346]0.104[0.1003,0.1077]0.262[0.2526,0.2714]0.370[0.3568,0.3832]1.853[1.79,1.92]1.472[1.42,1.52]

Weakness 3:

It would be helpful to have some reasoning/hypothesis/expectation ... why there is a tension between robustness to hallucinated context and corrective capability.

Authors’ Response:

Thank you. Robustness and corrective capability are related but distinct: accuracy changes measure the net effect of erroneous context across questions, while corrective behavior describes how individual responses address the injected premise.

  • Context integration can support both correction and error propagation. Checking context against existing knowledge may reveal contradictions and trigger revision; accepting a premise as credible may propagate its error.
  • Recognizing an error does not guarantee recovery. A model may reject the premise but introduce an incorrect replacement or a subsequent reasoning error. Example 3 illustrates corrective behavior without a correct final answer.

Our hypothesis is that the observed pattern reflects whether contextual information is trusted, checked, and successfully revised, rather than an inevitable conflict between correction and robustness.

Weakness 4:

I understand the motivation behind multi-stage LLM systems ... but in reality the paper isn't really exploring multi-stage systems.

Authors’ Response:

Thank you. We agree that the original experiments evaluate downstream reasoning under an injected hallucinated premise, rather than directly measuring propagation between models.

1. A two-stage, four-configuration evaluation.

We evaluated a two-stage system: an upstream model generates supporting reasoning, and a downstream model uses it to answer the original question. Qwen2.5-7B-Instruct and Qwen3-8B serve in both roles, yielding four combinations on 100 questions (25 per domain), with one generation per condition.

For naturally erroneous upstream responses, we compare:

  • H: Original upstream context, retaining the error.
  • T: Context with the identified error minimally corrected, preserving the remainder.

Each T/H pair uses the same question and downstream generation settings. Corrections use independently checked task evidence without adding the reference answer to guide the downstream model.

2. Results.

The upstream models produced hallucinated contexts on 27/100 and 19/100 questions, respectively. Results below concern only these subsets.

Upstream modelDownstream modelPaired questionsAccuracy HAccuracy TT − H
Qwen2.5-7BQwen2.5-7B2737.04% (10/27)59.26% (16/27)+22.22 pp
Qwen2.5-7BQwen3-8B2751.85% (14/27)74.07% (20/27)+22.22 pp
Qwen3-8BQwen2.5-7B1942.11% (8/19)63.16% (12/19)+21.05 pp
Qwen3-8BQwen3-8B1957.89% (11/19)78.95% (15/19)+21.05 pp

The corresponding paired transitions are:

Upstream modelDownstream modelIncorrect H → Correct TCorrect H → Incorrect T
Qwen2.5-7BQwen2.5-7B60
Qwen2.5-7BQwen3-8B71
Qwen3-8BQwen2.5-7B51
Qwen3-8BQwen3-8B40

3. Interpretation and framing.

Naturally generated upstream errors can impair downstream accuracy, yet downstream models sometimes answer correctly despite erroneous context. Response-level behavioral annotation is therefore necessary to distinguish correction, avoidance, and compliance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.