acceptodds
Withdrawn from ICLR 2027

RL-Driven Autonomous Hallucination Reduction in High-Precision Agentic Applications

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reliable hallucination detection for deployed conversational AI requires both accuracy and real-time latency—constraints existing approaches satisfy only in isolation. Generative LLM judges achieve strong accuracy but require a full autoregressive generation per turn; lightweight probes on uncertainty features offer sub-millisecond inference but degrade on the hardest cases. We identify and formally characterise a previously undocumented failure regime—confident-fluent (CF) hallucination—in which a language model assigns near-zero entropy at the verdict token (0.0003 nats, a 140x compression relative to the entropy of hallucinations that are caught) while producing factually incorrect output. Through controlled ablations on a human-annotated benchmark spanning four failure taxonomies, we show this constitutes a hard feature-space ceiling: we prove that the Youden index of any decision rule measurable in the self-assessment features is bounded above by the total-variation distance between the CF and correct-confident feature distributions, which we measure at . No linear or interaction combination of standard self-assessment features can separate CF hallucinations from true PASS turns, regardless of probe architecture or training-data scale. To address this ceiling we introduce two components and keep them distinct: POLYGRAPH, the probe that scores each turn and serves as the reward model, and VERITAS (Verification via Embedded Representations in Interactive Tuning for Agentic Safety), the autonomous Reinforcement Learning + Reward Model (RL+RM) loop that consumes it to adapt the agent's generation policy without per-turn human labels. The probe supplies a dense per-turn reward to a PPO fine-tuning loop: the agent is penalised for confident-fluent hallucinations and rewarded for grounded responses. Against an 11-method benchmark on 4,600 AI-coaching conversations, the static probe reaches 85.5% accuracy at a 12.3% hallucination rate, surpassing LLM-as-judge baselines by 5.5 points at 100x lower latency. After 500 RL steps the autonomous loop reaches an 8.0% hallucination rate—below the 10% production safety target with zero human labels beyond the initial bootstrap. Code, synthetic training data, and a replication package are released.

Primary area:
foundation or frontier models, including LLMs
Keywords:
hallucination reduction, reward modeling, reinforcement learning, probing, representation learning, large language models, agentic safety, PPO

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_H6WaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper mainly introduces two components. First, Polygraph is a gradient-boosted detector using 16 statistics and the second is Veritas, which is PPO loop using detector as a reward model. The hallucination that no confidence-based detector motivates the changing in the generation policy. In the AI-coaching conversations dataset, the detector accuracy is 85.5% and a reduction from 12.3% to 8.0% in the stated hallucination rate.

Critical strengths

1.The research direction is useful, which is combining factual monitoring and policy adaptation can be used in conversational agent depolyment, instead of relying on improving the generator. 2. The proposed detector implementation is sound, including feature groups in use, and deployment details. 3. Besides the main experiments, this paper also performs feature ablations, difficult-case analysis, and an external FActScore audit to further valid the components of the methods.

Critical weaknesses
  1. The hallucination rate is defined as the false-negative rate on the FAIL class from Section 6, and this does not measure how often the generator can cause errors. So reducing this quantity from 12.3% to 8.0% does not demonstrate improved generation correctness. Also, the summation of Hall% and Rec does not equal to 1, given hallucination rate definition, is a confusion.

  2. The setting has drawbacks. Only judge-flagged examples receive human review. Unflagged judge errors may therefore enter the reference labels as correct, potentially biasing both training and comparisons against those judges.

  3. On experiment perspective, the RL evidence comes from one run and one domain, and the authors does not include baseline comparisons, such as DPO. A controlled PPO comparison with and without the CF penalty is especially important to add.

Questions for authors

In the autonomous retraining mechanism, an unlabeled MMD drift test can detect distribution change, but it does not supply factual labels for retraining the supervised reward model. Could you explain where the label comes from?

Reviewer_dbjpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose to use POLYGRAPH (a 16-dimensional hidden-state probe) as reward model and VERITAS (an autonomous RL+RM loop using the probe as reward model) for hallucination detection and reduction in high-precision agentic applications. Specifically, this paper first formalizes a confident-fluent(CF) hallucination failure mode, and then show that such CF failure mode can only be effectively addressed via the proposed RL-driven hallucination deduction method.

Critical strengths

This paper identifies a new CF failure mode of hallucination, which reveals the limit of uncertainty-based detection. The online RL-based hallucination deduction method(the main contribution is using probe as RM for training) is interesting, and the experimental results show some improvements.

Critical weaknesses

It is not a new idea to train LM via RL for a specific ablity/task, the main advantage of this paper is to use online probe as RM, therefore the contribution is incremental. For experiments, currently the authors only compare with limited baselines, the authors should also compare other SoA hallucination detection methods. For RL results, because all RL results come from a single run, it is suspicious where the performance improvement comes from. Furthermore, all experiments are based on one LM, the robustness and the generalization should be further verified.

Questions for authors

See weaknesses in above.

Reviewer_8Kk8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents an RL-based method for fine-tuning LLMs to reduce hallucinations. It consists of a pre-trained hallucination-detection probe based on hidden states, and an RL loop which tunes the deployed model. FActScore and an MMD test on the feature distribution are used for reward hacking monitoring. The developed hallucinations detector is compared against Perplexity threshold, single-layer Token entropy, SelfCheckGPT, NLI entailment score, LLM-as-a-judge (Gemini 3.{5,7} Flash) and Turn-by-Turn probe.

The primary dataset consists of 4,600 AI-coaching conversations (46,000 turns) from a deployed business-coaching product, labelled with a cascade: a Gemini 3.7 Flash judge on all turns, then human review of all flagged cases.

Critical strengths

The submission describes a running production system as opposed to a research prototype.

The setup itself is reasonable - probe and the RL loop are clearly separated, there are online reward hacking checks.

Critical weaknesses

Misrepresented novelty

A. The paper claims to contribute a “previously undocumented failure regime—confident-fluent (CF) hallucination”. ML models in general and LLMs in particular being confidently wrong is common knowledge. For example, see Guo et al. (2017) for general neural networks, and Simhi et al. (2025) for high-certainty hallucinations in LLMs.

To give the authors credit where it’s due, they do reference the relevant prior work inside the main text:

SelfCheckGPT (Manakul et al., 2023) and semantic entropy (Kuhn et al., 2023) sample multiple responses and measure consistency; these degrade when the model is consistently wrong—precisely the CF regime.

Theorem 2 is also acknowledged as not novel result:

This is Le Cam’s two-point inequality (Le Cam, 1986)

B. Comparison with existing RL-tuned anti-hallucination work misses several highly relevant papers, see the Questions

Misleading reproducibility claims

evaluate_benchmarks.py uses functions evaluate_static_probe and evaluate_post_rl_veritas to compute the reported metric values.

evaluate_static_probe(features: np.ndarray, ground_truth: np.ndarray, probe: PolygraphProbe) doesn’t use the ground_truth and probe arguments! It simply returns the hardcoded metric values.

evaluate_post_rl_veritas runs some computations - but then again returns the hardcoded metric values (line 87).

Inadequate evaluation

Evaluation is only presented on an unpublished custom dataset. This described dataset:

  1. Apparently doesn’t differentiate between the error types, it counts every wrong answer as hallucination. Abstract promises “four failure taxonomies” - but they are not mentioned in the main text. This is especially important given the domain, business-coaching, where a lot of questions are a matter of subjective opinion, not a black-and-white factual correctness.
  2. Labels come from a Gemini judge, with humans reviewing only the turns it flagged (sec. 4.2) - so presumably hallucinations not flagged by Gemini went unchecked.

The files included in supplementary information include only metadata and numpy arrays, not the raw datasets

The objective of the RL pipeline is to reduce the hallucinations - but the actual hallucination rate of the fine-tuned model on the task in question is not reported; Tables 2&3 only present the metrics quantifying the detector accuracy, not that of the underlying model. Using an independent control, FActScore, is a right thing to do - but it doesn’t answer how much did the performance of fine-tuned model improve in production.

Reducing hallucinations can be achieved by increasing hedging and reducing overall model confidence. KL penalty and drift guard mitigate potential loss of competence - but it’s not measured.

Questions for authors

How does the proposed approach innovate upon Prasad et al. (2026), Liang et al. (2024), and Cundy & Gleave (2025)?

Do I understand correctly that in order for a sample to be labelled FAIL, it must have been first flagged by Gemini, and then marked as incorrect by a human? Tables 2 reports Hall% 13.5 for Gemini - does this mean that its verdicts for those 13.5% samples were inconsistent?

In equations 5 and 6, does g_CF penalty depend on correctness? Line 315 says that gCF fires precisely on the signature of Definition 1 (aka the Confident-fluent regime), but whether or not the model is correct doesn’t appear in the equations, they penalise confidence in general.

Figure 1b includes Grading call at 166 ms, but Table 2 lists POLYGRAPH & VERITAS at 0.0125 ms. Could you please elaborate how is latency defined in Table 2 and the abstract?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.