acceptodds
Withdrawn from ICLR 2027

Ruleusion: Generator-Free Metamorphic Testing and Answer-Conditioned Probes for Hallucination Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Without a reference answer or access to a model's internals, a hallucination has to be detected from the model's own replies. Sampling detectors ask the same question many times and read disagreement as a warning. When every sample is the same, the agreement can mean the model is right or that it repeats one error; we call this decoding collapse, and these detectors then have little to measure. Detectors that change the input instead, such as SAC³ and MetaQA, pay a large language model (LLM) to write the tests. We present Ruleusion, which checks an answer by asking questions whose answers follow from it by logic. Each test is either a fixed rewrite of the question or a short follow-up about the model's own answer. The rewrite states a logical relation, such as negation, presupposition, or entailment, so the expected answer is known before the model is queried, and the reply is graded locally. No generative model writes or grades any test. On answers that five LLMs gave to questions from five short-answer datasets, Ruleusion achieves the highest average AUROC of seven detectors, 0.744, ahead of the strongest baseline, SAC³ (0.722), and it is the most accurate detector at every call budget: with 6.5 calls per question it already matches SAC³ at 27 calls. It uses 54% fewer calls and 70% fewer tokens than SAC³. On the half of answers where all twelve samples coincide, every other detector loses accuracy, SelfCheckGPT and Semantic Entropy falling to 0.53–0.56 AUROC, while Ruleusion keeps 0.69, the same as on the other half.

Primary area:
foundation or frontier models, including LLMs
Keywords:
hallucination detection, large language models, reference-free evaluation, black-box detection, metamorphic testing, consistency-based detection, natural language inference, factuality

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_MryeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Ruleusion for black-box hallucination detection of short factual answers. It identifies “decoding collapse,” where repeated sampling produces the same incorrect answer, as a limitation of SelfCheckGPT and Semantic Entropy. Ruleusion instead uses hand-written logical transformations to generate related questions with supposedly known expected answers, requiring LLM calls only for answering these questions. The method is evaluated on 5,995 answers from five LLMs across five QA datasets.

Critical strengths
  1. The paper considers a practically relevant black-box hallucination detection setting.
  2. The evaluation covers multiple LLMs and QA datasets, providing a reasonably broad experimental setting.
  3. The motivation of reducing the number of LLM calls is reasonable.
Critical weaknesses
  1. The presentation quality is somewhat low. Why are there no references in the first paragraph of the Introduction? Actually, only one paragraph in the Introduction contains citations. After reading through the Introduction, I still cannot clearly understand what the authors are talking about. Are you serious?

  2. You mentioned a phenomenon called “decoding collapse,” but there is no further justification for it. A few cases may not be sufficient to demonstrate that this is a meaningful and general phenomenon.

  3. In the Introduction, you take SAC3 and MetaQA as examples, but provide almost no detailed description of what they actually do. This makes the discussion quite confusing.

  4. What exactly do the authors mean by “test”? This term is repeatedly used, but its meaning is not clearly defined.

  5. Is this work mainly vibe-coded? The whole framework feels like a collection of manually designed heuristics, especially given that the transformation rules are simply taken for granted without sufficient justification.

Questions for authors

Please see Weakness and check what you have written in the Introduction.

Reviewer_gqjZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work introduces Ruleusion, a metric to evaluate hallucination based on certainty. The idea behind the metric is to rephrase the question using static rephrasing (they have rules on how to rephrase) and probe the probability of the answer. One aspect of this metric is to use consistent phrasing to reduce calls to the models. They evaluated on the datasets Natural Questions, StrategyQA, FreshQA, and Mintaka. Overall, averaged across models (open and closed), they achieve mostly slightly higher results compared to other techniques and larger differences in the results when they reduce the number of calls the other metrics can use.

Critical strengths
  1. Show good detection of hallucination results with a low number of calls to the model.
  2. The metric is simple and shows good results on average model evaluations.
Critical weaknesses
  1. The results seem relatively similar to the SAC3 metric, which also uses paraphrasing in its metric.
  2. This work seems to be applicable only to setups where one can use simple rules to paraphrase the question, reducing its applicability.
  3. The differences in Table 2 seem small between Ruleusion and SAC3 and MetaQA; why are they important? Also, these tables are missing standard deviations. Making it inaccurate to comper this numbers and evaluate if there is a real differences in their score.
Questions for authors
  1. Do the authors use an exact match to evaluate the model's correctness? If yes, how do they evaluate its accuracy? If not, what do they use?
  2. Does Ruleusion have fewer calls than SAC3 because it uses consistent phrasing of the question? What are exactly the differences between the two metrics? And why did the authors make each of the modifications?
  3. Please add std to Tables 2 and 4.
  4. Also, Table 2 shows the mean AUROC across models; how different are the models? On which model does Ruleusion work better?
  5. Why should the differences in Table 2 be considered large differences between the metrics? Do they hold after adding std?
  6. If Table 9 is important to discuss in the main paper, please move the table to the main paper.
  7. Please clarify in the paper what the differences are between LOGIC, WRAPPER, probes, and wobble.
  8. Figure 2 is too big and is outside the page limits; please decrease the size.
  9. What is the ground truth? What constitutes a hallucination that each metric is evaluated on? This is important to understand to evaluate what the metrics are evaluated on. I believe it is whether the answer is correct or not, while the metrics do not have this information and evaluate based on certainty. Please explain this better in the paper.
Reviewer_brAjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Ruleusion detects potentially incorrect factual answers by asking the model a set of related questions. It uses fixed rules (LOGIC and WRAPPER) to rewrite the original question and construct follow-up probes, then checks the responses using polarity matching, year matching, and a local NLI model. Some probes also contribute confidence scores. The method requires neither reference answers nor access to the target model’s internals. Across five models and five datasets, it achieves an average AUROC of 0.744 with fewer calls and tokens than SAC³ and MetaQA. The paper also examines cases where repeated sampling returns the same answer.

Critical strengths
  1. The cost advantage is substantial. Ruleusion uses 12.5 additional calls and 870 tokens per question, compared with 26.9 calls and 2,934 tokens for the implemented SAC³ baseline.

  2. The evaluation covers five models and five datasets, with component ablations and statistical comparisons.

Critical weaknesses
  1. The practical value of the work appears limited. Without reference evidence, a rule violation reveals a conflict among the model’s responses rather than establishing that the original answer is factually incorrect. The original answer may be correct while a subsequent probe response is wrong, whereas a consistently incorrect answer may escape detection. Conversely, when a reliable reference answer is available, correctness can be assessed directly against it rather than inferred indirectly from consistency. How should users who lack those references interpret a detected violation? The paper does not clearly identify a concrete application scenario or explain how the inconsistency signal supports a useful decision. Consequently, the approach risks adding complexity to hallucination detection without sufficiently establishing its practical benefit.

  2. The methodological contribution is limited relative to existing work. MetaQA already uses meaning-preserving mutations, inverse relations, and negation to test whether related answers satisfy expected relations. BTProp also uses logical relations between claims to estimate factual confidence. Ruleusion's fixed rules and local grading offer a practical way to reduce generation cost, but the current comparisons do not isolate the value of this design from the broader idea of consistency checking.

Questions for authors
  1. What concrete deployment scenario makes authoritative evidence impractical to obtain while allowing users to make useful decisions from Ruleusion’s inconsistency score without knowing which response is incorrect?

  2. How frequently does Ruleusion flag correct original answers because of incorrect probe responses or miss incorrect original answers that remain consistent across tests?

  3. How are the assumptions underlying the NO OTHER probe and the WH SUPERL rewrite verified to prevent false positives on questions with multiple valid answers or tied superlatives?

  4. Could the authors provide successful and unsuccessful examples from each dataset showing the original question and answer, applicable rewrites, probes, expected and actual replies, final score, and a separately identified reference answer?

  5. Could the authors compare Ruleusion with a simple MetaQA variant using fixed templates or a small set of verification prompts under comparable call budgets, using the same local grader and aggregation scheme where feasible?

  6. What evidence shows that Ruleusion’s fixed rules provide diagnostic capabilities beyond those of MetaQA and BTProp, rather than primarily improving efficiency and reproducibility?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.