acceptodds
Under review as a conference paper at ICLR 2027

Inference-Time Guardrails: Reducing Hallucinations via Structured Invariants

Abstract

Large language models (LLMs) and vision-language models (VLMs) often produce outputs that are internally inconsistent: a high-level hypothesis conflicts with finer-grained fields that should determine it under explicit domain policy. Such failures persist under chain-of-thought prompting and under tool- or solver-mediated scaffolding when the model must itself nondeterministically generate fragile code or specifications at inference time. We propose *Inference-Time Guardrails* (ITG), a post-inference scaffolding framework that requires no weight updates. Given that LLM/VLM performs a single structured generation that jointly predicts the answer to a high-level hypothesis and answers to a fixed set of auxiliary probes, lightweight, human-authored predicates then check consistency, and a deterministic correction repairs violations when possible. This decouples probabilistic extraction from auditable policy enforcement. We implement ITG across a diverse set of domains. ITG is compared with structured baselines without guardrails, hypothesis-only structured outputs, unconstrained chain-of-thought, and with related work (when applicable) such as program-style (PAL) and solver-style (SatLM) scaffolding. The evaluation is performed across multiple generalist and specialist LLMs and VLMs. ITG often yields large accuracy gains over these baselines. Moreover, for smaller models, ITG sometimes closes the performance gap with larger models.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
llm scaffolding, mlsys

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_MHp8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ITG which jointly generates an answer and auxiliary facts then applies human-authored rules to correct inconsistencies without updating model weights. Experiments across contracts, charts, arithmetic, and driving scenes report substantial gains in selected settings, particularly chart medians, but mixed performance against CoT and occasional harmful corrections.

Critical strengths
  1. Substantial chart gains: On FigureQA, Llama-4-Scout’s upper-median accuracy increases from 51.8% with Base+A to 96.2% with ITG (Table 8). This is useful evidence that deterministic computation can recover information the model extracts but fails to use correctly.

  2. Inspectable enforcement: Fixed schemas and correction functions make the decision process explicit and reusable without retraining. This provides practical value for narrowly specified tasks where the rules are valid and can be independently audited.

Critical weaknesses
  1. Limited established contribution: For chart tasks, the corrected answer is effectively a deterministic function of extracted values; the benefit of additionally predicting and reconciling an answer is unestablished. GSM8K still executes model-generated expressions, weakening the claimed distinction from PAL. The experiments do not establish a substantial advance over extraction followed by conventional computation.

  2. Unreliable enforcement and validation: Some “invariants” are unjustified implications, such as third-party permission implying employee permission (Eq. 4). Separately, GPT-4o’s NDA-5 accuracy falls from 91.3% with Base+A to 78.7% with ITG (Table 5), but FCR is only 7.97% (Figure 3). A 12.6-point decline cannot arise from that harmful-correction rate on the same paired predictions.

Questions for authors
  1. Evaluation: Can you reconcile the accuracy/FCR contradiction above and provide the underlying paired counts? Additionally, Table 9 contains 325 decisions per run, whereas §4.1 describes 123 contracts × five hypotheses, implying 615 decisions.

  2. Rule validity: What justifies third-party permission implying employee permission, or absent brake lights implying a vehicle is not stopped? Are additional conditions omitted from the equations, and how are conflicting rules ordered? These assumptions can fail even with correct extraction.

  3. Baseline failures: What fractions of PAL/SatLM failures arise from formatting, execution, or incorrect reasoning? SatLM’s roughly 1–5% contract accuracies and PAL’s roughly 10% chart accuracies require explanation. Where are the GSM8K PAL/SatLM results promised in §4.3?

  4. Incremental benefit : Does predicting and reconciling H improve over extracting only A and applying the same deterministic functions? An equal-budget comparison, including input/output tokens and retries, would materially affect whether ITG offers an advantage beyond a conventional extraction-and-computation pipeline.

Reviewer_ycNROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether fixed, human-authored rules can reduce inconsistencies in LLM and VLM outputs without updating model weights. The proposed framework, Inference-Time Guardrails (ITG), asks the model to jointly generate a target prediction and answers to a predefined set of questions in a structured format. Deterministic predicates then check whether the prediction is consistent with the answers under domain-specific rules. Then a correction function modifies the prediction when a violation is detected. The schemas, auxiliary questions, and correction rules are designed manually for each domain.

The paper evaluates ITG on contract review, chart reasoning, driving-scene analysis, and mathematical problems. The reported gains vary across models and tasks, with notable improvements for weaker models in contract review and for median-related questions in chart reasoning. The evaluation also reports output token usage and the frequency of helpful and harmful corrections to characterize the trade-offs of rule-based post-processing.

Critical strengths

S1. The paper addresses a practically relevant problem: model predictions can contradict information extracted from the same input. The proposed framework is simple and interpretable, requires no model retraining, and makes the rules governing corrections explicit and auditable.

S2. The evaluation covers several text and vision tasks and multiple models, providing evidence that fixed rule-based correction can improve performance in some structured settings. The paper also acknowledges its dependence on extraction quality and human-authored specifications, which helps clarify its intended scope.

Critical weaknesses

W1. My main concern is that the methodological contribution is not sufficiently distinguished from a standard pipeline of structured extraction. ITG asks the model to produce auxiliary fields and then applies manually specified rules to obtain a consistent prediction. When these values uniquely determine the answer, checking and correcting the model’s prediction produces the same result as directly computing the answer from the auxiliary fields. Although Base+A isolates the effect of correction, it does not establish the value of jointly generating the target prediction over a simpler extraction-only pipeline.

W2. The practical applicability of ITG beyond the evaluated configurations remains insufficient. Sections 3.3 and 7 acknowledge that schemas, auxiliary probes, and correction rules require domain-specific design. However, the evaluation does not quantify this effort or show how well a fixed specification continues to work across a broader range of inputs and hypotheses within a domain. Designing and maintaining these specifications may require substantial expert effort, particularly when task requirements change or new requirement arise. This leaves uncertainty about how much additional engineering is needed to extend the reported gains and when the rules require revision.

Questions for authors

Q1. How much expert effort was required to design and refine the schemas, auxiliary questions, and rules for each domain? Which components could be reused in a related task, and which would need to be redesigned?

Q2. How robust are the fixed rules to changes in input distributions or task requirements? Could the authors provide evidence beyond the selected settings, such as broader ContractNLI coverage, and clarify when the rules would require revision?

Q3. Could the authors compare ITG with a simpler baseline that generates only the auxiliary fields and computes the final answer using the same rules? When those fields uniquely determine the answer, what benefit comes from also generating and correcting the target prediction?

Reviewer_S5R8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Inference-Time Guardrails (ITG), which extracts structured predictions and auxiliary information from LLMs, then applies predefined, human-written rules to check and correct the predictions. The authors evaluate ITG on ContractNLI, FigureQA, GSM8K, and nuScenes, reporting accuracy, token usage, and correction outcomes.

Critical strengths
  1. The research problem is worth studying. Facts extracted by models can contradict their final predictions, which is particularly concerning in applications such as legal document analysis and autonomous driving.

  2. The experiments cover four domains and multiple models. The Base+A ablation helps distinguish the effects of auxiliary information extraction and rule-based correction, while FCR and FBR show both the benefits and harms of correction.

Critical weaknesses
  1. Limited novelty. ITG mainly combines structured extraction with fixed Python rules, offering limited novelty over existing programmable validation tools such as NeMo Guardrails (which supports custom Python actions) and Guardrails AI (which provides programmatic on-fail corrections). The Related Work overlooks these capabilities while spending too much space on less relevant topics like System One models and agentic AI (along with minor typos like "complexity.gio" on page 4). The paper does not establish what new technical capability or reliability property ITG adds beyond existing constraint-checking methods.

  2. Rule design and validation. The paper lacks a systematic way to construct or validate hand-written rules in complex scenarios. Section 3.3's discussion of ISO standards and the Delphi method is overly general, while the actual nuScenes traffic rules in Appendix B.2 conflict with each other: Eq. (18) marks acceleration as unsafe if a vehicle is within 8m ahead, whereas Eq. (20) marks it as safe whenever there are no pedestrians or red lights, and the 8m/12m rule thresholds do not match the 10m/15m label and prompt thresholds. Moreover, the internal consistency checks and abstention mechanisms mentioned in Section 3.2 are never evaluated.

  3. Missing simpler baseline and unreliable corrections. The evaluation misses a simpler baseline that directly computes answers from extracted fields—for example, in FigureQA (Eqs. 12–13), the rules overwrite H in both equality and inequality cases, so predicting H first has no effect at all. ITG also loses to Base+CoT on most models in ContractNLI (Figure 1) and GSM8K (Table 6), and can severely hurt strong models when extracted facts are wrong (on NDA-5 in Table 5, Qwen3-235B drops from 96.9% to 79.4% and GPT-4o from 94.5% to 78.7%). When both LLM-extracted facts and human rules can be flawed, having more fixes than breaks on average is still far from a reliable guardrail.

Questions for authors
  1. What are the specific technical differences between ITG and existing approaches such as NeMo Guardrails and Guardrails AI? Beyond using fixed rules, what new methodology or capability does ITG introduce? A convincing explanation supported by direct comparisons could lead me to raise my rating.

  2. How are the correctness and coverage of the hand-written rules verified? In particular, how are potentially conflicting rules in Appendix B.2 handled, and why are different distance thresholds used? Have the proposed consistency checks and abstention mechanisms been implemented in the experiments?

  3. For the min/max/median questions in FigureQA (Eqs. 12–13), what role does the initial prediction H play? Would directly extracting the auxiliary information and applying the same fixed functions produce identical results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.