Inference-Time Guardrails: Reducing Hallucinations via Structured Invariants

Large language models (LLMs) and vision-language models (VLMs) often produce outputs that are internally inconsistent: a high-level hypothesis $H$ conflicts with finer-grained fields that should determine it under explicit domain policy. Such failures persist under chain-of-thought prompting and under tool- or solver-mediated scaffolding when the model must itself nondeterministically generate fragile code or specifications at inference time. We propose *Inference-Time Guardrails* (ITG), a post-inference scaffolding framework that requires no weight updates. Given that LLM/VLM performs a single structured generation that jointly predicts the answer to a high-level hypothesis $H$ and answers $A$ to a fixed set of auxiliary probes, lightweight, human-authored predicates $I(H,A)$ then check consistency, and a deterministic correction $C(H,A)$ repairs violations when possible. This decouples probabilistic extraction from auditable policy enforcement. We implement ITG across a diverse set of domains. ITG is compared with structured baselines without guardrails, hypothesis-only structured outputs, unconstrained chain-of-thought, and with related work (when applicable) such as program-style (PAL) and solver-style (SatLM) scaffolding. The evaluation is performed across multiple generalist and specialist LLMs and VLMs. ITG often yields large accuracy gains over these baselines. Moreover, for smaller models, ITG sometimes closes the performance gap with larger models.