acceptodds
Under review as a conference paper at ICLR 2027

Evidence-Gated Retrieval Separates Grounding from Answerability in Stateless LLM Systems

Abstract

Large language models often conflate whether they can answer a prompt with whether that answer is actually grounded in evidence, collapsing two separable concerns into one brittle refusal decision. We introduce Human Interaction Profiles (HIP), a stateless, inference-time control layer that conditionally injects retrieved evidence and enforces predefined behavioral boundaries using a fixed, deterministic routing scheme, with no fine-tuning and no persistent memory. We evaluate whether this deterministic design behaves as specified across four complementary evaluations spanning exactly 2,630 model outputs and three model providers: a paired answerability comparison (500 no-retrieval vs. 500 evidence-gated outputs) showed 0% refusal in both conditions; a grounding-signal validation (50 vs. 50 outputs) showed a 0% vs. 100% operational grounding-signal rate; EXP-G1 (30 profiles × 41 prompts; 1,230 outputs) held 100% on all predefined deterministic evidence-signal metrics with 0% refusals; and a three-provider Phase-B suite (300 calls) showed a predefined refusal boundary enforced by the control plane remained exact across all 120 refusal-required trials. Together, these results verify the intended pipeline behavior under the evaluated configurations; the Phase-B result additionally demonstrates provider-invariant enforcement of the predefined refusal boundary. This determinism is boundary-specific, not universal, as an exact five-word formatting constraint succeeded in only 15/30 trials. Grounding here is an operational, deterministic evidence-signal measure, not a semantic-correctness judgment, and these results should be read as a systems-verification finding rather than an independent discovery about model behavior or semantic quality.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
retrieval-augmented generation, LLM alignment, inference-time safety, grounding, refusal behavior

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 4 reviews, above 0% of the others.
Reviewer_7RpdOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Human Interaction Profiles (HIP), a stateless inference-time control layer that separates conditional evidence injection from predefined refusal-policy enforcement. Routing uses fixed, pre-established per-prompt classifications, and the resulting context is deterministically assembled without updating model parameters or maintaining interaction history. The evaluation includes paired retrieval/no-retrieval comparisons, a larger synthetic-profile study, and a cross-provider enforcement suite. The results demonstrate the presence of citation-like and domain-anchor signals following evidence injection, unchanged refusal rates in the paired comparisons, and exact refusal enforcement on 120 pre-tagged refusal-required trials. The contribution is framed as systems verification; the grounding metrics do not assess semantic correctness or whether evidence actually supports the generated claims.

Critical strengths

HIP provides a modular, stateless design for controlling evidence injection and enforcing predefined refusal policies without modifying model parameters. The cross-provider evaluation offers a concrete, although narrow, demonstration of this design: the specified refusal output was enforced in all 120 refusal-required trials across three providers and two configurations, while the answerable controls continued to receive non-refusal responses. This is a useful engineering property for systems requiring explicit response boundaries.

Critical weaknesses
  1. The research contribution beyond pipeline verification is insufficiently established. Separating answerability from grounding is a reasonable design principle, but the reported results mainly verify behaviors that the configured pipeline is designed to produce. Although the authors clearly acknowledge this limited scope, the evaluation does not establish what additional insight or operational benefit HIP provides over a simpler pipeline using the same evidence and explicit retrieval/refusal rules. The connection between the motivating problem and a substantive research advance therefore remains unclear.

  2. The core routing policy is insufficiently specified. Sections 3 and A.1 describe pre-established per-prompt classifications of “evidence dependence” and “stability,” but do not provide concrete classification criteria, an assignment procedure, or sufficient detail on how these classifications determine the evidence namespace and whether evidence is strictly required. Phase-B directly supplies refusal-required tags and verifies their enforcement, so it does not resolve this upstream specification gap. These omissions hinder understanding and independent implementation of the decision policy even within the evaluated fixed-prompt setting.

Questions for authors
  1. Clarification of contribution: Within the stated fixed-prompt setting, what distinct conceptual insight or operational benefit does HIP provide beyond a simpler pipeline using the same evidence and explicit retrieval/refusal rules? What evidence in the submission supports that distinction?

  2. Clarification of method: What concrete criteria and procedure establish the “evidence dependence” and “stability” classifications for the evaluated prompts? How do these classifications determine the evidence namespace and whether evidence is strictly required?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 7Rpd

Thank you for a precise and fair review. Please see the correction and the follow-up note at the top of this thread: the evidence signals behind Table 3 and EXP-G1 were measured on files that included the injected passages, and the refusal counts for those sets were wrong, so we have withdrawn them. Your summary of what the remaining results show is accurate.

What does HIP provide beyond a simpler pipeline with the same evidence and explicit retrieval and refusal rules? In the submission, nothing demonstrates that distinction, and we won't argue otherwise. The original 50 prompts can't answer it either: none is labelled as needing or not needing evidence, and in the archived runs retrieval was set by the run, not decided per prompt. So we are testing it on a new, small preregistered set where every prompt has an explicit evidence-required label and fixed routing before anything runs. Each prompt runs with no retrieval, with retrieval gated by that label, with a pipeline that always retrieves from the same routed source with the same passage count and the same refusal instruction, and with broad retrieval without routing. On prompts that don't need evidence, we test whether gating reduces declines and input tokens. On prompts that do, gated and always-retrieve receive exactly the same evidence, so we test whether their claim-level support is equivalent within the preregistered ±10-point margin, and if it is, we will report that gating does not improve grounding once the evidence is the same. The decision rules, including what we report if there is no benefit on any measure, are frozen, hashed and posted here before any results are seen.

If the answer is that HIP adds an efficiency benefit on prompts that don't need evidence and nothing on grounding, that is a much narrower contribution than the paper claimed, and we will present it as exactly that.

How the classifications are established and how they determine the namespace. You are right that Sections 3 and A.1 leave this underspecified, and "classification" made it sound more general than it is. Each prompt in the two frozen packs carries fixed routing metadata set when the packs were built: the evidence index. No component reads an arbitrary prompt and assigns it a policy. In the archived runs, retrieval was set by run-level triggers: the Frozen-41 run switched on both the HD and the PD trigger, so every prompt received evidence from both domains, and the Evidence-Required-9 run used PD only. The namespaces searched came from the run configuration (five HD namespaces and one PD namespace), and the model was told to answer only from the injected passages or give the canonical refusal sentence. The revision will give the criteria and the mapping, with a worked example of one prompt going through routing, retrieval and compilation into the final model input, and we will publish the prompt files.

Phase-B does not resolve the upstream gap. Agreed. Phase-B supplies the refusal-required tag directly, so it verifies enforcement of a given boundary, not the routing decision that would produce it. We will say this explicitly in Section 4.5.

Thank you for the clarifications and for correcting the evaluation errors. Your response addresses some of my concerns, but HIP’s added value over a simpler pipeline remains unestablished, especially after the grounding results were withdrawn. The planned comparisons are relevant, but results are not yet available. I therefore maintain my rating of 1 for now and would be willing to reconsider if results addressing this concern become available during the discussion period.

Reviewer_ytguOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Human Interaction Profiles (HIP), an inference-time wrapper that uses predefined prompt classifications to control evidence retrieval and enforce refusal responses, fitting in the retrivial augmented generation (RAG) framework. The main idea is to let RAG handle evidence request and refusal separately in two stages

Critical strengths

The design is rather simple and clear to me: two gates handle refusal judge and evidence retrival separately.

Critical weaknesses

Problem definition is not clear to me. The paper set the goal as separating the answerability and evidence grounding, by why that is better is not clarified sufficiently. Does it improve answering quality or enforcing the safety better, or improving efficiency? The end benefit is neither discussed or demonstrated in evaluation.

The evaluation looks incomplete. No clearly introduced experiment setup (what dataset / benchmark used), no clear metric design (I saw refusal rate as metric but I do not know why). Importantly the evaluation fails to build the connection to the research question. Why we need to separate the answerability and evidence grounding and what is the benefit for doing so? What I can see from evaluation is that the separate gatng did not make safety guardrail worse.

Questions for authors

The major question: What is the benefit of separating the answerability and evidence grounding? Without answering that I cannot process the contribution of the paper correctly.

2 comments
AuthorsOpenReview ↗

Response to Reviewer ytgu

Thank you for your review and for pushing on the most basic question. Please also see the correction and the follow-up note at the top of this thread: the grounding results in Table 3 and EXP-G1 were mismeasured and have been withdrawn, which makes your question about the benefit even more important to answer honestly.

What is the benefit of separating answerability from grounding? The benefit we had in mind is practical. If every prompt is forced through retrieval, prompts that don't need evidence still get passages, which costs input tokens and, under an instruction to answer only from the evidence, can push the model to decline questions it could have answered. Separating the two lets a system retrieve only when evidence is needed, and enforce a refusal boundary independently of what retrieval returned. You are right that the submitted paper does not demonstrate this. Table 2 only shows that gated retrieval did not increase the decline rate compared with no retrieval, and Phase-B shows that the refusal boundary can be enforced independently, but neither tests the benefit over a simpler design.

We are testing it with a new, small preregistered set of prompts, each labelled in advance as needing evidence or not. It compares retrieval gated by that label with a pipeline that always retrieves from the same routed source, with the same passages and instruction. On prompts that don't need evidence, the question is whether always retrieving causes more declines and more input tokens. On prompts that do, both see exactly the same evidence, and if their claim-level support is equivalent, we will say that gating does not improve grounding itself. If gating shows no benefit on any measure, that is what we will report.

Experimental setup, datasets and metrics. You're right that the setup was not clearly introduced. The paper's prompts are 50 frozen prompts in two packs (Frozen-41 and Evidence-Required-9), written for this study rather than taken from a public benchmark. They carry routing metadata, not an evidence-required label, and in the archived runs retrieval was set by the run: every Frozen-41 prompt received passages from both evidence domains, and every Evidence-Required-9 prompt from one. The revision will describe the packs, the runner, how evidence was injected (twelve passages per evidence domain, cut to previews of about 500 characters) and the instruction the model receives. On metrics: we used refusal rate because the failure mode on answerable prompts is declining to answer. For grounding, claim-level support judged by two blind raters replaces the pattern-based signal.

The connection between the evaluation and the research question. We agree this was missing, and the correction makes the gap larger. The revision will state each research question next to the experiment that answers it, and will only keep the questions the evidence actually supports.

Thanks for the rebuttal.

As the message mostly agrees with the weaknesses discussed in my review, and the new support evidence is not yet available. I am going to keep the rating until new results come in.

Reviewer_M8bFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses an important distinction between grounding and answerability. However, the proposed deterministic routing mechanism has limited technical novelty. The evaluation primarily verifies predefined system behavior rather than demonstrating improvements in factual correctness or grounding quality. Stronger comparisons and meaningful RAG benchmarks are needed.

Critical strengths

The distinction between response availability, evidence injection, and refusal policy is useful for systems engineering. The paper clearly acknowledges that its operational grounding measure does not establish factual correctness. It also evaluates refusal-boundary enforcement across three providers.

Critical weaknesses

The grounding metric measures evidence-related signals, not whether generated statements are actually supported by retrieved documents. A reported 100% grounding-signal rate therefore does not demonstrate successful grounding.

The routing validator relies on fixed, pre-established classifications of prompts. Its deterministic behavior is largely expected by construction.

There is no convincing demonstration of improved factual accuracy, reduced hallucination, better retrieval quality, or generalization to unseen queries.

Questions for authors

Can the system improve claim-level factual support on a recognized RAG benchmark? What happens with unseen prompts, noisy retrieval, conflicting evidence, and ambiguous evidence needs? How does HIP compare against a simple rules-based retrieval pipeline?

1 comment
AuthorsOpenReview ↗

Response to Reviewer M8bF

Thank you for your review. Please see the correction and the follow-up note at the top of this thread first: the evidence signals behind Table 3 and EXP-G1 were measured on files that included the injected passages, and on the answers alone the reported 100% grounding-signal rate does not hold. We have withdrawn those results. Your point that the metric does not demonstrate grounding was right.

Claim-level factual support. We now measure it directly, in two places. In a reconstruction of the archived runs, two raters judge each atomic claim against exactly the evidence those runs received, with the original 500-character previews and, separately, the full passages, blind to condition and provider. In a new preregistered evaluation, the same rating is applied to prompts labelled as needing evidence. We will post the preregistration hashes here before anything is run, and the results here if they are complete in time, otherwise in the revision.

A recognized RAG benchmark. We won't have a recognized benchmark result within this discussion period, and we don't want to promise one. We agree that a standard benchmark is needed before any general grounding claim, and we will say so in the limitations.

Unseen prompts, noisy retrieval, conflicting and ambiguous evidence. None of these are evaluated in the paper. Routing uses fixed metadata assigned to known prompts, so its determinism is, as you say, expected by construction, and it says nothing about how HIP behaves on new prompts. The revision will state this directly. Comparing previews with full passages in the reconstruction gives a small, indirect look at whether the injected evidence was sufficient, but we don't claim it addresses noisy or conflicting evidence.

Comparison with a simple rules-based retrieval pipeline. This is the central comparison of the new evaluation. The original 50 prompts can't support it, since none is labelled as needing or not needing evidence and the archived runs set retrieval by run rather than per prompt. The new set gives every prompt an explicit evidence-required label and fixed routing before anything runs. A pipeline that always retrieves from the same routed source, with the same passage count and instruction, then differs from HIP only in the gating. We test whether gating reduces declines and input tokens on prompts that don't need evidence, and whether claim-level support differs on prompts that do, where both see exactly the same evidence. If they come out equivalent, we will report that gating itself does not improve grounding. A broad baseline without routing tests what the routing adds. All decision rules are frozen and hashed before results are seen.

What remains of the paper after the correction is narrower: gated retrieval did not increase the decline rate in the matched 500 vs. 500 comparison, and the control plane enforced a predefined refusal boundary across three providers, with a clear limit at 15/30 for the five-word constraint. We will be careful not to present more than that.

Reviewer_VuSfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Human Interaction Profiles (HIP), a stateless inference-time framework that uses predefined prompt classifications to decide when to retrieve external evidence and applies fixed rules to control refusal behavior. The authors evaluate HIP through paired retrieval and no-retrieval experiments, a larger study with synthetic profiles, and a cross-provider test. The results show that evidence gating increases citation and domain-anchor signals while keeping the refusal rate unchanged. The cross-provider experiment also achieves 120/120 exact refusals for predefined cases.

Critical strengths

The paper makes a clear distinction between answerability, grounding, and refusal enforcement. The authors also acknowledge that the reported grounding signals do not measure factual correctness, which helps avoid overstating the experimental findings.

The evaluation includes several experimental settings and three model providers. The paper reports both successful and unsuccessful cases, such as the 120/120 exact refusals and the 15/30 five-word formatting results. These results help show which constraints the system can enforce reliably.

Critical weaknesses

Overall, the paper is closer to an implementation and functional validation study than a substantial research contribution. I have the following major concerns:

  1. The paper only checks whether answers contain citation markers and domain-related words or patterns. It does not check whether the answers are correct or supported by the retrieved evidence. Therefore, the 0% to 100% improvement in Table 3 does not show that HIP improves actual grounding. Also, both Tier A and Tier B have 0% refusal rates, so the experiments do not show any improvement in answerability.

  2. The proposed method mainly combines predefined routing labels, conditional retrieval, and fixed output rules. The technical novelty seems limited. The paper also lacks comparisons with simple baselines, such as always-on retrieval, making it difficult to see the benefit of the proposed design.

  3. The cross-provider refusal experiment is not very convincing. The refusal-required prompts are tagged in advance, and HIP simply maps these tags to a fixed refusal response. The 120/120 result is therefore largely expected. Similarly, the routing mechanism relies on predefined classifications, and the paper does not evaluate how it handles new prompts or incorrect classifications.

  4. The paper is unnecessarily repetitive in several places, especially when discussing the difference between grounding signals and semantic correctness. At the same time, the method description lacks concrete examples. For instance, it is unclear how the complete HIP input is constructed, and the differences between HIP_01 and HIP_03 are not explained. These details would help readers better understand the experiments.

Questions for authors
  1. Can the authors provide an independent assessment of whether the retrieved evidence actually supports the generated answers? Even a small evaluation of factual correctness and citation support would help determine whether the observed evidence signals reflect meaningful grounding.

  2. Can the authors clarify how the predefined prompt classifications are constructed and validated? In particular, what would happen when a new prompt has no existing classification or is incorrectly classified?

2 comments
AuthorsOpenReview ↗

Response to Reviewer VuSf

Thank you for such a careful review. Before answering, we want to point you to the correction and the follow-up note at the top of this thread. The evidence signals in Table 3 and EXP-G1 were measured on files that also contained the injected passages, and the refusal counts for those sets were wrong. On the answers alone, the 0% to 100% result does not hold, so we have withdrawn Table 3, the EXP-G1 results and the claims built on them. Your first concern was right, and in a stronger form than you put it.

Grounding is only pattern-based. We agree, and after the correction the paper no longer claims that Table 3 or EXP-G1 demonstrate grounding. Your first question asks for an independent check of whether the evidence supports the answers. We now do this in two places. In a reconstruction of the archived runs, answers are split into atomic claims and two raters judge each claim against exactly the evidence those runs received, once with the original previews and once with the full passages, working blind to condition and provider. In a new preregistered evaluation, the same claim-level rating is applied to the prompts labelled as needing evidence. Agreement is reported before reconciliation.

Answerability shows no improvement, since both tiers are at 0%. This is a fair reading of Table 2. What Table 2 can show is that gated retrieval did not make the model decline more often than no retrieval in the matched 500 vs. 500 comparison. It cannot tell us whether gating helps, because it has no always-retrieve condition.

Novelty and the missing simple baseline. We accept that each piece of HIP is simple, and that the paper did not show what the combination adds. The original 50 prompts cannot answer this, because none of them is labelled as needing or not needing evidence, and in the archived runs retrieval was set by the run, not decided per prompt. So we are writing a new, small set of prompts, each with an explicit evidence-required label and fixed routing, frozen and hashed before anything runs. It compares no retrieval, retrieval gated by that label, always-retrieve from the same routed source with the same passage count and instruction, and broad retrieval without routing. On prompts that don't need evidence, it tests whether always retrieving leads to more declines and more input tokens. On prompts that do, gated and always-retrieve see exactly the same evidence, and if their claim-level support comes out equivalent, we will say plainly that gating itself does not improve grounding. The decision rules, including what we report if gating shows no benefit, are frozen before any results are seen.

The 120/120 refusal result is expected. We agree, and we tried to say so in Section 4.5. It verifies that the control plane enforced a boundary the harness supplied, across three providers and two presets, and not that the models detected it. We still think that is worth showing as a systems property, and the 15/30 five-word result shows where that determinism stops. We will make this framing more direct.

How the classifications are constructed, and what happens with a new or wrong one. Each prompt in the two frozen packs carries fixed routing metadata set when the packs were built: the evidence index to retrieve from. No classifier reads an arbitrary prompt and decides its policy. In the archived runs, retrieval was set by run-level triggers. The Frozen-41 run switched on both the HD and the PD trigger, so every prompt received evidence from both domains, and the Evidence-Required-9 run switched on PD and turned HD off. Which namespaces were searched came from the run configuration, not from the prompts: five HD namespaces and one PD namespace. The study only ran prompts that carry this metadata, so it doesn't show what happens with an unlabelled or wrongly labelled prompt, and we will state this as a limitation. We will publish the prompt files with the revision.

Repetition, and HIP_01 vs. HIP_03. You're right that the paper repeats the caveat about grounding signals too often while leaving out concrete detail. The revision will cut the repetition, show a full HIP input end to end, and explain the two presets, HIP_01 (Emotional · Reflective) and HIP_03 (Analytical · Evidence-heavy · Structured), including exactly what each one changes in the input the model receives.

Thank you again. Your review asked exactly the questions the new evaluation is built to answer.

Thank you for the additional clarification and experiments. I remain concerned about the fundamental weaknesses. The withdrawal of Table3 and EXP-G1 weakens the support for the central claims of the paper. While authors propose additional results with simpler baseline, these are not concrete results that can be assessed at this stage.

Moreover, the clarification reinforces my concerns about the limited novelty and lack of evidence for the proposed method. The refusal results also remain largely expected by construction. Overall, thank for the clarifications, the rebuttal does not provide sufficient new results to change my assessment. I therefore remain my original rating.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Correction to the grounding-signal validation (Table 3) and EXP-G1 results (1/2)

Correction to the grounding-signal validation (Table 3) and EXP-G1 results

While preparing our responses, we found a mistake in how two of our four evaluations were measured. We wanted every reviewer to see it before anything else, so we are posting it first. It affects Table 3, the EXP-G1 results in Section 4.2, and the sentences in the abstract, contributions and discussion that build on them. Table 2 and the Phase-B results are not affected. We have started a replication and will add its results to this thread as soon as they are ready.

What went wrong

The script that measured evidence signals for Table 3 and EXP-G1 read each stored run file as a whole. For the evidence-gated runs, that file holds both the model's answer and the passages we injected into its context. So when the script found citation-like markers and domain words, it was mostly finding them in the injected passages, not in the answers.

The refusal numbers for these sets were wrong too. Most of the outputs are our system's own refusal sentence, "I don't have enough evidence in the provided context to answer that.", which our runner itself counts as a refusal. Yet the paper reports 0% refusals. The refusal audits archived with these runs explain the Table 3 figure: they report 0 of 50 in each condition. Their pattern looked for that sentence written with a typographic apostrophe (’), while the model's outputs use a plain one ('), so it never matched. That apostrophe is what explains the Table 3 refusal figures. It is a separate problem from the grounding signals described above, which come from the whole-file audit. The audits archived with the EXP-G1 runs use the same pattern and count 69 of 1,230. That is still far below the 987 refusal sentences, and it isn't the 0% we reported either, so EXP-G1's reported 0% refusal remains unexplained: we have not found where that figure came from.

We also left out of the method section how evidence was injected in these runs. In the Table 3 runs, each Frozen-41 prompt received 24 passages, twelve from each evidence domain, and each Evidence-Required-9 prompt received twelve, all cut to a preview of about 500 characters, along with an instruction to refuse if the context didn't contain enough evidence. In most of the evidence-gated cases, the model decided these short excerpts weren't enough and gave the refusal sentence.

The corrected numbers

We reran the paper's own regular expressions on the answers only. The recount reads the archived files without changing them, and both the script and its input manifest are hashed, so anyone can rerun it and get the same table.

SetnCitation-like markerSecondary-domain anchorPrimary-domain anchorRefusals
Grounding validation, no retrieval500/50 (reported 0/50)0/50 (reported 0/50)0/50 (reported 0/50)50/50 (reported 0%)
Grounding validation, evidence-gated500/50 (reported 50/50)2/50 (reported 50/50)10/50 (reported 48/50)40/50 (reported 0%)
EXP-G1, evidence-gated1,2300/1,230 (reported 100%)48/1,230 (reported 100%)243/1,230 (reported 100%)987/1,230 (reported 0%)

The "100% non-empty evidence previews" figure in Section 4.2 describes what we gave the model, not what it produced, so we no longer report it as a result.

AuthorsOpenReview ↗

Correction to the grounding-signal validation (Table 3) and EXP-G1 results (2/2)

What this changes

The paper's grounding claim no longer has evidence behind it. We withdraw Table 3, the EXP-G1 results, the "0% vs. 100% operational grounding-signal rate" and EXP-G1 sentences in the abstract, the EXP-G1 part of contribution 3, and the sentence in the discussion saying Tier B reached a 100% grounding-signal rate at 0% refusal. Together, these account for 1,280 of the 2,630 outputs.

Separating answerability from grounding is still the idea that motivates HIP, but the paper doesn't demonstrate it as a result. We will rewrite the discussion paragraph about repeated 100% and 0% outcomes, since Phase-B is the only one of its three examples still standing, and narrow the abstract's claim about the 2,630 outputs to the evaluations that hold. The revised PDF will reflect all of this, including how evidence was actually injected.

What stays the same

Table 2 (paired answerability, 500 vs. 500). We checked these outputs again directly. Every pair is there, and neither condition has a single refusal or empty answer. These runs used a different runner and prompt from the Table 3 sets, and the revised paper will say so. Neither condition contains citation-like markers, so Table 2 shows that gated retrieval keeps the model answering, not that it grounds the answers. As an exploratory, after-the-fact observation, the secondary-domain anchor goes from 73/500 without retrieval to 198/500 with gated retrieval, but we don't treat this as a confirmed result.

Phase-B (Tables 5 and 6). Phase-B has its own harness and evidence chain and measures enforced refusals, so this issue doesn't touch it. The 120/120 result still verifies that the control plane enforces a boundary the harness supplies, and the 15/30 five-word result is still its limit.

Table 4 (latency). We are checking exactly what Table 4 timed, and we won't rely on it until we've confirmed that.

The replication

The replication tests grounding directly, and it also answers the question several of you raised: what does gating add over a simpler pipeline that uses the same evidence?

We use the same 50 frozen prompts (Frozen-41 and Evidence-Required-9) with the evidence-required labels they already have, so nothing gets relabelled after we see results. Each prompt runs in three conditions: no retrieval, HIP's evidence-gated retrieval, and a simple pipeline that always retrieves from the same routed namespace with the same number of passages and the same instruction. The only difference between the last two is the gating. On evidence-required prompts they see the same evidence, so if their grounding is the same there, that tells us gating itself is not what improves grounding, and we will report it that way. Separately, a broad retrieval baseline that draws on all namespaces without routing tests what the routing adds. We run one small model from each of three providers, two repeats per prompt, and report each provider separately. The main condition uses the original previews of about 500 characters so the results compare with the archive, and full passages are a second, pre-registered condition.

This time, every call stores the injected passages, the exact model input and the raw answer separately, and every measure is computed on the answer alone. We measure refusals, both as the exact refusal sentence and as any short decline, the paper's evidence-signal patterns, input tokens and latency. On a fixed random sample, two raters judge each claim against the stored passages, working from a shuffled packet that hides the condition, the provider and whether an answer is original or from the replication, with a fixed rubric.

We freeze and hash the protocol, prompts, code, scoring rules and decision rules, including what we will report if gating shows no benefit, before we look at any new results. We will post those hashes here first, then the results as soon as they are ready, before the discussion closes.

We're sorry for this mistake, and we thank you for reading the paper so carefully. Our responses to each review follow, and they rely only on the results that still stand.

AuthorsOpenReview ↗

Follow-up to our correction: prompt labels and the replication plan

While preparing the replication runner, we went back to the archived prompt files and run scripts, and we need to correct one statement in our earlier comment. We wrote that the replication would use the 50 prompts with the evidence-required labels they already have. That is not accurate. The two prompt files (Frozen-41 and Evidence-Required-9) contain only an id, the prompt text and routing fields. None of the 50 prompts carries an evidence-required label, and none is marked as not needing evidence.

The archived scripts also show how retrieval actually worked in those runs. All 50 prompts are routed to the HD evidence index. For Frozen-41, the run script also switched on the PD retrieval trigger, which overrides the per-prompt routing, so every Frozen-41 prompt received evidence from both domains. For Evidence-Required-9, PD retrieval was forced on and the HD index was switched off for that run. In practice, the evidence condition in Table 3 was fixed by the run settings rather than decided per prompt.

Because of this, the three-condition plan described in our correction is replaced by two separate parts. The first is an archival reconstruction. We keep the original 50 prompts unchanged and reproduce the retrieval the archived runs actually performed, with the original previews and, separately, the full passages taken from the source documents. We will report this as a reconstruction and evidence-sufficiency analysis, not as a test of gating, and we will not assign evidence-required labels to these prompts after the fact.

The second is a new, small preregistered evaluation for the gating question. Every prompt in it will have an explicit evidence-required label, fixed routing metadata and frozen scoring rules before anything is run. It compares no retrieval, retrieval gated by that label, always-retrieve from the same routed source, and broad retrieval without routing. For prompts that require evidence, the gated and always-retrieve conditions receive exactly the same evidence.

We will post the hashes of a forensic note on the archived runs, the prompt sets and the protocol before any results are generated. Our correction said results would follow before the discussion closes. We will post them here if they are complete in time, and otherwise they will be in the revision. No new results have been run or inspected.

AuthorsOpenReview ↗

Hashes of the forensic note, reconstruction inputs and current protocol snapshot

As promised in our follow-up, here are the hashes of the documents and inputs behind the replication. Nothing new has been run: no model call has been made for either part, and no new results have been generated or inspected. We will publish all of these files with the revision, so anyone can check them against the hashes below.

Archival forensic note. This note records what the archived runs actually did: the prompt files, the run scripts and their trigger settings, how those triggers overrode per-prompt routing, and which namespaces were searched. It also keeps apart what was intended, what was configured, what the frozen runner implements and what happened in the runs.

  • Forensic note, revision 2 (17 November): e746a82f23fbbe4cd8227d84f1a23a3eccfb927abd724072cd7acb3bedfe20d5
  • It replaces the first version (16 November), which had the same findings: 8a30acb8dd48d55da53677607b8d8e74e54f2f8ab6c2daaa30060804744205a7

The two original prompt files, unchanged and without any evidence-required labels:

  • Frozen-41: 3dbe42bd24526c24e3d5efe7f5efd1d7ddea3fbfbdb5ff72ab584833378660f0
  • Evidence-Required-9: 6482ef5ee3791d7688bb2a146d3c1cea3be3af4c36956b3f758f8bd67a480e33

The archived runner and run scripts:

  • run_hip_pilot_for_submission.py: 5650619f947e4fb23c7cc15009d04ce89fb9328c75f0340fc0b9b93f2d4d94b5
  • run_tier_compare_v14.sh: fdfa230c9bd76cc18ac6f4f2a21d3df761d6f730b7e92e8617030165b5ab37fd
  • pilot_run_env.hip_v1_1_gated.sh: b30b11d0596c8d9bdfee90e61df647b534f7bfbdb9ee21ac1c2f0175b3ef1f93

Archival reconstruction inputs. The profile values passed to the archived runs were reconstructed from the historical command that created the original file, which we no longer have, and we will report them as reconstructed. The reconstruction plan reproduces the archived evidence for all 100 context blocks.

  • Reconstructed profile file: eb4b0dd25d865035854b8a1950e92321fa2f19dad2219760a91e57601b9521cd
  • Reconstruction plan (200 calls): 6124d9e449e776acae7c549aa93c37c0da98c5dc3d7c7f4e6bd2df821b6bac2f

Current pre-run protocol snapshot. This is version 3 of the replication protocol, dated 17 November. It is not the final frozen preregistration. Six practical items in it are still open, mainly the source documents for the full-passage condition, the models and their access for the new gating evaluation, its prompt set, which has not been written yet, its labellers and raters, and the call budget.

  • Replication protocol, version 3: 7f456dcbd8ea170d762806c71a8fe5c00e165f1978633fcbf5950c66fe44b9e7

If resolving any of these items changes the protocol, we will post the new and final protocol hash here, together with the hash of the new prompt set and its labels, before anything is executed. Only once that final hash is posted will we treat the preregistration as frozen and make any model call. Until then, no model calls have been made and no results have been generated or inspected.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.