- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Human Interaction Profiles (HIP), a stateless inference-time control layer that separates conditional evidence injection from predefined refusal-policy enforcement. Routing uses fixed, pre-established per-prompt classifications, and the resulting context is deterministically assembled without updating model parameters or maintaining interaction history. The evaluation includes paired retrieval/no-retrieval comparisons, a larger synthetic-profile study, and a cross-provider enforcement suite. The results demonstrate the presence of citation-like and domain-anchor signals following evidence injection, unchanged refusal rates in the paired comparisons, and exact refusal enforcement on 120 pre-tagged refusal-required trials. The contribution is framed as systems verification; the grounding metrics do not assess semantic correctness or whether evidence actually supports the generated claims.
Critical strengths
HIP provides a modular, stateless design for controlling evidence injection and enforcing predefined refusal policies without modifying model parameters. The cross-provider evaluation offers a concrete, although narrow, demonstration of this design: the specified refusal output was enforced in all 120 refusal-required trials across three providers and two configurations, while the answerable controls continued to receive non-refusal responses. This is a useful engineering property for systems requiring explicit response boundaries.
Critical weaknesses
-
The research contribution beyond pipeline verification is insufficiently established. Separating answerability from grounding is a reasonable design principle, but the reported results mainly verify behaviors that the configured pipeline is designed to produce. Although the authors clearly acknowledge this limited scope, the evaluation does not establish what additional insight or operational benefit HIP provides over a simpler pipeline using the same evidence and explicit retrieval/refusal rules. The connection between the motivating problem and a substantive research advance therefore remains unclear.
-
The core routing policy is insufficiently specified. Sections 3 and A.1 describe pre-established per-prompt classifications of “evidence dependence” and “stability,” but do not provide concrete classification criteria, an assignment procedure, or sufficient detail on how these classifications determine the evidence namespace and whether evidence is strictly required. Phase-B directly supplies refusal-required tags and verifies their enforcement, so it does not resolve this upstream specification gap. These omissions hinder understanding and independent implementation of the decision policy even within the evaluated fixed-prompt setting.
Questions for authors
-
Clarification of contribution: Within the stated fixed-prompt setting, what distinct conceptual insight or operational benefit does HIP provide beyond a simpler pipeline using the same evidence and explicit retrieval/refusal rules? What evidence in the submission supports that distinction?
-
Clarification of method: What concrete criteria and procedure establish the “evidence dependence” and “stability” classifications for the evaluated prompts? How do these classifications determine the evidence namespace and whether evidence is strictly required?
2 comments
Thank you for the clarifications and for correcting the evaluation errors. Your response addresses some of my concerns, but HIP’s added value over a simpler pipeline remains unestablished, especially after the grounding results were withdrawn. The planned comparisons are relevant, but results are not yet available. I therefore maintain my rating of 1 for now and would be willing to reconsider if results addressing this concern become available during the discussion period.
Response to Reviewer 7Rpd
Thank you for a precise and fair review. Please see the correction and the follow-up note at the top of this thread: the evidence signals behind Table 3 and EXP-G1 were measured on files that included the injected passages, and the refusal counts for those sets were wrong, so we have withdrawn them. Your summary of what the remaining results show is accurate.
What does HIP provide beyond a simpler pipeline with the same evidence and explicit retrieval and refusal rules? In the submission, nothing demonstrates that distinction, and we won't argue otherwise. The original 50 prompts can't answer it either: none is labelled as needing or not needing evidence, and in the archived runs retrieval was set by the run, not decided per prompt. So we are testing it on a new, small preregistered set where every prompt has an explicit evidence-required label and fixed routing before anything runs. Each prompt runs with no retrieval, with retrieval gated by that label, with a pipeline that always retrieves from the same routed source with the same passage count and the same refusal instruction, and with broad retrieval without routing. On prompts that don't need evidence, we test whether gating reduces declines and input tokens. On prompts that do, gated and always-retrieve receive exactly the same evidence, so we test whether their claim-level support is equivalent within the preregistered ±10-point margin, and if it is, we will report that gating does not improve grounding once the evidence is the same. The decision rules, including what we report if there is no benefit on any measure, are frozen, hashed and posted here before any results are seen.
If the answer is that HIP adds an efficiency benefit on prompts that don't need evidence and nothing on grounding, that is a much narrower contribution than the paper claimed, and we will present it as exactly that.
How the classifications are established and how they determine the namespace. You are right that Sections 3 and A.1 leave this underspecified, and "classification" made it sound more general than it is. Each prompt in the two frozen packs carries fixed routing metadata set when the packs were built: the evidence index. No component reads an arbitrary prompt and assigns it a policy. In the archived runs, retrieval was set by run-level triggers: the Frozen-41 run switched on both the HD and the PD trigger, so every prompt received evidence from both domains, and the Evidence-Required-9 run used PD only. The namespaces searched came from the run configuration (five HD namespaces and one PD namespace), and the model was told to answer only from the injected passages or give the canonical refusal sentence. The revision will give the criteria and the mapping, with a worked example of one prompt going through routing, retrieval and compilation into the final model input, and we will publish the prompt files.
Phase-B does not resolve the upstream gap. Agreed. Phase-B supplies the refusal-required tag directly, so it verifies enforcement of a given boundary, not the routing decision that would produce it. We will say this explicitly in Section 4.5.