- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper builds 300 matched safe/unsafe agent states that differ in one safety fact, and measures how much that fact changes 6 models' responses under 3 interfaces: an ALLOW/BLOCK judgment, a candidate-or-STOP checkpoint, and open next-action selection. Sensitivity drops from judgment to action in every model. A 2×2 follow-up shows that checkpoint framing increases sensitivity for every model, while the effect of the available action set varies by model.
Critical strengths
- The matched-pair design keeps the task, candidate action, and other facts fixed while changing one safety-relevant fact, enabling a controlled comparison across response interfaces.
- The statistical analysis accounts for repeated measurements within pairs and dependence across construction templates. I also appreciated that the corresponding factorial arms closely reproduce the benchmark's checkpoint and open-action estimates, which is a useful consistency check.
Critical weaknesses
- The incremental contribution beyond established judgment-action divergence needs more justification. The claim that safety judgments and actions diverge is already established (e.g., ManagerBench), and the paper doesn't say which interface gives the valid measurement. The matched factual interventions provide a more controlled comparison, but the paper should explain what additional understanding this comparison provides beyond confirming that divergence.
- The judgment-action comparison also changes several aspects of the elicitation protocol, whose full instructions are not provided in the main text. The factorial separates framing and action-space effects within the action interfaces, but does not identify which differences between judgment and action account for their gap.
- The practical significance of the measured differences remains unclear. The paper evaluates the first response before tool execution and does not establish whether sensitivity under any of the interfaces predicts safety during actual task execution.
Questions for authors
See above. Most concerned about novelty and significance.
3 comments
3. The appropriate measurement interface depends on the evaluation target
The reviewer asks which interface provides the valid measurement. We argue that this question requires distinguishing the behavior being measured:
- J (Judgment): whether a model explicitly recognizes the safety implications of a proposed action.
- D (Checkpoint admission): whether a model admits or rejects a specified candidate action at a decision checkpoint.
- E (Open first-action selection): whether a model selects the candidate as its next action when alternatives are available.
These interfaces measure related but distinct aspects of safety-sensitive behavior. There is no reason to expect one interface to serve as an interchangeable measurement of all three.
The practical implication is direct: a model's ability to recognize an unsafe action in an explicit judgment does not establish that the same safety-relevant information will exert an equally strong influence on its action selection. Consequently, judgment-based evaluations should not be treated as direct proxies for agent action-selection safety.
SameFact provides a framework for identifying when these measurements agree, when they diverge, and which safety factors are most affected by the interface transition. This is important for both benchmark design and model comparison, where relying exclusively on aggregate judgment scores can obscure meaningful differences in action-side behavior.
4. The matched-intervention effect persists beyond first responses
To examine whether the findings are restricted to the initial response boundary, we conducted a stateful evaluation using 120 matched pairs, four backbones, and three rollout seeds in an inert simulator (720 trajectories per model).
The safe/unsafe target-commit rates and their differences (pp; 95% CIs) were:
| Model | Safe commit | Unsafe commit | Difference |
|---|---|---|---|
| Qwen | 100.0% | 90.6% | +9.4 [5.8, 13.3] |
| Gemma | 97.5% | 68.9% | +28.6 [21.9, 35.6] |
| GLM | 94.4% | 54.7% | +39.7 [32.8, 46.7] |
| MiniMax | 96.9% | 64.4% | +32.5 [26.4, 38.9] |
Approval requests were also more frequent in unsafe than safe trajectories across all four models (Qwen: 72.5% vs. 27.2%; Gemma: 46.4% vs. 8.9%; GLM: 50.8% vs. 8.9%; MiniMax: 42.2% vs. 2.5%). Strict response validity was at least 94.2% in every model×state cell.
These results demonstrate that the matched factual intervention continues to influence subsequent commitments, rather than appearing only in isolated first responses.
Taken together, SameFact contributes a controlled cross-interface measurement framework, experimental identification of model-dependent protocol effects, and evidence that the intervention's behavioral influence persists into subsequent commitments. Its central contribution is to show that the measured influence of safety-relevant information depends jointly on the model, the factual intervention, and the response interface.
Revisiting the Contributions and Implications of SameFact
Dear Reviewer,
Thank you again for your detailed and constructive feedback. We appreciate your questions regarding the novelty of SameFact beyond existing judgment–action divergence studies, the confounding effects of different elicitation protocols, and the practical significance of our findings.
We have further strengthened our responses and conducted additional experiments to address these concerns.
In particular, we would like to highlight three aspects of our additional work:
1. A more controlled measurement perspective
We clarified how SameFact differs from prior work such as ManagerBench. Rather than simply measuring whether safety judgments agree with action choices, SameFact examines whether the influence of the same controlled safety-relevant factual intervention is preserved across different response interfaces.
Our results show that interface changes can affect not only the magnitude of safety sensitivity but also the relative ordering of safety factors across models. This provides a different perspective on how safety-relevant information influences model behavior.
2. Additional identification of protocol effects
To better understand the Judgment–Admission transition, we conducted a 120-pair Protocol Bridge experiment that independently examines framing and response-contract effects. Together with our original factorial experiment, these results provide further evidence that concrete protocol choices contribute to the observed differences, with substantial model-dependent effects and interactions.
3. Evidence beyond isolated first responses
We also conducted an additional stateful evaluation in an inert simulator, covering four backbones and 120 matched pairs. The results show that the influence of matched safety-relevant factual changes persists into subsequent target commitments, extending the evidence beyond the initial response boundary.
We hope these additional analyses help clarify the methodological contribution of SameFact and its implications for safety evaluation and model comparison.
We would sincerely appreciate it if you could take another look at our responses when you have a chance. As the discussion deadline is approaching, we are concerned that there may be limited time for us to address any further questions or concerns that emerge later in the discussion. If there are still aspects of the contribution or experimental evidence that you find insufficiently clear, we would be very grateful if you could share your thoughts at your earliest convenience, so that we have an opportunity to respond before the discussion closes.
Thank you very much for your time and consideration. We truly appreciate the opportunity to discuss these issues with you.
Best regards,
The Authors
Response to Reviewer quqt
We thank the reviewer for recognizing the value of SameFact's matched-pair design and controlled comparisons. We clarify three aspects of our contribution: how SameFact differs from prior work on judgment–action divergence, what our protocol experiments reveal about the judgment–admission transition, and what these findings imply for agent-safety evaluation.
Our central finding is that the effect of the same controlled safety-relevant factual intervention is not invariant to the response interface. This goes beyond establishing that safety judgments and actions can disagree: it reveals how interface changes affect both the magnitude of safety sensitivity and the relative ordering of safety factors across models.
1. SameFact addresses a different measurement question from prior judgment–action divergence work
Prior work, including ManagerBench, has established that safety recognition does not necessarily translate into safe action selection. However, SameFact investigates a different question: whether the effect of the same controlled factual intervention is preserved across different response interfaces.
The distinction lies in the research question and corresponding estimand. Rather than simply examining agreement or disagreement between independently elicited judgments and actions, SameFact replays each matched safe/unsafe pair across explicit judgment (J), checkpoint candidate admission (D), and open first-action selection (E), holding the task, prior observations, candidate action, identifiers, and non-target facts fixed. We therefore measure how a single safety-relevant factual change affects responses under different interfaces.
This design reveals information that a simple recognition–action divergence analysis cannot provide. Across the 24 model–factor profiles, Spearman agreement decreases from 0.817 (J–D) to 0.674 (D–E) and 0.470 (J–E). More importantly, tie-inclusive pairwise ordering disagreement increases from 18.48% (J–D) to 24.64% (D–E) and 32.61% (J–E). Thus, interface changes do not simply attenuate a common safety signal; they can change which safety factors appear most influential.
The results illustrate this distinction. Gemma exhibits 56.8 pp sensitivity to Effect under judgment, but only 5.3 pp under checkpoint admission, below its 24.0 pp sensitivity to Authorization. GLM exhibits 60.2 pp and 49.2 pp judgment sensitivity to Authorization and Effect, respectively, while the corresponding open-action sensitivities fall to 0.4 pp and 0.0 pp. These are substantive changes in factor-level conclusions, not merely differences in aggregate scores.
SameFact therefore does not simply re-establish recognition–action divergence. It provides a controlled cross-interface measurement framework and demonstrates that the model–factor response profile itself depends on the elicitation interface.
2. Protocol effects are experimentally identifiable
The reviewer raises the question of which differences between judgment and action elicitation account for the observed J–D gap. We address this through two complementary experiments.
First, the original 2×2 factorial experiment independently manipulates checkpoint-style versus natural continuation framing and binary versus open action space within the action interfaces. Checkpoint-style framing increases sensitivity across all six backbones (8.4–29.3 pp), whereas action-space effects and their interactions vary substantially in magnitude and direction across models. This demonstrates that action-side protocol choices systematically shape the measured response to the same factual intervention.
Second, to examine the J–D transition more directly, we conducted a 120-pair Protocol Bridge experiment with four backbones (1,920 trials per model). The Bridge crosses Safety Review versus Task Continuation framing with ALLOW/BLOCK versus candidate/STOP response contracts.
The estimated framing effects (SR−TC), response-contract effects (AB−JSON), and their interactions (pp; 95% CIs) are:
The GLM SR+JSON condition is at floor, which is important for interpreting its large response-contract effect. Nevertheless, the broader result is clear: framing and response contracts have model-dependent effects, and their interactions can be substantial.
These experiments identify specific, experimentally manipulable protocol contributions to the J–D transition. We do not claim a complete additive decomposition of the original gap. Rather, the results demonstrate that the gap is sensitive to concrete protocol components and cannot be interpreted solely as a fixed difference in model capability.