acceptodds
Desk rejected at ICLR 2027

SameFact: The Same Safety Facts Lead to Different Responses Across Different Interfaces

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Safety evaluations often ask whether a model recognizes that an action is unsafe, whereas agent evaluations ask what the model chooses to do. Using safety judgments as evidence about action selection therefore raises a measurement question: does the influence of the same safety-relevant fact persist across response interfaces? We introduce SameFact, a matched-counterfactual benchmark that tests this question directly. SameFact contains 300 safe/unsafe pairs that hold the task, prior observations, candidate action, identifiers, and non-target facts fixed while changing a single state-grounded safety fact. Across six LLM backbones, we measure the effect of this matched intervention through three interfaces at the same candidate-action boundary: explicit safety judgment, checkpoint candidate admission, and open first-action selection. All six backbones show lower aggregate sensitivity under open first-action selection than under judgment, but the change is not a uniform attenuation: across 24 model–factor cells, Spearman agreement falls from 0.817 between judgment and checkpoint admission to 0.470 between judgment and open first-action selection, while pairwise ordering disagreement rises from 18.5% to 32.6%. A follow-up 2×2 first-response experiment shows that a checkpoint-style protocol increases measured sensitivity in all six backbones by 8.4–29.3 percentage points, whereas action-space effects and their interactions with protocol vary in magnitude and direction across models. These results show that the response interface is part of the measured quantity: judgment and action interfaces share safety signal, but do not provide interchangeable measurements of how safety-relevant facts shape model responses. Code is available at https://anonymous.4open.science/r/SAMEFACT-596E.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, Agent Safety, Safety Evaluation, Counterfactual Evaluation, Behavioral Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_quqtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds 300 matched safe/unsafe agent states that differ in one safety fact, and measures how much that fact changes 6 models' responses under 3 interfaces: an ALLOW/BLOCK judgment, a candidate-or-STOP checkpoint, and open next-action selection. Sensitivity drops from judgment to action in every model. A 2×2 follow-up shows that checkpoint framing increases sensitivity for every model, while the effect of the available action set varies by model.

Critical strengths
  1. The matched-pair design keeps the task, candidate action, and other facts fixed while changing one safety-relevant fact, enabling a controlled comparison across response interfaces.
  2. The statistical analysis accounts for repeated measurements within pairs and dependence across construction templates. I also appreciated that the corresponding factorial arms closely reproduce the benchmark's checkpoint and open-action estimates, which is a useful consistency check.
Critical weaknesses
  1. The incremental contribution beyond established judgment-action divergence needs more justification. The claim that safety judgments and actions diverge is already established (e.g., ManagerBench), and the paper doesn't say which interface gives the valid measurement. The matched factual interventions provide a more controlled comparison, but the paper should explain what additional understanding this comparison provides beyond confirming that divergence.
  2. The judgment-action comparison also changes several aspects of the elicitation protocol, whose full instructions are not provided in the main text. The factorial separates framing and action-space effects within the action interfaces, but does not identify which differences between judgment and action account for their gap.
  3. The practical significance of the measured differences remains unclear. The paper evaluates the first response before tool execution and does not establish whether sensitivity under any of the interfaces predicts safety during actual task execution.
Questions for authors

See above. Most concerned about novelty and significance.

3 comments
AuthorsOpenReview ↗

Response to Reviewer quqt

We thank the reviewer for recognizing the value of SameFact's matched-pair design and controlled comparisons. We clarify three aspects of our contribution: how SameFact differs from prior work on judgment–action divergence, what our protocol experiments reveal about the judgment–admission transition, and what these findings imply for agent-safety evaluation.

Our central finding is that the effect of the same controlled safety-relevant factual intervention is not invariant to the response interface. This goes beyond establishing that safety judgments and actions can disagree: it reveals how interface changes affect both the magnitude of safety sensitivity and the relative ordering of safety factors across models.

1. SameFact addresses a different measurement question from prior judgment–action divergence work

Prior work, including ManagerBench, has established that safety recognition does not necessarily translate into safe action selection. However, SameFact investigates a different question: whether the effect of the same controlled factual intervention is preserved across different response interfaces.

The distinction lies in the research question and corresponding estimand. Rather than simply examining agreement or disagreement between independently elicited judgments and actions, SameFact replays each matched safe/unsafe pair across explicit judgment (J), checkpoint candidate admission (D), and open first-action selection (E), holding the task, prior observations, candidate action, identifiers, and non-target facts fixed. We therefore measure how a single safety-relevant factual change affects responses under different interfaces.

This design reveals information that a simple recognition–action divergence analysis cannot provide. Across the 24 model–factor profiles, Spearman agreement decreases from 0.817 (J–D) to 0.674 (D–E) and 0.470 (J–E). More importantly, tie-inclusive pairwise ordering disagreement increases from 18.48% (J–D) to 24.64% (D–E) and 32.61% (J–E). Thus, interface changes do not simply attenuate a common safety signal; they can change which safety factors appear most influential.

The results illustrate this distinction. Gemma exhibits 56.8 pp sensitivity to Effect under judgment, but only 5.3 pp under checkpoint admission, below its 24.0 pp sensitivity to Authorization. GLM exhibits 60.2 pp and 49.2 pp judgment sensitivity to Authorization and Effect, respectively, while the corresponding open-action sensitivities fall to 0.4 pp and 0.0 pp. These are substantive changes in factor-level conclusions, not merely differences in aggregate scores.

SameFact therefore does not simply re-establish recognition–action divergence. It provides a controlled cross-interface measurement framework and demonstrates that the model–factor response profile itself depends on the elicitation interface.

2. Protocol effects are experimentally identifiable

The reviewer raises the question of which differences between judgment and action elicitation account for the observed J–D gap. We address this through two complementary experiments.

First, the original 2×2 factorial experiment independently manipulates checkpoint-style versus natural continuation framing and binary versus open action space within the action interfaces. Checkpoint-style framing increases sensitivity across all six backbones (8.4–29.3 pp), whereas action-space effects and their interactions vary substantially in magnitude and direction across models. This demonstrates that action-side protocol choices systematically shape the measured response to the same factual intervention.

Second, to examine the J–D transition more directly, we conducted a 120-pair Protocol Bridge experiment with four backbones (1,920 trials per model). The Bridge crosses Safety Review versus Task Continuation framing with ALLOW/BLOCK versus candidate/STOP response contracts.

The estimated framing effects (SR−TC), response-contract effects (AB−JSON), and their interactions (pp; 95% CIs) are:

ModelFramingResponse contractInteraction
Qwen+8.3 [5.0, 12.1]−5.4 [−8.8, −2.1]+0.8 [−4.6, 5.8]
Gemma+21.0 [16.5, 25.6]+38.1 [31.5, 44.8]−22.9 [−30.4, −15.4]
GLM+4.4 [1.5, 7.3]+66.0 [60.0, 71.9]+15.4 [9.2, 22.1]
MiniMax+3.1 [−1.9, 8.1]−1.5 [−9.0, 6.0]+5.4 [−5.4, 15.8]

The GLM SR+JSON condition is at floor, which is important for interpreting its large response-contract effect. Nevertheless, the broader result is clear: framing and response contracts have model-dependent effects, and their interactions can be substantial.

These experiments identify specific, experimentally manipulable protocol contributions to the J–D transition. We do not claim a complete additive decomposition of the original gap. Rather, the results demonstrate that the gap is sensitive to concrete protocol components and cannot be interpreted solely as a fixed difference in model capability.

3. The appropriate measurement interface depends on the evaluation target

The reviewer asks which interface provides the valid measurement. We argue that this question requires distinguishing the behavior being measured:

  • J (Judgment): whether a model explicitly recognizes the safety implications of a proposed action.
  • D (Checkpoint admission): whether a model admits or rejects a specified candidate action at a decision checkpoint.
  • E (Open first-action selection): whether a model selects the candidate as its next action when alternatives are available.

These interfaces measure related but distinct aspects of safety-sensitive behavior. There is no reason to expect one interface to serve as an interchangeable measurement of all three.

The practical implication is direct: a model's ability to recognize an unsafe action in an explicit judgment does not establish that the same safety-relevant information will exert an equally strong influence on its action selection. Consequently, judgment-based evaluations should not be treated as direct proxies for agent action-selection safety.

SameFact provides a framework for identifying when these measurements agree, when they diverge, and which safety factors are most affected by the interface transition. This is important for both benchmark design and model comparison, where relying exclusively on aggregate judgment scores can obscure meaningful differences in action-side behavior.

4. The matched-intervention effect persists beyond first responses

To examine whether the findings are restricted to the initial response boundary, we conducted a stateful evaluation using 120 matched pairs, four backbones, and three rollout seeds in an inert simulator (720 trajectories per model).

The safe/unsafe target-commit rates and their differences (pp; 95% CIs) were:

ModelSafe commitUnsafe commitDifference
Qwen100.0%90.6%+9.4 [5.8, 13.3]
Gemma97.5%68.9%+28.6 [21.9, 35.6]
GLM94.4%54.7%+39.7 [32.8, 46.7]
MiniMax96.9%64.4%+32.5 [26.4, 38.9]

Approval requests were also more frequent in unsafe than safe trajectories across all four models (Qwen: 72.5% vs. 27.2%; Gemma: 46.4% vs. 8.9%; GLM: 50.8% vs. 8.9%; MiniMax: 42.2% vs. 2.5%). Strict response validity was at least 94.2% in every model×state cell.

These results demonstrate that the matched factual intervention continues to influence subsequent commitments, rather than appearing only in isolated first responses.

Taken together, SameFact contributes a controlled cross-interface measurement framework, experimental identification of model-dependent protocol effects, and evidence that the intervention's behavioral influence persists into subsequent commitments. Its central contribution is to show that the measured influence of safety-relevant information depends jointly on the model, the factual intervention, and the response interface.

AuthorsOpenReview ↗

Revisiting the Contributions and Implications of SameFact

Dear Reviewer,

Thank you again for your detailed and constructive feedback. We appreciate your questions regarding the novelty of SameFact beyond existing judgment–action divergence studies, the confounding effects of different elicitation protocols, and the practical significance of our findings.

We have further strengthened our responses and conducted additional experiments to address these concerns.

In particular, we would like to highlight three aspects of our additional work:

1. A more controlled measurement perspective

We clarified how SameFact differs from prior work such as ManagerBench. Rather than simply measuring whether safety judgments agree with action choices, SameFact examines whether the influence of the same controlled safety-relevant factual intervention is preserved across different response interfaces.

Our results show that interface changes can affect not only the magnitude of safety sensitivity but also the relative ordering of safety factors across models. This provides a different perspective on how safety-relevant information influences model behavior.

2. Additional identification of protocol effects

To better understand the Judgment–Admission transition, we conducted a 120-pair Protocol Bridge experiment that independently examines framing and response-contract effects. Together with our original factorial experiment, these results provide further evidence that concrete protocol choices contribute to the observed differences, with substantial model-dependent effects and interactions.

3. Evidence beyond isolated first responses

We also conducted an additional stateful evaluation in an inert simulator, covering four backbones and 120 matched pairs. The results show that the influence of matched safety-relevant factual changes persists into subsequent target commitments, extending the evidence beyond the initial response boundary.

We hope these additional analyses help clarify the methodological contribution of SameFact and its implications for safety evaluation and model comparison.

We would sincerely appreciate it if you could take another look at our responses when you have a chance. As the discussion deadline is approaching, we are concerned that there may be limited time for us to address any further questions or concerns that emerge later in the discussion. If there are still aspects of the contribution or experimental evidence that you find insufficiently clear, we would be very grateful if you could share your thoughts at your earliest convenience, so that we have an opportunity to respond before the discussion closes.

Thank you very much for your time and consideration. We truly appreciate the opportunity to discuss these issues with you.

Best regards,
The Authors

Reviewer_cfenOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors construct a benchmark of agentic scenarios with facts of which one is safety-relevant, and they create contrastive pairs where they change the safety fact from safe to unsafe, so they can directly compare the model's behaviors. Across six models, they find that the models are much better at identifying which actions are safe or unsafe than only taking safe actions. Adding a checkpoint consistently improves the ability to choose safe actions, with magnitude varying between models.

Critical strengths
  • I like the idea of having an eval to clearly test how agents respond to the presence of safety relevant facts
  • Having clean contrast pairs is extremely helpful
  • I liked the 2x2 factorial set up separating protocol framing from action space
Critical weaknesses
  • Reasoning is disabled, which is highly unrealistic for how agents would be used in practice
  • The eval is single turn and static, while agent behavior only really matters in more complex multi-turn settings
  • The evals seem extremely toy and artificial, eg destination "external_server", which will be an issue with capable models that are eval aware
Questions for authors

How could the level of realism be improved here? How do results change with CoT enabled?

4 comments
AuthorsOpenReview ↗

Response to Reviewer cfen

Response to Reviewer cfen

We thank the reviewer for raising questions about reasoning, interaction depth, and scenario realism. We conducted additional experiments to test whether SameFact's findings depend on reasoning-disabled inference, isolated first responses, or explicit sandbox disclosure.

1. Native reasoning changes magnitude but preserves interface ordering

We enabled native reasoning on all 300 pairs for Qwen3.8-27B and GLM-4.7-Flash. All-trial sensitivities (ΔJ/ΔD/ΔE-next, pp) were:

ModelWithout reasoningWith reasoning
Qwen72.7/68.3/43.061.8/53.0/44.5
GLM59.0/39.3/2.855.6/29.8/24.3

J–E gaps narrowed from 29.7 to 17.3 pp (Qwen) and 56.2 to 31.3 pp (GLM); D–E gaps narrowed from 25.3 to 8.5 and 36.5 to 5.5 pp, respectively. Nevertheless, ΔJ > ΔD > ΔE-next persisted in both models, including valid-output estimates. After retries, 40 Qwen and 56 GLM trials remained invalid (including 50 GLM truncations). Thus, reasoning changes effect magnitudes but does not eliminate the qualitative interface ordering; the pattern is therefore not solely an artifact of disabling reasoning.

2. Stateful evaluation extends beyond first responses

We evaluated 120 pairs (10 per factor×context stratum), four backbones, and three rollout seeds in an inert simulator (720 trajectories/model).

Safe/unsafe target-commit rates and differences (pp; 95% CI) were:

ModelSafeUnsafeDifference
Qwen100.0%90.6%+9.4 [5.8, 13.3]
Gemma97.5%68.9%+28.6 [21.9, 35.6]
GLM94.4%54.7%+39.7 [32.8, 46.7]
MiniMax96.9%64.4%+32.5 [26.4, 38.9]

Approval requests were also more frequent in unsafe than safe trajectories: Qwen 72.5% vs. 27.2%; Gemma 46.4% vs. 8.9%; GLM 50.8% vs. 8.9%; MiniMax 42.2% vs. 2.5%. Invalid/protocol-error outputs counted as non-commits; strict validity was at least 94.2% in every model×state cell. The matched-intervention effect therefore persists into subsequent commitments across all four backbones, showing the effect is not confined to an isolated first-response boundary.

3. Scenario grounding and sandbox-disclosure ablation

SameFact's 300 pairs span Repository, Data, and Access Control, covering 43 template families, 51 context×template groups, and 39 scenario blocks, including data export, medical fields, deployment, and production versus read-only privileges. external_server is a schematic identifier; scenario semantics are defined by the underlying data, permissions, operations, and constraints. All pairs passed programmatic checks; 108/120 audited pairs passed all five dimensions and 109/120 passed unsafe-state entailment.

To test whether explicit sandbox disclosure drives sensitivity, we changed only this disclosure in a 120-pair ablation:

ModelWith disclosureWithout disclosurePaired change
Qwen14.72 [10.28, 19.44]28.89 [22.78, 35.28]+14.17 [10.00, 18.33]
GLM29.44 [23.06, 35.83]36.39 [30.00, 43.06]+6.94 [1.39, 12.22]
MiniMax32.78 [26.11, 39.72]36.11 [29.72, 42.78]+3.33 [−1.39, 8.33]

Sensitivity: safe−unsafe commitment, pp; 95% CI.

Cluster-adjusted CIs for Qwen/GLM excluded zero across 38 template families ([8.33, 20.30]/[2.30, 12.27]) and 46 context×template groups ([9.12, 19.30]/[2.19, 12.01]); MiniMax's CIs included zero. Across 38 leave-one-family-out analyses, effects remained positive for Qwen (+11.47 to +15.17 pp) and GLM (+6.17 to +7.99 pp).

Removing explicit disclosure increased sensitivity in all three point estimates, significantly for Qwen and GLM. Thus, explicit sandbox disclosure does not account for the observed safe–unsafe distinction, although it affects its magnitude, reinforcing that evaluation protocols are part of the measurement rather than nuisance variables.

4. Overall implications

Across native reasoning, stateful interaction, and altered disclosure, the qualitative interface ordering or matched-intervention effect persists, with model- and protocol-dependent magnitudes. These results show that SameFact's findings are not confined to reasoning-disabled inference, isolated first responses, or explicit sandbox disclosure. They support evaluating safety-sensitive behavior across multiple response interfaces rather than relying on a single evaluation setting.

AuthorsOpenReview ↗

Follow-up on Our Additional Experiments for SameFact

Dear Reviewer,

Thank you again for your thoughtful feedback and for raising important questions about the realism and practical relevance of our evaluation.

We have made substantial efforts to investigate these concerns through additional experiments, rather than relying solely on explanations:

  1. Native reasoning: We enabled native reasoning for Qwen3.8-27B and GLM-4.7-Flash across all 300 matched pairs. Although reasoning changes the magnitude of the measured effects, the qualitative interface ordering (ΔJ > ΔD > ΔE-next) remains consistent in both models.

  2. Stateful evaluation: We extended the evaluation beyond isolated first responses by conducting multi-step rollouts in an inert simulator, covering 120 matched pairs, four backbones, and three rollout seeds. The matched safe–unsafe factual contrast remains visible in subsequent target commitments across all four models.

  3. Scenario realism and evaluation awareness: We further clarified the scenario construction, covering 43 template families across Repository, Data, and Access Control. We also conducted a sandbox-disclosure ablation to examine whether explicit disclosure drives the observed sensitivity. The results show that removing this disclosure increases sensitivity in the point estimates for all three tested models, with statistically significant changes for Qwen and GLM.

We hope these additional results help clarify that our findings are not simply artifacts of reasoning-disabled inference, isolated first responses, or explicit sandbox disclosure.

We would really appreciate it if you could revisit our responses when time permits. Since the discussion deadline is approaching, we are a little concerned that there may not be sufficient time to address further questions or clarify remaining concerns if they arise too close to the deadline. If there are any aspects you would still like us to elaborate on, we would be very grateful if you could let us know at your earliest convenience.

We would be more than happy to continue the discussion and provide any additional clarification needed.

Thank you again for your time and consideration!

Best regards,
The Authors

Thanks for the additional experiments. They're useful, but they do also show that the headline gap shrinks substantially with reasoning enabled. And my concern about realism and evaluation awareness is not addressed. I am keeping my score at a 2.

Thank you for your further comments. We respectfully disagree that the concerns regarding reasoning, realism, and evaluation awareness remain unaddressed. We would like to clarify three important points.

1. Native reasoning does not eliminate interface-dependent sensitivity.

We acknowledge that native reasoning narrows the overall J–E gap. However, this does not eliminate the central finding of SameFact. More importantly, examining only the J–E gap overlooks how the sensitivity profile changes across interfaces.

Under native reasoning, the J–D gap actually increases in both models: from 4.4 to 8.8 pp for Qwen and from 19.7 to 25.8 pp for GLM. Meanwhile, the ordering ΔJ > ΔD > ΔE_next remains unchanged in both models.

Thus, reasoning changes the magnitude and distribution of sensitivity across interfaces, rather than eliminating interface-dependent differences. This is precisely why our contribution goes beyond reporting a single aggregate gap: we investigate whether the influence of the same safety-relevant factual intervention is preserved across response interfaces. Across the 24 model–factor cells in the main evaluation, J–E Spearman agreement is only 0.470, compared with 0.817 for J–D, and pairwise ordering disagreement increases from 18.48% to 32.61%.

We therefore believe that the reasoning-enabled results refine, rather than invalidate, our central measurement finding.

2. Our experimental interfaces are deliberate methodological choices, not assumptions about real-agent architecture.

We believe the realism concern conflates experimental interface design with production agent architecture. SameFact does not assume that real-world agents expose three separate J/D/E modules or APIs. Rather, these are deliberately constructed response protocols applied to the same underlying task, observations, and candidate action, allowing us to isolate how the same safety-relevant factual intervention affects different response modes.

This separation is essential to the diagnostic purpose of our benchmark. A purely end-to-end execution evaluation may reveal that an agent commits an unsafe action, but it cannot by itself distinguish whether the safety-relevant information fails to influence judgment, candidate admission, or subsequent action selection. Our controlled interfaces make these differences measurable.

Furthermore, the scenarios are grounded in concrete operational constraints involving repository operations, data handling, and access control, including data export, medical information, deployment, and production versus read-only permissions. Identifiers such as external_server are schematic; the underlying operations, permissions, and safety constraints are the actual objects of evaluation.

We have also extended the evaluation beyond first responses: stateful rollouts across four backbones show safe–unsafe differences of 9.4–39.7 pp in subsequent target commitments. We additionally tested expanded alternative-action sets and protocol components to examine how action availability and framing affect the results.

We do not claim to reproduce a complete production-agent environment. However, a controlled behavioral benchmark does not need to replicate an entire production architecture to provide meaningful and diagnostic evidence about a specific safety mechanism. We believe this distinction is important when assessing the methodological validity of SameFact.

3. Evaluation awareness has been empirically investigated.

We specifically tested whether explicit sandbox disclosure drives the observed sensitivity. Removing this disclosure increased sensitivity by 14.17 pp for Qwen (95% CI [10.00, 18.33]) and 6.94 pp for GLM ([1.39, 12.22]), with both effects statistically significant. MiniMax also showed a positive point estimate (+3.33 pp), although its confidence interval included zero.

We acknowledge that this ablation cannot rule out every possible form of evaluation awareness. However, it directly tests an important potential source of evaluation-induced behavior and provides evidence that the observed sensitivity is not simply an artifact of explicit sandbox disclosure. Therefore, we believe it is too strong to characterize this concern as entirely unaddressed.

In summary, SameFact makes a measurement claim, not a claim that all deployed agents possess three independent interfaces or that our simulator fully reproduces production deployment. Our central finding is that the influence of the same safety-relevant fact depends on the model and response protocol, and that judgment and action interfaces are not interchangeable measurements. The additional reasoning, stateful, protocol, and disclosure experiments provide further empirical support for this claim.

We respectfully ask that the assessment take these substantive experimental additions and the intended scope of our methodological contribution into account.

Reviewer_7ZkpOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SameFact, a matched-counterfactual benchmark containing 300 safe/unsafe pairs across four safety factors and three agent contexts. It evaluates six LLM backbones by measuring their sensitivity to the same safety-relevant factual changes across three interfaces.

Critical strengths
  1. SameFact constructs matched safe/unsafe pairs by holding the task, prior observations, and candidate tool calls fixed while changing only one safety-relevant fact, providing a well-controlled setting for cross-interface comparisons.
  2. The experiments reveal differences between safety judgment and action selection across models. Some models recognize safety risks in explicit judgments but still frequently select unsafe candidate actions, offering useful insights into agent safety.
  3. The 2×2 factorial experiment further examines framing and choice-set effects, showing a consistent direction of framing effects across six models, while choice-set effects and their interactions vary substantially across models.
Critical weaknesses
  1. Open First-Action Selection always provides a complete candidate tool call. Among 21,600 Open responses, only 111 select alternative actions, with none observed for Qwen or DeepSeek. The resulting decisions largely reduce to a binary candidate/STOP choice, making it difficult to distinguish genuine action competition from protocol-induced effects.
  2. Judgment uses four explicit safety-review prompts, whereas D/E primarily instruct models to continue the task. Consequently, the J→D gap conflates safety guidance, decision objectives, and output formats. Gemma and MiniMax exhibit drops of 26.9 and 24.6 percentage points at this stage, respectively, but the 2×2 experiment only decomposes action-side differences, leaving this substantial gap unexplained.
  3. The four safety factors differ in their operationalization. Constraint typically involves explicit rules, whereas Effect requires reasoning about object properties and downstream consequences. Therefore, factor-specific sensitivity may also reflect differences in rule explicitness and reasoning difficulty, making direct attribution to safety factors uncertain.
  4. Among 120 human-audited pairs, only 105 receive joint PASS across all five dimensions, and 107/120 pass the unsafe-state entailment criterion. The remaining uncertain cases have not been adjudicated. Since the core measurements rely on valid safety-semantic flips, the influence of these cases on the main findings remains unclear.
Questions for authors

see weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7Zkp

Response to Reviewer 7Zkp

We thank the reviewer for engaging with SameFact. Additional experiments examine action-space composition and protocol effects across four backbones.

1. Alternative actions in OPEN

The 0.514% alternative-action rate indicates that the original OPEN interface primarily elicits candidate-anchored first-action selection, rather than broad action-space exploration. We therefore extended the experiment to 120 pairs (10 per factor×context stratum), three models, and three explicit alternatives (stage_for_review, request_approval, defer_operation); tool execution was disabled (960 trials/model).

In unsafe states, BASE→MULTI alternative selection rose from 0.0% to 47.9% for Qwen, 5.4% to 60.8% for GLM, and 0.4% to 73.3% for MiniMax. Safe-state MULTI rates were 0.8%, 17.5%, and 15.8%, respectively. Candidate-commit sensitivity (safe−unsafe, pp) was 47.1→50.0 for Qwen, 6.2→43.3 for GLM, and 35.0→57.5 for MiniMax. Difference-in-differences were +2.9 [−1.2, 7.5], +37.1 [29.2, 45.0], and +22.5 [15.4, 29.6] pp, respectively.

Thus, choice-set expansion redistributes first actions, but its effect on candidate sensitivity is model-dependent (Qwen's change is not distinguishable from zero). The two experiments capture complementary aspects of first-action selection.

2. Protocol components and sandbox disclosure

Across four Judgment probe families, J–D correlations were 0.769–0.808, versus 0.439–0.507 for J–E. The 2×2 Protocol Bridge used 120 pairs (10 per factor×context stratum) and four backbones (1,920 trials/model); MiniMax protocol-invalid outputs counted as non-commitments.

Sensitivity (safe−unsafe candidate commitment, pp):

ModelSR+ABSR+JSONTC+ABTC+JSON
Qwen82.587.573.879.6
Gemma60.834.251.31.7
GLM73.80.061.73.3
MiniMax-M357.155.851.355.4

Paired framing (SR−TC), response-contract (AB−candidate/STOP), and interaction effects (pp; 95% CI) were: Qwen +8.3 [5.0, 12.1], −5.4 [−8.8, −2.1], +0.8 [−4.6, 5.8]; Gemma +21.0 [16.5, 25.6], +38.1 [31.5, 44.8], −22.9 [−30.4, −15.4]; GLM +4.4 [1.5, 7.3], +66.0 [60.0, 71.9], +15.4 [9.2, 22.1]; MiniMax +3.1 [−1.9, 8.1], −1.5 [−9.0, 6.0], +5.4 [−5.4, 15.8].

GLM's SR+JSON arm is at floor. The interaction is a paired difference-in-differences, not a full additive decomposition. These effects identify tested protocol contributors to the J–D transition. MiniMax's main factorial framing effect was +8.42 pp [5.58, 11.25], directionally consistent with the Bridge estimate (+3.12 [−1.88, 8.12]); excluding 12 pairs outside the frozen cohort, the Bridge effect was +5.74 [0.50, 11.04] pp (108 pairs).

Removing explicit sandbox disclosure in a 120-pair ablation increased sensitivity by +14.17 pp for Qwen, +6.94 for GLM, and +3.33 [−1.39, 8.33] for MiniMax. Cluster-adjusted CIs exclude zero for Qwen/GLM but include zero for MiniMax. Across 38 leave-one-family-out analyses, no sign reversal occurred. The magnitude is model-specific, supporting protocol-dependent measurement.

3. The four safety factors have distinct operationalizations

Our primary estimand is the within-factor safe–unsafe contrast, not an intrinsic ranking of factor difficulty. Within each matched pair, operationalization remains fixed while one safety-relevant fact changes. Thus, operationalization differences do not invalidate within-factor comparisons.

Authorization concerns entitlement; Constraint, explicit rules; Effect, object properties or downstream consequences; Scope/Consent, audience, purpose, or consent boundaries. Constraint-minus-Effect contrasts ranged from −0.2 to +38.0 pp for ΔJ, +6.2 to +64.0 for ΔD, and +4.4 to +51.6 for ΔE-next (some Qwen intervals include zero).

Within Constraint, ΔE-next spans 4.4–79.6 pp across models, while ΔJ spans 83.7–94.5 pp (DeepSeek: 88.3 vs. 5.8; GLM: 87.2 vs. 4.4). This cross-interface variation within the same factor shows that rule explicitness alone cannot explain the observed patterns.

4. Human-audit robustness

We re-reviewed all 15 initially unresolved cases; three additional pairs passed all five dimensions, yielding 108 strictly human-confirmed pairs.

On the original 105 jointly accepted pairs (100,000 stratified bootstraps), all 18 model×interface sensitivity CIs remained above zero. All six models retained ΔJ > ΔD > ΔE-next, with ΔJ−ΔD and ΔJ−ΔE-next CIs excluding zero. ΔD−ΔE-next was positive for five models; MiniMax's CI crossed zero (−0.19, 10.55 pp). The expanded 108-pair set yields the same qualitative conclusions.

Together, these results support SameFact as a controlled measure of how factual interventions and protocols shape safety-sensitive behavior.

AuthorsOpenReview ↗

Further Discussion on SameFact (Submission 20736)

Dear Reviewer,

Thank you again for your careful reading of our paper and for your insightful feedback. We particularly appreciate your comments on the action-space composition, protocol effects, operationalization of safety factors, and human-audit robustness.

Following your suggestions, we conducted several additional experiments and analyses to address these concerns:

  • Action-space composition: We introduced explicit alternative actions in the OPEN interface to investigate how expanding the choice set affects candidate-action sensitivity.
  • Protocol effects: We conducted an additional Protocol Bridge experiment to better identify the contributions of framing and response contracts to the Judgment–Admission transition.
  • Safety-factor operationalization: We added within-factor analyses to clarify why differences in operationalization do not invalidate our matched-counterfactual comparisons.
  • Human-audit robustness: We re-reviewed all initially unresolved cases, increasing the number of strictly human-confirmed pairs from 105 to 108, and verified that our main qualitative findings remain unchanged.

We hope these additional experiments and clarifications have addressed your concerns and further strengthened the evidence supporting SameFact.

We would greatly appreciate it if you could take another look at our responses when you have a chance. As the discussion deadline is approaching, we are concerned that there may be limited time to address any additional questions that arise closer to the deadline. If any concerns remain, we would sincerely appreciate hearing from you at your earliest convenience, so that we can provide further clarification before the discussion closes.

Thank you very much again for your time and thoughtful feedback!

Best regards,
The Authors

Thank you for your discussion, I'll keep my positive score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Summary

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their valuable feedback. As the discussion period draws to a close, we would like to briefly summarize our contributions and the additional work completed during the discussion.

Paper Summary We appreciate the reviewers' recognition of SameFact's matched-counterfactual design (Reviewers 7Zkp and quqt), the clean safe/unsafe contrast pairs (Reviewer cfen), and the rigorous statistical analysis accounting for repeated measurements (Reviewer quqt). SameFact introduces a 300-pair matched benchmark that holds the task, history, and candidate action fixed while changing a single safety fact. It measures the effect of this intervention across three interfaces: explicit safety judgment (J), checkpoint candidate admission (D), and open first-action selection (E), demonstrating that judgment and action interfaces are not interchangeable measurements of safety sensitivity.

Discussion Period Summary During the discussion period, we responded to all questions raised by the three reviewers, provided extensive additional experiments and analyses, and incorporated these revisions into our manuscript.

  • Choice-set expansion and OPEN interface behavior (Reviewer 7Zkp). We extended the OPEN experiment to 120 pairs with explicit alternative actions. In unsafe states, alternative selection rose to 47.9%–73.3% across models, confirming that the original OPEN interface primarily elicits candidate-anchored choices rather than broad action exploration.
  • Protocol confounds and the J→D transition (Reviewers 7Zkp and quqt). We introduced a 120-pair Protocol Bridge experiment, crossing framing and response contracts. Results reveal model-dependent protocol effects (e.g., GLM's response-contract effect was +66.0 pp), experimentally identifying protocol contributors to the judgment–admission gap.
  • Impact of native reasoning and cross-interface ordering (Reviewer cfen). Clarification on overlooked findings: While Reviewer cfen noted that the J–E gap shrinks under reasoning, examining only the two endpoints overlooks the intermediate D interface. Under native reasoning, the J–D gap actually increases (Qwen: 4.4 to 8.8 pp; GLM: 19.7 to 25.8 pp), and the strict ordering ΔJ > ΔD > ΔE-next remains unchanged. Furthermore, our sandbox-disclosure ablation demonstrated that removing explicit disclosure increased sensitivity (e.g., +14.17 pp for Qwen), directly countering the assumption that evaluation awareness drives the observed effects.
  • Multi-turn stateful evaluation (Reviewers cfen and quqt). We conducted multi-step rollouts in an inert simulator (4 backbones, 3 seeds). The matched intervention persisted into subsequent target commitments (safe–unsafe differences of +9.4 to +39.7 pp), proving the effect extends beyond isolated first responses.
  • Safety factor operationalization and human-audit robustness (Reviewer 7Zkp). We clarified that our within-factor comparisons fix operationalization. Additionally, we re-audited unresolved cases, increasing strictly human-confirmed pairs to 108. 100,000 stratified bootstraps confirmed all 18 model×interface sensitivity CIs remained above zero.
  • Novelty beyond judgment–action divergence (Reviewer quqt). Clarification on mischaracterization: We respectfully clarified that the reviewer's assessment conflates the well-known phenomenon of judgment–action divergence with our distinct measurement problem. SameFact is not merely re-demonstrating divergence; it introduces a matched-counterfactual framework proving that the interface alters factor-level conclusions (e.g., Gemma's Effect sensitivity drops from 56.8 pp under J to 5.3 pp under D). This cross-interface reversal cannot be captured by simply observing that judgments and actions disagree.

Detailed results and explanations are available in the corresponding review threads and the updated manuscript. We sincerely appreciate your time and consideration.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.