acceptodds
Under review as a conference paper at ICLR 2027

The Sensing Gap: Counterfactual Trajectory Forks for Contract-Relative Pre-Commitment Sensing

Abstract

Single-world agent evaluations can reward a write even when its trace contains no current supporting evidence. We introduce counterfactual trajectory forks, an evaluation unit that replays a represented pre-commitment prefix after mutating one contract dependency. Each fork provides an executable read that separates the resulting continuations. Mechanical witnesses verify prefix identity, mutation provenance, candidate-precondition divergence, separator executability, and benchmark-state effects. We then measure separator acquisition (NSR), commitment timing (PCR), and query cost. In a controlled retail study, six model–scaffold configurations achieve 72.7%–95.5% static success, yet only 9.1%–31.8% grounded static success. On 19 selected stale-evidence forks, removing dependency-bearing read turns changes NSR by +0.656 [0.500, 0.738], PCR by -0.656 [-0.748, -0.500], and query cost by +0.754 [0.503, 0.975]. This intervention identifies a rendered prompt-package effect, not stale semantics alone. In 15 costly synthetic service-operations forks, all 90 repeated invalid-world continuations perform simulator-accepted, benchmark-state-changing writes before separation. Together, these results show that outcome scoring can conceal large differences in evidence acquisition and commitment timing. The method yields reproducible contract-relative measurements, not estimates of empirical ATIS, correctness, safety, or prevalence.

Primary area:
datasets and benchmarks
Keywords:
agent evaluation, tool-using language-model agents, counterfactual trajectory forks, pre-commitment sensing, stale information, temporal validity, benchmark construction

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_m2cAOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies an interesting problem, when an agent completes a task, is that success the correct metric? did the agent check the relevant evidence and execute accordingly, or does it just happen that action is the correct one?
Authors took some agentic trajectories and made two versions, where one fact changed, and they evaluated if the agent checks that fact before acting. The experimental results shed some lights, and not too surprisingly, agents show significant performance drop when considering if they checked the current or changed state.

Critical strengths
  • The research questions raised in the study are valid. Focusing just on the task success doesn't tell the full story of agent performance.
  • The way authors created the data to study this problem is simple and clever. They keep the history and with one changed fact, two different worlds are obtained to assess if the agent check that relevant fact.
  • They have some interesting findings. In many cases, the agent don't check the state, and though often harmless, in risky cases agent performs real state-changing write before checking.
Critical weaknesses

Writing: The paper's language is too complex and dense, it's very hard to follow (I can't tell how much this is due to AI writing). There are a lot of jargons and dense terminology. The paper can be much simplified to convey the same message.

All the appendix provided seem to be just agent logs. This is not the right content for a paper.

Authors introduced many metrics, NSR, PCR, SQP, SFT, JointATIS, etc, 7 primary metrics and a half dozen others (some are in Appendix). Too many results are undermining the paper's real message, that is, outcome success looks high, but evidence grounded success is low. Readers can't follow all the results. And not all of those are necessary.

The experimental setup and results left some open questions. The absence of human validation of the contracts is a big gap. AI audits are not enough, and even the paper own AI auditors showed the risky cases having major issues. The validity of the results is thin, partly due to the limited environment used.

Questions for authors

About the experimental setup, the agents used are relatively small models, do we know if this is a real agent problem? With the changed agent trajectories, is there a way to measure using frontier models to help us understand whether it's an agent or a small model problem?

Reviewer_gjv5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes counterfactual trajectory forks for evaluating whether an agent obtains current evidence before a contract-designated write. A represented pre-write history is held fixed while one dependency is changed, creating worlds in which the candidate action is respectively valid and invalid. Executable witnesses check the shared prefix, mutation, action-precondition divergence, and a separating read. The primary measurements concern separator acquisition, write timing, and query cost, rather than safety or task correctness. Experiments cover six local model-scaffold configurations, a retail panel, external public trajectories, and a selected costly-action service-operations subgroup.

Critical strengths

S1 A precise evaluation target. The distinction between successful outcomes and evidence-supported action is important. Mechanical witnesses make the paired intervention more inspectable than an unconstrained judgment of whether an agent should have checked something. S2 Careful claim calibration. The abstract and limitations correctly identify the stale-read deletion result as a rendered prompt-package effect, not an isolated effect of stale semantics. The underidentified semantic experiment and the problems uncovered by the audits are reported rather than hidden. S3 Useful diagnostic breadth. The paper includes irrelevant-change and no-risk controls, external-source replays, provenance diagnostics, a generic TOCTOU construction baseline, and accepted state-changing actions in a second synthetic domain. These additions help separate several otherwise conflated failure modes.

Critical weaknesses

W1 policy validity remains the main obstacle. Appendix E reports that all 48 canonical risky retail invocations are rejected without benchmark-state changes. The stricter model-assisted audit marks every risky unit as major, chiefly because the strict policy excludes state-preserving rejected invocations; dependency completeness also remains disputed. These are not failures of mutation validity, and the paper acknowledges the distinction. Nevertheless, mechanical validity alone does not validate the normative commitment boundary. Without independent domain judgments or a justified cost model, the strict behavioral scores cannot establish that an agent made a practically undesirable decision. The costly subgroup is helpful, but its 15 selected forks from eight clusters do not resolve the validity of the main panel. W2 The distinctive value over simpler temporal tests is not yet quantified. Appendix A.9 shows that all four construction-source variants produce 232 of 232 valid forks, while generic TOCTOU construction also achieves 100 percent risky validity in this status-heavy domain. Contract guidance supplies additional mutation coverage and controls, which is a legitimate advantage, but budgets differ. A matched-budget comparison of diagnostic coverage, discovered failure types, and ranking stability would more directly justify the proposed construction procedure. W3 Evidence for the motivating semantic mechanism is limited. The large effect on 19 selected stale forks is explicitly a prompt-package replacement effect. On the 23-fork semantic panel, the direct estimate is approximately 0.007 with only two discordant clusters out of nine and is underidentified. The external replay results improve sensing while increasing query cost and reducing SQP. These results support a measurement framework, but not a robust causal account of stale-evidence reliance or an intervention that reliably improves overall agent behavior. W4 Generalization and effective sample size need sharper emphasis. Forks are selected from successful stored trajectories and much of the evidence comes from authored environments. Ninety repeated costly-action runs are not ninety independent task samples. Cluster-level uncertainty is appropriate, but more independent tasks, on-policy rollouts, and domains with naturally consequential writes are needed before drawing broader conclusions.

Questions for authors

Q1 Can independent, outcome-blind domain reviewers validate dependency completeness and the acceptable W1 behavior set? What fraction of risky units remains eligible under a policy that permits harmless rejected calls? Q2 At equal construction and replay budgets, what diagnostic information does contract guidance add over generic TOCTOU tests? Does the additional coverage change model rankings or reveal distinct failures? Q3 Can the costly-action experiment be repeated on a prospectively sampled task set with more independent clusters, reporting accepted writes, actual state changes, recovery, sensing cost, and task utility separately? Q4 What is the minimal publicly accessible artifact needed to reproduce one fork end to end, including its contract, original prefix, mutation, separator, replay, and frozen scorer?

Reviewer_RWaFOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that outcome-based agent benchmarks cannot tell whether an agent checked the relevant current state before taking a consequential action, or just happened to land in a world where the action was valid. To measure this, the authors propose "counterfactual trajectory forks." They take a recorded trajectory, cut it just before a write action and create two worlds with identical visible history that differ in one fact. A specific read would tell the two worlds apart. They then measure how often the agent makes that check before writing, how often it writes first, and how many reads it makes. On a small synthetic retail environment with six 7-14B open models, task success is 72.7-95.5%, but only 9.1-31.8% of runs checked the relevant state before acting

Critical strengths

The paper raises a crucial issue. Outcome success does not tell you whether an agent acted on current information, and the illustrative example with the cancel-order example is reasonable.

Critical weaknesses

Overall, I think the paper may have some good ideas. But in it's current form of writing it is very hard to parse the contributions of the paper.

  • The writing makes the paper very hard to read. I read the abstract and the introduction several times and still could not tell what problem the paper solves clearly (Please rewrite the abstract and the introduction). The core message "outcome scoring hides whether an agent checked current state before acting" is not clear and the abstract should first give a reasonable context about the problem and the setup. The key terms like ATIS are not defined.

  • The contribution seems incremental relative to prior work. The related work section says the paper does not claim novelty for stale-state failures, rechecking, authorization or abstention (TOCTOU-Bench, Atomicity for Agents, ContractBench, AgentAbstain, etc.), and that the contribution is the combination. The paper also shows that task and contract information alone build the same 232/232 forks as recorded trajectories, which undermines the case that trajectories are central to the method, despite the title.

  • The experiments are very small and synthetic, and the ground truth isn't validated. Evaluation is just two toy environments with hand-written rules for what an agent should check, 48 test pairs, a key history experiment on 19 selected cases and only 7-14B models with a basic agent loop.

  • The rules for what counts as grounded were audited only by Codex agents. There was no human or domain-expert review.

Questions for authors
  • Can you state what problem the paper solves and what the main contribution is (benchmark, method or finding)?

  • Since contract-only construction gives the same forks as recorded trajectories, what do trajectories add in practice beyond recording which reads already happened?

  • How sensitive is the grounded-success gap to the hand-written contracts? Did any humans check whether the required reads are actually necessary?

  • Do the findings hold for stronger models and on a standard benchmark such as τ2\tau^2-Bench beyond retail?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.