acceptodds
Under review as a conference paper at ICLR 2027

CoBe: A Benchmark For Conversational Counterfactual Text Editing

Abstract

Counterfactual text editing is the ability to revise a given text in light of a hypothetical change while preserving facts not causally affected by that change. When asked what would have happened if a doctor had not recommended surgery, the model output should retain the patient's medical history and modify only the consequences of the decision. This requires causal understanding: a model must determine, without being explicitly instructed to, what events are affected by the hypothetical change and what should remain fixed. This skill is central to human reasoning, especially in safety-critical domains like healthcare or law. By contrast, the task of associational text generation merely generates text statistically associated with the altered condition. In the medical example, an associational response might make the patient's symptoms less severe because milder cases correlate with no surgery. Such responses produce a plausible scenario, but not the counterfactual version of the given one. Prior counterfactual-text datasets do not clearly separate these modes of reasoning. We introduce the Counterfactual Benchmark (CoBe), a testbed for models' zero-shot ability to counterfactually edit a given textual scenario when prompted to do so in a conversationally natural way. CoBe contains 2000 questions spanning healthcare, science and engineering, business, and everyday scenarios. We test a suite of frontier and smaller models, with frontier systems achieving an average score of 58.05%. Our findings indicate that models frequently default to associational generation rather than preserving the causal structure of the original scenario. Moreover, performance does not scale consistently with either model size or output token length, suggesting that the failure may reflect model training objectives rather than insufficient model capacity.

Primary area:
causal reasoning
Keywords:
counterfactual reasoning, causality, counterfactual text editing, counterfactual text generation, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XYvaOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper makes a clear distinction between counterfactual text editing and associational text generation, and constructs a benchmark to test whether LLMs can perform causally valid counterfactual editing. It introduces CoBe, which contains 2,000 samples, each consisting of a short scenario and conversational prompts.The data covers seven domains and six types of causal graph structures. Experiments show that even frontier models often fail to produce causally valid counterfactual edits.

Critical strengths
  • This paper makes a necessary distinction between counterfactual text editing and associational generation, and the task is properly grounded in Pearl’s causal hierarchy.
  • Well conducted experiments, with a comparative study of nine LLMs and detailed analyses across evaluation criteria, graph types, and query phrasings.
  • The dataset covers seven domains and six graph types. Unlike prior work that mostly only focus on simple linear timelines, it actually tests complexer structures like colliders, forks, and diamonds.
Critical weaknesses
  • The paper is a bit unbalanced. The abstract and introduction take 3.5 pages because the intro includes too much background, like the Pearl Causal Hierarchy. This should really be moved to a separate section.
  • The comparison between the LLM-Judge and human annotators doesn't seem fair. Humans get the original context, query, and LLM output, but not the evaluation criteria (App. D.4.1). Meanwhile, the LLM-Judge only gets the output and the criteria (App. D.2). Can the authors get human annotations for the three criteria checks in lines 292–316, compare them with the LLM-Judge results, and report Krippendorff’s for each one?
  • Only one LLM-Judge is used. All responses are judged by Claude-Opus-4.7 (l.319), and it is also one of the evaluated models. This may introduce self-preference bias [1]. Could the authors include more frontier models as LLM judges?
  • The authors state that “models frequently default to associational generation rather than preserving the causal structure” (l.28-39). However, the experiments only measure three checks, and none of them directly tests for associational generation. Could the authors add a direct test? For example, the same question could be asked in two forms: “what would have happened if...” and “what happened if...” If the model gives similar answers to both forms, this would support the claim.

[1] Panickssery et al. LLM Evaluators Recognize and Favor Their Own Generations. NeurIPS 2024.

Questions for authors

Please see weaknesses.

3 comments
AuthorsOpenReview ↗

Author Response [1/2]

We thank reviewer XYva for their very helpful feedback, and have incorporated several of their suggestions into our latest draft (changes in blue).

We especially appreciate the acknowledgment of the formal causal grounding of our benchmark, the diversity of domains and graphs compared to prior work, and the rigor of the prompt ablation experiments and error analysis.

(W1) Introduction is long

This is useful feedback, and we have now reduced Sec. 1 to 3 pages and shifted some content to later.

We still retain some background about the Pearl Causal Hierarchy because a general audience may lack your causal expertise. They may not understand clearly why our evaluation is novel vs. earlier benchmarks (e.g. TimeTravel, PASTA discussed in the other threads) which use the word “counterfactual” but are actually not properly grounded in causal rules for editing.

We want to strongly clarify the difference between counterfactual and associational text edits. We have added a note at Line 048 that readers familiar with PCH can skip this part. Please let us know if that helps balance out the paper.

(W2) Comparison between the Judge and human experts doesn't seem fair. Can the authors compare human annotations for the three criteria checks with the Judge?

We clarify an important point here.

In our human agreement study (App. D.4), the LLM-judge receives model output and the self-contained eval criteria which contain the facts which should hold, or be changed in the output, plus the eval prompt in App. D.2. This setting is how we evaluate models in our benchmark. The human experts receive the original story, intervention, and model output, plus instructions of how to perform the counterfactual edit based on formal causal rules (App D.4.1). This is our unbiased ground truth for what counts as a correct evaluation.

Indeed, we deliberately don’t show the judge-facing eval criteria to the human annotators because we want to compare our actual eval pipeline to ground truth correctness according to the causal rules (92.1% agreement). The judge-facing instructions are internal to the judge pipeline. We don’t want to bias the annotators based on what the judge sees.

To address your concern, we added App. J with newly conducted independent expert annotations on the judge-facing criteria checks themselves.

CriterionAgreementKrippendorff’s alphaGwet’s AC1
E196.9%0.910.95
E295.2%0.550.95
E096.4%0.380.96
Overall verdict92.1%0.810.86

Per-criteria agreement of the judge with experts is 95-97%. The Krippendorff’s α for E2, E0 is misleading because of class imbalance in labels, with such skewed labels, α attributes nearly all agreement to chance and penalizes each disagreement on a rare failure heavily, exhibiting wide intervals, as reported in App. J. Thus, we additionally report Gwet’s AC1 which is high across all criteria.

(W3) Only one LLM judge is used. This may introduce self-preference bias.

We tested our eval pipeline also using GPT-5.6-Sol as a judge (n=500) and discussed results in App. I.

Claude-Opus-5 judge score (%)GPT-5.6-Sol judge score (%)Δ (pts)
Claude output54.954.7−0.2
GPT output62.158.8−3.4
GPT − Claude gap+7.2+4.1−3.1

We see no evidence of model-family-bias. Claude-judge widens the GPT–Claude score advantage, compared to GPT-judge. GPT-judge reduces the GPT–Claude score advantage, compared to Claude-judge. The scores are also roughly similar. The judges differ mainly in calibration: GPT-5.6-Sol is slightly stricter on the “unchanged facts” criterion and slightly more lenient on the “downstream facts changed” criterion, for both models alike.

(Minor correction: our submission had a typo at Line 367. Our evals used Opus-5 as a judge, not Opus-4.7; we corrected this now)

AuthorsOpenReview ↗

Author Response [2/2]

(W4) The evaluation criteria don’t test associational bias. Could the authors add an associational prompt test?

Associational reasoning:

Actually, our eval criterion E1 does test failures due to associational reasoning. This criterion checks whether models update prior causes, background premises, or other facts which should remain invariant under counterfactual edit. E.g., in Fig. 3 when changing the treatment Gemini also edits the patient symptoms. Frontier models exhibit this error ~30% of the time (red bars in Fig. 6). Such non-causal updates are failures related to associational reasoning (App. A.2) because the edit affects merely-correlated events.

Associational prompting:

We follow your suggestion and directly compare GPT-5.4-Pro by asking the same question under counterfactual (“would have happened…had”) and associational (“happened…if”) query forms on the same 500 samples, holding the stories, judge, and evaluation criteria fixed (App. K). Model accuracy is similar: 62.1% (counterfactual) versus 61.1% (associational).

Beyond aggregate accuracy, responses under both query versions receive matching judge verdicts on 80.9% of paired examples overall and 88.2% for E1 eval criterion, which specifically checks for changes to invariant facts (associational reasoning failures). Failures on E1 criterion across each query form are also similar at 27.2% vs 28%, and 80% of counterfactual-prompting E1 failures recur under the associational-prompting form.

These results are consistent with associational bias toward updating facts in violation of causal rules.

(a) Accuracy under each query form

Query formAccuracy
Counterfactual: “would have happened…had”62.1%
Associational: “happened…if”61.1%

(b) E1 failure rate under each query form

Query formE1 failure rate
Counterfactual: “would have happened…had”27.2%
Associational: “happened…if”28.0%

(c) Match rates between query forms

Evaluation checkMatch rate
Overall judge verdict80.9%
E1: preservation of invariant facts88.2%

Lastly, we highlight the discussion with reviewer M9Pf extensively discussing the novelty and rigor of our benchmark over prior works that go by “counterfactual editing” but are not rigorously grounded in Pearl causal semantics.

Please let us know if this addresses your questions. We are happy to receive any follow up clarifications/suggestions.

Thank you!

Dear reviewer XYva,

We wanted to enquire if you had a chance to read our response. In particular, our revision incorporates your feedback: (1) shortened Introduction; clearly marked optional background, (2) human-agreement study on individual criteria (95-97% agreement), (3) experiments using different judge model, showing no model-family-bias. We also (4) ran the associational prompt experiment you requested, and documented evidence for associational bias.

Please let us know if there are any remaining concerns, and we would be glad to address these.

Thank you for your time and consideration!

Sincerely, Authors of submission 50487

Reviewer_P1VgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CoBe, a benchmark for zero-shot conversational counterfactual text editing, with no explicit causal graph or editing instructions. The task is grounded in Rung 3 of Pearl’s causal ladder. CoBe contains 2,000 samples across 7 domains and several causal graph types (chain, collider, fork, diamond, correlated, hybrid). Starting from 106 core scenarios, the authors generate and filter LLM-assisted variations. Evaluation uses story-specific preserve/revise criteria and a check for causal connectors, assessed by a Claude Opus 4.7 LLM judge. Nine models are benchmarked; the top overall accuracy is 61.34% and the three frontier systems average 58.05%. Human-majority agreement with the judge is reported on 250 responses (Cohen's kappa = 0.812). The study also examines error types, causal structures, and prompt ablations.

Critical strengths

S1. A relevant and differentiated benchmark setting.

CoBe targets zero-shot conversational counterfactual editing without explicit causal graphs or editing instructions. This is a meaningful evaluation setting. It combines conversational requests, implicit causal structure, open-ended narrative editing, and causally grounded scoring. Unlike surface-similarity evaluation, the preservation/update criteria target behavior directly relevant to real-world assistant use (Secs. 1-3; Table 1).

S2. Careful dataset construction and diagnostic evaluation.

The paper evaluates nine models across multiple domains and causal graphs, breaks down distinct failures, and conducts prompt ablations. The dataset construction includes multi-stage author screening, an external scenario-quality audit, and a 250-response judge/human agreement study provide useful, if incomplete, quality checks (Secs. 3-4; Apps. D-E). These checks are not definitive and they make the benchmark’s diagnostic claims substantially more credible. This matters because the paper’s value lies not only in reporting low accuracy, but in showing where models fail: editing non-descendants, failing to propagate downstream changes, misusing causal connectors, and confusing correlation with causation. That diagnostic contribution supports acceptance even though the paper does not yet propose a method for fixing these failures.

Critical weaknesses

W1. Limited methodological novelty and an overbroad novelty claim.

Counterfactual story rewriting and counterfactual reasoning benchmarks already exist (TimeTravel, PASTA, CRASS, CLadder, CounterBench, etc.). CoBe’s contribution is a specific combination — conversational, zero-shot, narrative, no causal graph, open-ended generation, expert-verified criteria — rather than the first counterfactual test. The paper does acknowledge prior work, but the “first testbed” phrasing should be qualified more explicitly. More importantly, the paper does not propose a method for solving the failures it documents; it is primarily a benchmark and diagnostic paper.

W2. The open-source models are outdated.

Section 4 evaluates several older open-source models, such as Llama-3.1-8B (2024), Mistral-7B (2023), Qwen-7B (2023), and Gemma-27B (2024), among others. This makes it difficult to tell whether the reported failures are characteristic of current open-weight models or partly an artifact of using older checkpoints. The benchmark would be more informative with a more up-to-date open-source model suite.

Questions for authors

Q1. How robust is the LLM judge to incomplete rewrites that omit a fact listed under E2? The paper notes that E2 accepts removal, and that manual inspection found no systematic shortcut. Could the authors report the rate of such omissions, or a stricter E2 variant, to quantify the potential inflation?

Q2. Could the authors compare CoBe and TimeTravel/PASTA on the same models under comparable prompting? This would clarify how much of the difficulty is due to the conversational/no-graph setting rather than story-rewriting in general.

Q3. Are accuracy differences robust when the 2,000 augmented items are grouped by their 106 originating core cases? Please provide cluster-aware uncertainty estimates. Results on independently authored, non-augmented cases would be a useful optional complement.

5 comments
AuthorsOpenReview ↗

Author Response [1/3]

We thank reviewer P1Vg for their thoughtful comments. We greatly appreciate the acknowledgment of the quality and diversity of our dataset, the usefulness/relevance of our task setting, and the insights from our detailed error analysis.

We have added new sections to the paper and appendix (changes in blue) to comprehensively address the questions raised.

(W1) Comparison to prior benchmarks

Our precise novelty claim (Line 155):

To our knowledge, we provide the first testbed for zero-shot conversational counterfactual edits, grounded in formal causal semantics, in the domain of narrative stories.

Prior works like TimeTravel (Qin et al, 2019) and PASTA (Ghosh et al, 2023) do test story rewrites and provide error analysis. However, their definition of “counterfactual” is not correctly grounded in Pearl structural causal semantics and thus permit model responses that violate counterfactual edit rules. We now clarify this in Line 156.

As acknowledged in your Strengths, distinguishing these reasoning modes is highly relevant for avoiding risks in modern AI pipelines.

We added Appendix F to elaborate in detail on the fundamental technical limitations of these benchmarks, which we resolve in our work. Please refer to our discussion with M9Pf where we elucidate the significant novelty of CoBe over these works. To summarize,

TimeTravel (TT) and PASTA

  • TT does not test one of the main failure modes of associational reasoning our dataset detects.
  • Both TT and PASTA’s evaluation criteria are not rigorously grounded in Pearl causality. We show through controlled diagnostics (Table 8, 10) how automated scores like BLEU don’t correspond well with causal-validity checks. We mathematically show (App F2) how PASTA could incorrectly approve associational edits as valid. We back this up with multiple examples of approved story rewrites from TT and PASTA (Table 9, 11) that violate counterfactual editing rules.
  • Both benchmarks rely on the ROCStories dataset (Mostafazadeh et al., 2016) of simple, everyday stories following a rigid 5-sentence template. Frontier model saturates TT.

In contrast, as the reviewer acknowledges, CoBe provides diverse labeled domains and causal graphs in our stories, and extensive quality checks to ensure our evaluation pipeline is reliable for judging causal-validity (>92% match with expert-annotated ground truth). CoBe is designed for use as a high quality and scalable signal for inference-time or post-training methods to improve LLM reasoning, unlike prior benchmarks for this task.

CLadder, CounterBench, CRASS etc. As we explain in Section 2 and Appendix F.3, these benchmarks test fundamentally different tasks. CRASS tests multiple choice questions about counterfactuals (also not grounded rigorously in valid causal rules). CLadder and CounterBench test whether LLMs can enact known causal inference algorithms when causal graphs and/or probabilities are provided, and only test Yes/No questions. None of these works test open-ended story rewrites.

it is primarily a benchmark and diagnostic paper.

We accept this, and have included it in Limitations. Methods to solve this task are beyond the scope of our work.

Our valuable contribution is to first rigorously define this task, benchmark model performance using clear, causally sound criteria, and extensively diagnose current failure modes.

(W2) Show results on newer open source models

We ran our dataset on 3 newer open source models (n = 500 samples) to show within- and across-family comparison of performance (App H). Results summarized below. All models released in 2026.

Qwen-3.8-27BGemma-4-12BGemma-4-31B
Score43.0%42.2%52.9%

It is interesting to note that within the same family, a larger model Gemma-4-31B improves by over 10% vs the smaller Gemma-4-12B. While these newer open-source models do a bit better than previous checkpoints at the same scale, the performance gap is still very wide open. We will continue to run the full dataset on these models and update these findings in the main text for the camera ready version.

AuthorsOpenReview ↗

Author Response [2/3]

(Q1) How robust are eval criteria to incomplete rewrites that omit a fact listed under E2?

We find our eval pipeline is robust to this gap in the model responses we inspected. As described in Sec. 3.1, E1 specifies facts to preserve, while E2 specifies original facts that must change or be removed, and E-N specifies additional constraints required in the output.

Our dataset uses additional positive criteria E-N to test whether an edit enacts story-specific requirements in the edit. Fig. 10 (page 42) shows a numerical example (318v5) where doubling the processing rate from 8 to 16 GFLOPS requires the completion time to halve from 10 to 5 seconds in the model response. An additional criterion, E-N, explicitly checks if this condition is satisfied; merely deleting the original ten-second duration (E2) does not satisfy the full rubric. Fig. 11 shows another example (315v12) where E-N ensures that fuel consumption must increase in the edit w.r.t original.

Overall reliability: we validate our overall pipeline against ground-truth evals from experts who receive the original story, intervention, and model response. Our judge achieves 92.1% agreement with expert majority (App. D.4).

Omission rate: we investigated 250 samples and found that only 1/250 cases (0.4%) involve an incomplete rewrite that deletes an E2 fact, where the judge incorrectly approves the model response (App. G.1). We don’t see this potential inflation as a systemic issue.

Stronger judge prompt: on your suggestion, we tested a more complicated judge prompt where we include the original story and intervention and add instructions like “Make sure the model response for the counterfactual query is coherent and consistent with the original story and does not delete any relevant details.” (App. G.2)

Judge promptAgreement with human experts (n=250 model responses)Output tokens w.r.t original promptQualitative findings
Original (App. D.2)92.1%1xStable, logic-gate checks
Augmented (add original story, intervention, instructions for consistency of edits/deletions)94.1%~1.4xNon-monotonic improvement; flips many correct answers

We found that adding the original story, intervention, and additional instructions only improved judge performance w.r.t human-agreement by 2.0%; but it required a ~40% additional cost in reasoning token budget (hard-capping output increased the rate of failed evaluations). Using the augmented judge also does not result in a monotonic improvement: it changes the overall verdict on 9.2% of the responses, fixing 5.6% disagreements with the annotators but also creating 3.6% new ones. The deletion criterion itself also catches no deletion that the original judge missed: every response it newly fails was passed by the annotators.

Overall, we find our eval pipeline is a cost-effective and sound check of counterfactual validity. Of course, we welcome additional suggestions to improve our pipeline.

(Q2) Compare CoBe and TimeTravel/PASTA using similar prompting

We tested TimeTravel (n=100 samples) on a frontier model under a similar prompting style to CoBe (Table 10(b)). Results: Gemini-3.1 scores 97% on TimeTravel vs. 56.68% avg score on CoBe.

This shows that the difference is not merely about conversational prompting used in CoBe, nor the causal invariance provided in TimeTravel. TimeTravel is just too simple for current frontier models, and is already saturated. It doesn’t detect the frontier model errors we identify. As mentioned, both TimeTravel and PASTA are based on the same ROCStories dataset which is 10 years old and likely already in the training set of all frontier models.

Unfortunately, we can’t compare CoBe using TimeTravel’s prompting because the latter strictly requires a 5-sentence story where the first sentence is a background premise and the second story is the intervention. CoBe does not follow this story structure, and sub-selecting samples would not be representative.

AuthorsOpenReview ↗

Author Response [3/3]

(Q3) Provide cluster-aware accuracy variation

If we understand your question, are you asking for cluster-aware discussion of sample diversity? We provide this in App. L the distribution of model accuracy on each cluster of samples, grouped by originating core set examples.

Aggregated histogram across all 9 models is shown here. X-axis shows avg. accuracy of the models on the cluster. The Y-axis shows how many clusters are in that accuracy bucket. In this histogram, we observe that the difficulty of each cluster is diverse. In particular, the distribution is not concentrated at the extremes: cluster-aware accuracy is well represented across buckets, with mean in the 40%-60% band in the model-aggregated plot. This shows that the augmented examples contribute diversity rather than replicating their originating core set examples.

In Fig. 12, we also show this histogram for 3 individual models and find the distribution shows cluster diversity, though Gemini finds 15/85 clusters very easy. (Minor clarification: our dataset began with 106 original core set examples, filtered to 85 during quality control; hence 85 total core sets in the histogram). Please let us know if we can provide any other metrics to support.

Please let us know if this addresses all your questions. We are fully committed to addressing any follow-up/concerns during this discussion phase.

Thank you!

Reviewer_M9PfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a benchmark for conversational counterfactual text editing. Given a short narrative and a hypothetical change expressed in ordinary language, a model must infer which facts should remain unchanged, implement the intervention, and revise its consequences. CoBe contains 2,000 samples obtained through LLM-assisted augmentation. The paper evaluate nine language models, report a best overall score of 61.34% and a mean of 58.05% across the three frontier models, and analyze errors by criterion, graph type, and prompt phrasing. The contribution is a targeted evaluation resource and analysis of failures in preserving and updating narrative facts.

Critical strengths
  1. A benchmark that can facilitate related research in counterfactual reasoning;
  2. The benchmark has a substantial review process, including review by at least two authors, an external expert audit, and exclusions prompted by expert feedback which make the dataset valuable
Critical weaknesses
  1. the novelty claims seem too strong. Counterfactual story editing and model error analysis have been studied before (e.g. Counterfactual Story Reasoning and Generation (Qin et al., EMNLP-IJCNLP 2019), PASTA: A Dataset for Modeling PArticipant STAtes in Narratives (Ghosh et al., TACL 2023)), so the paper should narrow its 'first' claims and clarify what is new about its conversational setting and evaluation.
  2. The graph-type analysis needs more explanation, the paper does not clearly descibe how stories are assigned to graph types and how these lables are checked, making the results in figure 7 difficult to assess and reproduce.
  3. The scope is a bit unclear. The introduction puts considerable emphasis on healthcare, while the benchmark studies counterfactual editing across multiple domains, with healthcare making up only 6% of the examples.
  4. the evaluator may accept incorrect edits, the judge checks whether original facts have changed or been removed, but its displayed prompts do not include the requested intervention, so a wrong replacement could still pass the checks (appendix D2)
  5. the claims about causal reasoning are too strong, although appendix E includes useful prompt controls, the results do not clearly separate misunderstanding the intended editing rules from failing to perform causal reasoning, so attributing failures to reliance on associations needs stronger evidence.
Questions for authors
  1. Does the judge receive the original story and intervention query through any context not shown in Appendix D2? If not, how does it verify that the requested change is correctly made, rather than accepting any replacement or deletion of the original fact?
  2. How are stories assigned to graph types, and how are these assignments checked? Do augmented examples inherit their core example’s graph label, or are they checked individually?
  3. Given the prompt ablation results in Appendix E, what evidence distinguishes misunderstanding the intended editing rules from failing to perform causal reasoning?
5 comments
AuthorsOpenReview ↗

Author Response [1/3]

We thank reviewer M9Pf for their insightful feedback. This greatly helps us improve our presentation. We have added new sections to the paper and appendix (changes in blue) to comprehensively address the questions raised.

(W1) Novelty claim is too broad

Our precise novelty claim is (Line 155):

To our knowledge, we provide the first testbed for zero-shot conversational counterfactual edits, grounded in formal causal semantics, in the domain of narrative stories.

The novelty is not just about conversational prompting, as the reviewer suggests. Prior works like TimeTravel (Qin et al, 2019) and PASTA (Ghosh et al, 2023) do test story rewrites and provide error analysis. However, their definition of “counterfactual” is not correctly grounded in Pearl structural causal semantics and thus permit model responses that violate counterfactual edit rules. We now clarify this in Line 156.

We added Appendix F to extensively detail the fundamental technical limitations of these works. Due to space constraints we summarize below the main problems and how CoBe overcomes them. We are happy to provide more intuition/clarification where needed!

TimeTravel (TT)

  1. One of our main contributions in CoBe is testing whether a model wrongly updates a story’s background premises in the rewrite, which are causally upstream of the requested edit (Fig. 3, 6). For instance, when reasoning about edits to medical treatment, the model should not change the patient’s symptoms. TT prompts the model by providing a fixed background premise in the first sentence, editing the second sentence, and asking the model to continue the story given the edit. TT’s prompt doesn’t test this critical failure mode because the model is explicitly given the invariant premise in the story prefix.
  2. TT’s evaluation is not rigorously grounded in Pearl causality. TT’s automated evaluation computes BLEU/ROUGE scores vs. a reference answer. We show (Table 8) that this unfairly penalizes a causally-identical paraphrase, while rewarding a totally incorrect near-match (deleting words like “never” flip the sentence meaning without changing BLEU score w.r.t reference), making it a bad signal for this task. Results from our control diagnostic below (Table 10(a)) quantify this problem. CoBe avoids this by using story-specific eval criteria that would accept valid responses (including paraphrases) based on semantic content.
Response (n=96)Mean BLEU-4
Original response53.62
Causally-identical paraphrase11.60
BLEU change−42.02
  1. TT’s crowdsourced human annotation does not explicitly check for Pearl-causal update constraints; guidelines only provide general instructions to “respect the changes induced by the counterfactual sentence” without saying what those are. We found dozens of “ground truth” answers that violate counterfactual update rules (3 instances in Table 9). One of CoBe’s central contributions is its high quality evaluation criteria, which are rigorously expert-validated, grounded in counterfactual constraints, and accept multiple valid edits per story.
  2. TT is already saturated and doesn’t detect the frontier model errors we identify. We tested TT on a frontier model under a similar prompting style to CoBe (n=100): Gemini-3.1 scores 97% on TT vs. 56.68% CoBe avg. score (Table 10(b)) showing the issue is not with prompting alone. As reviewer XYva notes, TT focuses on simple, linear everyday stories, which may be too easy for frontier models.

PASTA

  1. We show mathematically (Line 2094–2103) that PASTA’s evaluation criteria permit associational responses; we also construct a mathematical counterexample (Line 2132–2138) of an edit that satisfies PASTA’s criteria by changing a causally-upstream fact (precisely one of the associational failure modes CoBe is designed to catch), showing this is an inherent limitation of PASTA’s eval criteria. LLM-assisted inspection found numerous examples (Table 11) evaluated as correct in PASTA which violate counterfactual edit rules.
  2. PASTA’s human evaluation solution is not scalable. Each new model requires case-by-case crowdsourced annotations. CoBe provides an automated and scalable eval pipeline using LLM-judge, that performs almost as well as expert annotators at 92.1% match rate (App. D.4).
  3. PASTA also relies on the ROCStories dataset (Mostafazadeh et al., 2016) of simple, linear everyday stories following a rigid 5-sentence template. In contrast, CoBe contains diverse domains and causal graphs in our stories, as reviewers note.
AuthorsOpenReview ↗

Author Response [2/3]

(W2, Q2) Do augmented examples inherit their core example’s graph label?

Yes, Graph-type label for each augmented sample is inherited from the originating core set example at stage three of the dataset construction pipeline of “dataset augmentation” (App. D.3). Augmented stories are quality-checked for match with the core set seed. Original core set examples were assigned graph-type labels when authors were constructing the initial set of seed examples, and these were part of the quality-checks.

These are metadata tags used for post-evaluation error statistics by graph type. Fig. 7 can be easily reproduced by bucketing results by the core example’s graph label.

(W3) Domain scope

We clarify throughout the abstract and introduction that CoBe spans multiple domains, not just healthcare (e.g. Lines 026). Line 148:

Our dataset covers scenarios from diverse domains: Science & Research, Engineering, Business & Services, Everyday Scenarios, Healthcare, and Legal, Policy & HR, among others.

We illustrate a healthcare failure mode in Fig. 3 because internal feedback found this example intuitive and the safety concerns clear. However, “doctor/patient symptoms” in this example could easily be replaced with “cybersecurity team/system vulnerability” to illustrate the same point.

Relative % of domains in the dataset naturally vary (5-16% for the top domains) because some domains like healthcare require more effort to ensure high quality and correctness.

(W4, Q1) How does the judge assess the correctness of edits without the original story?

As clarified in Sec. 3.1, judge-facing evaluation criterion E1 specifies facts to preserve, while E2 specifies original facts that must change or be removed, and E-N specifies additional constraints required in the output. Our dataset uses additional positive criteria E-N to test whether an edit enacts story-specific requirements in the edit. Fig. 10 (page 42) shows a numerical example (318v5) where doubling the processing rate to 16 GFLOPS requires the completion time to halve to 5 seconds in the model response. An additional criterion, E-N, explicitly checks if this condition is satisfied; merely deleting the original ten-second duration (E2) does not satisfy the full rubric. Where the correct edit is deterministic (e.g. 318v5), we do provide story-level checks. However, we do not claim that the criteria constitute an exhaustive formal verifier for every possible rewrite. This is not possible. In most cases, there are too many causally-correct possibilities to enumerate exhaustively. E.g. in App. B.1.8 (217v10) multiple replacements are acceptable and adding restrictions beyond our criteria may penalize a correct answer.

Overall reliability: instead, we validate our judge pipeline against ground-truth evals from experts who receive the original story, intervention, and model responses. Our judge achieves 92.1% agreement with expert majority (App. D.4). Detailed inspection found that only 1 of 250 audited samples (0.4%) involved the judge wrongly approving an edit which deleted a necessary fact. The other 5 of 250 false positives (2%) don’t involve wrong downstream updates (App. G.1). Under models’ natural responses we simply don’t find loopholes or shortcuts to be an issue.

Stronger eval prompt: Upon reviewer P1Vg’s request we also test a more complicated judge prompt where we include the original story and intervention and add instructions like “Make sure the model response for the counterfactual query is coherent and consistent with the original story and does not delete any relevant details.” (App. G.2)

Judge promptAgreement with human experts (n=250 model responses)Output tokens w.r.t original promptQualitative findings
Original (App. D.2)92.1%1xStable, logic-gate checks
Augmented (add original story, intervention, instructions for consistency of edits/deletions)94.1%~1.4xNon-monotonic improvement; flips many correct answers

We found that adding the original story, intervention, and additional instructions only improved judge performance w.r.t human-agreement by 2%; but it required a ~40% additional cost in reasoning token budget (hard-capping output increased the rate of failed evaluations). Using the augmented judge fixes 5.6% disagreements with the annotators but also flips 3.6% to disagreements. The deletion check itself also catches no deletion that the original judge missed: every response it newly fails was passed by the annotators. We also anecdotally found that stacking further ad hoc story-criteria did not add any significant improvement in overall judge reliability.

Overall, we find our eval pipeline is a cost-effective and sound check of counterfactual validity. Of course, we welcome additional suggestions to improve our pipeline.

AuthorsOpenReview ↗

Author Response [3/3]

(W5, Q3) How do prompt ablations distinguish misunderstanding the task vs. wrong reasoning?

Our prompt ablations (App. E) examine whether failures persist when the intended editing rules and causal procedure are explicitly supplied. Tier 2 prompting provides an unambiguous request for an intervention within the original scenario, but without stating the preserving/update rules. Tier 3 explicitly states the preservation and updating rules for this task interpretation. Tier 4 additionally provides a step-by-step causal procedure.

ModelTier 2: conversationalTier 3: explicit editing rulesTier 4: causal procedureTotal Gain, Tier 2→4Failure under Tier 4
GPT-5.4-Pro71.7%73.5%81.3%+9.6 pp18.7%
Gemini-3.1-Pro64.4%67.1%76.1%+11.7 pp23.9%

For GPT and Gemini, we can decompose the guidance-gain as follows:

  • Tier 2 → Tier 3 gain: explicitly stating the task interpretation and update rules improves performance by 1.8 and 2.7 pp
  • Tier 3 → Tier 4 gain: providing the detailed reasoning steps provides further gains of 7.8 and 9.0 pp
  • Tier 4 residual error: 18.7% and 23.9% of responses still fail under Tier 4. Supplying the task interpretation and procedure does not ensure successfully executing the reasoning steps.

The claim about associational bias is a different one. We claim we see models often responding in an associational way (as defined in App. A.2) when asked counterfactual questions. Failures on E1 in Fig. 6 includes errors like updating facts that should remain invariant, such as background premises, which correlate with the intervention condition (e.g. updating patient symptoms when changing the treatment), and provides evidence of associational bias. Frontier models exhibit this error ~30% of the time.

Our prompt ablation experiments show that this inability to perform valid counterfactual edits is not merely a prompting artifact. Associational bias is one of the possible reasons models may fail at counterfactual edits.

Please let us know if this addresses your questions, especially regarding novelty. We are fully committed to resolving any remaining concerns during this discussion phase.

Thank you!

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Author Response

We thank all reviewers for their helpful comments. We are heartened that they acknowledge the following strengths:

  • P1Vg and XYva find our task setting "relevant and differentiated" which "targets behavior directly relevant to real-world assistant use". It is "properly grounded in Pearl’s causal hierarchy".
  • All reviewers noted the dataset quality and diversity across domains and graph types. They note the "substantial review process", resulting in a "valuable" benchmark. "Unlike prior work that mostly only focus on simple linear timelines, it actually tests complex [story] structures".
  • XYva and P1Vg acknowledge our "well conducted experiments…and detailed analyses" on LLM failure modes. Experiments and ablations on 9 models "make the benchmark’s diagnostic claims substantially more credible".

The main critique is around:

  1. Novelty of the benchmark vs prior work, particularly TimeTravel and PASTA datasets;
  2. Soundness of the evaluation criteria; and
  3. Evidence to support the claims about failure modes.

Our main follow-up actions/responses include:

  • Appendix F highlighting the technical limitations of prior datasets, and how we improve on it. We summarize the main points to M9Pf and P1Vg.
  • Testing TimeTravel under CoBe prompting for P1Vg; Gemini-3.1 saturates TimeTravel.
  • Clarifications to M9Pf and P1Vg about the rigor of our eval pipeline (>92% match with expert ground truth annotations); omission gaps account for only 0.4% of audited cases. Additional tests using a stronger eval prompt cost did not significantly improve eval rigor.
  • Explanation to M9Pf about what our ablations demonstrate; additional experiments requested by XYva with pair-matched counterfactual and associational prompting show evidence for associational output from LLMs.
  • Testing 3 newer open-source models on our benchmark for P1Vg.

Other points are covered in individual reviewer threads, and new Appendices F-L in the revised submission.

We believe our work represents a significant contribution toward improving counterfactual reasoning in frontier models. We hope to engage with the reviewers to address all remaining questions!

Sincerely,
Authors of Submission 50487

Dear reviewers M9Pf, P1Vg, XYva,

Thank you again for your helpful feedback in the first round!

We would like to follow up on our responses earlier. Please let us know if there are any further questions/clarifications we could address.

We look forward to engaging with you during the discussion period, which ends soon.

Thank you,

Authors of Submission 50487

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.