acceptodds
Under review as a conference paper at ICLR 2027

Same Causal System, Different Graphs: Auditing Pretrained Causal Discovery Across Measurement Coordinates

Abstract

Pretrained causal discovery systems predict graph structure for new datasets, but aggregate accuracy does not establish whether their reported relationships persist across measurement representations. We introduce OrbitBench, a paired evaluation protocol that re-expresses the same observations through strictly increasing, invertible transformations of each variable while keeping the inference pipeline fixed. We evaluate three pretrained systems, with PC and GES as classical references, on linear and nonlinear synthetic tasks and measurements from one physical light-tunnel apparatus. On 16 controlled linear tasks, non-decreasing skeleton F1 can coexist with changed adjacencies; among the pretrained systems, Arrow remains near-best in both views while changing its skeleton in 63 of 64 task–view comparisons. Disagreement persists when predicted edge counts are fixed and extends to directed-path answers on fresh tasks. On 32 new nonlinear tasks, using two audit views, audit-assisted and stability-first selection reduce macroaverage held-out adjacency disappearance risk from 8.27% for score-only selection to 6.71% and 6.68%, respectively. These gains concern persistence within the tested transformation families, not established improvements in causal correctness. Our results motivate reporting relationship consistency alongside accuracy and model-choice summaries.

Primary area:
causal reasoning
Keywords:
Pretrained Causal Models, Measurement Reparameterization, Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rpr9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OrbitBench, a benchmark that can be used to evaluate whether causal discovery models can accurately still report the same causal relationships after the variables in the observational data undergo strictly monotonic, invertible numerical transformations. The paper mainly uses two types of transformations, asinh and cubic, then fixes the preprocessing rules and graph decoding process, and then compares the changes.

Critical strengths
  1. The problem studied in the paper is relatively important and has practical value.

  2. The transformatio processes are well controlled, ensuring randomness and equivalence.

Critical weaknesses

1.The paper’s testing and analysis cannot fully prove that the selected relationships are more correct or valuable. They can only guarantee that the stability observed under the same type of transformations can, to some extent, predict that this stability still exists under new transformations.

2.The data coverage of the benchmark proposed in the paper is slightly narrow.

3.The issue that causal relationships should remain unchanged under appropriate variable reparameterization is not a new issue, and there is already related research. The paper’s novelty is somewhat weak.

Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Response to rpr9 (1/3): At fixed retention, auditing improved edge precision on new nonlinear systems

Thank you for the time and care you put into reviewing our paper, and for recognizing the value of the controlled comparisons. Your distinction between persistence and correctness motivated a direct follow-up. On 32 new nonlinear systems with known graphs, the submitted two-view rule increased retained-edge precision from 91.04% to 92.59% relative to model scores alone, while keeping the same number of candidate edges.

Evidence in the submission. Table 5 showed that audit-assisted selection reduced the proportion of retained edges that disappeared in a held-out view from 8.27% to 6.71%. However, the audit and test views came from the same two transformation families, and the precision gain was uncertain (+0.41 percentage points, 95% CI [−0.06, 0.88]). We agree that the original experiment supported greater persistence, not a demonstrated gain in correctness.

New experiment with the submitted rule. We applied the original rule without retuning it. For each model and system, both selectors kept 80% of the candidate adjacencies, rounded up. We locked the selections without using the true graphs and before generating the held-out predictions. We then evaluated the same retained edges in two ways:

  1. Agreement with the true graph. The paired precision gain was 1.55 percentage points (95% bootstrap CI [0.95, 2.18]). The point estimate favored auditing in all six combinations of model and mechanism, with gains of 0.84–2.25 points. The interval is for the equally weighted average across the six groups.
  2. Persistence beyond the audit families. We kept the selections made from the asinh and cubic audit views and tested them on held-out sinh-asinh views. In this prespecified secondary comparison, disappearance fell from 12.40% with score-only selection to 11.71% with auditing (paired reduction: 0.69 points, pointwise 95% bootstrap CI [0.11, 1.30]). The submitted rule thus showed some transfer to one additional family without changing which edges it selected.

New comparisons with stability-first selection. In a separate confirmation on 32 fresh systems, an eight-view rule reduced disappearance on two non-audit families from 13.59% to 11.99% relative to score-only selection. Yet simply prioritizing edges that survived the audit produced exactly the same retained set in 95 of 96 system–model comparisons. This result supports checking additional views, not the necessity of our weighting formula.

We also tested a separately developed rule that predicts disappearance from edge scores and their variation across audit views. We froze it before evaluating two independent cohorts of 32 new systems each. Relative to stability-first, precision improved in both cohorts (+0.37 and +0.64 percentage points; pooled gain +0.50, adjusted 97.5% CI [0.22, 0.82]). Its risk advantage was less general: disappearance was 0.31 points lower on two families represented during calibration but absent from the audit views, and 0.15 points higher on two families absent even from calibration. This finding belongs to the new rule, not to the submitted formula.

For a researcher prioritizing a fixed number of candidate relationships, the practical result is that audit views can add information beyond model confidence alone. A simple stability-first ranking captures much of the persistence benefit. The accuracy evidence concerns whether two variables are adjacent; it does not establish the direction of an edge or the benefit of an intervention.

AuthorsOpenReview ↗

Response to rpr9 (2/3): Model rankings changed across generators, but F1 still masked changes in predicted edges

Changing the nonlinear data generator changed which pretrained model had the highest reference F1. It did not remove the finding that F1 can stay the same or improve while the model changes which variables it connects. This distinction is why we agree that adding seeds from the submitted generator alone would not have resolved your coverage concern.

New generator comparison. The submitted nonlinear panel used RFF generators released with AVICI. After submission, we tested two independently implemented mechanisms: a nonlinear additive-noise model with Laplace errors and a post-nonlinear model with heteroscedastic errors. We used the same 32-system panel as the two-view precision experiment in Part 1; here we evaluated the predicted graphs before screening.

AVICI no longer had the highest reference F1 among the pretrained systems on either new mechanism. Nevertheless, skeleton F1 did not decrease while the predicted skeleton changed in 55/128 comparisons for CDFM, 52/128 for Arrow, and 38/128 for AVICI. Each denominator represents four coordinate changes on each of 32 systems, not 128 independent datasets. We therefore do not generalize the original model ranking. What reproduced across generators was the gap between the accuracy summary and the individual relationships reported.

New graph-size and sample-size comparisons. The submitted panels used 20 variables and 800 observations. We varied graph size (10, 20, 40 variables) and sample size (200, 800, 1,600 observations) in separate controlled comparisons. At 800 observations, Arrow’s median number of changed pairs rose from 6.1 to 9.8 to 25.8 as graph size increased. Yet the fractions of all possible pairs fell from 13.6% to 5.1% to 3.3%. Larger graphs had more changed relationships in absolute terms, but a smaller proportion of possible relationships changed. We will report counts alongside their denominators.

Additional analysis of the original physical predictions. The 57 light-tunnel environments in the submission are settings of one apparatus. We compared the stored predictions with the published standard-configuration graph restricted to our 20 selected variables, after verifying that omitted recorded variables were constant in the evaluated windows. This provides a configuration-specific reference for interpreting the existing predictions. It is not a new physical dataset or an independently validated ground truth. The additional experiments broaden the simulated mechanisms and scales; generalization across physical domains remains untested.

AuthorsOpenReview ↗

Response to rpr9 (3/3): A higher F1 can hide an incorrect answer about a directed path

We agree that an invertible change of measurement coordinates does not create a new causal system, and that reparameterization sensitivity itself is not new. Jørgensen and Hauberg (2020) studied causal decisions under reparameterization; Yi et al. (2025) evaluated graph recovery under assumption violations including changes of scale. OrbitBench evaluates something these principles alone cannot tell us: how often the same model remains a good choice by F1 while its graph gives a different answer about a specific relationship.

Evidence in the submission. On the 16-system linear panel in Table 2, Arrow remained within 0.01 F1 of the best of the three pretrained models in both views while changing its skeleton in 63/64 comparisons. That is the frequency of disagreement, not its size: the median change was 10.5 of 190 possible pairs (5.5%). On a separate 16-system panel, Table 4 showed that 18.0% of Arrow’s native directed-path answers changed on average. Even when the decoder was required to return the same number of edges in both views, 15.7% changed.

A concrete example from the original predictions; no new model runs. On the first system in the path panel, Arrow ranked first in skeleton F1 among the three pretrained models in both views. Its F1 actually increased from 0.8955 to 0.9014. Yet its answer to “Is there a directed path from variable 1 to variable 6?” went from correct to incorrect. The generating DAG contains no such path, and Arrow’s reference graph correctly contained none. After a mild asinh re-expression of the same observations, its graph contained the path 1 → 5 → 6. Neither prediction contained a direct edge 1 → 6. Thus the model choice stayed the same and its score improved, but its answer about whether variable 1 was upstream of variable 6 became wrong.

Post-hoc check across the full path panel. We checked all 64 Arrow comparisons, not just the illustrative example. Skeleton F1 did not decrease in 11 comparisons spanning eight systems. In every one of those 11, at least one previously correct path answer became incorrect. “At least one” does not mean that most answers changed: across the full panel, using all ordered pairs of distinct variables as the denominator, 13.75% of answers went from correct to incorrect against the generating DAG and 4.24% went from incorrect to correct. Coordinate changes can introduce errors and correct them; we do not assume that the reference prediction is right.

This is the distinction we will foreground in the revision. Without ground truth, the paired audit can flag a specific conclusion that depends on the measurement coordinates even when the model’s standing gives little reason to reconsider it. With a known generating graph, we can additionally determine which answer was wrong. We will report both the frequency and the magnitude of disagreement in the abstract so that frequent changes are not mistaken for wholesale graph instability.

Reviewer_w37UOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether pretrained causal discovery systems report the same relationships when the same observations are re-expressed through strictly increasing, invertible, column-wise transformations — a coordinate change that preserves the target graph and population conditional independences while not preserving an estimator's functional-form or noise assumptions. OrbitBench pairs a reference table with transformed views (asinh asinh(λx)/λ, λ∈{0.5,1.5}; cubic x+βx³, β∈{0.05,0.15}) under a fixed inference pipeline, and measures skeleton symmetric difference, paired skeleton-F1 change, a winner-set flip indicator (0.01 F1 tolerance), and directed-path answer flips over all 380 ordered pairs.

Five methods are evaluated — AVICI, Arrow, CDFM, plus PC and GES as classical references — on 16 controlled linear-Gaussian tasks, 16 fresh-seed path tasks, 32 nonlinear RFF tasks, 32 held-out screening tasks, and 57 Causal Chambers light-tunnel environments, all at 20 variables × 800 observations. Main findings: non-decreasing F1 coexists with changed adjacencies in 41/9/14 of 64 comparisons for CDFM/Arrow/AVICI; Arrow stays near-best in both views while its skeleton changes in 63/64; disagreement persists at fixed predicted edge counts (Table 3) and extends to path answers (12.5%/15.7%/4.4%, Table 4); and an audit-based edge-screening rule lowers held-out adjacency disappearance risk from 8.27% to 6.71%/6.68% (Table 5). Appendices add an SNR sweep, a Rank-Gaussian canonicalization reference, native nonlinear accuracy, bootstrap details, and a Robustness Card implementation.

Critical strengths

Unusual discipline in separating what was measured from what it licenses. The paper prespecifies its primary endpoints and then reports against them even when they fail: the screening gain is 1.56 pp [1.03, 2.14] against a prespecified 2 pp reference, and the paper says plainly that the interval "does not establish that it exceeds the prespecified two-percentage-point reference" and that "audit-assisted selection has no demonstrated advantage over stability-first selection." In E.1 it reports that "all four paired F1-loss intervals include zero, so the prespecified directional increase from the low- to high-R² setting is not established." E.2 states that zero disagreement under Rank-Gaussian canonicalization "follows by construction" rather than presenting it as a robustness win, and C.2/E.3 twice disclose that the nonlinear panel "is therefore not an out-of-distribution evaluation for every pretrained system." Appendix F even notes that the Robustness Card collection "does not constitute additional experiments." This is the right way to write an audit paper, and it is rarer than it should be.

Critical weaknesses
  1. There is no same-coordinate perturbation control, so the paper cannot separate "coordinate sensitivity" from the ordinary instability of a weak estimator — and its own tables point to the latter. The declared controls (§3.3, §4.2) are an identity repeat, which measures execution nondeterminism, and positive affine maps, which preserve the linear/additive-noise model class; both "preserve skeletons for all five methods across the 48 synthetic tasks." What is missing is a perturbation of comparable magnitude that leaves the coordinates alone — resampling the 800 observations, or redrawing the noise at the same R² — against which 10.0–16.5 changed pairs out of 190 could be calibrated. Without it, three features of the paper's own data are more consistent with estimation variance than with a representation-specific failure. (i) The accuracy effect has no direction: CDFM's median F1 loss is −0.014 on the linear panel and −0.0248 at R²=0.25 (accuracy improves under transformation), GES is −0.003, PC is exactly 0.000, and E.1 reports all four paired intervals including zero; the illustrative card in Appendix F has F1 rising from 0.6598 to 0.7750 in the strong-cubic view alongside 21/190 changed pairs. (ii) Instability tracks inaccuracy across methods: PC attains the panel's highest reference F1 (0.925) with 0.75 changed pairs, and 0.25–1.25 across all three SNR settings (Table 8), while CDFM — lowest F1 at 0.656 — changes 12.25–22.25 pairs; so a 1991 constraint-based algorithm is both more accurate and roughly 14× more coordinate-stable than the best pretrained system. The paper notes only that "the observed sensitivity is not confined to pretrained systems"; the stronger reading its Table 2 supports — that these pretrained systems are simply dominated here, and that reshuffling ~5% of pairs is what an F1≈0.62–0.66 estimator does under any input perturbation — is never engaged. (iii) The method ordering reverses on real data: on the 56 light-tunnel environments (Table 6) CDFM is the most stable (2.50) and GES the least (7.75), exactly inverting the synthetic linear ordering (CDFM 16.5, GES 4.5). A design that produces opposite orderings on synthetic and physical data, with no matched within-coordinate reference, does not yet support the paper's framing.

  2. The nonlinear panel — which carries the fixed-edge-count confirmation and the paper's only claimed practical payoff — is confounded by generator provenance, and the payoff fails its own bar and is matched by the trivial baseline. Appendix C.2 states the 32 nonlinear tasks come from "AVICI 1.0.7's released rff-gauss.yaml and rff-gauss-heterosked.yaml generators," and the paper twice concedes these "are related to AVICI's training distributions." The effect is visible and large: AVICI is the least accurate pretrained system on the linear panel (F1 0.617 vs PC's 0.925) but the most accurate and most stable on the nonlinear panel (F1 0.7974/0.8114 and 4.50/3.25 changed pairs, versus PC at 0.6068/0.5132 and Arrow at 0.6033/0.5520, Table 11), with by far the lowest path-flip fraction (4.4% vs 12.5% and 15.7%). So every cross-method statement drawn from the nonlinear panel is at least as consistent with train–test overlap as with differential coordinate robustness, and no nonlinear generator independent of the evaluated models is reported. The screening experiment (Table 5) runs on this same panel and weights six model–noise groups equally, two of which are AVICI's in-distribution groups. On its own terms that experiment also does not deliver: 8.27% → 6.71% (audit-assisted) versus 6.68% (stability-first), i.e. the proposed composite score u(e)=r(e)−q(e) adds nothing over simply ranking by how often an edge vanishes; the 1.56 pp gain misses the prespecified 2 pp reference; retained-adjacency precision moves +0.41 pp [−0.06, 0.88], a CI containing zero, so greater persistence buys no measurable correctness; and audit and test parameters are drawn from the same two families over the same ranges (λ∈[0.5,1.5], β∈[0.05,0.15]), making the generalization tested in-distribution by construction. After the paper's own disclaimers, contribution (3) reduces to a small in-distribution persistence gain from a rule the paper itself declines to prefer.

Questions for authors
  1. Same-coordinate control. Can you add a perturbation of matched magnitude that leaves the measurement coordinates fixed - for example bootstrap-resampling the 800 observations, or redrawing the noise at the same local R-squared - and report changed-pair counts for all five methods under it? Without that reference, I cannot tell whether 10.0 to 16.5 changed pairs out of 190 is a coordinate effect or the ordinary sampling instability of an estimator whose F1 is 0.62 to 0.66. The identity repeat and the positive-affine control both preserve skeletons by design, so neither calibrates the scale of the reported effect.

  2. Is the effect coordinate-specific or accuracy-driven? In Table 2, PC attains the highest reference F1 (0.925) with 0.75 changed pairs, while CDFM has the lowest F1 (0.656) and the most changed pairs (16.5); Table 8 shows the same ordering across all three R-squared settings (PC 0.25 to 1.25, CDFM 12.25 to 22.25). Can you report the association between a method's reference F1 and its changed-pair count across methods and panels? If instability is largely explained by accuracy, the paper's framing as an audit of pretrained systems specifically would need revision.

  3. Direction of the accuracy effect. CDFM's median F1 loss is -0.014 on the linear panel and -0.0248 at R-squared 0.25, GES is -0.003, PC is 0.000, and E.1 reports all four paired F1-loss intervals including zero; the Appendix F card shows F1 rising from 0.6598 to 0.7750 under the strong-cubic view. Given that transformation is as likely to help as to hurt, what distinguishes the reported phenomenon from symmetric estimation noise?

  4. Generator provenance. The 32 nonlinear tasks use AVICI's own released rff-gauss and rff-gauss-heterosked generators, and the paper states twice that the panel is not out-of-distribution for every system. AVICI is the least accurate pretrained system on the linear panel (F1 0.617 versus PC's 0.925) but the most accurate and most stable on the nonlinear panel (F1 0.7974 / 0.8114, changed pairs 4.50 / 3.25) and has the lowest path-flip fraction (4.4%). Can you rerun the nonlinear confirmation and the held-out screening on a nonlinear generator not derived from any evaluated model, and report whether the cross-method ordering survives? As submitted, I do not think the nonlinear cross-method comparisons or the six-group screening average can be read as coordinate-robustness results.

  5. The screening contribution. Given that stability-first attains 6.68% versus audit-assisted's 6.71%, that the 1.56 pp [1.03, 2.14] gain misses the prespecified 2 pp reference, that the precision change is +0.41 pp [-0.06, 0.88], and that audit and test parameters are drawn from the same families over the same ranges, what do you claim is established beyond "edges that vanish in two extra in-distribution views are somewhat more likely to vanish in a third"? Does the composite score u(e) = r(e) - q(e) have any setting in which it beats stability-first?

  6. Scope and reporting. Every synthetic task and physical environment uses 20 variables and 800 observations, with two transformation families at two parameter values. How do the reported counts behave as d and n vary, and at larger transformation strengths? Separately: AVICI is absent from the SNR analysis (Tables 8 and 9) with no stated reason - why? And since the physical extension was, by your own account, "designed after inspecting the initial result" and carries no reference-graph accuracy, what does it establish beyond that predicted skeletons also change on this apparatus?

  7. Magnitude versus incidence. The abstract leads with "changing its skeleton in 63 of 64 task-view comparisons" for Arrow, which counts any nonzero change; the corresponding median magnitude is 10.5 of 190 pairs (5.5%) with a median F1 loss of 0.040. Would you consider reporting incidence and magnitude together in the abstract, since 63/64 alone reads as near-total instability?

3 comments
AuthorsOpenReview ↗

Response to w37U (1/3): Matched noise and full resampling give different baselines for coordinate disagreement

Thank you for identifying the missing control and for recognizing the distinction between what the audit measures and what it establishes. We ran two same-coordinate controls. For CDFM, re-expressing the same observations changed about 16 adjacency pairs, compared with 29 when we drew a fresh sample from the same system and 14 when we perturbed the noise at a comparable standardized input displacement. Full resampling changed more pairs than re-expression; standardized magnitude-matched noise changed fewer. The first comparison puts coordinate disagreement in perspective. The second tests whether the magnitude of the input change alone accounts for it.

Q1 — Matched perturbation and independent resampling. We evaluated 32 new linear-Gaussian structural causal models (SCMs), each with 20 variables, target local R² = 0.5, and three independent 800-row samples. We compared (i) the four submitted coordinate changes applied to exactly the same observations; (ii) a same-coordinate perturbation that mixed the reference Gaussian noise with independent noise from the same SCM, preserving population SNR; and (iii) independent samples from that SCM, with no coordinate change. For (ii), the mixing strength was calibrated using population moments, not model outputs. Entries are mean changed skeleton pairs out of 190, not the two-level medians in submitted Table 2.

MethodSame data, new coordinatesSame coordinates, matched noiseSame coordinates, fresh sample
CDFM15.6113.6128.96
Arrow10.936.1210.38
AVICI9.474.2510.63
PC0.910.663.10
GES6.464.8516.52

Full resampling measures ordinary sampling variability. Drawing a fresh sample produced more disagreement than re-expression for four of the five methods; for Arrow, the two counts were close. The original changed-pair counts therefore do not establish that coordinate changes dominate sampling variability.

The matched-noise result depends on the scale used for matching. In the initial 16 SCMs, we measured displacement relative to each variable’s fixed reference variance. Under that rule, CDFM changed fewer pairs under re-expression than under matched noise (difference: −1.61 percentage points of possible pairs). After CDFM’s own column standardization, however, the noise perturbations were larger: the median noise-to-coordinate displacement ratio ranged from 1.75 to 2.40 across the four views. We therefore fixed a second rule that matched expected displacement after column standardization and tested it on the 32 fresh SCMs above. The observed median ratio was 1.01 (2.5th–97.5th percentiles: 0.95–1.08), close to the target of 1. Under this rule, CDFM, the prespecified confirmation target, changed about two more pairs under re-expression than under matched noise: 1.05 percentage points of possible pairs (95% SCM-bootstrap CI [0.60, 1.51]). All five point estimates were positive, although PC’s absolute difference was small. For CDFM, perturbations of comparable standardized magnitude did not produce the same amount of graph change: how the input was perturbed also mattered. The reversal under reference-scale matching is why we qualify that conclusion by the metric used.

Q2 — Accuracy and disagreement are related, but their ordering is not universal. PC was both more accurate and more stable on the submitted linear panel. Across the five methods, Spearman correlations between median reference F1 and median changed-pair count were −0.50 there, −0.20 and −0.21 on the two submitted nonlinear panels, and −0.30 and +0.40 on the independent mechanisms in Part 2. These are descriptive correlations across only five methods. Within CDFM, the correlation across the 32 new SCMs was −0.59. In a post-hoc split, even its upper half by reference F1 (0.681–0.742) showed more disagreement under coordinate change than under standardized matched noise (+0.58 points; 95% CI [0.11, 0.96]). The excess was not confined to CDFM’s less accurate half. We do not claim a failure exclusive to pretrained systems.

Q3 — The sign of F1 change does not identify its cause. CDFM’s average transformed-view F1 rose relative to its reference on 23 of the 32 new SCMs and fell on nine. Replacing a false adjacency with a true one can improve F1 while changing which relationship a researcher would investigate. We agree that neither a gain nor a loss alone distinguishes coordinate sensitivity from sampling noise. F1 tells us whether agreement with truth improved; the paired controls tell us how much the reported relationships changed relative to ordinary perturbations.

AuthorsOpenReview ↗

Response to w37U (2/3): Independent generators separated accuracy from stability and tested what screening adds beyond stability-first

Q4 — The accuracy ranking changed on independent generators; the stability ranking did not change in the same way. We reran the structural comparison and the submitted screening rule on 32 new systems: 16 from an independently implemented nonlinear additive-noise model with Laplace errors and 16 from a heteroscedastic post-nonlinear model. Neither generator was derived from an evaluated model. The same cohort supplies both analyses; these are not two replications. One clarification: AVICI’s submitted 4.4% path-flip result (Table 4) came from fresh linear-Gaussian tasks, not from AVICI’s nonlinear generators. Your provenance concern applies directly to the submitted nonlinear comparisons.

Pretrained modelMedian reference F1: additive / post-nonlinearMean fixed-count disagreement (% of possible pairs): additive / post-nonlinear
CDFM0.710 / 0.6725.18 / 6.46
Arrow0.806 / 0.6605.92 / 11.43
AVICI0.696 / 0.5053.73 / 4.36

AVICI no longer had the highest median reference F1 on either mechanism: Arrow led on the additive systems and CDFM on the post-nonlinear systems. Yet AVICI still had the lowest fixed-count disagreement of the three on both. A changed native skeleton also coexisted with non-decreasing F1 in 55/128 comparisons for CDFM, 52/128 for Arrow, and 38/128 for AVICI (four views on each of 32 systems, not 128 independent systems). The original accuracy ranking did not generalize. What did generalize was the distinction the audit measures: a model could lose its accuracy lead without losing its stability lead.

Q5 — The submitted screening rule on the new generators. We used the original two-view rule without retuning. At 80% retention (rounded up), selections were locked before generating held-out predictions and without consulting the true graphs.

  • Same-family persistence. Compared with model scores alone, disappearance fell from 22.55% to 19.70% (paired reduction 2.85 percentage points; adjusted 97.5% CI [2.11, 3.65]). This exceeds the two-point reference on the new panel; it does not change the original 1.56-point result.
  • Correctness at equal retention. Retained-edge precision was 91.04% with scores alone, 92.50% with stability-first, and 92.59% with the submitted rule. The submitted rule gained 1.55 points over scores alone (95% CI [0.95, 2.18]); all six model-by-mechanism point estimates favored it over scores alone, including the four not involving AVICI. Thus the audit views helped select a higher fraction of true adjacencies, not just persistent ones. The submitted rule and stability-first selected exactly the same edges in 89 of 96 system–model comparisons, explaining why their precision was nearly identical.
  • Direct comparison of the original formula with stability-first. On held-out sinh–asinh views, a family absent from the audit, disappearance was 12.40% with scores alone, 11.71% with the submitted rule, and 11.81% with stability-first. This is a setting where the formula did slightly better than stability-first (0.10 points; pointwise 95% CI [0.03, 0.19], adjusted 97.5% CI [0.00, 0.20]). On the same-family test it did worse by 0.23 points. We therefore recommend using the audit information, not generally preferring the original weighting formula to stability-first.

Additional finding, distinct from the submitted formula. We separately developed a rule using edge scores and their variation across audit views, then froze it before testing two independent cohorts of 32 new systems each. Its precision gain over stability-first was +0.37 and +0.64 points (pooled +0.50; adjusted 97.5% CI [0.22, 0.82]). Its disappearance risk was 0.31 points lower on two families seen during calibration but absent from the audit views, and 0.15 points higher on two families absent even from calibration. What replicated was a gain in the fraction of selected adjacencies that were true at a fixed budget, not a universal reduction in disappearance. This result belongs to the new rule, not to the submitted formula.

AuthorsOpenReview ↗

Response to w37U (3/3): Graph size, sample size and transformation strength put the magnitude of disagreement in context

Q6 — Graph size, sample size and transformation strength (new experiments). We varied one factor at a time on linear-Gaussian systems, with 16 SCMs per dimension. Different dimensions use different SCMs; at 20 variables, the sample-size comparison uses nested samples from the same SCMs. Entries are median changed adjacency pairs, rounded to one decimal.

Methodd=10, n=800d=20, n=800d=40, n=800d=20, n=200d=20, n=1600
CDFM3.913.437.127.011.4
Arrow6.19.825.810.010.3
AVICI4.89.815.912.68.9
PC0.50.52.32.00.8
GES1.57.516.410.14.8

At n=800, Arrow changed more pairs as d grew, yet the fraction of possible pairs fell: 13.6%, 5.1%, 3.3%. At d=20, increasing n from 200 to 1,600 reduced CDFM’s count from 27.0 to 11.4, whereas Arrow’s remained near ten. Stronger transformations also mattered: at d=20, n=800, Arrow’s median count was 8.0, 14.5, 15.0 across asinh strengths 0.5, 1.5, 3.0 and 7.0, 9.0, 13.5 across cubic strengths 0.05, 0.15, 0.30. Thus “ten changed pairs” is not a transferable effect size without the number of possible pairs and the setting in which it was measured.

AVICI in the signal-to-noise analysis. The original SNR extension was frozen for four methods; AVICI was evaluated in a separate extension that did not cover the three SNR settings. We should have made that scope explicit. We have now run AVICI on the original tasks and views. At target local R² of 0.25, 0.50 and 0.75, its median changed-pair counts were 8, 10 and 9, respectively; its median reference F1 rose from 0.430 to 0.617 to 0.707. Better recovery at higher SNR did not produce a monotone decline in coordinate disagreement.

What the physical extension establishes (new analysis of stored predictions). The 56-environment extension was designed after inspecting the initial result and uses one apparatus, not 56 independent physical systems. We checked the standard configuration and verified that the 18 omitted recorded variables were constant in the evaluated 800-row windows. The published graph restricted to our 20 variables has 39 edges. Across the extension, CDFM and PC both changed few pairs (medians 2.50 and 2.00), but their median skeleton F1 against this reference was 0.730 and 0.281. GES changed 7.75 pairs with reference F1 0.638. The CDFM–GES stability ordering does reverse relative to the submitted linear panel; we do not use either ordering to recommend a model across domains. What the apparatus adds is a concrete case where low disagreement does not imply close agreement with a physical reference. These scores are relative to the published standard-configuration graph, not independently validated ground truth for each intervention.

Q7 — Incidence and magnitude belong together. We agree and would revise the abstract to read: “On 16 controlled linear tasks, Arrow remained near-best among the pretrained systems in both views while changing its skeleton in 63 of 64 task–view comparisons. On the same panel, the median disagreement was 10.5 of 190 possible pairs (5.5%), and the median skeleton-F1 loss was 0.040.” The first number says how often reported relationships changed; the second says how many changed. Frequent change is not wholesale graph replacement.

Reviewer_XF2rOpenReview ↗
Rating
2(−1)
Confidence
2(−1)
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

OrbitBench tests whether causal discovery models predict the same graph after strictly increasing, invertible transformations of each variable. It compares AVICI, Arrow and CDFM with PC and GES on synthetic data and measurements from a light tunnel. The paper tracks changes in edges and directed paths, showing that F1 and model rankings can miss changes in predicted structure. It also tests screening edges using two transformed views and provides a format for reporting consistency.

Critical strengths
  1. The paper shows why checking F1 alone is not enough. A model can keep its score while changing the relationships it predicts (Table 2). I would want to know this before using a discovered graph to decide which relationships to test experimentally.

  2. The controls support the finding. Disagreement persists when edge counts are fixed, and repeated runs help rule out execution noise. I also like that the authors distinguish consistency from correctness. An edge surviving a transformation does not mean it is correct.

Critical weaknesses
  1. I am less convinced by the screening step. Audit-assisted and Stability-first have almost the same disappearance risk: 6.71% versus 6.68% (Table 5). The confidence interval for the precision gain also includes zero. Rank-Gaussian preprocessing makes the inputs identical across these views, but its accuracy is only evaluated on linear tasks (Appendix E.2). Without the nonlinear comparison, I am unsure when I would choose auditing over preprocessing.

  2. The disagreement counts need more context about graph density. A method predicting few edges has fewer opportunities for disagreement under fixed-edge-count decoding. This matters when comparing methods in Tables 3 and 11. The light-tunnel results also lack edge counts and accuracy against a reference graph appropriate to the variables and intervention setting. This makes it hard to judge whether lower disagreement comes with predictions that are useful.

Questions for authors
  1. Comparison affecting my assessment: How does rank-Gaussian preprocessing affect accuracy on the nonlinear tasks? This would help establish the benefit of auditing.

  2. Clarification: Can you report reference edge counts alongside disagreement, and screening risk and precision separately for each model? For the light-tunnel data, is there a reason reference-graph accuracy cannot be reported for the evaluated settings?

  3. Clarification: How are Arrow’s directed scores constructed? For CDFM, does matching the edge count of a cyclic reference prediction require adding edges below its native threshold during DAG decoding? These details affect how I interpret the path results.

4 comments
AuthorsOpenReview ↗

Response to XF2r (1/3): Rank-Gaussian preprocessing improved nonlinear graph accuracy for some models and reduced it for others

Thank you for the care you put into this review, especially for asking what a researcher should do before taking a predicted relationship into an experiment. We ran the nonlinear comparison you identified as affecting your assessment. On the same 16 new heteroscedastic post-nonlinear systems, rank-Gaussian preprocessing raised Arrow’s skeleton F1 by 13.29 percentage points (95% bootstrap CI [9.57, 17.13]) but lowered AVICI’s by 6.10 points (95% CI for the change [−11.98, −0.76]). Making the inputs identical across measurement scales did not guarantee a more accurate graph.

Nonlinear accuracy on the tasks in the submission (Q1; new runs). We ran the native and rank-preprocessed pipelines on the same observations and compared each full predicted skeleton with the known graph, before selecting any edges. Entries are mean paired F1 changes in percentage points; positive values favor preprocessing. The structure and screening panels use different systems, each with 16 homoscedastic and 16 heteroscedastic tasks.

ModelStructure: homosc.Structure: heterosc.Screening: homosc.Screening: heterosc.
CDFM+0.69+2.14−2.87−2.74
Arrow−0.77−3.79−0.99+0.54
AVICI−14.44−6.31−12.10−18.71

The classical references were included in the structure panel: PC changed by +0.97 and +1.24 points, and GES by +1.53 and +1.13 (homoscedastic, heteroscedastic). The screening panel covers the three pretrained models.

The choice depends on the decision being made. If choosing a prediction pipeline, validate rank preprocessing for the model and setting; the Arrow result shows why it is worth considering. If the native graph has already nominated an edge for follow-up, audit that edge: does the same pipeline still recommend it when only the measurement scale changes? Preprocessing can give a useful alternative graph. Auditing tells us whether the original recommendation survives.

AuthorsOpenReview ↗

Response to XF2r (2/3): Audit views helped select true edges at fixed retention; reference counts put disagreement in context

Screening at a fixed retention budget (Q2; new experiment). On 32 new nonlinear systems with known graphs, retained-edge precision was 91.04% with model scores alone, 92.59% with the submitted two-view rule, and 92.50% with stability-first. All three kept the same number of candidate edges per model and system (80% retention, rounded up). The submitted rule’s paired gain over scores alone was 1.55 percentage points (95% bootstrap CI [0.95, 2.18]). For a researcher choosing a fixed number of edges to investigate, the practical result is that ranking edges by how often they survived the audit views, with model scores breaking ties, captured nearly all of the submitted rule’s precision gain.

Per-model breakdown of submitted Table 5. Each cell is held-out disappearance risk / retained-edge precision, in percent. Lower is better for risk; higher is better for precision. The two noise conditions are weighted equally within each model, conditional on nonempty candidate sets.

ModelScore-onlyAudit-assistedStability-first
CDFM7.75 / 73.906.37 / 74.216.30 / 74.21
Arrow10.88 / 81.118.24 / 81.908.24 / 81.82
AVICI6.17 / 94.965.54 / 95.105.48 / 95.10

Your reading of Table 5 was right: the two audit-based rules had nearly identical disappearance risk (6.71% vs. 6.68%), and the original precision gain over scores alone was uncertain.

A new rule improved precision beyond stability-first. We separately developed a rule using edge scores and their variation across audit views, then froze it before testing two independent cohorts of 32 new systems each. Precision improved over stability-first in both cohorts (+0.37 and +0.64 points; pooled +0.50, adjusted 97.5% CI [0.22, 0.82]). Disappearance risk was 0.31 points lower on two families seen during calibration but absent from the audit views, and 0.15 points higher on two families absent even from calibration. Thus the advantage that replicated was selecting a higher proportion of true adjacencies, not uniformly reducing disappearance.

Reference edge counts alongside disagreement (new breakdown of submitted predictions). Each cell gives the median predicted edge count on the reference view / median number of changed adjacency pairs, out of 190 possible pairs. Table 3 used score-based fixed-count decoding for the three pretrained models.

Fixed-count decoding (Table 3)LinearNonlinear: homosc.Nonlinear: heterosc.
CDFM57.5 / 14.536.5 / 9.540 / 9
Arrow34 / 1021 / 921 / 9.5
AVICI18 / 8.531.5 / 3.531.5 / 3

Density matters, but it does not explain every contrast. On the homoscedastic panel, AVICI predicted more reference edges than Arrow (31.5 vs. 21) yet changed fewer pairs (3.5 vs. 9). Table 3 tests whether a given method keeps the same pairs at the same edge count; it is not a density-adjusted ranking of methods.

Light-tunnel agreement with the published reference (new analysis of stored predictions). We verified the standard configuration and that the 18 omitted recorded variables were constant in each evaluated 800-row window. The published graph restricted to our 20 variables has 39 edges. Below are native-graph results for all five methods. The physical column gives medians across the 56 extension environments of one apparatus, excluding the separate initial environment.

MethodTable 11 homosc.: edges / changedTable 11 heterosc.: edges / changedLight tunnel: edges / changed / reference F1
CDFM36.5 / 10.2540 / 10.2525 / 2.50 / 0.730
Arrow21 / 921 / 10.2518.5 / 6.00 / 0.288
AVICI31.5 / 4.531.5 / 3.2510 / 4.50 / 0.235
PC26 / 5.2524.5 / 518 / 2.00 / 0.281
GES39 / 1344.5 / 13.531 / 7.75 / 0.638

If we looked only at disagreement, PC and CDFM would appear similar on the apparatus: both changed few pairs. Their reference F1 scores, 0.281 and 0.730, tell a different story. Low disagreement means a model reports similar adjacencies across scales; it does not mean those adjacencies agree with the physical reference. Here that reference is the published standard-configuration graph, not an independently validated ground truth for each intervention.

AuthorsOpenReview ↗

Response to XF2r (3/3): CDFM path changes persisted when the decoder was forbidden to add edges

CDFM’s fixed-budget decoder admitted below-threshold edges (Q3). Yes: checking the stored predictions, we found below-threshold edges in 77 of 80 decoded graphs (16 linear systems, each with one reference and four transformed views), with a median of 17 such edges per graph. The decoder considered directed edges in descending score order, skipped any edge that would form a cycle, and continued until it reached the native reference prediction’s directed-edge count. The resulting DAG is a score-based projection, not CDFM’s native cyclic prediction.

Arrow’s directed scores. Arrow combines a symmetric edge-existence probability aij=ajia_{ij}=a_{ji} with node-order scores sis_i and sjs_j. Its score for i→ji\to j is pij=aij⋅sigmoid(si−sj)p_{ij} = a_{ij} \cdot \mathrm{sigmoid}(s_i-s_j), with zero diagonal. The two directions therefore share the same evidence for adjacency but receive different weights from the ordering. Our fixed-budget DAG decoder ranks these directed scores, not the symmetric adjacency probabilities, and skips edges that would create a cycle.

Path changes without edge additions (new analysis of the original CDFM scores). We repeated the comparison allowing the decoder only to remove native edges to break cycles, never to add an edge. At least one directed reachability answer still changed in all 64 reference–transformed comparisons. Of the 380 ordered variable pairs, the mean fraction whose answer changed was 10.24% with deletion only (95% bootstrap CI [8.26%, 12.66%]), compared with 12.45% under the submitted fixed-budget decoder.

Deletion-only decoding allows the edge count to vary, so the difference between those percentages does not isolate the effect of adding edges. What the check establishes directly is this: below-threshold additions are not necessary for measurement-scale changes to alter which variables are reachable from which others in the decoded DAG.

Thanks, the added preprocessing experiments and density and decoding checks address some of my questions. The screen contribution still looks limited to me, because stability first reaches about 92.5% precision versus 92.6% for the submitted rule. So actually the weighting formula adds little contribution here. I would also suggest to add the required caution regarding that the physical F1 uses the standard reference graph, which was not independently validated for each intervention, if I understand correctly. In light of the other reviewers' feedback, I see that I initially misunderstood parts of the paper and have updated my score accordingly. Overall, I still view the audit protocol itself positively, provided the above limitations are made clear in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.