- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors study whether a specific causal target can still be recovered when data fragmentation makes some conditional independence evidence unavailable. They characterize the combinations of missing evidence that prevent recovery and analyze the associated failure probabilities.
Critical strengths
- The authors study how data fragmentation can prevent recovery of a specific causal target even when the combined sample has enough support. They separate this effect from ordinary sample scarcity, which helps clarify what goes wrong under their local testing setup.
- The authors explain why adding more CI queries does not always provide useful backup evidence, since different queries may depend on the same poorly represented groups of samples.
Critical weaknesses
- The authors introduce minimal destructive sets and an exact risk calculation, but do not clearly explain what Theorems 1–3 add beyond ruling out incompatible candidate graphs and standard reliability analysis.
- The authors require each CI query to have complete support within at least one individual client, but do not give a sufficiently concrete use case to explain why their stricter setup is needed.
- The authors compare C6, C12aug, and Q40, as well as local and aggregate support, but these are comparisons of evidence sets and interfaces. They do not evaluate an existing federated causal discovery method, so they leave unclear whether the analysis predicts failures when a practical method selects and combines its CI tests, or helps improve that process.
- The authors do not clearly explain the overall analysis or the key experimental settings and results, making the main text difficult to follow and weakening the overall presentation.
Questions for authors
- Could the authors explain how a user should choose the query set and its support thresholds when the true causal graph is unknown?
- Could the authors state what information each client must share to carry out the risk calculation? They should also explain how the joint support probabilities used in the analysis would be obtained in practice.
- Could the authors add an overview diagram and comparative experiments with federated causal discovery methods that fit their setting, to make the analysis easier to follow and its practical value clearer?
4 comments
Part 2 of 3 — Practical method comparisons and experimental evidence (Weaknesses 3–4)
Weakness 3 — No comparison with practical federated methods
We do have end-to-end evidence-to-target evaluations, but not a published full-learner head-to-head. In submitted ALARM (, C12aug), 5,000 paired paths give 73.88% Local / 0% Aggregated clean-erasure failure under the fixed target decoder. For correct definite evidence, the difference decomposes into aggregate scarcity plus pure fragmentation. Submitted Fig. 1(b) additionally tests the C6 versus C12aug/Q40 evidence-family effect on the same target (W2).
A historical homogeneous-client comparison evaluates actual local voting, aggregated-count CI and visibility gating through target recovery.
| CI interface | Definite-answer coverage | Wrong-definite share | Exact target recovery |
|---|---|---|---|
| Local voting | 98.83% | 0.034% | 93.65% |
| Aggregated-count CI | 100% | 3.84% | 78.81% |
| Visibility gate | 41.86% | 0.044% | 5.89% |
This illustrates how missing and incorrect evidence can have different downstream consequences; the historical arms are not matched to the held-out experiment and are not the complete published FedC2SL learner. A new held-out experiment is the strict same-sample, same-readiness-mask/decoder Oracle–actual-test comparison across 1,000 ALARM paths. Local/Aggregated failure 75.3%/0% Oracle, 75.5%/0.8% conditional MC , 100%/100% original TV. With wrong labels the Local-minus-Aggregated risk difference is net, not necessarily pure fragmentation.
Complete FedCSL/FedC2SL learners have not been directly run under harmonised data/target metrics; their communication/statistical interfaces may differ. Thus we claim no complete-learner performance ranking, while emphasising that the existing evaluations do reach the downstream causal target output.
Weakness 4 — Experimental explanation and presentation
The requested overview is provided here, with the paper's running two-hospital example.
TARGET → edge direction / local target summary L_T
| competing DAGs disagree on L_T
v
SUFFICIENT CI CERTIFICATE → correct labels exclude competitors
| T1 → which joint CI losses admit a discrepant DAG?
v
MINIMAL DESTRUCTIVE CI SETS
| each query I requires stratum-block counts above m_I
v
T2 → SHARED SUPPORT-BLOCK CEILINGS
| Hospital A (m,m-1), B (m-1,m) → neither locally ready;
| pooled counts meet m on both strata.
v
T3 → JOINT CLIENT SUPPORT LAW → CLEAN-ERASURE TARGET RISK
| UNDER CLEAN ERASURE → Local failure = aggregate scarcity + pure fragmentation
+→ non-IID clients | extended-access blockers | ALARM thresholds
+→ historical CI interfaces | paired actual-CI/Oracle
The decomposition applies only under clean erasure; actual-CI experiments do not automatically inherit it when definite labels can be wrong. The diagram is provided in this reply, not in the submitted PDF. The submitted three-panel Fig. 1 shows (a) controlled 4.42%→27.10% matched-marginal risk separation, (b) paired ALARM C6 fails/C12aug-Q40 recover at , and (c) T3 predictions 26.31%/73.15% versus observations 26.62%/73.88% for the specified ALARM regimes. The submitted CHILD/GruntingReport contrast is also informative. All three queried families succeed at and fail at while Aggregated succeeds. Those are submitted results; the further CI/support audits are post-submission studies.
Part 3 of 3 — Query selection, information sharing and algorithmic comparisons (Q1–3)
Question 1 — How should users choose a query family and thresholds without the true graph?
Certificate verification need not identify the unique DAG. For correct CI reference labels, a declared candidate DAG class meeting Assumption 1, and a query family, require a nonempty compatible version space whose members are unanimous on the target summary. In finite oracle-audited benchmarks, reference-labelled enumeration plus redundant-query deletion can yield an inclusion-minimal certificate, but not a guaranteed minimum-cardinality, unknown-data trustworthy or efficiently scalable construction.
Choose readiness threshold before looking at test results, from the selected CI test's per-stratum sample/power requirements, then report sensitivity. Our ALARM threshold test shows that even ±5% shifts can drive a sharp transition. This threshold controls availability, not automatic label correctness; heterogeneous pooling must additionally preserve common certificate CI semantics.
Practical lesson from W2/W4. ALARM C6 is sufficient and minimal but fragile at , while C12aug/Q40 add alternative routes and recover. By contrast, CHILD C5/C8aug/Q40 share a Disease support bottleneck and all fail at . Query count alone cannot choose the family. Within an adequate declared class, verify target sufficiency, map destructive sets/shared support blocks, and compare their readiness. This is guidance within the paper's stated assumptions, not a generic automatic query-selection algorithm.
Question 2 — What must clients share, and where do the joint support probabilities come from?
The decision-only CI interface and T3 risk inputs are different. Under decision-only reporting, clients send CI labels or abstentions, not mergeable stratum-level test statistics. T3 separately requires a joint support-state model, given by client 's CDF at blocker-ceiling intersections, or a sampler for complete support vectors. Such inputs may be specified by a sampling law or estimated from joint support states; CI labels alone and per-query readiness marginals do not determine them (Proposition 3; Corollary 2). If full joint laws are unavailable, Corollary 6 (Sec. 4.4/App. B.4) supplies marginal-information bounds, not exact point risk.
In principle, clients could share only selected estimated joint CDF evaluations at those intersections rather than raw observations. Alternatively, given a verified blocker family and a sampler for complete client support vectors, event-indicator Monte Carlo avoids explicitly expanding the up-to- inclusion–exclusion terms. The submitted 17→12→6 ALARM reduction (W1) illustrates why only selected ceilings matter. These are conditional calculation routes, not demonstrated private estimators or communication protocols; statistical accuracy and blocker discovery remain separate challenges.
The paper's Conditional Fixed-Global Disintegration averages placement-conditional risk over realised global tables; the T3 product of distinct client CDFs assumes independent client support-state vectors. One must not use that product solely because pooled counts have been conditioned upon.
Question 3 — Could the authors add an overview and compatible algorithm comparisons?
The overview is already supplied in W4, and the same-target comparisons are explicit in W2/W3. They comprise submitted ALARM matched Local/Aggregated and certificate-family comparisons, natural count-threshold variation, historical local-vote/aggregated-count/visibility-gate CI interfaces through target recovery, and held-out same-path Oracle, TV, asymptotic and conditional MC using the same ALARM readiness and decoder. W3 reports answer coverage, wrong-definite share and target recovery together for the historical interface comparison rather than treating recovery percentages as unexplained rankings.
These validate evidence-formation and target-output consequences within their respective contracts, not an independent evaluation of complete FedCSL/FedC2SL learners. We do not rank such learners. These additional evaluations were conducted after submission, not included in the submitted PDF. The overview is provided in W4 of this response, rather than in the submitted PDF.
Thank you again for your thoughtful questions and suggestions. We have now uploaded the revised manuscript, including a clearer causal-to-support theoretical roadmap, the held-out finite-sample CI results, additional comparisons between local and aggregated evidence interfaces, and more explicit guidance on certificate verification, query selection, and the assumptions underlying the recovery-risk analysis.
We would appreciate your feedback on these revisions and would be grateful if you could reconsider your assessment and score in light of the clarifications and additional results. We would be glad to address any remaining questions.
Part 1 of 3 — Causal contribution and local evidence requirements (Weaknesses 1–2)
We provide the requested causal-to-support overview below and identify the precise evidence-formation interface. The submitted benchmarks and our historical/held-out CI comparisons follow evidence all the way to target recovery, but we do not equate these with running a published FedCSL/FedC2SL full learner or claim learner-level superiority.
Weakness 1 — What is added beyond graph elimination and reliability analysis?
The causal-to-support geometry is the contribution, not inclusion–exclusion. Proposition 3 gives two fixed-global configurations with the same query-wise erasure and blockwise support marginals but pure-fragmentation risk 1/3 versus 0. Corollary 2 strengthens this to matched margins with risk gaps tending to one. T1 derives the minimal joint CI-loss sets that admit target-discrepant DAGs; T2 maps them to shared conditioning-stratum blockers; T3 evaluates their union under joint client support-state laws. Per-query marginal reliability therefore cannot in general determine target risk, even for a fixed certificate.
In submitted Fig. 1(a), a causal five-variable target has 18 non-singleton minimal destructive sets, resilience 2, and identical single-query erasure probabilities 0.1. Avoiding, independent and aligned joint erasures yield 4.42% / 6.28% / 27.10% exact target risk. Submitted ALARM C12aug contracts 17 destructive sets → 12 shared-support regions → six undominated blockers. These are concrete structural examples, not a claimed general polynomial-time algorithm. The paper also proves inclusion-minimal certificates are singleton-fragile, so additional queries can create alternate exclusion routes, or fail to help when they share one support bottleneck (W2/W4).
Weakness 2 — Why assume complete support within one client?
This models a decision-only federation. Each client transmits a query ID and definite local CI label or abstention, not mergeable stratum counts/test statistics. Each query is therefore visible only if one client covers all required strata. Two clients at threshold m with counts (m,m−1) and (m−1,m) are locally unready even though pooled counts are ready. This is a stated interface scenario, not an assertion that all existing learners use it or a claim of deployment.
If statistics are combinable and the pooled CI test preserves the target-certificate semantics (not automatic under heterogeneous mixtures), pre-test aggregation can remove this placement-only loss. Aggregate shortage and wrong CI labels may remain. The matched comparator quantifies the difference under the specified interface.
Submitted Fig. 1(b) illustrates the target-level effect. At ALARM K = 8, N = 160,000, for 5,000 paired within-trial fixed-global repartitions, Local C6 fails 100%, but prespecified C12aug and Q40 succeed 100%, with matched Aggregated success throughout. This is evidence about the causal choice of evidence family under this interface, not universal preference for it. The paper's Conditional Fixed-Global Disintegration distinguishes conditional placement risk from averaging across randomly realised global tables.