- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies design-based minimax risk for network experiments under one-hop interference and a hard budget on outcome measurements. It uses an exposure-conflict graph to represent jointly feasible observations and gives matching rates on bounded-degeneracy networks, separating measurement cost from contrast-specific network difficulty. The theory distinguishes unrestricted from exactly design-unbiased Direct-effect estimation and extends to growing degeneracy. DS-CGD uses a finite library of compatible supports, an optimized mixture, and exact inclusion probabilities for design-unbiased generalized-difference estimation.
Critical strengths
-
The theory gives a useful and precise characterization of the measurement budget. Matching bounds for the measurement and network terms explain when further measurements can improve risk only within a constant factor. The distinction between unrestricted and exactly unbiased Direct-effect estimation also reveals the statistical cost of requiring unbiasedness.
-
The practical design connects feasibility and valid estimation clearly. Compatible supports ensure that the selected exposures can occur together. Conditional on the frozen predictions and design, positive inclusion probabilities and residual correction give design unbiasedness. The authors also explicitly distinguish the finite-library marginal variance objective from a minimax risk guarantee.
Critical weaknesses
My main reservation concerns the historical information used in the practical evaluation. Appendix C.3 uses 200 historical rounds for every node and target exposure to construct predictions and residual scores. Performance with much less history or less accurate residual scores is not established by the reported experiments. The matched methods share this information, so this concerns empirical scope rather than comparison fairness or design unbiasedness.
Questions for authors
Are results with fewer historical rounds or coarser residual scores already available? If so, a brief summary would help clarify how much historical information is needed for the reported gains.
6 comments
Response to Reviewer Yo3r (2/4): Historical sample size and Global effects
Global effects, n = 750, B = 150. Mean RMSE.
| Network | Method | R = 5 | R = 20 | R = 50 | R = 100 | R = 200 |
|---|---|---|---|---|---|---|
| ER | DS-CGD | 0.2356 | 0.2301 | 0.2304 | 0.2309 | 0.2312 |
| ER | Generalized CGD | 0.6797 | 0.6067 | 0.5848 | 0.5769 | 0.5731 |
| ER | Original CGD | 0.9835 | 0.8675 | 0.8320 | 0.8190 | 0.8126 |
| ER | CC | 0.3649 | 0.3468 | 0.3412 | 0.3390 | 0.3377 |
| ER | RGCR | 0.5725 | 0.5609 | 0.5587 | 0.5575 | 0.5568 |
| PA | DS-CGD | 0.1435 | 0.1413 | 0.1452 | 0.1426 | 0.1440 |
| PA | Generalized CGD | 0.9302 | 0.9142 | 0.9103 | 0.9147 | 0.9169 |
| PA | Original CGD | 1.3220 | 1.2981 | 1.2884 | 1.2901 | 1.2889 |
| PA | CC | 0.3895 | 0.3649 | 0.3580 | 0.3551 | 0.3533 |
| PA | RGCR | 1.1392 | 1.0250 | 0.9982 | 0.9887 | 0.9836 |
Global effects, n = 2,000, B = 400. Mean RMSE.
| Network | Method | R = 5 | R = 20 | R = 50 | R = 100 | R = 200 |
|---|---|---|---|---|---|---|
| ER | DS-CGD | 0.0827 | 0.0806 | 0.0804 | 0.0799 | 0.0799 |
| ER | Generalized CGD | 0.3517 | 0.3506 | 0.3505 | 0.3505 | 0.3502 |
| ER | Original CGD | 0.4973 | 0.4962 | 0.4958 | 0.4958 | 0.4955 |
| ER | CC | 0.1404 | 0.1406 | 0.1408 | 0.1410 | 0.1411 |
| ER | RGCR | 0.3386 | 0.3382 | 0.3382 | 0.3381 | 0.3381 |
| PA | DS-CGD | 0.0971 | 0.0962 | 0.0943 | 0.0934 | 0.0937 |
| PA | Generalized CGD | 0.6064 | 0.6047 | 0.6046 | 0.6045 | 0.6042 |
| PA | Original CGD | 0.8543 | 0.8509 | 0.8507 | 0.8503 | 0.8505 |
| PA | CC | 0.2053 | 0.2021 | 0.2014 | 0.2011 | 0.2010 |
| PA | RGCR | 0.6056 | 0.6221 | 0.6294 | 0.6333 | 0.6360 |
Direct-effect figure · Global-effect figure · Anonymous figure page
Response to Reviewer Yo3r (3/4): Residual score sensitivity: rationale and protocol
Q2: Are results with fewer historical rounds or coarser residual scores already available?
Yes. Replacing individual allocation scores with four shared scores per exposure increases mean estimation RMSE by at most 1.9% in our eight settings. Incorrectly assigning scores to other nodes has a larger effect in the covariate-predictable settings. The results for fewer historical rounds are above.
The question is whether DS-CGD needs an accurate score for every node. We test this by deliberately removing detail from the scores. Within each exposure, we sort nodes by their original scores and divide them into 8, 4, 2, or 1 groups of approximately equal size. Every node in a group receives that group's mean score. With two groups, the design sees only a lower-score group and a higher-score group. With one group, it receives no score differences between nodes in that exposure. We then rerun DS-CGD and measure the RMSE of the treatment-effect estimate. This tests whether rough measurement priorities are enough.
We separately test inaccurate priorities by randomly reassigning 25%, 50%, or 100% of scores to other nodes within the same exposure. The score values stay the same, but they no longer describe the same nodes. This tests what happens when the design gives measurement priority to the wrong events.
For event , the score estimates how much outcome variation remains after prediction. Write for the prediction and for historical outcome r. Before the numerical floor and normalization,
\widehat a_v^0&=\frac1R\sum_{r=1}^{R}(Y^{\mathrm{hist}}_{v,r}-\widehat m_v)^2,\\\\ \mathbb E[\widehat a_v^0\mid\widehat m_v] & =\underbrace{\sigma_v^2}_{\text{outcome noise}} +\underbrace{(\mu_v-\widehat m_v)^2}_{\text{prediction error}} =a_v. \end{aligned}$$ Here, $\mu_v$ and $\sigma_v^2$ are the outcome mean and variance. The expectation uses the independence of an event's historical outcomes from the data used to fit its prediction. After the design is fixed, let $Y_v$ be the current outcome, $I_v$ indicate inclusion, and $p_v$ be its probability. The residual correction for this event contributes $$\frac{1}{n^2}\mathbb E\\!\left[ \left(\frac{I_v}{p_v}-1\right)^2(Y_v-\widehat m_v)^2 \right] =\frac{a_v}{n^2}\left(\frac1{p_v}-1\right)$$ to the diagonal variance term. Thus, an inaccurate score matters when it gives too little measurement probability to an event with a large residual. A coarse score can still preserve useful priorities. Assigning the scores to the wrong nodes can lose those priorities. We evaluate the complete estimator's RMSE, which also includes covariance between events. We use ER and PA networks with $n=750$, $B=150$, and $R=200$. Predictions and current outcomes are shared across score variants. For each variant, we rerun both candidate selection and weight optimization. The tables report mean estimation RMSE.
Response to Reviewer Yo3r (1/4): Historical sample size and Direct effects
Rebuttal to Reviewer Yo3r
Thank you for your careful review and for asking how the results depend on historical information.
We now report
R = 5, 20, 50, 100, 200. Historical data help us predict outcomes and estimate allocation scores. Better predictions reduce residual variation. The scores guide measurement toward target exposures with larger expected squared residuals.Let R count historical observations per node–exposure event. Fix a randomization design independently of the historical and current outcome noise. For event v, write the current outcome as Yv=μv+εv and historical outcome r as Yv,rhist=μv+ηv,r. Assume centered noise, independent and identically distributed historical rounds, and independence between historical and current noise. Using the historical mean as the predictor gives mv=μv+R−1∑rηv,r.
Let Iv indicate inclusion of event v, with probability pv>0. Write sv=+1 for the treatment term and sv=−1 for the control term, and set cv=sv(Iv/pv−1)/n. The regression-adjusted estimator then satisfies
\widehat\tau_R-\tau(Y) &=\sum_v c_v(Y_v-\widehat m_v)\\\\ &=\sum_v c_v\varepsilon_v -\frac1R\sum_{r=1}^{R}\sum_v c_v\eta_{v,r}. \end{aligned}$$ Here, $\tau(Y)$ is the current finite-population treatment effect. Conditional on the randomization, the cross terms have zero expectation because the noise is centered and independent across rounds and between history and current outcomes. Thus, averaging over outcomes, history, and randomization, $$\begin{aligned} \mathbb E[(\widehat\tau_R-\tau(Y))^2] &=\mathbb E\\!\left[\left(\sum_v c_v\varepsilon_v\right)^2\right] +\frac1{R^2}\sum_{r=1}^{R} \mathbb E\\!\left[\left(\sum_v c_v\eta_{v,r}\right)^2\right]\\\\ &=V_{\mathrm{current}}+\frac{R}{R^2}V_{\mathrm{history}}\\\\ &=V_{\mathrm{current}}+\frac{V_{\mathrm{history}}}{R}. \end{aligned}$$ $V_{\mathrm{current}}$ is the first expectation: the variance contribution of current outcome noise. $V_{\mathrm{history}}$ is the corresponding contribution of one historical round. More history reduces the historical contribution at rate `1/R`. Once it is small relative to current outcome noise, further history brings little improvement. DS-CGD also updates its predictor and allocation with R, so its RMSE can fluctuate across R. ER and PA networks, n = 750 and 2,000, B = 0.2n, degree-aligned outcomes. Methods share historical inputs and current potential outcomes. Tables report mean RMSE. **Direct effects, n = 750, B = 150. Mean RMSE.** | Network | Method | R = 5 | R = 20 | R = 50 | R = 100 | R = 200 | | :--- | :--- | ---: | ---: | ---: | ---: | ---: | | **ER** | **DS-CGD** | **0.2421** | **0.2404** | **0.2467** | **0.2476** | **0.2466** | | ER | Generalized CGD | 0.3443 | 0.3037 | 0.2917 | 0.2873 | 0.2852 | | ER | Original CGD | 0.3950 | 0.3477 | 0.3315 | 0.3265 | 0.3238 | | ER | Independent set | 0.1924 | 0.1840 | 0.1816 | 0.1807 | 0.1803 | | **PA** | **DS-CGD** | **0.1758** | **0.1741** | **0.1751** | **0.1755** | **0.1751** | | PA | Generalized CGD | 0.3849 | 0.3771 | 0.3752 | 0.3743 | 0.3739 | | PA | Original CGD | 0.4313 | 0.4230 | 0.4212 | 0.4204 | 0.4199 | | PA | Independent set | 0.2992 | 0.2689 | 0.2613 | 0.2584 | 0.2567 | **Direct effects, n = 2,000, B = 400. Mean RMSE.** | Network | Method | R = 5 | R = 20 | R = 50 | R = 100 | R = 200 | | :--- | :--- | ---: | ---: | ---: | ---: | ---: | | **ER** | **DS-CGD** | **0.1045** | **0.0930** | **0.0944** | **0.0937** | **0.0940** | | ER | Generalized CGD | 0.1721 | 0.1713 | 0.1713 | 0.1712 | 0.1713 | | ER | Original CGD | 0.1963 | 0.1957 | 0.1954 | 0.1953 | 0.1953 | | ER | Independent set | 0.1092 | 0.1089 | 0.1088 | 0.1088 | 0.1088 | | **PA** | **DS-CGD** | **0.1047** | **0.1028** | **0.1007** | **0.1010** | **0.1013** | | PA | Generalized CGD | 0.2314 | 0.2306 | 0.2305 | 0.2304 | 0.2304 | | PA | Original CGD | 0.2602 | 0.2593 | 0.2591 | 0.2590 | 0.2590 | | PA | Independent set | 0.1614 | 0.1579 | 0.1570 | 0.1566 | 0.1564 |