- Rating
- 3
- Confidence
- 4(+1)
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper tackles the challenging problem of causal effect estimation in industrial decision-making, specifically dealing with multi-arm treatments, heavy-tailed and zero-inflated outcomes, and severely sparse treatment overlap in the context of e-commerce fulfillment packaging assignment, by proposing MATEL, an end-to-end causal deep-learning framework. Its key advantages lie in its hybrid architecture (combining a tabular backbone with Tweedie outcome heads and a decoupled propensity encoder) and its theoretical pairwise renormalization, which ensures bounded estimates even under extreme sparsity, alongside a non-negative least-squares program that enforces transitive counterfactual levels for a cost-minimization engine. The industrial grounding and empirical results are highly impressive: validated on a massive dataset of 2.5 billion shipments, it achieves 91% directional agreement with observational references and recovers 91%-92% of oracle uplift in synthetic tests, while a production-calibrated simulation reports a 3.74 percentage-point network cost reduction and a 25% cut in corrugated box usage. However, the paper has notable limitations: the theoretical novelty is largely incremental, relying on engineering adaptations of existing methods (e.g., TMLE, Tweedie loss, and NNLS), and the causal assumptions are overly strict, as it strictly assumes unconfoundedness and lacks any sensitivity analysis for hidden confounders, which are highly prevalent in real-world logistics. Overall, despite these theoretical and assumption-related shortcomings, this is a solid and highly practical industrial paper with strong real-world impact.
Critical strengths
The most compelling strength of this paper is its exceptional success in translating causal machine learning into a deployed, large-scale industrial solution that solves real-world pain points. The methodological design is highly effective, elegantly tackling the notorious "sparse overlap" and heavy-tailed data issues that routinely cause traditional estimators (like IPW/DML) to fail in real-world logistics. By introducing pairwise renormalization, the framework successfully provides bounded estimates even under extreme sparsity, ensuring reliable performance on long-tail data. More impressively, it bridges the gap between causal estimation and operational execution by coupling directly with a constrained packaging optimization engine. Validated on a massive scale of 2.5 billion shipments, it delivers a tangible 3.74 percentage-point cost reduction and a 25% cut in corrugated box usage. This high-impact deployment substantiates the practical utility of the approach and provides a highly valuable blueprint for applying causal inference to complex, messy industrial decision-making systems.
Critical weaknesses
While the paper demonstrates impressive industrial impact, it has several critical weaknesses. First, the causal assumptions are overly strict: it strictly assumes unconfoundedness and lacks any sensitivity analysis for hidden confounders, which are highly prevalent in real-world logistics (e.g., unrecorded operator judgment), potentially invalidating the claimed causal estimates. Second, the theoretical contribution is largely incremental, combining existing tools like Tweedie regression and TMLE, yet the paper presents a dense and cluttered array of complex mathematical formulas that obscures the core message. This lack of clarity makes it difficult to follow; since the primary strength lies in practical deployment, the authors should significantly reduce the overwhelming theoretical notation and instead focus more on clearly articulating the real-world engineering insights, the specific challenges of the logistics data, and how the proposed solution actually addresses them.
Questions for authors
I have a few questions and suggestions that could materially affect my assessment. To clarify the robustness of your claims, given that packaging assignment is highly likely driven by unrecorded factors (e.g., operator intuition, unobserved transit conditions), how robust are the reported production gains if hidden confounding exists, and can you provide any sensitivity analysis to quantify this risk? Regarding the empirical validation, since the real-world evaluation uses the model's own AIPW pseudo-outcomes as the reference for BLP and GATES diagnostics, could this self-referential evaluation potentially inflate MATEL's performance compared to the baselines? Furthermore, the synthetic data-generating process appears specifically tailored to favor the proposed pairwise renormalization; could you discuss how the method performs under different, less favorable data-generating distributions? Finally, as a recommendation for improving the manuscript's focus, given that the theoretical novelty is largely incremental (combining existing tools like TMLE and Tweedie), I strongly encourage the authors to streamline the dense array of mathematical formulas and instead dedicate more space to clearly articulating the practical engineering insights, the specific characteristics of the real-world data distribution, and the deployment pipeline, as the true impact of this work lies in its industrial application.
6 comments
Response for 19143
Thanks to the authors for the response; my assessment is now more accurate.
Q2's fair-comparison check and Q3's stress tests can be verified directly in the revision, and Q4's promised Section 4.4, Table 3, and Figure D.9 do appear in the main text. The only remaining uncertainty is Q1: the three robustness checks are informative but none places a causal bound on unobserved confounding. I suggest adding an E-value as the primary bound, which directly answers how strong an unobserved confounder would need to be to drive the 2.87 pp gain to zero, with OVB calibration and a negative-control test as complementary checks.
Overall, I consider this a 3-point paper, and I am willing to raise my confidence to 4.
Follow-up on H43D's feedback
Thank you so much for your prompt and thoughtful reply over the weekend, and for raising your confidence to 4.
Following your suggestion, we added an E-value for the 2.87 pp production saving (Section 4.4 and Appendix D.2). The saving disappears only if true damage under MATEL's decisions is at least 7.7 times its estimate, which corresponds to an E-value of 14.8, so a hidden confounder would need a risk ratio of at least 14.8 with both pack choice and damage to erase the saving. As an even more conservative evaluation, assuming the shadow decisions caused no damage at all, the E-value is still 13.2. In production, realized damage has stayed between 0.97 and 1.08 times MATEL's predictions.
We are grateful for the productive rebuttal discussion and for your kind advice in improving our manuscript. We hope the revision may improve your perspective on the paper quality, and if there is any further improvement that would help, we will be happy to devote our full effort to it before the end of the discussion period. We fully respect your assessment, and iterating on your comments has made the paper substantially stronger. Thank you again for all your efforts on our paper, and we wish you the best in your endeavors!
Rebuttal response to reviewer H43D
Thank you for your thoughtful review and for recognizing the translation of causal machine learning into a deployed, large-scale decision system. We sincerely appreciate the time and effort you devoted to our paper.
Q1. Given that packaging assignment is likely driven by unrecorded factors, provide a sensitivity analysis to quantify this risk.
We agree that unrecorded factors can threaten identification, and we added three analyses to assess this risk.
First, we use the simulation of Section 4.3 as a sensitivity test for biased damage estimates. A hidden factor could make MATEL under- or over-predict damage, which the simulation reproduces by scaling the damage weight from 0 to 10 (Table D.2). Across this range, the saving stays positive at 1.94–3.74 pp, and the 2.87 pp realized in production falls inside it.
Second, data from the five production facilities are consistent with unrecorded factors not materially affecting accuracy (Section 4.4, Table 3): realized damage stays within 8% of MATEL's predictions for both corrugate boxes and flexible bags. Associates follow the packaging recommendation for the large majority of shipments, so unrecorded judgment affects only a small share of decisions. Deviations are mostly upgrades to a corrugate box, the more protective option, which lowers realized damage in operation. Because they are rare, their influence on the estimates is limited, consistent with the agreement between predicted and realized damage.
Third, Appendix D.1 checks the validity of the observational reference. Within product × route, facility type, ship speed, region, and shipment composition all have low standardized mean differences between arms (Figure D.1), and all 45 pairwise contrasts keep their sign when route is held fixed (Figure D.2). The observed contrasts thus show no sign of major confounding.
Q2. Since the real-world evaluation uses the model’s own AIPW pseudo-outcomes, could this inflate MATEL’s performance?
In the original manuscript, all methods were scored against the same doubly robust pseudo-outcome. Following your comment, we now also score DML on a pseudo-outcome built from its own cross-fitted nuisances. DML's BLP slopes improve slightly, but the conclusions hold (Table D.1, Figure D.4; Appendix D.2), so the shared reference does not inflate MATEL's performance.
Q3. The synthetic data-generating process appears specifically tailored to favor the proposed pairwise renormalization.
Inspired by your comment, we added stress tests in Appendix E.1 that compare MATEL with matched baselines under mild and extreme settings (Table E.2). In three scenarios, every pack type is assigned from a model fitted on real data instead of our designed overlap; in two others, damage has lighter or heavier tails than our loss assumes. MATEL outperforms matched variants sharing its backbone and outcome heads when overlap narrows or tails grow heavier, and performs comparably under more uniform assignment and lighter tails (lower ATE error, slightly higher PEHE than HydraNet (adapted)). Across all five settings, its ATE error stays at 192–329 DPMO after reconciliation, with 87–93% of oracle uplift.
Q4. The theoretical novelty is incremental ...notation is dense... add engineering insights, the real-world data distribution, and the deployment pipeline.
We have added Section 4.4 to show the recent production impact of MATEL in five fulfillment facilities, our engineering insights from the deployment along with the pipeline (Appendix D.3, Figure D.9), and the prediction accuracy checks against realized damage data (Table 3). For the real-world data distribution, the original draft already includes Table D.3 and Figures D.5–D.8 in Appendix D. Additionally, we added Appendix D.1 to describe how shipments are aggregated per product, route, and pack type over a rolling one-year window refreshed at each weekly retraining, and to report the distribution of operational covariates across pack types (Figure D.1).
We understand that the dense theory and formulas in the original manuscript were distracting. Following your recommendation, we shortened the theoretical content and streamlined how it is presented. Section 3.3 now states each guarantee in words and cites only the three theorems, the reconciliation proposition, and its corollary. Its formulas and auxiliary results now sit in Appendix C.6, and the remarks were shortened or folded into the statements they qualify.
We would also like to be specific about the novelties. MATEL does build on existing components, including the Tweedie loss and TMLE targeting. Its main contributions are the renormalized pairwise propensity and targeting, whose bounds depend on relative rather than absolute overlap; the hybrid encoder, which keeps outcome-only features out of the propensity; and the transitive reconciliation.
Thank you again for your insightful comments; we hope the revision addresses them.