- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper identifies a real gap in the NTC literature: shortcut audits never feed back into the pre-training objective. The proposed InvarFlow addresses this issue.
Critical strengths
The paper identifies a real gap in the NTC literature: shortcut audits never feed back into the pre-training objective. The proposed InvarFlow addresses this issue.
Critical weaknesses
One main issue with this paper is the the results in Table 4. Table 4 compares seven different models. Every one of them loses exactly 14.3 points on the first dataset, 11.7 on the second, and 5.8 on the third when moved to a new environment, even though their starting accuracies differ by up to eight points. This is strange and should not happen in real systems. When it happens in synthetic tests, the fidelity of the anthetic test is questionable. I am unsure how much the results hold in realy networking system measurement.
Another issue is that this paper builds its environments by changing timing jitter, packet order, and padding, none of which touch server names, IP addresses, and ports leaking the labels. To teach the model to ignore those, the invariance penalty actually needs training environments where those fields vary. Threre is a mismatch here.
Questions for authors
none
Response to jDjL: Table 4 disclosure, corrected theory, and scope
We thank you for the precise criticism and for noting that the anomaly alone does not establish fabrication. Our audit confirms the cause; full disclosure and retraction are necessary.
Table 4 / ethics concern.
gen_placeholder_tables.py, not seven independent pipelines, generated the table:L1=L0-L1_DROP[dataset],L2=L0-1.3*L1_DROP[dataset]. Thus L0-L2 is forced to 14.3, 11.7, and 5.8, independent of architecture/seed. No independent per-seed runs exist. The script also generated the main ablations. We incorrectly described projections as measured five-seed results and apologize. We retract Table 4, affected/extended ablations, p-values/effect sizes, and all dependent conclusions. The generator and actual controlled-experiment CSVs accompany this response.Proposition 1. You are correct. With shared P(y|c), define r_g(c)=E_{y|c}[loss(g(c),y)]. The semantic Bayes predictor is optimal in every environment, but R^e(g)=E_{P^e(c)}[r_g(c)] depends on the semantic marginal. For loss in [0,B], the valid bound is |R^e(g)-R^e'(g)|<=B TV(P^e(c),P^e'(c)); a Lipschitz version uses W_1. Equal risk requires invariant P^e(c) or constant r_g. We will replace equality with predictor invariance plus marginal-shift control.
Proposition 2 also fails: I(Z;S|C) within environments does not bound divergence between P^e(Z|C) and P^e'(Z|C). We retract the derivation. A valid risk decomposition needs both conditional representation shift and semantic-marginal shift; any sqrt-MI term requires an additional explicit cross-environment assumption.
Cross-dataset / title. No cross-dataset experiments were conducted. We will remove “Generalizable” and restrict the scope to environment shift.
References. CLUB is ICML 2020, GroupDRO ICLR 2020, ET-BERT WWW 2022, and YaTC AAAI 2023. IRM remains a preprint; BiasSeeker is motivation, not peer-reviewed proof of our assumptions. We will state assumptions directly. The leakage result does not imply our floor-b construction; that linkage is retracted.
ET-BERT/YaTC. Not run. Architecture independence is only a formulation property, not an established empirical result; we withdraw that claim.
Gate A0. Ranking agreement cannot exclude shared-shortcut reliance, and semantic preservation of jitter/reordering/padding was not established. Our new stress test is negative: naive shortcut-varying proxies increase Delta from 7.29 (ERM) to 21.10 with high variance. Gate A0 is insufficient; independent coverage, semantic-preservation, and real-L2-alignment tests are required.
Field removal. The only complete controlled comparison is the toy study (5 seeds):
This uses the same generator, backbone, and budget, but is not a real-dataset comparison. We retract the seven-dataset removal claims. In overlap tests, suppression retains 1.24–1.37 more L0 than removal but closes only part of the ERM gap.
Identified fields. The artifacts contain no per-dataset profiler outputs, manual inspections, or false-positive/missed-shortcut analysis. We cannot honestly provide the requested lists and withdraw the validated-identification claim. AMI is only candidate screening; protocol inspection, cross-environment instability, and controlled occlusion/semantic-preservation tests are required.
We understand that these corrections remove much of the claimed empirical support and will not defend unsupported results.