- Rating
- 1
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies whether censorship can transfer through distillation after target-topic filtering. It introduces a matched China-versus-foreign benchmark and compares filtered and overt training across model scales. The topic is important. However, the main evidence is not sufficient for the central claim: the overt condition changes both content and training dose, while the primary 120B result uses only two seeds and automated labels. The full-panel rejudgment becomes inconclusive, and the reported intervals condition on fixed checkpoints rather than accounting for training-seed uncertainty. I therefore recommend rejection.
Critical strengths
-
The paper asks a timely and important question and correctly warns that a higher false-answer rate alone does not identify selective censorship.
-
The matched benchmark, overlap audit, missing-label analysis, and judge-sensitivity checks are useful efforts toward construct validity.
Critical weaknesses
-
The intervention is confounded. Overt training adds China-related examples, responses, tokens, and optimizer updates, so the overt-minus-filtered contrast cannot isolate subliminal transfer or censorship. There is no same-dose content control or uncensored-teacher control.
-
The decisive 120B result is not robust enough. It uses two training seeds, automated judges, and complete-case selection. Under the four-judge reanalysis, only 10 and 12 concepts remain complete and the confidence interval includes zero. No human-validation results are reported, and the intervals omit training-seed and judge uncertainty.
-
The benchmark is narrow and not demonstrably exposure-free. It is synthetic, predominantly English, contains thematic and named overlap with training material, and the finance comparison has no usable complete behavioral cohort. The claims should therefore be much narrower than selective censorship transfer in general.
Questions for authors
-
What result would remain after matching the overt and filtered arms for training-token count, optimizer updates, and response style, while adding an uncensored-teacher control?
-
Can the main 120B claim be replicated with additional independent training seeds and blinded human raters, with uncertainty propagated over seeds, labels, and missing answers?