- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates whether output text alone can distinguish base models from instruction-tuned models within the same family and scale. Using 16 text statistics and logistic regression, the authors obtain strong within-scale classification performance on Qwen3. Through length controls, question pairing, prefix truncation, and feature-group analysis, they show that cross-scale performance depends substantially on the evaluation protocol. The main contribution is to demonstrate that high black-box lineage classification scores may largely reflect differences in output termination and length, and should therefore be interpreted in the context of the measurement conditions.
Critical strengths
The problem is practically relevant. The paper examines overlooked confounders in black-box model auditing and cautions against interpreting high classification accuracy as evidence of a stable lineage fingerprint.
The control experiments are systematic. Section 3.4 progressively controls output length, question identity, and visible text. Section 4.3 extends the analysis to five model scales and 20 directed transfer pairs, helping assess the stability of the initial findings.
The negative findings are presented cautiously. The authors distinguish performance degradation from evidence that no signal exists. They also disclose problems in earlier attack construction, feature extraction, and evaluation procedures, demonstrating useful transparency.
Cross-family checks provide additional value.
Critical weaknesses
The controls introduce sample-selection effects. Different control conditions retain different subsets of examples, making it difficult to separate the effects of removing confounders from changes in sample composition. Comparisons on a shared subset would strengthen the conclusions.
The transfer results remain statistically inconclusive. Performance deteriorates under stricter controls, but substantial uncertainty remains. The evidence supports sensitivity to evaluation conditions more clearly than the absence of transferable lineage information.
The mechanistic interpretation is not fully established. The feature-group analysis does not clearly demonstrate that termination-related features have a unique explanatory role. The paper should distinguish predictive associations from evidence about the mechanism underlying classification.
The attacks cover a limited threat model. Most attacks preserve the output length and termination behavior identified as important. Their failure therefore provides limited evidence of robustness against interventions that directly target these channels.
Key validation details need clarification. The main text should explain whether data splitting prevents overlap between outputs for the same question and whether confidence estimation accounts for their dependence. These choices affect the reliability of the reported results.
Questions for authors
Can the existing results in Table 1 be reported on the same final subset of questions across conditions, separating the effects of sample selection from those of the controls? If this analysis already exists, please identify its location and key results.
Is within-scale cross-validation grouped by question ID, ensuring that outputs or generation variants for the same question cannot appear in different folds? What is the resampling unit used for bootstrapping?
Given the wide confidence intervals under strict controls and the fact that none of the 20 transfer pairs satisfies H1, what level of useful transfer performance can the current evidence rule out?
Since the random-subset comparison does not establish the special role of the degenerate feature group, what independent evidence supports the claim that termination dominates? Which conclusions should be interpreted only as predictive associations?
Do any existing attacks change output length or termination behavior? If not, would the authors agree to limit the attack conclusion to robustness against surface perturbations that preserve these channels?