- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether behavioral evidence validated for one model version still holds for an update with unchanged accuracy, which it calls evidence inheritance. It proposes a claim-aligned drift measure (FD) and uses controlled toy experiments (small MLPs and an MNIST–CIFAR CNN) to show that an update can keep accuracy stable yet break an old claim. Weight change alone doesn’t predict this: one update moves the weights further and still leaves the claim intact. Its conclusion is that continuity must be judged per claim, not by performance.
Critical strengths
The conceptual problem is novel. Prior work on prediction churn, backward compatibility and shift detection compares outputs or test results. This paper instead asks whether a specific historical claim , validated for a reference model , still holds for an updated model .
The formulation is clear and easy to follow. It cleanly separates performance continuity, , the ground-truth consequence, , and claim-aligned drift, . It also distinguishes evidence inheritance from evidence refresh.
Critical weaknesses
The experiments are small in scale, as the authors use tiny MLPs on 2–3-dimensional synthetic inputs and a three-layer CNN on an MNIST/CIFAR mosaic, with hand-designed updates and a single claim type. Whether the phenomenon arises in realistic updates such as fine-tuning or compression remains untested.
The headline evidence is partly driven by construction. Because the Level 1 arms are exact reparameterizations, is guaranteed, so the significant gaps in Tables 3 and 5 are largely arithmetic. Although FD and DTL are measured on independent samples and are not equivalent by definition, the hand-designed alignment among the migration, the intervention, and the claim may favor FD. Its usefulness under less controlled updates thus remains unestablished, and the more informative Level 2 control is limited to a synthetic MLP.
Questions for authors
How does FD compare with simply rerunning the frozen diagnostic, i.e. computing directly? Since mapping FD to DTL already requires DTL labels, in what practical setting would FD be preferable?
Can the Level 2 orthogonal control be extended beyond the synthetic MLP, for instance to the stacked-image setting of Section 4.3 or to a realistic update chain on a standard spurious-correlation benchmark?