- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Paper Summary
This paper addresses the challenge of reusing historical training updates across disparate neural network architectures without requiring explicit parameter correspondence between the source and target models.
Contributions: The authors propose the Update Tomography (UT) framework, which translates parameter updates from different models into linear measurements within a shared supervision space. The key technical contributions include:
- Theoretical Foundation: A proof demonstrating that for loss functions exhibiting an affine relationship between output gradients and the supervision signal, source model updates can be exactly formulated as linear observations of the supervision signal.
- Target-Relative Observability: The introduction of a novel theoretical condition that characterizes the circumstances under which the gradient contributions required by a target model can be uniquely determined, even when the original supervision signal cannot be fully recovered.
- Algorithmic Realization: A practical mechanism that constructs supervision proxies from the update histories of multiple source models, utilizing the target model's native backpropagation and optimizer to generate the final updates.
Validation: The framework is empirically evaluated across a diverse set of vision and language models. The experiments demonstrate that, provided the information sufficiency conditions are met, UT can accurately reproduce the parameter updates of directly supervised training with minimal error. Furthermore, the validation confirms that UT successfully enables the reuse of historical learning events even in scenarios where the original source models are no longer accessible.
Critical strengths
1. Novel Conceptual Framework for Cross-Architecture Transfer The paper addresses a valuable problem by shifting update transfer from relying on strict parameter correspondence to a shared supervision space. This approach effectively captures common learning information across disparate architectures, offering a conceptually clear and novel analytical framework for reusing historical updates without architectural constraints.
2. Systematic Validation of Theoretical Claims The experiments rigorously validate the target-relative observability condition. By manipulating observation ranks and mappings (Table 2), the authors clearly demonstrate the success and failure regimes predicted by their theory. Furthermore, the study accounts for practical optimizers (e.g., AdamW) and proves that compiled historical records can accurately reproduce target training trajectories without accessing the original source models (Table 3).
Critical weaknesses
1. Unclear Necessity and Lack of Baseline Comparisons The paper fails to justify why exact gradient reconstruction via UT is necessary compared to simpler alternatives, such as saving source model predictions (logits) for knowledge distillation. In core experiments with full-rank measurement operators, UT effectively reduces to label recovery and replay, obscuring its unique advantage. The authors should identify specific scenarios where saving updates is viable but saving labels is not, and directly compare the computational cost, storage efficiency, and downstream performance of UT against these simpler baselines.
2. Disconnect Between Reconstruction Fidelity and Performance Gains High reconstruction fidelity does not inherently translate to downstream improvements. For example, in CIFAR-100 experiments (Appendix D.1), both UT and direct training degrade accuracy, and certain settings (Appendix Table 19) yield a zero update coefficient. The paper lacks a systematic analysis linking reconstruction precision to actual performance gains. The authors should investigate how update fidelity correlates with downstream improvements across varying error levels to substantiate the practical value of exact reconstruction.
Questions for authors
Q1. Please further clarify the practical application scenarios and core value of UT. Specifically, in what real-world scenarios does UT offer irreplaceable advantages over simply saving the original supervision signals or source model predictions (e.g., logits) for subsequent training?
4 comments
Response 2/2: Update Fidelity and Downstream Task Performance (W2)
R2: Update Reconstruction Fidelity and Downstream Task Gains (W2)
We thank the reviewer for the thoughtful analysis of the relationship between update fidelity and downstream performance. Your comments prompted us to distinguish faithful update reconstruction from the usefulness of the learning event itself, and to add long-horizon training and perturbation experiments along different error directions.
W2: Relationship Between Update Fidelity and Task Performance
Weakness 2 (summary): High-fidelity update reconstruction does not necessarily improve downstream performance. In the original CIFAR-100 experiments, both UT and direct supervision sometimes reduced accuracy, while some update coefficients in Appendix Table 19 were even zero. The paper should examine how reconstruction errors affect task gains to clarify the practical value of faithful reconstruction.
Response W2:
1. Update fidelity and learning benefit are different questions. UT aims to faithfully reconstruct the target update induced by historical supervision; whether that update is beneficial also depends on the target's current state. Table 28 of the revised manuscript already illustrates this distinction: for the original target head at 91.753% accuracy, validation selected a zero update for both UT and direct supervision. For the refitted target head, both selected a 1.5× update, increasing accuracy from 77.062% to 78.519% (Table R3). Thus, a zero update does not indicate a failure of reconstruction; under the same validation criterion, it reflects whether the learning event should be applied.
To examine benefits over sustained training, we additionally ran three-seed, long-horizon CIFAR-100 experiments. UT improved test accuracy from 72.253% to 80.837%, closely matching direct supervision at 80.833%. This shows that, when the historical supervision supports useful learning, UT can preserve its task benefits over thousands of optimization steps rather than merely reproduce a local parameter change.
2. How do reconstruction errors affect final performance? We conducted two complementary experiments. First, at nine training states spanning early, middle, and late training across three seeds, we injected random errors and errors along weakly observable directions at different magnitudes, then compared parameter deviations and validation performance over the following five steps. With the same 1% measurement perturbation, weakly observable directions produced 2.79–4.86× larger trajectory errors than random directions, while the resulting NLL changes did not increase strictly with trajectory deviation.
We then applied persistent perturbations throughout one complete 3,872-step CIFAR-100 run (Table R4). Standard UT had a trajectory error of only 7.495×10⁻⁶; persistent 10% perturbations along weakly observable directions increased it to 0.2477, with test accuracy decreasing from 80.51% to 80.31%. Random perturbations of the same magnitude also caused noticeable trajectory deviation, yet accuracy remained 80.54%. Thus, both the magnitude and direction of reconstruction errors affect learning trajectories, while parameter distance and final task gains do not have a simple one-to-one relationship.
These results clarify the connection between our theory and its use: UT's information-sufficiency condition determines whether the historical supervised update is identifiable, while the target state and training process determine whether applying that update improves performance. The revised Section 6.3 and Appendix D.3.4 report this analysis and distinguish update fidelity from downstream task gains.
Table R3. Original Table 19: Validation-selected update application (Pet, n=800)
| Target initialization | Initial accuracy | UT / direct update coefficient | Accuracy after UT | Accuracy after direct supervision |
|---|---|---|---|---|
| Original target head | 91.753% | 0 / 0 | 91.753% | 91.753% |
| Refitted target head | 77.062% | 1.5 / 1.5 | 78.519% | 78.519% |
Table R4. New CIFAR-100 long-horizon perturbation experiment
| Training condition | Final relative trajectory error | Test accuracy (%) | Test NLL |
|---|---|---|---|
| Direct supervision | 0 | 80.51 | 0.764024 |
| Standard UT | 7.495×10⁻⁶ | 80.51 | 0.764024 |
| 1% random perturbation per step | 5.862×10⁻³ | 80.51 | 0.764097 |
| 1% weak-direction perturbation per step | 2.640×10⁻² | 80.53 | 0.764132 |
| 10% random perturbation per step | 5.705×10⁻² | 80.54 | 0.764977 |
| 10% weak-direction perturbation per step | 2.477×10⁻¹ | 80.31 | 0.770376 |
Table R4 reports error interventions from one complete training seed; local five-step comparisons were also conducted at nine training states across three seeds.
I thank the authors for their clarifications and additional experiments, which address my main concerns, I maintain my positive score.
Thank you for your initial positive assessment and careful review. We are also glad that these clarifications and additional experiments addressed your concerns. Your feedback was very important for improving the paper, and we sincerely appreciate your time and constructive comments.
Response 1/2: Research Value, Baselines, and Applications (W1/Q1)
R1: Research Value, Baseline Comparisons, and Practical Applications (W1 / Q1)
We thank the reviewer for recognizing our cross-architecture learning perspective and target-relative sufficiency theory. Your suggestions concerning label archival, distillation from source predictions, and practical applications helped clarify the roles of different forms of learning information. We have added long-horizon task comparisons, compute and storage measurements, selective-sharing tests, and experiments on reusing historical supervision.
W1: Why UT, and how does it compare with simpler baselines?
Response W1:
UT's central contribution is not a cheaper way to archive labels, but a mathematical account of historical heterogeneous updates as measurements in a common supervision space, including whether those measurements suffice to determine a future target update. Unlike simply storing labels, this framework characterizes target-relative sufficiency even when supervision is not fully identifiable. Unlike distillation from stored source logits, UT retains constraints imposed by the historical learning event on supervision; when these constraints are sufficient, it reconstructs the target's native update under the original supervision rather than treating source predictions as new training targets.
We added a three-seed CIFAR-100 long-horizon cross-architecture experiment (Table R1). UT improved target accuracy from 72.25% to 80.84%, closely matching direct supervision (80.83%), while source-output KD reached 79.50%. We also measured the costs: compiling a two-source event took 10.14 ms; target steps took 245.39 ms with UT versus 249.70 ms with direct supervision. A general record required 108.4 KB per event, reducible to 21.2 KB by archiving a recovered supervision representative when supervision was fully identifiable.
Q1: Concrete applications and core value
Response Q1:
1. Selective sharing of learning information. When supervision remains private at the source and future targets can access only a restricted shared view, selective measurements on public inputs can convey the retained learning information to heterogeneous targets. In a new three-client collaboration experiment, UT and equal-information coarse-class KD both achieved 77.80% mean test accuracy. A further leakage audit found that full records allowed 100% fine-label recovery under the tested attack, whereas selective official-20 records yielded 84.23%, matching the coarse-label baseline.
2. One-to-many reuse of valuable historical supervision. When future models cannot revisit expert assessments or multi-annotator supervision used in a past learning event, its recorded measurements can be reused across architectures. In our new CIFAR-10H experiment, we captured one real ViT-B/32 update using human soft supervision and reused the record for 20 full-parameter steps on each of three heterogeneous targets, comparing against direct soft labels, hard labels, and source-output KD (Table R2). UT closely matched direct soft supervision on all three targets, without aligning source and target parameter spaces.
These examples illustrate UT's role in interpreting, selectively retaining, and reusing historical learning information. The revised Sections 6.2–6.3 and Appendix D.3 report the access conditions, task baselines, costs, and reuse experiments.
Table R1. New CIFAR-100 long-horizon comparison (mean ± SD)
Three fixed seeds; the EfficientNet-B0 target underwent full-parameter training for 3,872–7,392 steps with validation-based early stopping.
Table R2. New one-to-many reuse of human soft supervision (Soft NLL ↓)
Means over three data splits; task-level benefits may vary across targets.