acceptodds
Under review as a conference paper at ICLR 2027

Update Tomography: Reusable Learning Records via Supervision Space

Abstract

Unlike homogeneous models, updates from heterogeneous architectures cannot be directly compared in a shared parameter space. We therefore turn to supervision space and show that updates from different architectures can be represented exactly as linear measurements of the same supervision contrast. Moreover, a target update remains uniquely determined as long as the supervision ambiguity left unresolved by these measurements cannot affect that target, even when the full supervision itself is not uniquely identified.We formalize these results as Update Tomography (UT), a framework for constructing updates across heterogeneous architectures without parameter alignment. Experiments on vision and language models agree with the theoretical predictions: even when full supervision cannot be recovered, UT reproduces direct-supervision learning whenever the information required by the target is retained. This learning information remains usable by heterogeneous models introduced after the original models are no longer available, with relative trajectory error remaining on the order of across ten successive learning events.Thus, cross-architecture update reuse need not treat model-specific parameter changes as the object to be transferred, but can instead characterize and reuse the information that a learning event leaves in supervision space.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Update Tomography, Heterogeneous Model Updates, Supervision Space, Target-Relative Observability, Cross-Architecture Learning.

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_4wfYOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Paper Summary

This paper addresses the challenge of reusing historical training updates across disparate neural network architectures without requiring explicit parameter correspondence between the source and target models.

Contributions: The authors propose the Update Tomography (UT) framework, which translates parameter updates from different models into linear measurements within a shared supervision space. The key technical contributions include:

  1. Theoretical Foundation: A proof demonstrating that for loss functions exhibiting an affine relationship between output gradients and the supervision signal, source model updates can be exactly formulated as linear observations of the supervision signal.
  2. Target-Relative Observability: The introduction of a novel theoretical condition that characterizes the circumstances under which the gradient contributions required by a target model can be uniquely determined, even when the original supervision signal cannot be fully recovered.
  3. Algorithmic Realization: A practical mechanism that constructs supervision proxies from the update histories of multiple source models, utilizing the target model's native backpropagation and optimizer to generate the final updates.

Validation: The framework is empirically evaluated across a diverse set of vision and language models. The experiments demonstrate that, provided the information sufficiency conditions are met, UT can accurately reproduce the parameter updates of directly supervised training with minimal error. Furthermore, the validation confirms that UT successfully enables the reuse of historical learning events even in scenarios where the original source models are no longer accessible.

Critical strengths

1. Novel Conceptual Framework for Cross-Architecture Transfer The paper addresses a valuable problem by shifting update transfer from relying on strict parameter correspondence to a shared supervision space. This approach effectively captures common learning information across disparate architectures, offering a conceptually clear and novel analytical framework for reusing historical updates without architectural constraints.

2. Systematic Validation of Theoretical Claims The experiments rigorously validate the target-relative observability condition. By manipulating observation ranks and mappings (Table 2), the authors clearly demonstrate the success and failure regimes predicted by their theory. Furthermore, the study accounts for practical optimizers (e.g., AdamW) and proves that compiled historical records can accurately reproduce target training trajectories without accessing the original source models (Table 3).

Critical weaknesses

1. Unclear Necessity and Lack of Baseline Comparisons The paper fails to justify why exact gradient reconstruction via UT is necessary compared to simpler alternatives, such as saving source model predictions (logits) for knowledge distillation. In core experiments with full-rank measurement operators, UT effectively reduces to label recovery and replay, obscuring its unique advantage. The authors should identify specific scenarios where saving updates is viable but saving labels is not, and directly compare the computational cost, storage efficiency, and downstream performance of UT against these simpler baselines.

2. Disconnect Between Reconstruction Fidelity and Performance Gains High reconstruction fidelity does not inherently translate to downstream improvements. For example, in CIFAR-100 experiments (Appendix D.1), both UT and direct training degrade accuracy, and certain settings (Appendix Table 19) yield a zero update coefficient. The paper lacks a systematic analysis linking reconstruction precision to actual performance gains. The authors should investigate how update fidelity correlates with downstream improvements across varying error levels to substantiate the practical value of exact reconstruction.

Questions for authors

Q1. Please further clarify the practical application scenarios and core value of UT. Specifically, in what real-world scenarios does UT offer irreplaceable advantages over simply saving the original supervision signals or source model predictions (e.g., logits) for subsequent training?

4 comments
AuthorsOpenReview ↗

Response 1/2: Research Value, Baselines, and Applications (W1/Q1)

R1: Research Value, Baseline Comparisons, and Practical Applications (W1 / Q1)

We thank the reviewer for recognizing our cross-architecture learning perspective and target-relative sufficiency theory. Your suggestions concerning label archival, distillation from source predictions, and practical applications helped clarify the roles of different forms of learning information. We have added long-horizon task comparisons, compute and storage measurements, selective-sharing tests, and experiments on reusing historical supervision.

W1: Why UT, and how does it compare with simpler baselines?

Weakness 1 (summary): The paper does not sufficiently explain why UT is needed instead of simply storing labels or source-model logits for knowledge distillation. Full-rank experiments resemble label recovery. Task performance, compute and storage costs, and UT's distinctive contribution need clearer comparisons.

Response W1:

UT's central contribution is not a cheaper way to archive labels, but a mathematical account of historical heterogeneous updates as measurements in a common supervision space, including whether those measurements suffice to determine a future target update. Unlike simply storing labels, this framework characterizes target-relative sufficiency even when supervision is not fully identifiable. Unlike distillation from stored source logits, UT retains constraints imposed by the historical learning event on supervision; when these constraints are sufficient, it reconstructs the target's native update under the original supervision rather than treating source predictions as new training targets.

We added a three-seed CIFAR-100 long-horizon cross-architecture experiment (Table R1). UT improved target accuracy from 72.25% to 80.84%, closely matching direct supervision (80.83%), while source-output KD reached 79.50%. We also measured the costs: compiling a two-source event took 10.14 ms; target steps took 245.39 ms with UT versus 249.70 ms with direct supervision. A general record required 108.4 KB per event, reducible to 21.2 KB by archiving a recovered supervision representative when supervision was fully identifiable.

Q1: Concrete applications and core value

Question 1 (summary): In what practical settings does UT offer distinctive value relative to directly storing supervision or source-model predictions?

Response Q1:

1. Selective sharing of learning information. When supervision remains private at the source and future targets can access only a restricted shared view, selective measurements on public inputs can convey the retained learning information to heterogeneous targets. In a new three-client collaboration experiment, UT and equal-information coarse-class KD both achieved 77.80% mean test accuracy. A further leakage audit found that full records allowed 100% fine-label recovery under the tested attack, whereas selective official-20 records yielded 84.23%, matching the coarse-label baseline.

2. One-to-many reuse of valuable historical supervision. When future models cannot revisit expert assessments or multi-annotator supervision used in a past learning event, its recorded measurements can be reused across architectures. In our new CIFAR-10H experiment, we captured one real ViT-B/32 update using human soft supervision and reused the record for 20 full-parameter steps on each of three heterogeneous targets, comparing against direct soft labels, hard labels, and source-output KD (Table R2). UT closely matched direct soft supervision on all three targets, without aligning source and target parameter spaces.

These examples illustrate UT's role in interpreting, selectively retaining, and reusing historical learning information. The revised Sections 6.2–6.3 and Appendix D.3 report the access conditions, task baselines, costs, and reuse experiments.


Table R1. New CIFAR-100 long-horizon comparison (mean ± SD)

MethodTest accuracy (%)Test NLL
No update72.253 ± 0.0311.1160 ± 0.0012
Direct supervision80.833 ± 0.2880.7532 ± 0.0094
UT80.837 ± 0.2910.7532 ± 0.0094
Same-information representative80.837 ± 0.2910.7532 ± 0.0094
Source-output KD79.503 ± 0.0990.9335 ± 0.0024

Three fixed seeds; the EfficientNet-B0 target underwent full-parameter training for 3,872–7,392 steps with validation-based early stopping.

Table R2. New one-to-many reuse of human soft supervision (Soft NLL ↓)

TargetNo replayDirect soft labelsUTHard labelsSource-output KD
MobileNetV3-small0.53930.53380.53390.53590.5419
ResNet-180.57880.59250.59240.67980.6146
EfficientNet-B00.65280.56560.56560.63040.5434

Means over three data splits; task-level benefits may vary across targets.

AuthorsOpenReview ↗

Response 2/2: Update Fidelity and Downstream Task Performance (W2)

R2: Update Reconstruction Fidelity and Downstream Task Gains (W2)

We thank the reviewer for the thoughtful analysis of the relationship between update fidelity and downstream performance. Your comments prompted us to distinguish faithful update reconstruction from the usefulness of the learning event itself, and to add long-horizon training and perturbation experiments along different error directions.

W2: Relationship Between Update Fidelity and Task Performance

Weakness 2 (summary): High-fidelity update reconstruction does not necessarily improve downstream performance. In the original CIFAR-100 experiments, both UT and direct supervision sometimes reduced accuracy, while some update coefficients in Appendix Table 19 were even zero. The paper should examine how reconstruction errors affect task gains to clarify the practical value of faithful reconstruction.

Response W2:

1. Update fidelity and learning benefit are different questions. UT aims to faithfully reconstruct the target update induced by historical supervision; whether that update is beneficial also depends on the target's current state. Table 28 of the revised manuscript already illustrates this distinction: for the original target head at 91.753% accuracy, validation selected a zero update for both UT and direct supervision. For the refitted target head, both selected a 1.5× update, increasing accuracy from 77.062% to 78.519% (Table R3). Thus, a zero update does not indicate a failure of reconstruction; under the same validation criterion, it reflects whether the learning event should be applied.

To examine benefits over sustained training, we additionally ran three-seed, long-horizon CIFAR-100 experiments. UT improved test accuracy from 72.253% to 80.837%, closely matching direct supervision at 80.833%. This shows that, when the historical supervision supports useful learning, UT can preserve its task benefits over thousands of optimization steps rather than merely reproduce a local parameter change.

2. How do reconstruction errors affect final performance? We conducted two complementary experiments. First, at nine training states spanning early, middle, and late training across three seeds, we injected random errors and errors along weakly observable directions at different magnitudes, then compared parameter deviations and validation performance over the following five steps. With the same 1% measurement perturbation, weakly observable directions produced 2.79–4.86× larger trajectory errors than random directions, while the resulting NLL changes did not increase strictly with trajectory deviation.

We then applied persistent perturbations throughout one complete 3,872-step CIFAR-100 run (Table R4). Standard UT had a trajectory error of only 7.495×10⁻⁶; persistent 10% perturbations along weakly observable directions increased it to 0.2477, with test accuracy decreasing from 80.51% to 80.31%. Random perturbations of the same magnitude also caused noticeable trajectory deviation, yet accuracy remained 80.54%. Thus, both the magnitude and direction of reconstruction errors affect learning trajectories, while parameter distance and final task gains do not have a simple one-to-one relationship.

These results clarify the connection between our theory and its use: UT's information-sufficiency condition determines whether the historical supervised update is identifiable, while the target state and training process determine whether applying that update improves performance. The revised Section 6.3 and Appendix D.3.4 report this analysis and distinguish update fidelity from downstream task gains.

Table R3. Original Table 19: Validation-selected update application (Pet, n=800)

Target initializationInitial accuracyUT / direct update coefficientAccuracy after UTAccuracy after direct supervision
Original target head91.753%0 / 091.753%91.753%
Refitted target head77.062%1.5 / 1.578.519%78.519%

Table R4. New CIFAR-100 long-horizon perturbation experiment

Training conditionFinal relative trajectory errorTest accuracy (%)Test NLL
Direct supervision080.510.764024
Standard UT7.495×10⁻⁶80.510.764024
1% random perturbation per step5.862×10⁻³80.510.764097
1% weak-direction perturbation per step2.640×10⁻²80.530.764132
10% random perturbation per step5.705×10⁻²80.540.764977
10% weak-direction perturbation per step2.477×10⁻¹80.310.770376

Table R4 reports error interventions from one complete training seed; local five-step comparisons were also conducted at nine training states across three seeds.

I thank the authors for their clarifications and additional experiments, which address my main concerns, I maintain my positive score.

Thank you for your initial positive assessment and careful review. We are also glad that these clarifications and additional experiments addressed your concerns. Your feedback was very important for improving the paper, and we sincerely appreciate your time and constructive comments.

Reviewer_UdnPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Update Tomography (UT), a framework for transferring learning information across heterogeneous neural network architectures without access to the original training labels. The method converts source model updates into reusable supervision records, which are used to construct target-specific updates. Experiments demonstrate accurate target-update reconstruction and reuse of archived learning records across different architectures.

Critical strengths
  1. Novelty: The idea of constructing reusable supervision records from heterogeneous model updates and using them to transfer learning information to different target architectures is interesting and appears to offer a novel perspective on cross-architecture learning transfer.
  2. The experimental results demonstrate that the proposed framework can reconstruct target updates with high numerical accuracy across heterogeneous neural network architectures and multiple learning events.
Critical weaknesses
  1. The authors suggest applications involving rare learning events, expensive human annotations, and expert interventions. However, it is unclear when preserving supervision measurements would be more useful than simply storing the original training inputs and labels. Since UT still requires the original inputs and target-side backpropagation, the practical advantage is not clearly established.
  2. Storage scalability is unclear. The framework generally constructs a separate record for each learning event. Although the authors demonstrate source-independent compilation and retain only selected measurements, the total storage requirement may become substantial for training involving thousands or millions of distinct batches. The paper does not sufficiently evaluate archive-size growth or demonstrate efficient compression across learning events.
  3. Theorem 4.1 characterizes when a retained record contains sufficient information for a specified target model. However, it does not provide a general method for constructing compact records that remain sufficient for arbitrary future architectures. This limits the practical applicability of the proposed historical-replay framework, particularly when the future target model is unknown at record creation time.
  4. While the authors demonstrate highly accurate reconstruction of individual target updates and short learning trajectories, the experiments do not evaluate complete end-to-end model training till convergence using UT records
Questions for authors
  1. Could the authors provide a concrete use case in which storing compiled supervision records is more efficient or advantageous than retaining the original labeled training data?
  2. How does the total storage requirement scale with the number of training events? Can records from different batches be compressed or consolidated without losing the information needed for future target updates?
5 comments
AuthorsOpenReview ↗

Response 1/3: Value and Reuse of Historical Learning Records (W1/Q1)

R1: Value and Reuse of Historical Learning Records (W1 / Q1)

We thank the reviewer for recognizing the cross-architecture learning perspective. Your comments clarify the theory and its practical conditions. We added sufficiency, archival, selective-sharing, and human-supervision experiments.

W1: Learning Records Versus Original Supervision

W1 (summary): Why retain supervision measurements rather than original inputs and labels for rare events or costly annotations, when UT still requires inputs and backpropagation?

Response W1:

UT does more than convert heterogeneous updates into reusable records: it characterizes what supervision information they retain and whether it suffices for a future target. Proposition 3.1 relates actual optimizer transitions to measurements in a common supervision space. Their rank and null space characterize identifiable supervision directions; Theorem 4.1 further determines whether a target update is unique even when supervision is not. This characterizes both what a record contains and which targets it can serve.

New AG News and real CIFAR tests (Table R1) illustrate this distinction: one incomplete record suffices for a 96-token deep target but not a longer-context target, while source-invisible changes in CIFAR alter target gradients. The key is not simply how much supervision is recovered, but whether the retained information suffices for the intended target.

When original labels can be retained and reused, direct archival is cheaper. We also tested a natural simplified UT realization at full identifiability: recovered representatives reduce Pet/DomainNet records from 2.38/17.71 MiB to 0.084/2.54 MiB with equivalent updates. Record form can therefore depend on information completeness and target needs.

Table R1. New tests: retained information and target-relative sufficiency

ExperimentSource informationTarget response
AG News, 96 tokens3D supervision ambiguity66M DistilBERT gradient 5.44–6.41×10⁻⁸
AG News, 256 tokensSame record; longer contextGradient 0.0422
CIFAR-100Real rank deficiency 127/128Source measurement nearly unchanged; target gradient changes 15.07%

AG News: absolute L2 response; CIFAR: relative gradient change.

Q1: Concrete Use Cases and Experimental Evidence

Q1 (summary): In what concrete setting do compiled supervision records offer an efficiency or practical advantage over keeping the original labeled data?

Response Q1:

1. Selective sharing across institutions. Where labels cannot be shared, institutions can train locally and release selective measurements on public inputs. In our new three-client experiment, these public-probe records trained a heterogeneous ViT target; UT and same-information coarse KD both reached 77.80% mean test accuracy.

We also audited the information exposed by selective records. Across 4,997 images and 40 real source events, the tested attack recovered all fine labels from full records, while official-20 records matched the coarse-label baseline and preserved accurate authorized updates (Table R2). This demonstrates selective information retention, not a formal privacy guarantee.

Table R2. New selective-sharing test: exposure and target fidelity

Released informationFine-label inferenceExtra vs. coarseAuthorized update error (max)
Inputs + coarse labels84.23%0 pp0
Inputs + official-20 record84.23%0 pp1.955×10⁻⁵
Inputs + full fine-100 record100.00%+15.77 pp1.568×10⁻⁵

Leakage uses the tested attack; update errors use a separate prespecified event subset.

2. One-to-many reuse of valuable human supervision. When future models cannot reaccess past expert or human annotations, the event record can still serve different architectures. In a new CIFAR-10H experiment, one real ViT-B/32 AdamW event was reused by MobileNetV3-small, ResNet-18, and EfficientNet-B0 for 20 full-parameter updates each, compared with direct soft supervision, hard labels, and source KD (Table R3).

Table R3. New one-to-many human-supervision reuse (Soft NLL↓)

TargetNo replayDirect softUTHard labelsSource KD
MobileNetV3-small0.53930.53380.53390.53590.5419
ResNet-180.57880.59250.59240.67980.6146
EfficientNet-B00.65280.56560.56560.63040.5434

Means over three splits; utility varies across targets.

The same event reproduces the effect of human soft supervision across architectures without parameter-space alignment. This applies when access is restricted; it does not replace direct training when labels remain available. The revised Sections 5–7 and Appendix D.3 cover the retained information and these applications.

AuthorsOpenReview ↗

Response 2/3: Storage Scaling and Future-Target Sufficiency (W2/Q2/W3)

R2: Storage Scaling, Cross-Event Compression, and Future-Target Sufficiency (W2 / Q2 / W3)

We thank the reviewer for examining the storage cost and future applicability of learning records. These questions also highlight a central issue in supervision space: how to preserve sufficient learning information while avoiding redundant storage. We added per-event encoding measurements, cross-event consolidation, and targeted information-completion experiments.

W2: Storage scaling across learning events

Weakness 2 (summary): UT stores separate records for individual events, potentially leading to substantial costs over thousands or millions of batches. Storage growth and compression efficiency need evaluation.

Response W2:

Storing distinct events independently incurs a cost that grows with the number of events; compiling source records does not itself remove cross-event redundancy. We therefore audited actual encoded sizes in the CIFAR-100 long-run experiment (Table R4). Across 3,872 / 7,392 events, general U/Z records total 389.8 / 744.2 MiB. When supervision is fully identifiable, archiving its representative instead reduces these totals to 76.4 / 145.7 MiB, a 5.11× reduction. We also report a linear projection to 10,000 events to make the scaling explicit.

Q2: Can records from different batches be compressed or merged?

Question 2 (summary): How does storage grow with the number of events, and can records across batches be compressed or consolidated without losing information needed by future targets?

Response Q2:

Cross-event consolidation is possible when events share reusable supervision information. With known sample identities and unchanged supervision, repeated representatives need only be stored once. In a new experiment involving three real events, 384 sample occurrences, and 256 unique samples, merging by sample ID reduced archival size from 60,022 B to 39,298 B (−34.5%), with reconstruction difference 1.087×10⁻⁷ (Table R5).

The gain comes from verified overlap, not from indiscriminately merging optimizer updates. Distinct samples, changing supervision, or unknown correspondences require their own constraints. These conditions are now stated explicitly in Appendix D.3.2 of the revised manuscript.

W3: Records for future, as-yet-unknown targets

Weakness 3 (summary): Theorem 4.1 establishes sufficiency for a specified target, but the paper does not provide a compact record guaranteed to serve arbitrary future architectures.

Response W3:

A target being unknown at capture time is different from requiring one compressed record to suffice for every possible target. If supervision is fully identified, a recovered representative can serve architectures selected later. Under partial identification, sufficiency depends on which missing supervision directions a future target can detect. For a target family T\mathcal{T}, Theorem 4.1 yields the joint condition

ker⁡RL⊆⋂T∈Tker⁡TL.\ker R_{\mathcal L}\subseteq\bigcap_{T\in\mathcal T}\ker T_{\mathcal L}.

Thus, keeping more measurements broadens the set of supported targets, while target-aware retention permits further compression; lossy compression cannot guarantee sufficiency for arbitrary unknown targets without additional assumptions. We also tested targeted completion on a real rank-deficient CIFAR event: adding just 619 B of informative measurement reduced the target's relative gradient error from 7.537×10⁻² to 6.019×10⁻⁷. Such completion requires the source side to remain able to supply that measurement.

The revised Section 6.3 and Appendix D.3.2 report the storage and consolidation results, while Sections 4–5 explain how retained information determines future-target sufficiency.


Table R4. New archive-size measurements across learning events

EventsGeneral U/Z recordFP32 supervision representative
3,872 (measured)389.815 MiB76.358 MiB
7,392 (measured)744.173 MiB145.704 MiB
10,000 (linear estimate)1,033.56 MiB202.39 MiB

Measured totals sum actual per-event encodings, excluding shared inputs, model weights, and optimizer states; the 10,000-event row is an extrapolation.

Table R5. New cross-event consolidation experiment

Archival scheme (three events)Storage
Independent U/Z records318,222 B
Independent FP32 representatives60,022 B
Representatives merged by sample ID39,298 B
Known fixed smoothing family: indices + rule2,408 B

Three events contain 384 sample occurrences and 256 unique samples; ID-based consolidation assumes unchanged supervision.

AuthorsOpenReview ↗

Response 3/3: Long-Horizon Target Training (W4)

R3: From Short-Horizon Update Reconstruction to Full Target Training (W4)

We thank the reviewer for suggesting an evaluation of full-training outcomes. We have added a three-seed, long-horizon cross-architecture experiment on CIFAR-100 to examine whether UT's update reconstruction translates into task-level gains during sustained optimization.

W4: Full Target Training and Task-Level Gains

Weakness 4 (summary): The paper mainly demonstrates accurate reconstruction of individual updates and short trajectories, without showing whether UT records can support long-horizon target training and yield stable task performance.

Response W4:

We have extended our evaluation from short-horizon reconstruction to full target fine-tuning. In the new CIFAR-100 experiment, MobileNetV3-small and ResNet-18 continuously update their classification heads as heterogeneous sources, while their learning records drive full-parameter AdamW training of an EfficientNet-B0 target. With data augmentation and a consistent validation-based early-stopping rule, the three seeds complete 3,872, 7,392, and 7,392 updates, respectively, followed by evaluation on the test set.

UT maintains essentially the same learning outcomes as direct supervision over thousands of training steps. As shown in Table R6, UT increases test accuracy from 72.253% to 80.837%, closely matching direct supervision (80.833%), with the same reported test NLL and higher accuracy than source-output KD (79.503%). The final relative trajectory errors across the three seeds range from 7.495×10⁻⁶ to 5.010×10⁻⁴. Thus, heterogeneous historical updates can support sustained target optimization and preserve the long-horizon task gains of the original supervision, beyond reconstructing local parameter changes.

Table R6. New CIFAR-100 long-horizon cross-architecture fine-tuning (mean ± SD)

MethodTest accuracy (%)Test NLLFinal trajectory error
No update72.253 ± 0.0311.1160 ± 0.0012—
Direct supervision80.833 ± 0.2880.7532 ± 0.00940
UT80.837 ± 0.2910.7532 ± 0.0094(2.986 ± 2.584)×10⁻⁴
Same-information supervision representative80.837 ± 0.2910.7532 ± 0.0094(2.986 ± 2.584)×10⁻⁴
Source-output KD79.503 ± 0.0990.9335 ± 0.00241.930 ± 0.008

Three fixed seeds. Source models continuously update their classification heads, while the target updates all parameters. Training uses random cropping and flipping, fixed BatchNorm running statistics, and validation-based early stopping.

The revised Section 6.3 and Table 4A report the long-horizon outcomes, with the full protocol and per-seed trajectory results in Appendix D.3.5.

Reviewer_ABNsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how to reproduce, in a target model with a different architecture, the update that a source model would produce with one training step on a batch, when the batch’s labels are unavailable but the inputs and the source models’ states before and after the step are known. Since the cross-entropy and MSE gradients are linear in the labels, each source update gives linear equations in the unknown labels. These equations may leave part of the labels undetermined. The main theorem states that they still determine the target’s update exactly if and only if the target’s update does not depend on that undetermined part. So a target can be trained correctly even when the labels themselves cannot be recovered. For linear classification heads, this becomes a rank condition: if the pooled source features have rank equal to the batch size n, the labels are recovered exactly; otherwise, a linear target head is covered only if its features lie in the span of the pooled source features. The authors propose Update Tomography, which solves for a label estimate, constrained to valid probabilities when the equations are underdetermined, and trains the target with its own backward pass and optimizer. Experiments on Pet, CIFAR-100, DomainNet and AG News measure the relative error between the target update built by UT and the update obtained by training the target on the true labels. This error is around 1e-5 when the condition holds and much higher when it does not. For example, a CIFAR-100 record keeping only superclass labels reproduces a superclass target’s update but fails for a fine-class target.

Critical strengths

1 Interesting idea and theoretical result The fact that you can look at one optimizer step of a source model and use it to train a new network with a different architecture, without having the labels, is for me an interesting way to possibly transfer learning between models, because it does not need any alignment between the parameters of the source and the target. The target just computes its own update with its own backward pass and optimizer. The authors provide a theorem that states the conditions under which this mechanism works. It also covers cases where the labels can’t be recovered, as long as the target does not depend on the missing part. I think this is the most interesting contribution of the paper. The negative examples in Table 2 are convincing because the authors show the exact label changes that make it fail. The superclass experiment on CIFAR-100 is a clear example: a record that keeps only the 20 superclass labels is enough to train a 20-class target (error around 1e-6) but not a 100-class target (error around 2.8), because it cannot distinguish fine classes inside the same superclass.

2 Experiments coverage The experiments cover different architectures (ConvNeXt, Swin, ViT, EfficientNet, BERT, RoBERTa) and both image and text data. Some targets are updated over the whole network, up to 87M parameters. The authors ablate the details that could break the method. For example, with Momentum or AdamW the change in the weights is not the gradient of the step, so they use the optimizer buffers saved before and after the step to recover it. If they use only the weights, the error goes from about 1e-7 to order one or more. They also show that the records can be reused later by new models without loading the source models. Everything is reported in detail, including results that miss their own thresholds, so the paper is easy to reproduce.

Critical weaknesses

1 Hard to understand and unclear motivation The abstract does not say what the setting is or what the method does, and I could not understand the point of the paper from it. When the pooled source features have rank equal to the batch size, as in the Pet and DomainNet experiments of Table 3, UT simply recovers the labels and trains the target on them. Since UT and the reference update then come from the same labels, the low errors in Table 3 are expected by construction and only show that label recovery works. The paper should say this clearly and explain what is new compared to label recovery from gradients. It is also not clear when this setting happens in practice. To build a record, the method needs the inputs of the batch, the source weights before and after every step (plus the optimizer states for Momentum or AdamW), and a shared label space or a known label map, but not the labels. If the labels existed when the record was made, storing them would be simpler and cheaper: the compiled records take 2.4 MiB on Pet and 17.7 MiB on DomainNet per batch, while the labels of 800 images take a few KB. When the rank equals the batch size, the record also contains the labels anyway, so it does not even hide them. Section 7 mentions rare events or costly annotations as possible uses, but does not explain why the labels would not be kept in those cases, and no experiment shows such a scenario.

2 No transfer experiment and no comparison in the main text The paper positions itself against task-vector transport, but there is no table where a capability is transferred from one model to another, for example a CIFAR-100 fine-tuning. All experiments cover 1 to 60 steps and measure how close the update is to the reference, not how much the target improves. The method also cannot be applied to an existing fine-tuned model alone, since it needs the inputs and the transition of every training step recorded at training time. The main text has no comparison with other methods. Comparisons with THESEUS, TransFusion and GradFix appear only in the appendix, use different information, and are not cited. For me this is the main reason for my score: the idea is interesting, but I do not see how it can be used in practice to transfer a fine-tuning between models.

Questions for authors
  1. Can you give a concrete scenario where the inputs and the source optimizer states of every step are available but the labels are not, and explain why storing the labels is not an option there? Also, when the pooled rank equals the batch size, is UT different from recovering the labels and training the target normally?

  2. Can you add an experiment where a full fine-tuning is transferred, for example CIFAR-100 to a target with a different architecture? I would like to see the target accuracy, compute and storage compared with task-vector transport, and direct fine-tuning.

  3. How does the method behave in more realistic training, with many steps, data augmentation and BatchNorm in train mode? And how does the constrained solver work with soft labels, where the true labels are far from one-hot? When the rank is below the batch size and the target is a deep network, how can a user check in practice whether a record is enough for a given target before using it? The paper says this is still open, but this is the case where UT goes beyond label recovery.

  4. The abstract mentions language models, but the text experiment is a 4-class classification with BERT, RoBERTa and DistilBERT. Does the method work for language modeling, where the number of classes is the vocabulary size? How large can the batch be before the rank condition fails?

7 comments
AuthorsOpenReview ↗

Response 1/3: Research Motivation, Label Recovery, and Applications (W1/Q1)

Response 1: Research Motivation, Label Recovery, and Applications (W1/Q1)

We thank the reviewer for recognizing our cross-architecture perspective and theory of target-relative sufficiency. Your suggestions clarified our contributions and application conditions. We added same-information archival, partial-sufficiency, selective-sharing, and real historical-event reuse experiments.

W1: Research Motivation and Distinction from Label Recovery

Weakness 1 (summary): The setting is unclear. Full-rank Pet/DomainNet largely recover labels, so low errors are expected. Direct label storage is cheaper, and full-rank records do not hide labels. What is new beyond gradient-based label recovery, and what is the practical value?

Response W1:

At full rank, UT is equivalent to recovering supervision and training the target, so the low errors in Table 3 are expected. The revised Section 6.3 presents these full-rank tests as checks of reconstruction accuracy. Our central contribution is supervision space as a common mathematical reference for heterogeneous updates, and the question: Can a future target update be uniquely determined without uniquely recovering supervision? New AG News soft-supervision ambiguity and real CIFAR rank-deficiency tests (Table R1) show both sufficiency and failure with incomplete information.

Beyond target-update determinacy, recovering and archiving a supervision representative is a natural simplified UT realization when supervision is fully identifiable. Directly storing available labels remains cheaper. New same-information archival tests (Table R2) show that recovered representatives substantially reduce compiled-record size while preserving equivalent updates. UT can adopt different realizations according to information completeness and practical needs. The revised abstract emphasizes target-relative sufficiency, while Sections 5–7 distinguish recovered supervision from general records and clarify the role of full-rank tests.

Table R1. New experiments: Target-relative sufficiency beyond label recovery

ExperimentInformation conditionTarget response
AG News, 96 tokens3D soft-supervision ambiguity66M DistilBERT gradient response 5.44–6.41×10⁻⁸; 10-step trajectory error 7.345×10⁻⁶
AG News, 256 tokensSame source record; longer target contextGradient response 0.0422
CIFAR-100Real rank deficiency, 127/128Source measurement nearly unchanged; target gradient changes by 15.07%

AG News reports absolute L2 gradient responses; CIFAR reports relative gradient change.

Table R2. New same-information archival comparison

DatasetOriginal compiled recordRecovered FP32 supervision representativeUpdate difference
Pet2.3796 MiB0.0841 MiB2.876×10⁻⁸
DomainNet17.7085 MiB2.5378 MiB2.803×10⁻⁹

Representatives are recovered solely from existing source records, without additional access to original labels.

Q1: Concrete Use Cases and Full-Rank Equivalence

Question 1 (summary): When are event inputs and source optimizer states available but labels unavailable to future targets? At full rank, is UT different from recovering labels and training normally?

Response Q1:

One scenario is cross-institution selective sharing: a source trains with private labels and optimizer states locally, but releases only selective measurements on public inputs. Capture and downstream sharing have different access permissions; labels were available during source training. In our new three-client heterogeneous collaboration test, UT and same-information coarse-class KD both reach 77.80% mean test accuracy.

Another scenario is one-to-many reuse of valuable historical supervision. A real ViT-B/32 CIFAR-10H event produces a record reused for 20 full-parameter updates on each of three heterogeneous targets, without giving them the original human soft labels. UT closely matches direct soft supervision (Table R3).

Table R3. New one-to-many reuse of human soft supervision (Soft NLL ↓)

TargetNo replayDirect soft supervisionUTHard labelsSource-output KD
MobileNetV3-small0.53930.53380.53390.53590.5419
ResNet-180.57880.59250.59240.67980.6146
EfficientNet-B00.65280.56560.56560.63040.5434

At full rank, training on recovered supervision is mathematically equivalent to UT, including the archival approach in W1. These scenarios illustrate selective sharing and cross-architecture reuse, not superiority over accessible labels. The revised Sections 3–5 and Appendix D.4 specify these access conditions and limitations.

AuthorsOpenReview ↗

Response 2/3: Full Fine-Tuning, Baselines, and Costs (W2/Q2)

Response2: Full Fine-Tuning Utility, Baselines, and Costs (W2 / Q2)

We thank the reviewer for highlighting that short-horizon update fidelity alone does not establish practical transfer, and for requesting full training, baseline comparisons, and resource measurements. We added three-seed CIFAR-100 long-horizon training, transfer baselines with matched target-side tuning budgets, and measured compute/storage costs.

W2: Full Fine-Tuning and Task-Level Utility

Weakness 2 (summary): The paper evaluates only 1–60-step update errors, without full fine-tuning utility or main-text comparisons. UT also requires training-time event records and cannot transfer from final fine-tuned weights alone.

Response W2:

1. Long-horizon learning. We ran new CIFAR-100 cross-architecture experiments using continuous classification-head updates from MobileNetV3-small and ResNet-18 as source records to train all parameters of an EfficientNet-B0 target. Training used augmentation, AdamW, and a common validation-based early-stopping rule; the three seeds ran for 3,872 / 7,392 / 7,392 steps.

UT improved test accuracy from 72.25% to 80.84%, closely matching direct supervision (80.83%) and exceeding source-output KD (79.50%, Table R4). Thus, the recorded heterogeneous updates reproduce not only local update trajectories but also the task-level effect of thousands of training steps. Final trajectory errors ranged from 7.50×10⁻⁶ to 5.01×10⁻⁴; the larger errors in two seeds correspond to a real rank-deficient source event, which we also analyzed.

2. Scope of transfer. UT reuses learning events captured during source training, not task vectors inferred from final weights. Supervision-space measurements construct target-native updates without aligning source and target parameter coordinates. The revised Sections 3 and 5 now state the capture requirements explicitly.

Q2: Transfer Baselines and Compute/Storage Costs

Question 2 (summary): Can you show full CIFAR-100 fine-tuning across architectures, with target accuracy, compute/storage costs, and comparisons against direct fine-tuning and task-vector transfer?

Response Q2:

1. Baselines. CIFAR-100 compares direct supervision, UT, an equal-information supervision representative, source-output KD, and no update (Table R4). We also added a THESEUS and KD comparison on Pet, matching target-side tuning budgets (18 development-set evaluations per method). At the target initialization requiring further learning, UT and direct supervision both reached 78.33%, versus 77.06% for THESEUS and 82.36% for KD. UT reproduces the supervised update; it is not intended to outperform every distillation baseline. The revised Section 6.3 and Appendix D.3 distinguish this single-event Pet comparison from the CIFAR-100 long-horizon protocol.

2. Measured costs. On a single RTX 5090, target-step times for UT and direct supervision were comparable (Table R5); additional work arises mainly from source capture and record compilation. A general record occupies 108.4 KB/event, versus 21.2 KB/event for a recovered supervision representative when supervision is fully identifiable—a 5.11× reduction, with identical test accuracy. The representation can therefore be chosen according to information sufficiency and storage needs.

The revised Section 6.3 reports the long-horizon outcomes and baselines; Appendix D.3 provides the measured costs and specifies the information available to each method.

Table R4. New CIFAR-100 long-horizon cross-architecture fine-tuning (mean ± SD)

MethodTest accuracy (%)Test NLLFinal trajectory error
No update72.253 ± 0.0311.1160 ± 0.0012—
Direct supervision80.833 ± 0.2880.7532 ± 0.00940
UT80.837 ± 0.2910.7532 ± 0.0094(2.986 ± 2.584)×10⁻⁴
Equal-information representative80.837 ± 0.2910.7532 ± 0.0094(2.986 ± 2.584)×10⁻⁴
Source-output KD79.503 ± 0.0990.9335 ± 0.00241.930 ± 0.008

Three fixed seeds; 45k/5k/10k train/validation/test. Sources update classification heads continuously; the target trains all parameters with BN running statistics fixed.

Table R5. New compute/storage measurements (RTX 5090; batch size 128)

OperationMeasurement
Source capture overhead, MobileNet / ResNet (head only)+0.481 / +0.551 ms/step (+12.31% / +4.28%)
Two-source compilation / record encoding10.139 / 2.602 ms
Target step: direct / UT / representative249.699 / 245.389 / 239.100 ms
Per-event storage: general / representative108,377 / 21,222 B
Encoded total over 7,392 steps: general / representative744.2 / 145.7 MiB

Target timing includes full-parameter forward/backward and optimizer updates. Encoded totals exclude shared inputs and model storage.

AuthorsOpenReview ↗

Response 3/3: Training Robustness and Language Modeling (Q3/Q4)

Response 3: Realistic Training, Supervision Sufficiency, and Language Modeling (Q3 / Q4)

We thank the reviewer for the questions on training conditions, soft supervision, and language modeling. We added experiments to clarify UT's scope.

Q3: Realistic Training and Sufficiency Checks for Rank-Deficient Records

Question 3 (summary): How does UT perform with long training runs, data augmentation, and BatchNorm in train mode? Can the constrained solver handle soft labels far from one-hot? For rank-deficient records and deep targets, how can sufficiency be checked before reuse?

Response Q3:

1. UT supports long-horizon learning with data augmentation, although training states affect numerical reconstruction accuracy. Beyond the thousands of steps reported under W2, we ran 200 consecutive AdamW steps with both sources and target updating all parameters, reaching a final trajectory error of 1.101×10⁻⁵. With matched cropping, flipping, and stochastic depth, the 20-step errors were 2.357×10⁻⁵ with frozen BN running statistics and 3.379×10⁻³ with BN in train mode (Table R6). These tests are reported in revised Section 6.1 and Appendix D.1.

2. UT's supervision-measurement relation also applies to soft labels; it does not require nearly one-hot targets. On Pet, we tested label smoothing from 0.1 to 0.9 and teacher-provided soft supervision. Reconstruction remained accurate at full rank. Under rank deficiency, the ADMM solution depends on the supervision distribution and available constraints: at n=1,600, smoothing 0.1 and 0.9 yielded full-network relative gradient errors of 1.000×10⁻² and 3.889×10⁻⁴, respectively. Thus, what matters is whether the source measurements retain the information required by the target, rather than label hardness alone.

3. For rank-deficient records, Theorem 4.1 gives the target-relative condition ker⁡R⊆ker⁡T\ker R\subseteq\ker T, which we further examined numerically for deep networks. On AG News, gradient computations with an independent DistilBERT and linear programming over a three-dimensional ambiguity space gave response upper bounds of 8.178×10⁻⁹ and 6.464×10⁻³ for 96- and 256-token targets, respectively. For a real CIFAR rank-deficient event, adding 619 bytes of targeted source measurements reduced the target gradient error from 7.537×10⁻² to 6.019×10⁻⁷. These are concrete checks and targeted completions; efficient high-dimensional certification remains open.

Table R6. New experiments under realistic training conditions

ExperimentSettingMain result
Continuous full-network learningSources and target update all parameters; 200 stepsFinal trajectory error 1.101×10⁻⁵
Augmentation and BNFrozen / train-mode BN; 20 stepsTrajectory errors 2.357×10⁻⁵ / 3.379×10⁻³
Pet soft supervisionn=1,600; smoothing 0.1 / 0.9Relative gradient errors 1.000×10⁻² / 3.889×10⁻⁴

BN comparisons use cropping, flipping, and matched stochastic depth. Sufficiency bounds use an ideal compatible record; targeted completion requires access to an additional source measurement.

Q4: Full-Vocabulary Language Modeling and Rank Limits

Question 4 (summary): The original text experiment is only four-class classification. Does UT extend to full-vocabulary language modeling, and at what batch size does the source-feature rank condition fail?

Response Q4:

1. We added a WikiText-2 next-token prediction experiment with SmolLM2-135M/360M as heterogeneous sources and SmolLM2-1.7B as a fully trainable target, using the entire 49,152-token vocabulary. At 1,024 effective prediction positions, the pooled source record was full rank and the target relative gradient error was 2.33×10⁻⁶ (Table R7).

2. Batch size and information complementarity. The two sources have feature widths of 576 and 960, giving a pooled width of 1,536. The record remained full rank at n=1,536, although numerical conditioning worsened; from n=1,537, it was no longer full row rank, and target errors increased. The rank condition therefore depends on the number of effective prediction positions and the pooled source features, not directly on vocabulary size. Complementary sources can increase the number of samples whose supervision is fully identifiable.

Table R7. New next-token language modeling results (49,152-token vocabulary)

Effective positions nPooled rankRelative target gradient errorRelative actual-update error
1,0241,0242.33×10⁻⁶3.96×10⁻⁴
1,5361,5369.36×10⁻⁵2.40×10⁻³
1,5371,5368.70×10⁻³2.56×10⁻²
2,0481,5363.16×10⁻¹3.26×10⁻¹

Median over three independent text events; each example predicts the next token from a 32-token prefix.

The revised Section 6.3 and Appendix D.3.1 state these results and their single-step limits.

Reviewer_U4MNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents Update Tomography (UT), which represents corrected updates from heterogeneous models as linear measurements of shared supervision under losses with supervision-affine output gradients. It characterizes when these measurements determine a target model’s supervision-dependent gradient, even without identifying the complete supervision, and constructs target updates through the target’s own backward pass and optimizer. Experiments test measurement correction, matched and mismatched information requirements, cross-architecture reconstruction, and replay from compiled records after source models are unavailable. The setting retains the event inputs and requires suitable source state at capture time. The central empirical outcome is update and trajectory fidelity to direct supervision.

Critical strengths
  1. The paper provides a clear framework connecting interpretable source updates, retained supervision information, and target-side learning. The distinction between recovering all supervision and determining a particular target response is useful. The stated assumptions and the separation between information sufficiency and numerical stability make the scope assessable.

  2. The experiments include meaningful controls for the proposed explanation: optimizer-state correction, matched versus mismatched class projections, and targets inside versus outside the retained source span. Whole-network vision experiments, text-domain results, and source-independent replay extend the validation beyond a linear toy setting. The appendix also reports costs, perturbation sensitivity, and cases where accurate reconstruction does not justify applying an update.

Critical weaknesses
  1. The incremental theoretical contribution needs sharper positioning. Proposition 3.1 follows directly from the chain rule under the stated affine-supervision assumption. The kernel-inclusion criterion in Theorem 4.1 is equivalent to a row-space containment condition, closely related to established functional observability; for example, Montanari et al. (PNAS 2022), Results Eq. (3), gives the corresponding rank condition for reconstructing a target function. The paper cites this literature, and its specialization to learning records may be useful, but it is not yet clear which theoretical insight extends beyond this specialization. This limits the strength of the theoretical novelty supporting acceptance.

  2. The distinctive practical benefit over recovering and archiving supervision is not sufficiently established. In the full-row-rank Pet and DomainNet settings, the compiled representation in Appendix C.2 permits recovery of the supervision contrast itself. In the official-20 CIFAR experiment, fine-100 supervision remains ambiguous, but the target’s coarse supervision is recoverable. Thus these central successes are also consistent with recovering the available supervision at capture time and training later targets on it. I recognize that this can be viewed as a special implementation of UT, rather than an unrelated competitor; the issue is what additional capability motivates retaining the general record. The Sensorless example demonstrates genuine partial-information sufficiency, but uses deliberately constructed target spans, while the rank-deficient Pet results establish approximate fidelity rather than general exact coverage. A clearer separation of these cases, and evidence of the value beyond supervision archival under the same information access, would materially strengthen the contribution. This concern does not presume that the paper claims compression or privacy guarantees.

Questions for authors
  1. Could you identify the precise theoretical increment over the cited functional-observability and optimal-recovery results, distinguishing the learning-specific formulation from new mathematical conclusions? In particular, how should Theorem 4.1 be interpreted relative to the established row-space/rank criterion?

  2. For the full-rank Pet/DomainNet records and the official-20 CIFAR archive, is recovering and storing the available supervision at capture time equivalent to the reported replay construction, up to numerical effects? If so, what functionality or cost advantage does retaining the general record provide in these experiments? A comparison should use the same inputs and source information; it need not grant access to the original labels.

  3. Beyond the constructed Sensorless targets, do the existing experiments establish a practically representative case where supervision needed as an explicit training target remains unidentified, yet an independently chosen deep target has certified or reliably demonstrated target-relative coverage? Please distinguish exact coverage, event-specific approximate success, and full recovery of a projected label space. These clarifications would affect my assessment of the framework’s significance.

5 comments
AuthorsOpenReview ↗

Response 1/3: Theoretical Novelty and Functional Observability (W1/Q1)

R1: Regarding the Theoretical Novelty (W1 / Q1)

We thank the reviewer for the careful analysis of our theoretical contribution, particularly for highlighting the connection between Proposition 3.1, Theorem 4.1, and classical functional observability. This observation helps us clarify the theoretical contribution of our work and its relationship to prior research.

Weakness 1: “The incremental theoretical contribution needs sharper positioning. Proposition 3.1 follows directly from the chain rule under the stated affine-supervision assumption. The kernel-inclusion criterion in Theorem 4.1 is equivalent to a row-space containment condition, closely related to established functional observability; for example, Montanari et al. (PNAS 2022), Results Eq. (3), gives the corresponding rank condition for reconstructing a target function. The paper cites this literature, and its specialization to learning records may be useful, but it is not yet clear which theoretical insight extends beyond this specialization. This limits the strength of the theoretical novelty supporting acceptance.”

Question 1: “Could you identify the precise theoretical increment over the cited functional-observability and optimal-recovery results, distinguishing the learning-specific formulation from new mathematical conclusions? In particular, how should Theorem 4.1 be interpreted relative to the established row-space/rank criterion?”

Response W1 / Q1:

Our central theoretical contribution is to establish a precise mathematical connection between actual learning updates from heterogeneous models and a common supervision space, and thereby develop a theory of target-relative information sufficiency. This connection allows updates that were previously difficult to compare to be analyzed in a unified space, and characterizes when historical learning information can be transferred and where its limits lie.

This contribution consists of two closely connected theoretical results:

1. From heterogeneous learning updates to common supervision measurements.

The parameter spaces of different architectures do not directly correspond, and their actual updates are also affected by optimizer history. Proposition 3.1 shows that, under the stated supervision-affine and state conditions, corrected optimizer transitions can be expressed as exact linear measurements of the same supervision contrast, without parameter alignment or linearizing the nonlinear networks.

This result establishes a common mathematical reference for describing and analyzing information from heterogeneous learning processes. The optimizer-state experiments in the Table 1 further verify this connection.

2. From supervision recovery to target-relative information sufficiency.

Building on this common representation, Theorem 4.1 addresses a further question: when the original supervision cannot be fully recovered, is the historical record still sufficient to uniquely determine the learning response of a future target?

We establish the necessary and sufficient condition, ker⁡RL⊆ker⁡TL\ker R_{\mathcal L}\subseteq\ker T_{\mathcal L}, which characterizes full supervision identification, partial information sufficiency, and cases in which the target response cannot be determined. Thus, transferability depends not only on how much information the source record retains, but more importantly on the relationship between the missing information and the target's learning requirements.

The reviewer correctly points out that the classical row-space criterion provides an important mathematical foundation for this condition. Our theoretical increment is to derive the common representation of source measurements and target responses from actual heterogeneous learning processes, so that the classical observability condition can characterize information sufficiency and transfer boundaries in cross-architecture learning.

In summary, our work establishes the theoretical connection “heterogeneous learning updates → common supervision space → target-relative sufficiency.” UT follows naturally as a constructive realization of this theory. The revised Sections 3–4 make this theoretical positioning explicit, with positive and negative tests reported in Section 6.2 and Appendix D.2.

AuthorsOpenReview ↗

Response 2/3: Supervision Archival and General Learning Records (W2/Q2)

R2: Supervision Archival and the Value of General Learning Records (W2 / Q2)

We thank the reviewer for the careful assessment of supervision archival and its practical value. In particular, the reviewer notes that recovering and archiving supervision can be one implementation of UT, while the additional capability of a general record beyond a single supervision representative calls for further evidence. This suggestion has been helpful in clarifying our results. We therefore conducted new same-information archival comparisons, multi-source complementarity tests, and targeted additional-measurement experiments on a real training event.

Weakness 2 (summary): Full-rank Pet and DomainNet records allow supervision recovery, while CIFAR official-20 identifies the coarse supervision required by its target. These results could therefore also be achieved by archiving recovered supervision. The reviewer recognizes archival as a possible implementation of UT but asks what additional capability general records provide, and requests a clearer distinction between constructed sufficiency in Sensorless and approximate results in rank-deficient Pet.

Question 2 (summary): Under identical inputs and source information, without access to original labels, is recovering and archiving supervision equivalent to UT replay for Pet, DomainNet, and CIFAR official-20? If so, what functional or cost advantage remains for general records?

Response W2 / Q2:

When supervision is fully identified, directly storing the recovered supervision representative is the more economical choice. Following the reviewer's suggestion, we conducted new archival comparisons under identical information-access conditions (Table R1). No original labels were used; the representatives were recovered entirely from the existing source records. For Pet and DomainNet, representative archival reduced storage by approximately 28× and 7×, respectively, while preserving essentially identical learning updates. CIFAR official-20 likewise permits archiving the recovered coarse-class supervision, without identifying the original fine-class supervision. These results validate the practical value of the reviewer's proposed archival approach and provide a more efficient implementation choice within UT.

Table R1 | New experiment: archival and replay using identical source information

DatasetArchive formatSize (MiB)Read + head update (ms)Relative update difference
PetGeneral record, FP322.379610.3960
PetSupervision representative, FP320.08411.0472.876×10⁻⁸
DomainNetGeneral record, FP3217.708584.0530
DomainNetSupervision representative, FP322.537817.7482.803×10⁻⁹

The complete comparison also includes FP64 representatives and their readout costs. Timings exclude whole-network training.

When supervision is not fully identified, a general record provides a further capability: assessing what a future target requires and how missing information can be supplied. A single representative can produce an update but cannot, by itself, determine whether other supervision signals compatible with the record would change that target's response. A general record retains the original measurement constraints, enabling target-sufficiency checks and the pooling of additional measurements. We tested this in a real rank-deficient CIFAR training event (Table R2): the same record supported a pair-averaged-supervision target but left a substantial error for the original fine-100 target. Adding redundant measurements had no effect, whereas a 619-byte targeted measurement raised the joint rank from 127 to 128 and reduced the target AdamW update error from 4.37% to 1.92×10⁻⁵.

Table R2 | New experiment: target sufficiency and targeted measurement in a real CIFAR event

Record conditionTarget supervisionJoint rank / 128Added bytesTarget gradient errorAdamW update error
Original recordPair-averaged supervision12705.849×10⁻⁷1.995×10⁻⁵
Original recordFine-10012707.537×10⁻²4.366×10⁻²
Redundant measurementFine-1001276127.537×10⁻²4.366×10⁻²
Targeted measurementFine-1001286196.019×10⁻⁷1.916×10⁻⁵
Full weighted-head measurementFine-100128465,5525.101×10⁻⁷1.705×10⁻⁵

The revised manuscript reports the completion tests in Section 6.2 and Appendix D.2, and the archival comparisons in Section 6.3 and Appendix D.3.2. Our separate Q3 response addresses partial sufficiency for an independent deep target.

AuthorsOpenReview ↗

Response 3/3: Target-Relative Sufficiency in Deep Models (W2/Q3)

R3: Target-Relative Sufficiency in Independent Deep Models (W2 / Q3)

We thank the reviewer for the thoughtful suggestion to distinguish full supervision recovery, constructed partial sufficiency, and event-specific approximation. It motivated us to test whether the theory also predicts both outcomes for independently instantiated deep targets.

Weakness 2 (relevant concern) & Question 3 (summary): The target feature spans in Sensorless are deliberately constructed, while rank-deficient Pet mainly shows event-specific approximate reconstruction. Beyond Sensorless, is there a representative case where supervision remains unidentified yet an independently chosen deep target has reliably demonstrated target-relative sufficiency? Please distinguish exact coverage, approximate success, and full recovery of projected supervision.

Response W2 / Q3:

We added positive and negative tests on AG News and a real CIFAR-100 training event to examine both the sufficiency and failure conditions characterized by Theorem 4.1. The original CIFAR official-20 experiment demonstrates full recovery of projected supervision; Sensorless shows exact sufficiency for constructed target spans; and rank-deficient Pet shows event-specific approximate reconstruction. Our new experiments examine these distinctions in real deep networks.

1. AG News: An independent deep target can have sufficient information despite supervision ambiguity (Table R3).

We use a natural property of text processing: different articles can become identical inputs after 96-token truncation. On these inputs, we construct valid soft-supervision events with a three-dimensional ambiguity. For an independently instantiated 66M-parameter DistilBERT, the unresolved supervision directions cancel structurally in the full-network gradient when the target also reads 96 tokens, and ten-step replay retains high fidelity. When the same model instead reads 256 tokens, previously invisible text differences affect its gradient. Thus, incomplete supervision need not prevent a target update from being determined; sufficiency depends on which information the target actually uses.

2. Real CIFAR training: Target-side augmentation makes missing information consequential (Table R4).

We analyzed a rank-deficient event encountered during long-horizon CIFAR-100 training. Two pixel-identical images have different labels and are indistinguishable to the source, while random augmentation at the target breaks this symmetry. Swapping their labels leaves the ideal source measurement essentially unchanged, but changes the full-network target gradient by 15.07% and the AdamW update by 8.72%; the actual trajectory error also grows following this event. The target update is therefore not uniquely determined by the source record, even with ideal measurements: this is an information boundary, not merely numerical solver error.

The revised Section 6.2 and Appendix D.2 now report the experimental settings, measurements, and scope of these tests.

Table R3 | New AG News experiment: target-relative sufficiency in a 66M-parameter DistilBERT

Measurement96-token target256-token target
Null-direction gradient response (absolute L2 norm)5.44–6.41×10⁻⁸0.0422
Ideal-record LP response bound (absolute L2 radius)8.178×10⁻⁹6.464×10⁻³
Ten-step relative trajectory error7.345×10⁻⁶—

The LP bound concerns ideal compatible records with smoothing 0.1. All three null directions were tested at 96 tokens; one prespecified direction was tested at 256 tokens.

Table R4 | New real CIFAR experiment: insufficient information and divergent target updates

MeasurementResult
Source record rank127 / 128
Ideal source-measurement difference after label swap (absolute L2 norm)2.623×10⁻¹⁴
Relative change in full-network target gradient15.07%
Relative change in AdamW update8.72%
Relative trajectory error (one step before / at event / ten steps after)7.454×10⁻⁶ / 5.082×10⁻⁵ / 3.399×10⁻⁴

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response: Theoretical Contributions and Additional Evidence

Global Response

I. Acknowledgments

We thank all reviewers for their thoughtful evaluations. We are encouraged that the supervision-space perspective and target-relative sufficiency were recognized as our core contribution. Below we summarize how the revised version addresses the main concerns.

II. Central Contribution: A New Theoretical Framework

Reviewers: U4MN, ABNs

Concern: The theory may appear as a restatement of the chain rule or classical functional observability.

Response: Our contribution is not merely establishing a connection, but introducing a new theoretical framework: we prove that heterogeneous updates, under explicit state-correction conditions, provide exact measurements of a shared supervision contrast, and we establish target-relative sufficiency as a new criterion for cross-architecture learning. While individual steps use classical tools, the framework itself and its testable consequences (AG News, real CIFAR event) are novel. We have revised Sections 3–4 to make this distinction explicit.

III. Practical Value and Scope

Reviewers: U4MN, ABNs, UdnP, 4wfY

Concern: Why not store labels directly? Does this work beyond short tests?

Response: When labels remain accessible, direct storage is indeed simpler. Our focus is on what historical updates preserve and when incomplete records suffice. The revised version adds:

  • Long-horizon CIFAR-100: UT achieves 80.837% vs. 80.833% for direct supervision
  • 200 continuous full-network source-training events
  • Single-step 1.7B-parameter language model with 49,152-token vocabulary

UT serves as a constructive realization of the theory, demonstrating that the predicted measurements can be obtained from actual training transitions.

IV. Selective Retention

Reviewers: 4wfY, UdnP

Concern: Full records may reveal fine-grained supervision.

Response: We agree this is not the primary contribution. The selective-retention experiments (Appendix D.4) demonstrate a direct consequence of our theory: when a target only requires coarse supervision, fine-grained measurements can be discarded without affecting the target's learning. We have revised the text to present this as an application of the theory, rather than a standalone privacy method.

V. Changes in the Revised Version

  • Abstract/Introduction: Reordered to present theory → UT, clarifying that UT realizes the theoretical framework
  • Section 3–4: Explicit distinction between classical observability and our target-relative sufficiency criterion
  • Section 6.3: Added explanation of rank limits in language models, connecting back to Theorem 4.1
  • Appendix D.4: Selective retention presented as a theoretical application
  • Discussion: Clarified the scope of the current implementation

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.