- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a setting called Data-Free Untraining in TTA, which aims to retrospectively remove the influence of sensitive or corrupted data contained in specific past batches from a model undergoing continual TTA. A request specifies only batch indices, without access to the original raw inputs, labels, or class information, and the goal is to approximate the Retain-Only Replay model, obtained by re-running TTA from the source model while excluding the requested batches. Following the distinction made by Triantafillou et al. (2026), the authors differentiate untraining, which reverses the influence of a specific set of samples, from unlearning, which removes a class or concept itself, and state that their setting belongs to the former. Built on parameter-efficient TTA methods such as TENT and ViDA, the proposed approach stores per-batch model states and predicted class sets, keeping recent records intact and compressing older ones into averages, and accumulates class-wise Fisher importance scores. When a request arrives, the method reconstructs feature tokens whose class responses are amplified in the stored state of the requested batches relative to a reference state taken before the earliest requested batch, and then optimizes a forget loss that reduces the feature–classifier alignment of these tokens, combined with Fisher-weighted EWC, a feature gradient projection that removes the classifier direction, and an optional retain loss. Experiments with ViT-B/16 on ImageNet-C and CIFAR-100-C streams, into which batches containing FAIR1M aerial images or synthetically perturbed images are injected, report that the proposed method achieves a smaller confidence gap to Replay than existing MU methods adapted to the PEFT parameter space, better preserves same-class accuracy, and remains stable across successive requests.
Critical strengths
The paper raises a novel and practical problem in TTA. To our knowledge, it is the first to formalize the scenario in which the influence of past batches must be retroactively removed from a model under continual TTA. It clearly defines the setup, including batch-level requests aligned with the TTA update unit, a well-defined Retain-Only Replay oracle, and successive request scenarios. In particular, by distinguishing untraining from unlearning, it explicitly establishes an evaluation perspective that requires removing the influence of sensitive samples while preserving recognition of the corresponding classes.
The component design and supplementary analyses are thorough. The idea of reconstructing tokens that capture the traces of the requested batches, by contrasting the stored state of those batches with a reference state that has not seen them, is intuitive. Table 7 shows that using the reconstructed tokens yields results nearly identical to using the actual images (Score 89.85 vs. 90.00), supporting their suitability as substitute inputs. Furthermore, the appendix provides detailed reports on the MIA evaluation protocol, sample-level agreement with Replay, runtime, and memory usage.
Critical weaknesses
The advantage over simple control baselines is not convincingly demonstrated. In Table 2, Clean Rollback and Source Reset achieve Scores of 88.50 and 88.43 on ImageNet-C, only modestly below the proposed method (89.85), and 87.98 and 87.43 on CIFAR-100-C, higher than the proposed method (85.05). On CIFAR-100-C, their CF-Gap (0.000 and 0.001) is also clearly lower than that of the proposed method (0.053), and on ImageNet-C they are comparable. In the sequential setting of Table 3, both controls also show a lower CF-Gap than the proposed method at the second request. However, since the bold entries in the tables mark only the best among the adapted MU methods, this is not immediately apparent. This issue is also related to the evaluation design. CF-Gap and MIA-Gap naturally become small simply by reverting to a state that has never seen the requested batches, and since the ImageNet-C stream uses only mild corruptions at severity 1, the adaptation gain from TTA is inherently small. The number of retained batches between the last requested batch and the arrival of the request is also not specified. Moreover, the proposed method already stores the reference state, which is essentially the state that Clean Rollback restores, so rollback is available at no additional cost. The paper reports that the proposed method retains a few more percentage points of SC-Acc and R-Acc than Clean Rollback. However, rollback achieves a lower MIA-Gap on both datasets and a lower CF-Gap and a higher Score on CIFAR-100-C. The paper also does not examine conditions under which rollback should fail, such as many retained batches after the requested ones or stronger corruptions. It therefore remains unclear when the more complex reconstruction and optimization procedure is preferable to simple rollback.
There are inconsistencies between the claims and the actual results regarding storage cost, privacy, and the choice of baselines. Although the paper states that storage does not grow with the stream length when R, L, and H are fixed, Table C.3 shows the persistent state growing from 12.11 GiB to 18.80 GiB, which is dozens of times larger than the ViT-B/16 model itself and is difficult to reconcile with the claims of parameter or storage efficiency. In addition, the fact that tokens carrying the class responses of a requested batch can be reconstructed from its stored state implies that the stored state itself encodes information about the sensitive data, yet the paper describes no procedure for deleting the exact records of the requested batches or their accumulated contributions to the Fisher bank after untraining. Regarding the baselines, the existing MU methods operate only on proxies without stored states, whereas the proposed method relies on per-batch state records, creating a substantial gap in available information. Furthermore, applying the existing methods in a class-conditional manner is likely to degrade same-class accuracy by design. SSD-PEFT produces results identical to No-Untrain on all ImageNet-C metrics, which raises doubts about whether it was applied correctly. Meanwhile, the paper does not compare against Amnesiac ML (Graves et al., 2021), which subtracts the updates of the requested batches and is the most natural baseline when per-batch states are stored.
Questions for authors
-
In each experiment, how many retained batches lie between the last requested batch and the arrival of the request? Can you show that the proposed method clearly outperforms Clean Rollback as this gap and the corruption severity increase, that is, in situations where rollback discards a substantial amount of subsequent adaptation? If not, what is the justification for using this method instead of rollback, given that reference state is already stored?
-
Most SC-Acc and R-Acc values in Tables 2, 3, 5, and C.1 correspond to multiples of 1/16, suggesting that accuracy was measured on about 16 images. Could you provide the actual number of evaluation images for each metric and the per-seed raw values?
2 comments
Response to Reviewer tUwC (Part4)
Q3: Stored states and the Fisher bank may retain information about the requested data, but the paper does not provide a complete procedure for deleting these auxiliary traces.
A3: We have clarified the scope of the current untraining guarantee in Appendix C.3. The current method corrects the active model’s behavior on the requested data but does not guarantee complete removal of every trace from auxiliary historical states and Fisher statistics.
We also added the auxiliary-state cleanup procedure to Appendix B. Exact state records associated with the requested batches are deleted, archived records overlapping the requested interval are invalidated, and Fisher entries associated with the affected classes can be reset. This conservative cleanup reduces residual auxiliary traces, but resetting class-level Fisher entries may also discard useful retained information. We therefore identify complete and selective auxiliary-state erasure as a limitation of the current framework rather than claiming that active-model correction alone provides complete system-level deletion.
Q4: The adapted machine-unlearning baselines have less historical information, SSD-PEFT matches No-Untrain, and Amnesiac ML is missing.
A4: Tables 2 and 4 now report paired no-history and matched-history versions of SCADA-UL, DELETE, SSD, SalUn, and SCRUB. The history-enabled rows use the same stored-state reconstruction as Ours.
The results show that providing history does enable these baselines to erase the response associated with the target category more aggressively. For example, at the first request, history reduces SCADA’s F-Conf from 0.481 to 0.091 and its SC-Acc from 0.391 to 0.109. For SCRUB, F-Conf decreases from 0.560 to 0.004. These large reductions indicate that the reconstructed historical information successfully identifies the target influence and allows the original class-directed objectives to suppress it more completely.
However, stronger target-category suppression is not necessarily equivalent to accurate batch-specific untraining. Retain-Only Replay still produces a nonzero response for legitimate retained examples from the same class. History-enabled SCADA therefore increases JS mean from 0.1031 to 0.3519 and reduces Score from 60.59 to 41.65. History-enabled SCRUB similarly increases JS mean from 0.1125 to 0.5401. These results indicate that the baselines often remove not only the requested-batch influence but also useful responses to retained examples from the same class.
The comparison therefore shows that access to historical information improves the ability of these methods to locate and suppress the target category, but their class-level objectives can produce over-forgetting. Effective batch-specific untraining requires both informative historical inputs and a sufficiently selective correction objective. The complete comparison and discussion are provided in Tables 2 and 4 and Appendix C.1.
We also added an Amnesiac-style baseline that subtracts the stored EMA parameter change accumulated over the requested interval. At the first request, it obtains JS mean 0.0100, R-Acc 0.617, and Score 81.97, compared with 0.0080, 0.641, and 84.19 for Ours. Across three requests, its Scores are 81.97, 78.68, and 73.67, whereas Ours obtains 84.19, 84.22, and 79.10. These results are reported in Tables 2 and 4.
Q5: How many evaluation images are used for SC-Acc and R-Acc?
A5: Appendix D.4 now specifies the evaluation-set sizes. Each request uses 128 same-class retained images for SC-Acc and 128 retained-stream images for R-Acc.
Response to Reviewer tUwC (Part3)
We have completed the experiments described in our previous response and revised the manuscript accordingly.
We first revised the calculation of the overall Score, as updated in the experimental section and in Appendix D.4. The original Score used CF-Gap, which compares the mean requested-image confidence of the evaluated model with Retain-Only Replay. Because it reduces each model’s output to a single averaged confidence value, different changes across classes or samples can cancel one another.
We therefore replace CF-Gap in the Score with JS mean. For every requested image, JS mean computes the Jensen-Shannon divergence between the full softmax probability distributions of the evaluated model and matched Retain-Only Replay, and then averages it across images. A lower value indicates that the corrected model more closely reproduces the oracle model’s complete output behavior, rather than merely matching one confidence statistic. Tables 2, 4, and 5 report results using this revised Score.
Q1: The proposed method does not convincingly outperform Source Reset and Clean Rollback, particularly under mild corruption and short request delays.
A1: We redesigned the ImageNet-C experiment to test the intended correction-retention trade-off more directly. The requested interval uses severity-1 corruptions. During the delay between the end of that interval and the arrival of the untraining request, online adaptation continues on previously unseen severity-3 domains selected from snow, fog, frost, brightness, and contrast.
Table 2 shows that Retain-Only Replay and Ours both obtain an R-Acc of 0.641, whereas Source Reset and Clean Rollback obtain 0.555 and 0.539. Thus, reset and rollback lose 8.6 and 10.2 percentage points of retained-stream accuracy, respectively. Ours also achieves an SC-Acc of 0.867 and a Score of 84.19, compared with 0.782/82.16 for reset and 0.782/81.97 for rollback.
For comparison, in the original experiment without a deliberately inserted domain transition, reset and rollback trailed Replay in R-Acc by only 6.7 and 3.5 percentage points. We moved that original experiment to Appendix C.4, Table C.8. These results support our central claim that selective correction can remove the requested influence while preserving useful adaptation acquired afterward.
We further define delay as the number of retained online-adaptation batches processed between the end of the requested interval and the arrival of the untraining request. Table 5 evaluates delays of 30, 50, and 70:
Ours preserves substantially more of the intervening adaptation, but it is not unaffected by increasing delay. Its Score decreases from 84.75 at delay 30 to 84.19 at delay 50 and 84.14 at delay 70. At delay 70, its R-Acc also falls 4.7 percentage points below Replay, whereas it matches Replay at delays 30 and 50. Nevertheless, its R-Acc of 0.664 remains clearly higher than reset at 0.512 and rollback at 0.555. The results therefore support a qualified conclusion that our method better preserves post-request domain adaptation than restoring an earlier state, although its correction accuracy also gradually degrades under sufficiently long delays.
Q2: Persistent storage grows from 12.11 GiB to 18.80 GiB, which appears inconsistent with the storage-efficiency claim.
A2: We extended the stream to 5,000 batches and report the measurements in Appendix C.3, Table C.7. With 50 exact records and 10 archives per rank, total history storage is 14.56 GiB at batch 100, 18.44 GiB at batches 1,000 and 2,000, and 18.56 GiB from batch 3,000 through batch 5,000. The Fisher class union also remains at 936 classes from batch 3,000 onward.
Thus, storage grows during early class coverage and then reaches a measured plateau. We have corrected the manuscript claim from “does not grow with stream length” to the more precise statement that storage is bounded by the fixed history budget and finite class-wise Fisher bank. We also explicitly acknowledge that the current 18.56 GiB bound remains large and is not yet optimized for edge deployment.