acceptodds
Under review as a conference paper at ICLR 2027

CROSS-TASK GAINS WITHOUT AUXILIARY LABELS: ARCHITECTURE-PRESERVING ABLATIONS UNDER DISJOINT SUPERVISION

Abstract

When two tasks are labelled on different datasets and no example carries both, coupling them through a shared module often improves the target task, a gain credited to the auxiliary labels. But the module also changes the architecture available to the target task, so comparing coupled with separate heads cannot separate supervision transfer from architectural benefit, and none of the fourteen multi-objective methods and systems we audit reports a control that can. We resolve this attribution problem with architecture-preserving label ablations, which need no new model or data: the coupling module and its readout are kept while the auxiliary labels are removed, the module is frozen at random initialization, or the auxiliary task is dropped with the readout kept. A gain that survives without the labels is architectural by construction, and the observed gain splits exactly into architectural and label parts; an exact single-task tie certifies the split. Our testbed is vision-only deepfake localization, where spatial masks and temporal boundaries come from separate corpora, with frozen features and cross-task attention. The control overturns the default reading. Attention improves official [email protected] at every seed, by at ten-fold budget, yet of the pilot gain survives with patch access but no masks (and most of it on a second encoder and dataset pair). At ten-fold, attention without masks matches or exceeds full attention at every seed on the main encoder, and even with the spatial task dropped and the readout kept, most of the gain remains at both budgets (both claims post hoc). A parameter-matched module with masks but no patch access reaches half of the pilot gain; the tested output-space consistency term, a loss through which mask labels do reach the temporal head, gives no consistent advantage. The gain shrinks with budget, is not reliably retained under fine-tuning, and does not predict better localization of unseen manipulation types. A cross-task gain under disjoint supervision should pass this control before it is credited to labels; we release it with scoring code.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
multi-task learning, auxiliary tasks, negative transfer, disjoint supervision, label ablation, cross-task attention, deepfake localization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9Xh5OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether target-task improvements under disjoint supervision arise from auxiliary labels or changes to the target architecture. It proposes architecture-preserving label ablations that retain the coupling module while removing auxiliary supervision. The testbed combines spatial and temporal deepfake localization using frozen visual features, separate annotation sources, and cross-task attention. An exact single-task baseline tie supports an operational decomposition of the observed gain into architectural and label-dependent components.

Critical strengths
  • Clear attribution framework. Section 3.2 identifies the confounding between auxiliary supervision and architectural changes, then provides an operational decomposition with a testable baseline identity. This makes the proposed control straightforward to interpret and reuse.
  • Complementary mechanism controls. Table 2 and Figure 2 combine label removal, shuffled masks, frozen attention, parameter matching, and a single-task attention readout. Together, these experiments show that substantial gains survive without mask supervision and help distinguish label dependence from feature-access benefits.
Critical weaknesses
  1. The writing has a strong AI-generated tone. For example, the captions of Tables 1 and 2 are very long and overly detailed—typical of AI writing. This paper requires extensive rewriting to improve readability.
  2. The experimental portion is insufficient; there are only two tables and one figure.
  3. I do not fully understand the paper's idea. Do the limitations of existing work, as described in the paper, actually exist? The paper claims that studies of transferability between two tasks must control for shared parameters—i.e., only when the parameters trained for the two tasks are identical can one conclude that true transferability exists between tasks. Existing studies did not control for parameter sharing, so they can only suggest that improved performance on one task may be due to a better model design rather than genuine transferability between tasks. I question whether this limitation actually exists; can the authors provide detailed references to demonstrate that this is indeed a real limitation?
  4. The paper uses only the FaceForensics++ and AV-Deepfake1M datasets for the study; for the hypothesis being proposed, two datasets are insufficient. Multiple algorithms, model architectures, and datasets should be used to validate the hypothesis.
Questions for authors

See Weaknesses

1 comment
AuthorsOpenReview ↗

Response to weaknesses

W1 — AI-generated tone; long captions; rewriting

We take this seriously; the density you describe is ours to fix. We rewrote, simplified, and restructured some of the main text, captions, and charts (with moving some details to the appendix with pointers) as described in the general comment.

W2 — two tables and one figure

The main text carries two tables and two figures (Figure 1 the setup, Figure 2 the per-seed controls) because the nine-page limit put the evidence in the appendix: 58 numbered tables give every per-seed level, paired difference and interval. The revision adds Table 2a to the main text. It gives the label ablation in one row per setting: three frozen encoders, two budgets, two learning rates, two mask-loss weights, a mean-pool readout, the second dataset pair, two fine-tuning recipes and one end-to-end system. The mechanism controls (shuffled masks, frozen attention, parameter matching, a single-task attention readout) are in Table 2 and Figure 2, and the discussion-period runs are in Appendices A.1–A.3.

W3 — does the limitation exist? references

The limitation we claim is narrower than the paraphrase. We do not say that parameters must be identical for transfer to exist. We say: when a coupling module and an auxiliary loss are added together, the gain over separate heads cannot be credited to the auxiliary labels unless the labels are removed with the module kept, because the module alone changes what the target head can compute (Section 3.2, Equation 1). The revised introduction states the question in one sentence: what the auxiliary labels add once their module, data and training budget are held fixed (Section 1).

Published work does make the label reading without that control, and the revised Appendix A.7 quotes where. OmniFD (Liu et al. 2025, Section IV-C) writes that tasks with fine-grained annotations "provide detailed supervision that facilitates knowledge transfer", and UVIF (Section V-C1) reads its image-plus-video row as "showing the benefit of fine-grained image annotations". In each case the labels are added together with their data, head or task.

Table 47 (Appendix A.7) documents, paper by paper from each paper's own ablation tables, that no cited system reports this control under a frozen backbone. OmniFD removes tasks together with their data, module kept. MTPSL (Li et al. 2022) removes the cross-task loss together with its mapping network. NexusFlow (Lin et al. 2025) removes the module with both label partitions kept. Knowledge Assembly (Spinola et al. 2023), named in the revised Section 2, removes one task's labels with its images kept, but under a shared backbone trained end to end, so its contrast is the whole label effect, not the two-term split. The nearest control is Zamir et al. (2020), whose frozen random constraint networks in a loss-side coupling are the analogue of our frozen-module arm.

Whether the limitation matters is what the paper tests. In an OmniFD-style module, the gain a reader would credit to mask labels survives their removal at every seed (86% at pilot; Table 2), which the coupled-versus-separate comparison cannot reveal. We do not claim that the cited systems' gains are architectural, only that their reported comparisons cannot tell (Appendix A.7).

W4 — two datasets; more algorithms, architectures, datasets

The hypothesis is tested in several blocks, each separately: three frozen encoders (OpenCLIP ViT-B/16, DINOv2 ViT-B/14, an ImageNet ResNet-50), two training pairs with FaceForensics++ masks (AV-Deepfake1M segments; TVIL segments, Table 27), two budgets, five seeds at pilot, and learning-rate and mask-loss-weight sweeps. The ordering repeats in each: the label-free arm is above naive at every seed on all three encoders, both budgets and both pairs. Algorithms: feature-space attention, output-space consistency in two formulations, and cross-dataset pseudo-labelling (Appendix A.6), plus one end-to-end system (OmniFD's recipe retrained on our pair, Appendix A.7). Four datasets (TVIL, DF40, OpenForensics, AV-Deepfake1M++) are also used as zero-shot readouts. We agree that this establishes the counterexample for one domain and one coupling family over frozen features, not a law of multi-task learning. The paper claims the former (abstract, introduction, Section 7). A general-vision pair is the natural follow-up and is future work.

Reviewer_p6xqOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether improvements in multi-task learning come from auxiliary supervision or changes in the model architecture. The authors use deepfake localization, where spatial and temporal labels come from separate datasets. They use frozen visual encoders and cross-task attention to connect the two tasks. By removing spatial supervision while keeping the architecture unchanged, they find that most of the temporal localization improvement remains. The results suggest that the additional access to spatial features, rather than auxiliary labels, explains much of the improvement. However, the gains decrease with larger training budgets and do not consistently transfer to unseen manipulations.

Critical strengths

The paper addresses an important question in multi-task learning by examining whether performance gains actually come from auxiliary supervision. The architecture-preserving label ablations provide a useful way to measure the additional contribution of auxiliary labels in a fixed architecture.

Critical weaknesses
  • The paper addresses an important question, but its main contribution is primarily an experimental control rather than a fundamentally new methodology. Prior studies, such as [1] and [2] have already investigated task interactions, information transfer, and the role of feature integration in multi-task learning. It remains unclear what fundamentally new insight or methodological contribution this work offers beyond these existing studies.

  • The study is largely restricted to deepfake localization, with both dataset pairs relying on FaceForensics++ for spatial supervision. Since the paper makes broader claims about multi-task learning under disjoint supervision, it is unclear whether the findings are specific to these datasets and their task relationships. A more comprehensive evaluation across general-purpose vision benchmarks and other domain-specific multi-task settings is needed to establish the generalizability of the conclusions.

  1. Which Tasks Should Be Learned Together in Multi-task Learning? (ICML 2020)
  2. Residual Feature Integration is Sufficient to Prevent Negative Transfer (ICLR 2026)
Questions for authors
  1. Does the limited benefit of spatial supervision persist with a weaker backbone, such as ResNet-18 or ResNet-50, rather than strongly pretrained CLIP and DINOv2 encoders?

  2. Since the backbone is frozen, spatial supervision cannot improve the shared encoder representations. Could the authors extend their fine-tuning experiments to determine whether auxiliary labels become more useful when the encoder is jointly trained?

  3. Could the authors compare cross-task attention with simpler trainable aggregation of the same spatial patch features to determine whether attention itself is necessary for the observed improvement?

  4. Do the conclusions about architectural and supervision gains hold when the models are trained to comparable convergence?

  5. How sensitive are the label-ablation results to learning rates, training schedules, and auxiliary loss weights? Could the authors clarify whether these settings were adequately tuned for both supervised and label-free models?

3 comments
AuthorsOpenReview ↗

Response to weaknesses 1 and 2, and question 2

W1 — novelty against Standley et al. 2020 and REFINE (ICLR 2026)

Both papers are cited in Section 2.1 and are the nearest prior work. The difference is the question and the control, which the revised introduction now states in one sentence (Section 1). Standley et al. (and Fifty et al. 2021) ask which tasks to train together on one dataset, and measure a task's contribution by removing it under shared parameters. REFINE (Xu et al., ICLR 2026) shows that a trainable residual encoder over frozen source features prevents negative transfer, which is an architectural result. Neither removes a second label source's loss while keeping its data and the coupling module that source fed. That is the one comparison that separates supervision transfer from architectural benefit when the two change together.

The new insight is what this comparison shows. In a published coupling family (OmniFD-style cross-task attention), most of the gain one would credit to the auxiliary labels does not need them: 86% of the pilot gain survives without mask labels; at ten-fold the label-free arm is at or above full attention at all three seeds on the main encoder; and 76% and 85% survive without the spatial task at the two budgets (Table 2). This holds on three frozen encoders and two temporal datasets. We audited the fourteen multi-objective methods and systems the paper cites, plus three more, from their own ablation tables (Appendix A.7, Table 47, which lists what each one ablates instead): none reports this control under a frozen backbone, where the split is exact, and the revised abstract, introduction and Section 2 state the claim with that condition. The contribution is a control rather than a new learning method, as contribution 1 says; its value is the result above, which the usual comparison cannot give.

W2 — one domain; FaceForensics++ in both pairs

Agreed. Section 7 and contribution 2 scope the claim to deepfake localization over frozen features, and the revised abstract and introduction say so where the testbed is introduced ("Our testbed is one domain"; "as a counterexample and a reusable control"). Within that domain, several things vary: the temporal dataset (AV-Deepfake1M; TVIL as the second pair, Table 27), the frozen encoder (OpenCLIP ViT-B/16; DINOv2 ViT-B/14; an ImageNet ResNet-50 at pilot budget, five seeds), the budget (pilot; ten-fold), the seeds (five at pilot) and the coupling family (our module; OmniFD's end-to-end recipe, Appendix A.7). The spatial source is FaceForensics++ in both pairs, which Section 7 states as a limitation, naming the one domain and the untested general-vision pair. The natural next test of the control is a general-vision pair with disjoint labels, such as segmentation with depth or detection with pose from different datasets. It cannot be run in the discussion window and we do not promise it. What is reusable is the control itself: the same experiment on any coupling module whose forward pass is unchanged when a loss is removed.

Q2 — auxiliary labels when the encoder is jointly trained

We ran this twice before submission and twice more during the discussion period, all at pilot budget, three seeds, official metric, with recipes fixed before any run: the backbone at a tenth and at a hundredth of the heads' learning rate (Tables 28–30), and, with the one-tenth recipe, a tenth and a hundredth of the temporal training pool (revised Appendix A.3). Labels do not become reliably more useful. The label part (full − label-free) is within 0.01 at every seed at both learning rates (positive at 1/3 seeds each); with a tenth of the pool it is +0.011 ± 0.008 (3/3 positive), and with a hundredth −0.010 ± 0.004 (0/3 positive), so its sign changes across settings. On OmniFD's end-to-end recipe retrained on our pair, where mask gradients reach the whole trunk, removing the mask loss lowers [email protected] at every seed, by 0.0071 ± 0.0077 on average, about 1.4% of its 0.523 (Appendix A.7). As the abstract states, our module's attention gain over naive is itself not reliably retained under fine-tuning (positive at 1/3, 2/3, 1/3 and 1/3 seeds in the four settings). We report direction and uncertainty and conclude nothing more (Section 7).

AuthorsOpenReview ↗

Response to question 1 and 3

Q1 — a weaker backbone (ResNet-18 / ResNet-50)

Run during the discussion period (revised Appendix A.3, a separate backbone block). We retrained the three label-ablation arms on a frozen ImageNet ResNet-50 read at its stride-16 stage (layer3: 1,024 channels, a 14 × 14 grid at 224 × 224, the same grid as ViT-B/16, so only the features change), at pilot budget, official metric, five matched seeds (seeds 4–5 added after the three-seed read, under a rule fixed before they ran). Over seeds 1–5:

  • attention − naive: +0.110 ± 0.008, positive at 5/5 seeds;
  • label-free − naive: +0.113 ± 0.012, positive at 5/5 seeds, so the label-free arm retains 103% of the mean attention gain (100% over seeds 1–3);
  • attention − label-free: −0.003 ± 0.010, positive at 1/5.

So the limited benefit of the spatial labels persists on a weaker encoder: the gain over naive is there at every seed, and removing the labels does not remove it. Across encoders we compare orderings, never levels, since absolute AP is not comparable across feature spaces; the two gains over naive keep their unanimous signs on all three frozen encoders run so far (OpenCLIP, DINOv2, ResNet-50). The control does not depend on the encoder: with any frozen backbone the auxiliary loss reaches the target head only through the coupling module, so Equation 1 stays exact and the same three arms answer the question. ResNet-18 was not run.

Q3 — a simpler trainable aggregation of the same patch features

We ran exactly that arm during the discussion period (revised Appendix A.2). The spatial-to-temporal attention of the s→t-only arm was replaced by the mean of the same projected patch tokens followed by one linear layer, trained with mask labels and with the mask-loss weights set to zero, at three matched seeds, pilot budget, official metric. Both mean-pool arms are above naive at every seed, but they carry only about a quarter of the gain: +0.017 ± 0.006 with labels and +0.016 ± 0.003 without, against +0.070 ± 0.007 for full attention (25% and 23%). The s→t attention readout beats the mean-pool readout at every seed, with and without labels, by 0.029 to 0.048 [email protected]. Removing the labels changes the mean-pool arm by +0.001 ± 0.003 (positive at 2/3 seeds).

So the answer has two parts. First, a plain mean of the same tokens is not enough: the attention-shaped aggregation over the patch tokens carries most of the gain. Second, trained attention weights are not what carries it: the module frozen at random initialization (attention weights never trained; the patch and pooled-feature projections train) retains 67% of the official gain at pilot and 126% at ten-fold, positive at every seed (Table 2). What does not change in any of these arms is the label part: with the mean-pool readout, as with the attention readout, removing the mask labels leaves the gain over naive at every seed.

AuthorsOpenReview ↗

Response to question 4 and 5

Q4 — comparable convergence

Every arm trains for the same batch budget and is selected on the same internal validation readout, so arms are compared at matched budget, not at convergence; the paper says the ten-fold point is unconverged (Section 5.1). Three checks address the question.

First, the conclusion holds at two budgets ten times apart. The gain over naive is smaller at ten-fold (Section 5.1), but in every frozen block of Table 2a, at both budgets, the label-free arm keeps at least 73% of the mean gain (OpenCLIP 86% at pilot and 138% at ten-fold; DINOv2 98% and 73%).

Second, during the discussion period we retrained the naive, attention and label-free arms at seed 1 for 30 epochs instead of 10, everything else unchanged (revised Appendix A.1). Official full-split AP is 0.148 / 0.174 / 0.173, and each arm last improved within its final ten epochs. Attention − naive is +0.026 and label-free − naive +0.025 (+0.033 and +0.034 at 10 epochs); full − label-free is +0.002. This is one seed by design, so we give no sign count or fraction and never rank the 30-epoch rows with the 10-epoch rows.

Third, we re-scored the last-epoch checkpoints of all 53 runs whose selected epoch is not the final one, with the official code (revised A.1). This keeps the signs of 39 of 42 contrasts, weakens two to two seeds of three, and makes one not robust (full − label-free on DINOv2 at ten-fold). Every attention − naive and label-free − naive contrast keeps its sign at every seed in every frozen block.

None of this establishes the asymptote, and Section 5.1 says so; at every point we measured, the ordering is unchanged.

Q5 — sensitivity to learning rate, schedule and loss weights; tuned for both arms?

The label reading holds across a tenfold range of both knobs the question names. We retrained the three arms with the heads' learning rate at × 0.3 and × 3 of the inherited value, and the full attention arm with its mask-loss weight at × 0.3 and × 3 (three seeds, pilot budget, official metric; revised Appendix A.2; all four rows are in Table 2a). For the mask-loss weight, the label-free arm is the weight-0 end of the axis and naive receives no mask gradient, so both are the printed rows.

  • At every setting, attention and the label-free arm are both above naive at every seed.
  • The label-free arm retains 103% and 102% of the mean attention gain at the two learning rates, and 92% and 93% at the two mask-loss weights.
  • Attention − label-free is −0.002 ± 0.003 and −0.001 ± 0.008 at the two learning rates (positive at 1/3 seeds each), and +0.005 ± 0.009 and +0.004 ± 0.014 at the two mask-loss weights (2/3 each; both means below the pre-set 0.015 bar).

The visual subset gives the same sign counts.

No setting was tuned for any arm, so tuning favours neither. Every arm inherits one configuration (constant learning rate, no scheduler, the released loss weights; Appendix A.1), and the guard tests in the released snapshot pin each ablation arm to its parent configuration: the label-free arm differs from full attention only in the two spatial loss weights set to zero. Section 7 states this for the fine-tuning recipes. The schedule and the temporal loss weights were not swept, and adaptive task weighting was not tried (Section 7).

Reviewer_wGsDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper asks whether gains credited to auxiliary labels under disjoint multi-task supervision instead arise from the coupling architecture itself. In a frozen vision-only deepfake localization testbed, it holds the cross-task module/readout fixed while removing mask labels, freezing the module, or removing the spatial task. It examines temporal AP, matched seeds, training budgets, a second frozen encoder/dataset pair, limited fine-tuning and out-of-domain readouts. The contribution is primarily an evaluation methodology and a concrete counterexample to unjustified label-transfer attribution, rather than a new state-of-the-art detector. The paper is an evaluation-methodology study: when architecture and supervision change together, observed downstream improvements cannot automatically be assigned to the auxiliary labels. Its main conclusion is strongest for a frozen-encoder and particular coupling/readout arrangement. The attribution counterexample should be judged on whether controls isolate that claim, not on whether the resulting detector wins the task leaderboard.

Critical strengths
  1. The central control is direct and informative. On the official ten-fold-budget AV-Deepfake1M metric (Table 2), full attention scores 0.171/0.170/0.172 [email protected] across three seeds, whereas attention without mask labels scores 0.171/0.180/0.191, compared with 0.138/0.152/0.149 for the naïve baseline. The result strongly challenges the claim that mask supervision primarily explains the gain in this frozen regime. The fixed-coupling, no-mask-label arm offers a genuine counterfactual: keeping architectural access while removing semantic auxiliary supervision still yields an improvement over the naïve baseline. The seed-level AP values are useful because they show both full and label-free attention outperforming naïve prediction, while any residual label contribution is small and less consistent.

  2. The paper distinguishes a valid algebraic attribution decomposition from claims of information transfer. Parameter-matched attention, frozen-module and same-readout single-task controls (Section 3.2, Table 2) help identify patch access and optimization as competing explanations. The study separates the architectural contrast from the incremental label contrast, preventing one number from answering two causal questions. Same-readout, frozen-module and shuffled-label controls probe whether spatial access or training paths explain the gain. This decomposition can improve evaluation practice even if its exact effect sizes do not transfer to other datasets.

  3. Methodological caution is unusually strong: the paper discloses post-hoc arms, uses the official evaluation harness, separates pilot from ten-fold budgets, warns that video-bootstrap intervals do not capture seed-to-seed training variability, and reports unfavorable transfer and fine-tuning evidence rather than hiding it. Explicit disclosure of post-hoc arms, seed-level signs and the limitation of clip bootstrap increases trust in the methodology. Reporting the weak absolute detector and unfavorable cross-domain orderings is particularly important: these facts make the paper less flashy but more honest about what its negative attribution finding does and does not establish.

Critical weaknesses
  1. The central empirical conclusion is condition-specific. The sharpest controls are in a frozen-backbone, particular coupling/readout architecture with FaceForensics++ mask data; it is not demonstrated that the same partition of gains holds in general multi-task learning or jointly fine-tuned shared backbones. The authors largely acknowledge this; I consider it a limit to scope rather than an unsupported universal claim. With a jointly fine-tuned backbone, auxiliary labels may change representation geometry rather than only the task head. That is a distinct mechanism absent from the main experiment, so the result must not be generalized to claim auxiliary supervision is usually irrelevant. Stating which parameters are frozen and which gradients remain in each ablation would make this limit precise.

  2. Several attribution arms, including the ten-fold label-free and same-readout controls, were run post hoc, and the dataset has only three principal paired training seeds. At pilot budget the sign of full-minus-label-free attention varies by seed. Small residual effects should not be declared zero merely because sample-level intervals are tight; the paper's own optimizer/data-order controls show effects of comparable size (Section 6.1). A clip-level bootstrap resamples observations conditional on already trained weights and cannot measure optimization-seed variation. Since a portion of the experimental menu was chosen post hoc, small differences between full and label-free attention need special caution. A paired five-seed analysis of the residual label contribution would be more credible than interpreting failure to reject a difference as equivalence.

  3. Practical model strength and external transfer are limited. The authors report an internally retrained OmniFD recipe scoring 0.523 [email protected] on the same harness versus roughly 0.171 for the main frozen attention arm (different training/backbone protocols, so not a fair model ranking). The in-domain ordering also fails to reliably carry into TVIL/other zero-shot evaluations. This matters to downstream relevance, while leaving the attribution counterexample intact. The 0.523 OmniFD recipe is not a controlled competitor for the 0.17 frozen-arm setting. Its value is as a reminder that practical transfer to stronger jointly trained systems is unknown. The scientific attribution counterexample may still stand, but downstream readers need to know the method has not demonstrated the same confound across the broad space of modern detectors.

Questions for authors
  1. Give paired full-minus-label-free and label-free-minus-naive values for all available training seeds, not just clip resamples, with sign counts and variability. Please avoid inferring exact equivalence from a confidence interval that includes zero.

  2. Clarify gradient pathways and training data exposure in the frozen-module and shuffled-mask controls. Which parameters still receive a mask-derived gradient, and which comparisons strictly preserve both model capacity and optimization budget?

  3. Identify the assumptions required for the architecture-preserving attribution decomposition and which fail under shared-backbone fine-tuning. If limited fine-tuning arms exist, report their direction and uncertainty without generalizing beyond the tested setting.

3 comments
AuthorsOpenReview ↗

Response to weaknesses

W1 — condition-specific; state which parameters are frozen and which gradients remain We agree, and the paper scopes the claim this way. Contribution 1 says the control applies in every training regime but exactness needs a frozen backbone, and Section 7 opens with the regime: frozen CLIP and DINOv2 ViT-Bs, one head design, FaceForensics++ masks in both pairs. We also agree that a jointly trained backbone adds a mechanism the frozen setting excludes: the mask loss can change the shared features. The one system we trained end to end, OmniFD's recipe retrained on our pair, shows a small effect in the direction that mechanism predicts: removing its mask loss lowers [email protected] at every seed (Table 2a; Appendix A.7). Our fine-tuning arms and their direction are in our answer to Q3. To make the limit precise, the revised glossary (Section 3.2) lists, for each arm, the coupling module, the losses that train it, whether it has patch access, and which parameters the mask loss reaches. The revised introduction names the frozen module as a bound on what needs trained attention weights, not as a label ablation (contribution 1). The per-arm gradient paths, and which comparisons hold capacity and budget fixed, are in our answer to Q2.

W2 — post hoc, three seeds, sign varies at pilot; a paired five-seed analysis; no equivalence Agreed on every point. The paired five-seed analysis is in the submitted PDF (Table 17, unchanged in the revision); seeds 4–5 were run post hoc under a rule fixed before any value was read (Appendix A.2). Over seeds 1–5 on the official full split:

  • attention − naive: +0.065 / +0.067 / +0.078 / +0.064 / +0.078 (5/5; +0.070 ± 0.007)
  • label-free − naive: +0.047 / +0.075 / +0.059 / +0.058 / +0.081 (5/5; +0.064 ± 0.014)
  • full − label-free: +0.017 / −0.008 / +0.019 / +0.006 / −0.003 (3/5; +0.006 ± 0.012) The visual subset gives the same sign counts. What the data support is the ordering. The label-free arm is above naive at every seed in every frozen block (5/5 at pilot, 3/3 at ten-fold, 3/3 on DINOv2 at both budgets, 5/5 on ResNet-50), so most of the gain needs no labels. We do not read the residual label part as zero; Section 7 says small differences do not show zero label benefit. The revision also states the test's resolution, measured during the discussion period (Appendix A.2): a spatial label that transfers by construction (a per-frame mask equal to the frame label, on the target clips) gives with-minus-without +0.013 ± 0.016 (3/3), below the pre-set bar of 0.015. So three seeds cannot resolve a label part of the seed spread's size, and Section 6.1 now says so. The clip bootstrap is reported as conditional on the trained weights (Section 4); the claims rest on per-seed signs and seed spread.

W3 — practical strength and transfer Agreed, and we read it as you do: it matters for downstream relevance while leaving the attribution counterexample intact. The testbed is a controlled probe, not a competitive detector. Table 1's context block reports the retrained OmniFD row on the same harness (0.523 against 0.171) as a level gap that we describe but do not attribute (Appendix A.7). Our claim is an attribution within a family of arms that differ only in coupling and supervision, and contribution 2 states that the in-domain ordering does not carry to the zero-shot readouts. We agree the confound is shown for one coupling family over frozen features, not across modern detectors. The one end-to-end system we trained (OmniFD's recipe) is a context row, not the architecture-preserving split; the revised Section 6.1 says so beside that result, and that no stronger end-to-end detector was tested.

AuthorsOpenReview ↗

Response to questions 1 and 3

Q1 — paired values for all training seeds

Our reply to Weakness 2 gives all five pilot seeds on OpenCLIP. The other frozen blocks (Tables 13 and 15), official full split:

  • Ten-fold OpenCLIP: label-free − naive +0.033553 / +0.027868 / +0.042413 (3/3); full − label-free −0.000059 / −0.009685 / −0.018969 (0/3; −0.00957 ± 0.00946).
  • DINOv2 pilot: label-free − naive +0.070946 / +0.068897 / +0.056618 (3/3; +0.06549 ± 0.00775); full − label-free −0.003268 / +0.000006 / +0.006466 (2/3; +0.00107 ± 0.00495).
  • DINOv2 ten-fold: label-free − naive +0.055763 / +0.024036 / +0.027728 (3/3; +0.03584 ± 0.01735); full − label-free +0.006552 / +0.019117 / +0.013747 (3/3; +0.01314 ± 0.00630).

The revised Table 2a gives, for every setting, the seeds at which each gain is positive, the per-seed retained fractions, and the label part as mean ± sd with its sign count (per-seed levels: Tables 2 and 15). We infer no equivalence from any of these. The paper says "inside the seed spread" and "most of the gain", never "no label effect", and where the label part is consistent (DINOv2 at ten-fold) Section 5.1 says so.

Q3 — assumptions of the decomposition; what fails under fine-tuning; the fine-tuning arms

Three assumptions give Equation 1. (i) The auxiliary loss has no trainable path to the target head other than the coupling module. With a frozen backbone and task-specific projections this means A(N,1) = A(N,0), a testable identity that Section 5.2 verifies to the last digit. (ii) Removing the auxiliary loss leaves the module's forward pass and parameter set unchanged, so A(C,0) is the same architecture. (iii) Arms are compared at matched seed and budget, with seed spread as the uncertainty. The decomposition is operational: it measures what the loss changes here, not information transfer (Section 3.2).

Under shared-backbone fine-tuning, (i) fails: the mask loss moves the backbone, so A(N,1) ≠ A(N,0), and the label part folds in whatever the mask loss did to shared features. (ii) and (iii) still hold, so the control still answers whether the gain needs the labels, read as a paired per-seed contrast with no fraction. Section 6.1 now states A(N,1) ≠ A(N,0) under fine-tuning (Appendix A.3).

The fine-tuning arms, all at pilot budget, three seeds, official full split, recipes fixed before any run and not tuned (direction and uncertainty):

  • Backbone at a tenth of the heads' learning rate (Tables 28–30; arm seed sd about 0.03): attention above naive at 1/3 seeds with the uninterrupted seed-1 rerun (2/3 with the resumed run); the label part (full − label-free) within 0.01 at every seed, positive at 1/3.
  • At a hundredth (same tables): attention above naive at 2/3; label part within 0.01 at every seed, positive at 1/3.
  • A tenth of the temporal training pool (revised A.3): attention above naive at 1/3; label part +0.011 ± 0.008 (3/3 positive).
  • A hundredth of the pool: attention above naive at 1/3; label part −0.010 ± 0.004 (0/3 positive).
  • OmniFD's end-to-end recipe, mask loss removed: −0.0071 ± 0.0077 (lower at 3/3 seeds; Appendix A.7).

Direction: in none of the four fine-tuned settings of our module is the attention gain itself consistent, and the label part is of the seed spread's size and changes sign across settings. The one end-to-end system shows a small, consistent label part. We generalize nothing beyond the paper's statement, in the abstract, contribution 2 and Section 5.1, that the gain is not reliably retained under fine-tuning; Section 7 keeps the per-recipe seed counts.

AuthorsOpenReview ↗

Response to question 2

Q2 — gradient pathways; which comparisons preserve capacity and budget

With the backbone frozen, the mask loss can reach the temporal head only through a parameter both losses update. Per arm, on FaceForensics++ batches:

  • Summed losses (naive): no shared trainable parameter. The mask loss trains only the spatial projection and the mask decoder, so A(N,1) = A(N,0): the temporal-only arm reproduces naive's validation AP to all digits at every seed (Table 2).
  • Full attention: the mask loss reaches the patch projection, the spatial-to-temporal readout and its normalizations (through the temporal-to-spatial direction, which feeds only the mask decoder), and the mask decoder (Section 6.1).
  • Label-free attention: the same network with the two spatial loss weights set to zero. The mask decoder receives zero gradient and the same weight decay; forward pass, parameter count and batch schedule are unchanged.
  • Shuffled masks: every gradient path of full attention. Only the mask targets are permuted within each batch, so the spatial gradient is kept and image–mask correspondence is destroyed (Appendix A.2). Each batch's mask distribution is preserved, so common mask geometry is not ruled out.
  • Frozen module: the attention weights stay at initialization. The patch projection and the pooled-feature projection the temporal head reads stay trainable and still receive mask gradients through the frozen module (Appendix A.2). So it is not an architecture-only arm: trained with mask labels, it bounds what needs a trained module; the same arm without mask labels was not run.
  • Single-task attention readout: spatial-to-temporal attention only, temporal loss only, no mask loss anywhere.
  • Parameter-matched self-attention: full attention's parameter count, attending to own-task features only; no patch access.
  • Stop-gradient, added in the revision (Appendix A.1): full attention with the mask-loss gradient into the patch projection set to an exact zero; the readout, its normalizations and the mask decoder still train on the mask loss. At ten-fold it is above naive at 3/3 seeds (Appendix A.1).

Budget: every arm trains for the same batch budget (1,000 mixed batches × 10 epochs at pilot; 10,000 × 10 at ten-fold) under the same seeded source schedule. Capacity: the label-free, shuffled-mask and stop-gradient arms are parameter-identical to full attention; the frozen module has the same count with fewer parameters trained; parameter-matched self-attention has the same count by construction; the spatial-to-temporal-only arms omit the temporal-to-spatial direction. Data order is matched within an architecture only, because model construction advances the RNG (Appendix A.2); retraining the readout and the label-free arm under matched per-source order shrinks the gap between them at every seed (revised A.2).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the three reviewers. We answer each point under its review; this comment summarizes what changed in the revised PDF.

Readability. We rewrote the main text in plainer language, so the diff touches every page. The abstract is shorter, as are the captions; a glossary names each arm once, and a summary table (Table 2a) gives one row per setting. Detail that moved is in the appendix, where the text cites it.

Framing. In response to the reviews, the introduction now states what the frozen-module comparison does and does not show, and Section 2 names a closely related labels-removed baseline from prior work.

New runs. The appendices add the discussion-period runs (A.1–A.3), each under a rule fixed before it ran. They test the existing results, for example with official re-scoring, longer training, more fine-tuning settings, sensitivity to learning rate and loss weight, a third frozen encoder, another readout, and bootstrap intervals. Each per-reviewer answer points to the runs it uses.

Results and claims. The submitted tables keep their numbers. One count changed: at a tenth of the heads' learning rate, fine-tuned attention beats naive at 1/3 seeds, not 2/3, because the count now uses an uninterrupted rerun of seed 1 (A.3). No other value or sign changed. "Post hoc" flags moved into a glossary column. Three claims are stated more narrowly: the audit, the shuffled-mask reading and the overall verdict.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.